Jobs search

Senior Experimentation Engineer

Confidential • Sonoma, CA • 1d ago

Key Responsibilities

Design and build the core experimentation platform infrastructure: experiment assignment service, randomization and traffic splitting, multi-layer experiment conflict detection, experiment configuration management (low-code, =30 min setup), and full lifecycle tooling (creation, monitoring, graduation, rollback) — supporting App, Web, and backend surfaces simultaneously
Build automated metric pipelines that connect experiment assignments to full-funnel business outcomes — Signup CVR, eFTD CVR, eFTT CVR, deposit amount, trading volume — with sub-day latency; solve the cross-device identity bridging problem (device ID to user ID across the registration boundary) and implement "time-to-convert" metrics (e.g., eFTD within N minutes of page exposure) that are currently unavailable
Implement advanced statistical methods: CUPED and stratified variance reduction to improve experiment sensitivity without increasing sample size, sequential testing for early stopping (mSPRT / always-valid inference), network interference correction for referral and social experiments, and pre-experiment SRM (Sample Ratio Mismatch) checks; design and execute conversion lift studies to quantify the causal impact of product changes on business metrics
Build self-serve experiment creation and analysis tooling for product managers, growth marketers, and data scientists — including experiment design wizards, power calculators, automated significance reporting, and decision support dashboards; reduce the experiment launch cycle from "requires 2 days of engineering" to "self-serve in under 30 minutes" for all three platforms (App, Web, backend)
Establish experiment quality standards: event instrumentation requirements, guardrail metric monitoring, automated anomaly detection to prevent shipping regressions, and an experiment knowledge base that documents winning patterns, failed hypotheses, and domain-specific learnings — ensuring teams learn from each other rather than rediscovering the same findings
Partner with Growth Product, Personalization (????), TradeGPT, ByX Community, and Asia-Pacific data engineering teams to design and analyze high-impact experiments across personalization, campaign optimization, user journey, push notifications, community feed, and AI product features; serve as the internal authority on causal inference and experiment design across all business units
Build the experimentation platform as the feedback engine for Bybit's AI strategy: design automated attribution systems and causal inference pipelines that deliver real-time feedback signals to AI models (TradeGPT response ranking, personalization algorithms, push notification optimization) — enabling AI models to self-iterate based on causal experiment results rather than correlation-only metrics; build experiment bloodline tracking that traces how each AI model version performs across user segments, and end-to-end observability for recommendation and growth systems that accelerates the iteration cycle from research to production deployment
Define engineering standards, conduct design reviews, and mentor junior engineers; drive cross-team adoption of experimentation best practices as the US team grows

Major Requirements

5+ years of industry experience in experimentation engineering, data engineering, or growth engineering at a consumer-scale internet company
Proven track record building and operating large-scale A/B testing infrastructure: experiment assignment, metric pipelines, statistical analysis, and self-serve tooling serving hundreds of experiments simultaneously
Deep expertise in causal inference and experimental statistics: hypothesis testing, power analysis, CUPED/variance reduction, sequential testing (mSPRT, always-valid inference), network effects and interference correction, conversion lift studies, and treatment effect estimation; ability to apply statistical test theories to optimize user experience and validate business decisions
Strong proficiency in Python and SQL; hands-on experience with real-time data processing frameworks — Flink Streaming or Spark Structured Streaming — as well as data warehouses (Hive, ClickHouse, or equivalent), real-time messaging (Kafka), and cross-device identity resolution (device ID to user ID mapping across the registration boundary)
Experience building data products and self-serve analytics tooling; strong sense for developer experience and product design for internal platforms; ability to reduce experiment launch cycles from days to minutes through thoughtful platform design
Strong bilingual communication skills in both English and Mandarin Chinese; ability to collaborate effectively with Asia-Pacific engineering and product teams in Mandarin, bridging the US R&D Center with Asia-Pacific teams. This role involves cross-timezone collaboration with teams in Singapore, Dubai, and other Asia-Pacific locations (UTC+8 to UTC+4); candidates may occasionally have important cross-timezone meetings in the early morning or evening
Nice-to-have: experience in fintech/crypto with understanding of financial user behavior and conversion funnels; experience with causal ML methods (DML, IV, synthetic control, uplift modeling) for observational analysis; experience with intelligent marketing or subsidy optimization experiments; familiarity with experiment platforms at Meta (PlanOut/Ax), Netflix, Airbnb (Experimentation Platform), or LinkedIn (XLNT); publications at top venues (KDD, NeurIPS, WWW, SIGIR, WSDM, CIKM, ICLR, ICML, or statistics journals); prior founding team or early-stage R&D center experience