Skip to main content

How to Answer A/B Testing Questions

Premium

A/B testing questions test more than your knowledge of experiment mechanics. They test whether you can make decisions under uncertainty, navigate results that don't give you a clean answer, and hold a defensible position when the interviewer pushes back.

Check out how companies are asking A/B testing questions in their interview loops:

  • Uber: Embedded in a product design question. After designing a gift card redemption flow: "How would you design an A/B test for this?"
  • OpenAI: Framed as a rollout question. Lead the launch of GPT-6 into ChatGPT, requiring you to reason through phased rollout, metric selection, and launch decisions under uncertainty.
  • Capital One: Direct. "How do you A/B test a new feature?"

Notice that none of them test the mechanics of experimentation in isolation. Even the plainest version is really asking how the experimentation drives a decision.

A strong answer does five things: states a clear hypothesis, designs a sound experiment, chooses the right metrics, explains how the results drive a decision, and acknowledges the experiment’s limits.

Step 1: State your hypothesis

A strong hypothesis has two parts: a prediction (what you expect to happen) and a rationale (why you expect it). Ground the rationale in a specific user behavior or source of friction so it holds up under pushback.

Prediction onlyWeak"We believe this change will increase user satisfaction."
Prediction + rationaleStrong"We believe this change will increase user satisfaction, because users are currently abandoning the conversation when responses feel overwhelming, and this change addresses that friction directly."

Step 2: Design the experiment

Cover three parts:

Treatment and control. Say exactly what changes for the treatment group and what the control group sees. Be specific enough that an engineer could build the treatment without needing to ask clarifying questions.

Population. Focus on the users most likely to be affected. Note who you'll exclude, what percentage you'll include, and why.

Product typeGuidance
Global consumer app1-5% of eligible users is typically enough for statistical significance
Regional / smaller-scale appMay need 20-50%+, so run a power calculation first
Dating appsAlways test regionally first since engagement varies sharply by geography
B2B / enterpriseConsider account-level randomization to avoid contamination within a team

Duration and sample size. State how long the test will run and how many users it needs. A common starting point is 2–3 weeks; enough to cover full weekly cycles and get past the novelty effect. Size the test to detect the smallest lift that matters, often 3–5%, at 95% confidence.

Step 3: Choose your metrics

Choose metrics that show whether the experiment worked and detect whether it caused any unintended side effects.

Name three metrics to track your experiment: a primary metric, a tracking metric, and a guardrail metric.

TierDefinitionGuards against
PrimaryThe single number that tells you whether the experiment worked. Must map directly to your hypothesis.Answering a different question than you set out to answer
TrackingLeading indicators that the primary metric is moving for the right reasonsWinning on a metric for the wrong reason
GuardrailSignals that something has gone wrong even if the primary metric is upLocal wins that create downstream problems

For AI features, the standard three-tier framework isn't enough on its own. The core problem is that classic engagement metrics can't distinguish between a user engaging with the correct answer and one with the wrong one. They look identical in behavioral data.

This means you need to add a parallel layer of measurement that tracks whether the AI is actually doing its job well, not just whether users are clicking:

  • Output quality: hallucination rate, task completion rate, and factual accuracy. These almost always require offline evaluation, meaning sampled human review of outputs, running alongside the experiment.
  • Trust: user override rate (how often users edit or dismiss the model's output), escalation to a manual path, and return rate to the same query.
  • Latency and cost: p50/p95 response time, cost per query. AI features often trade speed for quality, and that tradeoff has to be measured explicitly, or you won't see it.

An A/B test shows how users behave, but not whether the model's output is accurate or safe. Pair it with an offline evaluation of sampled outputs to measure quality.

Step 4: Define the decision

Explain how the results would affect your launch decision. Be clear about what would make you launch, hold back, or keep testing and connect that choice to the company’s broader goal.

You should cover three possible outcomes:

Primary MetricGuardrailsAction
UpStableShip. Name your rollout plan and explain why you're starting with that slice.
UpDownDon't hedge. Name the tension, decide which signal matters more in this specific context, and commit to a call with a specific reversal condition.
No effect—Distinguish "wrong hypothesis" from "bad experiment." Low engagement may mean a discoverability problem, not a value problem. Check whether users even saw or used what you tested.

Step 5: Call out blind spots

Every experiment has blind spots. The data can tell you whether a metric moved, but not always whether the change made the product better.

Call out one thing the experiment might miss before the interviewer asks. It could be a qualitative effect, a longer-term behavior change, or another consequence that won't show up in the dashboard.

Common blind spots:

  • Quality vs. quantity. More comments don't always mean better conversations, and more clicks don't always mean users are happier.
  • Trust and satisfaction. A change may look neutral in the data while making users less confident in the product.
  • Cannibalization. A larger button may increase its click-through rate, but what did users stop clicking instead?
  • Novelty. A feature may produce an early spike that fades once it is no longer new.
  • Changes that can't be A/B tested. Large infrastructure or brand decisions may be too broad for a controlled experiment.
Next Lesson