Skip to main content

Making & Defending Decisions

Premium

Most analytical interviews end the same way. You've diagnosed a root cause or interpreted an A/B result, and the interviewer asks: "So what do you do?"

Clear cases: Your primary metric is up, guardrails are stable, and there are no obvious second-order risks. Or your root cause analysis (RCA) points unambiguously to a measurement artifact with a defined fix. Call it directly and move on. Layering unnecessary hedging onto a clear result reads as uncertainty, not rigor.

Genuine tension cases: These are the scenarios that require more care. You're in genuine tension when:

  • A primary metric is up, but a guardrail is meaningfully down
  • The data support two different hypotheses that imply different actions
  • Two product directions both have real merit and real cost, and the team is genuinely split

The rest of this lesson is about these genuine tension cases.

In these cases, a strong answer chooses a direction, explains why the benefits outweigh the costs, and names the evidence that would make you change course.

① Name the Tension

② Compare Cost of
Being Wrong

③ Commit with a
Reversal Condition

We'll walk through each step using one running example, based on a question pattern from Meta's analytical interviews:

You've been tracking your notification feature for six weeks. Notification engagement is up across all users, all geographies, all platforms. But time-on-site is flat or declining. Your team wants to keep pushing the feature. What do you do?

Step 1: Name the tension

Start by explaining what pulls you in each direction. Present the strongest case for both options before choosing one; otherwise, you risk making the decision sound easier than it is.

Step 2: Compare cost of being wrong

Step 1 established why both directions are worth considering. When both offer credible benefits, comparing the upside again may not resolve the decision. What often separates them is the downside of being wrong and how easily that mistake can be reversed.

For each direction, ask:

  • If I choose direction A and I'm wrong, what does that cost?
  • If I choose direction B and I'm wrong, what does that cost?

Don't assume the same action carries the same cost in every product. The stakes depend on the users affected, the scale of the decision, and how easily it can be reversed.

Take a common choice: ship or wait. The actions are the same across products, but the stakes can be very different:

  • Classic software: Shipping a poor UI or content change may hurt a metric temporarily, but the team can usually roll it back.
  • AI products: Shipping a model that takes incorrect actions may damage trust or create safety risks that persist after rollback.

This doesn't mean defaulting to the safer option. A direction with greater upside may still be worth the risk. It means that a larger, harder-to-reverse downside should require stronger evidence.

Step 3: Commit with a specific reversal condition

Finish with three things:

  1. Your decision
  2. Why you chose it
  3. What evidence would make you reverse it

A reversal condition should name the metric, threshold, and time window that would make you change course. “I'll monitor closely” isn't specific enough.

AI product nuances

AI products introduce a few considerations that don't show up as strongly in traditional software decisions:

  • Engagement doesn't guarantee quality. Users may engage with outputs that are inaccurate or unsafe. Pair behavioral metrics with evaluations of output quality, accuracy, and safety.
  • Set safety stop conditions before launch. Decide which metric, threshold, and response would trigger a pause before exposing users. For example: "If the high-severity incident rate exceeds X in the first cohort, pause the launch regardless of engagement."
  • The company's product promise changes the cost of being wrong (Step 2). The same failure can carry a very different cost depending on what users trust the company to protect.
Next Lesson