Product execution questions are one of the most popular questions you’ll encounter during PM interviews, including interviews for Facebook. These questions aim to evaluate how you measure success, and distill user problems into a metric that a team can rally around. These questions appear easy at first, but in reality, are challenging because the question isn’t about naming metrics - it’s about first taking an educated guess at the north star goal for the team and quantifying it with a metric that expresses it succinctly.
There are three main execution question types:
- Success: How would you measure success for X product?
- Debug: You’re the PM for X. Y metric dropped by Z%. How would you figure out what happened?
- Trade-off: Y metric went up but Z metric went down. What do you do?
These Success, Debug, and Tradeoff questions closely mirror the critical decisions PMs are accountable for on a regular basis. For this memo, we’ll focus on defining success metrics and cover the others in future memos.
Here are three things interviewers are looking for that you can focus on to improve your interview performance:
1. It’s not a math problem, it’s a social science problem.
It’s easy to fall into a trap and treat it like a math problem to be solved. A candidate might believe there is a correct metric the interviewer is looking for, and try listing every single aspect of the problem, leading to rabbit holes. Your metrics will differ, however, depending on the goals you define and the assumptions you make. The interviewer is looking for your ability to define a reasonable north star goal for a product, and suggest a metric that is aligned with the goal.
Example: Once, I was defining success metrics for a language learning app in an interview, and I suggested weekly active users (WAUs) as our goal metric. The interviewer pushed back on this choice.
- Interviewer’s response: Why weekly, and not daily? If you’re trying to learn a new language, you would practice daily.
- My response: Using a language learning app daily would be ambitious for your average user, especially when competing with apps such as Instagram and TikTok, and that daily wouldn’t capture the fluctuations in people who use it over the course of a week — for example, people who only have time to learn a new language on weekends.
My interviewer agreed with my justification. I could have, however, easily gone the other route, and responded with:
Actually, you’re right. Our goal should be to make language learning a daily habit, so I’m going to change the measurement cadence from weekly to daily and measure daily active users (DAUs) instead.
This is why it’s important to be really clear up front about what the product goals are, and the assumptions you are making. Here’s an example on how to do that:
Example: Define success for a language learning app.
2. The interviewer is not looking for a long list of metrics, but how you form an opinion on the few metrics that matter most.
When interviewees are asked to say, Define success for Bumble (an online matchmaking app), candidates will often list 10+ metrics, worried about covering all aspects of the problem. In this case, however, less is more. It’s more interesting to have a discussion on 5-6 metrics than 20-30 metrics. Then, it’s best to end with an opinion on what the one primary metric should be, and 2-3 secondary metrics that aid the primary metric in showing the full picture.
Here’s how to do that in action:
Example: Define success for Bumble.
* 1: Use percentiles, which are not skewed like averages are. If some of your users have very fast response times and others have very slow response times, an average response time would not be representative of your users' experience from the data. Read more on our previous memo: The Allure and Trap of % Goals.
3. Define the downsides of metrics.
While an OK candidate might list general downsides to their metrics, and talk about counter metrics such as cannibalization of the company’s other products or abusive behavior, the best counter metrics are specific to the north star metric you’ve defined. It’s about asking: If we were to over-optimize for our north star, what are ways our decision-making would be flawed?
Example: Venmo is launching a debit card. Define success.
Questions are often intentionally vague!
Metrics questions are hard because you can take them in so many directions, and interviewers are often intentionally vague. They are seeking to understand: does this candidate dive into coming up with metrics (OK interviewee) or do they take the time to define the landscape and the goals (great interviewee)?
It’s less important that you cover every single aspect, and more important to define all the pieces you need to solve the problem. As a product manager, most of the time you’ll have enough resources and people to help you, so it’s less about correctness, than about being able to frame what is success to your team in a way that enables them to frame decisions later on -- and this involves walking through how you think about success.
To read more on how to ace PM interviews and be at the top of your game as a PM, see more of Alexis and Adrienne’s memos here.