Three Key Learnings from our Experiment Critiques
Imagine you are a product manager in the midst of launching an experiment. You’ve put a lot of effort into planning. However, you have your doubts. Is my hypothesis well thought through? Am I using the right testing method? Am I targeting the right segment of users? Have I chosen the right key metrics to measure? Have I thought through each and every risk?
Now, imagine having a group of peers you can present your work to, get feedback from, and learn if similar experiments have worked in the past.
My colleague Gary recently covered the L&D practices at carwow. Today I wanted to focus on one of those learning techniques: experiment critiques. Experiment critiques help get regular feedback from peers with different perspectives and backgrounds. They also help to continuously improve our product processes, expand our product knowledge, and keep everyone aligned.
What is an experiment critique?
As product managers at carwow, our goal is to keep improving the overall experience of our consumers and partners. Experimentation is the backbone behind achieving this fundamental goal.
An experiment critique is a monthly event at carwow where we discuss new experiment ideas or the results of completed experiments. The objective is to get constructive feedback on the thinking behind the experiment.
How this works:
- Each month two Product Managers volunteer to discuss experiments they are considering or have completed
- Volunteers share the experiment write-up or documentation before the meeting as a pre-read
- Each volunteer gets 30 minutes to present their experiment to the group
While there are no strict guidelines on the structure of the meeting, the idea is to have a healthy debate on the problem statement, hypotheses, and risks. We then discuss any insights and challenges on the experiment.
Three Key Learnings from our Experiment Critiques
Experiment critiques can be truly rewarding, irrespective of the experience or background of the PM. I have personally found them to be an enriching learning experience. Here are my three key learnings from being part of experiment critiques at carwow:
1. A great way to build trust
Imagine presenting your work to your peers from Product, Design and Data. It can be daunting, and will force you to be vulnerable. According to Dr Jeff Polzer, a professor of organisational behaviour at Harvard, vulnerability and trust are deeply intertwined. Experiment critiques create a vulnerability loop and a shared exchange of openness amongst the team. The critiques are treated as a safe place to discuss work and ideas. The objective is to critique the work and not the presenter.
If you’re interested in learning more, do read Jack’s post on building trust and motivation at carwow: “Braving Connection” — how to build trust and team motivation
2. A quick way to leverage group knowledge and experience
The product team at carwow has a wide array of experience both inside and outside carwow. Learnings could cut across product teams, and at times there could be a lot of context that might be missing while designing or running experiments. An experiment critique helps solve this problem.
During the critique, team members often bring up learnings from similar experiments that they were involved with in the past. This has its benefits: problems are viewed from multiple perspectives which helps shape the final solution better. This also ensures key elements that might improve the outcome for our consumers and partners are not missed.
3. Continuous improvement
The critiques are a good opportunity to answer a few fundamental questions. For example, some of the questions that are asked quite often are:
- “If you had to run the experiment all over again, what would you do differently?
- “What consumer or partner problem are you trying to solve?”
- “What are the main takeaways of the experiment?”
Answering these questions during the critique facilitates better retrospection and refinement of the problem statement, which in turn improves the future outcome of the experiment. It also forces us to become better in future experiments that we conduct while staying true to our product principles and processes.
Experiment critiques are one of the most interesting sessions that we conduct as a Product team at carwow. Not only are they an exercise in improving critical thinking, but also help develop better product sense by leveraging the learnings of the team, and in turn build trust and openness.
These meetings are a clear and easy way to elevate product capabilities, and the value from attending them is 10x the time they take. I would love to hear if you have found experiment critiques useful at your organisation, or if you have any feedback to share.
Thanks for reading! If you enjoyed this blog post, and want to join the carwow team, why not take a look at our jobs page?
