Episode 35: Dear Melissa - Answering Questions About Experimentation
In this Dear Melissa segment, Melissa dives into the world of experimentation, answering subscribers’ questions about metrics and signals for internal applications, measuring the success of company transformations, and the best way to track experiments.
Have a product question for Melissa? Submit one here and Melissa may answer it in a future episode.
Subscribe via your favorite platform:
Q: Do you have any recommendations for metrics signals for internal applications, particularly where we are trying to change behaviors over the long term? How can an organization measure the success of a product-led company transformation? [2:05]
A: When I think about internal applications, I immediately go to work flows and behavior change. Internal applications are usually there to help people do their jobs better, run the company, things like that. So how do you measure the success of someone doing their job? Can your application help them do it faster, better, and cheaper? I'd start there, then I would think about the elements in your flow that contribute to that… Another internal application could be giving people information or data they didn't have before to make better quality decisions. Then, you want to measure the success of the decisions and how they're using that data. [3:09]
I [measure the success of a product-led company transformation] by looking at behavior change throughout the organization, so I want to see that there is a product strategy tied to the company strategy and that you're talking about it and visiting it constantly… We talk about product transformations a lot, but why do it? You're doing it so you can scale technology products to enable your company to grow and meet customer demands and needs in a better way. [6:41]
Q: How do you track experiments and the results in a central way? [8:44]
A: I see this a lot in companies that are trying experiments for the first time but they haven't really embraced a true learning culture of actually taking those learnings and acting on them… You may want to invest in some kind of user or market research platform that allows you to get in touch with users in different ways. [8:59]
Q: How do I set a timeframe for measuring success and performance of a product before I pivot or iterate? How do I know when to kill a product, if after a couple more tries the initial idea didn't work? [11:36]
A: I think what it comes down to is if you’re testing on the right variable that's going to drive change. What I've found is that if you're experimenting and it's not working, it’s because you don't understand the problem well… You should get feedback from users and talk to them constantly. A lot of times we launch experiments out in the world, never keep tabs on them, and then we’re like ‘What do we change? What went wrong?’ [11:56]
Resources