Better approach to measure impact and important input metrics
Two different perspectives to measure analytics impact, and how Amazon use metrics to operate.
Today is the second edition of Data and Decision! So far, I have enjoyed creating this newsletter and now have a medium I can use to pour my thoughts. I also hope the readers get inspired by this email and have a productive weekend.
Now, let’s get down to business, shall we?
2 Insights
Analytics. Credit: Unsplash
A better way to measure analyst impact - Time to decision vs. Inspiration per hour
I’m working as a data analyst right now, and these few weeks, I realized that I played a different game than my previous role. As a data team, my job now isn’t to make a decision (like in my previous role), but to help my stakeholder make well-informed actions. So my impact, unlike my last position, is somewhat blurry now.
To speak more broadly, it’s also quite hard to measure the impact of analytics in general. It’s so subtle (we know it’s important) but even the most data-driven person that you’ve met has a hard time coming up with a number.
That’s why I found these articles from Benn Stancil (co-founder of Mode) and Cassie Kozyrkov (Chief Decision Scientist of Google) interesting. Both pieces try to answer the main question above, “How we can measure analytics impact?” from two different perspectives.
To visualize it, let’s create a structure for it first (sorry, a force of habit:P).
Let’s start with Benn’s first.
Benn’s perspective: focus on creating more decisions per hour. Blue square.
Then, let me put Cassie’s below.
Cassie’s perspective: focus on generating more Inspiration. Green square.
You can see the path the two of them try to aim at. Benn (the blue square) tries to optimize the “Average Decision per Working Hour” while Cassie (the green square) focuses on “Average Inspiration per Hour”.
Side note: I believe I need to come up with a better visualization technique for this hmm.
I prefer the Cassie approach because i t’s something I can act upon. It’s more concrete than the term “decision” from Benn’s perspective. On a side note, the exciting gap I found is still at what kind of proxy we can use to measure “Average Output per Decision” (purple square) which is essential but somehow overlooked by most of the data practitioners I know.
Track and manage your input - Controllable input metric
Working Backwards by Colin Bryar and Bill Carr is one of the best books I have ever read about Amazon. It’s quite a short book but very dense with insight. It’s also the source where I got a unique metric I mentioned above.
Controllable input metrics, as the name says, can be controlled directly by the company. The controllable input metric that Amazon has, for example, are includes how much stock is left for specific products or pricing schemes. It’s different from output metrics - things like orders, revenue, and profit- that are also quite important, but you can’t directly manipulate them.
The writers argue that so many companies waste their time keeping track of the output metrics without really knowing how the input metrics correlate with them. For me, it’s a refreshing concept that offers more clarity to our actions.
1 Big Question
What is the hardest question in your field?
I found the question from Shane Parrish’s tweet (sadly, I forgot the source). I remember staring at my laptop for the next 30 minutes while trying to answer it. It forces me to reflect that while I already have all my career’s age in this field, I still don’t know the most challenging question in my area. I now have a few, but I believe I could share more about them in the next post.
1 Quote
“As we do new things, we accept that we may be misunderstood for long periods.”- Colin Bryar from Working Backwards