The \"Where's Eric Dale?\" Problem

Can AI answer “Who did this analysis and can we trust them?”. Oh and short update

Hello there! Woah, how long has it been? But yep, I decided to open this newsletter and will start to write consistently again.

There’s no better topic for a “comeback post” than talking about AI. I have so many things to catch up on, but fortunately, I kept my laundry list of “observations” around AI that I have gathered over the last 1.5 years. Please bear in mind that this is a (very) short list and only specifically pertains to the data profession — I have no capability or interest to think outside of my domain of expertise.

• ChatGPT has brought the “AI” discussion back to life and introduced “LLM” and “GPT” terms to a broader audience.
• This LLM thing, to quote Bill Gurley, is really more like an “advanced Wikipedia that can also code, (and) it has sucked in the world’s information”.
• One of the killer use cases of LLM is becoming a human-to-machine translator. Just give your prompt if you want to create something (ranging from a single-page application to a complete app) and GPT will provide the code for you.
• Well, if this thing can translate your average friend’s “I have an idea, bro” prompt into a working app, why can’t we deploy it also to the analytics field? After all, what’s the difference between your friend’s prompt to the stakeholders’s “Can’t you pull the revenue data and break it down by user cohort?”
• Cue the music. Hundreds of startups are trying this thing (I think it’s called a human-to-SQL translator?). Some experts are excited. Some experts disagree with it.

That “Cue the music” part is something that I want to talk about today.

But as you may guess, I have a dog in this fight. As a data practitioner, I cannot help but think of the amount of joy I would get if any data request could be handled by ChatGPT. I could focus on more important stuff like doing advanced statistical analysis or complex data gathering. And I’m not concerned, well at least not right now, about the chance of my work being automated. On the contrary, these new tools might help me and my friends deliver more output, which will translate to better productivity for the company.

No, no. What I’m afraid of is that most practitioners in data overlook the nature of the analytics work we usually do in our daily jobs, and AI might not be suitable for that. This is what I call the “Finding Eric Dale” problem.

Finding Eric Dale (well, this is actually Stanley Tucci but you get the point)

Finding Eric Dale

Well, most of you might already know, but this “Eric Dale” comes from a character in the movie “Margin Call”. It’s a great movie, and you folks who love data and finance should watch it. Wikipedia has a great description of this movie: “The principal story takes place over a 24-hour period at a large Wall Street investment bank during the initial stages of the 2007–2008 financial crisis. It focuses on the actions taken by a group of employees during the subsequent financial collapse”

That leads us to Eric Dale. His role, by the way, is quite short. Eric is the Head of Risk Management. He’s the guy who knows “Okay, things are going to get ugly for this company” and then reports it to the higher-ups of the firm — which ends badly for him. Several months after his warning, he gets fired from his job and is told to leave the company. His incomplete analysis, the one that says “DANGER!” all over it, is passed to his subordinate, Peter. This Peter character then completes Eric’s calculation, proves that he’s right about the danger, and warns the higher-ups. This time, they finally listen.

Now, it’s easy to chalk this topic up to ethics, banking regulation, or just another argument about why limitless capitalism is bad. I’m in no way capable of doing that — and not particularly interested in doing that either.

What I found fascinating is how Eric’s name, despite already being gone from the company, is kept being mentioned. I can count at least four times that his name is mentioned in high-stakes conversations — and he’s not sitting in those meetings.

Let’s look at my summary above. I simplistically mentioned that the exec “finally listened” here. It’s more complex than that.

Well, when Peter completes Eric’s calculation, he shares that analysis with their (Eric and Peter’s) boss. His boss, forgetting that the company had already booted off Eric, says “You sure about the analysis? Where’s Eric Dale?”. This is the 1st time Eric’s name is mentioned (video here)

Now, Eric and Peter’s boss need to mention this to his boss. Reading the analysis, surprised by the findings, this other boss mentions “Is this thing right? Where’s Eric Dale?”. There you go, two.

This boss’s boss also mentions it to his boss. Eric Dale’s name is mentioned again. The same words, asking if the analysis is correct, and where’s Eric Dale. Three. And the final meeting with the CEO is the one where they agree to take drastic action based on that analysis. The CEO briefly mentions to the audience, again, “Where’s Eric Dale?”. Four.

Four times. His name keeps being mentioned. Without his presence.

The Socio-Technical Aspect of Analytics, or “Who Did This and Can I Trust Them?” Question

This is all fun stuff. But what exactly am I trying to say?

I’m arguing that replacing analysts with AI might not be possible, and it’s not because of purely technical issues (LLM is prone to hallucination, so translating human language to SQL couldn’t get 100% accuracy), but because of a subtler aspect of the socio-technical. Or in simpler terms: AI can’t deal with “Who made this analysis and can we trust them?”

My favorite writer, Cedric, has a good way of characterizing this phenomenon.

I would imagine that Eric’s analysis, the one used as a basis for the drastic action the company will take, can be replicated if you know the correct prompt in the current LLM (GPT-4, Anthropic, you name it). My favorite writer, Ethan Mollick, demonstrated that here. But unlike Ethan’s case, most of the time our analysis will be reviewed by decision-makers, and they need to check if this analysis is sound.

They have two ways to do that.

One straightforward way to solve this is if the decision-maker also has the correct understanding of the statistical (“You sure logistic regression is the correct way to model this?”) or technical method (“You used the ‘platform_fee_v2’ column for the gross revenue, right?”). They can be the authenticator themselves.

However, decision-makers cannot check the analysis forever. So, they use a shortcut for it: they ask, “ Who made this analysis?”

The “who made this” question is seemingly simple but really really important to how we operate as people. To stretch this point a bit too far, you wouldn’t trust that easily if an analyst from your competitor mentioned “There’s this potential and you folks should jump on it,” no matter how sound the analysis seemingly is. You wouldn’t have the same suspicion for your company’s analyst — who has skin in the game and have every incentive to not screw things up.

I bet this level of detail is also the same reason why, for example, a job that is seemingly easy to automate is proven to still exist and grow (take the Radiologist case, for example).

Epilogue

I opened this essay with how many startups try to automate away the analyst with LLM, banking on its ability to translate human language into technical stuff like code or SQL queries. I mentioned how this might not happen, precisely because of the “Finding Eric Dale” problem, to which I proposed that the socio-technical aspect of analytical work might be the biggest barrier that we have to achieve perfect “Human language to SQL query” translation.

Interestingly, this whole thread reminds me of the “Believability” concept that Ray Dalio mentioned in his book. The similarity between these two arguments is “One way to judge someone’s analysis (or in the believability case, advice) is to look at their past record in a certain domain and gauge it”. But, it’s still quite different — believability is used to gauge “Is this person truly competent and should I act on his/her advice”. The Eric Dale problem is, uh, a lesser problem than this.

I cannot also help but think how this is all related to Ronald Coase’s famous “The Nature of the Firm” essay. The question that Coase tries to answer is “Why and under what conditions should we expect firms to emerge?”, articulating how while it seems cheaper to just hire outside contractors instead of recruiting employees, the firm also wants to reduce other costs that might not be captured easily, such as “search and information costs, bargaining costs, keeping trade secrets, and policing and enforcement costs”. Analysts are probably one of the most important roles that contribute to “search and information” costs, so could we assume that firms will have every incentive to hire in-house analysts to reduce this cost?”

But nope. This post is already long enough.

Short Update

I received a promotion to become Senior Data Analyst last October (yea!). Unfortunately, that same month my company faced a strong challenge internally — which made me feel like it’s not ethically sound to keep my work here while I can focus more on solving the problem.

Nonetheless, I think I will try to re-ignite my writing again in the coming months. You can expect a new edition once every two weeks now. Oh, and I’ll make the post also shorter but denser — multiple links, short summary, and my thoughts on it.

See you later!