The Black Box "Problem"
Our understanding of how AI exactly works is limited. But the so-called black box problem isn't quite the fundamental issue it's often made out to be.
“The black box problem isn't just about our lack of understanding; it's about our need for transparency in the AI-driven world.” I have heard this sentiment, expressed in various forms throughout recent years - in the media, at conferences, on panels, and in discussions.
At its core, the black box problem of AI highlights the reality that we don't fully grasp how most AI models manage to do what they do. Yes, we have weights and biases, layers, encoders, and decoders – structures that we've constructed. However, the increasingly powerful capabilities of AI models are mostly emergent phenomena. We can observe their outcomes, but the exact process remains a mystery. Our understanding falls short when it comes to explaining the inner workings of these increasingly vast, multi-billion parameter models.
Engineering, meet science
From an engineering perspective, this situation is quite unusual. Engineers build things based on their understanding of how something works. When we build cars, for instance, we know exactly how they work - and that is precisely why we build them the way we do.
This scenario has remained consistent throughout the history of engineering. We've never constructed rockets by simply optimizing for flight capabilities, only to end up with an incredibly complex machine that flies, but nobody really knows why.
In engineering, we construct because we understand.
Scientists, on the other hand, largely work in the opposite direction. We don't construct - we dissect to understand. Nature has already built everything; our job is to figure out how it works.
Of course, the two disciplines often work together. For example, scientists may discover how a certain biological process works, and bioengineers might use this new understanding to build a new product. Still, we will understand how the new product works - it's built based on our new understanding.
The fact that we don't fully comprehend how these incredibly complex AI models function is disconcerting to many. We're simply not accustomed to having built something we don't fully understand. Indeed, many scientists are now trying to decode how these models work, and I'm sure we'll make substantial progress in the coming years. But before we get too enthusiastic, remember that we still don't fully understand how the brain functions.
So, is this a problem?
How does each neuron contribute to human thought?
My view is that the black box problem is an intriguing scientific challenge, in the sense I alluded to earlier. However, it's not a fundamental issue that should prevent us from utilizing AI systems.
The lack of a complete understanding of how a system operates should not hinder us from using that system. This is true on both individual and collective levels.
On an individual level, I don't fully comprehend how a car functions. I understand the concept of small explosions driving pistons in an engine, which in turn rotate the wheels, but that's the extent of my understanding. The same applies to computers. I have a basic knowledge of my laptop's architecture, but my hardware expertise is limited. These gaps in my understanding don't prevent me from driving a car or using a computer.
Of course, some people have an intricate understanding of how cars and computers operate. Indeed, those who build and design these technologies possess a deep comprehension of their workings. Even though it might be increasingly challenging for a single individual to understand everything, there are systems in place to compile and organize this knowledge collectively.
On the collective level, it’s instructive to look at medications. In many instances, we don't - meaning none of us - fully understand how certain drugs function. This often surprises people, but it's true. We generally understand the principle behind their functioning, but our knowledge is quite limited. Take aspirin, for instance. This drug has been on the market for over a century. Even so, in 2023, researchers continue to discover new insights about how and why aspirin works. We established that aspirin works a long time ago - by today's standards, that means it is more effective than a placebo and is reasonably safe. However, our understanding of precisely how it works remains somewhat limited.
In many ways, drugs are often a black box. But that does not prevent us from using them. Instead, we’ve developed processes to ensure that drugs are only allowed on the market if they can be shown, through clinical trials, that they’re safe and effective. I assume we will develop similar processes for AI.
Machines are faster to correct than humans
This brings me to my final point. It's more crucial to prove that something works than to understand how it works. If you give me a potentially lethal drug, my prime concern isn't about how exactly it would kill me. I'm more interested in ensuring it doesn't kill me in the first place. Sure, there's great value in knowing why the drug could be dangerous. We could leverage this knowledge to avoid similar issues with other drugs. But my initial concern is about whether the drug works, meaning whether it's safe and effective.
This is how we should handle AI systems. We shouldn't demand explainability right out of the box (no pun intended), but rather, we should want to test the systems to see if they do what we want them to do, without causing any unwanted side effects, like creating biases. As these systems are increasingly multi-purpose, they need to be tested all the time, and continuously at that.
The strength of this continuous, systematic, and thorough testing is that it shines a light on any faults these systems might have. If biases are uncovered, we can adjust the systems. This is the beauty of this technology - it can be adjusted relatively quickly. Fixing biases in humans, on the other hand, is much harder and usually takes many generations.
That's why I'm extremely optimistic about using such systems in the public sector. But this optimism rests on the assumption that there's appropriate regulation in place. Such regulation should mandate continuous, systematic, and thorough testing, ensuring the AI system does precisely what we intend. With this approach, understanding the intricate inner workings of these systems afterwards becomes a valuable added bonus. But it’s not our most pressing concern.
CODA
This is a newsletter with two subscription types. You can learn why here.
To stay in touch, here are other ways to find me:
Writing: I write another Substack on digital developments in health, called Digital Epidemiology.