AI as an Existential Opportunity

The claim that AI represents an existential threat that could wipe out humanity is staggering and without basis in reality. If history is a guide, AI is an existential opportunity.

Imagine an urn brimming full of balls, where each ball symbolizes a human invention. Humanity is in the continuous process of drawing balls from this urn. Predominantly, these balls are glowing brightly - inventions that have served us well, like medicine or the printing press. Some, however, are just dimly lit, which means they have a mixed impact - partly beneficial, partly detrimental. Occasionally, we stumble upon those that are relatively dark, which typically means a lot of trouble for humanity.

Most importantly, among the balls in the urn are a few so pitch dark that they carry a profound existential risk to our species.

This is the urn analogy proposed by the philosopher Nick Bostrom. He asks us to consider that if we pull out a profoundly dark ball, we will not be able to put it back into the urn. As a consequence, our world is fundamentally vulnerable to humans creating their own demise through invention.

Bostrom calls this the vulnerable word hypothesis (VWH):

“If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition.”

Nick Bostrom

The “semi-anarchic default condition” has three characteristics:

• no global governance

• limited capacity for predictive policing

• actors who have an interest to destroy civilization even at great cost to themselves.

In other words, it’s the world we currently live in.

‘Tis the season of doom

Bostrom’s arguments are clearly laid out, and make for depressing reading. They are the basis of much of the AI doomerism that is popular these days.

It should go without saying that the existence of the extremely dark ball - the one that make the extinction of humans very likely - is pure speculation. Because it’s a speculation that can’t be falsified, it’s hard to argue against, and impossible to counter with science. That doesn’t make it any less relevant. Religious beliefs can’t be falsified either, and yet they have shaped much of humanity’s history. So we must take these speculations seriously.

Indeed, with the rise of powerful large language models, panic has griped many people, including some that were themselves deeply involved in creating the technology in the first place. Many seem to experience a kind of “what have I done” moment, similar to Oppenheimer’s guilty feeling of having blood on his hands.

Their reasoning mostly goes as follows. AI is developing much more rapidly than anyone thought. It’s a technology that can improve itself, further accelerating the advancement. Very soon, in a few years, we’ll have AI that will surpass human intelligence (so called Artificial General Intelligence, or AGI).

Up to this point, it’s hard to disagree. I have yet to meet anyone who hasn’t been impressed and surprised by the extremely rapid development in this space. Most people now think that it is indeed just a question of a few years for AGI to emerge.

However, when we look beyond AGI, we venture into the realm of fantasy. The doom argument posits that once we encounter an intelligence vastly superior to ours, the end of humanity is imminent. Humans have dominated the world by being the most intelligent beings - if AI surpasses us in intelligence, it will inevitably dominate the world. In the same way that we have driven many species to extinction because their goals were not aligned with ours, AI will do the same to us. The crux is that there seems to be no escape, because the smarter entity always wins.

This is the existential threat scenario, where AGI inevitably means that humanity is at the risk of extinction.

A staggering claim

Fears of human extinction are not new.

Looking back a few decades, each period had its own ideas about the existential threats they faced - always represented by the latest technological breakthrough of the time. In the mid 20th century, this threat was nuclear power, and its horrifying potential for destruction. In the latter part of the century, genetic entering was perceived as playing God, setting us on a path of biological destruction. In the early 21st century, climate change is seen by many as an existential threat, reflected by activity groups such as the “last generation”, whose name is a clear indication of the perceived existential threat.

In the past 9 months, mostly with the rise of ChatGPT, AI has now become the poster child of the existential threat du jour. Time magazine even published an op-ed by Eliezer Yudkowsky, where he suggested the following plan (emphasis added):

Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.

Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.

The piece ends by saying that only with such a plan would he and his wife see a chance that their toddler daughter “would live”, ending with the following:

We are not ready. We are not on track to be significantly readier in the foreseeable future. If we go ahead on this everyone will die, including children who did not choose this and did not do anything wrong.

To say that this is on the extreme end of the spectrum is an understatement. This is the kind of stuff that people in dangerous cults believe. I’m seriously worried about the well-being of Eliezer Yudkowsky and his family. Being deeply convinced that everyone will die soon is a very concerning state to be in.

Many luminaries of AI have chimed in, from George Hinton to Yoshuia Bengio, expressing deep concern about the destructive potential of AI.

The most recent consensus statement - signed by hundreds of leaders in this field, including the CEOs of DeepMind and OpenAI - simply states the following:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Statement of AI Risk

On the surface, the 23-word statement of AI risk seems as reasonable as it can be. Notwithstanding our demonstrated inability to be serious about pandemic preparedness, who wouldn't want to mitigate the risk of extinction? Do we really need to sign a statement that says, “hey, just to be clear, I am against human extinction”? Something else must be going on here.

The purported risk of a human extinction event is a staggering claim with extraordinary consequences, but with no evidence whatsoever.

Pandemics do indeed pose a serious risk (not for extinction, but that’s a detail). We’ve known this for centuries, and yet, we have failed to prepare seriously against them.

Back in the day when I was still on Twitter, I responded to a tweet by Ethereum creator Vitalik Buterin who argued:

A maximally terrible engineered pandemic is more likely to kill ~99.99% of humanity, leaving civilization room to recover from the remainder; AI will kill everyone. So it's another argument for caring about AI over bio.

Look at this tweet. “AI will kill everyone”. It’s stated as a fact. How can otherwise smart people make such ludicrous statements in public?

I counter-argued that the number of deaths by AI so far is 0, while the number of deaths by infectious disease are in the millions every single year, and so I would rather prioritize biological threats. Call me old-fashioned, but I think that things that happened millions of times in the past are likely to happen again in the future, whereas the case is less clear for things that have never happened before.

Let’s be clear: That something would pose a serious risk of human extinction is a staggering claim with extraordinary consequences. Were we to take it seriously, nothing else would matter. Any action that would help avoid extinction would be justified. When life fights for survival, all rules fall but one: survive or die.

It’s easy to dunk on Eliezer Yudkowsky, but he is the most consistent in his argument. His suggested plan of nuclear attack in the case of large AI development runs are completely sensible if you accept the premise that everyone will die otherwise.

It is correspondingly concerning to hear politicians like the EU’s Margaret Verstager say, “probably [the risk of extinction] may exist, but I think the likelihood is quite small.” Based on what evidence does she think such a risk exists? And even if it is quite small, why isn’t she focusing all her attention exclusively on this risk, alongside with the entire EU, and everyone alive on the planet? We’re not talking about just another serious problem here - we are talking about human extinction. Death. The end.

My simple answer is that most people don’t believe that there is an existential risk. My interpretation is that they are simply signaling to others that they’re aware of the concerns about negative effects, and that they would like to be on the record that somebody should address these concerns.

I won’t go more into the doom scenario. It’s without basis in reality, and judging by how people act, rather than what they say, the vast majority does not take this claim seriously. In addition, as others have said many times, AI, like any other technology, can cause harm if misused, and it’s important to focus on these issues today, and in the future.

The case for optimism

In all this doom and gloom, is there a case to be made for optimism?

I think so, and I would even go one step farther. I agree with the doomers about the epoch-changing impact of this technology. It’s just that I am optimistic about the outcome.

Indeed, I believe AGI to represent an existential opportunity.

What does that even mean? It means two things:

• It is existential in the sense that it radically changes our existence as human species in the future.

• It is an opportunity in the sense that this change will overall be very positive.

If technology represents power so profound that its consequences can be existential, shouldn’t we at least entertain the possibility that these existential consequences can be positive?

For those worrying deeply about AGI, this might be quite a head twister. Going from an existential threat to an existential opportunity seems like too much of a jump. Let’s first look at why AGI is unlikely to lead to our demise, and then consider its positive existential potential.

Why we are the most powerful species on the planet

There is a reason why humans dominate the earth. Although intelligence plays a role, it's not the central one. The primary reason why we have become so powerful is our extraordinary capability for cooperation. This kind of cooperation has ultimately allowed us to build a knowledge base of “reusable facts” (technology, science), which has turned us into the most powerful entity this planet has ever seen.

If we go back to the time before the invention of agriculture, we will find small bands of hunter-gatherers, who had to follow their food. Their survival depended entirely on finding food - everything else was absolutely secondary. They were utterly dependent on whatever nature provided. If you were to jump into a time machine and travel back 20,000 years, it would be impossible to conclude that humans ruled the planet. We may have already been the most intelligent species on the planet - depending on how you define intelligence - but it did not make us the all-powerful species we are today.

Approximately 10,000 years ago, the invention of agriculture allowed humans to secure their food supply. This, in turn, permitted them to build larger communities, first villages, then towns, and eventually cities. Work could be divided among large groups. While most individuals were engaged in agriculture or defense, a few had the opportunity to strategize, think, and develop new technology. This is the moment where human intelligence begins to shine brightly. Thanks to our language capability, and the invention of writing and eventually printing, we did not have to reinvent everything from scratch, but could immediately build from where the previous generation had left off. This led to an accelerating cycle of technology development, which ultimately resulted in us inventing machines that are stronger than us - first mechanically, and now intellectually.

When AGI comes onto the scene, it will find itself in the rich, complex technological infrastructure that we’ve built over generations. It will be entirely dependent on it, just as we are. Those who fear AGI often posit that at this point, it will be us versus them. But why would that be? We invented AGI. We may not be able to fully control all of it, but we will control most of it, and we will use it for our purposes and goals. And if that goal is to ensure that rogue AGI doesn’t harm us, then that’s where the focus will be.

I find it quite strange how the envisaged conflict is always presented as “humans vs. AI”. We humans will never be without AI. If there is ever going to be a conflict that involves AI, it will be one of two forms: 1) humans with AI vs. other humans with AI; or 2) humans with AI vs. AI without humans. The danger about the first option is real, and this is what we should really be worrying about - humans doing bad things with AI. But it is the second option that underlies all existential threat scenarios. However, given that AI will depend entirely on the infrastructure that humans built, it’s overwhelmingly likely that humans with AI will win that conflict. In fact, I believe this conflict to be so unbalanced that we will likely be able to prevent it in the first place.

We will merge with AGI

Historically, when humans invented a new powerful technology, we integrated it deeply into society. Today, most people use technology daily in almost every aspect of their lives, either directly or indirectly. During the process, we completely forget how amazingly powerful the technology is. I have direct and instant access to the world’s combined knowledge. I can travel extraordinary distances within hours. I am leveraging extraordinarily powerful machines to enhance my professional know-how. If I have the slightest physical discomfort, I have access to a wide range of evidence-based medicine. I am almost never seriously ill because the society I grew up in protects me through public health measures, and I am doing my part as well (plus luck in my case). My life is so different from my ancestors' 20,000 years ago that one might conclude we are different species. But of course, we are genetically almost as similar as I am to any random person living today.

In recent decades, we have begun the next phase of integrating technology, by merging it with our physical bodies. What started with prosthetics, pacemakers, and insulin pumps, continues today with subdermal RFID chips, retinal implants, and cochlear implants. These technologies are addressing physical issues. However, over the past few decades, we've seen the emergence of implants that interface directly with the brain. Recent work at EPFL, for example, used a digital bridge between the brain and spinal cord to enable an individual with chronic tetraplegia to stand and walk naturally in community settings. Neuralink has recently gained FDA approval to begin trials to implant brain chips into humans. And while products like Apple’s Vision Pro don’t interface directly with the brain, their many sensors can track your eye movements and other physical motions, enabling you to manipulate objects in the digital realm.

It’s not a stretch to imagine that we will eventually manage to have machines interact with the brain rather flawlessly. The opportunities are enormous. For instance, a person could mentally pose the question, 'What is 234,532,234 times 982,342,324?' and a brain interface could capture this query, relay it to a calculator implanted in the body, and transmit the correct answer back to the brain within milliseconds. From that moment on, it should also be possible to connect to a powerful AI model, instead of just a calculator.

The ethical challenges here are, of course, enormous. Enhancing brain functionality like this fundamentally alters who we are as humans. It’s entirely possible that this change will be too profound for us to accept, at least initially, leading to its prohibition. Martin Rees, the previous president of the Royal Society, believes that human explorers on Mars will look more favorably on this technology as it will enable them to survive in the new environment. But I think that we will not be able to prevent this on Earth. Many interactions with biology seem incompatible with human ethics at first, only to be used regularly a generation later.

For this reason, I think the merger with AI will be inevitable. The details are not predictable, but history gives us no reason to assume that this won’t happen. As a consequence, who we will be as humans will forever have changed. You may see this as a threat today. But imagine stepping into the shoes of, say, a 20th generation human merged with AI. For them, their all-biological human ancestors will be of historical and cultural interest, but their lives will be far superior from ours, just as ours are far superior to those that lived 500 years ago (unless you prefer being a peasant with little individual rights, an average life expectancy of 35 years, rampant childhood mortality, constant war, little sanitation, frequent hunger, and other such things).

This is the existential opportunity of AGI. The benefits this new type of humans will enjoy will have been made possible by AGI. Humans of the future might look back at our era and chuckle over our fears of emerging technology, much as we find humor in historical concerns that the invention of the printing press spelled doom.

What to do

Based on these reflections, what should we do in the face of emerging AGI?

My first suggestion is simple: strengthen institutions and international cooperation. Again, I might be old-fashioned here. But in the face of an extremely powerful technology that could easily be misused by bad actors, working together across borders has served us well. Why would we throw out that model? What format this international collaboration should take is an open question that many people are working on. My deepest concern there is many of these institutions are stuck in the pre-digital age. Nobody seriously believes that UN agencies, for example, are technologically strong enough today to drive the agenda. But perhaps that’s not necessary. What’s more necessary is for governments to have a vehicle in which they can come to mutual and binding agreements. 

My second suggestion is to look at the full spectrum of possible outcomes, focusing our efforts on promoting the positive rather than simply preventing the negative. Amidst widespread disagreement, there is a near-universal consensus that AGI will be changing everything. It’s therefore unsurprising that projections about the future vary widely. However, we should not allow global attention to become consumed by the more panic-inducing scenarios. Instead, let's ground our discussions in serious, evidence-based arguments. Today, technology is already causing harm to some groups - not because we lack laws to prevent this, but because we fail to enforce those laws consistently or even acknowledge the harm inflicted on certain communities. Rather than rushing to devise new policies, we should focus on reinforcing and improving what we already have in place.

When it comes to taking action, we should channel our energy into enabling obviously beneficial outcomes. The potential applications for AI are virtually limitless and hold immense promise for societal good. Let's commit to enabling AI's beneficial power: fostering universally accessible medical AI; harnessing AI's potential to combat climate change; transforming education through personalized learning; fortifying sustainable agriculture with precision farming; boosting accessibility with AI-assisted tools; enhancing public safety and disaster responses; addressing mental health issues on a global scale, and so many other ambitions. Let’s use this historic technology to bring about unprecedented societal progress.

My third suggestion is to be more rigorous about the extremely bad scenarios. If there are potentially very dangerous outcomes where things could seriously go wrong, what are the paths to get there, what are the different steps, and what are the risks of each step? These things can be calculated, even if there are many uncertainties (tale a look at this calculation about the potential risk of pathogenic lab leaks). Then, consider what needs to happen to minimize the risk in each step. It’s a difficult exercise because unlike lab leaks, we have no historical evidence to fall back on for losing control of AI. But that doesn’t mean we can’t think about all the necessary steps that would be necessary. Rogue AI would need an independent energy source - can we make that harder? Rogue AI would need to replicate itself - can that be prevented, or even be prohibited? It’s important that we start to think more like ecologists and evolutionary biologists, because our fear of AGI that is out of human control is ultimately a fear of AGI becoming a part of nature.

CODA

This is a newsletter with two subscription types. You can learn why here.

To stay in touch, here are other ways to find me:

• Social: I’m mainly on Mastodon, and occasionally on LinkedIn.

• Writing: I write another Substack on digital developments in health, called Digital Epidemiology.