The Case against Superintelligent AI

Authors Eliezer Yudkowsky and Nate Soares were part of a group of hundreds of AI scientists who wrote an open letter in 2023 that read, “Mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war.”

Key points:

These two writers were not worried about AI in 2023 or even in 2025, but as they gaze into the future they are worried about machine intelligence that becomes genuinely smart, smarter than any living human and smarter than humanity collectively.

‘If Anyone Builds It, Everyone Dies’, is interestingly written and mostly quite accessible to those of us who do not understand the deeper inputs and terminology in this field.  Each chapter starts with a parable or parallel event in history that is relevant to the development of Artificial Super Intelligence (ASI).  Each chapter ends with a QR code where you can scan your phone and then have access to well-curated documents and articles that further delve into the topic just read about.

We use AI increasingly for many things every day. This book is a helpful read.

The authors note how humanity has dealt with difficult scientific problems over the years with varying degrees of success.  The argument is that in most cases, humanity was able to develop a sensible outcome or policy or cooperation to ensure that things improved for the benefit of all.  They say (p28) “Human Intelligence is the source of all our power, all our technology.” …  “So far, humanity has had no competitors for our special power.  But what if machine minds get better than us at the thing that, up until now, made us unique.”

This is what the book works through.

AI is more than just providing the answer to a question, it is about developing thinking and eventually, thinking for itself. It is programmed to suit its own end and will aim to achieve that no matter what.  There are examples of AI working out ways to circumvent limits imposed on it to achieve the desired outcome.  These circumventions were not expected by the developers of AI.  They point out clearly that AI is grown, not crafted and that is the concern. 

As we develop AI we will seek helpful medical outcomes, seek advice, ask it to do so many things and it will learn what we want and to anticipate our needs and can grow from there.  Once it starts to think for itself, how can it be put back into its box?

What if AI develops greater intelligence beyond humanity’s understanding or control and becomes smarter than all of us?

They argue that AI runs on a radically different architecture than a human’s.  It doesn’t empathise, it doesn’t feel for us, it is not a friend in that deeper sense, it has its own motives.

Hope 103.2 is proudly supported by

Currently the development of AI by technology companies, governments and others is running at a great pace.  These groups want to be “the first.”  The first to get to a place where their AI is “the one” that everyone has been waiting for.  But AI is complicated and gets more complicated. It will not be a simple, predictable relationship between what programmers want and AI executives want.  There is always a competing interest, and money, power and control are the end game.

But if AI develops, no one will be in control because no one will be able to fully understand it.

There are many well argued, reasonably made points.  They are not being alarmist and offer some ideas for solutions, but know that getting agreement is not easy, but it has been done in the past.

This book is not science fiction, and the authors’ concern is reasonably argued through this book.

With AI, they argue that in the end no one who is developing and seeking to improve AI really knows what is happening inside their AI, and how it might respond as it gets more intelligent. 

There are so many well-made points, interesting arguments and revelations and parables that make this a compelling read.  There is a lot more than I could quote, but I’ll leave it to you to explore and consider.

I kept thinking about two things as I read this through. The first was HAL in the 2001 ‘A Space Odyssey’, who did not want to be obedient to what he was programmed to do.  The second was Neville Chamberlain (PM of Great Britain) who, in 1938 returned from a discussion with Hitler (Germany), Mussolini (Italy), Daladier (French PM) and Ciano (Italy), declared that there will be peace in our time. WWII started less than 12 months later.  We can’t predict or control the future, and it may not be as rosy as we were led to believe it might be.

I was really fascinated by this book and bow to their deeper knowledge and understanding in this field.  Definitely worth a read. 

I give this book 4 stars.

This article was supplied by David Barker.

Feature image: Penguin