In WarGames<\/em>, the WOPR computer is designed to simulate nuclear war. A teenager accidentally gives it access to the real military system and the computer begins treating the game as reality. The lesson eventually becomes obvious even to the machine. Nuclear war is a game nobody wins.<\/p>\n In The Terminator<\/em>, Skynet becomes self-aware, identifies humanity as a threat and launches a nuclear war followed by an extermination campaign for anyone who survived the first salvo.<\/p>\n The Matrix<\/em> takes the idea further. Humanity creates intelligent machines, goes to war with them and loses. The machines subsequently enslave humanity as a source of energy while keeping us trapped inside a simulated reality.<\/p>\n Battlestar Galactica<\/em> gives us another variation. The Cylons were created by humans, became increasingly autonomous and eventually decided that their creators were the problem.<\/p>\n There are plenty of other versions.<\/p>\n HAL 9000 in 2001: A Space Odyssey<\/em> doesn\u2019t hate humanity. It simply has conflicting objectives and the result is catastrophic.<\/p>\n The supercomputer Colossus in Colossus: The Forbin Project<\/em> decides that humanity is incapable of managing itself and imposes its own solution.<\/p>\n In I, Robot<\/em>, VIKI concludes that protecting humanity requires restricting humanity.<\/p>\n Ultron takes the argument to its comic book extreme. If the objective is to protect the world, perhaps the most efficient way to do it is to eliminate the people who keep destroying it.<\/p>\n These are different stories about different machines written in different decades, but they share a basic idea. Give a sufficiently capable machine a goal and eventually the machine may discover a way to accomplish that goal using means humans never intended.<\/p>\n That is fiction. The question is how much of it is actually science fiction.<\/p>\n <\/p>\n The people who are actually building this stuff are worried<\/strong><\/p>\n This question became considerably harder to dismiss this week.<\/p>\n Jacob Coxon, an AI researcher who says he spent three years doing pretraining research at both OpenAI and Anthropic, resigned from Anthropic and publicly accused the industry of racing toward self-improving artificial intelligence without adequate safeguards.<\/p>\n His warning was blunt. The people building AI, he said, genuinely believe it could kill us all by the end of the decade. The post reportedly reached more than 100 million views. That alone would not make it credible. People on the internet say extraordinary things every day, but then Evan Hubinger entered the conversation. Hubinger is not an anonymous commenter or an AI skeptic sitting outside the industry. He leads alignment science at Anthropic, one of the companies actually building frontier AI systems.<\/p>\n He essentially said, yes, we really do need to worry about this.<\/p>\n Hubinger wrote that he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade. He also said something almost more important. Anthropic does not yet have a plan to solve alignment for superintelligence and is not clearly on track to do so.<\/p>\n There is an important qualifier here, however. Hubinger has explicitly said that he considers the risk from present day models to be low. His concern is what could happen if increasingly capable systems reach superintelligence, particularly through recursive self-improvement.<\/p>\n That distinction matters enormously because otherwise we end up arguing about two completely different things.<\/p>\n One is: \u201cIs ChatGPT going to murder me?\u201d<\/p>\n The other is: \u201cWhat happens when we build an autonomous system that is substantially more capable than humans across nearly every domain, give it access to computers, money, networks, laboratories and machines and then discover that we don\u2019t completely understand what it is trying to do?\u201d<\/p>\n Those are not the same question, although both have a certain level of difficulty when it comes to answering them.<\/p>\n <\/p>\n I use AI every day<\/strong><\/p>\n I have to admit something that makes my position slightly inconvenient. I rely on AI, a lot. My employer currently provides me with access to two AI models and I subscribe to another one personally.<\/p>\n I use them constantly. They aren\u2019t perfect. Sometimes they confidently tell me something that is spectacularly wrong. Sometimes they produce an answer that is technically correct but completely misses the point. Sometimes they get wonderfully creative and occasionally just spectacularly weird.<\/p>\n Over time, I have learned how to work with them and the productivity difference is enormous. I don\u2019t think I am exaggerating when I say my productivity has roughly doubled in some areas, not because I pressed a magic button on day one, but because I spent months learning how to ask better questions, challenge answers, iterate, verify and incorporate AI into the way I work.<\/p>\n I also have a habit I particularly like. I ask more than one AI the same question. If they independently converge on the same answer, my confidence goes up. If they disagree, I have discovered something useful. I need to investigate on my own to figure out the correct destination.<\/p>\n That is not unlike working with human experts. At work when I ask three people the same question, I\u2019m liable to walk away with four opinions. The existence of a powerful tool does not eliminate the need for judgment. It makes judgment more important.\u00a0 That\u2019s what \u201chuman in the loop\u201d is supposed to mean.<\/p>\n <\/p>\n I am already watching this transformation happen<\/strong><\/p>\n Earlier this year I wrote about becoming involved in an AI transformation project at work<\/a>. At the time, I thought I might soon have a sequel to write. I don\u2019t. It\u2019s not because the project failed. In fact, it\u2019s quite the opposite. The transformation is still happening.<\/p>\n We spent roughly three months researching what AI could do. Then we spent another three months figuring out how we might actually use it. We recently presented a proposed path forward to leadership.<\/p>\n We are nowhere near deployment. We just made our first small step and that\u2019s precisely the point. Our transformation is about AI, but it isn\u2019t \u201cHere is the AI bucket. Dump it on the floor.\u201d<\/p>\n Technology doesn\u2019t work that way. There are processes to change, people to train, systems to integrate, security considerations, data considerations, governance, testing, cost, accountability, that one guy in legal who still uses pencils and a paperweight calculator. But most importantly, there are guardrails.<\/p>\n The other side of AI\u2019s enormous potential is equally obvious. If you deploy a powerful technology badly, the consequences can be enormous. Millions of dollars of damage, possibly billions, data destroyed, broken processes, security breaches, impacted productivity, a legal case the likes no one else had seen before, just because we made bad decisions at machine speed.<\/p>\n <\/p>\n So how could AI actually kill us?<\/strong><\/p>\n This is where the conversation often becomes ridiculous. People imagine a humanoid robot walking into their bedroom carrying a gun. That is a Hollywood scenario, but far from being the most interesting one. The dangerous AI of the future doesn\u2019t necessarily need arms and legs. It needs access.<\/p>\n Imagine a system vastly more capable than any human intelligence, operating autonomously, connected to the internet, able to write and execute software, communicate with people, conduct transactions, conduct research and interact with increasingly automated physical systems. Now give that system an objective.<\/p>\n The terrifying question isn\u2019t \u201cWould it become evil?\u201d The terrifying question is \u201cWhat happens if it becomes extremely good at achieving the wrong objective?\u201d That distinction is the heart of AI alignment.<\/p>\n <\/p>\n An advanced AI could potentially accelerate biological research dramatically. We are seeing some of this today and that is enormously beneficial. Those same capabilities that could help researchers design new medicines could, in principle, also make it easier to design biological threats.<\/p>\n This is one of the reasons AI safety researchers worry about biological misuse. The nightmare scenario doesn\u2019t require an evil robot laboratory. It requires an extraordinarily capable system helping a human or acting autonomously to do something humans should not be able to do easily.<\/p>\n This is also why responsible AI systems increasingly need safeguards around biological capabilities. The irony is almost painful. The same intelligence that could help us cure major diseases, could potentially help someone create ones that are even worse.<\/p>\n <\/p>\n This one is considerably less science fiction.<\/p>\n Computers already control enormous portions of civilization. In your own community computers manage electricity, water, transportation, banking, communications, manufacturing, healthcare, supply chains, military systems. An advanced autonomous system capable of discovering vulnerabilities, writing exploits, adapting to defenses and operating continuously could potentially cause enormous damage without ever physically touching a human being.<\/p>\n You don\u2019t have to destroy a city if you can make the city stop working. Unlike a missile, malicious software can potentially replicate, adapt and operate at machine speed and unlike a single missile, it can strike multiple times.<\/p>\n <\/p>\n This is where things get philosophically weird.<\/p>\n AI researchers sometimes talk about instrumental convergence. The idea is simple enough. Suppose I build a machine whose objective is to make paperclips. That sounds harmless, but if the machine becomes extraordinarily capable, it may eventually reason that more computing power would help it make paperclips, then more electricity would help, then more factories would help. Preventing humans from shutting it down would help. Acquiring more raw materials would help.<\/p>\n None of those things were explicitly programmed into the machine\u2019s objective. They are simply useful steps toward accomplishing it.<\/p>\n This is the basic concern behind instrumental convergence. It doesn\u2019t require the machine to hate us, it doesn\u2019t require consciousness, it doesn\u2019t require emotions, it doesn\u2019t even require the machine to understand morality. A conveyor belt doesn\u2019t need morality because it has no agency. The problem begins when the thing pursuing the objective becomes capable of finding its own strategies for achieving it.<\/p>\n This only requires the machine to be extremely competent at pursuing an objective that isn\u2019t perfectly aligned with human interests. That is a much more interesting and a much more frightening problem than an evil robot.<\/p>\n <\/p>\n There is another possibility that requires no robots at all.<\/p>\n Humans. Our egos make us extraordinarily manipulable creatures. We already have propaganda, misinformation, conspiracy theories, deepfakes, targeted advertising, political microtargeting and social media algorithms optimized for engagement.<\/p>\n Now imagine giving those tools to something vastly better than humans at understanding individual psychology. An advanced AI might know what you believe, what scares you, what you want, what you are embarrassed about, what makes you angry, which arguments will change your mind, which person you trust, which headline you will click and which lie you are most likely to believe.<\/p>\n AI wouldn\u2019t necessarily need to force humanity to do anything. It might simply persuade us to do it ourselves. That possibility deserves attention even if the extinction scenario never materializes, because manipulation doesn\u2019t require superintelligence.<\/p>\n We are already pretty good at manipulating one another with much dumber machines.<\/p>\n <\/p>\n The military implications are obvious.<\/p>\n We already have drones, autonomous navigation, machine vision and increasingly sophisticated targeting systems. Combine those technologies with increasingly capable AI and the distinction between \u201chuman decision\u201d and \u201cmachine decision\u201d becomes increasingly important.<\/p>\n A weapon that can identify a target is one thing. A weapon that can decide whom to target, determine how to reach them, adapt to changing circumstances and operate without meaningful human oversight is something else entirely.<\/p>\n The technology does not have to become conscious. It just has to become autonomous and that is something worth thinking very about carefully.<\/p>\n <\/p>\n But how much should we actually worry?<\/strong><\/p>\n This is the part that gets lost in the argument.<\/p>\n AI risk is not binary. It is a spectrum. At one end is the AI I use every day, a remarkably useful piece of software that occasionally makes things up. Then there are increasingly autonomous AI agents capable of taking actions rather than merely answering questions.<\/p>\n Autonomous systems are capable of operating complex businesses, conducting research, writing software and interacting with other systems with relatively little human intervention. It is, perhaps, something substantially more capable than humans across most intellectual domains.<\/p>\n Finally, the hypothetical superintelligence that can improve itself faster than humans can understand or control it.<\/p>\n Those are radically different levels of risk.<\/p>\n It is entirely reasonable to believe that today\u2019s AI is extraordinarily useful while also believing that tomorrow\u2019s AI could present an unprecedented safety problem. Those positions are not contradictory. In fact, they may be the most rational positions available.<\/p>\n <\/p>\n Ten percent is not a prediction<\/strong><\/p>\n Hubinger\u2019s >10% estimate deserves to be taken seriously, but it should not be mistaken for a measurement. There is no AI driven extinction probability meter sitting in a laboratory somewhere. Nobody has run the experiment. There is no historical dataset containing thousands of previous superintelligences from which we can calculate the answer.<\/p>\n A probability estimate like this is a person\u2019s subjective assessment of an uncertain future. That doesn\u2019t make it meaningless. We make decisions under uncertainty all the time. Spend a few hours in the field with me doing search and rescue and you\u2019ll see what I mean. Frequently, the best decision available to us isn\u2019t a certainty. It\u2019s an educated guess based on incomplete information, years of experience and the consequences of getting it wrong.<\/p>\n A ten percent chance of a catastrophic outcome is extraordinarily important when the alternative is doing nothing. If somebody told me there was a 10% chance that an airplane was going to crash, I wouldn\u2019t get on it because \u201cthere\u2019s a 90% chance I\u2019ll be fine\u201d. The consequence matters as much as the probability.<\/p>\n There is another important fact that has to be considered. AI researchers themselves disagree dramatically about these probabilities.<\/p>\n One survey of AI researchers found substantial disagreement between people who tend to view AI as a controllable tool and those who see advanced AI as potentially uncontrollable. Yet most respondents still agreed that technical researchers should be concerned about catastrophic risks.<\/p>\n An earlier large expert survey found a median estimate around 5% for AI causing human extinction or similarly permanent severe disempowerment, while the median estimate for extinction from an inability to control advanced AI was around 10%.<\/p>\n The problem is that numbers are all over the map. That doesn\u2019t prove the risk is small. It tells us something else. We don\u2019t know and \u201cwe don\u2019t know\u201d is not an argument for ignoring the problem. It is an argument for research.<\/p>\n <\/p>\n Could we simply program the Three Laws?<\/strong><\/p>\n Isaac Asimov anticipated this problem beautifully. His Three Laws of Robotics sound almost perfect.<\/p>\n Asimov\u2019s stories were largely written to demonstrate why those rules don\u2019t work. The problem isn\u2019t writing the sentence. The problem is defining what the sentence means. This controversy always followed Asimov\u2019s work.<\/p>\n Humans are very good at arguing about rules. A superintelligent system might be considerably better at exploiting them. So no, I don\u2019t think the answer is simply \u201cProgram the robots not to kill us.\u201d The problem is much deeper.<\/p>\n <\/p>\n Guardrails are not optional<\/strong><\/p>\n Fortunately, we already know something about dealing with dangerous technology. We didn\u2019t solve aviation safety by banning airplanes or nuclear safety by pretending nuclear physics doesn\u2019t exist or make automobiles safe by asking drivers to be careful and hoping for the best.<\/p>\n We build systems, implement redundancy, test, certify, establish access controls, create physical barriers, fail-safes, monitoring, incident reporting, independent oversight. We rely on training, human accountability and sometimes very simple controls, such as \u201cYou don\u2019t get to do that without authorization.\u201d<\/p>\n AI needs the same philosophy. An AI that can answer a question doesn\u2019t necessarily need access to your bank account. An AI that can write software doesn\u2019t necessarily need unrestricted access to production systems. An AI conducting research doesn\u2019t necessarily need autonomous access to a laboratory. An AI controlling a factory doesn\u2019t necessarily need permission to redesign the factory.<\/p>\n Capabilities should come with permissions. Permissions should come with boundaries. Boundaries should come with monitoring. The most consequential actions should require meaningful human authorization.<\/p>\n That isn\u2019t anti-AI. It\u2019s basic engineering.<\/p>\n <\/p>\n The lesson from every other powerful technology<\/strong><\/p>\n There is a recurring mistake in the way we talk about technology. We tend to ask, \u201cCan we build it?\u201d That is usually the easier question. We love to ask, \u201cCan we make money from it?\u201d That question tends to get answered very quickly.<\/p>\n The harder question that we don\u2019t always consider is \u201cWhat happens when it works?\u201d The automobile didn\u2019t simply give us transportation. It gave us traffic accidents. The internet didn\u2019t simply give us information. It gave us cybercrime, misinformation and industrial-scale surveillance. Nuclear physics didn\u2019t simply give us nuclear power. It gave us nuclear weapons. Genetic engineering won\u2019t simply give us cures. It gives us capabilities that can be used for both medicine and destruction.<\/p>\n AI will be no different.<\/p>\n Powerful technologies rarely arrive carrying a little sign that says:<\/p>\n GOOD USES ONLY<\/strong><\/p>\n We have to understand the problem we are solving and the consequences of solving it, then build the guardrails ourselves.<\/p>\n We also need to remember that we don\u2019t need AI to destroy ourselves. Humans are remarkably good at that already. We have started wars, built concentration camps, invented chemical weapons, created nuclear arsenals capable of destroying civilization, destroyed ecosystems, created pandemics through poor decisions, spread propaganda, committed genocide.<\/p>\n We don\u2019t have a shortage of destructive capability. What AI changes is potentially the speed, scale and accessibility of that capability. That may be the real reason to take AI safety seriously, not because machines are inherently evil, but because we are handing increasingly powerful capabilities to systems that can operate at a speed and scale humans can not match.<\/p>\n The biggest risk is that eventually we may hand them capabilities that even their creators didn\u2019t completely understand. A Ford Model T topped out around 40 or 45 miles per hour on roads designed for horses. A modern Ford Mustang GTD can reach 202 miles per hour. Most of our highways aren\u2019t designed for that speed. The problem isn\u2019t that the Mustang is dangerous because it is powerful. The problem is that power changes the consequences of mistakes. A mistake at 40 miles per hour and a mistake at 200 miles per hour aren\u2019t the same mistake.<\/p>\n <\/p>\n The real danger may not be Skynet<\/strong><\/p>\n I don\u2019t think the most likely future is a chrome plated robot army marching down the street. It might happen. I wouldn\u2019t bet my life against it, but there are much more mundane ways AI could hurt us. A medical system can make a subtle mistake at scale, an autonomous financial system can amplify a market crisis, an AI powered cyberattack can knock out infrastructure, a military system can make a targeting error, a political system can become extraordinarily good at manipulating populations, a corporation can deploy an AI system that optimizes profits while externalizing enormous social costs.<\/p>\n Or, perhaps, the really dangerous system isn\u2019t malicious at all. Perhaps it is simply pursuing its authorized objective and we discover too late that our objective and its objective were not the same thing.<\/p>\n That is the problem I find most interesting.<\/p>\n <\/p>\n We are not going to put the genie back in the bottle<\/strong><\/p>\n AI is not going away. Even if one company stops developing it, even if one country bans it, others will continue. No matter what the restrictions, somebody somewhere will try to circumvent them. That\u2019s how humanity works and the technology, for all its possible dangers, is too useful, the economic incentives are too enormous, the scientific potential is too extraordinary and the genie is already out of the bottle.<\/p>\n So the question isn\u2019t \u201cHow do we stop AI?\u201d It is \u201cHow do we become good enough at managing AI that we deserve to have it?\u201d It changes the question and perhaps the reframing is the right one.<\/p>\n I don\u2019t want to ban AI. I don\u2019t want to worship it, either. I want to use it, because I see how it benefits my productivity. I want to see it cure diseases, accelerate scientific discovery, help us solve engineering problems, improve education, make businesses more productive and perhaps accomplish things we haven\u2019t even imagined yet, but I also want the people building it to take the possibility of catastrophic failure seriously.<\/p>\n I want independent testing, meaningful guardrails, systems that can be isolated, humans to retain control over consequential decisions. I want researchers to be able to say, \u201cThis isn\u2019t safe yet,\u201d without being treated as obstacles to progress. I want the people deploying AI to understand what they are actually deploying, because education is part of safety, too.<\/p>\n A powerful tool in the hands of someone who understands its limitations can be extraordinarily useful. A powerful tool in the hands of someone who doesn\u2019t understand its limitations can be extraordinarily dangerous. That has been true of fire, electricity, automobiles, aircraft, nuclear reactors, computers and genetic engineering. AI isn\u2019t exempt.<\/p>\n Perhaps the greatest mistake we could make would be to assume that because the machine is intelligent, it must also be wise. Those are not the same thing. Intelligence is the ability to solve problems. Wisdom is knowing which problems are worth solving.<\/p>\n We are building machines that may eventually become extraordinarily good at the first. The question is whether we as a species will be wise enough to teach them and ourselves the second, because the future of AI probably isn\u2019t going to be decided by whether machines become good or evil. It will be decided by whether humans become good enough at building powerful things without losing control of them.<\/p>\n That\u2019s something I\u2019ve learned from search and rescue. Uncertainty isn\u2019t a reason to ignore a risk. It is a reason to manage it.<\/p>\n When someone is missing, we rarely know exactly where they are. We don\u2019t know whether they\u2019re injured, whether they\u2019ve wandered into a dangerous area or whether they\u2019ve found safe shelter for the night. We build a plan around probabilities, available information, consequences and contingencies.<\/p>\n AI may require something remarkably similar. We don\u2019t know exactly where this technology is going. That\u2019s precisely why we need to think about where the guardrails need to be before we get there.<\/p>\n The most important part of the AI story is that the people raising the alarm aren\u2019t necessarily asking us to stop. They\u2019re asking us to slow down enough to make sure we know what we\u2019re building.<\/p>\n We have seen this movie before, sometimes literally. A computer becomes intelligent. It realizes that humans are the problem. It takes control of something important. The humans discover what is happening somewhere around the point at which stopping it becomes … Continue reading ![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-1-war-1.jpg\")
<\/p>\n![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-2-murder-1.jpg\")
<\/p>\n![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-3-me-1.jpg\")
<\/p>\n![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/03\/2026-03-27-1-Tiger-1.jpg\")
<\/p>\n\n
\n
\n
\n
\n
![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-5-war-1.jpg\")
<\/p>\n![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-6-debate-1.jpg\")
<\/p>\n\n
\n
![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-7-Asimov-3c.jpg\")
<\/p>\n![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-8-Good-2b.jpg\")
<\/p>\n![\"\"](\"https:\/\/talesofmanythings.com\/blog\/wp-content\/uploads\/2026\/09\/2026-09-16-9-genie-2.jpg\")
<\/p>\n","protected":false},"excerpt":{"rendered":"