{"id":2090,"date":"2026-10-21T13:52:41","date_gmt":"2026-10-21T13:52:41","guid":{"rendered":"https:\/\/redzine.co.uk\/index.php\/2026\/10\/21\/why-the-most-dangerous-ai-may-be-the-one-we-trust-too-much\/"},"modified":"2026-10-21T13:52:41","modified_gmt":"2026-10-21T13:52:41","slug":"why-the-most-dangerous-ai-may-be-the-one-we-trust-too-much","status":"publish","type":"post","link":"https:\/\/redzine.co.uk\/index.php\/2026\/10\/21\/why-the-most-dangerous-ai-may-be-the-one-we-trust-too-much\/","title":{"rendered":"Why the most dangerous AI may be the one we trust too much"},"content":{"rendered":"<p>Machines don\u2019t have to become superintelligent to cause catastrophic harm. As artificial intelligence spreads through military decision-making, they just have to be wrong \u2013 and trusted by the humans in charge.<\/p>\n<p>In 1991, Terminator 2: Judgement Day gave us one of popular culture\u2019s most enduring images of AI gone wrong. Humanity builds Skynet, an artificial intelligence given control of strategic defence. It learns at a \u201cgeometric rate\u201d, becomes self-aware and, when its human operators try to shut it down, it starts a nuclear war.<\/p>\n<p>Versions of that story now feature in serious debate about AI. Increasingly capable large language models (LLMs), the argument goes, could develop into artificial general or super intelligence (AGI\/ASI), acquire goals that diverge from ours and ultimately threaten humanity.<\/p>\n<p>That debate deserves attention, but it depends on a chain of large and unproven steps. Today\u2019s AI systems must somehow transform into genuinely general intelligences, develop independent goals and gain control over the physical world.<\/p>\n<p>Meanwhile, a more immediate risk is taking shape that requires none of those steps.<\/p>\n<figure><figcaption><span class=\"caption\">The genesis of Skynet.<\/span><\/figcaption><\/figure>\n<p>In the film, Skynet builds the Terminators, machines that find, track and kill humans. Strip away the science fiction and the idea is simple: combine sensors, software and weapons and reduce the human role.<\/p>\n<p>Human beings are doing exactly that, without anything resembling a malicious superintelligence like Skynet. In August 2026, Ukraine\u2019s Ministry of Defence <a href=\"https:\/\/mod.gov.ua\/en\/news\/smart-weapons-more-than-70-systems-using-ai-and-computer-vision-are-already-striking-enemy-targets\">said<\/a> its forces were using more than 70 AI and computer vision systems to strike targets. Algorithms can recognise equipment and personnel and lock onto those targets autonomously, so that a drone can complete its attack even if jamming cuts the link to the operator.<\/p>\n<p>These systems need no general or super intelligence. They need sensors, software that can recognise a target, navigation and a weapon. And the more targeting is automated, the more the human\u2019s role shrinks to approving what the machine has already identified.<\/p>\n<p>That\u2019s precisely where trust becomes the weak point.<\/p>\n<h2>The war that AI almost started<\/h2>\n<p>The second risk is more subtle: AI shaping the decisions of the people who pull the triggers. <\/p>\n<p>According to a recent <a href=\"https:\/\/edition.cnn.com\/2026\/09\/18\/politics\/us-military-ai-false-intelligence-china-ship\">CNN investigation<\/a>, in spring 2026, during the war with Iran, a US special operations analyst used an AI chatbot to assess intelligence about a Chinese ship in the Middle East. The chatbot, drawing on both open-source and classified material, wrongly identified the ship\u2019s cargo as components of a nuclear weapons programme. The analyst used AI again to package this into a standard intelligence report. <\/p>\n<p>Armed personnel were preparing to board the ship and military aircraft were airborne before an official looked more closely and found the error. One source said the false report \u201calmost started a war\u201d. Any US operation against a Chinese vessel could risk spiralling into an armed conflict.<\/p>\n<p>The AI was not conscious, hostile or out of control. It was simply wrong \u2013 but people trusted and acted on it.<\/p>\n<hr>\n<p>\n  <em><br \/>\n    <strong><br \/>\n      Read more:<br \/>\n      <a href=\"https:\/\/theconversation.com\/ai-needs-to-be-disarmed-pope-leo-sees-threat-to-humanity-in-technological-arms-race-283432\">\u2018AI needs to be disarmed\u2019: Pope Leo sees threat to humanity in technological arms race<\/a><br \/>\n    <\/strong><br \/>\n  <\/em>\n<\/p>\n<hr>\n<p>The US military is pushing to <a href=\"https:\/\/www.bbc.co.uk\/news\/articles\/cy02gjq2987o\">weave AI<\/a> into almost every part of its work, from analysing intelligence to selecting targets. CNN\u2019s sources said errors of this kind are not uncommon across the intelligence community, and that pressure to produce intelligence faster is opening the door to mistakes. The ship incident was caught in time. The next error may not be.<\/p>\n<p>At its heart, this is a problem of trust. LLMs produce fluent, confident language effortlessly \u2013 and, repackaged as an official report, an error gains an authority it never earned. Colleagues and I have <a href=\"https:\/\/link.springer.com\/article\/10.1007\/s00146-026-03021-w\">argued<\/a> that these systems are built to produce an answer rather than to say \u201cI don\u2019t know\u201d. For example, in one <a href=\"https:\/\/link.springer.com\/article\/10.1007\/s43681-026-01391-x\">experiment<\/a>, we found that non-specialists could not tell a chatbot\u2019s convincing but wrong physics answers from genuine expertise.<\/p>\n<figure class=\"align-center \">\n            <img decoding=\"async\" alt=\"A depiction of working robots and communication networks\" src=\"https:\/\/images.theconversation.com\/files\/764978\/original\/file-20261008-50-phdctp.jpg?ixlib=rb-4.1.1&amp;q=45&amp;auto=format&amp;w=754&amp;fit=clip\"><figcaption>\n              <span class=\"caption\">The more military targeting is automated, the more the human\u2019s role shrinks to approving what the machine has already identified.<\/span><br \/>\n              <span class=\"attribution\"><a class=\"source\" href=\"https:\/\/www.shutterstock.com\/image-photo\/working-robots-communication-networks-2356921807?trackingId=62d181af-0168-40f2-869b-469dca7b4528&amp;listId=searchResults\">metamorworks\/Shutterstock<\/a><\/span><br \/>\n            <\/figcaption><\/figure>\n<h2>Why a \u2018human in the loop\u2019 isn\u2019t enough<\/h2>\n<p>Policymakers often reassure us that a human makes the final decision. In the Chinese ship case, people were in the loop throughout. They simply became another stage through which the error was passed.<\/p>\n<p>That fits decades of research on <a href=\"https:\/\/journals.sagepub.com\/doi\/10.1177\/0018720810376055\">automation bias<\/a>, the tendency to over-rely on automated systems. It affects experts as well as novices. In 2003, US Patriot missile batteries <a href=\"https:\/\/www.cnas.org\/press\/press-release\/cnas-releases-report-on-automation-and-the-patriot-air-and-missile-defense-system\">shot down<\/a> two allied aircraft after operators accepted its misclassification.<\/p>\n<p>A person in the loop works only when that human is able, adequately equipped and willing to challenge the machine. <\/p>\n<p>That means verification. AI output should be clearly labelled, so its origin is visible to everyone who relies on it, and treated as a claim to be tested against independent sources before anyone acts on it. In war, the cost of not checking could hardly be higher.<\/p>\n<p>None of these concerns over trusting AI in the short term mean long-term questions about advanced AI in the future should be dismissed. But much public anxiety focuses on what these systems might become, and less on what they are already doing.<\/p>\n<hr>\n<p>\n  <em><br \/>\n    <strong><br \/>\n      Read more:<br \/>\n      <a href=\"https:\/\/theconversation.com\/why-police-have-been-told-to-pause-their-use-of-ai-in-court-cases-289731\">Why police have been told to pause their use of AI in court cases<\/a><br \/>\n    <\/strong><br \/>\n  <\/em>\n<\/p>\n<hr>\n<p>Perhaps Terminator 2 got more right than we appreciate. Skynet\u2019s catastrophe doesn\u2019t begin with self-awareness. It begins with trust, when humans hand a machine control because they believe it will make better decisions than they can.<\/p>\n<p>We don\u2019t need a conscious machine to repeat that mistake. Fallible AI, misplaced human trust and increasingly autonomous weapons offer a far more mundane route to catastrophe.<\/p>\n<p>The danger is not that our machines become too intelligent. It is that we hand responsibility to machines that are far less reliable than we assume. As the film puts it: \u201cThere\u2019s no fate but what we make for ourselves\u201d.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/counter.theconversation.com\/content\/293887\/count.gif\" alt=\"The Conversation\" width=\"1\" height=\"1\" \/><\/p>\n<p class=\"fine-print\"><em><span>Simon Thorne does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.<\/span><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Machines don\u2019t have to become superintelligent to cause catastrophic harm. As artificial intelligence spreads through military decision-making, they just have to be wrong \u2013 and trusted by the humans in charge. In 1991, Terminator 2: Judgement Day gave us one of popular culture\u2019s most enduring images of AI gone wrong. Humanity builds Skynet, an artificial [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2090","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/posts\/2090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/comments?post=2090"}],"version-history":[{"count":0,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/posts\/2090\/revisions"}],"wp:attachment":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/media?parent=2090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/categories?post=2090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/tags?post=2090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}