{"id":1332,"date":"2026-08-07T14:09:56","date_gmt":"2026-08-07T14:09:56","guid":{"rendered":"https:\/\/redzine.co.uk\/index.php\/2026\/08\/07\/ai-is-making-disinformation-harder-to-spot-but-weve-found-a-new-way-to-catch-it\/"},"modified":"2026-08-07T14:09:56","modified_gmt":"2026-08-07T14:09:56","slug":"ai-is-making-disinformation-harder-to-spot-but-weve-found-a-new-way-to-catch-it","status":"publish","type":"post","link":"https:\/\/redzine.co.uk\/index.php\/2026\/08\/07\/ai-is-making-disinformation-harder-to-spot-but-weve-found-a-new-way-to-catch-it\/","title":{"rendered":"AI is making disinformation harder to spot \u2013 but we\u2019ve found a new way to catch it"},"content":{"rendered":"

Do you ever see comments on social media that seem way off topic, but still manage to wrench the discussion around to divisive political debate?<\/p>\n

A discussion about the cost of living suddenly becomes an argument about immigration. A conversation about the war in Ukraine turns into claims about government corruption. It can feel jarring \u2013 and sometimes this is deliberate.<\/p>\n

As generative AI becomes more powerful, malicious groups are increasingly using it<\/a> to produce and spread disinformation<\/a> online. Automated accounts can flood social media with convincing comments designed to sow division<\/a>, inflame political debate and undermine trust in reliable information.<\/p>\n

But our latest research<\/a> offers a way to spot these attempts. Rather than trying to identify whether a post was written by AI, we focus on something different: whether it\u2019s trying to derail the conversation.<\/p>\n

Until recently, identifying malicious accounts was often quite straightforward. Many campaigns relied on people writing in a second language. So, posts sometimes contained grammatical mistakes or unusual word choices. Detection systems could look for these patterns in the language used.<\/p>\n

But generative AI has changed that. AI systems can now produce fluent, natural-sounding text that is much harder to distinguish from human writing. For example, patterns like use of em-dashes<\/a> and the word \u201cdelve\u201d used to be telltale signs<\/a> of a text being generated by AI. But AIs are adapting, and these older systems are increasingly ineffective<\/a>. <\/p>\n

Trying to detect AI purely from the words people use is becoming a losing battle. We believe the better approach is to look at what a message is trying to achieve. <\/p>\n

Looking for signs<\/h2>\n

Attempts to spread disinformation often work by steering conversations away from their original topic, towards more polarising issues. So, instead of analysing individual words, we set out to build a system that could recognise this phenomenon in online discussions.<\/p>\n

We analysed comments posted beneath BBC News videos on YouTube, a platform that has previously been targeted<\/a> by organised disinformation campaigns.<\/p>\n

For example, imagine a comment about Ukraine\u2019s president, Volodymyr Zelensky, interacting with senior UK political figures: \u201cZelensky must be wondering how many foreign secretaries the UK goes through.\u201d Now, imagine another person responding: \u201cMind you, Zelensky has barely been president for four years. Maybe that\u2019s why the little tyrant bans his opposition.\u201d <\/p>\n

Whether that second point is true or false is not the issue. Instead of responding to the original comment, it redirects the conversation towards a different, more divisive topic. <\/p>\n

This is known as a red herring: introducing an unrelated issue that distracts from the original discussion. These kinds of shift are difficult for conventional disinformation detection systems to identify, because they are not tied to particular words or phrases.<\/p>\n

\n \"Composite
\n 36% of derailing messages online included \u2018red herrings\u2019.<\/span>
\n Roman Samborskyi\/Shutterstock<\/a><\/span>
\n <\/figcaption><\/figure>\n

We manually analysed more than 1,600 comments under BBC News videos, labelling them according to 25 different features of online discussion \u2013 and discovered some clear patterns. <\/p>\n

We found that 36% of derailing messages had red herrings, 65% had leaps in logic known as \u201cnon sequiturs\u201d, and 20% contained personal attacks. They were also much less likely to acknowledge previous comments or express empathy.<\/p>\n

Spotting manipulation<\/h2>\n

The next step was to see whether an AI system could recognise these patterns automatically. We used an AI to catch an AI. <\/p>\n

For every genuine online comment, we asked an AI large language model to generate several reasonable, relevant responses. Returning to the example of UK foreign secretaries, the AI suggested replies such as: \u201cThe current situation in this country must come as quite a shock\u201d or \u201cOne too many?\u201d. Both responded directly to the original point. <\/p>\n

The system then compares the real response with our AI-generated replies. If the actual comment differs substantially, it may indicate that someone is attempting to steer the conversation in a different direction. So, rather than searching for suspicious words, our system looks for unexpected changes in the flow of the discussion.<\/p>\n

How the system works:<\/strong><\/p>\n

\n \"A
\n Discourse derailment is measured by the distance between the real reply and a set of expected replies generated by an AI.<\/span>
\n Krykoniuk, Hopkin-King & Roberts: Using LLMs to identify discourse derailment as a potential cue for disinformation in social media posts (2026).<\/a>, CC BY<\/a><\/span>
\n <\/figcaption><\/figure>\n

We tested this approach using our manually labelled dataset. In our second study<\/a>, the system correctly identified derailing comments around 77% of the time.<\/p>\n

That\u2019s far from perfect, but no detection system is \u2013 particularly when analysing something as complex as human conversation. However, our approach performed around twice as well as existing systems based on word-level sentiment analysis. It also achieved results comparable with the level of agreement between human researchers.<\/p>\n

Our approach is effective because the AI learns what a typical response to a conversation looks like. When a reply unexpectedly changes the discussion, the system can identify that change and analyse patterns that earlier methods couldn\u2019t detect.<\/p>\n

\n

\n
\n
\n Read more:
\n Why science GCSEs matter more than we think in a post-truth age<\/a>
\n <\/strong>
\n <\/em>\n<\/p>\n

\n

Of course, going off topic isn\u2019t necessarily a sign of malicious intent or disinformation. People naturally take conversations in unexpected directions, and there are many legitimate reasons why discussions evolve.<\/p>\n

For that reason, this technology may act as an early-warning system rather than a replacement for human judgment. It could help moderators identify conversations that deserve closer attention \u2013 but any final decisions should remain with trained experts.<\/p>\n

There are also important ethical questions to address. AI systems can reflect biases in the data they are trained on, and they still do not understand conversations in quite the same way that people do. Improving how AI represents and interprets human discussion remains a challenge.<\/p>\n

As AI-generated content becomes increasingly difficult to distinguish from human writing, detecting disinformation requires more than simply searching for telltale words. It requires understanding how conversations work, how they are manipulated, and when someone is trying to quietly steer them off course.<\/p>\n

\"The<\/p>\n

Se\u00e1n Roberts was funded by the AI and autonomy for intelligence, surveillance and reconnaissance (A2ISR) project at the Defence Science and Technology Laboratory (Dstl) through the Defence and Security Accelerator (DASA), which is part of the UK Ministry of Defence, United Kingdom.<\/span><\/em><\/p>\n

Kateryna Krykoniuk was funded by the AI and autonomy for intelligence, surveillance and reconnaissance (A2ISR) project at the Defence Science and Technology Laboratory (Dstl) through the Defence and Security Accelerator (DASA), which is part of the UK Ministry of Defence, United Kingdom.<\/span><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Do you ever see comments on social media that seem way off topic, but still manage to wrench the discussion around to divisive political debate? A discussion about the cost of living suddenly becomes an argument about immigration. A conversation about the war in Ukraine turns into claims about government corruption. It can feel jarring […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1332","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/posts\/1332","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/comments?post=1332"}],"version-history":[{"count":0,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/posts\/1332\/revisions"}],"wp:attachment":[{"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/media?parent=1332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/categories?post=1332"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redzine.co.uk\/index.php\/wp-json\/wp\/v2\/tags?post=1332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}