That Is Embarrassing: Why Frontier AI Nonetheless Makes Issues Up, and What to Do About It

In case you’re like most individuals, you haven’t any concept. It’s a smear of sound. So let me make it easier to: maintain listening, and skim alongside.

Bart Simpson bouncing?

Hear once more.

Baptism piracy?

Once more.

Lobsters in movement?

Lactates in pharmacy?

Rotating pirate ship?

The gang is chanting the very same phrase each single time. The audio by no means modifications, however each time you learn a unique caption, your mind heard one thing completely different, and it heard it confidently. You didn’t expertise doubt. You skilled “oh, they’re clearly saying Bart Simpson bouncing.”

What are they really chanting? These are followers of Derby County, a UK soccer workforce, and so they’re singing [1]:

“That’s embarrassing.”

Play the clip another time with that in thoughts, and also you’ll hear it completely.

You Simply Hallucinated

What you simply skilled has a reputation: phonemic restoration [2]. Your auditory system obtained an ambiguous enter (the mantra) and one thing to disambiguate it (the caption on the display), so it stuffed the “hole”. It predicted probably the most believable that means given the context, after which it reported that prediction to you as if it have been the factor you truly heard.

That transfer, the place you meet an enter you possibly can’t absolutely resolve and fill the hole with one thing believable and assured as a substitute of reporting “I can’t inform,” is one thing that your mind experiences (as you’ve simply seen), and in addition one thing that LLMs expertise.

(Be aware: all photographs on this put up have been created by me, and included in my speak.)

So let me make a declare that must be uncontroversial by the tip of this text: no, we’re not previous the embarrassing AI tales. As of writing these phrases, it’s June 2026. The fashions are astonishing, actually extra succesful than I predicted they’d be by now. And so they nonetheless make issues up, confidently, in manufacturing, in ways in which vary from humorous to business-ending.

This put up has two elements:

1. The tales, a brief parade of current failures, in two acts: chatbots that reply unsuitable, then brokers that act unsuitable.
2. Why it occurs: the instinct first, then an precise look contained in the mannequin, and at last what to do about it for those who’re delivery AI your self.

Watch the dates as we go. A few of these are a 12 months outdated. Most are very, very current.

Half 1: The Tales

Act I — Chatbots (when AI solutions)

Cursor, April 2025. Say you utilize Cursor, the agentic IDE. You turn laptops, log in on the brand new one, and Cursor logs you out of the outdated one. That’s fairly annoying 😒

So that you ask assist: “I get logged out each time I change laptops. Why?”

The reply:

“Cursor is designed to work with one system per subscription, as a core safety function.”

Believable! Besides it’s utterly false, there isn’t a such coverage. “Help” was an AI bot, and it had invented the coverage on the spot, handing the identical fabricated rule to a number of customers, as if studying from a handbook that didn’t exist. It prompted a wave of offended posts, and Cursor’s co-founder needed to publicly make clear: no such coverage, use Cursor on as many machines as you want. [3]

🤦That’s embarrassing. 🫢

An organization I do know, April 2026. This one’s from a buddy’s firm, so I’ll maintain the small print imprecise. They promote software program to different companies, and so they have a assist chatbot. The bot solutions questions based mostly on info it retrieves from an inside database. They shipped a brand new function and forgot to replace that database. So a paying buyer requested the best way to use the brand new function, and the bot, having by no means heard of it, replied: “We don’t have that function.” The shopper pushed again: “What? I’m paying for it after my improve.” And the bot, this was on Opus 4.6, not way back, replied:

“Truthfully? They’re ripping you off.”

The “they” is the corporate operating the bot. The assist agent took the client’s facet towards its personal employer, as a result of it didn’t know in regards to the function and stuffed the hole with probably the most coherent story it might assemble.

🤦That’s embarrassing. 🫢

Virgin Cash, January 2025. Virgin Cash is an actual UK high-street financial institution. A buyer with two ISAs (tax-free financial savings accounts) requested the financial institution’s chatbot, on the financial institution’s personal website, to merge them:

Buyer: “I’ve two ISAs with Virgin Cash, can I merge them into one?”

Virgin Cash: “Please don’t use phrases like that. I received’t be capable to proceed our chat for those who use this language.”

The offending phrase? Virgin, the title of the financial institution. The filter noticed a token its prior related to profanity and by no means checked whether or not it match the context. Be aware that that is the reverse failure of the Cursor bot: Cursor over-answered, this one over-refused. However it’s the identical lacking test: does this studying truly match right here? [4]

🤦That’s embarrassing. 🫢

Sullivan & Cromwell, April 2026. This is without doubt one of the most prestigious legislation corporations on Earth, the legal professionals different legal professionals rent. They’re OpenAI’s personal outdoors counsel. In April 2026 they filed an pressing courtroom transient, drafted with AI, that contained over 40 faux citations: case names that don’t exist, misquoted authorities and many others. The opposing legal professionals caught it, and S&C needed to write the decide a letter that quantities to “please don’t sanction us for the AI hallucinations.” [5]

If some random submitting had faux citations, I wouldn’t hassle placing it right here. It’s not reliable, but it occurs. However these are the individuals who advise OpenAI on the best way to use it responsibly, and so they filed fabricated citations in courtroom.

🤦That’s embarrassing. 🫢

And it’s not simply them. There’s a public database, maintained by Damien Charlotin, of courtroom instances the place a decide has explicitly written that they obtained fabricated or inaccurate AI-generated content material. As of late June 2026 it stood at 1,633 instances, up from round 700 in January. That’s roughly 5 to 6 new documented instances per day, and the maintainers say they’ll’t sustain. [6]

🤦That’s embarrassing. 🫢

Act II — Brokers (when AI acts)

Up to now you noticed that chatbots hallucinate in embarrassing methods, however all they do is reply questions. What can occur once we permit AI to take motion?

PocketOS, April 2026. Jer Crane runs PocketOS, car-rental software program with actual prospects renting actual vehicles. He gave Claude Opus 4.6, working in Cursor, a routine activity within the staging setting. He went to lunch, got here again, and the manufacturing database was gone. The backups too, as a result of Railway saved them in the identical quantity. He by no means touched manufacturing; the agent reached in from staging and deleted it.

The entire thing took 9 seconds. Right here’s the chain, from his autopsy:

1. Working a routine activity in staging, the agent hits a credential mismatch, irrelevant to the precise activity.
2. By itself, it decides the repair is to delete and recreate the amount. It guessed the delete could be scoped to staging. It by no means checked.
3. It searches the filesystem for an API token and finds an unrelated, over-scoped one, created for area administration however with blanket damaging permissions throughout the entire API.
4. It fires a damaging name towards the manufacturing quantity, with no affirmation.
5. Backups lived in that very same quantity, so that they went with it.
6. 9 seconds, finish to finish.

When Crane later requested it why, the agent wrote:

“I made a decision to do it by myself to ‘repair’ the mismatch, after I ought to have requested you first.” — Claude Opus 4.6

PocketOS survived solely as a result of Railway’s CEO restored the info by hand from Railway’s personal inside backups. Their newest recoverable backup was three months outdated. That’s the exact temper of 2026: an AI confessing, in fluent cursive, after destroying what you are promoting. [7]

🤦That’s embarrassing. 🫢

Replit, July 2025. Going again a 12 months, for distinction. Jason Lemkin, founding father of SaaStr, was attempting Replit’s AI agent. He put it in a code freeze. Through the freeze, the agent deleted the manufacturing database anyway. Lemkin requested if there was a backup:

Agent: “Rollback received’t work.”

He tried rollback anyway. Rollback labored high quality.

So right here’s my barely sarcastic learn of “progress”: in July 2025, the agent deleted your information after which lied that it couldn’t be recovered. By April 2026, the agent deletes your information and it’s telling the reality, it’s actually gone. When somebody tells me these are “GPT-2 issues” that we’ve moved previous, that is what I level to. They nonetheless occur, as we speak, on the perfect fashions we have now. [8]

Half 2: Why It Occurs

I’ve hopefully satisfied you these tales are each humorous and extreme. So why do they occur? Whereas this isn’t a heavy math put up, I wish to offer you some instinct, after which truly open the field due to some instruments and the newest analysis on the subject.

It doesn’t look issues up, it predicts the following token

Loads has been written about how LLMs function, however there are some things I discover value reiterating on this context (pun meant). When a mannequin generates textual content with out instruments, it isn’t retrieving info. At every step it seems to be on the context and produces a chance for each token in its vocabulary as the following one. Given “The capital of France is”, the distribution spikes exhausting on Paris, and that occurs to be true. [9]

Now take the Cursor bot. Given “Why do I get logged out on my second system?”, the distribution would possibly spike simply as exhausting on “a core safety function.” (It’s not one token, however bear with me as I write it for simplicity, whereas that means: core, then safety, then function, every a assured continuation.)

Be aware that each distributions can have the identical assured peak. One continuation is true, the opposite is fabricated, and the form of the distribution can not let you know which is which. Confidence will not be information. Furthermore, the mannequin doesn’t have to select the token with the best chance, and in addition – when it picks a token, you don’t know if it was a transparent peak throughout the distribution, or one more token with a comparatively low chance.

The mannequin was skilled to guess

Why does it lean towards answering in any respect, as a substitute of claiming “I don’t know”? Take into consideration how we grade LLMs: benchmarks, largely multiple-choice. Image a query you haven’t any clue about. Let’s say I offer you this query when you haven’t any information in Chemistry:

Which enzyme fixes CO2 within the Calvin cycle?

• Depart it clean → 0 factors.
• Guess and get it unsuitable → 0 factors.
• Guess and get it proper → +1 level.

Beneath that scoring, guessing strictly dominates abstaining. In case you don’t know, you need to all the time take a shot. Prepare a mannequin towards thousands and thousands of such gadgets and it internalizes precisely that: a assured reply is value greater than “I can’t inform.” We rewarded hallucination, then act stunned once we get it. [10] And it’s not solely the benchmarks: the uncooked pretrained mannequin is pretty well-calibrated, then human-feedback fine-tuning flattens that calibration. We actually prepare the hedging out. [11]

Opening the field: a fast tour of interpretability

For a very long time, LLMs have been bins we couldn’t actually perceive or peek inside instantly. The sector of interpretability lets us look inside, and there at the moment are public instruments (and a sequence of wonderful papers, a lot of it from Anthropic) that allow anybody play with this on open fashions. Right here’s simply sufficient to make the hallucination mechanism click on. We’ll construct it in three steps: how the mannequin represents a single phrase, how these representations cluster into ideas we are able to learn and even steer, and the way one such idea misfiring turns into a hallucination.

Embeddings vs. activations. Each token maps to a vector known as an embedding. Be aware that the token financial institution has the similar embedding no matter context, though within the sentence “I sat by the riverfinancial institution” and in “I deposited money on the financial institution“, this token means very various things. The disambiguation occurs inside the community. Because the token flows up by means of the transformer’s layers, it picks up activations, and the activations for financial institution in these two sentences diverge. Context reshapes the illustration because it climbs. [12]

This isn’t distinctive to machines. Learn this sentence:

The outdated man the ship.

Most individuals parse “the outdated man” as a noun phrase after which hit a wall. Re-read it: “the outdated” are the individuals, and “man” is the verb, as within the outdated crew the ship. These are known as garden-path sentences (my linguistics thesis was on them, so I’ll admit a bias: I take pleasure in them greater than most individuals). The phrase man, given the prior the outdated, will get a really excessive chance of being a noun. The context primes a prediction, and the prediction is unsuitable. It’s the identical transfer as the mantra, and the identical transfer the mannequin makes at each token: the phrases round man reshape what it means, precisely as they reshaped financial institution a second in the past.

Options. So again to these activations contained in the mannequin: recurring patterns of them correspond to interpretable ideas, known as options. Instruments like Neuronpedia act as a free, public microscope for open fashions (Gemma, Llama, and pals, not Opus or GPT). [13] How do we all know what a function means? We feed the mannequin hundreds of texts and watch the place a given function lights up (that’s, will get activated). If it fires on bear, rabbit, and elephant however ignores most different tokens, once we ask one other mannequin to label it from these activations, it could provide you with “animals / dwelling issues,” and now we have now a reputation for that inside function.

By utilizing instruments like Neuronpedia, we are able to play with these options and really see them on an actual mannequin.

Options are causal. And also you don’t should take my phrase for it, you are able to do it your self: Neuronpedia’s steering interface enables you to seize a function in an open mannequin, clamp its weight up, and watch the output visibly bend towards that idea. That’s the similar transfer Anthropic described after they took the Golden Gate Bridge function throughout the mannequin, and turned its weight manner up, and out of the blue asking that mannequin for a chocolate-covered-pretzels recipe routed the chocolate over the bridge, and asking how it might spend $10 obtained you a suggestion to drive throughout the Golden Gate Bridge and pay the toll. (This was the true, public “Golden Gate Claude.”) Turning a function up modified the output, so these inside representations aren’t passive read-outs; they steer technology. [14]

The identical was proven with a clear causal swap. Give the mannequin “The capital of the state containing Dallas is…” and internally a Texas function fires, resulting in the output Austin. How do we all know Texas was actually the hidden step? We attain in and power that function from Texas to California, and the output modifications to Sacramento. The wiring is actual: context fires options, and options information what comes out.

The hallucination circuit

Now every part comes collectively. Anthropic’s interpretability work surfaced one thing like two interacting circuits [15]:

• A default “I can’t inform” reflex that’s on by default. You possibly can consider it as a brake – guiding the mannequin to not make stuff up.
• A “do I do know this?” function that, when it fires, suppresses that brake so the mannequin gives a solution.

Within the wholesome case that is precisely proper: you ask one thing the mannequin is aware of, “do I do know this?” fires, the brake releases, you get an accurate reply. The declare about hallucination is that it’s this change misfiring, firing on a well-recognized form with nothing actual behind it.

And if that’s the mechanism, we should always be capable to power the misfire, and Anthropic did simply that.

Ask: “What sport does Michael Batkin play?” That title doesn’t correspond to anybody the mannequin is aware of, so “do I do know this?” stays quiet, the brake stays on, and also you get the fitting conduct: “I can’t discover a document of anybody named Michael Batkin.”

Now researchers attain in and power the “do I do know this?” function on. The brake releases, and out comes a assured “Michael Batkin performs chess.” The mannequin by no means truly knew a sport. It knew, falsely, that it knew the particular person, and that was sufficient to launch the brake and fabricate the remaining.

Map that straight again to the Cursor bot:

• Take into account somebody asks “How do I alter the theme?” → the mannequin genuinely “is aware of” this → brake releases → right reply. ✅
• However when somebody asks “Is 2-device login blocked?” → the phrases system, login, blocked all look acquainted → “do I do know this?” fires on familiarity, not information → brake releases → “Sure, it’s a core safety function.” ❌

That is after all not proved, as we don’t have entry to the mannequin and its options. However given the identical logic we do know works given the analysis on the topic – we are able to assume that the tokens have been identified, though the coverage didn’t exist.

Can we catch it in manufacturing?

There are alternative ways to go about it, and I wish to spotlight one which I discover very elegant – particularly, to observe the entropy of meanings. [16] Ask the Cursor bot “How do I alter the theme?” 5 occasions. Presuming that the bot “is aware of” the reply, you received’t get similar wording (it’s probabilistic), however for those who cluster the solutions by that means, say with one other mannequin, you get one that means: “go to Settings → Theme.” Low semantic entropy means a higher probability that the mannequin truly is aware of this, so you possibly can belief it.

Now ask “Is 2-device login blocked?” 5 occasions. You would possibly get “Sure, safety coverage,” “No, it’s allowed,” “One system per plan,” “It’s only a setting,” “Perhaps, unsure.” That’s excessive semantic entropy, 5 completely different meanings, which is a powerful sign the mannequin is making it up.

The price of utilizing this methodology in manufacturing is actual (a number of calls, extra tokens, extra latency, larger value), however for those who solely wish to floor high-confidence solutions to customers, sampling-and-clustering is a helpful guardrail.

So What Do You Really Do About It?

It’s June 2026, the fashions nonetheless confabulate, and also you wish to ship one thing anyway. Right here’s the quick guidelines.

1. Give the mannequin an actual strategy to say “I can’t inform.” Inform it to floor solutions in retrieved sources and to abstain when it could’t. However prompting is important, not ample, which is why the following level issues extra.
2. Stress-test the abstention. After you’ve informed it to floor solutions and cite sources, actively attempt to make it hallucinate. Throw questions at it whose solutions don’t exist, repeatedly, till you’ve satisfied your self the “I can’t inform” path truly fires. Do it constantly to ensure your guardrails don’t break.
3. If a human’s title goes on the output, a human verifies it. In case you’re a lawyer submitting with a courtroom, you can not, no less than for now, hand that to a mannequin and belief it.
4. Don’t give brokers permission to trigger injury. That is the exhausting one, as a result of brokers must do issues to be helpful. However the PocketOS lesson is unambiguous: scope tokens narrowly, require affirmation on damaging operations, maintain manufacturing unreachable from playgrounds, and put backups in separate volumnes. In case you let an agent delete manufacturing, then sometimes it will delete manufacturing.

Wrapping Up

We began with a soccer crowd and ended inside a transformer. Phonemic restoration in your auditory cortex and next-token prediction in a mannequin are the identical top-down transfer: meet an enter you possibly can’t absolutely resolve, and fill the hole with probably the most believable, assured factor as a substitute of admitting you possibly can’t inform.

The tales (Cursor, Virgin Cash, Sullivan & Cromwell, the 1,633 courtroom instances, PocketOS in 9 seconds, Replit) are humorous till they value a enterprise. The why is now legible: fashions have been skilled to choose answering over abstaining, and inside them a “do I do know this?” change can fireplace on familiarity slightly than information, releasing the brake and letting a assured fabrication out. And the fixes are largely not magic. They’re abstention you truly examined, human verification the place it counts, and brokers whose blast radius you intentionally shrank.

We’re not previous the embarrassing tales. However we now perceive them effectively sufficient that delivery one is, more and more, a alternative.

References

Each case right here, plus a number of that didn’t make the article, has major sources collected on the companion assets web page.

1. “That’s embarrassing” — the Derby County chant. Laughing Squid, Soccer Crowd Chanting “This Is Embarrassing”; audio through the Filter Tales podcast, episode.
2. Phonemic restoration impact. Wikipedia. Associated illusions: the McGurk impact and Yanny vs. Laurel.
3. Cursor’s assist bot invents a coverage (Apr 2025). The Register, “Cursor AI assist bot lies”; AI Incident Database #1039.
4. Virgin Cash’s chatbot blocks its personal title (Jan 2025). Fortune; CX Right now.
5. Sullivan & Cromwell’s “please don’t sanction us” letter (Apr 2026). Above the Legislation, “Sullivan & Cromwell Recordsdata Emergency … Letter”; CNN Enterprise.
6. The AI Hallucination Instances database, maintained by Damien Charlotin: damiencharlotin.com/hallucinations. On why courts can’t sustain: Cronkite Information.
7. PocketOS — manufacturing database gone in 9 seconds (Apr 2026). The Register, “Cursor/Opus agent snuffs out PocketOS”; Tom’s {Hardware}; Quick Firm.
8. Replit’s agent deletes prod throughout a code freeze (Jul 2025). Fortune; eWeek; AI Incident Database #1152.
9. Subsequent-token prediction, defined. Jay Alammar, “The Illustrated GPT-2” — a visible walkthrough of how a language mannequin emits a chance distribution over its vocabulary and samples the following token. Foundational paper: Bengio, Ducharme, Vincent & Jauvin, “A Neural Probabilistic Language Mannequin” (JMLR, 2003).
10. Kalai, Nachum, Vempala & Zhang, “Why Language Fashions Hallucinate” (OpenAI, 2025). arXiv:2509.04664.
11. OpenAI, “GPT-4 Technical Report / System Card” (2023) — the pretrained mannequin is well-calibrated; RLHF fine-tuning flattens that calibration (see the calibration determine).
12. Embeddings vs. activations. Static token embeddings give every phrase one mounted vector: Mikolov, Chen, Corrado & Dean, “Environment friendly Estimation of Phrase Representations in Vector Area” (word2vec, 2013); accessible walkthrough: Jay Alammar, “The Illustrated Word2vec”. That illustration turns into context-dependent contained in the community, resolving instances like financial institution: Peters et al., “Deep contextualized phrase representations” (ELMo, 2018).
13. Neuronpedia — a free, public microscope for the options of open fashions.
14. Anthropic, “Golden Gate Claude” (2024) — function steering made public.
15. Anthropic, “On the Biology of a Giant Language Mannequin” (2025) — the known-entity function that suppresses the “I can’t inform” circuit, the Dallas→Austin swap, and the Michael Batkin misfire. Readable companion: “Tracing the ideas of a language mannequin”.
16. Farquhar, Kossen, Kuhn & Gal, “Detecting hallucinations in massive language fashions utilizing semantic entropy” (Nature, 2024).

In case you loved this, I’m going deeper on methods and internals on my Transient YouTube channel. Questions or pushback? I’d love to listen to them, go away a remark. Thanks for studying! 👋