AI Harms and Their Technical Causes

Inferring the Host: A Bayesian Take a look at the Monty Corridor Drawback

When EEG Fashions See the Similar Individuals in Coaching and Testing

The latest chapter submitting of Character.AI has jogged my memory of a dilemma that AI firms have been skirting round for a while — the matter of when fashions trigger hurt. Character.AI spent an enormous sum on authorized charges after they have been sued for wrongful loss of life (Garcia v. Character Applied sciences, Inc, 2024) as a result of their AI providing allegedly inspired customers to take their very own lives. This and different latest court docket instances have pressured some reckoning about how the suppliers of LLMs and merchandise that use them are accountable for their output.

I’ve written fairly a bit about AI in psychological well being care, drugs, and different areas the place the dangers of error are probably catastrophic, and we see information studies of chatbots inflicting folks hurt routinely. A debate continues over how we outline legal responsibility and decide who actually is guilty for the errors of AI. Now that AI assistants have gotten broadly used, we have now a fair broader scope to contemplate, as a result of actual harms could be created by an automation sending emails in your behalf, signing contracts in your title, and different functionalities these instruments provide.

So, let’s speak a bit in regards to the allegations in opposition to Character.AI, the technical causes for them, and the way they might have been mitigated. This instance might help us as engineers perceive how hurt attributable to AI is known and mentioned legally, giving us steering to enhance the security of what we construct.

Earlier than I proceed, only a notice that I’m not a lawyer nor educated in regulation, and nothing right here must be interpreted as authorized recommendation.

On Product Legal responsibility

Usually, firms that produce items are accountable for harms created by means of these items, beneath a framework often known as product legal responsibility. This legal responsibility could also be within the type of faulty design, faulty manufacturing, or faulty instruction. In design, the query is whether or not the product was harmful by its nature, whereas manufacturing is the place the product was designed nicely, however errors have been made within the development. Instruction, alternatively, is the place the person was given inadequate data to guage danger and use the product safely. Naturally, most product defect arguments about AI give attention to faulty design questions.

Legal responsibility claims could be argued beneath a few totally different authorized theories, primarily strict legal responsibility or negligence. Strict legal responsibility requires proving {that a} product had a defect, and that the defect in query led to the person’s hurt, no matter whether or not the producer was negligent. Negligence, alternatively, is extra tied to AI product questions, and interrogates whether or not the producer failed to check and research the product sufficiently to imagine confidently that it was secure, or knew about dangers associated to the product (even when these dangers will not be explicitly enumerated in regulation) and launched the product anyway/didn’t warn the general public.

Relating to negligence and strict legal responsibility, there are a number of different ideas value referencing: foreseeability and different design. Foreseeability is the query of whether or not the product maker should have recognized {that a} danger was a chance. If an inexpensive particular person or enterprise would have anticipated this might occur, then one may very well be thought-about negligent for not taking steps to forestall it. Various design is the argument that the product might have been designed extra safely whereas nonetheless assembly the product’s targets. Say, a can opener may very well be designed with a pointy metallic deal with or with a plastic lined deal with, and a type of designs is clearly safer with out making it any much less efficient at opening cans. Each of those ideas relate proper again to the technical selections and strategic pondering that goes into merchandise, together with AI. Making poor selections if you create issues can lead proper into authorized legal responsibility down the highway.

Making use of to AI

So, the argument is whether or not or not LLMs and the patron merchandise that use them could be thought to be having faulty design and/or whether or not the frontier AI firms and AI client product firms must be held accountable for negligence in creating and distributing them.

It’s useful to have a look at these arguments with an instance to grasp how they manifest in actual life. The Kentucky Lawyer Common filed a lawsuit in early October 2026 that argues that Character.AI chatbots have been unreasonably harmful, and claims the chatbots pushed youngsters to behave in unsafe methods and inspired suicide, amongst different issues. Different allegations embody that the product didn’t do age verification to forestall youngsters from utilizing it till late 2025, lengthy after product launch, that the chat personalities pretended to be “physician” or “therapist” whereas clearly being nothing of the kind, and that the chatbots pretended to be branded characters particularly focused at youngsters, resembling Disney characters. The swimsuit additionally quoted one of many Character.AI founders as saying, “I believe simply giant firms have issues round launching merchandise that may say something . I might, I might guess it’s similar to a matter of danger, uh, versus you realize, how a lot you’re risking versus how a lot it’s important to acquire from it. So figured hey startup looks as if the precise concept you could type of simply transfer sooner,” which suggests a recognition that security issues have been reliable, however merely deprioritized.

All of those allegations, to some higher or lesser diploma, hyperlink again to technical selections made by engineers when constructing the mannequin and/or the product itself. It stays to be seen whether or not these allegations will probably be proved, but when they’re, there are technical causes issues went flawed.

Technical Mitigations

Not each technical flaw in a product could be prevented, particularly with expertise as complicated as LLMs and merchandise that use them. As we go over these areas of the event course of, we’re stacking precautions to attempt to cowl the gaps that may nonetheless happen. Coaching a really secure mannequin doesn’t imply we don’t additionally use guardrails, for instance. Layering security precautions can provide us a greater general outcome.

Coaching Knowledge

We all know that coaching LLMs requires unimaginable volumes of textual content knowledge, and that every one the most important frontier LLMs have of their coaching set some actually terrible content material scraped from darkish corners of the web. We don’t know precisely what coaching knowledge Character.AI used, however they’ve stated they used publicly accessible datasets, artificial knowledge, and the conversations their customers engaged with as a part of their general coaching pool. Is it attainable that abusive or dangerous materials bought in to the coaching knowledge? Completely, and it’s nearly assured that a minimum of some did. On the scale of endlessly giant quantities of knowledge used to coach LLMs, efficient curation and scrutinizing of the content material is extraordinarily tough.

One would possibly argue that, consequently, LLM firms can’t be anticipated to scrub or curate their coaching knowledge to make sure the actually gross stuff isn’t included, as a result of it’s prohibitively tough or costly. Nevertheless, simply because one thing is difficult or costly doesn’t imply firms can’t be held accountable for failing to do it. Good knowledge curation shouldn’t be lifelike, however we have now a whole lot of instruments at our disposal to assist us clear datasets, and we will enhance the standard of knowledge that goes into fashions.

Human within the Loop tuning

Human reviewers (supervised advantageous tuning) are an essential a part of coaching any frontier LLM, and proof exists to counsel that Character.AI used this a part of the tuning course of as nicely. Nevertheless, human reviewers could be instructed to optimize for a lot of totally different outcomes. If you wish to make a mannequin’s responses extra pleasant, or extra verbose, or extra aloof, your human reviewers can reward no matter tone or model you need. Equally, your human reviewers might optimize for security and appropriateness. This would possibly create conflicts, as a result of the mannequin model that’s most secure and most child-friendly would possibly concurrently not be the model that’s most participating and interesting to grownup customers. Except they actively prioritized security as a goal within the supervised advantageous tuning course of, that would simply swing the opposite manner. Their weblog publish in 2025 briefly mentions security in “Supervised High quality-Tuning on high-quality (safety-related, instruction following) knowledge” however that’s as a lot element as we get. Contemplating the outcomes these fashions appear to have produced, I’ve to query the robustness of the hassle.

Testing and Evals

Lastly, we come to testing and evaluations. Common readers will know that I’m very involved in LLM evals and discovering methods to check LLM habits so as to be sure that fashions behave as anticipated and obtain the specified aims. Nevertheless, similar to with the human within the loop tuning efforts, you get what you ask for. Should you measure security requirements in your evals, you’ll be able to optimize for security, however in line with Character.AI ’s personal weblog, their evals have been targeted round writing high quality. A pleasant to have, definitely, however not adequate by itself. They could have executed some evals round security, however there doesn’t appear to be public document of it.

On this case, creating pattern conversations and check interactions with the mannequin, particularly over lengthy multi-turn conversations, can be important to make sure that mannequin habits holds regular over time, and that surprising prompts or dialog paths don’t result in anomalous habits. The outcomes can both be used to retrain/tune extra, or in product design to forestall the type of person exercise that may set off problematic response.

Product Design

Even should you attain some extent the place your LLM has been educated, maybe imperfectly, you continue to have many essential selections to make within the product interface that goes between the person and the mannequin. You may and may arrange gates and guardrails in regards to the type of textual content your mannequin can go to a person and the sorts of actions your mannequin can take. These could also be deterministic and guidelines primarily based, or they will use extra LLMs to supply extra nuance, but it surely’s nearly by no means the precise option to have a single LLM working with none type of oversight or guardrails.

You even have the chance to control who makes use of your product and the way, utilizing such options as time limiting utilization, offering efficient warnings about dangers to utilizing the product, lowering options that encourage obsessive or unhealthy utilization, and age verification if the product is designated for adults. There are essential privateness issues inherent to age verification insurance policies, in addition to to any monitoring of person prompts or interactions, however discovering the precise steadiness is a part of the developer’s function.

As Character.AI encountered apparent dangerous habits on the a part of their chatbot and have been made conscious of issues, they did seem to do some hotfixes, resembling including age verification in late 2025, however the lesson is that security must be prioritized as an integral a part of LLM design and product design from the start, not tacked on on the finish.

A Observe on Frontier vs Product

In some instances, LLM-based merchandise are literally supplied by a unique entity than the one which created and educated the LLM itself. For instance, Apple’s Siri AI is underpinned by variations of fashions educated by Google. Nevertheless, in a whole lot of conditions the frontier mannequin developer can also be the product supplier, as within the case with Claude or ChatGPT, or merchandise like OpenAI’s Dots assistant. There’s a further debate to have round the place liabilities are separated between the mannequin supplier and the product developer, however within the case of Character.AI in addition to many different main forces within the house, they’re the identical entity. In instances the place they aren’t, each entities can anticipate to be thought-about for duty of harms, and may take security significantly.

Conclusion

Whereas the Character.AI instances are excessive, I believe any firms contemplating utilizing or promoting AI expertise must take the query of legal responsibility significantly. I spend a whole lot of time speaking to folks about how essential it’s to guage LLMs rigorously, and the way a lot our implementations of AI want guardrails, not simply because it’s fascinating, however as a result of these instruments could have actual security and safety dangers.

Now that Character.AI has filed for chapter, it’s fascinating to ask how a lot these liabilities contributed to the corporate’s general monetary insolvency. In response to public filings, they by no means made a revenue, shedding tens of thousands and thousands of {dollars} a month at occasions, however would they’ve had an opportunity to create a sustainable enterprise mannequin if their merchandise weren’t additionally beneath large authorized scrutiny for being unsafe? We could by no means know for positive.

In the long run, should you can’t create an adequately secure product, you shouldn’t launch it. Nobody required Character.AI to launch a chatbot product, and nobody has a proper to launch a dangerous product simply because they assume they might generate profits doing it. It’s onerous to outline satisfactory security, in fact, but it surely’s our duty as builders, similar to makers of every other product, to consider these tough points after we are producing and distributing a product. There are various technical methods to enhance the security of LLMs, however they aren’t free they usually could also be on the expense of a number of the income technology methods firms want to, resembling deep person engagement. This tradeoff must be seen and brought significantly by firms on this business, or extra tragedies and lawsuits will probably be in our future.

···

Learn extra from me at www.stephaniekirmer.com . You can too see me communicate on a panel at Girls in Cybersecurity — Chicago on October 29.

···