We dwell in an attention-grabbing technological period, with so many questions on the morality and legality of utilizing AI for inventive content material. As such, many researchers have taken it upon themselves to seek out patterns in what AI generates. They tried to place into numbers what we really feel after we learn or see one thing AI-generated.
Principally, they supplied a solution to the query, “can one thing be too neat? Too clear? Too to the purpose?”
To reply this, let’s take a step again and picture studying a brief story a few girl returning to her childhood house after twenty years.
“She finds a letter hidden behind a unfastened floorboard. The letter reveals a household secret. She confronts her mom, lastly understands why her childhood was so difficult, and learns an necessary lesson about forgiveness”
At first sight, that appears like a wonderfully affordable storyline! The plot is sensible. The characters appear to have logical motivations, and the ending ties all the things collectively.
Nice, however what if we suggest one other model of that very same story? What if the story opens on the mom’s funeral? We soar backward twenty years (as a result of who doesn’t love a time soar?!), then ahead three days, then sideways right into a reminiscence that originally appears irrelevant. The letter contradicts one thing we discovered earlier. The daughter comes to a decision that may be courageous or egocentric, relying on whom you ask. And the ending refuses to clarify what any of it means.
Now, can we are saying for certain which one sounds extra human?
That query is subjective, in fact. Many human writers love tidy endings, and lots of AI-generated tales are delightfully unusual. However what if these intuitions mirror measurable patterns?
On this article, I wish to stroll you thru an enchanting paper I learn not too long ago that solutions that query. The paper is titled StoryScope: Investigating Idiosyncrasies in AI Fiction. As a substitute of searching for suspicious vocabulary, repetitive sentence buildings, or everybody’s favourite AI punctuation mark (and mine!), the em sprint, the authors investigated the narrative selections that form a narrative.
And what they discovered was fairly attention-grabbing! With out additional ado, I am going to stroll you thru the paper.
The issue with present AI detectors
Plenty of AI detection work focuses on how textual content is written. They search for sure phrases, sentence predictability, and whether or not the prose follows recognizable statistical patterns.
These alerts will be helpful. The issue is that they will additionally change. A language mannequin can be taught to keep away from widespread AI clichés, a human editor can rewrite a suspicious paragraph, and immediate could make the machine-generated prose sound way more pure.
We are able to signify an editorial utilizing a characteristic vector:
The place every part would possibly signify one thing measurable concerning the textual content, equivalent to phrase frequency, sentence size, punctuation, or syntactic construction, but when a brand new mannequin modifications its vocabulary and sentence construction, the characteristic distributions shift, and our beforehand helpful choice boundary might now not work as anticipated.
StoryScope asks whether or not we are able to look past these surface-level options by asking questions like:
-
Who drives the plot ahead?
-
How predictable is the sequence of occasions?
-
Does the story comprise subplots?
-
How a lot does the narrator clarify the which means of occasions?
-
Does the story soar round in time?
-
How morally simple are the characters’ selections?
These options describe a narrative’s structure, and altering it’s a a lot larger job than changing a couple of phrases.
Turning tales into knowledge
Right here is the place the analysis turns into attention-grabbing from an information science perspective, as a result of how can we quantize “creativity”? The StoryScope researchers constructed a pipeline to do precisely that.
They used a dataset that incorporates 10,272 writing prompts. For every immediate, they paired a human-written story with 5 AI-generated variations from Claude, DeepSeek, Gemini, GPT, and Kimi. So, 61,632 tales!
The tales averaged roughly 4,753 phrases, lengthy sufficient to review narrative construction at a stage shorter texts won’t assist. They then developed a 3-step course of to extract 304 narrative options from every story.
Step 1: Flip prose into structured representations.
An AI mannequin extracts info equivalent to characters, motivations, occasions, settings, plot construction, and temporal group. As a substitute of evaluating two tales solely by their unique wording, the system creates structured representations of their content material.
Stage 2: Evaluate tales written from the identical immediate.
As a result of every human story has 5 AI-generated counterparts, the researchers can evaluate tales that begin from the identical premise. This helps management for contextual variations.
Stage 3: Convert narrative observations into measurable options.
The researchers establish recurring variations and formulate them as questions with outlined reply choices. For instance, a characteristic would possibly ask whether or not the protagonist’s selections or exterior occasions drive a narrative’s decision, or how usually the narrative jumps throughout time.
The result’s a characteristic vector describing a narrative’s narrative selections, which the researchers then used XGBoost classifiers and feature-importance evaluation to establish the choices that greatest distinguish human and AI writing.
The outcomes
The researchers educated a classifier utilizing narrative options whereas intentionally excluding stylistic options, which achieved a 93.2% macro-F1 rating on the human-versus-AI classification activity.
The mannequin additionally achieved 68.4% macro-F1 when distinguishing amongst all six sources (human authors and the 5 AI fashions), which is tougher to do! As a result of detecting whether or not one thing is AI-generated will not be the identical as figuring out which AI mannequin generated it.
You may be questioning now: what did they discover? They distinguished three patterns AI makes use of when writing fiction.
1. AI likes to clarify the purpose
They discovered that AI-generated tales explicitly acknowledged their themes 77% of the time, in contrast with 52% for human-written tales. Take into consideration the distinction between these two endings:
She lastly understood that forgiveness was not about forgetting the previous, however about discovering the braveness to maneuver ahead.
And:
She folded the letter, positioned it again beneath the floorboard, and left the home earlier than dawn.
The primary explicitly spells out the ethical of the story, and the opposite leaves it open to the reader’s interpretation. Typically a narrative advantages from spelling out its message, however when a mannequin repeatedly chooses specific explanations, that tendency turns into a measurable sample.
2. AI prefers a tidy plot
Additionally they discovered that AI tales have been extra more likely to have simple causal chains and fewer subplots. About 79% of AI tales had no subplots, in contrast with 57% of human tales. AI-generated tales additionally extra ceaselessly resolved conflicts by the protagonist’s selections: 69% versus 46% for human-written tales.
By comparability, human tales have been extra more likely to embrace morally ambiguous characters, nonlinear timelines, and unresolved tensions. For instance, think about two thriller novels: Within the first, each clue factors to the identical conclusion. Every revelation follows logically from the earlier one, and the detective explains the answer on the finish.
Within the second, the primary chapter describes a funeral. The following takes us fifteen years again. A seemingly unimportant dialog turns into essential midway by the novel, and the ultimate revelation makes an earlier scene imply one thing fully completely different.
The second construction is tougher to summarize as a series of trigger and impact, which human writing tends to comply with. This makes AI writing have an attention-grabbing narrative form, a really uniform form.
3. AI will be dramatic!
One of many extra amusing findings was about sensory descriptions! AI-generated tales extra ceaselessly conveyed emotion by bodily sensations and bodily metaphors: 81% versus 38% for human tales.
Consider the acquainted literary mixture of a tightening chest, chilly sweat, trembling palms, and the room abruptly seeming darker. Of their dataset, human writers usually used specific emotion labels relatively than bodily descriptions.
The sample persists
Many individuals generate AI-written textual content after which edit it to sound extra human. They will take away clichés, range sentence lengths, substitute generic transitions, and delete paragraphs that repeat what the earlier paragraph already stated.
However what occurs for those who enhance the textual content with out altering the story beneath?
The researchers answered this utilizing a rewriting technique known as LAMP (an modifying method proposed by completely different researchers in Might 2025). They utilized surface-level edits that take away widespread AI tells to 278 Gemini-generated tales. This achieved 95.5% macro-F1 on the unique tales and 93.9% on the edited variations! Not a really important distinction as a result of it solely modified the writing model, not the underlying story construction.
One necessary qualification, although: the modifying experiment coated 278 tales from one mannequin, Gemini. It’s proof that narrative options can survive a specific form of stylistic modifying, not proof that each AI-generated story will stay detectable after each form of revision.
Can we really use this to detect AI writing?
As a researcher, I state a research’s limitations first. No research is ideal or able to encompassing the entire image. So crucial factor right here is that StoryScope studied long-form fiction. Its outcomes do not routinely apply to emails, essays, documentation, social media posts, or brief articles.
Second, the dataset is constructed in a selected manner: it pairs tales from human writers with AI-generated tales primarily based on reconstructed prompts. It is a managed comparability, nevertheless it would not mirror each manner folks use AI in the true world.
Third, the researchers assign narrative options utilizing language fashions! Lastly, as at all times, a excessive classification rating would not show {that a} specific particular person used AI.
The research’s 93.2% macro-F1 end result applies to its analysis setup. It doesn’t imply a detector can look at any novel textual content and reliably affirm its authorship and originality.
The paper additionally studies stronger outcomes from supervised text-based baselines, together with near-perfect classification scores. That issues as a result of the contribution of StoryScope will not be that narrative evaluation beats each different detector. As a substitute, interpretable narrative options retain substantial predictive energy with out counting on floor model.
Closing ideas
It’s tempting to show analysis like this into an oversimplified conclusion: “AI-generated textual content is at all times unhealthy.” I don’t suppose the info helps that. AI-generated fiction will be fulfilling; a linear plot will be precisely what a reader desires. Specific themes aren’t a literary flaw, and ambiguity is not routinely an indication of genius.
Language fashions can produce convincing sentences whereas nonetheless exhibiting systematic preferences in how they manage tales. We see that on a regular basis after we cope with ML fashions! A mannequin can carry out nicely on a activity whereas exhibiting biases within the patterns it makes use of to get there. Wanting solely on the remaining output rating can cover these patterns.
The authors argue that evaluating generative AI requires greater than asking whether or not its output sounds believable; in addition they encourage us to consider methods to make a narrative extra attention-grabbing by going past changing predictable phrases with uncommon ones, and as a substitute query the construction itself.
Maybe the following step in generative AI growth is not simply producing extra convincing phrases; it’s studying when to not take the apparent narrative path. Which, I’d argue, is a way more attention-grabbing drawback than counting em dashes. So writers (I primarily imply me) can return to utilizing em dashes with out being accused of utilizing AI.
Reference
Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., and Wieting, J. (2026). StoryScope: Investigating Idiosyncrasies in AI Fiction. Convention on Language Modeling (COLM 2026).
