StoryScope detects AI fiction that reads perfectly human

AI fiction detection usually leans on style: em dashes, the word “delve,” a taste for purple prose. StoryScope, a pipeline from the University of Maryland and Google DeepMind, throws all of that away. It scores 93.2% macro-F1 separating human stories from AI ones using plot structure alone, and rewriting the prose barely dents the score.

What is StoryScope and how does it detect AI fiction?

Hachette pulled the horror novel Shy Girl after it was flagged as roughly 78% AI-generated, the first commercially published novel canceled over AI claims . Nearly 20% of a 14,000-book sample of self-published Amazon novels got the same flag, a share that climbed 41% in a year. So publishers want a check that a rewrite cannot defeat.

StoryScope is that attempt. The paper from Russell and colleagues describes a three-stage pipeline. First it turns each story into a structured template. Then it compares templates across the six writers who answered the same prompt. Finally it keeps only the narrative features that tell them apart.

The features come from the NarraBench taxonomy. StoryScope uses 10 of its aspects: agent, social network, event, plot, structure, setting, time, revelation, perspective, and style. GPT-5.1 handles the templating and the feature discovery. Gemini 3 Flash then scores every story against all 304 surviving features.

The corpus is the expensive part. Researchers took 10,272 writing prompts already answered by a human author. Five models then wrote their own version of each one: Gemini 3 Flash, Kimi K2.5, DeepSeek V3.2, Claude Sonnet 4.6, and GPT-5.4. That gives 61,608 stories averaging 4,753 words, at a compute bill of $4,400.

The narrative variant deletes all 47 style-related features and runs on the remaining 257. A style-only variant keeps just 39. The StoryScope code and data are public, including the prompts and all 51,336 AI stories. The human originals stay behind, because they come from Books3.

How well does narrative-only detection actually work?

The binary human-versus-AI results, by feature set:

Narrative features alone recover 97% of what the full model scores. Thirty core features still hold about 91% of the narrative model’s number. Those thirty are printed in the paper’s appendix, so an editor can read them as a checklist.

Plain text classifiers beat this handily: ModernBERT hits 99.9 macro-F1, stylometry with XGBoost 99.8, TF-IDF with XGBoost 99.7. Only zero-shot Binoculars lags, at 55.9. Tools like Pangram and GPTZero work the same surface ground. Pangram is what flagged those Amazon novels. StoryScope is slower, pricier, and less accurate than all of them. Its selling point is explanation. The classifier is XGBoost with SHAP attribution, so every verdict traces back to named narrative choices.

Length is not doing the work either. On a test subset matched to human story lengths, the narrative model scores 93.2 before and after matching. Guessing which of the six sources wrote a story is harder. Narrative alone still hits 68.4 macro-F1, and 77.3 with style added back, against a 16.7% chance line.

The narrative tells that give AI fiction away

Each feature is a plain question about the story, and the gap between human and machine answers is often enormous.

AI finishes the job for you. It states the lesson, resolves the plot through the hero, and lands on acceptance. Humans leave more of it unresolved and write more ambivalent protagonists.

Timelines diverge just as sharply.

“Show, don’t tell” is what every workshop teaches, and the models have absorbed it too well.

Human writers also point at the world outside the page. They cite actual books, authors and places, break the fourth wall at nearly twice the AI rate, and speak to the reader directly four times as often. AI writes as though nobody is in the room.

Why editing the prose does not hide the AI

Style-based detection is losing ground fast. GPT-5.4 cut its em-dash habit sharply, and fine-tuning a model to imitate human style drops detection on creative writing from 97% to 3%. Every surface tell has a shelf life.

So the researchers tried to break their own classifier. They ran 278 Gemini stories through LAMP , a span-level rewriting framework. LAMP hunts seven kinds of AI writing tic, among them cliché, padded exposition, and purple prose. It works from 25 examples by professional writers. Gemini itself did the rewriting.

The edited stories still got caught at 93.9 macro-F1, against 95.5 on the originals. A professional-grade de-slopping bought 1.6 points.

Every model has its own storytelling fingerprint

Ask the classifier which of the six sources wrote a story and the sources separate unevenly.

Claude is the calmest writer in the set, with the least escalation of events and the most even voice of any source. In 62% of its stories it honors literary tradition rather than subverting it, against 39% to 56% elsewhere. Epilogues are a habit, dream sequences almost never appear.

GPT leans social. Gossip and rumor drive the plot in 64% of its stories, versus 44% to 55% for the rest, and a favorite frame is someone looking back on events from decades ago. GPT also subverts expectations more than its peers and leaves reconciliations hanging.

Gemini writes the bleakest worlds, with 88% of its settings tagged bleak and oppressive. It still produces the tidiest endings and the longest wind-downs. DeepSeek gives away crucial context early that other models hold back.

The six most-confused source pairs are all AI against AI. The biggest, Gemini against DeepSeek, accounts for 222 and 207 stories. The most common human mix-up, human read as Kimi, hits only 46.

AI models have converged on one way to tell a story

Turn each story into a vector of narrative features and the geometry gets blunt. Mean distance between the human centroid and an AI centroid is 1.6 times the mean AI-to-AI distance, 6.6 against 4.3. Even the closest human-AI pair sits farther apart than the two most distant AI models, 6.2 against 6.0.

Human stories also spread wider. Their mean distance to their own centre runs 22% above the average AI radius, 33.2 against 27.4. Their median 10-nearest-neighbor radius is 1.13 times larger.

Scoring each story by how far it sits from its 25 nearest neighbors gives the same picture. Human stories average a rarity percentile of 0.71 against 0.49 for AI, a Cohen’s d of 0.83. Humans also hold 24.7% of the rarest tenth of the corpus, against 7.1% for AI. Given one prompt and six answers, the human answer was the strangest of the six 57.8% of the time, against 16.7% by chance.

That last number is why the paper reaches for copyright law. US precedent asks for a minimal degree of originality. Copyright Office guidance ties protection to real human creative control. The authors offer rarity in narrative feature space as a number you could measure against that bar.

What this changes for writers, editors and publishers

For an editor screening submissions, the cheapest manual checks are the ones with the widest gaps. Watch for a moral the narrator spells out, a single-track plot with no subplots, emotion carried only by the body, and an absence of named real books, places, or brands. Any one of those proves nothing on its own, but three together are worth a conversation.

For writers using AI, polishing sentences leaves the signal intact. The edits that help are structural, and they take real work: rebuilding the timeline, adding a subplot that pays off, or leaving the ending open.

A 93% classifier gives a probability, so it belongs at the start of a review. The pipeline also needs long text, roughly 5,000 words per story, so it says nothing about a blog comment. It costs real money to run, and cheap text classifiers still beat it on raw accuracy.

The human stories come from Books3, a dataset of pirated books. The authors decline to release those stories over copyright concerns while using them to build a copyright argument. They also disclose that Claude Code and Codex helped write the paper and generate some of its tables and plots.