If you have used AI writing tools for more than a year, you have probably felt something that the benchmarks refuse to confirm. The models got smarter. They plan better, they follow instructions better, they can hold a legal brief or a codebase in their heads. And somewhere along the way, their fiction got worse. Not broken, exactly. Worse in a specific, slippery way: every story arrives in the same voice, wearing the same phrases, reaching for the same tidy endings. A shiver runs down someone's spine. The air is thick with tension. Little does she know.
Writers noticed this before researchers did, because writers read output the way editors read slush. You cannot fool someone who has seen four hundred first pages. The pattern they spotted has a real, mechanical cause, and it is not a mystery or a conspiracy. It is a training method doing exactly what it was designed to do, at a scale nobody fully priced in. The method is called reinforcement learning from human feedback, and its effect on prose is mechanical enough to trace step by step. At the end, we will tell you what we built, briefly, because this page is the receipt for a claim our homepage makes.
The three-stage assembly line
Every modern language model goes through some version of the same assembly line, and each stage leaves fingerprints on its prose.
Stage one is pretraining. The model reads a colossal amount of text, trillions of words, and learns one skill: predict the next token. A model that has only done this is called a base model, and base models are strange, feral things. Ask one to continue a scene and it will not try to please you. It will try to continue the scene the way the internet might have, which means it can veer brilliant, boring, obscene, or unhinged, sometimes in the same paragraph. What a base model has is the full distribution of human writing. Every register, every cadence, every bad habit and every good one, weighted roughly as they occur in the wild. Base models have range. What they lack is obedience.
Stage two is supervised fine-tuning. The lab collects tens of thousands of example conversations, prompt and ideal response, often written by hired contractors, and trains the model to imitate them. This is where the model learns to answer questions instead of continuing them. It is also the first place a house style creeps in, because those contractors were following a style guide.
Stage three is the one this article is about. Reinforcement learning from human feedback, RLHF. The lab shows human raters two responses to the same prompt and asks which one is better. Do this a few hundred thousand times and you can train a second model, called a reward model, that predicts which response a human would prefer. Then you point the language model at the reward model and let it practice: generate a response, get scored, adjust, repeat, millions of times. The model is no longer learning to imitate human text. It is learning to maximize a score. There are technical variants (PPO, DPO, and a family of successors), each with some mechanism that punishes the model for drifting too far from its starting distribution, so it cannot simply collapse into gibberish that happens to score well. But the essential shape is the same everywhere: prose is now being optimized against a proxy for human approval.
This method is why chatbots are usable at all. Before RLHF, getting a model to do anything took prompt sorcery. After it, you ask and it does. InstructGPT proved the recipe in 2022, ChatGPT took it mainstream, and every serious lab has run some version of it since. For most tasks, a little uniformity is a fair price for obedience. For fiction the price is quietly catastrophic.
What the reward model actually rewards
Picture the rater. They are doing piecework, maybe a minute or two per comparison, judging responses to prompts they did not write, in domains they may not know. Which response do they prefer? Overwhelmingly, predictably, the one that is clear, confident, well-organized, agreeable, and complete. The one with a beginning, middle, and end. The one that resolves.
None of those preferences are wrong. Compounded over millions of updates, they become a gravitational field. Researchers have documented the biases that emerge: reward models favor longer responses, favor lists and headed structure, favor hedged safety over risky specificity, favor agreement with the user's framing. The model under training discovers these preferences the way water discovers a slope. It is not trying to game anyone. Gaming the score is the entire objective it was given.
Now apply that field to a scene. The rater compares two continuations of a thriller beat. One is jagged, withholding, a little confusing on purpose, the way tension actually works on a page. The other is smooth, vivid in a familiar way, emotionally labeled so nobody could miss the point, and it lands on a note of resolution. The rater, reading fast, picks the smooth one. Almost anyone would, at comparison speed. But a novel is not read at comparison speed, and the qualities that win a sixty-second judgment are close to the opposite of the qualities that keep a reader up at three in the morning. Menace requires withholding. Voice requires risk. A chapter that resolves every paragraph has no reason to be turned.
There is a name for what happens to the model's output distribution under this pressure: mode collapse. The base model held thousands of ways to write a knock at the door. Preference training concentrates probability on the narrow band of continuations that score well, and the rest of the distribution withers. You can turn the temperature up, but sampling harder from a collapsed distribution just gets you the same house style with more typos. This is why prose from different labs converges on the same tics. The em dash cadence. The “not just X, but Y” construction. Rule-of-three sentences, everywhere, because the reward model learned that humans find triads satisfying. Characters who let out breaths they did not know they were holding, whose hearts pound in their chests, the anatomically redundant chest included, because those phrases sat in the high-probability center of “vivid emotional writing” and the training sharpened them from common to near-mandatory. The compounding runs through the labs' own pipelines too: each generation of models is tuned partly on data produced or filtered by the last, so a stylistic bias, once in, tends to be inherited and sharpened rather than washed out. And to the extent AI text now floods the open web and leaks past pretraining filters, the loop may be tightening from the outside as well. The clichés compound, model to model, like interest.
One more effect deserves a name, because writers feel it daily: the flinch. Push a preference-tuned model toward anything genuinely dark, morally uncomfortable, or unresolved, and it slides toward safety. Villains get redemption arcs nobody asked for. Grief scenes get silver linings. The knife stops an inch short. Human raters preferred the safer continuation, and the automated feedback that increasingly supplements them, models grading models against written principles, prefers it even more consistently. The model learned that danger is a defect. In a customer support bot it is. In a thriller it is the whole product.
Then the reasoning era made it sharper
In May 2024, GPT-4o was the state of the art in the chat paradigm. By late 2024, the frontier had pivoted to something structurally different: reasoning models, starting with OpenAI's o1 and followed within months by DeepSeek's R1, Google's thinking-mode Gemini variants, and Anthropic's extended-thinking Claude models. That pivot rewired the optimization target, and fiction lost.
Reasoning models are trained with reinforcement learning too, but of a different flavor: reinforcement learning with verifiable rewards, RLVR. Instead of a reward model guessing what a human would prefer, the reward comes from things a computer can check. Did the math answer verify? Did the code pass the tests? Did the final answer match? The model is trained to generate a long private chain of thought, thousands of tokens of working, before it answers, and the RL shapes that thinking toward whatever gets checkable answers right. This is why reasoning models jumped so dramatically on math, code, and logic. The reward signal was finally honest. You cannot flatter a unit test.
But notice what happened to the optimization target. RLHF at least pointed at human taste, however badly its raters proxied it. RLVR points at verifiability itself. Fiction is the least verifiable artifact humans make. There is no unit test for menace. And a detail that matters: reasoning models are not built from scratch. They are trained on top of chat models that already carry the RLHF house style, so the collapsed prose comes along as inherited cargo. The preference tuning these models still receive is real but secondary; the dominant training pressure now pours into structured, checkable, stepwise correctness. Worse, the habits of verified thinking leak into the prose register through both available channels: the reasoning RL updates the same shared weights that produce every visible sentence, and the chain-of-thought format itself trains a stepwise, explanatory register that bleeds into the answer. Ask a reasoning model for a chapter and you can feel it solving the chapter: the plot beats arrive like a satisfied checklist, the metaphors are chosen the way an engineer chooses a fastener, and the voice sits at a polite, explanatory altitude that never quite descends into a character's skin. The model plans better than any writing tool in history, and the plan is exactly what it sounds like when a plan writes prose.
None of this is an accusation of laziness at the labs. It is a description of the economics. Benchmarks are verifiable, enterprise workloads are verifiable, agentic coding is verifiable, and that is where the frontier competes. Fiction ships in the same weights as an afterthought, inheriting whatever style the reward signals happened to sculpt. The smartest models got worse at fiction because nobody was paying them to be good at it, and everybody was paying them, in effect, to be good at things that pull in the opposite stylistic direction.
Why open models come in thinking and non-thinking flavors
If you have browsed open-weight models lately, you have seen the split everywhere. DeepSeek ships V3-class chat models and R1-class reasoners from the same lineage. Qwen3 shipped with a hybrid switch, thinking mode on or off per request. Labs publish “instruct” and “thinking” variants of the same base like a sedan and the sport package.
The split exists because the two behaviors are trained differently and priced differently. Thinking is expensive: a reasoning model may burn thousands of hidden tokens before its first visible word, which costs real money and real latency, and for most everyday requests buys nothing. So the ecosystem settled on offering both: a fast, direct variant tuned with ordinary preference methods, and a deliberate variant with RLVR-shaped chains of thought for the problems that need it.
For creative writing, this split is unexpectedly good news, and it points at a kind of choice the closed labs never really offered. A closed lab sells you its aligned chat model, thinking or not; the alignment itself is not negotiable. The open world publishes the whole ladder, base weights, instruct tunes, reasoning variants, and lets you pick how much alignment sits between you and the base distribution. That ladder feeds a thriving after-market of fiction tunes, community models built on open bases and instructs, trained further on curated novels and roleplay data, and often explicitly de-tuned away from assistant habits. These are the models with names you will not see in an enterprise deck, and on prose they routinely embarrass systems fifty times their size, for the simple reason that the frontier abandoned this territory. Their distributions stayed wide where it matters. A well-chosen 70B fiction tune does not reach for the same sentence every time, does not flinch at a knife, and does not sound like the minutes of a meeting about a scene. What it will not do is hold ninety thousand words of continuity in its head. Voice and memory turn out to live in different models now.
That sentence is the entire design problem of AI-assisted fiction in 2026. The models that remember cannot write. The models that write cannot remember.
What we built instead
Unsloppy is an answer to exactly that split, so here is the architecture in one paragraph, offered as engineering rather than as a pitch.
We never ask one model to be both the writer and the clerk. Reasoning-class models do what RLVR made them good at: they read your draft as you write, maintain a story bible nobody has to type, and assemble a small, curated brief for each scene, who is in the room, which promise from chapter six is still open, what rule your world runs on. That brief, and not your whole manuscript, goes to a voice: one of a hand-picked catalog of fiction-tuned open models, each a distinct register, chosen by you. Keeping the context small is not a cost dodge; a focused brief is what keeps a voice model in character at word ninety thousand, because the drowning-in-context failure is real even for models with million-token windows. And because even the best fiction tunes carry residual tells, a final structural pass, we call it the Humanizer, strips the known artifacts of preference training out of finished prose: the em dash cadence, the negative parallelism, the breath she did not know she was holding. It maintains a kill list. It is public, on our homepage, and it is the reason this article exists: several thousand words of theory, posted as the receipt for ten struck-through phrases.
The deeper point survives without the product. Preference optimization gave us assistants worth talking to, and it quietly standardized the most personal thing text can carry, which is a voice. The reasoning era then aimed the industry's entire optimization budget at everything fiction is not. None of that reverses soon, because the incentives that caused it are getting stronger, not weaker. If you write, the practical conclusion is simply this: the era of one model for everything was a two-year anomaly. Pick your thinkers for what thinking is for. Pick your voices from the corners of the ecosystem where the distributions are still alive. And read your own pages back cold, the way you would read anyone else's, because the tells are learnable, and once you see them you cannot unsee them.
That last habit costs nothing. It is also, not coincidentally, how you will know whether any of this, ours included, deserves your name on the cover.
Unsloppy is a writing desk for novelists: your words, a story bible that maintains itself, and a catalog of fiction-tuned voices with the machine tells stripped out. No credit card required. Bring the chapter that stalled and watch the kill list go to work on it.