Next-concept modeling writes the gist first, 21x faster

2026-10-21 9 minutes

Stack of glass layers with a few big blocks on top and a detailed tile landscape at the bottom, beside a snail machine laying single tiles

Next-concept modeling (NCM) writes text the way an artist sketches: a rough gist of the whole passage first, then finer detail. In 12 steps instead of 128, it beats the best diffusion model on a standard test and runs 21x faster. It also solves 86% of hard Sudoku boards. Still, it’s a small research model.

Key Takeaways

NCM writes a rough version of the whole text first, then sharpens it.
It matches or beats diffusion models while running 10 to 31 times faster.
A tiny model solves 86% of hard Sudokus, against 1% for word-by-word models.
On translation, the plain word-by-word model still wins at this size.
The biggest model tested has 725M parameters, far below chatbot scale.

What is next-concept modeling?

Next-concept modeling comes from the NCM paper, published on arXiv on 9 October 2026. The authors work in Christopher Ré’s Hazy Research lab at Stanford. One of them, Simran Arora, is also at Together AI.

The system has two parts. First, a tokenizer squeezes a passage into a stack of codes. The top level holds just a few codes, and each level below holds more. The bottom level has one code per token, which is a word or piece of a word. Second, a transformer model learns to build that stack from the top down.

Inside each level, the model guesses every code at once in a single pass. So a 128-token passage takes 12 steps, not 128.

“Concept” has a narrow meaning here. It means the rough information in the top levels of the stack. Meta’s Large Concept Models, a paper from December 2024, uses the word for whole-sentence summaries instead.

The idea comes from image generation. An image model called VAR, from a paper of April 2024, draws a picture as a blurry thumbnail and then sharper versions. NCM brings that to text. That is harder, because a sentence has no natural “blurry” version the way a photo does.

Why one word at a time became the bottleneck

Most language models are autoregressive: they write one token per step, and each step waits for the one before. A 1,000-token answer means 1,000 calls to the model, one after another.

The best-known fix keeps that model and guesses ahead. A small draft model proposes a few tokens, and the big model checks them all in one pass. Google’s look back at speculative decoding, posted on 6 December 2024, puts the gain at 2x to 3x on translation and summary tasks.

Reasoning models make the wait longer. Their hidden “thinking” tokens are billed as output tokens and take up room in the model’s memory.

Diffusion language models are the other camp. They start from a fully blanked-out sequence and fill in many positions per step. They already run at scale: LLaDA, from February 2025, reaches billions of parameters, and Mercury, from June 2025, is sold as a paid service.

Three panels comparing text generation: one token added per step, diffusion filling scattered positions across all slots, and NCM filling one wide block then 2, 4 and 8 narrower blocks

Diffusion has three weak spots:

Many steps. Cut the number of passes and quality drops fast.
Costly steps. Each pass reworks the whole text. The caching trick that speeds up word-by-word models doesn’t work here.
Huge training bills. Strong diffusion baselines train on more than 500 billion tokens of web text.

How NCM builds a sentence from coarse to fine

Here is NCM translating a German sentence into English, shown after selected steps:

Step 1: “However, it is a important to the the the and the.”
Step 3: “However, it is a concern for small small small medium-sized and-sized-sized and and and…”
Step 6: “However, there is a concern that medium medium-sized enterprises will remain remain remain surveillance surveillance surveillance capabilities”
Step 11 (final): “However, there is concern that small and medium-sized enterprises will remain vulnerable to hacking and surveillance.”

Against the human translation, the final output scores 77.8 BLEU, a translation score where 100 means a word-for-word match.

By step 3 the topic is in place, and later steps only clean up repeats.

Three design choices make this work:

Each level fixes the last one’s errors. A level stores only what the levels above it got wrong. During training, the stack is cut short at random. That forces the top levels to carry the key meaning.
Early steps are cheap. The top levels hold very few codes, just one for the TinyStories model. Each level also looks only at the combined picture from the levels above.
The model learns to recover from mistakes. Guessing a whole level at once lets several errors slip in together. So in training, some input codes are swapped for random ones, and the model learns to fix them.

NCM vs diffusion models: the benchmark numbers

The paper tests NCM on five tasks against the strongest rival on each. Perplexity measures how natural a judge model finds the text, so lower is better. For summaries (ROUGE-L), translation (BLEU) and Sudoku, higher is better.

Task	Rival	NCM steps	Rival steps	NCM score	Rival score	Speedup
News sentences	DUO	12	128	93.05	97.30	21x
Web text	DUO	32	1,024	83.12	77.69	10x
Summaries	AURORA-LM	8	128	28.0	26.6	31x
Translation	E2D2	11	64	22.6	24.8
Hard Sudoku	FMLM+	12	16	86.1%	81.3%

The news sentences come from LM1B, the web text from OpenWebText, the summaries from XSum, and the translation from WMT14 German to English. Speed is wall-clock time to write 1,024 sequences on one NVIDIA H100 GPU.

Three charts from the NCM paper: NCM reaching lower perplexity 21x faster than DUO on LM1B, higher ROUGE-L 31x faster on XSum, and the highest Sudoku accuracy from easy to hard — Quality against wall-clock time on LM1B and XSum, and Sudoku accuracy by difficulty

Image: Figure 1 of the NCM paper

The LM1B win also shows up under four other judge models. There, NCM averages 83.24 against DUO’s 92.52, even though NCM is the smaller model.

NCM is also cheaper to train. On LM1B it sees 4.9x fewer training tokens than the diffusion baselines: 13.36 billion against 66 billion. On web text, it uses about 127 billion against more than 500 billion.

The headline numbers skip two limits:

Web text is a near miss. DUO wins there, with 77.69 perplexity to NCM’s 83.12.
Translation is a loss. A plain word-by-word model of the same size scores 25.2 BLEU against NCM’s 22.6. NCM only ties it by writing 12 drafts and picking the best, which takes 132 steps to the plain model’s 64.

The saved time can buy quality. NCM can write many drafts and keep the one most like the rest. On XSum, 16 drafts reach 29.6 ROUGE-L, ahead of AURORA-LM’s best of 28.6 at fewer total steps.

Why the Sudoku result is the one to watch

The Sudoku test uses 9x9 boards with 40, 35 or 30 given digits for easy, medium and hard. A board counts only when every cell is right.

At about 25M parameters, NCM solves 99.5%, 96.5% and 86.1% of boards in 12 steps. A word-by-word model of the same size solves 14.6%, 5.1% and 1.0% in 128 steps. The best diffusion rival, FMLM+, lands at 97.7%, 92.8% and 81.3%.

A left-to-right model commits to each digit and can never take it back. NCM refines a picture of the whole board at every level, so later levels can fix earlier guesses.

NCM’s coarse-to-fine generation is well suited to tasks that require global planning

Hermann Kumbong et al. (NCM paper)

On one hard board, 16 of 81 cells are right after step 1. The count dips to 15 as the model reworks its guesses, then reaches all 81 at step 12. The tokenizer also rebuilds every Sudoku board without error. One wrong digit would ruin a solution.

Fifteen Sudoku grids from the NCM paper: the puzzle, its solution, and the board after each of 12 steps turning green as cells become correct, plus a line chart of correct cells rising to 81 — A hard board decoded after each NCM step, from 16 of 81 cells right to all 81

Image: Figure 20 of the NCM paper

Where NCM fits next to Coconut and hybrid reasoning agents

NCM arrived the same week as another paper about models that think without words.

Coconut, a Meta paper from December 2024, lets a model reason in “continuous thought.” It feeds its own inner state back in as the next input, so the reasoning never turns into words.

Hybrid Reasoning Agents, published on arXiv on 13 October 2026, switch between silent thinking and normal text whenever the model needs a tool. Across 3 model sizes and 11 tests, they score 10.4% better and use 35.5% to 54.7% fewer tokens. The code lives under Together AI’s GitHub, which links it to NCM through Arora.

Coconut and hybrid agents hide the thinking, while NCM produces ordinary words and changes only the order in which the answer is built.

Silent thinking is hard to check. Every NCM level can be turned back into text, so you can watch it work.

What NCM cannot do yet

The paper’s own limits section and error tables show NCM is early research:

Small models. The main models have about 100M parameters. Quality improved from 97M to 317M to 725M, but nobody has tried billions.
One model per task. Each of the six datasets gets its own tokenizer and generator. There is no general NCM and no demo to talk to.
Short outputs. Targets run from 48 to 1,024 tokens.
A fixed schedule. Even a short reply walks the full stack of levels.
Two training stages. The tokenizer and the generator train separately.
Glitches. Outputs show odd seams where rough levels hand off to fine ones.

NCM fixes both the hierarchy depth K and the per-level lengths ℓk, so even short continuations traverse the full hierarchy

Hermann Kumbong et al. (NCM paper, limitations)

The tokenizer’s slips are the most worrying for factual writing. On web text it rebuilds 99.99% of tokens exactly. But its rare errors fall on names and numbers: “Melbourne” comes back as “Brisbane” and “1957” as “1956.”

Trying the released NCM checkpoints

The trained models live in the hazyresearch/ncm collection on Hugging Face, released under the MIT licence. Test data is in ncm-datasets.

You need Python 3.10 or newer and PyTorch built for an NVIDIA GPU. Models this small fit on a home GPU.

git clone https://github.com/HazyResearch/ncm.git
cd ncm
pip install -r requirements.txt
hf download hazyresearch/ncm-checkpoints --local-dir artifacts
hf download hazyresearch/ncm-datasets --repo-type dataset --local-dir data
python evaluate.py --config config/eval/<task>.yaml --checkpoint-dir artifacts/<task> --data-dir data/<task>

Replace <task> with one of the checkpoint folders. XSum and translation also accept a --cfg setting that ties the output more tightly to the input. The paper’s best XSum value is 0.5.