Next-Concept Modeling Explained

Current text generation methods

Most language models use next-token prediction. The model writes one token, reads it, and writes the next one. A 1,000-token answer needs 1,000 steps. A cache makes each step cheap, but the steps must happen one after the other.

Diffusion language models work differently. They start with masked or noisy tokens and repair all positions together. This work is parallel, so the number of steps can be much smaller than the number of tokens. The paper names three problems with this approach:

• Many steps. Quality falls quickly when you remove steps.
• Expensive steps. Each step reads the full sequence. The usual next-token cache does not apply directly, because attention goes in both directions.
• Large training budgets. Strong baselines read over 500 billion tokens on OpenWebText.

Visual Overview of NCM

Think of a large image on a slow connection. First you see a blurry version of the full image. Then the detail comes in. NCM does the same for text. The first step gives a very short description of the full text. Each next step is longer and adds detail. Only the last step has one code for each token.

The word concept in the name means the information in these first, short levels. A concept here is a learned code. It is not an idea that a person can read.

The method comes from image generation, where it is called next-scale prediction. Images make this easy, because you can shrink an image if you average its pixels.

So NCM must learn its own low-resolution version of text. This gives the method two stages. Stage 1 learns how to store text in layers. Stage 2 learns how to generate those layers.

Step 1: Store Text in Layers

The first stage trains a multiscale residual VQ-VAE. This name is long, but each part of it is simple.

Vector quantization

An encoder changes each token into a vector of numbers. Vector quantization then replaces this vector with the nearest entry in a fixed list, the codebook. You store only the position of that entry in the list. So a long vector becomes one integer. NCM uses 4,096 or 8,192 entries per level for text.

Residual

Rounding loses information. Residual quantization stores this loss in a second level, and the loss of the second level in a third. A small example with one number shows the idea:

Multiscale

The levels have different lengths. For LM1B, the released config uses 12 levels with these lengths:

1 8 16 24 32 40 52 64 76 92 108 128

So the first level is one code for 128 tokens. To use a short level, the tokenizer shrinks what is still missing to that length. It then snaps each vector to the codebook and stretches the result back to full length.

Nested dropout

One problem remains. The tokenizer can put almost all information in the last levels, and then the early levels are useless as a sketch. Nested dropout prevents this. During training, it keeps only the first k levels and tells the decoder to rebuild the text from them. The value of k changes at random, so each early level must carry useful information.

After training, the tokenizer is frozen. With all levels, it recovers 98.79% to 99.99% of tokens on the text tasks and 100% on Sudoku. This matters, because one wrong digit breaks a Sudoku solution.

Step 2: Predict the Next Layer

Each text is now a stack of integers. The generator is a transformer that predicts these integers, one level at a time. At each level, its input is the running total of the earlier levels, resized to the length of the current level.

The main assumption is simple. If you know the sketch so far, the codes inside one level are independent. So the model can sample all of them in one forward pass:

p(all codes) = p(level 1)
             × p(level 2 | sketch 1)
             × ...
             × p(level K | sketch K-1)

p(level k | sketch) = p(code 1 | sketch)
                    × p(code 2 | sketch)
                    × ...

Generation is a short loop:

# start with nothing
sketch = zeros(L, d)

# K = 12 for LM1B
for k in range(K):
    # fit this level
    inputs = resize(sketch, length[k])
    # one forward pass, all codes at once
    codes = sample(transformer(inputs))
    # add the correction
    sketch += stretch(codebook[k][codes])

# sketch to text
tokens = decoder(sketch).argmax(-1)

Forward pass during training

During training, the frozen tokenizer gives the correct codes for each text. So the inputs for all levels are known before the model starts. The transformer predicts all K levels in one forward pass, with a cross-entropy loss.

Self reading

Without a prompt, attention stays inside the current level. This looks like a strong limit, but the input of each level already contains the earlier levels. In the paper’s test on TinyStories, this pattern gave the best mix of quality, sample diversity and speed. With a prompt, each level can also read the prompt.

How a bad code in training helps

Training gives the model perfect codes, but generation gives it its own codes, which contain errors. All codes in a level arrive together, so several errors can enter the next input at the same time. To prepare for this, NCM replaces some input codes with random entries during training. The targets stay correct. A moderate rate improves quality by a large amount, and too high a rate makes it worse.

Two (very) important parts

• Number of levels. More levels give better text, but each increase helps less. On TinyStories, the Gen. PPL score (defined under Results, lower is better) falls by 52% from 4 to 8 levels, then by 32%, then by 20% at 32 levels.
• Growth of level lengths. Slow growth at the start works best in this test. After 10k training steps, exponential and quadratic growth reach a Gen. PPL of about 58 and 60. Linear growth reaches 88, and square-root growth reaches 197.

Reason behind the speed

NCM saves time in two ways: it uses fewer steps, and its early steps are short. The picture below shows both at the same scale.

This count is mine, and it is not a benchmark. Fewer steps give 128 / 12 = 10.7×. Shorter rows give 1,536 / 641 = 2.4×. Together they give about 26×. The paper measures 21× against DUO in wall-clock time, on one H100 GPU. The two numbers are close, but the count ignores model size, attention cost and the final decoder pass.

Training also costs less. On LM1B, NCM reads 13.36 billion token positions, and this includes the tokenizer. The baselines read 66 billion. On OpenWebText, NCM reads 127 billion, against over 500 billion for MDLM and DUO.

Level by level generation

The paper follows one German-to-English translation through all 11 levels. It decodes the sketch after each level and scores it against a human translation.

After level 1, only the opening word is correct, and the rest is mostly repeats of “the”. Around level 6, the main topic words are present, but they repeat. Level 11 gives a nearly exact translation. The score also falls at level 9, so progress is not smooth. Later levels can repair the errors of earlier levels.

Results

The results use four measures:

• Steps (NFE in the paper): the number of times that the model runs for one output.
• Gen. PPL: how surprising the text is to a reference model, GPT-2-large. Lower is better. Repetitive text can also score low, so the paper reports text diversity with it.
• BLEU and ROUGE-L: overlap with a human translation or summary. Higher is better.
• Solved boards: the percentage of Sudoku boards that are fully correct.

The baseline is the one that the paper compares against for each task. The better score is in bold. Speed is the paper’s wall-clock measurement.

NCM does not win every row. It loses to E2D2 on translation. DUO scores better on OpenWebText, but it uses 32× more steps. Next-token models also stay ahead on raw quality: a next-token baseline reaches 35.45 on OpenWebText, against 83.12 for NCM. That baseline needs 1,024 steps.

Sudoku. A next-token baseline solves about 1% of hard boards, because it must commit to each digit from left to right. NCM refines the full board at each level, so later levels can correct earlier choices.

Saved time can give quality. NCM can generate many candidates and select the one that is most like the others. This is MBR decoding. On XSum, it reaches ROUGE-L 29.6 with 128 model runs. The best guided baseline reaches 28.6 with 184.

Effect of coarse levels on the high level content

Do the coarse levels really hold the high-level content? The authors test this on short children’s stories, in three ways:

• Keep the first levels, resample the rest. The broad theme of the story stays. Names, objects and wording change.
• Replace two early codes. Characters, objects and events can change, and parts of the plot stay.
• Mix the early codes of two stories. The output contains details from both.

The paper calls these probes qualitative. They suggest that coarse levels capture global structure, but they do not prove it.

Limits

• Fixed depth. Short text must also go through all levels, because the schedule does not change with the input.
• A separate tokenizer. It trains before the generator, so it does not learn from the generator’s goal. The size comparisons also count only the generator. On LM1B, the 97M generator has a 111M tokenizer with it.
• Small models. Most tests use about 100M parameters. Only LM1B goes further, to 725M, where quality improves. The paper does not test the sizes that production models use.
• Artifacts. Text is generally fluent, but errors sometimes appear where coarse levels meet.

Final Thoughts

Earlier work on fast diffusion keeps the full-length representation and reduces the steps. NCM changes the representation, and this reduces both the number of steps and the cost of the early steps. For me, this is the main idea of the paper. The next test is whether the method keeps its advantage at much larger model sizes.

Credits

The method and results are the work of Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini and Christopher Ré.

• Paper: arxiv.org/abs/2610.14162
• Code: github.com/HazyResearch/ncm

I drew the figures for this post. The numbers come from the paper, with two exceptions. The LM1B level lengths come from the released config. I calculated the position-pass count from those lengths.