Skip to main content
ES CAT EN IT FR DE 中文
Method

From 4,096 to 32,000 tokens: how to design the context window in an AI (LLM)

A model pretrained on 4,096 tokens can't read a much longer text well, say 32,000 tokens. There is a process of 'shrinking' the positions and then a short fine-tune of about 1,000 steps, and a needle test (hiding a fact far away inside the context) will tell us whether it can really take in the whole text.

What we found

  • With a 128-dimension head and base 10,000, 18 of RoPE's 64 rotation pairs don't complete a turn in 4,096 tokens. At 32,000, the slowest one reaches 212° when training only took it to 27° (498A's own calculation, October 2026).
  • Scaling the positions by multiplying each one by 0.128 (4,096/32,000) and fine-tuning for about 1,000 steps extends the window. On LLaMA 7B, direct fine-tuning to 8,192 tokens went from 2,048 to 2,560 effective after more than 10,000 steps; with interpolation it reached 8,192 in 200 (Meta, June 2023).
  • In an illustrative GEORadar example, a ten-interaction conversation takes about 5,400 tokens: it doesn't fit in 4,096 and it fits in 8,192. Five of them side by side, 27,000, fit in 32,000; seven, 37,800, don't.

In 2025, talking about factorization or model distillation sounded like computing archaeology. In 2026 they come up in our product meetings. The reason is practical: once tasks move to local models, somebody has to decide what the provider used to decide. The temperature. The embedding. The reranker. How much context window you need and how you get it.

To me that's good news. We're taking the reins of our tools back.

In GEORadar's analytics module we had to redefine the context window, and this article walks through the reasoning. Our thread is a classic classroom setup: a model pretrained on 4,096 tokens that has to read documents of 32,000. The GEORadar examples are illustrative. They use our scale figures and assumptions we state each time, and no production log sits behind them. When a figure is real and already published, we link it.

01The pieces that are ours again#

Training and fine-tuning a model matter, but they aren't the only levers. A local system is also tuned with pieces that never touch the weights, and with a cloud model those came decided at the factory.

The pieces of a local AI system and what each one decides. Illustrative examples, with no production inventory behind them
PieceWhat it decidesIllustrative example in a GEORadar analysis
TemperatureHow much the output varies between two runs0 when extracting brands and sentiment from an answer; higher only when suggesting alternative phrasings
Context windowHow much the model sees at onceOne answer, one conversation or a comparison across surfaces
EmbeddingWhat counts as similarSpotting near-duplicate prompts in a corpus of 30,000
RerankerWhat reaches the window and in what orderPicking the relevant interactions of a conversation before reading them
DistillationHow heavy the reading model isA small model that classifies at scale using the labels of a large one
Low-rank factorization (LoRA)How much it costs to adapt itAdapting a model to a vertical by training small matrices

Almost all of them have a parameter you set per task. The window is the one people take most for granted, because it's printed on the model card. It's also the one you can move the most, if you know how.

02What happens when a 4,096-token model meets a 32,000-token document#

Here's the setup. A model was pretrained on sequences of 4,096 tokens (a token is a fragment of a word) and uses rotary embeddings, RoPE. We hand it a 32,000-token document. Perplexity, which measures how surprised the model is by each next token, looks normal for the first 4,000 tokens and then shoots up. We have budget for about 1,000 fine-tuning steps on long documents.

What happens to a model when it runs into positions it has never seen? To find out, look at what RoPE does. To each pair of dimensions of the query and the key it applies a rotation whose angle is proportional to the token's position. Each pair turns at its own frequency. Some make full turns every few tokens, others barely move.

With an attention head of 128 dimensions (64 pairs) and base 10,000, typical values, for example, in the LLaMA family, the numbers come out like this:

  • The fastest pair makes one turn every 6.3 tokens.
  • The slowest, one every 54,410.
  • In 4,096 tokens, 46 pairs complete at least one turn. The other 18 don't.
  • At 32,000 tokens, 60 of them do.

The problem sits with the slow pairs. The fast ones repeat angles the model already knows, because they go around many times. The 18 slow ones don't: during training the slowest one only got to 27°. At 32,000 tokens it would reach 212°. The model has never seen those angles, and the attention scores built from them mean nothing to it.

Chart of the angle turned by RoPE's slowest pair according to token position. With the positions untouched, the line rises from 0° to 212° at token 32,000, and only the first 4,096 tokens, up to 27°, were seen in training. With the positions multiplied by 0.128, the line reaches 27° at token 32,000, inside the range already seen.
RoPE's slowest pair, with and without scaled positions · Source: 498A own calculation using the RoFormer definition, November 2026
Technical detail · how those numbers come out

The frequency of pair i is θᵢ = 10000^(-2i/d), with d = 128 and i from 0 to 63. The wavelength is 2π/θᵢ. For i = 0, θ is 1 radian per token and the wavelength is 6.3 tokens. For i = 63, θ is 0.000115 radians per token and the wavelength is 54,410 tokens. The angle that pair covers is 0.473 radians (27°) at 4,096 tokens and 3.695 radians (212°) at 32,000. These are our own calculations using the RoFormer definition (Su et al., 2021). They change if the model uses a different base or head dimension.

03Four options, and only one survives a 1,000-step budget#

With about 1,000 fine-tuning steps, the setup offers four paths.

The four options and what the evidence says about each
OptionWhat it doesVerdict with about 1,000 steps
A · Fine-tune on long documents without touching the positionsAsks the model to learn new anglesNot enough: on LLaMA 7B, more than 10,000 steps moved the effective window from 2,048 to 2,560
B · Scale the positionsShrinks the indices so 32,000 fits in the range already seenWorks: 200 steps were enough in the original paper to reach 32,768
C · Replace RoPE with ALiBi and fine-tuneSwaps the positional model for a linear bias by distanceThrows away what was learned about positions; a thousand steps won't restore it
D · Sliding window of 4,000Limits attention to the last 4,000 tokensCan't see what falls outside: it arrives chained through the layers, with no direct access

Option A. The natural temptation. In the Position Interpolation paper (Chen et al., Meta, June 2023) they tried exactly that with LLaMA 7B: fine-tune to reach 8,192 tokens without touching the positions. They measured the effective window with the passkey retrieval test, and it went from 2,048 to 2,560 after more than 10,000 steps. Without scaling, the model has to learn from scratch what the new angles mean, and a thousand steps don't get you there.

Option B. The one we pick, known as position interpolation, and one we can picture as 'shrinking the fabric'. Each position index m is multiplied by 4,096/32,000, which is 0.128. Token 32,000 is read as if it sat at position 4,096, and every angle falls inside the range the model knows. Chen et al. applied it to LLaMA models from 7B to 65B, with windows of up to 32,768 tokens and at most 1,000 fine-tuning steps. With the 7B model, 200 steps were enough for the effective window to reach 32,768. Their analysis adds a reason why: the upper bound for interpolation is at least about 600 times smaller than the one for extrapolation.

Two charts of a RoPE wave by token position. Top, unscaled: the wave runs past the pretrained range of 0 to 4,096 and the red dots land on positions never seen, up to 8,192. Bottom, with positions divided by 2: the same dots are squeezed and all land inside the seen range, on a wave the model already knows.
Extrapolating or interpolating positions · Source: adapted from Figure 1 of Chen et al. (Meta, 2023), CC BY 4.0, redrawn by 498A

Option C. ALiBi (Press et al., 2021) is a good method, but you choose it before training. In a model that already learned to read positions through rotations, removing RoPE throws that part of what it learned away. That's our reading, since none of the papers we cite measures this specific swap. The asymmetry shows anyway: option B changes a scale, option C changes the mechanism.

Option D. Keeping the window at 4,000 tokens leaves the model on familiar ground, but the question was how to read all 32,000. Mistral 7B uses a sliding window of 4,096 with 32 layers, and its theoretical reach is about 131,000 tokens because information passes from layer to layer. It's a legitimate way to save compute. A fact from the beginning gets to the end by going through layers, with no head ever looking at it head-on, and that's a poor basis for retrieving an exact fact. Again, that's our reading.

Three sketches. Left, a 5 by 5 full-attention mask: every token attends to all earlier ones. Middle, a sliding-window mask with W equal to 3: every token attends only to the last 3. Right, a stack of layers over 10 tokens: the effective reach of the last token grows by W tokens per layer, so after 3 layers it reaches 9 tokens back. With W of 4,096 and 32 layers in Mistral 7B, the theoretical reach is about 131,000 tokens.
Full attention and sliding window · Source: adapted from Figure 1 of Jiang et al. (Mistral 7B, 2023), CC BY 4.0, redrawn by 498A

04What scaling positions costs#

Scaling isn't free. Multiply by 0.128 and two neighboring tokens go from one unit of position apart to 0.128 apart. The slow pairs couldn't care less. For the fast ones, the pairs that tell a token from its neighbor, the world changes: what was a one-radian turn between adjacent words becomes 0.128. The model has to relearn to tell them apart, and that short fine-tune of 1,000 steps is what it takes. A 'fine-tuning step' is a small correction of the model's internal values, computed from a batch of examples: each step nudges all of its values a little. A thousand steps is little next to training a model from scratch (Meta's paper calls that cost negligible, and so do we, having weighed the alternatives).

In the original paper, the loss inside the original window was between 0.01 and 0.05 of perplexity on Proof-pile. Small, and expected. There are variants that interpolate the high frequencies less: YaRN (Peng et al., 2023) needs 10 times fewer tokens and 2.5 times fewer steps than earlier methods.

Chart of the wavelength of each RoPE dimension pair, on a log scale, from the fastest pair (1) to the slowest (64). Original RoPE runs from about 6 tokens to about 54,000. Position Interpolation shifts every curve up by the same factor, 7.8. NTK-aware shifts them progressively. NTK-by-parts, the YaRN method, leaves the fast pairs untouched, those whose wavelength is shorter than the 4,096-token training length, and interpolates only the slow ones.
Wavelength of each RoPE pair under each method · Source: 498A's own calculation using the definitions of Peng et al. (YaRN, 2023), CC BY 4.0, based on the EleutherAI blog chart

That said, memory is what does get more expensive. From 4,096 to 32,000 tokens, full attention costs 61 times more per layer, because it grows with the square of the length. And the key-value cache, which grows in a straight line, goes from 2 GiB to 15.6 GiB on a typical 7B model. The Position Interpolation authors used 128 A100 GPUs for the longest cases, and they explain that the GPU requirement comes from memory while the steps stay few. On a local machine, memory runs out before patience does. To fine-tune with fewer resources there's LongLoRA (Chen et al., 2023), which combines the low-rank factorization of LoRA with sparse attention during fine-tuning.

Technical detail · where the key-value cache comes from

Each token stores, per layer, one key and one value for every KV head. With 32 layers, 32 heads of 128 dimensions and 2 bytes per value (fp16), that's 2 × 32 × 32 × 128 × 2 = 524,288 bytes, or 0.5 MB per token. For 4,096 tokens that comes to 2 GiB; for 32,000, 15.6 GiB. With grouped-query attention of 8 KV heads (GQA), it's 128 KiB per token and 3.9 GiB at 32,000. This is our own calculation for a typical 7B architecture without GQA. Real models vary.

05How to check that it uses the whole text#

Changing 4,096 to 32,000 in the config proves nothing. You have to show that the model uses what sits far from the question, beyond 4,096 tokens. Five tests, from least to most demanding:

  1. Perplexity by segment. The symptom in the setup is a jump past 4,000 tokens, and a global average hides it. Measure perplexity by position segment (0 to 4,000, 4,000 to 8,000, and so on up to 32,000) on documents longer than the window. It has to come out flat. A step means the extension doesn't work.
  2. Needle in a haystack. Hide a fact in a long document and ask about it, varying length and depth. Chen et al. tested 32 positions spread uniformly, 10 trials at each one, with a random 5-digit key. The effective window is the longest distance at which the model keeps getting it right.
  3. Counterfactual test. The same question over the full document and over the document without the stretch that holds the answer. If the correct answer disappears when you remove it, the model was using it. If it keeps answering the same thing, it was guessing.
  4. Tasks harder than a needle. RULER (Hsieh et al., 2024) evaluated 17 models that advertised 32,000 tokens or more. Almost all of them scored near perfect on the simple needle, and only 10 held up satisfactorily at 32,000 with multiple needles and chained questions. And Lost in the Middle (Liu et al., 2023) showed that performance peaks at the beginning and the end of the context and drops in the middle, even in long-context models. Hide facts in the middle too.
  5. Regression in the short range. The same battery inside the 4,096 tokens, before and after. An extension that improves the long range and breaks the short one is useless.
Line chart of the accuracy of three models on a question with 20 documents, by position of the document that holds the answer (1, 5, 10, 15 and 20). GPT-3.5-Turbo goes from 75.8% at position 1 to 57.2%, 53.8% and 55.4% in the middle and rises to 63.2% at 20, a U shape. LongChat-13B also dips. Claude-1.3 stays between 55.9% and 60.1%. A dashed line marks GPT-3.5-Turbo's 56.1% with no documents.
Accuracy by position of the document holding the answer · Source: adapted from Figure 1 of Liu et al. (TACL, 2024), CC BY 4.0, with the data from its 20-document table

We already run a small version of this. In the local models benchmark we published on October 19, 2026, one of the 17 tests hides a fact among more than 20,000 tokens of filler, with 32,768 tokens of context, temperature 0.2 and seed 42 on a Mac with 48 GB. It's one needle. Just one. An analytics module needs the whole grid.

06GEORadar examples: the unit of analysis sets the window#

The numbers in this section are illustrative. We start from GEORadar's scale figures (studies of 6,000 to 30,000 prompts, seven AI surfaces, conversations of up to ten interactions) and from two assumptions: 40 tokens per question and 500 per answer, so 540 per interaction. No client appears. We talk about verticals.

Size of each unit of analysis under the assumptions of this article (540 tokens per interaction, no instructions)
Unit of analysisCalculationTokensFits in 4,096?Fits in 32,000?
One answer from one surface40 + 500540YesYes
A ten-interaction conversation10 × 5405,400NoYes
Five conversations side by side5 × 5,40027,000NoYes
The same conversation across the seven surfaces7 × 5,40037,800NoNo
A 6,000-prompt study, individual answers42,000 × 54022,680,000NoNo

Add the instructions and the output schema to every row. In a structured analysis we assume about 1,000 tokens more.

A whole conversation fits in 8,192

Among other things, GEORadar measures whether an AI accepts a negative premise in the prompt or corrects it (we call it Premise Validation). Picture a conversation in the Travel vertical with ten interactions, where the second question takes something false about a destination for granted. To judge whether the assistant ended up correcting the premise or dragging it along to the ninth answer, the reading model has to see both at once.

With 4,096 tokens you have to trim, and you lose the first three interactions, right where the premise lives. With instructions and schema, the conversation needs about 6,400 tokens. It fits in 8,192. Going up to 32,000 would mean paying about 15 times more attention per layer to use almost none of it.

Comparing surfaces is what justifies going further

To see whether the seven surfaces drag along the same premise, the model needs the answers side by side. Five complete conversations add up to 27,000 tokens and fit in 32,000 with 5,000 to spare. Seven add up to 37,800 and don't fit.

There are two ways out. Extend the window to 40,000 and absorb the cost. Or ask what's really needed. The interaction where the premise appears and the answer that corrects it or drags it along make two interactions per surface. Seven surfaces, 14 interactions, 7,560 tokens. It fits in 8,192. The work is done by whoever picks those interactions: an embedding to locate the premise and a reranker to order the candidates.

A whole study fits in no window

A study of 6,000 prompts against seven surfaces is 42,000 answers and about 22.7 million tokens under our assumptions. With 30,000 prompts, about 113 million. No window reaches that, and none needs to. The aggregate figures come from SQL queries on entity_metrics, and the BIS (Brand Impact Score) is read from there and never recalculated by hand. The model reads samples and classifies them, and the database does the summing. That division of labor is what lets the window stay small.

The needle test, with GEORadar data

If the reading model were one extended from 4,096 to 32,000, here's how we'd set up the test. A synthetic dossier of 32,000 tokens made of AI answers about a fictional Travel brand. One planted sentence with a specific negative assessment, placed at five depths (5, 25, 50, 75 and 95%) and four lengths (4,000, 8,000, 16,000 and 32,000 tokens). The question: which surface does it appear on. We know the answer because we planted it, and we add distractors, similar sentences about competing brands.

With ten repetitions per cell that makes 200 runs per configuration, and the result is a heat map. If the model works up to 4,000 and switches off beyond that, the extension hasn't worked. If it fails at 50%, that's the Lost in the Middle effect.

The same rule for the model that reads

GEORadar measures with the same frozen set of prompts, against the same engine and the same model version. That lets us attribute a change to the action plan. The same rule holds for the model that reads the answers: if it changes between two measurements, you can't tell whether the brand moved or the rule did.

With a local model you can fix everything: weights, window, instruction template and temperature. At 0, variability drops a lot. It doesn't vanish entirely, because depending on the hardware and the batching, two runs can differ on a token. That's why in our October benchmark we also fixed a seed, 42, and a temperature of 0.2 to compare two models. For a repeatable analysis we'd go down to 0.

07When a longer window helps and when it hurts#

Models advertise ever longer windows, and their useful accuracy at those lengths doesn't grow at the same pace. In RULER, the 17 models evaluated advertised 32,000 tokens or more, and only 10 performed satisfactorily at 32,000. So the advertised window and the effective window are two different numbers, and the second one is the one that counts.

Bar chart of 17 models with their claimed and effective context length according to RULER. Gemini-1.5-Pro claims 1M and exceeds 128K effective; GPT-4 claims 128K and performs up to 64K; Llama 3.1 70B, 128K and 64K; GLM4, 1M and 64K; Qwen2, Command-R-plus, Llama 3.1 8B, Yi, Phi3 and Mixtral perform up to 32K; GradientAI Llama 3 and Mistral v0.2, up to 16K; DBRX, 8K; Together, 4K; LWM, LongChat and LongAlpaca, below 4K. Ten of the 17 hold up satisfactorily at 32K.
Claimed and effective length of 17 models · Source: adapted from Table 3 of Hsieh et al. (RULER, 2024), CC BY 4.0, redrawn by 498A

I'm one of those who defend long windows when the task calls for them. Extending helps when the task needs to relate pieces that sit far apart and chopping them up would break the relation. The premise in interaction two against the answer in interaction nine. Five conversations side by side. A study report you have to read in one go to keep the thread.

Extending hurts when it's a way of skipping the retrieval design: you stuff everything in because it fits. You pay 61 times more attention per layer, you add memory, and the model reads worse whatever lands in the middle. It also hurts when the window gets confused with a memory. The window stores nothing between runs.

08How we approach it in the analytics module#

A stumble of our own brought it home. In the October benchmark, Qwen spent its whole 4,096-token generation limit thinking and never got to answer in three tests. That limit was for generation; the context window is a separate thing. The lesson holds anyway: a token budget is a decision, and the default is almost never the right one.

This is the order in which we make the call, and it works for any reading task with a local model:

  1. Define the unit of analysis. One answer, one conversation, one comparison. The window follows from that, whatever the spec sheet says.
  2. Calculate the tokens of the unit plus the instructions, the output schema and a margin. If it fits in 8,192, don't ask for more.
  3. If it doesn't fit, ask what you can drop before extending. Embedding to locate, reranker to order, and only then the window.
  4. If you have to extend, scale the positions and fine-tune for about 1,000 steps on long documents from the task itself rather than generic ones, with LoRA if memory is tight. That last part is our own recommendation.
  5. Test with the needle grid (length by depth), with distractors, and rerun the short battery to catch regressions.
  6. Fix the rest of the instrument: weights, template, temperature and seed. And don't change them between two measurements.

This fits how we work on local models and data sovereignty, and what GEORadar measures: what seven AI surfaces say about a brand, with studies of 6,000 to 30,000 prompts. A reliable reading of those answers is part of the instrument. For the other side of the same coin, retrieval, there's our piece on The most expensive mistake in a vector rollout.

For us, the context window has stopped being a number on the model card and become a decision that can be justified with a simple test. If you're fine-tuning local models for specific tasks, I'd like to hear which window you ended up choosing, and why.

Sources
  1. RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al.). arXiv 2104.09864, April 2021. https://arxiv.org/abs/2104.09864
  2. Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., Meta). arXiv 2306.15595, June 2023. https://arxiv.org/abs/2306.15595
  3. YaRN: Efficient Context Window Extension of Large Language Models (Peng et al.). arXiv 2309.00071, August 2023. https://arxiv.org/abs/2309.00071
  4. Extending the RoPE (EleutherAI blog on YaRN). EleutherAI, 2023. https://blog.eleuther.ai/yarn/
  5. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Press et al.). arXiv 2108.12409, August 2021. https://arxiv.org/abs/2108.12409
  6. Mistral 7B (Jiang et al.). arXiv 2310.06825, October 2023. https://arxiv.org/abs/2310.06825
  7. Lost in the Middle: How Language Models Use Long Contexts (Liu et al.). arXiv 2307.03172, July 2023. https://arxiv.org/abs/2307.03172
  8. RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al.). arXiv 2404.06654, April 2024. https://arxiv.org/abs/2404.06654
  9. Landmark Attention: Random-Access Infinite Context Length for Transformers (Mohtashami and Jaggi). arXiv 2305.16300, May 2023. https://arxiv.org/abs/2305.16300
  10. LoRA: Low-Rank Adaptation of Large Language Models (Hu et al.). arXiv 2106.09685, June 2021. https://arxiv.org/abs/2106.09685
  11. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (Chen et al.). arXiv 2309.12307, September 2023. https://arxiv.org/abs/2309.12307
  12. Distilling the Knowledge in a Neural Network (Hinton, Vinyals, and Dean). arXiv 1503.02531, March 2015. https://arxiv.org/abs/1503.02531
  13. 498A. The best local AI for marketing and communication: Glimmer against Qwen. 498advance.com, October 19, 2026. https://498advance.com/en/blog/mejor-ia-local-marketing-comunicacion.html
Carlos Ortet
Carlos Ortet
Co-founder and CEO of 498 Advance

Runs 498A, the applied-AI lab of the Zoopa group, and GEORadar. Guest lecturer at UAB, where he supervises master's theses in AI with IIIA-CSIC.