In 2025, talking about factorization or model distillation sounded like computing archaeology. In 2026 they come up in our product meetings. The reason is practical: once tasks move to local models, somebody has to decide what the provider used to decide. The temperature. The embedding. The reranker. How much context window you need and how you get it.
To me that's good news. We're taking the reins of our tools back.
In GEORadar's analytics module we had to redefine the context window, and this article walks through the reasoning. Our thread is a classic classroom setup: a model pretrained on 4,096 tokens that has to read documents of 32,000. The GEORadar examples are illustrative. They use our scale figures and assumptions we state each time, and no production log sits behind them. When a figure is real and already published, we link it.
01The pieces that are ours again#
Training and fine-tuning a model matter, but they aren't the only levers. A local system is also tuned with pieces that never touch the weights, and with a cloud model those came decided at the factory.
| Piece | What it decides | Illustrative example in a GEORadar analysis |
|---|---|---|
| Temperature | How much the output varies between two runs | 0 when extracting brands and sentiment from an answer; higher only when suggesting alternative phrasings |
| Context window | How much the model sees at once | One answer, one conversation or a comparison across surfaces |
| Embedding | What counts as similar | Spotting near-duplicate prompts in a corpus of 30,000 |
| Reranker | What reaches the window and in what order | Picking the relevant interactions of a conversation before reading them |
| Distillation | How heavy the reading model is | A small model that classifies at scale using the labels of a large one |
| Low-rank factorization (LoRA) | How much it costs to adapt it | Adapting a model to a vertical by training small matrices |
Almost all of them have a parameter you set per task. The window is the one people take most for granted, because it's printed on the model card. It's also the one you can move the most, if you know how.
02What happens when a 4,096-token model meets a 32,000-token document#
Here's the setup. A model was pretrained on sequences of 4,096 tokens (a token is a fragment of a word) and uses rotary embeddings, RoPE. We hand it a 32,000-token document. Perplexity, which measures how surprised the model is by each next token, looks normal for the first 4,000 tokens and then shoots up. We have budget for about 1,000 fine-tuning steps on long documents.
What happens to a model when it runs into positions it has never seen? To find out, look at what RoPE does. To each pair of dimensions of the query and the key it applies a rotation whose angle is proportional to the token's position. Each pair turns at its own frequency. Some make full turns every few tokens, others barely move.
With an attention head of 128 dimensions (64 pairs) and base 10,000, typical values, for example, in the LLaMA family, the numbers come out like this:
- The fastest pair makes one turn every 6.3 tokens.
- The slowest, one every 54,410.
- In 4,096 tokens, 46 pairs complete at least one turn. The other 18 don't.
- At 32,000 tokens, 60 of them do.
The problem sits with the slow pairs. The fast ones repeat angles the model already knows, because they go around many times. The 18 slow ones don't: during training the slowest one only got to 27°. At 32,000 tokens it would reach 212°. The model has never seen those angles, and the attention scores built from them mean nothing to it.
Technical detail · how those numbers come out
The frequency of pair i is θᵢ = 10000^(-2i/d), with d = 128 and i from 0 to 63. The wavelength is 2π/θᵢ. For i = 0, θ is 1 radian per token and the wavelength is 6.3 tokens. For i = 63, θ is 0.000115 radians per token and the wavelength is 54,410 tokens. The angle that pair covers is 0.473 radians (27°) at 4,096 tokens and 3.695 radians (212°) at 32,000. These are our own calculations using the RoFormer definition (Su et al., 2021). They change if the model uses a different base or head dimension.
03Four options, and only one survives a 1,000-step budget#
With about 1,000 fine-tuning steps, the setup offers four paths.
| Option | What it does | Verdict with about 1,000 steps |
|---|---|---|
| A · Fine-tune on long documents without touching the positions | Asks the model to learn new angles | Not enough: on LLaMA 7B, more than 10,000 steps moved the effective window from 2,048 to 2,560 |
| B · Scale the positions | Shrinks the indices so 32,000 fits in the range already seen | Works: 200 steps were enough in the original paper to reach 32,768 |
| C · Replace RoPE with ALiBi and fine-tune | Swaps the positional model for a linear bias by distance | Throws away what was learned about positions; a thousand steps won't restore it |
| D · Sliding window of 4,000 | Limits attention to the last 4,000 tokens | Can't see what falls outside: it arrives chained through the layers, with no direct access |
Option A. The natural temptation. In the Position Interpolation paper (Chen et al., Meta, June 2023) they tried exactly that with LLaMA 7B: fine-tune to reach 8,192 tokens without touching the positions. They measured the effective window with the passkey retrieval test, and it went from 2,048 to 2,560 after more than 10,000 steps. Without scaling, the model has to learn from scratch what the new angles mean, and a thousand steps don't get you there.
Option B. The one we pick, known as position interpolation, and one we can picture as 'shrinking the fabric'. Each position index m is multiplied by 4,096/32,000, which is 0.128. Token 32,000 is read as if it sat at position 4,096, and every angle falls inside the range the model knows. Chen et al. applied it to LLaMA models from 7B to 65B, with windows of up to 32,768 tokens and at most 1,000 fine-tuning steps. With the 7B model, 200 steps were enough for the effective window to reach 32,768. Their analysis adds a reason why: the upper bound for interpolation is at least about 600 times smaller than the one for extrapolation.
Option C. ALiBi (Press et al., 2021) is a good method, but you choose it before training. In a model that already learned to read positions through rotations, removing RoPE throws that part of what it learned away. That's our reading, since none of the papers we cite measures this specific swap. The asymmetry shows anyway: option B changes a scale, option C changes the mechanism.
Option D. Keeping the window at 4,000 tokens leaves the model on familiar ground, but the question was how to read all 32,000. Mistral 7B uses a sliding window of 4,096 with 32 layers, and its theoretical reach is about 131,000 tokens because information passes from layer to layer. It's a legitimate way to save compute. A fact from the beginning gets to the end by going through layers, with no head ever looking at it head-on, and that's a poor basis for retrieving an exact fact. Again, that's our reading.
04What scaling positions costs#
Scaling isn't free. Multiply by 0.128 and two neighboring tokens go from one unit of position apart to 0.128 apart. The slow pairs couldn't care less. For the fast ones, the pairs that tell a token from its neighbor, the world changes: what was a one-radian turn between adjacent words becomes 0.128. The model has to relearn to tell them apart, and that short fine-tune of 1,000 steps is what it takes. A 'fine-tuning step' is a small correction of the model's internal values, computed from a batch of examples: each step nudges all of its values a little. A thousand steps is little next to training a model from scratch (Meta's paper calls that cost negligible, and so do we, having weighed the alternatives).
In the original paper, the loss inside the original window was between 0.01 and 0.05 of perplexity on Proof-pile. Small, and expected. There are variants that interpolate the high frequencies less: YaRN (Peng et al., 2023) needs 10 times fewer tokens and 2.5 times fewer steps than earlier methods.
That said, memory is what does get more expensive. From 4,096 to 32,000 tokens, full attention costs 61 times more per layer, because it grows with the square of the length. And the key-value cache, which grows in a straight line, goes from 2 GiB to 15.6 GiB on a typical 7B model. The Position Interpolation authors used 128 A100 GPUs for the longest cases, and they explain that the GPU requirement comes from memory while the steps stay few. On a local machine, memory runs out before patience does. To fine-tune with fewer resources there's LongLoRA (Chen et al., 2023), which combines the low-rank factorization of LoRA with sparse attention during fine-tuning.
Technical detail · where the key-value cache comes from
Each token stores, per layer, one key and one value for every KV head. With 32 layers, 32 heads of 128 dimensions and 2 bytes per value (fp16), that's 2 × 32 × 32 × 128 × 2 = 524,288 bytes, or 0.5 MB per token. For 4,096 tokens that comes to 2 GiB; for 32,000, 15.6 GiB. With grouped-query attention of 8 KV heads (GQA), it's 128 KiB per token and 3.9 GiB at 32,000. This is our own calculation for a typical 7B architecture without GQA. Real models vary.
05How to check that it uses the whole text#
Changing 4,096 to 32,000 in the config proves nothing. You have to show that the model uses what sits far from the question, beyond 4,096 tokens. Five tests, from least to most demanding:
- Perplexity by segment. The symptom in the setup is a jump past 4,000 tokens, and a global average hides it. Measure perplexity by position segment (0 to 4,000, 4,000 to 8,000, and so on up to 32,000) on documents longer than the window. It has to come out flat. A step means the extension doesn't work.
- Needle in a haystack. Hide a fact in a long document and ask about it, varying length and depth. Chen et al. tested 32 positions spread uniformly, 10 trials at each one, with a random 5-digit key. The effective window is the longest distance at which the model keeps getting it right.
- Counterfactual test. The same question over the full document and over the document without the stretch that holds the answer. If the correct answer disappears when you remove it, the model was using it. If it keeps answering the same thing, it was guessing.
- Tasks harder than a needle. RULER (Hsieh et al., 2024) evaluated 17 models that advertised 32,000 tokens or more. Almost all of them scored near perfect on the simple needle, and only 10 held up satisfactorily at 32,000 with multiple needles and chained questions. And Lost in the Middle (Liu et al., 2023) showed that performance peaks at the beginning and the end of the context and drops in the middle, even in long-context models. Hide facts in the middle too.
- Regression in the short range. The same battery inside the 4,096 tokens, before and after. An extension that improves the long range and breaks the short one is useless.
We already run a small version of this. In the local models benchmark we published on October 19, 2026, one of the 17 tests hides a fact among more than 20,000 tokens of filler, with 32,768 tokens of context, temperature 0.2 and seed 42 on a Mac with 48 GB. It's one needle. Just one. An analytics module needs the whole grid.
06GEORadar examples: the unit of analysis sets the window#
The numbers in this section are illustrative. We start from GEORadar's scale figures (studies of 6,000 to 30,000 prompts, seven AI surfaces, conversations of up to ten interactions) and from two assumptions: 40 tokens per question and 500 per answer, so 540 per interaction. No client appears. We talk about verticals.
| Unit of analysis | Calculation | Tokens | Fits in 4,096? | Fits in 32,000? |
|---|---|---|---|---|
| One answer from one surface | 40 + 500 | 540 | Yes | Yes |
| A ten-interaction conversation | 10 × 540 | 5,400 | No | Yes |
| Five conversations side by side | 5 × 5,400 | 27,000 | No | Yes |
| The same conversation across the seven surfaces | 7 × 5,400 | 37,800 | No | No |
| A 6,000-prompt study, individual answers | 42,000 × 540 | 22,680,000 | No | No |
Add the instructions and the output schema to every row. In a structured analysis we assume about 1,000 tokens more.
A whole conversation fits in 8,192
Among other things, GEORadar measures whether an AI accepts a negative premise in the prompt or corrects it (we call it Premise Validation). Picture a conversation in the Travel vertical with ten interactions, where the second question takes something false about a destination for granted. To judge whether the assistant ended up correcting the premise or dragging it along to the ninth answer, the reading model has to see both at once.
With 4,096 tokens you have to trim, and you lose the first three interactions, right where the premise lives. With instructions and schema, the conversation needs about 6,400 tokens. It fits in 8,192. Going up to 32,000 would mean paying about 15 times more attention per layer to use almost none of it.
Comparing surfaces is what justifies going further
To see whether the seven surfaces drag along the same premise, the model needs the answers side by side. Five complete conversations add up to 27,000 tokens and fit in 32,000 with 5,000 to spare. Seven add up to 37,800 and don't fit.
There are two ways out. Extend the window to 40,000 and absorb the cost. Or ask what's really needed. The interaction where the premise appears and the answer that corrects it or drags it along make two interactions per surface. Seven surfaces, 14 interactions, 7,560 tokens. It fits in 8,192. The work is done by whoever picks those interactions: an embedding to locate the premise and a reranker to order the candidates.
A whole study fits in no window
A study of 6,000 prompts against seven surfaces is 42,000 answers and about 22.7 million tokens under our assumptions. With 30,000 prompts, about 113 million. No window reaches that, and none needs to. The aggregate figures come from SQL queries on entity_metrics, and the BIS (Brand Impact Score) is read from there and never recalculated by hand. The model reads samples and classifies them, and the database does the summing. That division of labor is what lets the window stay small.
The needle test, with GEORadar data
If the reading model were one extended from 4,096 to 32,000, here's how we'd set up the test. A synthetic dossier of 32,000 tokens made of AI answers about a fictional Travel brand. One planted sentence with a specific negative assessment, placed at five depths (5, 25, 50, 75 and 95%) and four lengths (4,000, 8,000, 16,000 and 32,000 tokens). The question: which surface does it appear on. We know the answer because we planted it, and we add distractors, similar sentences about competing brands.
With ten repetitions per cell that makes 200 runs per configuration, and the result is a heat map. If the model works up to 4,000 and switches off beyond that, the extension hasn't worked. If it fails at 50%, that's the Lost in the Middle effect.
The same rule for the model that reads
GEORadar measures with the same frozen set of prompts, against the same engine and the same model version. That lets us attribute a change to the action plan. The same rule holds for the model that reads the answers: if it changes between two measurements, you can't tell whether the brand moved or the rule did.
With a local model you can fix everything: weights, window, instruction template and temperature. At 0, variability drops a lot. It doesn't vanish entirely, because depending on the hardware and the batching, two runs can differ on a token. That's why in our October benchmark we also fixed a seed, 42, and a temperature of 0.2 to compare two models. For a repeatable analysis we'd go down to 0.
07When a longer window helps and when it hurts#
Models advertise ever longer windows, and their useful accuracy at those lengths doesn't grow at the same pace. In RULER, the 17 models evaluated advertised 32,000 tokens or more, and only 10 performed satisfactorily at 32,000. So the advertised window and the effective window are two different numbers, and the second one is the one that counts.
I'm one of those who defend long windows when the task calls for them. Extending helps when the task needs to relate pieces that sit far apart and chopping them up would break the relation. The premise in interaction two against the answer in interaction nine. Five conversations side by side. A study report you have to read in one go to keep the thread.
Extending hurts when it's a way of skipping the retrieval design: you stuff everything in because it fits. You pay 61 times more attention per layer, you add memory, and the model reads worse whatever lands in the middle. It also hurts when the window gets confused with a memory. The window stores nothing between runs.
08How we approach it in the analytics module#
A stumble of our own brought it home. In the October benchmark, Qwen spent its whole 4,096-token generation limit thinking and never got to answer in three tests. That limit was for generation; the context window is a separate thing. The lesson holds anyway: a token budget is a decision, and the default is almost never the right one.
This is the order in which we make the call, and it works for any reading task with a local model:
- Define the unit of analysis. One answer, one conversation, one comparison. The window follows from that, whatever the spec sheet says.
- Calculate the tokens of the unit plus the instructions, the output schema and a margin. If it fits in 8,192, don't ask for more.
- If it doesn't fit, ask what you can drop before extending. Embedding to locate, reranker to order, and only then the window.
- If you have to extend, scale the positions and fine-tune for about 1,000 steps on long documents from the task itself rather than generic ones, with LoRA if memory is tight. That last part is our own recommendation.
- Test with the needle grid (length by depth), with distractors, and rerun the short battery to catch regressions.
- Fix the rest of the instrument: weights, template, temperature and seed. And don't change them between two measurements.
This fits how we work on local models and data sovereignty, and what GEORadar measures: what seven AI surfaces say about a brand, with studies of 6,000 to 30,000 prompts. A reliable reading of those answers is part of the instrument. For the other side of the same coin, retrieval, there's our piece on The most expensive mistake in a vector rollout.
For us, the context window has stopped being a number on the model card and become a decision that can be justified with a simple test. If you're fine-tuning local models for specific tasks, I'd like to hear which window you ended up choosing, and why.
- RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al.). arXiv 2104.09864, April 2021. https://arxiv.org/abs/2104.09864
- Extending Context Window of Large Language Models via Positional Interpolation (Chen et al., Meta). arXiv 2306.15595, June 2023. https://arxiv.org/abs/2306.15595
- YaRN: Efficient Context Window Extension of Large Language Models (Peng et al.). arXiv 2309.00071, August 2023. https://arxiv.org/abs/2309.00071
- Extending the RoPE (EleutherAI blog on YaRN). EleutherAI, 2023. https://blog.eleuther.ai/yarn/
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (Press et al.). arXiv 2108.12409, August 2021. https://arxiv.org/abs/2108.12409
- Mistral 7B (Jiang et al.). arXiv 2310.06825, October 2023. https://arxiv.org/abs/2310.06825
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al.). arXiv 2307.03172, July 2023. https://arxiv.org/abs/2307.03172
- RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al.). arXiv 2404.06654, April 2024. https://arxiv.org/abs/2404.06654
- Landmark Attention: Random-Access Infinite Context Length for Transformers (Mohtashami and Jaggi). arXiv 2305.16300, May 2023. https://arxiv.org/abs/2305.16300
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al.). arXiv 2106.09685, June 2021. https://arxiv.org/abs/2106.09685
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (Chen et al.). arXiv 2309.12307, September 2023. https://arxiv.org/abs/2309.12307
- Distilling the Knowledge in a Neural Network (Hinton, Vinyals, and Dean). arXiv 1503.02531, March 2015. https://arxiv.org/abs/1503.02531
- 498A. The best local AI for marketing and communication: Glimmer against Qwen. 498advance.com, October 19, 2026. https://498advance.com/en/blog/mejor-ia-local-marketing-comunicacion.html
Runs 498A, the applied-AI lab of the Zoopa group, and GEORadar. Guest lecturer at UAB, where he supervises master's theses in AI with IIIA-CSIC.