Practical context engineering for LLM applications: budgeting context, selecting what to include, ordering, compaction, and diagnosing quality loss from context bloat.
Context engineering for LLM applications is the discipline of deciding what a model sees on each call. It has largely displaced prompt wording as the main lever on quality, because in production systems the failure is usually what was included or omitted, not how the instruction was phrased.
Larger context windows have not removed the problem. They have made it easier to fill context with material that degrades reasoning.
Treat context as a budget
Every token in context has a cost in money, latency, and attention. Attention is the one teams underweight: models attend unevenly across long contexts, and material buried in the middle of a large context is often effectively ignored.
Set an explicit budget per call and allocate it. A workable default is a small fixed allocation for instructions, a bounded allocation for retrieved evidence, a bounded allocation for conversation history, and headroom for output.
When the budget is exceeded, something must be dropped deliberately. Systems without an explicit policy drop whatever falls off the end, which is frequently the most important item.
Select rather than include
The instinct to include everything relevant is the main source of degradation. Retrieval that returns twenty passages when three are relevant does not give the model more to work with; it gives it seventeen distractors.
Retrieve broadly, then rerank and cut hard. Precision matters more than recall for generation quality, which inverts the usual retrieval intuition.
Filter by metadata before similarity where you can: tenant, recency, document type, permissions. This removes wrong-scope material cheaply and is also a security control.
Order matters
Place the instruction and the immediate question where the model attends most reliably, which in practice means near the beginning and again near the end for long contexts.
Put the most relevant evidence closest to the question. If reranking produced an ordering, preserve it rather than shuffling.
Keep system instructions stable across calls so prompt caching can apply. Reordering stable content defeats caching and raises cost with no benefit.
Compaction for long-running sessions
Agent sessions and long conversations exceed any window eventually. Handle it with a deliberate compaction strategy rather than truncation.
Summarize completed segments into structured facts rather than prose. "Established: customer ID 4471, order 8823, refund policy allows 30 days, order placed 45 days ago" preserves what matters in a fraction of the tokens.
Keep the original available for retrieval when detail is needed, so compaction loses nothing permanently.
Never compact away commitments, constraints, or corrections. These are the items whose loss produces the most visible failures, where the system contradicts something it agreed to earlier.
Mark provenance
Label what came from where: system instruction, user input, retrieved document, tool result, memory. This supports two things.
It lets you apply security controls, treating retrieved and tool content as untrusted for the purpose of authorization decisions.
It lets you debug. When a model produces a wrong answer, the first question is which context item caused it, and unlabeled context makes that unanswerable.
Diagnose context problems
When quality degrades, check context before adjusting prompts.
Log the full assembled context for failing cases, and read it. The cause is usually visible: a missing document, an irrelevant one ranked first, stale memory contradicting current facts, or history that crowded out the evidence.
Test by ablation. Remove suspected distractors and re-run. If quality improves, the problem was inclusion, not instruction.
Watch for degradation as context grows. If accuracy falls as sessions lengthen, compaction is inadequate.
Frequently asked questions
Does a larger context window remove the need for this?
No. Cost, latency, and uneven attention persist. Larger windows raise the ceiling on what you can include, not the value of including it.
How much retrieved content should be included?
Fewer, better-ranked passages generally beat more. Tune on your evaluation set rather than on a default.
Should conversation history always be included?
Only what remains relevant. Compact completed segments into structured facts instead of carrying raw turns.
What is the fastest quality win?
Reranking retrieval results and cutting to the top few. It typically improves accuracy and reduces cost at the same time.
Explore our software and web capabilities or discuss your AI workflow.
Written by
Moueen Togarvi
Founder & CEO at Voquarn Code, focused on product engineering, search growth, and practical AI systems.