Context engineering: why a longer prompt doesn't always help

What should be included in the context?

Separate the application's permanent rules, the user's current command, and the materials used to generate the response. These elements serve different roles. A client document is a data source, not a place for defining application permissions. The outcome of a previous conversation may be helpful, but it should not automatically become a binding rule.

For each fragment, ask: what decision is it meant to support? If the material does not influence the task, it may not need to be included. For example, a draft response regarding a delivery deadline does not require the client's entire marketing history. A smaller, deliberately selected scope also facilitates data control and diagnosing unwanted responses.

Context and references: Anthropic: Effective context engineering for AI agents

Conversation history is not a source of truth

In a long conversation, the user changes assumptions, and the model presents hypotheses. Without organization, an old proposal may be treated as an approved decision. Store important facts in a structure describing their status: confirmed, working, or rejected. Keep a reference to the source if later verification is important.

Example: A client first requests implementation in three branches, then limits the pilot to one. The summary should reflect the current scope and indicate the change. Simply shortening the conversation is not enough if it removes information about the decision. Test summaries on tasks where the user corrects previous agreements.

Retrieve information when it is needed

Instead of putting the entire knowledge base into every task, the application can search for the right sources or run a limited tool. It is important to clearly distinguish between missing data and a retrieval error. The model should not fill in a system failure with a default assumption about order status.

Tool outputs should include units, dates, and field meanings. A value of “120” without a currency or identifier is not useful context. At the same time, do not send unnecessary technical fields. Design the format based on tasks and errors that actually occur. A shorter tool response may be more informative than a full record dump.

How to evaluate whether a change helps?

Compare the same set of tasks before and after changing the context selection method. Evaluate factual accuracy, missed requirements, time, and cost. Add long conversations, similar documents, and conflicting versions. Ensure that improvement for a short question has not worsened a complex task.

Record which sources and versions were included in a specific trial, with reasonable data retention limits. This ensures diagnosis does not end with the statement that the model “sometimes does that.” First, determine whether it received the right information. Only then assess whether you need a different model, search method, or better instruction.