Study: AI agents forget user rules when compressing context

When AI agents compress long conversations to free up context window space, user instructions get lost systematically. That is the finding of "Lost in Compaction," a new study from Pennsylvania State University: common compaction methods retain on average only 17 percent of so-called session constraints – 83 percent of the rules disappear (arXiv).

These are temporary rules such as "confirm with me before making changes" or "don't send emails without my approval." The researchers warn: "A task-centric compactor may preserve the task but drop the constraint, causing a silent integrity failure." The agent keeps working – while unknowingly violating the user's instructions.

For the study, the team built the CompInt evaluation suite with 750 evaluation instances per condition across three datasets – multi-turn chat, tool-using trajectories and long-horizon research tasks. More than six compactors were tested, from simple truncation to LLM-based methods.

The results are stark: non-LLM methods achieved 0 percent retention, open-source LLM compactors 0 to 36 percent. Only the best tested model reached 88 to 98 percent. Retention degraded further as context length grew from 10,000 to 100,000 tokens.

The researchers propose a fix: a plug-and-play extractor module that secures user rules separately lifts retention above 90 percent – 95.6 percent on agent trajectories, 95.1 percent on research tasks, 90.3 percent on chats – without modifying the compactor or the base model.

For the booming AI agent market, the finding is significant: especially in long, autonomous runs, context management determines whether a user's safety and approval rules are actually respected.