Post

Cutting the Cost of LLM and AI Agents: Trajectory Reduction as an Inference-Time Efficiency Lever

Cutting the Cost of LLM and AI Agents: Trajectory Reduction as an Inference-Time Efficiency Lever

The software engineering community has treated the compute cost of coding agents as an implementation detail rather than a research question. Reducing Cost of LLM Agents with Trajectory Reduction (Xiao, Gao, Peng, and Xiong — Peking University’s Key Lab of HCST and ByteDance, FSE 2026) argues that this is a mistake, and backs it with a number that reframes the whole problem: in September 2025, daily usage of Claude 4 Sonnet on OpenRouter reached 100 billion tokens, of which 99 percent were input tokens accumulated in trajectories and only 1 percent were generated by the model.

The cost of an agent is not what it writes. It is what it drags along behind it.

That matters commercially, not just academically. The authors cite a StackOverflow survey in which 53 percent of professional developers named agent cost as a barrier to adoption, while 24 percent were already using LLM agents daily or weekly in 2025. Adoption is outrunning the economics.

The snowball

The diagnosis is the ever-growing trajectory. In a standard agent loop, once the LLM calls a tool, the call and its result are concatenated into the trajectory and retained until the task completes. Open one large file, or run one command with verbose output, and those tokens are re-transmitted in every subsequent prediction. Cost snowballs by construction.

The anatomy is worth having in front of you. Using SWE-bench Verified and trajectories collected from Trae Agent on Claude 4 Sonnet, the average trajectory for a single GitHub issue looks like this:

SegmentTokens
Tool messages30.4K
Assistant messages13.7K (11.9K of it tool-call arguments)
System and user messages4.4K
Trajectory total48.4K across ~40 steps
Accumulated input per issue~1.0M

Forty-eight thousand tokens of state becomes a million tokens of billed input. That ratio is the entire paper.

The obvious objection is KV caching, and the authors take it seriously. Caching does not remove the need for trajectory reduction, for two reasons. It caches only part of the computation, and the remaining work still grows with trajectory length. And it consumes real hardware — VRAM and I/O bandwidth — so a shorter trajectory conserves resources even when the cache hits.

More interestingly, caching makes reduction harder to design, because modifying any token in the trajectory invalidates the cache for everything after it. That constraint drives the architecture.

There is also a quality argument. Long contexts degrade model performance through the well-documented “lost in the middle” effect, so removing content does not automatically cost you accuracy. It can pay for itself twice.

Three kinds of waste

Manual inspection of 100 Trae Agent trajectories found waste in almost all of them, in three categories.

Useless information — content irrelevant to the task. Nearly every trajectory opens with a tool call enumerating all repository files, and the response includes cache and build artifacts: __pycache__, .egg-info. Build and test commands are worse. GNU make prints make[2]: Entering directory '...' for every location it visits. Trae Agent truncates tool output at 16KB, but a fixed threshold only clips the extreme case; useless content sits comfortably under it.

Redundant information — the same content appearing more than once. The canonical case is str_replace_editor, Anthropic’s standard file-editing tool. Its arguments arrive as JSON — {"command": "str_replace", "path": "F", "old_str": "P", "new_str": "Q"} — and then the replacement string Q reappears in the tool’s response, often alongside an echo of the original P. You pay for the same bytes two or three times.

Expired information — content that was relevant and no longer is. The contents of a file that has since been edited. The output of a test run superseded by a later one.

The authors frame their design as four questions. Is there waste? A great deal. Can it be identified automatically? Yes, an LLM can do it. Will agents do it on their own initiative? No — which is why reduction has to be enforced externally rather than trusted to the agent’s judgment. Is calling an LLM for this ideal? Not really, but the overhead is controllable.

That third answer is the one I would underline for anyone building this in-house. Asking the agent to manage its own context is not a strategy.

AgentDiet

The resulting approach, AgentDiet, is an inference-time system with four design decisions that are each a response to a specific constraint.

An external reflection module. Compression lives outside the agent’s reasoning loop, in a separate LLM-based module that processes trajectory segments. The reduction policy stays independent of the task policy, and the agent’s decision-making is not perturbed.

A sliding window. The module processes a target step (s − a) using only local context spanning steps (s − a − b) through s. The delay parameter a keeps it away from the most recent steps, which are the most likely to still matter; the context parameter b gives it enough surroundings to judge correctly.

Threshold-gated filtering. Only steps longer than a threshold θ are processed. If a reduction fails to clear a minimum savings bar, the original segment is kept — a conservative fallback that refuses changes which do not pay for themselves.

Cache preservation. Because only a bounded, recent window is ever rewritten, the trajectory prefix stays byte-stable and KV cache entries for earlier steps survive. This is the direct answer to the cache-invalidation constraint identified up front, and it is the most quietly important choice in the design.

Tuning landed at θ = 500 tokens, a = 2, b = 1. Within the steps it processes, the reflection module removes 69.2–77.4 percent of the content.

What it delivers

The evaluation runs AgentDiet inside Trae Agent, with GPT-5 mini as the reflection module and either Claude 4 Sonnet or Gemini 2.5 Pro driving the agent — a deliberate split showing the compressor need not be the same model, or the same family, as the agent. Benchmarks are SWE-bench Verified (500 Python tasks) and Multi-SWE-bench Flash (300 tasks across seven languages).

Four arms are compared: Original, LLMLingua-2 (a learned prompt compressor), Random removal (a control), and Delete (aggressive whole-turn truncation). Including both a learned compressor and a random control is the methodologically right call — it separates the value of intelligent selection from the value of mere reduction.

MetricResult
Input token reduction39.9% – 59.7%
Total computational cost reduction21.1% – 35.9%
Pass% variation vs. original−1.0% to +2.0%

The gap between those first two rows is the honest part of the paper. Roughly a third of the gross token savings is eaten by the compression machinery itself, because the reflection module spends tokens to make its judgments. Any vendor quoting you the 59.7 percent figure without the 35.9 percent figure is quoting the wrong number.

Performance holds across both models and both benchmarks, including the multi-language set — so the waste categories found in Python trajectories are not a Python artifact.

Where it is soft

The authors are candid. The reflection step adds latency, which no token or dollar accounting captures. They note it could be removed by a design change, at the cost of shrinking the context available to the module. Experiments were confined to Trae Agent because API costs are real, which limits claims about generalization across scaffolds. Benchmark leakage into training data remains a threat, and validation depends on test cases, with the attendant overfitting risk — addressed, though not eliminated, by held-out data.

The argument this paper walked into

AgentDiet’s real conceptual contribution is reframing trajectory reduction as an inference-time, timing-sensitive problem rather than a one-shot input-compression problem. In single-turn compression you decide once. In an agent, tokens accumulate gradually, so when you compress becomes a research question in its own right. Coding agents also handle structured source code rather than prose, and most existing methods require modifying the inference process or the model — impractical against proprietary APIs. The authors note that Claude Code and Cursor apply LLM-based compaction sparingly, only when the window fills or the user asks, optimizing for robustness rather than efficiency.

That framing puts the paper directly into a fast-moving 2025–2026 literature, and it does not go unchallenged.

Simpler baselines may win. The JetBrains Research study The Complexity Trap found that plain observation masking halves cost relative to a raw agent while matching or exceeding LLM summarization solve rates. It explicitly criticizes AgentDiet for not testing against an observation-masking baseline, and notes that AgentDiet’s “Delete” arm breaks the reasoning trace across turns and degrades downstream performance. That critique is fair, and it names the obvious next experiment: AgentDiet versus observation masking, head to head.

Token savings may not be cost savings. Token Reduction Is Not Cost Reduction (Weinberger and Hozez, PointFive) measured provider-billed cost in Claude Code and found the most aggressive compression cut delivered tool-output tokens by 38.4 percent while raising the bill by 6.8 percent, with a near-zero per-task correlation (r = 0.15) between token reduction and billed-cost change. Their decomposition attributes about 87 percent of cost to prompt-cache creation and reads, leaving roughly 6 percent addressable by user-side compression. This lands squarely on AgentDiet’s headline, which is computed from token accounting rather than invoices. The mitigating detail is that AgentDiet’s cache-preserving sliding window is exactly the design that would protect against this — which makes a billed-cost replication of AgentDiet one of the highest-value experiments available right now. I wrote about the PointFive paper here.

Solve-rate parity can hide interaction cost. What Does Context Compression Cost an Agent? (Liu) shows compression driving up the cost of reacquiring dropped state while task completion stays statistically flat: in one configuration GPT-5.5’s completion was unchanged (80% → 85%, p = 1.0) while retrieval calls roughly tripled (21.0 → 63.9, p = .002). Control Under Compression (Hou and Yang) maps a nonlinear reliability frontier, with success falling from a 93.8 percent full-context baseline to as low as 19.9 percent at 35 percent retained context. Both bear directly on AgentDiet’s parity claim — a −1.0% to +2.0% Pass% band could be concealing a real increase in turns, tool calls, and re-exploration.

The mechanism is caching. Don’t Break the Cache (Lumer et al., PwC) measured prompt caching across 500+ agent sessions, finding 41–80 percent API cost reductions while showing that naive full-context caching can paradoxically increase latency. Cache-Aware Prompt Compression (Song) shows query-aware compression mechanically invalidating prefix-strict caches — on tau-bench it was the most expensive strategy tested, at +40.1 percent over vanilla. TokenPilot (Xu et al.) builds a dual-granularity framework to reconcile text sparsity with cache alignment, reporting 56–87 percent cost reductions.

Taken together, the frontier has moved from “how much text can we remove” to “how do we remove text without breaking cache continuity.” AgentDiet anticipated that question with its sliding window. It did not evaluate against billed cost.

What to take from it

AgentDiet opens a direction rather than closing one, and it does so on solid ground. Establishing that 39.9–59.7 percent of input tokens in coding-agent trajectories are removable waste — and that removing them nets 21.1–35.9 percent in computation — is enough to make inference-time trajectory reduction a real lever rather than a hypothesis. The cache-preserving window and the threshold-gated fallback are sensible engineering, not just paper decoration.

The open questions are the ones the follow-on literature has already started answering: whether token savings survive contact with provider billing, whether solve-rate parity conceals interaction cost, and whether observation masking gets you most of the way there for a fraction of the complexity. The paper calls itself a preliminary study. That is the right label, and the direction has proven productive.

If you are running agents at scale, the practical order of operations is unchanged by any of this. Instrument billed cost with cache splits before you compress anything. Then measure cost per successful task, not tokens removed. Trajectory reduction is a genuine lever — just make sure you are watching the invoice while you pull it.


References

Primary paper

Xiao, Y.-A., Gao, P., Peng, C., and Xiong, Y. “Reducing Cost of LLM Agents with Trajectory Reduction.” Proceedings of the ACM on Software Engineering, Vol. 3, FSE056, July 2026. DOI: 10.1145/3797084. arXiv:2509.23586.

Related work

  • Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., and Zharov, Y. “The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management.” arXiv:2508.21433, 2025 (v3, October 2025). 4th DL4C Workshop, NeurIPS 2025.
  • Weinberger, S., and Hozez, A. “Token Reduction Is Not Cost Reduction: An Empirical Study of End-to-End Efficiency in API-Based Coding Agents.” arXiv:2607.12161, July 2026 (v5, August 2026).
  • Liu, S. “What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics.” arXiv:2608.16370, August 2026.
  • Hou, Y., and Yang, Z. “Control Under Compression: Reliability Frontiers for Tool-Using Agents.” arXiv:2608.01056, August 2026.
  • Kang, M., Chen, W.-N., Han, D., Inan, H. A., Wutschitz, L., Chen, Y., Sim, R., and Rajmohan, S. “ACON: Optimizing Context Compression for Long-Horizon LLM Agents.” arXiv:2510.00615, October 2025 (v3, June 2026). ICML 2026.
  • Lumer, E., Nizar, F., Jangiti, A., Frank, K., Gulati, A., Phadate, M., and Subbiah, V. K. “Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks.” arXiv:2601.06007, January 2026.
  • Xu, B., Xue, Z., Chen, D., Fu, C., Wu, C., Huang, C., Jiang, C., Fang, J., Deng, X., Chen, Y., Yao, Y., Wang, X., Shang, J., Yu, G., and Zhang, N. “TokenPilot: Cache-Efficient Context Management for LLM Agents.” arXiv:2606.17016, June 2026.

Background

  • Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv:2310.06770, 2023.
  • Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. “ReAct: Synergizing Reasoning and Acting in Language Models.” arXiv:2210.03629, 2022.
  • Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. “Lost in the Middle: How Language Models Use Long Contexts.” arXiv:2307.03172, 2023.
  • Gao, P., et al. “Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling.” 2025.
  • Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., Zhao, H. V., Qiu, L., and Zhang, D. “LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression.” arXiv:2403.12968, 2024.
  • Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” arXiv:2309.06180, 2023.
This post is licensed under MIT by the author.