LLM Wiki — Synaptic Lattice

Token Reduction Is Not Cost Reduction (Paper)

Properties1
tagsprompt-engineering, token-usage-reduction, research-gap

Leitura de 2 min

Token Reduction Is Not Cost Reduction (Paper)

Definition

A measured study (arXiv 2607.12161, Weinberger & Hozez) asking which context-reduction interventions actually reduce end-to-end billed cost while preserving task success — not just how much context they remove. Methodology: a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs (2,848 analyzed), 103 tasks, 7 repositories, 3 models, comparing baseline vs two generations of hook-based compression and an API-boundary proxy, within a ~5,500-run measured program.

Key Findings

  • Billing asymmetry dominates: prompt-cache traffic is ~87% of reconstructed cost and ~80% of the actual bill — so removing local payload tokens barely moves the bill.
  • Local payload reduction does not predict billed cost (r = 0.15).
  • Aggressive compression destroys evidence: corrupted verbatim edit anchors caused task failure.
  • The headline number: −38% tool-output tokens → +6.8% paired cost (95% CI +2.8% to +11.3%); SWE-bench-derived Go patch success fell 27/40 → 15/40.
  • Proposed metric: evaluate context-reduction systems by success-adjusted billed cost, not token reduction.

Implications

This paper is the formal, measured version of the rtk-rust-token-killer issue #582 lesson and the reason token-usage-reduction is marked contested. It generalizes the anti-pattern: any input-compression proxy (headroom included) must be judged on success-adjusted billed cost. It also explains why token-optimizer-mcp measures prompt-cache economics from transcripts — cache traffic, not payload size, is where the money is. The r = 0.15 correlation is the core empirical warning: token counts are the wrong metric. The paper’s failure mode is specific to compressing live tool evidence; skill-reducer shows the opposite result for compressing authored skill instructions (less-is-more, +2.8% quality) — the distinction is what content is removed.

Open Questions

  • Whether the findings hold for newer models with different cache pricing (version-sensitive).
  • Whether output-token shaping (Headroom’s approach) avoids the “destroyed edit anchors” failure mode.

rtk-rust-token-killer — concrete instance of the paper’s anti-pattern token-usage-reduction — contested cost-lever concept the paper refines token-optimizer-mcp — tool built to measure cache economics headroom — compression proxy whose claims should be read against this paper prompt-caching — the cost-dominant mechanism (~80-87% of the bill)

Sources

  • raw/external/arxiv-org-2607-12161-a5d415f4.md

Visão de gráfico

Backlinks