Original Summary

I audited 73 public benchmark agent trajectories for wasted tokens — 74% carried measurable waste. Then I ran live agents to check. chewme-audit (MIT) is a CLI I built that reads agent traces — Langfuse, LangSmith, plain JSONL, and Claude Code transcripts — and reports where tokens were wasted, priced per provider, with a copy-paste fix for each pattern. It runs locally: no API calls, nothing uploaded. Repo: https://github.com/littleman3002/ChewMe-Audit Install: pip install chewme-audit, then chewme-audit your_traces.jsonl --provider deepseek Clearest case: one trajectory sent the same malformed edit command 283 times before hitting its step limit. I applied the recommended dedup fix and re-ran it with a real model call: the loop broke on the next call and the agent went back to investigating the actual bug. Waste on that trace dropped 77% (13,193 → 3,041 tokens), with a tamper-evident SHA-256 receipt for the result. Cost per million wasted tokens (priced as cache misses, which is what these waste spans were on the measured traces; prices as of Oct 2026): DeepSeek $0.15 · Gemini $0.75 · Grok $1.25 · Claude Sonnet $2.00 · GPT-5 class $4.00 Then I ran live agents on a real task: - Solo agent: 35% of its I/O was re-verification — the same "find files" script re-run 7 times. Competent, paranoid. - 4 identical agents, same task, no coordinator: waste ranged 15–37%. One never finished. One deleted 117 lines of real docs after skimming files with head -3 and mistaking a 60-line docstring for a stub (root cause in the write-up: https://github.com/littleman3002/ChewMe-Audit/blob/main/traces/live\_runs/WORKING\_NOTE.md ). - A debugging run passed 12/12 for $0.013 — but the trace shows the fix arrived through an accidental git checkout -- (a rerun passed 12/12 cleanly for $0.0077). Pass/fail would have certified the first run; the trace didn't. Honest scope: it reads trace exports, so cache behavior and reasoning-token overspend need your API logs, and the report says so rather than guessing. Every dollar figure is labeled estimated unless you provide logs. The 74% comes from benchmark trajectories; benchmark agents loop more than production ones, so your own traces are the real test. Try it on your traces and tell me what it finds. I'm especially curious whether the re-verification pattern dominates elsewhere.   submitted by   /u/Impressive_Laugh_176 [link]   [comments]

  • 情报分类：商业与市场研究
  • 分类依据：内容涉及商业、投资或市场动态
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/10/20 04:45:31