Caveman AI skill claimed 65% savings, JetBrains got 8.5%

Caveman AI skill token savings are real, but far smaller than the 65% its README advertises. JetBrains ran 86 paired SkillsBench coding tasks and measured 8.5% fewer output tokens, with no quality loss. The author agreed, then rebuilt Caveman as a local proxy that compresses input instead. Nobody outside the project has re-run the new number yet.

What is the Caveman AI skill for Claude Code?

Caveman is a Claude Code skill that rewrites the agent’s narration in clipped, grammar-free prose. Filler gets dropped, while code, commands, and error strings stay byte-exact. The repo’s tagline: why use many token when few token do trick.

It started as a student’s joke and turned into one of the most-starred agent skills on GitHub, now past 98,000 stars with 5,600 forks. You pick an intensity level (lite, full, or ultra) and trigger it with a phrase like “caveman mode” or “be brief”.

Install the CLI:

npm install -g @caveman-ai/cli
caveman setup --install

The 65% number comes from a ten-row table in the project’s README. It averages 1,214 output tokens down to 294 across chat-style questions: explain a React re-render bug, fix auth middleware, set up a PostgreSQL pool. Individual rows range from 87% down to 22%.

What JetBrains measured across 86 coding tasks

JetBrains ran the first independent paired benchmark of the skill and published the harness, the trial count, and the bill. The team used Harbor 0.17 Docker sandboxes with task-level verifiers, SkillsBench at 86 of its 87 tasks, and Claude Sonnet 5 at low reasoning effort. That came to roughly 240 billed trials for about $106.

In the straight A/B design, arm A ran stock Claude Code while arm B force-activated Caveman on every reply, so the skill could not sit out and flatter the result.

Across 82 clean paired tasks, the skill arm used 8.5% fewer output tokens: 542k against 592k. Cost per task tracked that at roughly 10% lower. Full-run totals flipped the other way, because one dependency-audit trial crossed the 200k long-context pricing tier and billed $8.29 against $0.33. That tier crossing is a property of the task itself.

Quality came out level: eight tasks scored better with the skill, ten scored worse, and 64 tied. The mean gap was 0.015 on a 0-1 scale, with a sign test p of 0.82. The JetBrains AI blog explains why the headline number shrank so far:

Forced activation is the ceiling, so auto-triggered use saves less or nothing. And an early 10-task run showed about 30% savings that collapsed as the sample grew, which JetBrains summed up as: never trust a k=1 eval.

Why chat benchmarks overstate Caveman AI skill token savings

The gap between 65% and 8.5% comes from measuring two different things. In a chat window the model’s prose is nearly the whole output, so squeezing prose squeezes almost everything. In a coding agent, the stream is dominated by diffs, tool calls, file contents, and verbatim error strings, and the skill is forbidden from touching any of it.

Output is also only one side of the bill. Files, logs, tool results, MCP responses, and the entire prior conversation get re-sent as input on every single turn. A style rule cannot reach any of that.

The skill also carries a fixed cost. The project’s own HONEST-NUMBERS.md states that the skill’s rules add roughly 1 to 1.5k input tokens per turn and that input reduction from the skill is zero. The same page records one Cursor A/B where the counters ran backwards: 4.3 million tokens with the skill against 1 million without.

If your sessions are already terse, the per-turn overhead is larger than the output you saved, and the skill costs you money.

Caveman v2 moved the compression to the input side

Instead of defending the number, the author rebuilt the tool around what the benchmark exposed and launched it on Product Hunt . Version 2 runs as a local proxy in front of your provider. It compresses tool output, logs, and JSON before the model reads them, and keeps the original recoverable byte for byte. The entry point is caveman wrap claude, and an existing Claude Code setup stays as it was.

The pinned result sits in docs/WRAP-BENCHMARK.md : 33.2% fewer provider-reported input tokens, 591,673 against 885,793, across 18 paired runs out of 54 total. All 18 exact-answer checks passed. The 95% interval runs from 14.6% to 48.5%.

Per fixture, the results spread much wider than that average:

The doc leaves the losing fixture in the aggregate rather than dropping it, and draws its own boundary too. The fixtures are deterministic large tool outputs rather than open-ended coding tasks or real traffic, and the report says the result is not a universal savings promise. That is a narrower, better-defined claim than v1 made. Still, no outside lab has re-run it, and JetBrains has said nothing about v2.

The compression is described as byte-safe and reversible, with requests still going only to the provider you configured, and that description is unaudited.

Skill, CLI, and SDKs stay MIT, while the engine, proxy, cache engine, and runtime ship under BSL-1.1 with a grant covering self-hosted first-party use. Those parts convert to Apache-2.0 on 2030-06-21 per LICENSING.md .

The announcement’s flashiest claim is undocumented: the author says every installed skill can load as an image at 70% fewer tokens while staying 99% accurate. The repo’s pixel mode is real and engine/pixel/ ships. But the published figures there are labelled inferred estimates, gated to two vision models. No paired benchmark backs them.

A compression proxy also rewrites the bytes going upstream, which changes what Anthropic’s prompt cache can hit. One commenter on the launch thread raised exactly that, noting that custom proxies and multi-account load balancing break cache stickiness and move the numbers again. Cached re-reads bill at a tenth the price of fresh input, so a proxy that trims raw input while spoiling cache hits can still lose.

JetBrains tested rtk too, and the bill went up

Caveman is not the only token-saver that shrank under measurement. rtk is a proxy that compresses shell output through a Claude Code PreToolUse hook, turning eleven lines of git status porcelain into three. Its README promised 60 to 90% less token consumption.

JetBrains put it on the same ladder: Harbor 0.18, Claude Code 2.1.201 headless, Claude Sonnet 5 at both low and high effort, the same 86 SkillsBench tasks. That ran to 425 billed trials for about $320, with endpoints pre-registered and Wilcoxon signed-rank tests.

At low effort the rtk arm was 7.6% more expensive at the median (p = 0.004), on 13.8% more turns and 14.3% more cache reads. At high effort the penalty vanished into a flat +0.1%. Quality tied in both arms.

Claude Code’s built-in Read and Grep tools bypass bash entirely, so the hook only ever sees about a fifth of tool-result characters, and cached context re-reads dominate cost anyway. Meanwhile rtk’s own analytics reported 96.2 million tokens saved, 99.8% of everything it touched, while the measured invoice rose.

The three claims side by side:

Every advertised figure came from a setting the tool controlled.

How do I test a token savings claim on my own bill?

None of the numbers above predict what your repo will do, because every one of them is workload-specific. The only figure that counts is a paired run on your real tasks, read off your provider’s billing page.

Pick 5 to 10 tasks from actual work and freeze them. Synthetic prompts will flatter whichever tool you are testing.

Run each task twice on the same model and settings, once with the tool forced on and once with it fully uninstalled, alternating the order. Then read the numbers from the provider’s own usage page and ignore the tool’s counter.

Count input, cache reads, and output separately. A tool that shrinks output while adding turns can still cost more.

Score the quality of each run as well as its tokens. The cheap version of the JetBrains method is a sign test: count how many tasks came out better, worse, and tied.

Finally, repeat the whole run at least three times. The 30% that became 8.5% was a sample-size artifact, and one run of anything tells you almost nothing.

What the community made of the climbdown

The author posted the correction himself in r/ClaudeCode , where it reached 112 upvotes. The top reply read the confession as AI-written PR:

The sharpest point came from u/En-tro-py, who pointed out that public benchmarks such as ContextBench, SlopCodeBench, and Terminal-Bench already exist. Skill projects substituting their own methods is how the over-claim happened in the first place.

When not to use Caveman

• Your plan bills per request rather than per token, so a shorter answer is still one Copilot-style request.
• Your sessions are already short and terse, and the skill’s per-turn overhead swallows the saving.
• You cannot run a local proxy in front of your provider for policy reasons, which v2’s input compression needs.
• You need a savings number you can put in a budget, and every figure here depends on the workload.