When using an AI coding agent, with Claude Code, OpenCode, or Cursor for example, every token sent as context has a cost. A verbose git log, a 500-line npm install, or a pytest -v run listing 600 tests all enter the context window and increase that cost. The idea of compressing this output before it reaches the LLM is therefore appealing. That is exactly what RTK does.

However, our analysis of real sessions points to a counter-intuitive result: when useful information disappears, the agent’s extra attempts can cancel out the expected savings.

TL;DR: for readers and LLM agents

• This analysis was conducted with RTK 0.30.0.
• Compression is useful for long, repetitive, and structurally stable output.
• When diagnosing an error or reusing an exact value, such as a traceback, token, identifier, or JSON response, prefer the unfiltered command.
• If an RTK result is empty, ambiguous, or unexpected, run the command once without RTK before changing code.
• Measure an agent’s total number of turns, not only the tokens saved by each command.

RTK: the idea is sound

RTK (Run Tool Kit) is a CLI proxy that intercepts system commands and summarises their output. Instead of returning 200 lines of pytest logs, it can produce one line: ✓ Pytest: 17 passed. Instead of a verbose ls -la, it can show a compact tree with sizes.

On paper, this is exactly what is needed. An 8,000-character pytest output (roughly 2,000 tokens) is replaced by 30 characters (roughly 8 tokens). Across hundreds of commands, the saving is real.

A real case: a 28-message loop

We analysed a pair-programming session using an AI-session observation tool. The numbers were:

• 848 messages, 103M input tokens, estimated cost: $12.90
• 261 bash commands, including 206 through RTK (79%)
• The agent used rtk pytest for almost every test run

Everything worked until the end of the session. The agent ran:

rtk pytest tests/backend/unit/ -x -q

RTK answered:

Pytest: No tests collected

The agent tried again. Same result. Then again. Four times in a row. It eventually changed the flag from -q to -v, and got:

✓ Pytest: 561 passed, 1 failed

The tests had been collected and executed all along. The -q option changes pytest’s output format, and RTK misread that format as “No tests collected”.

A second issue appeared when the agent moved to integration tests:

rtk pytest tests/backend/integration/test_api.py -x -v

Again: “No tests collected”. This time there was a real problem: an ImportError in the Python code prevented test collection. RTK reduced the full traceback to an innocuous one-line summary. The agent never saw the actual error.

It then made 17 attempts: changing flags, targeting specific classes, adding --tb=short, and using --co, without obtaining the traceback that would have explained what to fix.

Only after abandoning RTK and running python -c "from myapp.routes..." directly did it finally see:

ImportError: cannot import name 'router' from 'myapp.routes.session_routes'

The numbers behind the paradox

Here is the initial estimate:

Metric	Value
Average context during the loop	~163,000 tokens
Messages in the loop	28
Input tokens billed during the loop	~4.6M
Equivalent at full input price ($3/M)	~$14

By comparison, RTK’s savings for the entire session, not just this loop, were probably $2 to $3 of tokens, across the output of 261 commands.

The ~$14 figure is an equivalent at full price, not the actual billed cost: it applies $3/M to every token without distinguishing cached reads. The detailed estimate below puts the 24 observed turns at roughly $1.68, compared with $0.21 for a direct three-turn path. The cost attributable to this loop is therefore closer to $1.47. This single case does not demonstrate a 5:1 ratio; it shows that one lost signal can absorb a significant share of the expected savings.

The second case: RTK + curl and the loss of useful data

If the pytest case illustrates a loss of error signal, the curl case illustrates another limit: the loss of data needed for the next step.

We analysed a second session (348 messages, 41M input tokens) in which the agent was building a REST API. Naturally, it tested endpoints with curl, and naturally, it prefixed those commands with rtk:

rtk curl -s http://localhost:8000/api/auth/login -X POST -d '{"email":"test@test.com","password":"pass"}'

The server’s raw response was:

{
  "access_token": "<example-access-token>",
  "token_type": "bearer",
  "user": { "id": 42, "email": "test@test.com" }
}

What RTK sent back to the agent was:

{access_token: string, token_type: string, user: {id: number, email: string}}

The JWT was replaced by string. The actual values disappeared. RTK turned a response into a type schema, useful for documentation, but not for debugging.

The phantom JWT

The limitation becomes concrete when the agent needs the token to authenticate subsequent requests. It gets a token through /api/auth/login, but RTK masks it:

access_***REDACTED***

The agent no longer has the token. It cannot build the Authorization: Bearer <token> header, so it calls /api/auth/login again. RTK redacts it again. The agent retries.

There were 22 calls to /api/auth/login in that session. 15 were unnecessary, caused by RTK destroying the token each time.

A correct, but temporary, diagnosis

Halfway through the session, the agent wrote:

“rtk truncates the response, let me try without it”

It switched to direct curl. It worked: it got the token, used it, and progressed.

But a few exchanges later it went back to rtk curl. It had forgotten its own conclusion. This is characteristic of agents in long sessions: local discoveries do not reliably survive context compaction. The session had 6 compactions, moments when the provider summarised history to remain within the context window. At every compaction, learned lessons can disappear. The longest run without one was 53 messages, more than enough for “do not use RTK with curl” to be pushed out of context.

Numerical summary

Metric	Value
Total curl commands	41
Through RTK	14 (34%)
Direct	27 (66%)
Calls to /api/auth/login	22
Unnecessary calls caused by RTK	15
Context compactions	6

The paradox is the same as with pytest, but the mechanism differs. With pytest, RTK hid errors. With curl, RTK removed working data, including JWT tokens, identifiers, and response values the agent needed to chain requests.

Reproducible tests: a taxonomy of bugs

The previous cases are not anecdotes. We systematically tested RTK v0.30.0 against nine different commands to map what breaks and what works.

🔴 Hidden errors

rtk pytest -q (the -q flag): when tests pass, RTK summarises them correctly with -v. With -q, though, the output format changes and RTK reports “No tests collected” even though tests passed:

# Direct: actual result
$ pytest tests/ -q
2 passed, 1 failed in 0.03s

# Through RTK: the useful signal disappears
$ rtk pytest tests/ -q
Pytest: No tests collected

The agent sees “No tests collected” and retries. It does not know the tests already ran.

rtk pytest + ImportError: when a Python module has an import error, pytest prints a full traceback. RTK summarises it as “No tests collected”, with both -q and -v:

# Direct: the traceback is visible
$ pytest tests/ -v
ImportError while importing test module '/tmp/test_api.py'
ModuleNotFoundError: No module named 'myapp.routes'

# Through RTK: the error disappears
$ rtk pytest tests/ -v
Pytest: No tests collected

The agent cannot diagnose the issue and keeps trying without understanding that it is an import problem.

🔴 Loss of useful data

rtk curl (JSON responses): RTK replaces JSON values with their types:

# Direct: usable data
$ curl -s https://httpbin.org/get
{"headers": {"Host": "httpbin.org", "User-Agent": "curl/8.7.1"}, "origin": "1.2.3.4"}

# Through RTK: a type schema, not data
$ rtk curl -s https://httpbin.org/get
{headers: {Host: string, User-Agent: string}, origin: string}

rtk json (JSON files): the same behaviour as curl. A JSON file containing "token": "abc123" becomes token: string.

rtk env (environment variables): inconsistent masking. AWS_SECRET is partially masked (AK****LE), while API_KEY and DATABASE_URL disappear from output entirely: they are not merely redacted, but absent.

🟢 Working correctly

Command	Behaviour
rtk go test (failures)	Error message preserved in full
rtk go build (compile error)	Error preserved
rtk gh api (with --jq)	JSON passed through correctly
rtk read (files)	Full content preserved

The observed pattern is clear: RTK works well for tools with structurally stable output (Go, gh). It becomes less reliable when output depends on a format (pytest -q versus -v), contains working data (curl, JSON), or is subject to overly aggressive masking (env).

A recurring pattern: 168 attempt sequences across 40 sessions

The two case studies and reproducible tests could still look isolated. To check, we analysed 40 real assisted-programming sessions through the API of an AI-session observation tool.

Raw numbers

Metric	Value
Sessions analysed	40
Total messages	17,000+
Bash commands	20,593
Detected attempt sequences	168

An “attempt sequence” is the same command, or a minor variation, run three or more times consecutively without a change in result.

The most affected commands

RTK command	Affected sessions	Longest sequence
rtk pytest	~20 sessions	26 consecutive attempts
rtk tsc --noEmit	~12 sessions	the same command 3×
rtk git commit	~12 sessions	up to 8×
rtk grep	~5 sessions	5×

The longest sequence contained 26 consecutive attempts of rtk pytest in one session. The agent reran the same command with minor option changes, never receiving the information it needed.

Not an isolated incident

168 sequences across 40 sessions means 4.2 sequences per session on average. Each sequence, in a 100K–200K-token context, can add $1 to $5 of tokens without advancing the task. Across those 40 sessions, the associated additional cost is estimated at $200 to $800, for sessions totalling around $400 to $500 in normal cost.

The same trends appeared repeatedly: similar commands, masked errors, and renewed attempts. They sometimes occurred at the same message indexes in separate sessions because some sessions shared history from a branch or fork.

Why every extra attempt is expensive

To understand the paradox, it helps to understand how an AI agent is billed. The intuition that “one message equals one API call” is wrong. When you say “run the tests and fix the errors”, this is what happens:

You: "Run the tests and fix the errors"

  → API call #1: [entire context] → Claude reasons → tool_call: bash("pytest")
  → API call #2: [context + pytest result] → Claude reasons → tool_call: edit(file)
  → API call #3: [context + edit result] → Claude reasons → tool_call: bash("pytest")
  → API call #4: [context + pytest result] → Claude reasons → "It is fixed!"

Every tool call is a separate API call. At each call, the entire context is sent again: all previous messages, all tool-call results, and the system prompt. Each turn includes:

• Input tokens: the accumulated context. With prompt caching, previously seen tokens cost roughly $0.30/M rather than $3/M (Sonnet). But the new tokens at each turn, such as the last tool-call result, are charged at full price.
• Output tokens: what the model generates (reasoning plus tool call). In the observed pricing model, they cost $15/M and are not cached.

Even with caching, cached tokens are not free: they cost 10% of the normal price. And the context grows at every turn.

The concrete cost of an attempt sequence

Take a 160K-token context, typical in the middle of a session:

	Normal turn	Extra turn
Input (cached)	160K × $0.30/M = $0.048	Same
Input (new)	~2K × $3/M = $0.006	Same
Output	~1K × $15/M = $0.015	Same
Total	~$0.07	~$0.07
Progress	✅ moves forward	❌ none

$0.07 for a turn with no progress. But a new attempt is not just a rerun. The agent also spends turns analysing the result, trying variants, and looking for the cause. One visible rerun can therefore generate 3 to 5 API calls.

Here is the actual scenario when rtk pytest returns “No tests collected” even though tests pass:

Turn 1:  rtk pytest -q → "No tests collected"             $0.07
Turn 2:  Claude reasons, tries -v                            $0.07
Turn 3:  rtk pytest -v → "No tests collected" (hidden ImportError)
Turn 4:  Claude tries --tb=short                             $0.07
Turn 5:  rtk pytest --tb=short → "No tests collected"       $0.07
  ... 17 turns later ...
Turn 22: Claude abandons RTK → direct pytest → TRACEBACK     $0.07
Turn 23: Claude reads the error → edits the file             $0.07
Turn 24: pytest → passes                                     $0.07

Total cost: ~24 × $0.07 = $1.68. Without RTK, the traceback would have appeared on turn 1 and been fixed by turn 3. Cost: $0.21. Ratio: 8:1.

That is only one loop. With 168 observed sequences across 40 sessions and contexts sometimes exceeding 200K tokens near the end of a session, the extra cost can accumulate quickly.

The real-cost formula

An agent’s cost is not tokens × price. It is:

cost = Σ (context_size_at_turn_i) × price_per_token
       for every turn i

RTK optimises context_size by compressing outputs. But when it hides an error, it multiplies the number of turns. In a large context, the number of turns dominates. That is the aggressive-compression paradox: the larger the context, the more every wasted turn costs, and the more critical it becomes for the agent to see errors in full.

There is no single solution

It is tempting to seek the solution: a better compression algorithm, an automatic fallback, or an intelligent mode that knows when to compress. But the data shows that there is no single solution.

Every path has limits:

Categorising commands (“never compress pytest, curl, xcodebuild”) works for known cases. But RTK bugs also affect grep, npm, and git, commands we would like to compress. Every new tool and output format is another possible false positive.

Automatic fallback: if RTK detects “No tests collected”, rerun without filtering. This only works for known errors. The hidden-ImportError problem did not consistently return “No tests collected”: RTK could produce empty or truncated output. Generalising that detection without introducing false negatives remains difficult.

Prompt caching reduces cost per turn but does not change the underlying dynamic. Even at 10% of the price, 24 wasted turns in a 200K-token context cost more than three turns at full price.

Instructions in AGENTS.md are the most robust approach, but they also have limits. An agent can follow them, yet in a long session with context compactions even system instructions can be summarised away or evicted. Maintaining an exhaustive list of commands to exclude for every project is continuous work.

What works in practice is a combination of complementary approaches:

1. Clear rules in AGENTS.md. Two simple instructions cover 80% of cases:

1. Avoid RTK for: pytest, curl, xcodebuild, python -c
2. If a command through RTK fails or returns an unexpected result,
   rerun it without RTK to obtain the full output.

2. Post-session observation. An analysis tool can identify attempt sequences and quantify their cost after the fact. The goal is not real-time correction, but making the behaviour visible so that rules and tools can be adjusted. Analysing 40 sessions is how we found that rtk pytest returned “No tests collected” 51% of the time.

3. Discipline. Do not make a compression proxy the default for every command. Start without it, measure, and enable compression only for commands where the data demonstrates a net benefit.

The underlying issue is that token optimisation and agent reliability are in tension. Compression can remove potentially useful information. Sometimes that information is redundant, such as the 200th line of an npm install. Sometimes it is the traceback that would have solved the issue in one turn. It is difficult to know in advance.

Conclusion

Token optimisation is a healthy instinct when working with AI agents. But, like any optimisation, it follows the law of diminishing returns. Here, returns can become negative much sooner than expected.

The observations converge across two case studies, nine test scenarios, and 40 analysed sessions:

• Hidden errors (pytest): 51% of RTK runs returned “No tests collected”. The agent can no longer diagnose.
• Loss of useful data (curl, JSON): values were replaced with type schemas. The agent can no longer chain its steps.
• At scale: 168 attempt sequences, 100 times the agent itself abandoned RTK, 204 expirations, and 363 empty outputs.

Context compactions can reinforce this pattern. Our data contains 9 moments where the agent wrote “let me try without rtk”, then later returned to the filtered command. An agent’s working memory is volatile; persistent instructions in AGENTS.md can help retain the rule.

There is no single solution. Output compression is neither good nor bad: it is a trade-off. It requires measuring lost information as carefully as saved tokens. On RTK 0.30.0, data from 40 sessions, 20,000 commands, and 168 attempt sequences showed that the net benefit could become negative.

The real question is not “should outputs be compressed?” It is: do you have the tools to know when compression costs you more than the tokens it saves?

Update

This article is an experience report on RTK 0.30.0, published on 6 April 2026. A separate verification of RTK 0.45.0 was published on 24 August 2026. It covers what was fixed and the pytest case that remains worth watching: RTK 0.45: what changed since our analysis.