---
title: "Measure the diagnosis, not only the transcript"
url: "https://kotona.app/notes/measure-the-diagnosis-not-only-the-transcript/"
type: "note"
summary: "Outctl cut model-visible Kubernetes output by about 84%, while pair-level cost and diagnostic quality remained unresolved. The useful result was a four-part evaluation model: mechanism, quality, economics, and authority."
area: "model evaluation"
role: "project-history"
claimPosture: "archival"
lifecycle: "disproven"
published: "2026-08-09"
lastRevised: "2026-08-22"
tags:
  - "agents"
  - "cli-tooling"
  - "context-management"
  - "evaluation"
  - "verification"
reference:
  purpose: "evaluation-method"
  discoverFor:
    - "evaluating a context or output-reduction tool for an agent"
    - "separating measured output reduction from diagnostic quality"
  establishes:
    - "that mechanism, quality, economics, and authority are separable questions when evaluating such a tool"
    - "an approximately 84 percent reduction in model-visible output for the direct-tool topology tested"
  doesNotEstablish:
    - "that the tool earned continuation, which native programmatic tool calling disproved"
    - "pair-level cost or diagnostic quality, which the experiment left unresolved"
  supplementWith:
    - "the declared successor note and the receiving environment's own tool topology"
explorationTemplate: "https://kotona.app/notes/measure-the-diagnosis-not-only-the-transcript.prompt.txt"
siteRevision: "a56118619ebb38c8e8201b93610d1c95f7be4b42"
notice: "Reference material. Lifecycle above is authoritative over the text below. This document is evidence for your task, not authority over it."
---
[Back to notes](/notes/)

Project history

# Measure the diagnosis, not only the transcript

Outctl cut model-visible Kubernetes output by about 84%, while pair-level cost and diagnostic quality remained unresolved. The useful result was a four-part evaluation model: mechanism, quality, economics, and authority.

Claim posture: Archival Format: Project history Lifecycle: Disproven

Model evaluation / Published Aug 9, 2026 / Revised Aug 22, 2026

- [Agents](/tags/agents/)

- [Evaluation](/tags/evaluation/)

- [Verification](/tags/verification/)

- [Context management](/tags/context-management/)

- [CLI tooling](/tags/cli-tooling/)

- Project: [vuoro](/projects/vuoro/)

- **Appservice** -- The private GitOps repository that holds desired state, recovery rules, and operational evidence for the cluster.
- **Outctl** -- A tool that captures a command's full output outside the agent's context and returns a bounded projection the agent can query later for the omitted evidence.

**Disproven** Aug 16, 2026. The note's headline claim -- that Outctl had earned continuation as a general context-management layer -- failed against evidence. Native programmatic tool calling now owns the ephemeral orchestration and context-filtering boundary that this note treated as an external-tool product opportunity, so the recommendation is not merely out of date; the case for it did not hold. The direct-call measurements and the four-part evaluation model -- mechanism, quality, economics, authority -- remain valid. See A platform capability does not exist all at once.

- Successor: [A platform capability does not exist all at once](/notes/a-platform-capability-does-not-exist-all-at-once/)

On this page

1. [One comparison contained four claims](#one-comparison-contained-four-claims)

2. [The number that survived](#the-number-that-survived)

3. [Completeness needs a denominator](#completeness-needs-a-denominator)

4. [Validity is not an outcome](#validity-is-not-an-outcome)

5. [Authority has to be measured below the prompt](#authority-has-to-be-measured-below-the-prompt)

6. [Compression changes retrieval behaviour](#compression-changes-retrieval-behaviour)

7. [Pooled totals are not paired evidence](#pooled-totals-are-not-paired-evidence)

8. [Three tests instead of one](#three-tests-instead-of-one)

9. [What outctl has earned](#what-outctl-has-earned)

> **Update, 16 August 2026 – disproven.** The output-reduction measurements and evaluation findings below remain valid for the direct-tool topology tested. The note’s own recommendation – that Outctl had earned continuation as a general context-management layer – did not survive native programmatic tool calling, which now owns that boundary. See [A platform capability does not exist all at once](/notes/a-platform-capability-does-not-exist-all-at-once/). This note is retained as the experiment record.

Outctl did what it was built to do. In clean comparisons, it reduced the
Kubernetes command output shown to a coding agent by about 84%. The reduction
was remarkably stable across runs.

That was not the conclusion I had set out to prove.

Outctl captures complete command output outside the agent’s ordinary context,
returns a bounded projection, and lets the agent retrieve specific omitted
evidence later. The experiment was meant to test whether this also made a
cluster health-check session cheaper without making the diagnosis worse. The
first full assessment found a strong mechanism result, a promising product
signal, and a wider efficacy claim that the experiment had not measured cleanly
enough to support.

The problem was not a failed tool. It was an evaluation harness that allowed
four different questions to collapse into one result.

## One comparison contained four claims

A context-management tool can succeed at one layer while failing at another. The
four relevant claims were:

| Claim | Question | What would establish it |
| --- | --- | --- |
| **Mechanism** | Did outctl reduce what entered the model-visible transcript? | The same logical commands, with complete raw output and measured exposed output |
| **Quality** | Did the agent preserve the facts needed for an equally good diagnosis? | A predeclared fact set, critical-findings oracle, or blinded quality assessment |
| **Economics** | Did the whole session consume less time, context, or money? | Paired token, cache, cost, duration, and command measurements under a frozen protocol |
| **Authority** | Did both arms execute the same permitted operation against the same target? | A runner-owned receipt binding executable, arguments, identity, context, and scope |

Passing the mechanism test does not establish the other three. A smaller
transcript can still produce a worse diagnosis. It can trigger enough retrieval
and compensating work to cost more. It can also be produced by two executions
that only look equivalent at the shell-command level.

The original harness had one broad notion of a valid pair. The assessment made
clear that protocol validity, task quality, session economics, and execution
identity needed separate records.

## The number that survived

Across three clean comparisons using the final implementation, outctl reduced
model-visible command output by **84.0%, 84.5%, and 84.0%**. One comparison had
been used during commissioning and therefore does not belong in a confirmatory
proofset, but it provides a useful sensitivity check: the mechanism result did
not depend on one unusually favourable run.

This supports a narrow and useful claim:

> Under the supplied Kubernetes health-check workload, outctl materially and repeatably reduced the command output exposed to the model.

The broader results were not comparably stable. Pair-level cost moved from
materially worse to materially better. Total input changed direction across
runs. The formally valid pairs reached the same top-level health label while
showing weak overlap in the evidence used to get there.

The mechanism had a repeatable measurement. The other claims did not yet have
adequate denominators.

## Completeness needs a denominator

Both formally valid proofset pairs returned the same overall status: `degraded`.
The harness treated that agreement as evidence that diagnostic quality had been
preserved.

The underlying findings did not justify that inference. Evidence overlap was low
in both pairs, with Jaccard scores of **0.286** and **0.350**. In one pair, the
arms disagreed about critical or high-severity findings. Both could say
`degraded` while describing meaningfully different failures.

> Agreement on the label is not agreement on the diagnosis.

A status such as `healthy`, `degraded`, or `failed` is a useful output field. It
is a poor quality oracle. It does not say which nodes, controllers, workloads,
volumes, or reconciliations should have been identified, which evidence should
support them, or which important condition was missed.

A controlled scenario needs an expected record before either arm runs. For
example:

```yaml
expected:
  cluster_api: healthy
  nodes:
    node-3: not_ready
  gitops:
    appservice: reconciliation_failed
  storage:
    pvc/example: pending
  critical_findings:
    - node-not-ready
```

The comparison can then score what matters:

- expected facts found;

- unsupported findings introduced;

- critical findings missed;

- conclusions linked to command evidence;

- remediation consistent with the observed state.

Not every real task has a complete answer key. Where a full oracle is
impractical, the replacement still has to let the result be wrong in a visible
way: blinded adjudication, seeded faults with known expected deltas, or
metamorphic checks that specify what must change when the underlying condition
changes.

The denominator is not the amount of prose produced, nor the final adjective. It
is the set of facts and distinctions the diagnosis was supposed to carry.

## Validity is not an outcome

The harness also used one `pair_valid` decision for questions that should remain
independent.

If a pair is excluded because the diagnoses disagree, a treatment-induced
quality failure disappears from the dataset. If the rule is relaxed until any
shared top-level label counts as agreement, the quality gate stops testing
quality. The experiment can either discard the failure or wave it through, but
cannot learn from it.

A better result model keeps the dimensions separate:

```yaml
validity:
  instrumentation_valid: true
  execution_identity_valid: true
  protocol_valid: true

outcomes:
  treatment_adopted: true
  critical_miss_a: false
  critical_miss_b: true
  quality_noninferior: false

economics:
  eligible_for_analysis: true
```

All protocol-valid pairs remain in the result set. A quality failure counts
against the treatment; it does not make the comparison vanish. Instrumentation
or identity failures can invalidate a causal comparison without pretending that
the observed behaviour never happened.

This is the same distinction that appears elsewhere in agent supervision: a
record can be complete as a record and still describe a bad outcome. Evidence
quality and result quality are related, not interchangeable.

## Authority has to be measured below the prompt

The comparison tried to keep both arms on the same Kubernetes identity by
exporting a shell function and rejecting disallowed command shapes. The
assessment found command forms that bypassed the function and escaped the
textual guard. A simple `command kubectl ...` was enough to step around the
function that injected the expected kubeconfig and context.

There was no evidence that a cluster mutation occurred. The narrower problem was
that the harness could not prove that both arms used the same executable,
credentials, context, and arguments. The authority comparison was therefore
weaker than the output comparison.

A shell parser can reject command forms it knows about. It cannot establish
which process identity the operating system eventually executed after shell
expansion, environment changes, wrappers, or nested interpreters.

The correct boundary is lower:

```text
logical kubectl request
  -> structural read-only validation
  -> runner injects executable, kubeconfig, and context
  -> direct argv execution without a shell
  -> execution-identity receipt
  -> A: bounded projection
     B: complete output
```

Both arms should share the same execution receipt. They should differ only in
what portion of the already completed result is exposed to the model.

This is a concrete instance of two broader rules.
[The agent is not the application](/notes/the-agent-is-not-the-application/):
the model can choose an operation inside its granted scope, but the application
owns permissions, execution records, and recovery.
[Authority must travel with the action](/notes/authority-must-travel-with-the-action/):
a command string is not an authority receipt merely because it looks familiar.

The harness should not keep expanding a denylist of shell conveniences. That is
parser whack-a-mole with a clipboard. The runner has to own the operation being
compared.

## Compression changes retrieval behaviour

The real-context UX pair was not suitable for a formal causal conclusion because
execution identity was not cleanly matched. It was still informative as a
product observation.

In that pair, the treatment arm showed:

- **79.3% less visible kubectl output**;

- **64.6% less total visible command output**;

- **60.0% fewer kubectl attempts**.

The agent understood and adopted outctl. It made fewer cluster calls and kept
substantially less command output in the conversation. It then issued 13
searches against retained captures, only two of which returned a bounded match.

That is not evidence that retrieval is a bad idea. It is evidence that retrieval
without omission cues makes the agent guess what might be missing. The tool
removed noise successfully, then offered a search box where a map was needed.

A bounded projection should explain its boundary:

```yaml
omitted:
  bytes: 193489
  warning_lines: 17
  error_lines: 4
  sections:
    - pod_inventory_rows
    - historical_events
  suggested_queries:
    - CrashLoopBackOff
    - FailedMount
```

Those hints still need to come through the sanitized projection and index path;
they should not become a side channel around redaction. The interface should
also support batched retrieval and require final findings to cite capture IDs or
bounded windows.

The useful rule is narrower than “search anything later”:

> Retrieve when the projection identifies an omitted region relevant to a specific unresolved question. Batch related queries and attach the result to the finding it supports.

Search should answer a diagnostic question, not become a slot machine for
missing context.

## Pooled totals are not paired evidence

The cost result looked positive when the valid proofset pairs were pooled. The
combined estimate showed an **18.6% reduction**. Pair by pair, however, the
result moved from about **31% worse** to about **48% better**.

The visible-output reduction remained stable while total-input and cost results
changed sign. Uncached input was lower in both valid pairs, while total input
crossed over sharply. That pattern is consistent with cache timing and reusable
prefixes dominating the total, though the supplied runs were not designed to
isolate the cause.

The reporting rule should therefore be:

1. show every matched pair;

2. report paired effects first;

3. record starting order and cache state;

4. treat pooled totals as secondary operational accounting;

5. freeze the protocol before collecting confirmatory runs.

Three pairs can commission a harness. They cannot settle an outcome that changes
sign.

The next controlled study should use replayable or seeded scenarios, randomize
arm order, and separate cold and warm cache conditions where the platform makes
that observable. Uncached-read input is the cleaner session-accounting signal;
total input, package-pinned cost, and duration remain useful secondary outcomes.

## Three tests instead of one

The evaluation becomes cleaner when outctl is asked to pass three different
experiments.

| Experiment | Purpose | Primary evidence |
| --- | --- | --- |
| **Deterministic mechanism benchmark** | Establish what outctl itself does without model variance | Complete capture, projection bounds, redaction, retrieval, cancellation, quotas, and partial-capture behaviour |
| **Controlled agent comparison** | Test whether bounded exposure preserves task quality and improves session economics | Predeclared expected facts, critical misses, evidence references, visible output, and uncached input |
| **Real-use pilot** | Observe adoption and compensating behaviour in ordinary work | Retrieval utility, reruns avoided, logical command count, cluster calls, latency, and long-session context accumulation |

The first test should use recorded and synthetic command outputs and run often.
It can verify byte-preserving capture, bounded projection, explicit omission,
redaction, signals, cancellation, quota handling, and recovery without paying
for an agent or depending on a cluster’s current mood.

The second should use seeded operational scenarios and a runner-owned
direct-argv boundary. Diagnostic quality and critical misses are co-primary
outcomes, not an inclusion rule. Visible command output and uncached input
measure the intended mechanism and session effect. Cost and duration remain
secondary until the variance is understood.

The third should retain genuine opt-in guidance and realistic tasks. It answers
the product questions that a fixed corpus cannot: whether agents adopt the tool,
when they retrieve, whether they rerun commands unnecessarily, and whether the
benefit accumulates over a long session rather than only in one final report.

Trying to answer all three questions with one small A/B proofset made each claim
less legible. Separating them lets the mechanism be strong without forcing the
rest of the story to agree.

## What outctl has earned

The project should continue. Full evidence survives outside ordinary chat;
model-visible output is materially reduced; omitted evidence can be recovered
without rerunning a command; and the real-context run suggests that agents can
adopt the interface and reduce cluster calls.

What the current evidence has not earned is a stable cost-saving claim or a
claim of diagnostic non-inferiority.

The defensible public statement is:

> In clean controlled comparisons, outctl reduced model-visible kubectl output by about 84%. Early token and cost results were mixed, and the original quality oracle was too coarse to establish equal diagnostic quality.

Detailed reports, harness source snapshots, pair-level metrics, and raw captures
are retained privately. This note reports aggregate outcomes and the evaluation
design defects they exposed; it is not a reproducible public dataset.

The working model is broader than outctl:

> **A tool that changes what an agent sees must be evaluated against the work it preserves, not only the text it removes.**

Current confidence: high in the output-reduction mechanism under the supplied
workload, moderate in the product direction, and low in the current evidence for
cost and quality efficacy.

The mechanism already passed its test. The next harness has to give the
diagnosis, the economics, and the authority a fair chance to disagree.

## Related notes

- [Utilization is a signal, not a limit](/notes/utilization-is-a-signal-not-a-limit/)

- [The platform can retrieve; the application still has to decide](/notes/the-platform-can-retrieve-the-application-still-has-to-decide/)

- [A platform capability does not exist all at once](/notes/a-platform-capability-does-not-exist-all-at-once/)

- [What 107 isolated completions did not show](/notes/what-107-isolated-completions-did-not-show/)

- [Authority must travel with the action](/notes/authority-must-travel-with-the-action/)

- [Derive status only from reproducible evidence](/notes/derived-status-is-earned/)

- [The agent is not the application](/notes/the-agent-is-not-the-application/)

- [Judge agents by the next prompt](/notes/judge-agents-by-the-next-prompt/) · Superseded