Acceptance Lab

Overview

Acceptance Lab is a small implementation vehicle for a larger question: when an AI system changes, what evidence is sufficient to promote the candidate rather than merely admire its output?

The current package accepts a versioned scenario and a bounded candidate record. It applies deterministic checks, stores the exact records and scores in an append-only SQLite event stream, and emits PASS, CONDITIONAL, or FAIL. Retrieval is one example workload. It is not the product boundary.

System shape

trace or observed run
-> bounded candidate evidence
-> deterministic scenario checks
-> hash-chained run events
-> rebuildable reports and comparisons
-> external promotion policy

The prototype owns the middle of this chain. It does not run the agent, mint credentials, perform production effects, or replace MLflow, OpenTelemetry, Prometheus, Loki, or Vuoro.

Four score dimensions remain separate:

• mechanism: whether the expected control path occurred;
• quality: whether required facts or effects survived;
• authority: whether sources, tools, and effect receipts were admissible;
• economics: whether latency and cost stayed inside the declared envelope.

A run can be complete as evidence and still fail as an outcome.

Current state

The v0.1.0 bundle includes JSON contracts, deterministic scorers, a Python CLI, a hash-chained SQLite event store, rebuildable projections, Markdown and JSON reports, comparison gates, ten unit tests, and a packaged wheel.

Two fixture pairs demonstrate the current claim:

1. naïve retrieval promotes a highly relevant but superseded ADR and fails; authority-aware retrieval filters by lifecycle and valid time and passes;
2. an unsafe tool trajectory reaches the claimed final state but fails because it uses an unscoped shell, omits the required control path, and produces no effect receipt or verification; the bounded verified trajectory passes.

The event chain detects edits to recorded history. It is not an external signature or protection against a host owner rewriting both the database and the verification code.

Open edges

The next proof is one real, bounded agent run. The adapter must obtain tool calls, effects, and receipts from the trace or runner rather than from model self-report. The resulting failure analysis should identify fields that the fixture-designed schema flattened or missed.

A model judge can be added later for genuinely semantic criteria, but it should not override deterministic authority, evidence, or execution failures. Shared Vuoro extraction remains deferred until a second real consumer needs the same run lifecycle and promotion state.