Test External Source-of-Truth Invariants at the Seam

Context

Most testing advice quietly assumes you own the definition of correct: your code, your tests, your rules, and a failing test means something in your repository changed. Configuration tooling breaks that assumption. Pyxel Config Lab exists to help scientists write YAML that ESA Pyxel, a large and actively-developed framework on its own release cadence, will accept. Pyxel is the source of truth for what a valid config is. The tool only holds a copy of that truth: a bundled JSON schema, a set of tutorial examples, and a mapping of detector types to parameter bounds.

The instant Pyxel ships a new model, renames an argument, or tightens a numeric range, that copy is wrong, and nothing in the tool's own repository changed to signal it.

Problem

This is state drift: two systems that are supposed to agree, silently diverging, with no failing assertion at the moment of divergence. The tests still pass, the build still goes green, and the tool now blesses configs Pyxel will reject or flags valid ones as broken.

Fixtures do not save you here, because a fixture is also a frozen copy of the truth. If Pyxel drifts, the fixture drifts with it, and a green test built on that fixture is lying.

Decision

Each invariant will be tested at the seam where it actually lives, by deliberately reaching for the real external thing at the right layer rather than relying on frozen copies. The drift surface splits into three layers, each with its own defense:

• Schema freshness. The bundled pyxel_schema.json can fall behind Pyxel's published schema. Defense: at runtime, fetch the upstream schema and diff the set of definition names against the bundle, and tell the user the truth when the bundle is stale rather than pretending it never happens.

• Argument bounds. The bounds-extraction logic can drift from the real values inside the schema. Defense: run those tests against the real schema file, not a hand-written fixture.

• Tutorial and doc accuracy. The 70-plus tutorial examples can drift from the installed Pyxel Python API. Defense: introspect the actual Pyxel API and check every example against it.

The suite is deliberately split into automated deterministic logic and manually-verified rendered output, so the automated half never makes brittle assertions about presentation. Per-file coverage gates in CI turn "we should keep this tested" into a pipeline that will not let the team leave it untested.

Alternatives Considered

Test against fixtures only

Rejected. A fixture is a frozen copy of the external truth, so it drifts in lockstep with the code and hides the exact failure this decision targets.

Mock the upstream system

Rejected for the drift-detecting layers. A mock encodes today's understanding of the upstream contract, so it cannot detect the upstream contract changing.

Assume the bundled copy is always current

Rejected. The whole failure mode is a silent divergence with no local change, so an unchecked assumption of freshness is precisely the bug.

Consequences

Positive

• Divergence from the external source of truth becomes a detectable, reportable event.

• Each invariant is verified where it genuinely lives, not where it is convenient.

• Users are told when the bundled copy is stale instead of being misled.

• Coverage gates keep the drift defenses from quietly eroding.

Trade-offs

• Reaching for the real external artifact at test time is more complex than a static fixture and can depend on network or an installed dependency.

• The line between automated and manually-verified checks needs deliberate maintenance.

• Introspection-based checks must track the upstream API surface as it evolves.

Evidence & Related Work

Implementation Evidence

• Pyxel Config Lab:

• Tests:

• Live demo:

• Related article:

Relationship to the Broader Engineering Approach

ADR-001 establishes validation as a boundary. This ADR protects the honesty of that boundary over time: a validator is only as trustworthy as its agreement with the source of truth it enforces, so that agreement must itself be tested at the seam.

Decision Outcome

Correctness that depends on an external system is treated as a testable relationship, not an assumption:

External truth ↔ tested seam ↔ local copy

This ADR should be revisited if a dependency stops publishing a checkable schema or API, in which case an alternative drift signal must be found.