Context
A specification defines what a valid file is. An implementation is one reading of it. A test suite that takes its expected answers from the implementation it tests can only show that the implementation agrees with itself.
CellML is the XML format the Physiome community uses to share mathematical models of biology, from ion channels to whole heart cells. CellML 2.0 is defined by a normative specification (Release 2.0.1, January 2023), and libCellML is its reference library. In October 2026, libCellML issue 1505 listed places where real numbers are not yet handled as the specification describes: a leading + in a unit exponent, which rule 1.3.3.2 allows; strings with no digits, such as -.; numbers out of range, such as 1e999; and empty cn elements. It asks for every place a real number can be written to be covered: unit attributes, initial values, cn and degree.
The tests that settle questions like these already have a history. Michael Clerx's cellml-validation, begun at the University of Oxford and continued at the University of Nottingham, holds 928 CellML 1.0 files and 938 CellML 1.1 files, each one following or departing from a single rule of the specification, with generated reports for several tools. For CellML 2.0 it holds one file, and its issue 71, open since 2021, asks for libCellML results. No other CellML 2.0 conformance corpus was found.
This record captures the decision to carry that approach forward to CellML 2.0, and to settle every expected answer from the specification text before any tool reads a file.
Problem
Without a shared corpus, each implementation tests itself against its own reading of the specification. Where two readings differ, both test suites pass, and the difference appears only when a model written for one tool is opened in another: sometimes as an error message, sometimes as a value that quietly means something other than what was written.
The specification itself can point two ways. Normative rule 1.3.3.2 allows a leading + in a real number, while the informative specification's examples list <cn cellml:units="dimensionless">+4.0</cn> as not valid. A test suite that took its answers from either text without saying which would pass that ambiguity on as a certainty.
A validator is also a boundary in its own right. Every model it reads relies on it, so a gap in it lets an invalid model through without a word, or turns a valid one away. It cannot be measured against itself.
We need a clear boundary between what the specification says and what an implementation does.
Decision
The CellML 2.0 conformance corpus will take every expected verdict from the normative specification text, give each file exactly one rule, and report where implementations agree and where they differ, without ranking them.
Each file names one numbered rule of Release 2.0.1 in its file name and in its header comment, using the layout and naming of cellml-validation: models_2_0/valid, models_2_0/invalid, and names such as 2.6.2.3.unit_exponent_leading_plus.cellml. Each place where a value can be written has one valid control file, and every other file for that place differs from its control in exactly one value. A self-check in the corpus proves that one-value difference, and works out each number case's verdict again with a small grammar written straight from section 1.3 of the specification, independent of every implementation.
Files the specification does not settle go in a third folder, models_2_0/unclear. They are run and shown, but no implementation is counted as matching or differing from the specification on them, and each file says why. 1e999 is the first: its syntax is allowed and the specification sets no range, but it has no value as a 64-bit floating point number, and the specification does not say what a tool should do with it. Where the normative and informative texts differ, the normative text decides, and the file quotes the informative note beside it.
A runner gives every file to each implementation in a process of its own, with a time limit, starting with libCellML (its parser and validator) and Myokit (its CellML 2.0 reader). Each pairing has one outcome: accepted, reported an issue, or stopped unexpectedly, together with the tool's own message. The report is a disagreement matrix: one row per file, a column for what the specification expects, one column per implementation, and a summary of the rows where every implementation gives the same outcome. It records exact versions and nothing that depends on the time, so two runs produce identical reports.
Spec rule → one file per case → self-check
↓
libCellML Myokit (more later)
↓
disagreement matrix: where tools agree, where they differThe first slice is real numbers, the cases from libCellML issue 1505: five places (unit multiplier, unit exponent, initial value, cn, and cn inside degree) and seven values in each (2, +2, -., -, ., an empty value and 1e999), 35 files in all.
This establishes a boundary between a specification and its implementations: what a file is expected to do is settled from the text before any tool reads it.
Alternatives Considered
Take Expected Verdicts From libCellML
Rejected.
libCellML is the reference library, so its answers are the natural default. Taking them as expected verdicts would record today's behavior as the rule, including the four cases issue 1505 asks to change, and every other tool would then be measured against libCellML rather than against the specification.
Add the Tests Only to One Implementation's Own Suite
Rejected as the only home.
Tests inside one implementation's suite serve that implementation well and no other. A shared corpus lets every tool be measured by the same files. The corpus files carry the BSD Zero Clause License, as cellml-validation's test files do, so any implementation can copy them into its own suite as well.
Extend cellml-validation Directly
Deferred.
cellml-validation is the corpus this one grows from, and its test files have not changed since October 2022. A CellML 2.0 corpus needs a home that can grow week by week. The layout, naming and license match on purpose, so files can move into cellml-validation later if its maintainer wishes.
Put Several Cases in One File
Rejected.
cellml-validation's valid real number file holds seventeen values in one model, which is efficient while a tool accepts all of them. Once a tool reports an issue, the file no longer says which value it responded to. One case per file keeps each outcome tied to one rule.
Decide the Unclear Cases Ourselves
Rejected.
Choosing an answer for 1e999 would turn one reading into a rule, and the expected verdict would change with whoever wrote it. The corpus shows what each tool does with the case and leaves the answer to the specification's custodians.
Give Each Implementation a Single Score
Rejected.
A percentage of files says little about a tool: the corpus covers some rules more densely than others, and a tool built for simulation may accept more than the specification allows, on purpose. cellml-validation's own reports make the same point about their percentages. The matrix shows each file and each outcome instead, with counts but no ranking.
Check Each Tool's Message Against an Expected Message
Deferred.
cellml-validation keeps a list of expected messages for each tool, which shows whether a tool reported an issue for the right reason. Those lists need care with every release of every tool. The control files give much of that assurance now: a tool that accepts the control and reports an issue on a variant is responding to the one value that changed. Messages are shown in the report from the start.
Consequences
Positive
Every expected verdict can be traced to a numbered rule of the specification, never to what a tool happens to do.
An outcome on a variant file points at the one value that changed from its control.
Ambiguities in the specification are shown as ambiguities, with their reasons, rather than settled without a word.
Every implementation reads the same files, so a difference between two tools is visible before a model depends on it.
A crash or a hang is recorded as an outcome with its message, and the run carries on.
Reports can be produced again byte for byte from the pinned versions.
The corpus files can be copied into any implementation's own tests, with no paperwork.
cellml-validation's approach continues into CellML 2.0, with its origin credited in the project.
Trade-offs
Coverage starts small. The first slice covers real numbers only, and a count of files says little about how much of the specification is covered.
Where the normative and informative texts differ, the corpus follows the normative text, so a reader of the informative examples will see those files counted the other way.
Messages are shown, not checked. A tool could report an issue for a different reason and still be counted as matching the specification. The control files narrow that gap without closing it.
Each adapter decides what "accepted" means for its tool, which is a reading too. The adapters are kept small and documented so that reading stays visible.
A tool that accepts more than the specification allows, on purpose, will show many differences. The report says that leniency can be a feature, but the matrix alone does not.
Pinned versions mean a report describes those versions only. Moving to newer releases is a deliberate step.
One case per file means many small files, so the layout and names have to stay easy to navigate as the corpus grows.
Evidence & Related Work
This decision is implemented in the CellML 2.0 Conformance Corpus, and the work below provides concrete evidence of applying the principle in practice. The gap it fills was confirmed in cellml-validation and in libCellML's issue tracker first, on 28 October 2026.
Implementation Evidence
The CellML 2.0 Conformance Corpus
The project: test files, self-check, runner and report, with a README that credits cellml-validation and shows how the parts fit together.
The Test Files
One case per file, filed under valid, invalid or unclear, each naming its rule and the reason for its verdict in a header comment.
The Self-Check
Proves that each file differs from its place's control in exactly one value, and works out every number case's verdict again from a grammar written straight from section 1.3 of the specification.
The Runner
Gives each file to each implementation in a process of its own, with a time limit, and records accepted, reported an issue or stopped unexpectedly, with the tool's message.
The Disagreement Matrix
The generated report: what the specification expects for each file, what each implementation did, and where they agree and differ.
Upstream Evidence
libCellML Issue 1505: Dealing With Real Numbers Has Some Holes
Open. Lists the leading + in a unit exponent, degenerate strings such as -., out of range numbers such as 1e999 and empty cn elements, across unit attributes, initial values, cn and degree.
CellML 2.0 Normative Specification, Release 2.0.1
Section 1.3 defines integer, basic real number and real number strings. Rules 2.6.2.2, 2.6.2.3, 2.8.2.2 and 2.12.5 say where they apply.
The Informative Specification on cn
Its examples list +4.0 inside a cn element as not valid, "except in e-notation exponents", where the normative rule 1.3.3.2 allows the sign.
Related Work
cellml-validation by Michael Clerx
The corpus this one continues: 928 CellML 1.0 files and 938 CellML 1.1 files, each tied to one rule, with reports for Myokit, cellmlmanip, OpenCOR and three XML schema methods. Its layout, its naming by rule number and its practice of a generated report per tool come from here.
cellml-validation Issue 71: Add libCellML Validation for CellML 2
Open since March 2021. The first slice's report includes libCellML from the start.
Related Implementation
Social Resource Floor
Two independent runners, in Python and in Node, give the same answer on all 427 verdicts over 144 fixtures, with every expected verdict written down in the contracts before either runner reads it. The corpus applies the same method to tools written by other people (ADR-009).
Relationship to the Broader Engineering Approach
This ADR is part of a broader engineering principle:
Invalid states should be made difficult or impossible to reach the execution layer.
A validator is where that principle is put into practice for a file format: it is the boundary every model passes before it is simulated. This record is about the boundary's own reliability. ADR-005 treats error messages as first-class outputs, which is why the report shows each tool's own message beside its outcome.
Where ADR-001 places the boundary between configuration and execution, ADR-010 places it between write paths and stored state, ADR-012 places it between generation and presentation, and ADR-013 places it between a check and its export, this decision places it between a specification and the tools that implement it. The move is the same in each case: what cannot be assumed is settled at a named boundary, and everything past that boundary may rely on it.
The distinctive difficulty here is that the thing being measured is itself a checker. A validator cannot be measured against its own answers, and each tool's tests are written from its own reading. The expectation has to be settled somewhere no implementation reaches: in the specification text, one rule at a time, with the places the text is silent shown as silent.
Configuration → guarded before execution
Writes → guarded before storage
Generation → guarded before presentation
Verdicts → chained before they are taken away
Validators → measured against the spec before models rely on themDecision Outcome
The corpus is considered an intentional architectural boundary.
Future work on the corpus should preserve this principle where practical:
Spec rule → one file per case → self-check → every implementation → disagreement matrix
rather than:
Implementation → its own tests → its own answers
This ADR should be revisited if the CellML specification custodians publish a new release or settle one of the unclear cases, in which case files move between folders with the release noted. It should also be revisited if what "accepted" means for a new implementation proves hard to define, or if checking each tool's messages becomes worth its upkeep.