CellML is an XML format for describing mathematical models of biology: the equations behind a beating heart cell, a nerve impulse, a metabolic pathway. Researchers write a model once, and many different tools can read it, check it and simulate it.

That only works if the tools agree on what a valid model is. So I built a small project to find out where they do: the CellML 2.0 Conformance Corpus.

This article is also published on dev.to.

The problem with testing against yourself

Every CellML implementation has its own test suite, and every suite passes. But each one tests the tool against its own reading of the specification. Two readings can differ, and both suites still pass.

Nobody notices until a model that loads in one tool is rejected by another, and a researcher is left wondering which one is right.

A shared corpus fixes that. Every tool reads the same files, and every file's expected verdict comes from the specification text, never from what any one tool happens to do.

Standing on someone else's shoulders

This idea isn't new, and I didn't invent it. Michael Clerx built cellml-validation, begun at the University of Oxford and continued at the University of Nottingham. It covers CellML 1.0 and 1.1 in depth, names each test file after the rule it checks, and publishes a report for each tool.

CellML 2.0 arrived in 2020, and an open issue on that project asked for 2.0 support with libCellML in the comparison. My corpus is a CellML 2.0 iteration of that work. It follows the same layout and naming, and the README credits it as the origin. Every file in it is written fresh.

It is an independent project, not run by the CellML project or its editors.

Where the first slice came from

The libCellML maintainers had already listed a set of cases in an issue titled "Dealing with real numbers has some holes". Things like:

  • a leading +, as in +2 (which rule 1.3.3.2 of the specification allows)
  • strings with no digits at all, such as -, . and -.
  • an empty value
  • a number too large for a computer's double, such as 1e999

CellML 2.0 lets you write a real number in five places: a unit's multiplier, a unit's exponent, a variable's initial value, a cn element in the math, and a cn inside a degree. Seven cases in five places gives 35 files. That became the first slice.

How the corpus works

One rule per file. Each file is named after the numbered rule it is about, for example 2.6.2.3.unit_exponent_leading_plus.cellml. Its header comments state the rule in plain words and explain the verdict.

Control files. Each place has one ordinary, valid file (_plain, holding the value 2). Every other file for that place differs from it in exactly one value. So if a tool accepts the control and reports an issue on a variant, it is responding to that one value and nothing else.

Three folders. valid, invalid, and unclear. A file goes in unclear when the specification genuinely doesn't settle it. 1e999 is the first: the syntax is allowed and the spec sets no range, but the number has no IEEE 754 double value. Those files are run and shown, but no tool is counted as right or wrong on them.

A self-check. The test suite proves the one-value difference for every file. It also works out each number's verdict again with a small grammar written straight from section 1.3 of the specification, independent of every implementation. I tested the self-check by making mistakes on purpose in a copy (an invalid file moved into valid, a second value changed, a header naming the wrong rule) and confirmed it caught every one.

A runner. Each implementation reads each file in a process of its own, with a time limit. Every pairing gets one of three outcomes: accepted, reported an issue, or stopped unexpectedly, along with the tool's own message. Nothing is skipped.

That last part mattered more than I expected. Some files end a tool's process outright. Because each pairing runs on its own, the run carries on, records what happened, and moves to the next file.

What the first run shows

I compared libCellML 0.7.2 and Myokit 1.39.2. Of the 35 files, 30 have a verdict the specification settles.

libCellML 0.7.2 Myokit 1.39.2
Matches the spec 9 of 30 30 of 30
Differs from the spec 21 of 30 0 of 30

Myokit's outcomes match the specification on all 30 settled files. On 1e999 in a unit multiplier or exponent, it stops with an OverflowError.

libCellML's differences all sit in the areas its own maintainers listed in the issue that started this slice:

  • A leading + is reported as an issue everywhere except inside degree.
  • The strings -, . and -. end the process with an uncaught C++ exception in unit attributes and in cn.
  • In initial values and inside degree, those strings and an empty value are accepted with no message.
  • For 1e999, libCellML gives a clear explanation that the value is out of range of the double type.

That's encouraging, not discouraging. It means the maintainers' list was accurate, and now each item on it has a small, named file that shows it, plus a way to watch it change from one release to the next.

The corpus measures and reports. It doesn't rank tools. A difference is information for the people who maintain them, not a judgment of their work. Tools built for simulation often accept more than the specification allows, and sometimes on purpose.

Why this matters to me

My business is built on a standard I call No Silent Failure: software should never fail a person in silence.

Look at the three ways a tool can respond to a bad value. Reporting an issue tells the researcher exactly what to fix. Stopping unexpectedly is loud, but leaves them guessing. Accepting it with no message is the quietest of all: the model loads, the simulation runs, and nobody learns that a value was never a number.

A conformance corpus makes those differences visible, one rule at a time, so they can be talked about with evidence instead of impressions.

Built to be checked

  • The report is generated in a pinned Docker image, so anyone gets the same bytes.
  • GitLab CI runs the self-check on every push, confirms the committed report is still current, and only then publishes the web page.
  • The test files are under the BSD Zero Clause License, so any project can copy them straight into its own tests. The code is under Apache 2.0.

What's next

The next slice covers unit prefixes. After that, a coverage map showing which rules have files and which don't yet, and more implementations in the comparison: libOpenCOR, cellmlmanip and OpenCOR are the next ones to look at.

If you work with CellML, I'd love your thoughts, and new cases are welcome as issues on the project. You can read the full report any time; it rebuilds on every change.