Axis 09 · Vacancy: Did the test even reach the rule?

A smoke alarm test where nobody lit a match. The alarm did not go off, and the test passed, and you learned nothing. A test is vacuous when the data never reached the rule it was supposed to check.

Every rule held and the suite was green. But the generator only made pending orders, so the rule about confirmed orders was never checked. A green run, and no evidence at all.

You declare nothing. This axis is automatic, and it runs on every other axis.

It checks that every rule can fire (it finds or is given a value that breaks the rule, and only that rule). It checks that a rule that applies "when X" saw an X. And each other axis checks its own version of the same thing: the operation changed something, the value moved, an order had two items, a run passed the deadline.

Vacancy: the test passed without proving anything.

Rule confirmed-orders-have-lines Not exercised because its guard was never true for the 200 generated values, so the rule held without being exercised. Next give the auto-test starts where the guard is true (for example a generator that sometimes draws the case the guard names), or mix such values into it.

Law Every rule can fire, and every rule's guard is reached by the data the tests use. Consequence A rule that can never fire, or a test that never reached it, reports green while checking nothing.

How to read this:

• Rule names the rule that was never exercised, and Not exercised because says why.
• Next says what to change, usually the generator, so the data reaches the rule.

This is the one that makes the other eight trustworthy. A property test that passes has not told you the property holds; it has told you it did not find a counter-example. If it never looked, that means nothing.

In docs-builder, this axis found that the changelog entry generator never produced a marker entry, so none of the marker rules had ever run.

Previous: 08 Confluence · All nine axes