Property-based testing

What a property is

An example-based test pins one input to one expected output: sort([3,1,2]) equals [1,2,3]. A property states something that should hold for every input, and the library generates the inputs for you. fast-check (JS/TS) and Hypothesis (Python) both sample 100 inputs by default (numRuns and max_examples), run your check on each, and hunt for a value that breaks it.

The shift is to stop asking "what should f(2, 3) return" and start asking "what is true about f for all inputs". Your check returns true or false (or throws), and the framework tries to falsify it.

import fc from "fast-check";
// for any array of integers, reversing twice returns the original
fc.assert(fc.property(fc.array(fc.integer()), (xs) => {
  return deepEqual(reverse(reverse(xs)), xs);
}));

Properties you can usually find

You rarely need an exotic property. A handful of shapes cover most code:

• Round trip (there and back): decode(encode(x)) deep-equals x. The natural test for any serializer, parser, or codec.
• Invariant (something that never changes): sorting keeps the length and the multiset of elements; a transfer keeps the total balance.
• Idempotence: applying twice equals applying once, normalize(normalize(x)) equals normalize(x).
• Oracle: run your fast implementation and a slow, obviously-correct one on the same input and compare.

The round trip is the one to reach for first, because so much code encodes then decodes:

Wrong "A property test just re-runs my function and checks it matches what the function returned." That reimplements or snapshots the code, so it only proves the code equals itself and stays green while the logic is wrong. Right assert a relationship that does not depend on the implementation's own output. Round-trip, an invariant, or an independent oracle all qualify; comparing encode against a second copy of encode does not.

Generators (arbitraries)

The inputs come from generators, called arbitraries in fast-check and strategies in Hypothesis. You start from primitives (fc.integer(), fc.string(), st.integers(), st.text()) and compose them: .map to transform a value, .chain when the next generator depends on the previous value, fc.record({...}) or fc.tuple(...) to build objects and tuples.

Reach for the built-in constraints before you reach for .filter. A filter generates a value and throws it away when the predicate is false, so a strict filter wastes most of what it makes and slows the suite:

// wasteful: generates any integer, discards all but the small evens
fc.integer().filter((n) => n > 0 && n % 2 === 0);
// direct: every value already satisfies the shape, nothing is discarded
fc.integer({ min: 1 }).map((n) => n * 2);

Reading a failure: shrinking and the seed

When a property fails, the framework does not hand you the raw random input that first broke. It shrinks: it searches for the smallest input that still fails, so a failing array of forty messy elements collapses to two or three. A minimal counterexample is the difference between "somewhere in this mess" and a bug you can read at a glance.

fast-check prints the shrunk counterexample plus a seed and a path:

Property failed after 1 tests
{ seed: 1234567890, path: "5:2:1", endOnFailure: true }
Counterexample: [[2,1]]
Shrunk 42 time(s)

That seed and path are how you replay the exact failure: pass them back to fc.assert and you rerun the same case every time. Hypothesis does the same automatically, saving failures to a local example database (.hypothesis/examples) and replaying them on the next run. A red property is a reproducible bug, so fix it or capture it as a regression; a retry that turns it green just buries the counterexample the tool worked to find.

What a green run actually proves

A green property run raises your confidence. It does not prove the property. fast-check samples 100 inputs by default, Hypothesis 100 examples; both draw a range of sizes so small and large inputs both get exercised. Raising numRuns or max_examples widens the sample and buys more confidence, and it never covers an infinite input space. Treat the number as a knob on how hard you are looking. It does not certify the property everywhere.

Wrong "It passed 100000 runs, so the property is proven for all inputs." That is 100000 samples of an enormous space. The value of PBT is that a randomized search finds classes of bugs an example suite never enumerates. It does not certify correctness.

Shrinking and why the minimal counterexample matters

The first input that fails is usually noisy. Shrinking is the search back down toward the simplest input that still fails, and the quality of that search decides whether the failure is legible.

There are two designs. Type-based shrinking (classic Haskell QuickCheck) shrinks a value by its type alone, independent of how it was generated. Integrated shrinking (Hypothesis, and fast-check's model) ties the shrinker to the generator, so a value shrinks along the same path it was built. The practical payoff is that generator invariants survive shrinking: if your arbitrary only ever produces even numbers, integrated shrinking keeps the shrunk value even, whereas a type-based shrinker can hand you an odd number and a failure that has nothing to do with the original bug. It also composes, so anything you can generate you can shrink, with no separate shrinker to write per type.

Metamorphic relations and the oracle problem

The hard case is code where you cannot state the exact right answer for a given input: an image resizer, a search ranker, a floating-point routine. This is the oracle problem, and metamorphic testing is the usual way through it. Instead of checking one output, you assert a relation between the outputs of related inputs.

For a sort, one metamorphic relation is that sorting a permutation of the input yields the same result. For a shortest-path function, adding a constant to every edge weight should not change which path is chosen. You never name the expected output; you constrain how outputs must relate. Round-trip and idempotence are special cases of the same idea, which is why they are the easiest properties to find.

Stateful and model-based testing

Single-value properties do not reach bugs that only appear after a sequence of operations: an LRU cache that corrupts after a specific interleaving of put and get, a bucket that goes negative only after a refill lands between two takes. Model-based testing generates the sequence for you.

You define commands. In fast-check each command implements check(model) (is this action legal in the current state), run(model, real) (execute it against both the model and the real system and assert they agree), and toString() for readable output. fc.commands([...]) generates a sequence, and fc.modelRun (or asyncModelRun) drives it. Hypothesis models the same idea as a RuleBasedStateMachine: methods decorated with @rule are the operations, @initialize runs once up front, @invariant is checked after every step, @precondition gates a rule on state, and a Bundle threads values produced by one rule into the arguments of another.

The model is the load-bearing design decision. It should be a simplified reference, a plain map plus an ordered key list standing in for a real LRU, not a copy of the implementation. A line-for-line copy carries the same bugs and can never disagree, so it catches nothing.

Wrong "Model-based testing needs an accurate reimplementation of the system to compare against." The model is a simpler reference than the system; its job is to be obviously correct, and it need not be fast or complete. When the sequence fails, the framework shrinks it, tracking which commands actually ran so it drops the ones that did not contribute and hands you the shortest failing program.

Concurrency is a separate axis. A normal async run only sees one interleaving. fast-check's scheduler wraps promises so it controls when each one resolves (schedule, scheduleFunction, waitNext), and reorders them to explore interleavings a sequential run would never hit; scheduledModelRun combines this with the model-based approach to hunt for races.

Reproducibility and flakiness

Randomized does not mean unrepeatable. fast-check seeds its generator from Date.now() by default and prints the seed and path on failure; feeding both back into fc.assert replays the exact case, shrinking included. Hypothesis persists failures to its example database and replays them first on the next run, and @seed or @reproduce_failure pins a specific case in the source. So a red property is a captured, minimal, replayable bug.

The real flakiness risk is a property that is not actually deterministic: it reads the wall clock, depends on map iteration order, or leaks state between runs. Those fail intermittently regardless of the seed, and the fix is the same as for any flaky test, control the nondeterminism. Watch Hypothesis's deadline too (200ms per case by default, disabled on CI): a slow example can fail on timing rather than logic. Heavy filtering is the other trap, since discarding most generated values trips the filter_too_much health check or raises Unsatisfiable; build constrained values directly instead.

Where it pays and where it wastes

PBT earns its keep where a clean property exists and the input space is large: parsers and serializers (round-trip), encoders and compressors, data structures and their invariants, anything with a slow reference implementation to use as an oracle, and stateful or concurrent code through model-based testing. It wastes time where the only "property" you can write restates the implementation, or where writing a correct oracle is as hard as writing the code under test. On thin CRUD and glue, a couple of example tests are cheaper and clearer, and a forced property there tends to collapse back into the tautology of asserting the code against a copy of itself.