Regression-Testing a Payroll Engine Offline

The Problem: Every Change Could Change Someone's Pay

The salary engine I work on is a library consumed by a calculation service. It turns a period of recorded driver activity into pay, driven by dozens of configurable rules. Change one rule — or the pipeline they run in — and you could silently alter the pay of thousands of employees.

Unit tests cover each rule in isolation, but the question that kept us honest was different: does this version of the engine still produce the same numbers as the last one, on real data? Answering it meant reproducing a full calculation — which required a live environment with running APIs for holidays, places and master data. Too slow to do per change, so in practice it wasn't done per change.

Capture the Calculation as a File

The harness I built captures one real calculation as a single self-contained snapshot: the employee, the contract, the activities, the full rule configuration, and — crucially — the results the engine actually produced.

CapturedCalculation
├── EngineVersion        // which engine produced this
├── Employee + Contract  // who was calculated
├── Activities           // the raw input data
├── RuleConfiguration    // every parameter value used
└── Results              // what the engine said, per field

Because everything the engine needs is inside the file, the replay needs no environment at all — no databases, no upstream services, no network.

Replay Through the Real Engine, Not a Copy of It

The tempting shortcut is to re-implement the calculation inside the test tool. That produces a second source of truth that drifts from the first. Instead, the replay drives the engine's own execution path — the same orchestrator the production service calls.

The engine wasn't written to run detached from its infrastructure, so most of the work was finding the seams: where to inject a repository backed by the snapshot instead of a database, and where to intercept the outbound lookups (public holidays, places, previous balances) the engine makes while building its context. Each interception point became a stub fed from the captured file.

After the replay, the tool diffs the recomputed results against the stored ones, field by field. Any drift — a premium hours figure, an allowance, a rounding change — shows up as a named difference, not a vague failure.

Version Pinning: A Diff Means Two Different Things

Each snapshot records the engine version that produced it, and the tool warns when that differs from the version it is running. This matters more than it sounds:

• A diff within one version means non-determinism — a genuine bug.
• A diff between versions means the change altered pay — which might be intended, but must be a conscious decision, reviewed as such.

A related footgun surfaced while building it: accepting arbitrary start and end dates lets a capture cover a partial payroll period, which produces meaningless comparisons. The capture flow is moving to a single date resolved against the employee's configured period boundaries, so a snapshot always covers a complete period.