One-Axis TVC
A hard real-time control and telemetry stack: a 500 Hz control loop on Linux, measured as a latency distribution.
| p99.9 wakeup jitter | 7.5 µs at 500 Hz |
|---|---|
| Run-to-run spread | 0.1 µs across 3 × 300,000 cycles |
| Naive baseline drift | 1.9 s in 10 minutes |
| Mitigation levels measured in isolation | 7 |
The naive version of this loop drifts seconds off schedule in ten minutes and reports itself healthy the whole time, because measuring each wakeup against the previous one hides the drift. Seven levels, applied one at a time and each measured in isolation, take p99.9 wakeup jitter from that failure to 7.5 microseconds at 500 Hz on a stock Ubuntu generic kernel, with per-core isolation and every C-state disabled.
The most useful result was a wrong number. An earlier campaign measured 88.4 microseconds on the same code, and the only difference was that it left C-states enabled: the idle driver advertises 18 and 350 microsecond exit latencies, so the old headline was measuring C-state exit. One power-management knob carried a 12x difference, and the provenance system could not see it. Governor, energy-performance preference, AC state and package temperature are recorded per run, and every one of those fields reads identically across both campaigns.
Every number traces to a committed campaign summary, and a regression gate diffs new runs against those baselines. Per-cycle telemetry is a single-producer single-consumer ring feeding a binary frame codec, with a ThreadSanitizer stress lane that forces the drop path, and it costs nothing measurable: pooled p99.9 lands within 48 nanoseconds of the quiet configuration over 2.4 million cycles per arm with zero ring drops. A Python ground station, simulation, and fault injection follow.
Wakeup jitter in microseconds, median of three repeats for p99.9.
| Level | Adds | p99.9 | Worst max |
|---|---|---|---|
| L0 | nothing: sleep_for(period), allocating log | 1,876,951 | 1,946,157 |
| L1 | absolute deadlines | 9.6 | 222 |
| L2 | mlockall, prefaulted stack and heap | 10.0 | 107 |
| L3 | SCHED_FIFO 80 | 13.5 | 713 |
| L4 | pinned to the isolated core | 13.9 | 535 |
| L5 | allocation-free hot path | 7.5 | 471 |
| L6 | telemetry ring and drain thread | 9.6 | 408 |
L6 adds the telemetry ring. Its planned check, L6 within 10 percent of L5 over three repeats, failed at 9.6 against 7.5. Five more repeats of each reversed the sign, L5 at 9.7 and L6 at 8.4, because runs of both configurations intermittently carry a mode that puts a few hundred cycles in the 10 to 35 microsecond band. That mode owns the tail at this floor and which arm it visits is luck, so the cost claim rests on the pooled figure rather than on either three-repeat median.