My desktop engine killed a perfectly healthy server, and the log it left behind was a wall of successful requests. ZQ Engine is the always-on host that launches and supervises everything I run locally, and part of that job is a ready probe: after spawning an app, poll its port or URL until it answers, then flip the record from STARTING to RUNNING. The HTTP probe capped each attempt at a hardcoded 1.5 seconds. The dashboard it was probing renders its front page off a slow drive, and a response takes anywhere from 1.2 to 16 seconds. So every single attempt was abandoned mid-flight, the server dutifully finished each render anyway (that was the flood of 200s in its log), and after the 60 second overall deadline the engine declared "ready probe timed out" and tree-killed a process that had been serving correctly the entire time.
Read that failure mode again, because it is worse than a timeout. The probe was generating load against the app it was judging, then executing it for answering slowly. The watcher was the problem.
Polling is guessing
The morning I diagnosed this, I wrote the fix up as a spec before building anything, and the spec starts from one observation: polling is the engine guessing at something the app already knows authoritatively. Sorting the signals by who actually knows them settles the design.
Readiness ("I'm up and serving") is known by the app, so it should be pushed. Health ("degraded, queue backing up, last error") is known by the app, so it should be pushed too, as a heartbeat. Liveness ("the process hasn't wedged") is the one thing the app cannot be trusted to report, because a hung app lies by silence. That one stays with the engine and the OS.
This is not a new idea. It is exactly the systemd sd_notify pattern: the service calls READY=1, but systemd still watches the PID and runs a watchdog. So I did not rip out pull probes. They remain for everything I can't instrument: a Docker stack, a third-party tool, a server someone started from a terminal that the engine adopts after the fact. Push becomes the preferred probe for first-party apps only.
The rail
At launch, the engine now injects three environment variables into the child process, the same way it already passes ordinary launch env:
ZQ_ENGINE_URL where to POST (a loopback address on an ephemeral port)
ZQ_RUNNABLE_ID which manifest id is reporting
ZQ_ENGINE_TOKEN a per-launch nonce
The token is regenerated on every run (sixteen random bytes, hex-encoded), stored on the run record, and checked on every report; a mismatch gets a 403. Combined with the listener binding to the loopback interface only, that means a random local process can't spoof "dashboard unhealthy" or trick the engine into restarting something.
The listener itself is a small node:http server the host stands up and hands to the kernel. That split matters to me: the kernel stays free of the GUI shell (a standing rule in this project), so the kernel only gains one pure method, report(id, token, payload), and both the Electron host and the headless CLI harness inject the same transport around it. The endpoint is a single POST /report taking:
{ "id": "…", "token": "…",
"status": "ready|healthy|degraded|unhealthy|stopping",
"detail": "optional human string", "metrics": {}, "ts": 0 }
Each status maps to one behaviour. ready clears the ready deadline and flips STARTING to RUNNING, instantly, no render required, so the slow disk stops mattering. healthy refreshes the last-beat timestamp and updates a health record. degraded and unhealthy update that record but the app stays RUNNING: health is a sidecar attribute, not a run-state, and I refused to tear the state machine apart for it. stopping marks the shutdown as requested so the imminent exit is classified clean, which killed a whole category of "exited (unexpected)" noise on graceful stops.
The ready path itself no longer polls at all. When a manifest declares the report probe, the engine sets a deadline and waits. An app that never reports still times out into an error state, so the failure behaviour of the old probes is preserved; the difference is that the happy path costs zero requests.
A reaper that never kills on one missed beat
The heartbeat needed a backstop, and backstops are where health systems usually turn dangerous. If a manifest declares a heartbeat interval, a kernel timer checks the gap since the last beat, and after a threshold of missed intervals (default three) it marks the app unhealthy and emits an event. That is all it does by default. The onMiss behaviour defaults to "mark", never kill, because true process death is already caught by the existing exit handler, and a missed beat alone does not mean dead. Restart-on-miss exists, but it is opt-in per manifest. Killing on a single missed heartbeat is how you rebuild the original bug with extra steps.
The client that doesn't care who launched it
The other half is a zero-dependency ES module that any of my Node apps can import. Its entire activation logic is reading those three environment variables:
const enabled = !!process.env.ZQ_ENGINE_URL;
Not launched by the engine? Every call is a silent no-op. Launched by it? reportReady(), startHeartbeat() and a shutdown reporter just work. Errors are swallowed, because a reporting library must never crash the app it reports for. The payoff is that the exact same code runs under the engine, standalone from a terminal, and in CI, with no configuration and no conditionals at the call site.
Wiring it into the dashboard was one file: Next.js runs instrumentation.ts once at server startup, so reportReady() fires the moment the server is listening, long before the first slow render completes. The one wrinkle: Turbopack rejected the dynamic import specifier as "expression too dynamic", so the helper gets loaded through an indirect Function("return import"). Not proud, works fine.
There is one honest gap. An instance of a report-probe app started outside the engine has no injected token, so it cannot self-report. For those, the manifest can declare a nested pull-probe fallback so the engine's discovery pass can still adopt it.
Did it actually work
Same-day turnaround on this one, in stages. The morning session diagnosed the root cause, shipped a one-line stopgap (a TCP port probe, which succeeds the instant the socket listens, no render involved), and wrote the handoff spec. A build session later that day shipped all six steps of the rail. The test suite passed 21/21 including a real-child end-to-end run, the litmus suite 8/8, the helper's self-test 3/3.
The live verification is the part I cared about: the real dashboard, launched through the kernel with the rail active, reached RUNNING about five seconds after Next was ready, via a pushed report, with zero probe requests in its log, heartbeat green. The flailing-then-kill sequence from the original incident was gone.
The build also surfaced an unrelated bug, because builds always do: the dashboard's launch directory resolved through a directory junction to a path that doesn't exist, so the engine couldn't even spawn it on that machine. I verified the rail by pinning the root directory and filed the junction bug separately in my build records.
The rule I keep from this: poll only for what the process cannot tell you. Liveness stays external because a wedged app lies by silence. Everything else, the app already knows, and asking it from the outside every 1.5 seconds is how you end up load-testing your own dashboard and killing it for answering.