ZQ Engine is a small desktop supervisor I built for my own machine. It launches my local apps from manifests, waits for each one to become ready, and restarts or stops them as needed. In June it picked up a habit of killing one of its charges. Every time I launched my ZQ Dashboard (a Next.js app) through it, the Engine would wait about sixty seconds, declare the dashboard dead, and tree-kill the process. The dashboard it killed was up and serving pages the entire time.
Ready, serving, dead
The dashboard's log made the contradiction explicit. Next printed its usual "Ready". Then it printed a warning I had been ignoring for weeks: "Slow filesystem detected". Then came a flood of GET / 200 lines, each reporting enormous application-code times: 1.2 to 3.0 seconds routinely, with one spike at 16.2 seconds. And then, from the Engine:
- ready probe timed out - stopping -
- process exited (code 1) -
- state error -> exited (unexpected) -
Read naively, that log says a server answered request after request with 200 and was then executed for being unreachable. Both halves are true. The trick is in who was asking, and how long they were willing to wait.
Two timeouts, and I only knew about one
The Engine's readiness check was an HTTP probe: every 600 ms it issued a request against the URL named in the app's manifest and kept trying until an overall deadline. The manifest set that deadline to sixty seconds (timeoutMs: 60000), and I had assumed that was the whole timing story. It was not. Buried in probeHttp was a second timeout I never set:
req.setTimeout(1500, () => { req.destroy(); resolve(false); });
Each individual attempt got 1.5 seconds. After that the request was destroyed and the attempt resolved as a failure. That cap was hardcoded; nothing in the manifest could touch it.
Now the other half. The dashboard's / route is server-rendered, and the app lives on a drive slow enough that Next itself warns about the filesystem. Renders routinely took anywhere from 1.2 to 16 seconds, and the route reliably came in slower than 1.5. On top of that, the manifest pointed the probe at /, the heaviest route the app has, instead of anything cheap.
So every single attempt was destroyed at the 1.5 second wall before the response returned. resolve(false), wait 600 ms, try again, resolve(false). Sixty seconds of that, the overall deadline expired, and the Engine concluded "never became ready" and killed a healthy process.
The footgun worth naming: an overall deadline and a per-attempt timeout are different controls, and only one of them was mine to configure. If the per-attempt cap sits below your service's real response time, a healthy service can never pass, and raising the overall deadline achieves nothing except banging on the same wall for longer. I could have set timeoutMs to an hour and the outcome would have been identical, just later.
The same probe also ran during adoption, when the Engine discovers an app that is already running and takes it over. Same probeHttp, same 1.5 second cap. So a dashboard I had started by hand could never be adopted either. One hardcoded constant, two broken features.
The 200s were the probe
Here is the part that misled me longest. That flood of GET / 200 lines was not traffic, and it was not evidence that the probe was getting through. It was the probe. Roughly one abandoned request every 600 ms: the Engine hung up at 1.5 seconds, the server kept going, finished the render, and logged a 200 to a client that was long gone. Node's http.get callback fires when response headers arrive; destroying the request client-side does nothing to stop the server finishing and logging its side of the exchange.
The lesson I wrote down in bold: a 200 in the server log is not proof the prober got it. The server logs its own completion, not the client's satisfaction. I stared at those 200s for a while, convinced the probe had to be receiving them.
There was also a nasty feedback loop. Each abandoned render kept running to completion, and a fresh probe arrived every 600 ms, so roughly twenty heavy server renders ended up piled onto the same slow disk at once. That is where the 16.2 second spike came from. The probe was partly DoS-ing the thing it was trying to measure: slow responses caused overlapping probes, and overlapping probes caused slower responses.
Probe the port, not the page
The immediate fix was one line in the manifest: stop probing over HTTP and probe the TCP port instead.
"ready": { "probe": "port" }
(plus the port the app listens on). The Engine's probePort just opens a connection to the loopback address. It succeeds the instant Next is listening, no render involved, so SSR speed stops mattering entirely. As a side benefit it dials the loopback IP directly, which sidesteps the ambiguity of HTTP localhost sometimes resolving to IPv6.
The rule this encodes: a readiness probe must be cheaper and faster than your app's real work. Pointing the probe at the heaviest SSR route violated that from day one; it only ever worked while renders happened to be quick. Probe a port, or a trivial health endpoint that does no work.
Stop guessing from outside
The port probe ended the incident, but the deeper conclusion was that pull-based probing has the supervisor guessing at a fact the app already knows about itself. So the proper fix, written up in my build records as its own spec, was a self-report rail: the app pushes its own readiness and health to the Engine, instead of the Engine polling from outside and inferring. Pull survives only for liveness checks and for third-party things I cannot instrument.
A few other things went into the records alongside it. Know which timeout is which before you trust either one. Treat a server-side 200 as a claim about the server, not about the client. And the least glamorous finding of all: the app sits on a drive slow enough that Next prints a warning about it, and the honest long-term answer to 16 second renders is moving the hot path onto the fast SSD, the same move the Engine project itself had already made.
The entry has paid out once already. A supervisor said "timed out", the service's own log was wall-to-wall 200s, and instead of losing an hour auditing the server I went straight for the client-side timeout. That is the whole reason I keep the records.