In early June I did a small tidy-up. ZQ Engine is my always-on desktop host: it launches and supervises my other local projects, each described by a manifest that says, among other things, which directory to spawn the dev command in. Engine and siblings lived together under one hub directory on my data drive. I moved the engine, and only the engine, onto the SSD, leaving an NTFS directory junction at the old path. One project moved. The junction was the whole point: nothing else was supposed to notice.
A day later, while building an unrelated feature, a headless check turned up this:
process.cwd(): X:\hub\projects\engine\app
kernel.rootDir: C:\hub\projects\engine\app (canonicalised off the junction)
dashboard resolved cwd: C:\hub\projects\dashboard\app (does not exist)
The engine thought it lived on C:. It had been started from X:, through the junction, but Node canonicalises junctions when it resolves import.meta.url (and process.resourcesPath in a packaged build), so the kernel's idea of its own root came out as the real C: path, not the X: alias it was loaded through.
That alone would be harmless. The damage came from what the root is used for. The registry makes every manifest's relative launch directory absolute against rootDir. The dashboard's manifest says, schematically, "cwd": "../../dashboard/app": go up out of the engine, across to the sibling, down into its app. Resolved against a C: root, that lands on C:\hub\projects\dashboard\app. But only the engine had moved. The dashboard still physically lived on X:, and the C: side of the junction contained exactly one project. Spawning npm run dev in a directory that does not exist fails with ENOENT before the ready probe ever gets a say.
There was even a warning sign already in the code: a ZQ_HUB_ROOT environment override for one consumer (the agent ledger's hub root), added earlier for the same junction problem. The registry's rootDir was not covered by it. The bug had bitten before, and the patch had only covered the spot that itched.
Fix one: trust the working directory
The first fix leaned on a genuinely useful asymmetry: process.cwd() is not canonicalised. It keeps whatever drive the process was started from, and both dev (npm start) and the autostart shortcut's working directory sat on X:. So the plan was: discover the hub from a non-canonicalised anchor, then rebuild the app directory underneath it.
A new hub-root.js module did the work. findHubRoot(anchors) honours ZQ_HUB_ROOT if it points at a real hub, otherwise walks up from each anchor to the nearest ancestor that carries all of the hub's marker files. The markers matter: the C: side of the junction is a stub hub containing only the engine, so it never matches. reanchorRoot then computes the app directory's path relative to its own third-level ancestor and rebuilds it under the discovered hub, with a fail-safe that returns the original root untouched if nothing is found. One line in the kernel wires it in:
const rootDir = reanchorRoot(opts.rootDir || APP_ROOT, [process.cwd()]);
Verification looked solid. The headless probe showed the root, the dashboard's cwd and the ledger's hub root all back on X: and existing. The test suite passed 21/21, the litmus checks 8/8, and a default kernel launched the real dashboard from the right directory and reached RUNNING. In my build records I wrote that for the packaged exe this fix was a no-op, because its paths already pointed at the hub drive.
That sentence was false, and I had not actually tested it. I had checked that the packaged engine booted. I had never made it launch a sibling.
The exe that was "fine"
Under two weeks later, clicking Launch on a sibling project in the packaged GUI failed with:
process error: spawn C:\WINDOWS\system32\cmd.exe ENOENT
That message is a small cruelty of Node on Windows: when the spawn cwd directory does not exist, the ENOENT gets reported against the shell executable. There was nothing wrong with cmd.exe. The working directory was not there.
I added a boot diagnostic (one line to the console and a boot log: packaged or not, cwd, resourcesPath, the root as computed, the root after re-anchoring) and the ground truth turned out to hinge on how the exe was started. Launch it from PowerShell with an explicit working directory on X: and everything stays on X:. It works, which is exactly how I had verified it, and exactly how the bug stayed hidden. Launch it from Explorer, from a .lnk shortcut, or from the autostart entry, and the executable's own path gets canonicalised through the junction: process.resourcesPath comes out on C:, and this time process.cwd() is on C: as well.
Now the whole first fix unravels. The cwd walk climbs to the stub C:\hub, which fails the marker check, so findHubRoot returns null, the fail-safe keeps the C: root, and the registry resolves every project's ../../ path against it. All four sibling projects pointed at directories that do not exist. Every launch in the engine was dead, from one moved folder.
Fix two: scan the drives, match the realpath
The durable fix stops trusting anything the environment hands the process, because every anchor so far (module URL, resources path, working directory) had turned out to be shaped by how the process was launched. What cannot lie is the filesystem itself.
findHubByJunction(rootDir) takes the canonicalised root, derives what the hub path would be, and scans drive letters A to Z for the same-tail path (the X:\hub shape) that passes two tests: it looks like the hub, and, the decisive one, joining the app's relative path under that candidate and taking its realpath lands exactly on the rootDir's own realpath. In other words, find the drive whose junction physically is this app directory. A stub cannot pass, no environment variable is needed, and no working directory has to be trusted. It is wired in as findHubRoot's final fallback, after the env check and the cwd walk, so the cheap paths still win when they can.
Verification, this time done the worst-case way: tests still 21/21; a read-only A/B through the real kernel and registry with the env var unset and the cwd forced onto C: resolved all four projects to existing X: paths; and the rebuilt exe, launched from the C: path with the override removed, logged:
rootIn=C:\hub\projects\engine\app
rootResolved=X:\hub\projects\engine\app (recovered by the junction scan)
The ZQ_HUB_ROOT stop-gap came out entirely. The code fix stands alone, with one operational note in the records: any future change to hub-root needs a rebuild of the packaged exe, or it ships the old logic.
What the junction actually changed
A junction that moves one project changes how every ..-relative path out of that project resolves, because Node resolves to real paths. My mistake, both times, was anchoring cross-project resolution on process-supplied context instead of on the one stable thing I control: the hub, identified by what it contains and confirmed by realpath identity.
The other lesson cost more. "Verified" is per launch path. A packaged Windows exe started from PowerShell and the same exe started from Explorer or an autostart shortcut are different processes as far as path canonicalisation is concerned, and I had proof for only one of them. The boot log stays in permanently for that reason: if root resolution ever goes wrong again, the first line says which drive everything landed on.
I keep written records of bugs like this, and this entry does something I want more of my records to do: it preserves the wrong claim. The first fix's write-up confidently calls the packaged exe a no-op, and the recurrence section directly below it opens by quoting that claim and marking it false. The correction is worth more with the overconfidence still attached.