Skip to content

Fault recovery matrix ​

Knossos treats the active scan as the last known-good architecture graph. Failed work is never activated, derived caches are disposable, and worker processes are supervised within explicit protocol and resource limits.

FaultObservable diagnosticRecovery and preserved state
Worker crash or broken pipeWORKER_EXITED or WORKER_PIPE_BROKENWorker and Linux descendants are terminated; that language degrades to an error diagnostic and the rest of the scan proceeds.
Worker timeout or floodWORKER_TIMEOUT or WORKER_OUTPUT_LIMITRequest is aborted and the process tree terminated; that language degrades to a diagnostic, and the limits in force are reported in the scan's worker_execution block.
Worker failure on a rescanWORKER_DEGRADED_INCREMENTALThe scan aborts and the active graph is unchanged when the graph holds facts for the failed language. Fix the worker (run knossos doctor) and rescan. A full rescan whose failed languages hold no facts degrades them as a first scan does.
Cancellation or signalKNOSSOS_SCAN_CANCELLED / watch stopped eventWorker cleanup and transaction rollback run; lease is released.
Tree changed under a scanKNOSSOS_SCAN_SNAPSHOT_CHANGEDNo graph row is written and the previous active graph stays queryable; a failed scan row is recorded for the reaper. Rerun once writes to the tree have stopped.
File unreadable under a scanKNOSSOS_SCAN_SNAPSHOT_CHANGEDSame preserved state, different remedy: waiting does not restore readability, so make the named path a readable file again before rerunning.
Rejected path or configurationKNOSSOS_UNSAFE_PATHNo scan is activated and the previous active graph stays queryable. Correct the root, path or invalid knossos.json the message describes, then rescan.
Concurrent writerKNOSSOS_SCAN_BUSYCurrent active graph remains queryable; retry after the writer finishes or stale lease recovery.
SQLite write lock contendedKNOSSOS_DATABASE_BUSYNothing was written; retry once the writer, usually a scan, has committed.
SQLite full/I/O failureKNOSSOS_STORAGE_ERRORTransaction fails closed; free capacity, then retry.
Partial reconciliation writeKNOSSOS_STORAGE_ERROR or runtime errorThe graph transaction rolls back and prior active scan remains selected.
Corrupt contribution cacheno user-visible error; affected file is reparsedInvalid derived payload is discarded and rebuilt from read-only source.
Corrupt databasefailing doctor integrity checkStop writers and restore a verified atomic backup; do not continue scanning the corrupt file.
Stale writer leaseKNOSSOS_SCAN_BUSY until lease expiryA writer whose process on this host is gone loses its lease to the next scan at once. Otherwise a later writer removes the expired lease (an hour) and proceeds.

CLI execution failures use exit code 2 and a stable diagnostic prefix. MCP tool errors carry the same family in structured content. Details after the prefix are explanatory and may vary by operating system or SQLite version.

Worker request batching ​

The limits in the worker_execution block are per request, not per project: a worker request has its own cumulative output-byte budget and its own request_timeout_ms deadline. A language's files are therefore split across several requests.

The bounds are reported per language, under worker_execution.scan_batches, because they differ per language and the byte budget can differ per scan:

  • files: most files sent per worker request. This bound guards the deadline.
  • source_bytes: most source bytes sent per worker request. This bound guards max_output_bytes, because protocol output has a large per-file constant (roughly 0.8-1.8 KB) plus a term that scales with how much source the request covers. Neither axis alone is sufficient, and neither is both together, since how densely a file declares symbols also drives output and is not modelled at all. That unmodelled term is why the budget adapts.
  • source_bytes_used: the narrowest budget any of that language's requests ran at. Lower than source_bytes means at least one batch overflowed and was re-split; see below.

The default is 400 files and 4 MB per request. TypeScript uses a much larger file cap (2,000), because it rebuilds and re-checks a whole ts.Program on every request (a cost set by the program, however many files the request named), so splitting its work repeats the expensive part. Its byte budget is 3 MB, and so is Rust's.

Adaptive budgets ​

Measured expansion from source bytes to protocol output varies far too much for any fixed budget to be both safe and fast: 1.68x for KaTeX's TypeScript sources, 2.24x for this repository's PHP, but 15x for corpora of very small generated files and for code unusually dense in declared symbols. The byte budget is therefore set optimistically, and corrected when it is wrong.

When a scan request fails with a size signal, the budget is halved, that language's worker is restarted, and the failing batch is re-split and retried. The reduction applies only to that batch and its descendants: later batches start again at the configured budget, so one pathological directory costs that batch its own retries, and the rest of the language keeps its budget for the rest of the scan. A batch and its descendants may be halved up to max_scan_batch_halvings times (4), however many other batches also needed retries, after which the failure falls through to the ordinary degrade path below.

Three kinds of failure are retried this way: a size signal (WORKER_OUTPUT_LIMIT, WORKER_FRAME_TOO_LARGE, WORKER_REQUEST_TOO_LARGE), a TypeScript worker that exited from V8 heap exhaustion on a request that names no tsconfig, and a worker of any language killed by SIGTERM or SIGKILL, which Knossos never sends to a worker that still owes a response. Those two are what a host memory guard such as earlyoom or systemd-oomd, or the kernel's OOM killer, send to the largest process when memory runs low, and a smaller request on a fresh worker needs less memory at its peak. A worker killed by hand with kill is retried the same way; to stop a scan, cancel it instead. A worker that crashed with its own signal (SIGSEGV, SIGABRT, SIGBUS and the like), one stopped by SIGHUP or SIGINT, a crash for any other reason, a timeout, or a cancellation is never retried.

Each retry halves both the file count and the bytes the batch actually held. source_bytes_used reports the narrowest budget these ordinary retries settled on.

WORKER_FRAME_TOO_LARGE is searched for differently, because it is one file's answer that did not fit one frame. The files the worker had already answered are not the cause, so they go back as one batch and only the rest is searched: its first file alone, since a worker that answers in order failed on exactly that file, then the others in two halves. A split that confirmed at least as many files as it still suspects halves the search and is free. Every other split draws on a search allowance per language of max_scan_batch_halvings plus two binary searches over the files it scans (16 for 64 files, 26 for 2,000). A worker that fails before answering anything, or answers one file before each failure, is therefore not searched file by file; when the allowance runs out the language degrades with a diagnostic saying the line limit is probably too low for the project. These splits do not lower source_bytes_used. An oversized frame that is not a file's contribution (the worker's response, another notification, or anything after every file in the batch was answered) belongs to no file, and degrades the language instead of leaving a file out.

On top of both, a language may send at most two requests per file plus four per batch in one scan. Reaching that cap degrades the language with WORKER_REQUEST_CAP, naming the cap and the last failure.

A batch of one file cannot be split any further, so it is never retried. When that one file's own frame or output is what outgrew a limit, as a generated bundle can, the file is left out with a diagnostic naming the limit and the rest of the language is kept. The scan names it in its summary line and in a SCANNER_FILES_LEFT_OUT warning, and counts it in left_out_files and nowhere else among the processing counts: not in parsed_files, and not in unchanged_files when a rescan reuses it. Its content still counts in added_files or changed_files like any other file's. The left-out result is cached against the file's content and the limits that excluded it, so an unchanged file stays out without being sent again. When every file of a language (two or more) is left out, the language is reported as failed with WORKER_EVERY_FILE_LEFT_OUT, since that points to a limit set too low or a broken worker. Any other failure of a one-file batch degrades the language at once. When retries run out, the diagnostic says how many retries were made.

Sending a whole project in one request made max_output_bytes a project-wide ceiling: at roughly 14.9 KB of protocol output per PHP file, a scan of more than about 1,340 files aborted with WORKER_OUTPUT_LIMIT even though max_files permits far more. Batching keeps the reported budgets meaningful at any project size.

Degraded scans ​

A worker that fails or times out costs its own language. The remaining languages are still analysed and reconciled, so a dead TypeScript worker leaves a usable PHP and Python graph.

  • degraded_languages: owner keys of scanners that failed during this scan. Non-empty means the graph is partial: the listed languages contributed no facts, and the corresponding failure is also persisted as an error diagnostic against the scan.
  • Every failure is repeated in the envelope's warnings as CODE: message, so a caller reading only the warnings still learns the answer is incomplete.
  • Cancellation is not a degradation. KNOSSOS_SCAN_CANCELLED still aborts the entire scan, because the caller chose to stop, which is no fault of the scan.
  • A tree that changes mid-scan is not a degradation either. Discovery hashes every file and the language workers read those same paths again for themselves, so a write landing between the two reads leaves graph facts that no recorded hash describes, which every later drift check would then report as fresh. The scan re-hashes what it discovered before it writes anything to the graph and aborts on the first file that no longer matches, was removed, or can no longer be read. The message names that file, and its wording says which of the three happened, because the last one is not a race to wait out: a rewritten or removed file is resolved by rerunning once the tree has settled, while an unreadable path stays unreadable and fails the next scan in the same place until it is corrected. As with any terminal attempt, the aborted scan is recorded as a failed scan row so stale-scan cleanup can reap it; no graph row is touched.

Worker diagnostic codes ​

Every code a worker failure is reported under. Unless a row says otherwise, the failure degrades that language to an error diagnostic and the rest of the scan proceeds.

CodeMeaning
WORKER_CANCELLEDThe scan or enrichment was cancelled while a request was in flight. Reported as a cancelled scan, not a degraded language.
WORKER_CAPABILITY_MISMATCHThe worker's manifest does not declare a capability the request needs.
WORKER_CONTRIBUTION_INVALIDA contribution broke the protocol: a fact, read or path the core refuses. The worker's language degrades.
WORKER_DEGRADED_INCREMENTALA rescan failed for a language whose facts the active graph holds; the scan aborts and the graph is unchanged.
WORKER_EVERY_FILE_LEFT_OUTEvery file of a language was left out as too large for the limits in force.
WORKER_EXITEDThe worker process exited before answering.
WORKER_FAILEDA language failed with an error that carries no worker code of its own.
WORKER_FRAME_INVALIDThe worker wrote an empty frame or one that is not a JSON-RPC object.
WORKER_FRAME_TOO_LARGEOne response frame outgrew the frame limit; the batch is searched for the file that caused it.
WORKER_IO_FAILEDReading from or polling the worker's pipes failed. Retryable in watch mode.
WORKER_JSON_INVALIDThe worker wrote a line that is not JSON.
WORKER_MANIFEST_INVALIDThe worker's initialize response is not a valid manifest.
WORKER_MANIFEST_MISMATCHThe worker's manifest describes another worker than the language it was launched for; rebuild it.
WORKER_OUTPUT_LIMITThe worker's output for one request outgrew the output limit; the batch is split.
WORKER_OUTPUT_SCHEMA_MISMATCHThe worker declares an output schema the core does not read.
WORKER_PIPE_BROKENWriting a request to the worker failed. Retryable in watch mode.
WORKER_PROTOCOL_VERSION_MISMATCHThe worker speaks a protocol version the core does not.
WORKER_REQUEST_CAPA language needed more requests than the request cap allows.
WORKER_REQUEST_INVALIDA request could not be encoded as JSON.
WORKER_REQUEST_TOO_LARGEA request frame outgrew the frame limit; the batch is split.
WORKER_RESPONSE_INVALIDA response is valid JSON-RPC but breaks the shape the method returns.
WORKER_RPC_ERRORThe worker answered a request with a JSON-RPC error.
WORKER_START_FAILEDThe worker process could not be started.
WORKER_TIMEOUTA request ran past its deadline. Retryable in watch mode.
WORKER_UNEXPECTED_RESPONSEA response names a request id other than the one in flight.

Fault-injection coverage ​

The standard suite exercises malformed/crashed/slow/flooding workers, cancellation, stale and competing leases, reconciliation rollback, malformed frames, corrupt cache rebuilds, SQLite page exhaustion, database locks, and a worker that spawns a long-running child. The Linux supervisor test verifies both worker and descendant disappear after cancellation.

sh
vendor/bin/phpunit --group=worker
vendor/bin/phpunit --group=concurrency
vendor/bin/phpunit --group=fault-injection

On Linux, Knossos enumerates the supervised worker's /proc descendants before termination and sends graceful then forced signals. Other operating systems still terminate the direct worker; third-party scanners must not detach child processes. The bundled PHP, TypeScript, Python, and Rust scanners do not spawn analysis children.

For database damage, use maintain-database integrity and restore a backup as described in maintenance. Cache corruption never requires a backup because contribution payloads are derived and validated before replay.