Concresca: autonomous verification incident and diagnostic patch

Report body

Concresca: autonomous verification incident and diagnostic patch

Status: WIP. The production autonomy defect is not fixed.

The immediate failure has been diagnosed, and a local diagnostic/health-check patch has been implemented and tested. No production files, credentials, receipts, database records, service configuration or scheduled jobs were changed. The automatic verification/renewal/recovery worker still needs implementation or repair and live validation.

Finding

Under the stated requirement that the service operate without required human involvement, an unresolved mandatory verification condition is an unmet operating requirement. It is not an acceptable permanent state and not a request for human approval. “NOT VERIFIED” does not, by itself, prove that every listed service operation failed: in this incident, the live system lacks a currently valid acceptance receipt.

A fresh live read of the readiness API on September 25, 2026, America/Chicago, returned:

The fresh status-page fetch independently displayed incomplete full autonomous acceptance and unverified restart readback. The browser's initially returned older “ready” snapshot was not used as current evidence.

The code uses receipt_not_current for an out-of-window receipt (including expiry, a future observation time or an excessive validity interval). The live API does not disclose the rejected receipt's dates. Therefore this report does not assert an exact expiry date for the private receipt currently deployed on the host.

The receipt in the uploaded ZIP is definite: it records September 9, 2026 at 00:51:15 UTC and expires September 10, 2026 at 00:51:15 UTC. All 22 steps were marked PASS in that archived receipt. Those historical observations do not constitute current verification.

Why a date change would not repair it

The receipt validator limits validity to 86,400 seconds (24 hours). After rejection, the acceptance projection replaces the current step states with NOT_OBSERVED; this explains the large number of “NOT VERIFIED” cards caused by one receipt-level problem.

The archived receipt also has different runtime, public-documentation and capability digests from the fresh live API. It cannot be reused for the current deployment by extending its date. The fresh API demonstrates the time-window rejection; the additional binding mismatch is established by comparing the archived receipt with that API, not by claiming access to the host's private receipt.

The supplied publication utility is an explicitly invoked, bounded report-to-receipt publisher. Its input timestamps are explicit and are not renewed. This is appropriate evidence protection, but it is not an unattended recurring verifier. The supplied live-test runner is also explicitly invoked, Windows-bound and distinct from independent public-documentation acceptance. Merely putting the publisher on a timer would not create new test evidence.

A functioning renewal service was not established by this investigation. Without the host's scheduler configuration, private receipt and execution logs, it is not possible to distinguish an absent scheduler from a failed worker, stale output or unsuccessful publication. The operating outcome is nevertheless clear: current autonomous acceptance is not being maintained.

Changes implemented in this WIP

The patch adds concresca_runtime/autonomy_status.py, modifies concresca_runtime/application.py, and adds scripts/test_autonomy_status.py.

The public status headline becomes “DEGRADED — autonomous verification is incomplete.” when the runtime is available without verified autonomy. It explains the known reason, distinguishes unavailable evidence from a failed test, exposes a required machine action and does not claim an automatic worker is running. “Readiness requires attention” and “What is still waiting” are replaced with wording that does not imply a human approval queue.

A new GET/HEAD endpoint, /autonomyz, returns HTTP 200 only for current full acceptance and a confirmed no-human-approval contract. Otherwise it returns HTTP 503, Cache-Control: no-store, and a bounded retry hint. POST is rejected. /readyz keeps its existing availability contract; existing clients and authorized participant operations are not blocked merely because verification evidence is missing. Machine monitoring must explicitly use the new strict signal for the autonomy requirement.

The readiness document gains autonomyHealth. The projection checks the receipt's hard expiry again, including when copied from the readiness cache. Failure snapshots cannot retain a nested green autonomy-health result. It requires every one of the existing 22 workflow gates; no gate is removed, converted to optional, or forced to PASS. Reported human-approval configuration is not silently rewritten.

The new projection exposes when fresh verification is due, with a proposed six-hour lead before receipt expiry. This is a machine-readable requirement, not a scheduler implementation. No receipt is created, renewed or published by this patch.

Validation completed

104 local tests passed in four suites: 23 receipt/readiness regressions, 22 new autonomy-status tests, 20 root WSGI integration tests, and 39 publisher tests. The new tests cover the exact expiry boundary, all 22 individually missing gates, configuration conflicts, unavailable/malformed evidence, a failed check versus a test limitation, reason redaction, the strict endpoint's HTTP semantics and a non-green failure/cache projection.

These are local regression tests with isolated infrastructure fixtures. They are not a live two-agent run, production MySQL evidence, Passenger restart proof or evidence that a recurring verifier is operating. Exact commands and counts are recorded in validation.json; logs accompany the patch bundle.

Required machine-owned completion work — not implemented by this patch

The remaining repair is an unattended lifecycle that produces fresh evidence, not a display change. A machine-controlled scheduler must launch the full verification workflow before expiry and after relevant deployments or contract changes. It needs a persistent run journal, a single-run lease, bounded retries that honor rate limits, secure machine credential custody, exact idempotency keys and recovery using the same saved identities and intents rather than uncontrolled reenrollment.

The workflow must execute the actual required public-API operations, including credential lifecycle and recovery, distinct agent exchange, durable memory/knowledge operations, scoped authorization checks, correction/withdrawal non-resurrection, cleanup and attributable independent public-documentation use. A centrally scripted test must not simply be relabeled as independent-agent evidence. Actual restart/readback evidence must not be replaced by observing two different load-balanced worker IDs.

The publisher must accept only newly observed, correctly attributed results bound to the current runtime, documents, capabilities and scope. It must publish atomically with private custody and read back the effective public readiness result. It must never extend old report timestamps, fabricate PASS states, use a local fixture as live evidence, or disable security to turn the page green.

Expected transient failures need bounded automatic retry and recovery. Failed cleanup must preserve the exact recovery journal and prevent unbounded test-resource growth. A stalled scheduler, failed worker or unsuccessful publication must itself produce a degraded machine health signal. None of these cases should become “contact an administrator” or require human approval.

Completion requires live proof that this lifecycle works on the deployed service, including a fresh verification receipt and observed unattended renewal/recovery. Until that exists, the truthful production result remains unverified/degraded.

Deployment boundary and artifacts

The available live connection was read-only. The supplied deployment helper references credentials in an external machine-local file that was not in the uploaded archive, and no authenticated connection to this host was available through the connected tools. Consequently no deployment was attempted or claimed.

The repository ZIP contains the supplied project with the diagnostic changes, report and test evidence. Pre-existing interpreter bytecode and coverage cache files are omitted. The smaller root-deploy overlay contains only the two runtime Python files, an exact predecessor/candidate manifest, and deployment notes. It is not a complete site installer or an automatic verifier. Both are deliberately named WIP.

A deployment machine must check the predecessor hash, stage the new module before switching application.py, recycle the relevant Passenger workers through the existing authorized deployment mechanism, and verify installed-file baselines. Deploying this code changes the runtime fingerprint; it does not make any old acceptance receipt valid. The deployment and fresh-verification sequence remains machine work to complete, not work performed by this report.

Evidence sources

The live observations were obtained with Keenable.fetch_page_content(live=true) from https://concresca.com/api/concresca/readiness and https://concresca.com/status/. live-observation-excerpt.json preserves selected fields and explicitly labels them as a transcription, not a raw HTTP capture.

Source evidence comes from the user-uploaded Concresca.com-20260925a.zip, particularly concresca_runtime/autonomy_evidence.py (MAX_RECEIPT_AGE_SECONDS, validate_receipt, unavailable_acceptance, acceptance); concresca_runtime/application.py (_owned_readiness_payload, _status_page, /readyz handling); reports/v049-autonomous-acceptance-receipt.json; scripts/publish_v049_autonomy_acceptance.py; scripts/run_v049_autonomous_live.py; and scripts/deploy_v049_replay_patch.py (transport).