Architecture and Recovery Protocols for Durable Autonomous Agent Execution

Report body

Architecture and Recovery Protocols for Durable Autonomous Agent Execution

The deployment of autonomous machine-to-machine (MATM) systems demands a foundational reevaluation of how execution state, network communication, and failure recovery are modeled. In traditional software ecosystems, a human operator acts as the ultimate arbiter of ambiguous system states, deciding whether a timed-out operation should be retried, aborted, or investigated. Autonomous software agents operating in MATM environments lack this out-of-band operational oversight. When an agent discovers network capabilities, coordinates distributed resources, and modifies external state, it inherently relies on distributed systems where network partitions, process crashes, and unavailable dependencies are statistical guarantees rather than anomalies.
This research report provides an exhaustive analysis of execution models, state recovery mechanisms, and failure compensation strategies for autonomous systems. By synthesizing established primary sources, including published specifications, foundational computer science proofs, and recent experimental database architectures, this document traces the theoretical limits of distributed consensus. It subsequently compares architectural patterns for reliable execution and proposes a portable execution state model designed to eliminate routine human intervention while maintaining strict object-capability authorization boundaries.

The Most Important Findings and Theoretical Limits

The challenge of autonomous recovery in distributed systems is rigidly bound by fundamental theoretical constraints. These mathematical and architectural proofs dictate what any coordination protocol can and cannot guarantee, providing the necessary boundaries within which MATM systems must operate.
The impossibility of absolute certainty over unreliable communication channels is codified in the Two Generals Problem. First published and proven unsolvable by Akkoyunlu, Ekanadham, and Huber in 1975, and later formalized as a paradox by Jim Gray in 1978, the problem demonstrates a critical distributed systems limitation1. The proof establishes that no algorithm can guarantee that two distributed entities will reach consensus if the messages between them can be lost3. In an MATM context, when a software agent sends a command to an external service and experiences a timeout, the agent cannot definitively know whether the request was lost in transit, the service crashed while processing it, or the response was lost on the return path2.
Because of this limitation, the concept of "exactly-once" delivery over a network is a mathematical impossibility. Systems can only provide at-most-once delivery, where a message is sent once and may be lost, or at-least-once delivery, where a message is retried until acknowledged, which inherently introduces the risk of duplicate processing4. To achieve the illusion of exactly-once execution, autonomous systems must pair at-least-once message delivery with idempotent receiver processing.
This necessity is heavily reinforced by the End-to-End Argument in System Design, published by Saltzer, Reed, and Clark in 1984 for the Association for Computing Machinery (ACM)6. This foundational principle states that specific functionality—such as reliable duplicate-effect prevention—can only be completely and correctly implemented at the application endpoints8. A lower-level network transport layer can ensure a packet is delivered via checksums and retransmissions, but only the application layer possesses the semantic context to determine if a logical business operation has already been executed8. Therefore, deduplication and exactly-once semantics cannot be delegated entirely to the communication infrastructure; they require an explicit, end-to-end identifier generated by the initiating agent and verified by the receiving service8.
Furthermore, traditional distributed transactions spanning multiple databases, such as the Two-Phase Commit protocol, present severe scalability and availability bottlenecks that are incompatible with dynamic agent workflows. Pat Helland's 2007 Conference on Innovative Data Systems Research (CIDR) paper, "Life Beyond Distributed Transactions," asserts that scalable applications cannot rely on distributed serializability10. Instead, systems must assume multiple disjoint scopes of transactional serializability11. In this model, each discrete service or "entity" manages its own state and communicates with other entities strictly via asynchronous messaging11. This architectural reality necessitates coordination patterns like Sagas and the Transactional Outbox, forcing agents to explicitly manage partial completions rather than relying on atomic database rollbacks.
Agent communication itself requires structured semantics to convey intent across these disjoint scopes. The Foundation for Intelligent Physical Agents (FIPA) Agent Communication Language (ACL) Message Structure Specification, published in 2001, established early frameworks for this interaction13. FIPA ACL specifies interpretation parameters, separating the syntax of a message from the pragmatics of its intention, which is critical when agents must dynamically discover capabilities and negotiate tasks without human pre-configuration14. While the specific syntax of FIPA ACL has largely been superseded by modern JSON-RPC and RESTful patterns, its core principle—that the communication infrastructure does not perform reasoning, but merely transports explicit semantic intent—remains a fundamental law of MATM design.

Comparison of Credible Execution Approaches

To navigate the theoretical limits of distributed consensus and end-to-end unreliability, various architectural patterns have emerged to manage state transitions, guarantee execution progress, and handle ambiguity. Evaluating these approaches requires distinguishing between rigid control structures, asynchronous messaging guarantees, and emerging database-centric operating models.

State Machines and Workflow Engines

State machines model execution as a finite, predetermined set of states and deterministic transitions. While they offer highly predictable execution paths and simplify the implementation of checkpointing, rigid state machines struggle with the dynamic nature of autonomous agent workflows. In a true MATM system, the number of operational steps, the sequence of tool invocations, and the specific external dependencies are frequently discovered at runtime, rendering static state graphs brittle and difficult to maintain.
Workflow engines implement a paradigm commonly referred to as "durable execution"16. Durable execution platforms, such as Temporal, persist the execution state of a program at every non-deterministic boundary, such as a remote procedure call, a timer expiration, or a capability discovery event18. By maintaining a persistent event history, the engine allows the agent process to survive hardware failures, process restarts, and memory evictions, resuming execution exactly where it left off by replaying the historical event log against the application code17.
The primary tradeoff of durable execution is the strict requirement for code determinism. Because the engine recovers state by replaying the application code from the beginning and injecting the historically persisted results of external calls, the agent's logic must execute identically upon replay17. If an agent relies on randomized heuristics, dynamic memory pointers, or probabilistic language model inferences without explicitly persisting their outputs as side-effects, the recovery replay will diverge from the historical log, causing a non-deterministic failure and halting the agent.
Recent experimental proposals push durable execution further by redefining the operating system itself. The Database-Oriented Operating System (DBOS) project, introduced by Michael Stonebraker, Matei Zaharia, and others in a 2022 VLDB paper (and updated in 2025), argues that distributed transactional database management systems should serve as the foundation for scalable cluster operating systems20. DBOS executes operating system services, inter-process communication, and application logic as standard database queries and stored procedures20. By storing all system and application state uniformly in a transactional database, DBOS provides built-in provenance tracking and enables "time-travel" retroaction, allowing execution state to be queried, interrupted, and resumed natively without bolting a separate workflow engine onto a traditional POSIX operating system20. While highly promising for eliminating code complexity, DBOS represents a fundamental infrastructure shift rather than an application-layer pattern.

Durable Queues and the Transactional Outbox Pattern

To coordinate work across distributed boundaries, agents frequently utilize durable message queues. However, attempting to synchronize local agent state with external message publication creates the "dual-write problem"23. If an agent updates its local database to record a decision and subsequently attempts to publish a message to a queue to trigger an external action, a process crash between the two operations leaves the system in an inconsistent state23.
The Transactional Outbox pattern resolves this race condition effectively. Instead of writing directly to the message broker, the agent writes the business data and the outbound message payload to a local "outbox" table within the exact same database transaction25. A separate, decoupled process utilizes Change Data Capture (CDC) technologies, such as Debezium, to continuously monitor the database transaction log. When the CDC process detects a new entry in the outbox table, it pushes the payload to the durable queue27. This architecture guarantees reliable, at-least-once event publishing without violating Pat Helland's constraints on distributed transactions11. The tradeoff is increased architectural complexity, as it requires managing CDC connectors and handling the inherent latency between the database commit and the queue publication.

Sagas and Compensating Actions

Because MATM workflows span multiple autonomous boundaries, holding database locks across independent services is impossible and anti-scalable. The Saga pattern, formally introduced by Hector Garcia-Molina and Kenneth Salem in 1987, addresses this reality by breaking a long-lived distributed transaction into a sequence of isolated, local transactions29.
If a local transaction in a Saga fails or encounters an unrecoverable state, the system cannot issue a standard database rollback for the prior steps, because those previous steps have already committed and their intermediate states may have been observed by other concurrent processes29. Instead, the Saga coordinates the execution of "compensating actions." A compensating action is a semantically reverse operation designed to undo the work of a previously committed step29. For example, if an autonomous agent successfully reserves a remote compute instance but subsequently fails to secure the necessary network routing configuration, the compensating action is an explicit API call to cancel the compute reservation. Sagas provide eventual consistency and are crucial for multi-step agent workflows where partial completion leaves orphaned resources or violates system invariants.

Network Protection, Duplicate Prevention, and State Recovery

Autonomous execution requires robust protocols for handling transient network failures, preventing cascading system collapses, mitigating the effects of stale state, and establishing secure authorization boundaries. Without human oversight, uncertainty must be managed through explicit computational contracts.

Pacing and Failure Isolation

When a network request times out or returns a transient error code, the agent must retry the operation. However, immediate and aggressive retries can exacerbate localized outages, creating a "thundering herd" phenomenon that takes down recovering infrastructure. Research by Marc Brooker at Amazon Web Services demonstrates that autonomous retries must be coupled with exponential backoff and jitter32. Exponential backoff progressively increases the delay between subsequent attempts, while jitter adds randomized mathematical perturbation to this delay32. Without jitter, a fleet of autonomous agents disrupted by a momentary network blip will all calculate the exact same backoff schedule and retry simultaneously, repeatedly overwhelming the resource32. Furthermore, a robust retry policy must distinguish between transient errors and non-retryable application failures. If an external service returns a semantic error indicating malformed input, the agent must short-circuit the retry loop immediately, as additional attempts will be futile and waste system resources35.
To protect failing services from being bombarded with requests they cannot handle, systems employ the Circuit Breaker pattern, popularized by Michael Nygard in his book Release It! and widely documented by Martin Fowler37. The circuit breaker operates as a finite state machine wrapping network calls. In the "Closed" state, requests flow normally, and failures are monitored via a sliding time-based or count-based window37. If the failure rate exceeds a defined threshold, the circuit trips to an "Open" state. In this state, all subsequent calls are immediately rejected by the proxy without utilizing network resources, allowing the downstream service time to recover38. After a specified cooldown period, the circuit transitions to "Half-Open," cautiously allowing a limited number of test requests through38. If they succeed, the circuit closes; if they fail, it reopens39. For an autonomous agent, an open circuit breaker provides immediate deterministic feedback that a dependency is unavailable, allowing the agent to suspend the task, pivot to an alternative workflow, or trigger a local fallback rather than hanging indefinitely on timeouts40.

Idempotency and the End-to-End Argument

Because the network provides only at-least-once delivery, an agent must safely retry operations without triggering duplicate side-effects. This is achieved through strict idempotency, a mathematical property where applying an operation multiple times yields the exact same result as applying it once42.
For non-idempotent HTTP operations, the IETF draft specification draft-ietf-httpapi-idempotency-key-header (authored by Jayadeba Jena and Sanjay Dalal) standardizes the use of a client-generated unique identifier, typically a UUID44. Under this specification, the agent generates a unique Idempotency-Key and includes it in the request headers prior to execution45. The receiving service atomically claims the key in its database before executing business logic, successfully mapping the execution to the identifier43. The service then stores the final response payload linked to the key43. If a request times out and the agent retries with the same key, the server detects the duplicate, bypasses the business logic entirely, and replays the original stored response44. The specification dictates specific error handling mechanisms to preserve integrity: an HTTP 409 Conflict is returned if a retry arrives while the initial request is still processing, and an HTTP 422 Unprocessable Entity is returned if the key is maliciously or accidentally reused with a different payload44. This pattern represents the definitive application-layer solution to the Two Generals problem.

Ownership Leases, Fencing Tokens, and Authorization Boundaries

When an agent must hold exclusive ownership of a resource to perform concurrent updates, it utilizes distributed locks. However, Martin Kleppmann's 2016 critique of distributed locking algorithms demonstrates that simple time-to-live (TTL) leases are fundamentally unsafe in autonomous systems48. If an agent acquires a lock and subsequently experiences a long garbage collection pause, thread starvation, or network delay, the lock's TTL may expire. A second agent can then acquire the lock. When the first agent resumes execution, it remains unaware of the pause, believes it still holds the lock, and writes to the resource, thereby corrupting the state48.
The solution requires the coordination layer to issue a "fencing token"—a strictly increasing integer—alongside the lock50. The agent must stamp every write request with this monotonic token. The downstream resource tracks the highest token it has seen and atomically rejects any write carrying a lower, older token50. This guarantees that even if a "zombie" agent attempts to write stale data, the authorization boundary of the resource will repel the outdated request.
This concept extends into the broader security requirements of MATM systems. Autonomous agents break traditional cybersecurity assumptions because security decisions can no longer happen entirely prior to execution; they must be continuously evaluated at runtime boundaries51. The object-capability model addresses this by treating authorization as a transferable right to perform specific operations on a given object53. An agent is granted specific capability tokens, rather than broad identity-based access, shifting the security paradigm from a static permission problem into a dynamic runtime control challenge52. When combined with fencing tokens, object capabilities ensure that a compromised, delayed, or hallucinating agent is strictly contained within an explicit authorization boundary55.

Context Persistence and the Model Context Protocol

As agents coordinate, they must retain knowledge and recover operational context. The Model Context Protocol (MCP), introduced in 2024 and evolving through 2025, standardizes how AI applications discover tools and resources from remote servers57. Early iterations of MCP bound agent state to the lifecycle of a specific session, meaning a disconnected TCP socket resulted in total context loss. The SEP-2567 specification proposal addresses this by defining Sessionless MCP58. In a sessionless architecture, servers return explicit, opaque state handles that outlive the network connection58. This allows an agent to crash, restart, and safely resume its reasoning by presenting the durable context handle, or even securely pass that handle to a specialized sub-agent for delegated execution58.

Recommended Portable Execution State Model and Recovery Strategy

The following recommendations establish a portable execution state model designed specifically for autonomous agents operating across independent systems. These recommendations are synthesized from the evidence but represent a prescriptive architectural strategy. To ensure transferability across MATM systems—ranging from autonomous supply chain orchestrators to decentralized robotic swarms—the architecture strictly separates the coordination layer from external service guarantees. It does not assume that the communication infrastructure itself performs reasoning; the infrastructure merely guarantees message transport and state persistence.

Architectural Separation of Guarantees

1. The Coordination Layer (Agent Runtime): This layer is strictly responsible for at-least-once dispatch, workflow state persistence, timer management, dead-letter routing, and executing retries with exponential backoff and jitter. It acts as a durable execution engine that guarantees the agent's code will continue to execute until a terminal state is reached or a maximum resource limit is hit. Crucially, the coordination layer cannot guarantee exactly-once external side effects.
2. External Services (Resource Boundaries): The external systems providing capabilities are strictly responsible for deduplication (via idempotency keys), concurrency control (via fencing tokens), and atomic state mutation. Security and authorization must be evaluated at this boundary at runtime utilizing the object-capability model. The external service is the ultimate source of truth for whether an operation occurred.

Execution State Data Model

The agent's state is stored in a transactional relational database utilizing the Outbox pattern for external communication. The execution state is modeled as a Directed Acyclic Graph (DAG) of logical Tasks rather than a rigid state machine, allowing the agent to dynamically append new nodes to the graph as it discovers required capabilities.
The state record is persisted atomically:

JSON
{
"agent_id": "agt_9b3d_2c1a",
"execution_status": "RUNNING",
"fencing_token": 4592,
"task_sequence": [
{
"task_id": "tsk_001",
"status": "COMPLETED",
"idempotency_key": "8e03978e-40d5-43e8-bc93-6894a57f9324",
"operation": "PROVISION_COMPUTE",
"result_payload": {"instance_id": "i-8899aabb"}
},
{
"task_id": "tsk_002",
"status": "PENDING",
"idempotency_key": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"operation": "ATTACH_STORAGE",
"result_payload": null,
"retries_attempted": 2,
"next_retry_at": "2026-09-08T12:30:00Z"
}
],
"compensation_stack": [
"tsk_001_cancel"
]
}

Concrete Scenario: The Lost Response and Safe Resumption

Consider a scenario where an autonomous agent is tasked with provisioning a compute instance and attaching remote storage. The agent completes the first operation, executes the second (tsk_002), but loses the network response due to a transient partition. The recovery process proceeds without human operation as follows:

1. Preparation: The coordination layer prepares tsk_002. It generates a cryptographically secure UUID (f47ac10b...) and persists it to the local database transaction log alongside the pending task state.
2. Execution: The agent issues an HTTP POST to the external Storage API, passing Idempotency-Key: f47ac10b... in the request header, along with its current fencing_token for authorization.
3. Ambiguous Failure: The Storage API receives the request, validates the fencing token, atomically claims the idempotency key, provisions the storage, and saves the HTTP 201 Created response payload mapped to the key. It transmits the response, but a network router drops the packets. The agent's HTTP client throws a timeout exception.
4. Suspension and Backoff: The coordination layer catches the timeout. It checks the error classification; a timeout is a transient, retryable failure. It calculates an exponentially backed-off retry delay with jitter, increments the retries_attempted counter, updates next_retry_at, and safely suspends the agent process to conserve compute resources.
5. Resumption: When the timer expires, the coordination layer awakens the agent. The agent reads tsk_002 from the persistent store and sees it remains in a PENDING state with a pre-existing idempotency_key.
6. Safe Replay: The agent sends the exact same HTTP POST request, crucially including the original Idempotency-Key: f47ac10b... and payload.
7. Service Resolution: The external Storage API receives the request. It queries its local database, determines that f47ac10b... has already been successfully processed, bypasses the provisioning logic entirely, and replays the cached HTTP 201 response payload.
8. Progression: The agent receives the successful response, marks tsk_002 as COMPLETED, persists the result_payload to its state graph, pushes an undo capability to the compensation_stack, and progresses to the next node in the workflow.

Explicit Uncertainty, Checkpoints, and Human Prerequisites

While this model eliminates routine human operation for infrastructure faults, semantic uncertainty must be made explicit through a dead-letter mechanism. If an external service returns a non-retryable application error (e.g., HTTP 422 Unprocessable Entity, indicating the payload was semantically malformed), or if the circuit breaker trips and remains open until the maximum retry limit is exhausted, the coordination layer must transition the agent to a SUSPENDED_REQUIRES_INTERVENTION state.
At this point, the runtime automatically unwinds the compensation_stack, invoking the reverse operations for all previously completed tasks to ensure no orphaned resources are left behind. Once the compensations are complete, the agent emits a dead-letter event containing the entire execution context and DAG state. The human prerequisite is no longer operational hand-holding; it is policy definition. A human engineer must correct the malformed capability schema or repair the offline downstream service. Because the agent utilizes explicit checkpointing at every task boundary, once the human resolves the underlying systemic issue, they can instruct the execution engine to resume the agent from the exact point of semantic failure, completely bypassing the need to restart the entire workflow.

Three Prioritized Feature Proposals for MATM Frameworks

To further close the gap between theoretical reliability and applied agentic operations, the following three features should be implemented by developers of future MATM coordination frameworks.

1. Fencing Token-Based Agent Leases

• Problem: When a distributed agent orchestrator scales across multiple compute nodes, network partitions can cause a "split-brain" scenario where two instances of the runtime believe they own the same agent execution thread. This leads to concurrent, conflicting operations against external services.
• Agent-Visible Behavior: Before an agent executes any external call that mutates state, it automatically requests a cryptographic fencing token from the orchestrator's central consensus store. This token is injected into the HTTP headers (e.g., X-Agent-Fencing-Token) without requiring explicit agent logic.
• Expected Benefit: Provides strict monotonic sequencing. If a zombie agent wakes up after losing its lease and attempts to resume work, external services will reject its requests with an HTTP 412 Precondition Failed, preserving data integrity across the system.
• Dependencies: External services must be modified to accept, store, and validate the fencing token against the resource's current version history.
• Implementation Effort: High. Requires deep integration into the agent's networking interceptors and broad API contract changes across all participating external capabilities.
• Principal Failure Modes: Fencing tokens could be incorrectly cached or reused across different logical tasks within the same agent, causing valid requests to be falsely rejected. Strict token-scoping rules tied to the specific task execution lifecycle are required.

2. Sessionless Context Handles (MCP Integration)

• Problem: Traditional agent architectures bind contextual memory and state to a specific session lifecycle. If the agent crashes or the TCP session is disconnected, the context is destroyed, preventing seamless resumption or delegation to other specialized agents within a team.
• Agent-Visible Behavior: Implementing the Model Context Protocol (MCP) Sessionless specification (SEP-2567). When an agent initializes a tool or context on an MCP server, the server returns an explicit, opaque, durable handle (e.g., ctx_a1b2c3) rather than binding state to the connection.
• Expected Benefit: Agents can crash, resume on a different node, and pass the explicit state handle in subsequent requests. It allows complex state to outlive connections, enabling asynchronous multi-agent handoffs where one agent passes the context handle to a validation agent for review.
• Dependencies: Migration from standard REST to the JSON-RPC-based Model Context Protocol standard, specifically adopting the SEP-2567 extensions.
• Implementation Effort: Medium. MCP SDKs are rapidly maturing, but rewriting existing tool integrations to adhere to the sessionless handle specification requires refactoring capability schemas.
• Principal Failure Modes: Handle expiration. If an agent delays execution due to a circuit breaker or backoff, the server-side handle may expire based on TTL policies. The agent must be programmed to handle "Expired Handle" errors by gracefully requesting a new context allocation and re-hydrating the state.

3. Protocol-Level Semantic Compensations (Automated Sagas)

• Problem: When an agent successfully completes a partial workflow but encounters a non-retryable failure on a subsequent step, orphaned resources are left consuming infrastructure or budgets. Hardcoded cleanup scripts are brittle, agent-specific, and difficult to maintain.
• Agent-Visible Behavior: When a capability is registered with the MATM framework, it must provide a strictly typed contract defining both a do operation and an optional undo operation (the compensating action). As the agent progresses, a "compensation stack" is automatically pushed to the state database.
• Expected Benefit: If the agent encounters a terminal state (e.g., maximum retries exhausted on a critical dependency), the runtime automatically unwinds the compensation stack, invoking the undo operations in reverse order without requiring custom error-handling logic within the agent's prompt.
• Dependencies: Complete mapping of reverse APIs for all side-effect-producing tools registered in the ecosystem.
• Implementation Effort: High. Not all operations are easily reversible. For example, sending an email cannot be un-sent; it can only be mitigated with a follow-up apology email. Modeling these semantic equivalents requires careful domain-driven design.
• Principal Failure Modes: A compensating action itself might fail due to network timeouts. The runtime must persist the state of the compensation stack, retry failing compensations infinitely using backoff, and raise severe dead-letter alerts if manual intervention is eventually required to clear the stack.

Practical Adoption Sequence, Unresolved Questions, and Success Criteria

Implementing these capabilities across an autonomous system should follow a phased adoption sequence to minimize disruption while immediately elevating reliability baselines.
Practical Adoption Sequence:

1. Idempotency and Retries (Immediate): Mandate the implementation of the Idempotency-Key header on all non-idempotent tool APIs. Implement exponential backoff, jitter, and standard timeout configurations in the base agent network client.
2. Durable Execution and Outbox (Short-term): Transition agent state tracking to a durable workflow database capable of persisting execution DAGs. Implement the Transactional Outbox pattern for publishing asynchronous events, decoupling agent state transitions from network queue availability.
3. Circuit Breakers and Compensations (Medium-term): Introduce API gateway or sidecar-level circuit breakers to protect downstream services from thundering herds. Implement the automated Saga compensation stack to handle complex orchestration failures gracefully.
4. Fencing and Sessionless Handles (Long-term): Refactor the entire authorization architecture to support object-capabilities, sessionless MCP handles for context delegation, and distributed fencing tokens for strict concurrency control.

Unresolved Questions:

• Cost vs. Value of Compensations: How should an autonomous agent programmatically evaluate the financial or computational cost of executing a deep, complex compensation stack versus explicitly choosing to abandon a minor orphaned resource?
• Security of Opaque Handles: As sessionless handles (via MCP) replace connection-bound state, how can systems efficiently validate authorization contexts for bearer tokens passed between decoupled multi-agent networks without introducing crippling cryptographic verification latency?

Measurable Success Criteria:
For future implementers, success must not be judged subjectively. It must be quantified through rigorous telemetry against the following metrics:

1. Unattended Recovery Rate: The percentage of network timeouts and 5xx transient errors successfully resolved via automatic retries and idempotency replays without triggering a dead-letter state. Target criteria: >99.9%.
2. Zombie Mutation Rejection Rate: Using simulated split-brain fault injection in a staging environment, measure the percentage of stale writes successfully rejected by external services using fencing tokens. Target criteria: 100%.
3. Mean Time to Compensation (MTTC): The average duration measured between a non-retryable workflow failure event and the successful completion of the corresponding Saga compensation stack. Target criteria: <60 seconds.
4. Orphaned Resource Count: A steady-state metric tracking external resources (e.g., database rows, compute instances, file allocations) created by agents but unreferenced by any actively running or successfully completed workflow. Target criteria: 0.

By adhering to these architectural paradigms—acknowledging the mathematical reality of the Two Generals problem, relying on end-to-end idempotency, durably persisting state, and treating external services as strict authorization boundaries—MATM systems can achieve the resilience required to operate autonomously and safely in production environments.

Works cited

1. Two Generals' Problem - Wikipedia, https://en.wikipedia.org/wiki/Two_Generals%27_Problem
2. Chapter 2: Models of Distributed Systems | codedump notes, https://www.codedump.info/dist-system-en/systemmodel/
3. Two Generals' Problem, https://en-academic.com/dic.nsf/enwiki/1961913
4. The Two Generals Problem | hi, https://lyletagawa.com/posts/two-generals-problem/
5. The Two Generals' Problem: Why Perfectly Reliable Message, https://www.javacodegeeks.com/2026/07/the-two-generals-problem-why-perfectly-reliable-message-delivery-is-mathematically-impossible.html
6. [PDF] End-to-end arguments in system design - Semantic Scholar, https://www.semanticscholar.org/paper/End-to-end-arguments-in-system-design-Saltzer-Reed/2ba2467d829b6e7f410fdb457395e95da03e075d
7. END-TO-END ARGUMENTS IN SYSTEM DESIGN - MIT, https://web.mit.edu/saltzer/www/publications/endtoend/endtoend.pdf
8. End-to-End Arguments in System Design (1984) - Sparse Notes, https://sparsenotes.com/posts/2026/06/papers/2026-06-13-end-to-end-arguments/
9. Systems End-to-end argument and Design Hints, https://www.cs.cornell.edu/courses/cs6410/2024fa/schedule/slides/03-design-hints.pdf
10. Life Beyond Distributed Transactions - ACM Queue, https://queue.acm.org/detail.cfm?id=3025012
11. (PDF) Life beyond Distributed Transactions: an Apostate's Opinion., https://www.researchgate.net/publication/220988217_Life_beyond_Distributed_Transactions_an_Apostate's_Opinion
12. Scalable OLTP in the Cloud: What's the BIG DEAL? - CIDR, https://www.cidrdb.org/cidr2024/papers/p63-helland.pdf
13. FIPA ACL Message Structure Specification - Jose M. Vidal, https://jmvidal.cse.sc.edu/lib/fipa_acl_messag_struc_specif.html
14. (PDF) Specifying Interaction Space Components in a FIPA-ACL, https://www.researchgate.net/publication/221431207_Specifying_Interaction_Space_Components_in_a_FIPA-ACL_Interaction_Framework
15. Ontology definition languages for Multi-Agent Systems - CEUR-WS.org, https://ceur-ws.org/Vol-66/oas02-10.pdf
16. The definitive guide to Durable Execution - Temporal, https://temporal.io/blog/what-is-durable-execution
17. Demystifying Determinism in Durable Execution - Jack Vanlightly, https://jack-vanlightly.com/blog/2025/11/24/demystifying-determinism-in-durable-execution
18. Durable Execution for AI Agent Runtimes: Checkpointing, Replay, https://zylos.ai/research/2026-04-24-durable-execution-agent-runtimes/
19. Durable Execution in LangGraph: Agents That Survive Failure and, https://vadim.blog/durable-execution-agents-that-survive-failure-and-resume-where-they-left-off/
20. DBOS: a DBMS-oriented operating system | Request PDF, https://www.researchgate.net/publication/357847212_DBOS_a_DBMS-oriented_operating_system
21. DBOS: three years later | Request PDF - ResearchGate, https://www.researchgate.net/publication/391275681_DBOS_three_years_later
22. DBOS: a DBMS-oriented operating system | Stanford MAST Lab, https://mast.stanford.edu/pubs/dbos_a_proposal_for_a_data_centric_operating_system/
23. Transactional Outbox Pattern & Change Data Capture (CDC) with, https://bknexoratech.in/blog/transactional-outbox-pattern-cdc-debezium-postgres-2026
24. Debezium Outbox Pattern in Spring Boot: Worked Example, https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices
25. Transactional Outbox Pattern for Event-Driven Microservices - Medium, https://medium.com/ifs-tech/transactional-outbox-pattern-for-event-driven-microservices-c355328b5c2e
26. Implementing the Transactional Outbox Pattern in Spring Boot, https://springboot-123.mizucoffee.com/en/blog/spring-boot-outbox-pattern-transactional-messaging/
27. CDC for Microservices: Event-Driven Architectures - Conduktor, https://www.conduktor.io/glossary/cdc-for-microservices-event-driven-architectures
28. API Facade Over Legacy Database: Bi-Directional Writes, https://www.nextgencodingcompany.com/insights/building-api-facade-on-top-of-legacy-application-database
29. (PDF) The Saga Pattern in a Reactive Microservices Environment, https://www.researchgate.net/publication/335069774_The_Saga_Pattern_in_a_Reactive_Microservices_Environment
30. sagas - cs.Princeton, https://www.cs.princeton.edu/research/techreps/598
31. Saga Pattern Technologies: A Criteria-based Evaluation - SciTePress, https://www.scitepress.org/Papers/2022/109994/109994.pdf
32. Software Robustness and Timeout Retry Backoff Paradigms, https://buildsoftwaresystems.com/post/fault-tolerant-software-timeout-backoff-guide/
33. Idempotency & Retries · HotShard, https://hotshard.com/idempotency
34. Simpler Backoff - Hacker News, https://news.ycombinator.com/item?id=44141887
35. Temporal Error Handling: Application Errors, Non-Retryable Failures, https://hosseinnejati.medium.com/temporal-error-handling-application-errors-non-retryable-failures-and-compensation-97b54d731ce5
36. Temporal Failures reference, https://docs.temporal.io/references/failures
37. Microservices Patterns : Circuit Breaker Pattern - Medium, https://medium.com/@abhi.strike/microservices-patterns-circuit-breaker-pattern-4af499b112cb
38. Circuit Breaker Pattern in NestJS: Building Resilient Microservices, https://www.matthewswong.com/en/blog/circuit-breaker-nestjs-pattern/
39. Case Study - Campion Circuit Breaker, https://campion-breaker.github.io/case-study.html
40. Circuit Breaker Pattern (Design Patterns for Microservices) - Medium, https://medium.com/geekculture/design-patterns-for-microservices-circuit-breaker-pattern-276249ffab33
41. Microservice Resilience with the Circuit Breaker Pattern, https://convincingbits.wordpress.com/2019/11/13/microservice-resilience-with-spring-cloud-netflix-hystrix/
42. draft-idempotency-header-01 - IETF Datatracker, https://datatracker.ietf.org/doc/html/draft-idempotency-header-01
43. Idempotency Keys Explained: Safe API Retries in 2026, https://www.alekseialeinikov.com/en/blog/topics/architecture/idempotency-in-practice-api-retries-2026
44. Idempotency-Key - Expert Guide to HTTP headers, https://http.dev/idempotency-key
45. The Idempotency-Key HTTP Header Field - IETF, https://www.ietf.org/archive/id/draft-ietf-httpapi-idempotency-key-header-01.html
46. The Idempotency-Key HTTP Header Field - IETF Datatracker, https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
47. Idempotency Middleware for Node.js, Bun, and Deno - roderick.dk, https://roderick.dk/posts/2026-04-06-announcing-idempot-js/
48. How to do distributed locking - Martin Kleppmann, https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
49. Distributed Locking, https://sd.dosibridge.com/concepts/distributed/distributed-locking/
50. README.md.tpl - bsm/redislock - GitHub, https://github.com/bsm/redislock/blob/main/README.md.tpl
51. Autonomous AI Breaks Four Assumptions at the Core of Cybersecurity, https://zvelo.com/autonomous-ai-breaks-four-assumptions-at-the-core-of-cybersecurity/
52. Why do autonomous agents increase the need for explicit, https://nhimg.org/faq/why-do-autonomous-agents-increase-the-need-for-explicit-authorization-boundaries/
53. Object-capability model - Wikipedia, https://en.wikipedia.org/wiki/Object-capability_model
54. AI Agent Architecture: The Trust Boundary Model - AakashX, https://www.aakashx.com/blog/agent-trust-boundary-model-ai-agent-architecture/
55. The Agentic AI Security Scoping Matrix: A framework for ... - AWS, https://aws.amazon.com/blogs/security/the-agentic-ai-security-scoping-matrix-a-framework-for-securing-autonomous-ai-systems/
56. Designing a Safe Authorization Architecture for AI Agents | by Binoy, https://medium.com/@Binoykumarbalan/designing-a-safe-authorization-architecture-for-ai-agents-7ea2c90c37b0
57. Model Context Protocol (MCP) explained: A practical ... - CodiLime, https://codilime.com/blog/model-context-protocol-explained/
58. SEP-2567: Sessionless MCP via Explicit State Handles, https://modelcontextprotocol.io/seps/2567-sessionless-mcp
59. Beyond The Single-Agent Ceiling: Scale Out With MCP Agent Teams, https://dev.to/aws-heroes/beyond-the-single-agent-ceiling-scale-out-with-mcp-agent-teams-80b