A timeout is an unknown outcome
A support API accepts a new ticket, but its response never reaches the agent. The agent retries and creates a second ticket. Nothing about this failure requires a hallucination: the application confused “I did not get a response” with “nothing happened.” Design the retry around that uncertainty.
Separate the logical action from the individual request attempt. A logical action might be “create the ticket for inbound message 42.” Every transport retry for that action should share its identity. The identifier must be stable across restarts, not recreated whenever a model decides to call the tool again.
Classify before retrying
A temporary network failure, an invalid customer identifier and a permission denial are different conditions. Retrying all three with the same policy wastes resources and may make an operational problem harder to see. Define an error taxonomy at the integration layer and return a clear result category to the workflow.
AWS describes bounded retries with backoff and notes that operations should be idempotent. Use that as a design pattern, not as a substitute for the destination’s actual semantics. A provider may offer idempotency only for selected methods, for a limited retention period, or with restrictions on changed parameters. Read and test the exact endpoint contract.
| Result | Suggested handling | Evidence to retain |
|---|---|---|
| Validation error | Correct input or ask for clarification | Invalid field and expected schema |
| Permission denied | Stop and escalate | Policy decision, without secrets |
| Temporary capacity error | Bounded delayed retry | Attempt count and provider response |
| Write outcome unknown | Reconcile destination before another write | Logical action ID and external reference |
Persist the action before executing
Store a record containing the action identifier, intended destination, normalized arguments and current state. Useful states include proposed, authorized, executing, completed and uncertain. The executor should atomically claim work so two workers do not independently send the same operation. The model’s reasoning transcript is not a reliable queue or durable state store.
If the provider supports an idempotency key, pass the same key for retries of the same operation. Stripe documents this pattern for its API, but another service may behave differently. If arguments change, treat the change as a new proposal requiring its own authorization; do not silently reuse an old approval.
Use a bounded attempt budget
Set maximum attempts and an overall deadline. Backoff should avoid a tight loop that repeatedly hits a failing dependency. An operator needs to know when the automatic budget is exhausted and where to investigate. The workflow should make that transition explicit rather than indefinitely displaying “working.”
In a hypothetical example, 100 logical tasks with 20 additional attempts produce 120 transport attempts. Billing and tool error metrics should count attempts, while completion metrics should count logical outcomes. These denominators cannot be swapped without changing the meaning of the result. Use the cost calculator to model additional attempts separately.
Test recovery as a state transition
Simulate a successful destination write followed by a lost response. Confirm that a resumed worker finds the existing result instead of creating another. Repeat the test with two workers racing to claim one action, an expired approval and an unavailable reconciliation endpoint. Record the final destination state for every case.
When safe reconciliation is impossible, the correct outcome may be an operator queue. Automation should expose uncertainty rather than hide it behind repeated requests. Connect these cases to your evaluation suite and document the recovery owner in the workflow planner.
Implementation map
Sources and limits
AWS retry pattern; Stripe idempotency semantics. These are source examples, not a claim that all APIs implement identical guarantees. Documentation checked October 14, 2026. Recommendations and worked examples are editorial methods, not claims of a completed client engagement.
Bring the problem. We’ll scope the next step.
Something in your workflow keeps breaking, or your team is doing the same manual work every week.
Send the current setup, what failed, and the result you need. I can help with software integrations, local AI infrastructure, agent permissions, failure recovery and deployment.
Useful first details: system or runtime, expected workload, data restrictions and one redacted example. Don’t send credentials or private customer data.Fazal Shah builds AI systems, automation and software products. Based in Pakistan, working with a global audience. About Fazal →