Resume After Tool Failure in Agentic AI: Durable Execution Semantics (and OpenTelemetry GenAI Instrumentation)

An agent that can’t resume after tool failure is not an agent runtime; it’s a demo script with good marketing.

Most production “agent failures” we see are not LLM reasoning problems. They’re execution problems: a tool call times out after it already mutated state, a retry double-charges, a worker restarts mid-step with no durable cursor, or two copies of the same run compete and interleave actions. Prompts won’t fix that. Semantics will.

The missing layer: execution semantics, not more prompting

When you build an agent loop, you’re implicitly building a workflow engine. The difference is that workflow engines come with vocabulary and contracts: attempt numbers, deterministic replay, idempotency, leases, compensation, and clear “what happens on crash” behaviour.

In most agent stacks, those semantics are either absent, left to application code, or hidden in framework internals you can’t easily inspect. The result is a specific class of failure mode:

• The LLM decides “call tool X”.
• The tool call partially succeeds (or succeeds but the response is lost).
• The agent retries.
• You get duplicated side effects, inconsistent internal state, and no reliable audit trail.

Durability is what turns “best effort” into “safe enough to run unsupervised”. Observability is what turns “safe enough” into something your team can operate.

OpenTelemetry has started to meet teams where they actually are: not just single LLM calls, but agent and tool spans. The GenAI semantic conventions are now maintained in a dedicated repository, and include explicit agent/framework spans such as invoke_agent, plan, and execute_tool. (github.com)

What “resume after tool failure” needs to mean

If you want “resume” to be more than “re-run the whole thing and hope”, you need to define the durable unit of work and what counts as committed.

A pragmatic target is:

• Every tool invocation is either:
• committed exactly once (idempotent, or effectively-once via dedupe), or
• safe to compensate, or
• explicitly flagged as non-durable (and therefore not resumable).
• Every agent run has a durable cursor so you can continue from a known step boundary.
• Replays are intentional (for debugging or reconstruction), and do not cause side effects.

That breaks down into four semantics you can implement without pretending you’ve solved distributed systems forever.

1) Idempotency keys: make retries safe

Idempotency is the cheapest durability win you can buy.

For any tool that can mutate state (create ticket, send email, charge card, update record), the agent runtime should include an idempotency key on the call. The tool service must dedupe by that key and return the original result if it’s seen before.

Key design rules:

• The key must be stable for “the same logical intent”.
• The key must be unique across unrelated intents.
• Dedupe must outlive process restarts.

In agent terms, a good default is:

• run_id + step_id + tool_name + intent_hash

Where intent_hash is a hash of a canonicalised request payload (not raw JSON; ordering matters).

When not to do it:

• If the tool cannot meaningfully dedupe (e.g., “get current time”, “fetch latest stock price”), you still want a call ID for correlation, but idempotency as “return the same result” is semantically wrong. Prefer caching semantics, not idempotency.

What to instrument:

• An explicit “idempotency present?” attribute.
• The idempotency key (often as a hashed/opaque value) for joining events across retries.
• A “deduped” flag from the tool service.

OpenTelemetry GenAI conventions already give you a place to model tool execution as a first-class span (execute_tool). (github.com) Your durability attributes can sit alongside the GenAI attributes on that span.

2) Step leases: stop duplicate workers from interleaving

Once you run agents on queues, cron, or autoscaled workers, you will eventually have two workers trying to progress the same run:

• Worker A starts step 12 and stalls.
• The queue redelivers.
• Worker B starts step 12.
• Both call the tool.

Idempotency prevents double side-effects, but you still get wasted work and confusing traces unless you also define a concurrency contract.

A step lease is a time-bound claim on “I am the worker allowed to execute this step”. Implementation is usually:

• A durable store row keyed by (run_id, step_id).
• Atomically set lease_owner, lease_expires_at if empty/expired.
• Refresh heartbeats for long-running steps.

Trade-offs:

• Leases add write load and operational complexity.
• Without them, you’ll end up debugging “ghost retries” and partial execution forever.

When not to do it:

• If your entire agent run is short-lived and single-process (no queue, no horizontal scale), leases are overhead you may not need. But the minute you introduce redelivery, you’re in lease territory.

What to instrument:

• Lease acquisition as a child span under execute_tool (or under the step span if you model steps explicitly).
• Attributes:
• durable.step.id
• durable.lease.owner
• durable.lease.ttl_ms
• durable.lease.acquired (bool)
• durable.lease.conflict (bool)

This is where OTel’s standard trace model helps: a span is a unit of time-bound work; the lease is literally the contract for who is allowed to do that work. (opentelemetry.io)

3) Replay vs retry: they’re different operations

Most agent stacks blur “retry the tool call” and “replay the step” and “re-run the whole run”. Those are not interchangeable.

Define these explicitly:

• Retry: same step, same idempotency key, expected to be side-effect-free at the system boundary.
• Replay: reconstruct state from the event log/step log for debugging or audit; must not call side-effecting tools.
• Rerun: new run ID, new idempotency keys; side effects may happen again.

If you don’t separate these, you’ll either:

• accidentally cause side effects during “debug replay”, or
• make safe retries impossible because you treat every attempt as a new intent.

What to instrument:

• Attempt numbers.
• A “mode” flag (retry, replay, rerun).
• Parent-child relationships that make retries visible: the clean pattern is one execute_tool span per attempt, all linked to a stable step ID.

OpenTelemetry GenAI agent spans include invoke_agent and plan, and describe how tool/task spans sit under the same invoke_agent span. (github.com) Use that structure, then layer your execution mode and attempt metadata on top.

4) Compensation: admit you can’t have exactly-once

Some tools cannot be made idempotent in a meaningful way (or not quickly), and you still need to ship.

That’s where compensation comes in: if step N succeeds and step N+1 fails, you run a compensating action to unwind the effects of N.

Examples:

• “Create order” compensated by “cancel order”.
• “Provision resource” compensated by “deprovision resource”.
• “Send email” has no real compensation; you can only send a follow-up correction.

This is the line you must draw with founders and product teams: if a step has no sensible idempotency or compensation story, then “resume” is an illusion. You can still automate it, but you should treat it as a supervised workflow.

What to instrument:

• Compensation as its own span, linked to the original step/tool span.
• Attributes:
• durable.compensation.for_step_id
• durable.compensation.reason
• durable.compensation.outcome

And be honest in your dashboards: “compensated” is not “successful”. It’s “we cleaned up”.

How to map durability to OpenTelemetry GenAI spans

OTel GenAI conventions give you a standard vocabulary for what the agent is doing: planning (plan), invocation (invoke_agent), and tool execution (execute_tool). (github.com) The missing part is to make “durable execution” observable without inventing a parallel tracing universe.

A practical span model (minimal but effective):

1. Root span: your user request / job execution.
2. Child span: invoke_agent (GenAI convention).
3. Child spans under invoke_agent: - plan (only when you can reliably distinguish planning, per the spec). (github.com) - One logical “step” span per durable boundary (you define this).
4. Under each step span: - One execute_tool span per attempt. - Optional “lease acquire/refresh” spans. - Optional “compensation” spans.

Key attributes to standardise internally (names are illustrative; keep them consistent):

• durable.run.id
• durable.step.id
• durable.step.seq
• durable.attempt
• durable.idempotency.key_hash
• durable.execution.mode (normal|retry|replay|rerun)
• durable.commit.state (prepared|committed|compensated|abandoned)

If you’re also exposing tools over MCP, you’ll care about trace context crossing that boundary. The OpenTelemetry demo environment has an MCP service explicitly instrumented for inbound MCP tool calls, which is a useful reference pattern for “agent calls tool over MCP and still stays in one trace”. (opentelemetry.netlify.app)

The observability checklist that actually catches the bugs

If you only capture “LLM input/output” you will miss the failure mode. You need to see the durable cursor and the side-effect boundaries.

Instrument these, or accept you’ll debug from anecdotes:

• For every tool call:
• request/response integrity (did we get a response body; did we time out)
• whether the tool mutated state
• idempotency key present + dedupe outcome
• attempt number and backoff reason
• For every step:
• lease acquisition outcome + owner
• step start/commit timestamps
• step output persisted (yes/no)
• For every run:
• current durable cursor (latest committed step)
• termination reason (completed|failed|abandoned|human_required)

And one operational rule that saves teams months: make it possible to answer, from a single trace, “did we charge the customer once, twice, or zero times?” If your telemetry can’t answer that, your runtime isn’t durable.

Build vs buy: when to stop and choose a workflow engine

You should not build a durable agent runtime from scratch if:

• you need strict auditability (regulated workflows)
• you have high-cost side effects (payments, provisioning, external comms)
• you can’t tolerate duplicated actions

In those cases, put the agent inside a deterministic workflow engine and treat the LLM as a planner/selector, not the executor. The agent proposes actions; the workflow engine executes them with known semantics.

You might build your own if:

• the domain tools are mostly read-only
• side effects are reversible or low-stakes
• you can ship with “supervised automation” while you harden semantics

Where Codeversols fits (briefly, honestly)

At Codeversols we tend to be brought in when a team has proved “agent + tools” creates value, and then hits the reliability wall: retries, partial progress, and unclear ownership between services. The work is rarely exotic model engineering; it’s system design, careful contracts around side effects, and OpenTelemetry instrumentation that makes failures diagnosable across agent, tool, and downstream services.

Close

“Resume after tool failure” is not a feature you add at the end. It’s a set of durability semantics you decide upfront: idempotency, leases, replay boundaries, and compensation. OpenTelemetry’s GenAI agent/tool spans give you a shared language for observing agent behaviour, but you still have to make execution state explicit.

If you do, agents stop being mysterious. They become debuggable distributed systems with an LLM in the loop — which is the only version that belongs in production.