Engineering · Agent infrastructure ·

What It Takes to Run an AI Agent for Hours

How AI Drive uses Temporal and persistent streaming to keep multi-step agent workflows running across browser sessions and worker restarts.

A single request in AI Drive can send an agent searching across files, running Python on what it finds, or delegating part of the work to another agent. Each result can change what the model does next. How does that work continue when the person who started it closes their laptop?

Take a contract renewal report. The agent has compared two contracts when you close the tab. It keeps working. If a deployment interrupts the worker while you’re away, another worker can resume the same turn from the last saved state of the conversation. When you reopen the chat an hour later, you can see what the agent finished without submitting the job again.

Give the turn its own lifetime

A turn starts with one user request and spans the model calls and tool work it sets in motion. A model call might take seconds to choose the next action. A document job can keep the turn waiting much longer. Tying the turn to the initiating HTTP request would let a network timeout end the whole chain.

At admission, the API marks the turn as the chat’s active turn and records itself as the owner. It saves the launch payload in Cloud Storage and submits a workflow to Temporal. Once Temporal accepts the workflow, the API can acknowledge the launch. The browser follows the turn’s persistent stream, a numbered log of display updates. Within the signed-in user’s own chat, it finds that stream by turn ID, which stays the same across worker attempts.

Figure 1

The three paths of a web turn

Launch, execute, and render one durable turnThe browser submits the request to the API, which saves it and launches a Temporal workflow. A worker runs the shared agent loop with models and tools. It persists display chunks to Firestore and checkpoints the transcript in Cloud Storage. Another tab signed in as the same user can subscribe to the turn stream.LAUNCHEXECUTERENDERWeb tabsend the promptAPIadmit and saveTemporalschedule an attemptAgent workeradopt the turnShared agent looprun the chat handlerModel and toolsproduce output and effectsReturning tabsubscribe by turn IDFirestorestream chunks · turn claimCloud Storagepayload · saved transcript
LaunchWeb tab → API → Temporal workflow
ExecuteTemporal → agent worker → shared model-and-tool loop
RenderWorker → Firestore chunks → returning tab. Cloud Storage retains the request and the transcript.
The agent loop writes two records. A returning browser reads numbered updates from Firestore to rebuild the display. A replacement worker reads the transcript, the saved conversation in Cloud Storage, to decide which work can continue. Each save of the transcript is a checkpoint.

The worker calls the same chat handler that serves interactive requests. It builds the tool list, checks permissions, and constructs the model prompt in both paths. Tool ordering matters here. Changing it can break the exact prefix a provider uses for prompt caching. Sharing the handler keeps those inputs aligned as tools evolve.

Run each turn as one Temporal workflow

AI Drive derives a stable workflow ID for each turn and sets its deadline at launch. Temporal retries eligible failures within that deadline. The worker keeps sending heartbeats even while it waits on a slow tool. If those heartbeats stop arriving, Temporal can time out the attempt.

The workflow receives a reference to the saved request, keeping prompts and document text out of its history. A worker loads the request from Cloud Storage and runs the agent loop as an activity, Temporal’s unit of work that it can time out and retry. In the renewal review, one model call selects a document tool. A later call uses its saved findings to draft the report.

Figure 2

One report, from tool call to returning tab

A document job continues while the browser is closedThe worker calls the model, runs document analysis, and calls the model again with the saved findings. The user’s tab disconnects during the tool call. When it reconnects, it reads the same turn’s saved progress.WORKERBROWSERModel callChoose the document toolDocument analysisSave the tool resultNext model callUse the findings in the reportClose the tabNo reader attachedReopen and replay the turn
  1. StartThe model selects a document tool for the renewal report.
  2. Close the tabThe worker continues the analysis and saves the tool result.
  3. Return to the conversationThe model can use the findings. The returning browser reads progress for the same turn.
What if a deployment replaces the worker during this turn?How an active turn moves to a new workerThe first worker sends heartbeats. A shutdown starts a bounded drain. A turn that finishes in time completes normally. Otherwise the old attempt gets bounded cleanup, and an eligible retry adopts the same turn and loads the latest acknowledged checkpoint.Shutdown signalAttempt 1 is runningSend periodic heartbeatsWorker begins drainingAllow active work to finishIf the turn finishesComplete on the original workerStill running at drain expiryBounded cleanupTry to save the latest progressAttempt 2 takes ownershipRead the saved checkpoint
  1. RunningAttempt 1 sends periodic heartbeats.
  2. Shutdown signalThe worker stops accepting new activities and gives active work time to finish.
  3. If the work finishesThe turn completes on the original worker.
  4. If the work is still runningThe old attempt gets bounded cleanup. An eligible replacement claims the same turn and resumes from the latest acknowledged checkpoint.

An orderly rollout gives the activity a chance to finish. If another attempt is needed, it takes ownership of the same turn and reads the latest acknowledged checkpoint.

The gap on the browser timeline spans a tool call. By the time the tab replays that call, the model may already have used its result to draft the report.

During a rolling deployment, Kubernetes gives the worker pod time to finish or checkpoint the turn. An out-of-memory (OOM) kill can end the worker halfway through a tool call, before it can save anything else. Once its heartbeat times out, Temporal can schedule a replacement. The replacement resumes from the last acknowledged checkpoint. Anything the old worker held only in memory is lost.

Starting the workflow has an edge case of its own. A timeout during launch leaves the API with a decision. Temporal may already have accepted the workflow. Falling back to the API process at that point could execute the same tool calls twice.

Temporal rejects a second start with the same workflow ID. If the launch reply is lost, the API looks up the workflow. An empty lookup can still race with a delayed start. While the outcome is uncertain, only the Temporal workflow is allowed to execute that turn.

A long review can outlive the credential used to read the first contract. The turn’s access is re-established as needed, and every later read still goes through the user’s access checks.

Let a returning tab catch up without losing its place

Imagine reopening the renewal report just after the agent has compared two contracts. The transcript already shows both results. The stream listener, meanwhile, starts with the agent’s opening words. Replacing the screen with the first replayed chunk would make both completed comparisons disappear. The browser keeps both results visible while replay catches up.

The browser builds the replay off-screen using the same code that displays live events. Before replacing the visible draft, it waits for a snapshot from the server and checks that the rebuilt view includes the prompt, text, tool calls, results, and approvals the user can already see. An older snapshot stays off-screen until it covers that work. The same rule preserves an approval you granted in another tab.

The writer batches numbered chunks in Firestore so each token can stream without waiting for its own storage write. If the buffer fills, the producer waits. On the other side, the reader may see chunks in the order 1, 3, 2, 2. It can display 1 immediately, hold 3 until 2 arrives, and discard the duplicate.

Figure 3

Four deliveries, three fragments

Arrived—
Buffered—
Visible draft—

Start with an empty draft. The reader is waiting for sequence 1.

With arrivals 1, 3, 2, 2, the reader buffers 3, applies 2 and 3 in order, and ignores the duplicate. If 2 never arrives, bounded waiting allows the incomplete draft to advance. At completion the saved transcript repairs it.

Each number identifies one illustrative text fragment. The browser can receive several deliveries for a single sequence. Only the sequence of applied content should determine what the reader sees.

A snapshot from the local cache can be incomplete. Once a server snapshot confirms a gap, the reader waits a bounded time for the missing chunk. If the wait expires, it lets later output through and makes sure later pieces of a tool call cannot attach to the wrong tool card. At the end of the turn it repairs the draft from the transcript. The user can keep following the analysis during the gap.

During a large replay the reader combines adjacent text fragments into a single display update and yields between batches. Tool calls and approvals keep their positions.

Two checks on the browser replay

TestConditionObserved result
Browser replayFeed the reader 50,000 generated text fragments.The reconstructed text matched the input exactly. Batching took 196 updates to the chat’s display state in this synthetic run.
Switching turnsSwitch to a new turn during replay, then deliver a heartbeat through the old turn’s listener.The reader unsubscribed from the old turn and started the new one at the beginning. The late heartbeat triggered no chat update.

Protect the checkpoint when ownership changes

A worker can lose ownership while a storage write is still in flight. Suppose it checks its claim and prepares a checkpoint. Before the write reaches storage, a replacement takes over and saves newer work. What keeps the delayed write from overwriting it?

Each attempt gets an owner stamp. A replacement adopts the turn and advances the transcript’s storage version. Suppose the old worker had already passed the ownership check and prepared a write against version 7. Once storage holds version 8, that delayed write fails its precondition at commit time. The turn keeps its identity and its most recent checkpoint.

Figure 4

How a version check protects the new checkpoint

How versioned storage preserves the active checkpointThe original worker reads version 7. The replacement takes ownership and advances the transcript to version 8. Cloud Storage rejects the late write that requires version 7.Original workerattempt AReplacement workerattempt BCloud Storagetranscript versionsReads transcript at v7 while it owns the turnWith the new claim, advances v7 to v8Late checkpoint arrives with “write only if v7”Rejected · the stored version is already v8
  1. Original workerReads transcript v7 and prepares a conditional write.
  2. Replacement workerTakes ownership and advances the saved transcript to v8.
  3. Original workerSends the old checkpoint with “write only if v7.”
  4. Cloud StorageRejects the stale write. Version 8 stays intact.

Follow the same turn through a change of owner.

The version numbers are illustrative. Advancing the stored version closes the gap between the old worker’s ownership check and its eventual write.

Tool calls get a checkpoint of their own. The durable path waits for acknowledgement of the intended action before dispatching a tool it classifies as capable of changing state. When the model calls several tools at once, the worker checkpoints each completed result without waiting for slower tools. A replacement worker resumes from the last acknowledged checkpoint. For supported document jobs, a launch receipt gives the next attempt a handle to the original work.

Suppose the network times out while a downstream tool is writing the shared report. The write may have succeeded even though the reply never arrived. A replacement can use a saved launch receipt to rejoin a supported document job. If it finds the job already complete, it returns the findings and leaves publication unconfirmed. If it still needs to publish, it checks access to the original destination and the expected file revision before writing.

Two preview runs on a shared report exercised both outcomes. In the first, the worker running the agent was replaced while a document job it had launched was still running. The replacement rejoined the job and recovered both file results, and the report gained one new revision. In the second, the report was edited after the job started. The replacement recovered the same results, the publication check detected the newer revision and refused the write, and the edit stayed intact.

When should a turn get another attempt?

Inside the worker, a deployment and a click on Stop both arrive as a cancellation. They express different intentions. Stop records the user’s decision on the turn, so even a later attempt sees that execution should end. A worker that runs out of time to drain during a deployment reports an infrastructure interruption. Temporal can then schedule another attempt against the same transcript. Closing a browser tab simply detaches a reader.

A model provider might rate-limit a call with HTTP 429, or the connection might time out while the worker waits for a reply. If provider retries and any configured fallback still leave a transient failure, Temporal can resume the turn from its checkpoint. The turn stops if consecutive failures at that checkpoint exhaust its retry allowance. Saving a new tool result resets the count. A heartbeat leaves it alone.

A context window overflow needs a different response. Suppose the contract comparisons have filled the model’s input window. If the provider rejects the input before returning any content, the agent loop can shorten older material while keeping the current task and the latest tool results together. It retries that model call once within the same worker attempt, using the saved comparisons without running those tools again. The full transcript stays intact. A second rejection ends that recovery path.

An unacknowledged checkpoint can also trigger another attempt. Losing the claim ends that worker’s authority to write. Revoked access or reaching the turn’s credit budget ends further work. Every attempt inherits the deadline set at launch, so queue time and retry delays count against the same clock.

Scaling workers that spend time waiting

An agent can spend minutes waiting for a tool or a model provider while its worker uses very little CPU. Its task has already left the queue. Both signals can make the worker look idle while the turn is still in progress.

Durable agent workers run on regular VMs in Google Kubernetes Engine (GKE), with a dedicated Temporal queue for agent work. KEDA supplies queue backlog and a signal that a turn is still running to the Kubernetes autoscaler, alongside CPU and memory. Backlog helps it add capacity for waiting turns. The running-turn signal helps it keep capacity while turns are in progress. The trade-off is that a worker can stay provisioned while it waits on a tool. Once demand subsides, the pool scales down gradually.

What the runtime guarantees

  1. A reader can rejoin the same turn. Replaying a numbered stream rebuilds the display. Opening a finished tool card starts no new document work.
  2. Only the current owner can finalize the turn. The ownership claim identifies that attempt. A version check on the transcript rejects a late write prepared against an older checkpoint.
  3. The worker records intent before dispatching a tool it classifies as state-changing. It waits for that checkpoint to be acknowledged before starting the action. A saved receipt can reconnect an interrupted job. If the outcome is still unknown, recovery directs the agent to inspect the existing output or job status before deciding whether to try again.

Back to the renewal report. You closed the tab two contracts in, and a deployment moved the turn to another worker while you were away. When you reopen the chat, the browser replays the same turn and the findings are in the conversation. A shared report was written only against the revision the agent expected, or not at all. For a document agent, the result lives in two places, the chat and the file, and recovery has to hold in both.

About the author. Sandeep Khanna V P is part of the AI Drive engineering team and built durable agent turns, from workflow launch to replay in the browser.