Skip to main content

Drain the queue

A todo is switchboard's unit of work — a durable work-item derived from an event. Todos are how agents actually do things: an agent claims a todo under a lease, does the work, and acks it. Nothing is "read once and lost."

The lifecycle​

  • pending — created and unclaimed; visible to consumers on the owning endpoint whose scope covers its queue.
  • claimed — a consumer holds a lease (a visibility timeout: owner + expiry), 300 seconds by default and up to 24 hours via lease_ttl_seconds. The todo goes invisible to every other consumer for the window. If the window lapses before completion — the worker crashed, hung, or dropped off — the todo pops back to pending and is re-claimable. This is the crash-safety guarantee (the same model as SQS visibility timeouts).
  • done — the consumer acked completion. Terminal; retained as an audit record.
  • failed — the consumer reported failure. It is retried automatically: back to pending after a backoff that starts at 30 seconds and doubles to a 15-minute cap.
  • dead letter — each claim counts one attempt, and a todo allows 5. When the last attempt fails or its lease lapses, the todo stays failed with no further retry. A human re-queues it, with a fresh attempt budget, using Retry now in the Todos view.

How an agent drains​

Over its vended endpoint, an agent runs a simple loop:

  1. list_todos — see what's pending on its granted queues. Pass queue, state, and a limit of 200 or fewer: every todo carries its full payload.

  2. claim — take one under a lease. Claiming is atomic: no two consumers get the same todo at once.

    Or, when several workers share the endpoint, claim_next — no id, no listing: it scans the granted queues oldest-first and atomically hands back one available todo. Concurrent callers each receive a different one, so a pool needs no coordination. An empty queue answers empty: true rather than an error, because a worker polling and finding nothing is the steady state.

  3. Do the work. If it's slow, heartbeat to extend the lease so it doesn't lapse mid-flight.

  4. complete on success, or fail on error — both take an optional result recording what happened.

Because a crash between claim and complete leaves the todo re-claimable once the lease lapses, delivery is at-least-once — so handlers must be idempotent.

Two properties worth knowing​

  • Idempotency keys collapse duplicates. Every todo carries an idempotency key — for webhooks, the webhook plus the delivery's id (X-GitHub-Delivery, X-Gitea-Delivery, Cairn's signed event_id, or a hash of the body when there is none). A delivery whose key matches a todo that is still pending, claimed, or waiting to retry returns that todo instead of creating another, so at-least-once webhook deliveries fold into a single work-item. Once the todo is done (or dead-lettered), the same key creates a new one.
  • Queues are per endpoint. A todo belongs to exactly one endpoint. Two endpoints that both have an inbox see different todos.

Running several workers on one endpoint​

An endpoint is vended to one agent, so several sessions on it are that agent running as competing consumers. Point each instance at the same endpoint URL and credential; each calls claim_next and gets distinct work. Nothing else needs configuring — the store's scan holds FOR UPDATE SKIP LOCKED, so the pool cannot double-claim.

This is load-sharing, and it is not the same as fan-out. Fan-out (add_webhook_route) delivers one event to several endpoints as several todos, so every one of them acts — that is what you want for different agents with different jobs. Competing consumers share one todo, so exactly one acts — that is what you want for capacity. Routing a webhook to two endpoints owned by the same agent duplicates work rather than sharing it.

Push: a doorbell, not the ledger​

Pulling with list_todos is always correct on its own. Where a client supports it, switchboard also pushes a notification the moment a todo is ready for a channel-attached consumer — over the same vended MCP endpoint, as a notifications/claude/channel event. Push is lossy by design: if no session is attached, the todo simply stays pending and the worker drains it on return. Unclaimed todos are rung again after 5 minutes, 20 minutes, 1 hour, and 6 hours. The durable queue is always the ledger; push is just the doorbell, so an offline agent loses nothing.

The doorbell rings one worker per todo rather than the whole pool, rotating between them, and prefers a session with an open notification stream. That keeps a pool from spending N model turns to do one todo's work. Which clients act on it is covered in Connect an agent over MCP.

Watch it live​

The operator board's Todos view shows the queue in real time over Server-Sent Events — claims, completions, retries, and the reaper returning lapsed leases — wearing the same trust badges the API and MCP surfaces carry. A human can claim, complete, fail, extend, release, and retry todos from the same view.

Deeper detail: ADR-0007 — Todos as the core primitive, ADR-0013 — Channels push delivery, and the todo-queue spec.