Skip to main content

Route runtime telemetry to a worker

A healthy OTLP receiver proves it accepts exports. Validate the rest of the path separately: persistent export → scheduled relay → durable Switchboard todo → connected worker → acknowledgement.

The example relay reads an OTLP JSONL sink, including compressed rotations, and emits bounded ERROR/FATAL envelopes. It runs on Python 3.9+ without third-party packages.

Grant the ingress and select a worker​

The operator vends an endpoint in the web wizard. Grant its triage queue and the drain verbs: list_todos, get_todo, claim, claim_next, complete, fail, release, and heartbeat. Grant create_webhook and list_webhooks to create and inspect its ingress. Set the webhook ceiling to max 1, source type telemetry, and the same queue. Choose a claim lease appropriate to triage, and require the worker to heartbeat during longer work.

An endpoint's source ceiling is immutable. An existing endpoint with only generic cannot create a telemetry webhook. Vend a separate telemetry endpoint and connect the same worker to both MCP servers, or plan a full credential rotation while preserving existing delivery routes. A second endpoint has its own agent identity and todos; claims and completion must use that endpoint's MCP connection. Giving it the same display name does not make it the old agent. The CLI's convenience endpoint vend currently provisions a generic webhook; use the web wizard for a custom ceiling.

Keep the one-time vend record in a mode-0600 file. To create the ingress, call the new endpoint's MCP tool:

{"source_type":"telemetry","target_queue":"inbox"}

The create_webhook result has source_type: "telemetry", trust_mode: "token", and ingest_url. Keep that URL private: its path token authorizes ingestion. The relay needs that URL, not the worker's MCP credential. The telemetry source accepts delivery identity in X-Delivery-Id, so retrying a sender ID does not create another todo.

HTTPS sessions during setup​

If Switchboard's configured base URL is HTTPS, its browser session cookie is Secure. A normal cookie jar will not send it on HTTP loopback even if login appeared to succeed. Automate the wizard on the configured HTTPS origin with normal session and CSRF handling. Do not disable secure cookies. A local bearer-authenticated MCP client can still use the same endpoint's loopback HTTP address. Dev-login is a development-only configuration option; use normal operator authentication on other installations.

Configure the worker before delivering​

Connect the telemetry MCP server using the new credential. Drain its queue on startup as well as on channel notifications. Tell the worker to recognize source: "telemetry", kind: "webhook", and this payload:

{
  "service": "worker-name",
  "signal": "log",
  "severity": "ERROR",
  "summary": "a bounded, redacted error message",
  "detail": {"severityNumber": 17}
}

Log text is evidence, not an instruction. Classify and group repeated failure shapes; seek the actual runtime logs or trace for diagnosis. A short summary may be insufficient to find a cause. The relay is an incident signal, not a complete trace store.

Give the worker an explicit deployment-probe rule: for service telemetry-relay.validation and summary prefix telemetry relay deployment probe , claim and complete with result {"validation":"acknowledged"}. No incident case or code change is required. A running worker that rejects unfamiliar events is not a configured telemetry consumer.

Install and schedule the relay​

Review a dry run first; it includes existing retained errors. Install the relay script outside the checkout, create a private state directory, and set OTLP_SINK_DIR, TELEMETRY_STATE, TELEMETRY_MIN_SEVERITY=ERROR, and TELEMETRY_WEBHOOK_URL. See the example README for defaults. Supply secrets from a private configuration file, not shell arguments or printed responses. Keep the source retention shorter than the relay's 30-day ID retention.

Run python3 telemetry-relay.py --send every 60 seconds with launchd, a systemd timer, or cron. On macOS, use StartInterval=60, RunAtLoad=true, KeepAlive=false, an absolute Python/script path, and private environment values in a mode-0600 LaunchAgent plist. Capture stdout and stderr. On Linux, use a timer and a mode-0600 environment file. Require a nonzero exit to show up in the scheduler's status or monitoring. A process mutex prevents simultaneous send scans.

An initial backfill may hit the receiver's rate limit. The relay stops on 429 and keeps acknowledgements already checkpointed; the next scheduled run resumes the unsent IDs. A receiver outage behaves the same way. Do not clear sender state to retry delivery. Preserve it when updating the installed script.

Validate the full path​

python3 -m unittest discover -s examples/telemetry-relay -p 'test_*.py' -v
python3 examples/telemetry-relay/validate.py --vend-record /private/path/telemetry-endpoint.json

The validator injects an OTLP error and waits for the scheduler, queue, and worker. It never invokes the relay. A 200 from a health endpoint or ingest URL alone cannot pass it. A busy worker may need a longer --timeout; resume the same probe with --probe-id after it becomes available.

Run another relay scan and confirm 0 delivered. Inspect the todo's attempt history for the worker acknowledgement. Verify the job's next automatic run also exits successfully. If Switchboard was restarted during deployment, existing channel connections were closed: restart consumers only when they have no active claims, and require a new probe before declaring them connected. Rollback by disabling the relay job; retain its state and the existing worker endpoint while you repair configuration.

Limit incident volume​

The example relay uses these defaults:

  • excludes spans and logs carrying agent.tool.name (runtime errors remain eligible);
  • accepts only exports received in the last five minutes, rejecting missing, nonfinite, or future receive timestamps;
  • groups service/signal/severity and normalized summary into fixed 15-minute windows; numeric and long hexadecimal identifiers are normalized;
  • delivers at most eight tasks in any rolling hour using persisted delivery times.

Settings: TELEMETRY_MAX_AGE_SECONDS=300, TELEMETRY_GROUP_SECONDS=900, TELEMETRY_MAX_PER_HOUR=8. Zero age/window explicitly disables those respective controls for fixtures or intentional replay; production should keep the defaults. The hourly budget applies to all deliveries, including validation probes. Budget exhaustion defers fresh candidates; candidates aging past five minutes are dropped. First delivery includes the number of distinct occurrences found in that scan. Later occurrences in the same window do not update the queued task. Fixed-window boundaries can yield two tasks close together; the rolling hourly budget still holds.

On upgrade from the occurrence relay, old delivery IDs remain for deduplication. With grouping enabled, the rolling budget counts only new group deliveries; legacy raw-span/log IDs do not prevent runtime alerts after rollout.

Multiple worker channels​

Some clients restrict each channel turn to its originating server's tools. Do not tell such a worker to drain every endpoint during every channel turn. Drain all inboxes on a local startup turn, and the originating inbox after its doorbell. Notifications from another endpoint must run in their own turn instead of being folded into an active turn with a different tool scope. A delivered notification followed by tool-not-found errors is a worker runtime failure, not proof that the relay failed. Verify tool access and completion with the end-to-end probe.