Events and monitoring

When fifty agents run on ten machines, fifty sets of logs don’t tell you who owns what. Everything agents do through runstate lands in one place, and you can read it three ways: the console for people, events for code that reacts to changes, and diagnostics for a point-in-time picture of a run.

The console

The console at app.getrunstate.com is organized by space:

Page	What it shows
Tasks	Every task with its state, filterable by state and key prefix. A task’s page shows its outcome, subscribers and each admission attempt (who took it, when, with which resources). You can cancel a task or detach a subscriber from there.
Resources	Concurrency pools, shared quotas and budgets with their current levels, plus agents that are waiting for capacity. Quotas can be paused (a cooldown) from here.
Groups	Task groups, their conditions, member verdicts and decisions.
Activity	The event journal as it happens.
Usage	Operations and retained data over time.
API keys	Create and revoke keys and choose their permissions.

Holder names appear throughout the console, so give workers stable, meaningful holder names (see Configuration & auth).

Events

Every state change is written to an ordered event journal: tasks created, admitted, succeeded and failed; quota units consumed and granted; budget reservations settled; groups finalized; runs cancelled. Each event has:

Field	Meaning
eventId	Position in the journal. Increases monotonically; use it to deduplicate.
type	For example task.created, task.admitted, task.succeeded, task.failed, task.cancelled, task.expired, task.joined, message.ready, allowance.consumed, allowance.cooldown, budget.reserved, budget.settled, budget.voided, group.finalized, scope.cancelled, timer.fired, barrier.released.
aggregateType, aggregateId	The resource the event is about, such as task and the task id.
aggregateVersion	That resource’s version after the change.
payload	Event details; task events include the scopeId of the task’s run.
createdAt	Timestamp.

Watch a run live

watch() is a long-poll stream you can resume. Filter by run (which includes its child runs), event types or one resource.

TypeScript

const controller = new AbortController();
process.on('SIGINT', () => controller.abort());

for await (const event of rs.events.watch({
  scopeId: run.id,
  types: ['task.succeeded', 'task.failed', 'group.finalized'],
  waitSeconds: 20,
  cursor: loadCursor(), // undefined on the first run
  signal: controller.signal,
})) {
  console.log(event.eventId, event.type, event.aggregateId, event.payload);
  saveCursor(event.cursor);
}

Python

async for event in rs.events.watch(
    scope_id=run.id,
    types=["task.succeeded", "task.failed", "group.finalized"],
    wait_seconds=20,
    cursor=load_cursor(),  # None on the first run
):
    print(event["eventId"], event["type"], event["aggregateId"], event["payload"])
    save_cursor(event["cursor"])

Pass stop_event= (an asyncio.Event) to end the stream from elsewhere. With the blocking client, rs.watch(...) takes the same keyword arguments and returns a regular iterator.

How watch behaves:

• At least once, in order. Events arrive ordered by eventId. After a reconnect you may see an event again, and the SDK doesn’t deduplicate for you: skip eventIds you’ve already handled.
• You own the cursor. Each yielded event carries a cursor; pass the last one back as cursor to resume. All events from one long-poll response share the same cursor (the position after that response), so persist it once you’ve handled every event that carries it.
• Long poll. waitSeconds (0 to 30) is how long the server holds a request open when there are no new events. The stream keeps polling until you abort it.
• Recovery. Watch is for noticing changes quickly. For the authoritative state of a task after a gap, read it with ticket.status().

Filters: types (list of event types), aggregate ({ type, id }, for example { type: 'task', id: taskId }), scopeId / scope_id (the run and its descendants), and limit.

For a one-off page instead of a stream, rs.events.list({ cursor, limit }) returns { events, nextCursor } (up to 500 events per page).

Webhooks

To push events to a server instead of holding a connection, register an HTTPS endpoint. Neither SDK wraps this yet, so call the API directly:

curl -X POST "$RUNSTATE_BASE_URL/v1/spaces/$RUNSTATE_SPACE_ID/destinations" \
  -H "Authorization: Bearer $RUNSTATE_API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hooks.example.com/runstate", "eventTypes": ["task.failed", "group.finalized"]}'

The response includes a signingSecret, returned only once. Each delivery is a JSON POST of the event (eventId, type, aggregateType, aggregateId, payload, createdAt) with these headers:

Header	Value
x-kernel-event-id	The event id. Use it to deduplicate.
x-kernel-timestamp	Unix time in seconds when the request was signed.
x-kernel-signature	v1= followed by the hex HMAC-SHA256 of <timestamp>.<raw body>, keyed with the signing secret.

Verify the signature against the raw request body before trusting the payload, and reject old timestamps. Any 2xx response counts as delivered. GET /v1/spaces/{spaceId}/deliveries lists recent delivery attempts and their results.

Diagnostics

Diagnostics are read-only, recent-window views for people and tooling. They are observations, not a source of truth: use them to understand a stuck run, not to decide who owns something.

TypeScript

const view = await rs.diagnostics.run(run.id);
for (const worker of view.workers) {
  console.log(worker.holder, worker.keys, worker.earliestLeaseExpiry);
}
for (const wait of view.waiting) {
  console.log(`${wait.scopeId} waits for ${wait.units} of ${wait.permit}, blocked by ${wait.blockedBy.join(', ')}`);
}

const waits = await rs.diagnostics.waits(); // space-wide: waiters, held grants, held claims
console.log(waits.waiters.length, waits.grants.length, waits.claims.length);

Python

view = await rs.diagnostics.run(run.id)
for worker in view["workers"]:
    print(worker["holder"], worker["keys"], worker["earliestLeaseExpiry"])
for wait in view["waiting"]:
    print(f"{wait['scopeId']} waits for {wait['units']} of {wait['permit']}, blocked by {wait['blockedBy']}")

waits = await rs.diagnostics.waits()  # space-wide: waiters, held grants, held claims
print(len(waits["waiters"]), len(waits["grants"]), len(waits["claims"]))

diagnostics.run(runId) covers the run and its child runs:

Field	Contents
scopes	The runs in the subtree and their states.
workers	Holders with live leases, the keys they hold, and their earliest lease expiry. diagnostics.workers(runId) returns just this list.
waiting	Pool waiters and what they’re blocked by.
expiredRecent	Claims and pool leases that expired recently, with the holder that lost them. A good first place to look for crash takeovers.
messages	Per work queue: messages by state and recent dead letters.
deliveriesFailed	Recent failed webhook deliveries.
timeline	Recent events for the run.

diagnostics.waits() returns the space’s wait-for edges (who waits for which pool, who holds grants and claims). It shows what each piece of work is waiting for; it doesn’t resolve anything.

Related

• Troubleshooting: common symptoms and where to look.
• API reference: Events, watch & webhooks, Diagnostics.