Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Run your own

Short version: one hub image, one config file for at least one worker, and this site ships inside the hub’s own build.

The hub

The hub is a single container image. It refuses to start unless every required environment variable is set — there’s no insecure default to fall back on. Names only below; your operator (or you, deploying it) supplies the values:

Required:

  • OIDC_ISSUER, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET — the identity provider browser logins go through.
  • PUBLIC_BASE_URL — this deployment’s own externally-reachable URL.
  • SESSION_SECRET — signs the browser session cookie.
  • REQUIRED_GROUP — the identity-provider group a login must carry to pass.
  • INDEX_PATH — where the session/message index database lives.

Optional:

  • PORT, STATIC_DIR — network and asset-serving basics.
  • DOCS_DIR — where this site’s built output lives on disk; unset serves no docs at all.
  • DOCS_HOST — a second hostname that serves this site at its own root, IN ADDITION TO (not instead of) /docs on the app host, which keeps working exactly as before whether or not this is set (see Start here).
  • AGENT_OIDC_RESOURCE — the resource/audience value for OIDC-authenticated agent tokens, reusing the same issuer as browser login with a distinct resource indicator (see Authentication). The old shared AGENT_TOKEN is retired; if it is still set the hub ignores it and logs a warning.
  • ROOT_BINDINGS_JSON — optional JSON object {"<root id>": "<worker sub>"} pinning a root to the worker allowed to serve it. It overrides what the hub learned on its own (first-come and presence bindings), and each change is audited. Operator bindings are the exception: a binding made with PUT /api/remote/v1/workers/root-bindings/{root} (source admin) is never rebound, unbound or promoted by ROOT_BINDINGS_JSON, a null value included. A root the object names, even with null, is pinned, and that PUT answers 409 for it; remove the key and restart to make the root rebindable there. ROOT_BINDING_FIRST_COME (default true) lets the first worker to touch an unbound root own it; set it to false so only ROOT_BINDINGS_JSON binds roots. GET /api/roots shows each root’s owner_principal. Turns that name no root go only to a worker listed in UNROOTED_POOL_PRINCIPALS or one that owns a config binding or a binding bootstrapped from presence; a first-come or operator binding never grants them. Each key must be a valid root id: 1-128 characters from A-Z a-z 0-9 . _ : -. The hub refuses to start on any other key and names it in the error. The object is the full set of config bindings: a root whose key is removed from it is unbound at the next restart (its row is deleted and audited as unbind_config_removed with the previous owner), and an empty object unbinds every config binding. A null value still unbinds that root whatever bound it, except an operator binding. A key that names a root already bound to the same worker (first-come or bootstrapped from presence) turns that binding into a config binding at the next restart (audited pin_config), so removing the key later unbinds it. Only roots that are not pinned at all, whether bound first-come or bootstrapped, are outside the set and survive any edit of the object. Reservation roots (res-r, a 20-character reservation id, a dot and a harness, such as res-r<20 characters>.claude) are bound only by enrollment; a root that merely starts with res- (res-lab) is an ordinary root. The binding audit records every bind, rebind and unbind. Rejections (a worker naming a root bound to someone else, or any unbound root in strict mode) are coalesced to at most one row per worker and action per minute; the next row says how many similar rejections were suppressed and lists a few sample root ids. Rejection rows are pruned once they are older than 30 days or beyond the newest 100,000, at startup (after the listener is up) and every six hours. Bind and unbind rows are kept.
  • RATE_LIMIT_REMOTE_SUBMIT, RATE_LIMIT_REMOTE_READ, RATE_LIMIT_AGENT_CLAIM, RATE_LIMIT_AGENT_EVENTS, RATE_LIMIT_AGENT_COMPLETE, RATE_LIMIT_AGENT_PUBLISH — per-principal token buckets written <per_minute>[/<burst>] (defaults 30/10, 600/120, 300/60, 3000/600, 600/120, 600/120); 0 disables a class. The workers feature adds six more with the same syntax, listed under Reserved workers below.
  • HUB_CLIENT_IP_SOURCE — which address a request is attributed to where a limit is keyed by caller address instead of by token subject (enrollment, static check-in, and the token endpoint when no client is named). socket (default) uses the TCP peer and ignores every header. fly-client-ip reads the Fly-Client-IP header that Fly’s edge sets; if HUB_TRUSTED_PROXIES is also set, the header is believed only when the TCP peer is inside it. x-forwarded-for takes the right-most X-Forwarded-For hop that is not a trusted proxy, and only when the TCP peer is itself a trusted proxy; it needs HUB_TRUSTED_PROXIES. Any other value stops the hub at startup. Set a header source only when a proxy you run overwrites that header, otherwise a client chooses its own bucket. deploy/fly/fly.toml sets fly-client-ip.
  • HUB_TRUSTED_PROXIES — comma-separated IP addresses or CIDRs of the proxies allowed to speak for a client, for example 10.0.0.0/8,fdaa::/16. A value that does not parse stops the hub at startup. Unset by default.
  • TURNS_DB_PATH, TURN_REQUEUE_MAX_ATTEMPTS, PENDING_TURN_TTL_MS — turn queue storage and tuning.
  • TIERS_CONFIG — path to a TOML file defining the model tiers (quick/standard/heavy) turn creation can name (see Integrate as a UI). Unset means the hub has no tier table at all — a NEW-session or fork request naming a tier is rejected outright, 400 "this hub has no tier table configured (TIERS_CONFIG unset)". An EXISTING session’s follow-up naming a tier is NOT rejected the same way: continuity already pins its provider, so the hub just narrows to that provider’s own candidates (an empty list on the hub’s side, since there’s no tier table to narrow from) and forwards it to whichever worker claims the turn — the worker then falls back to its OWN locally configured tier table to pick a candidate, rather than being told “no” by the hub.
  • VAPID_PUBLIC_KEY, VAPID_PRIVATE_KEY, VAPID_SUBJECT — Web Push notifications on turn completion, if you want them.

The review profile

/api/integrations/turns* (Integrate as a dispatcher) has its own independent set of environment variables, all optional as a group but interdependent once you set any one of them:

  • API_RESOURCE — the resource/audience value for review:run OIDC tokens, distinct from AGENT_OIDC_RESOURCE above. Shared with the work profiles below — there’s no separate work-run audience.
  • REVIEW_PROFILE_PROVIDER, REVIEW_PROFILE_ROOT_ID, REVIEW_PROFILE_WORKING_DIRECTORY — the fixed provider, root, and directory every queued review runs against.
  • REVIEW_PROFILE_MODEL, REVIEW_PROFILE_EFFORT — optional overrides.
  • REVIEW_PROFILE_PERMISSION_MODE — optional, defaults to (and today must be) "plan".

Setting none of the REVIEW_PROFILE_* variables leaves /api/integrations/turns* entirely unconfigured — every route under it answers a plain 404. Setting any one of them requires API_RESOURCE and the three non-optional REVIEW_PROFILE_* variables above, or the hub refuses to start.

The work profiles

/api/integrations/work-runs* (Integrate as a dispatcher) is a separate, independently configured surface layered on top of API_RESOURCE above (the SAME resource/audience review:run tokens use — there is no distinct work-run audience):

  • WORK_PROFILES_JSON — a JSON object mapping each opaque target_ref a caller may name to one work profile: {"provider", "root_id", "working_directory", "model"?, "effort"?, "permission_mode", "branch_prefixes"?, "base_ref"?, "checkout"?, "repo"?}. Must contain at least one profile if set at all (an empty object is refused, same as leaving it unset in effect, but explicitly). Field limits, all in bytes: target_ref (the JSON key) non-blank and at most 500; provider non-blank and at most 200; root_id non-blank and at most 500; working_directory non-blank and at most 4096, and must be an absolute path; permission_mode non-blank and at most 100; model, when given, non-blank and at most 500. permission_mode must be one of "acceptEdits", "auto", or "bypassPermissions" — the unattended subset; unlike the review profile, "plan" is not accepted here. effort, when given, must be one of "low", "medium", "high", "xhigh", "max". provider must additionally name a provider capable of being driven end to end (not every registered provider qualifies). branch_prefixes (default ["aya-task/"]) lists the prefixes a caller’s own branch must start with; [] means callers never choose one. Each must make a safe branch name with something after it. base_ref, when given, is the base a run starts from when the request names none; same name rules as a branch. checkout is "workdir" (the default: runs use working_directory, as they always have) or "fresh", which gives every run its own new copy of repo and requires it. repo (only with "fresh", at most 4096 bytes) is a git URL (https, ssh, git or file, no password) or an absolute path to a local mirror on the helper’s machine. The hub stores the resulting {repo, branch, base_ref} with the run; helpers do not act on it yet (the claim does not carry it until the helper side is wired). Setting this leaves /api/integrations/work-runs* unconfigured when unset — every route under it answers a plain 404 — and requires API_RESOURCE plus the full WORK_REPORT_* group below when set, or the hub refuses to start.
  • WORK_REPORT_URL, WORK_REPORT_RESOURCE, WORK_REPORT_CLIENT_ID, WORK_REPORT_CLIENT_SECRET — all four together, or none: this is the credential and endpoint the hub uses to deliver a completed work run’s result back to you on its own initiative, a separate OAuth2 client-credentials client the HUB presents (not something a caller authenticates with) — see Run work through the hub: work runs. WORK_REPORT_URL must be an absolute HTTPS URL (or an HTTP loopback URL, for local development only) with no embedded credentials, query, or fragment. WORK_REPORT_RESOURCE must be an absolute HTTPS URI under the same restrictions — the resource/audience the hub requests for its own outbound token.

The remote API

/api/remote/v1/* (Remote API) lets a remote agent that cannot join your network read sessions and drive chats on your machines through the hub. It is off until REMOTE_API_RESOURCE is set; every path under it answers a plain 404 until then.

  • REMOTE_API_RESOURCE — the resource/audience value for its chat:read and chat:drive tokens. Must differ from API_RESOURCE and AGENT_OIDC_RESOURCE, or the hub refuses to start. In your identity provider, allow only the clients you mean to trust to request this resource and these scopes; that is the access control.
  • REMOTE_ALLOWED_SUBJECTS — required, comma-separated client-… subjects. No other subject is admitted even with a valid token, and the hub refuses to start with REMOTE_API_RESOURCE set and this empty or missing.
  • REMOTE_MACHINE_ALIASES_JSON — optional JSON object naming machines, keyed by the OIDC subject a machine’s worker authenticates as, for example {"client-abc":"Studio"}. Unnamed machines are listed by their subject.
  • REMOTE_MAX_ACTIVE_TURNS — optional, default 8: how many turns one caller may have waiting or running at once before the hub answers 429.

A remote caller can ask a chat for read-only (the default) or allow-edits and nothing stronger, however its token was issued.

Reserved workers

Workers lets a remote caller reserve a short-lived machine, which enrolls with the hub and serves only that caller. It is off by default; with it off every worker route answers 404 {"error":"workers unavailable"} and the hub behaves as before. The reservation and admin routes sit under /api/remote/v1, so the remote API above must be on too (REMOTE_API_RESOURCE and REMOTE_ALLOWED_SUBJECTS), and workers authenticate on /api/agent/*, so AGENT_OIDC_RESOURCE is needed as well. Every variable below is optional, and a value that is set but invalid stops the hub at startup.

To turn it on:

  • WORKLOAD_ISSUER=on — the switch. The hub becomes a second OIDC issuer and signs worker tokens itself.
  • PUBLIC_BASE_URL — becomes the issuer (iss) of those tokens, so it must be the URL workers and provisioners can reach. It must be https; loopback http is allowed for development.
  • Signing keys, one of two ways. Either set WORKLOAD_ISSUER_KEK to the base64 of exactly 32 random bytes and the hub generates and stores its RSA signing keys sealed under it, or point WORKLOAD_SIGNING_KEY_FILE at a PEM key (WORKLOAD_SIGNING_KEY_NEXT_FILE and WORKLOAD_SIGNING_KEY_PREVIOUS_FILE are optional and need the active file). Guard the KEK like any other secret; see KEK loss and rotation below.
  • WORKER_POOLS_FILE — a JSON file with the pools, either an array of pool documents or {"pools":[...]}. Any error in it stops the hub at startup. The pool document (harnesses, who may reserve, limits, backend, attestation and token grants) is described in docs/api/workers.md. Admins can also create pools at runtime with a workers:admin token.

A minimal static pool, for one registered machine:

[{"name": "dev", "harnesses": ["claude"],
  "access": {"principals": ["client-requester"]},
  "backend": {"kind": "static"},
  "attestation": {"kind": "static_psk"},
  "grants": [{"audience": "$hub_agent",
              "scopes": ["agent:claim", "agent:report", "execution:heartbeat"]}]}]

Other settings, all optional:

VariableDefaultMeaning
WORKLOAD_TOKEN_TTL_SECS600Lifetime of worker and service tokens, 60 to 3600.
WORKLOAD_EXTERNAL_TOKEN_TTL_SECS300Lifetime of tokens for external audiences, 30 to 900.
WORKLOAD_EXTERNAL_AUDIENCESemptyComma-separated https:// resources a pool grant may name beyond the hub’s own.
RELAY_RESOURCEnoneThe audience the $relay grant placeholder resolves to.
WORKERS_PROVISION_RESOURCE{issuer}/api/provision/v1The audience a provisioning service presents.
WORKERS_WEBHOOK_SIGNING_KEYnoneA whsk_... key for the signed webhook to a provisioning service. Needed only for webhook pools.
WORKERS_WEBHOOK_SIGNING_KEY_PREVIOUSnoneThe previous webhook key, kept while receivers rotate.
AGENT_ACCEPT_EXTERNAL_ISSUERtrueSet false so /api/agent/* accepts only hub-issued tokens.
WORKLOAD_KEY_ROTATION_DAYS30Automatic rotation period for stored signing keys, 1 to 365.

Key rotation: with stored keys the hub rotates automatically. A new key is published in the JWKS two days before it becomes active, and the old one stays published for seven days so tokens signed just before the switch still verify. An admin can rotate on demand with POST /api/remote/v1/workers/issuer/keys/rotate (it stages the next key), or with ?emergency=true to retire every older key and activate a new one immediately. With file-backed keys the route answers 409; replace the files instead.

Revocations reach identity-provider tokens too: with the issuer on, a token from the external issuer on /api/agent/* is refused when its client is disabled or its subject was revoked after the token was issued. A subject revocation is pruned an hour after it takes effect (the hub’s own token lifetime cap), so keep the identity provider’s access-token lifetime at one hour or less for the clients that call the agent routes, or a long-lived token issued before a deprovision would be accepted again once its revocation is pruned.

KEK loss and rotation

The KEK seals the stored signing keys at rest and nothing else can open them. If WORKLOAD_ISSUER_KEK is lost or changed, the hub refuses to start with the active signing key does not open under WORKLOAD_ISSUER_KEK (wrong KEK?). Putting the original value back is the only way to keep the existing keys. If it is truly gone, stop the hub, delete the rows of the workload_signing_keys table in the database at TURNS_DB_PATH, and start it again with a new KEK: the hub generates a fresh key set. Every token signed by the old keys then fails verification, so workers and provisioners must mint new ones, and any external verifier must refetch the JWKS.

To change the KEK while keeping the stored keys, re-seal them in place with WORKLOAD_ISSUER_KEK_PREVIOUS:

  1. Set WORKLOAD_ISSUER_KEK_PREVIOUS to the old KEK and WORKLOAD_ISSUER_KEK to the new one.
  2. Restart once. Every stored key that opens only under the old KEK is sealed again under the new one.
  3. Remove WORKLOAD_ISSUER_KEK_PREVIOUS.

WORKLOAD_ISSUER_KEK_PREVIOUS needs WORKLOAD_ISSUER_KEK and has the same format. The KEK also keys the workers audit chain, so the hourly chain check reports rows written under the old KEK as not verifying after a change; note the head the hub logs before you rotate. Keep the KEK in your secret store, never in the image or the repository.

Workers need an always-on hub

The issuer, the reconciler and the wake channels live in the hub process, so a hub that is stopped, or scaled to zero when idle, stops reservations, wakes and revocation. On Fly, with WORKLOAD_ISSUER on, set auto_stop_machines = "off" and min_machines_running = 1; the shipped deploy/fly/fly.toml keeps its defaults, so make that change in your own deployment. The hub logs an error at startup when FLY_MACHINE_ID is set and workers are on.

Rate-limit classes

Every class is written <per_minute>[/<burst>], with 0 disabling it, and is keyed by the token subject unless the table says otherwise. The six classes of the remote and agent surface are in the list at the top of this page; these are the six the workers feature adds, and the routes on each:

VariableDefaultRoutes
RATE_LIMIT_WORKERS_RESERVE30/10POST /api/remote/v1/reservations, POST /api/remote/v1/sessions/{pk}/move
RATE_LIMIT_WORKERS_ADMIN60/20every /api/remote/v1/pools*, /workers/* and /principals/* route
RATE_LIMIT_TOKEN_ENDPOINT120/30POST /oauth/token, keyed by client id, or by caller address with no client
RATE_LIMIT_ENROLL30/10POST /api/enroll/v1/enroll and /static/checkin, keyed by caller address
RATE_LIMIT_PROVISION120/30/api/provision/v1/*, keyed by the provisioner’s subject
RATE_LIMIT_AGENT_WAKE120/30GET /api/agent/wake

Reading a reservation (GET /api/remote/v1/reservations, .../{id}, DELETE .../{id}) draws from RATE_LIMIT_REMOTE_READ. On the agent surface, GET /api/agent/reservation and the worker heartbeat draw from RATE_LIMIT_AGENT_CLAIM.

On the machine itself, the worker is aya-helper with an [enrollment] table in its config, which is exclusive with the other root and hub-connection settings; see docs/api/workers.md for the keys.

The preview relay

The preview relay serves dev servers that run on your machines at https://<port>-<label>.<preview-domain>. It is a separate binary, aya-relay, with a small set of hub settings. It is off until the hub has a signing key; every /api/relay/* path, /r/s/* and the key document answer a plain 404 until then.

What you need

  • A preview domain that is not the hub’s registrable domain (the relay and the hub refuse to start otherwise; allow_same_site_preview and RELAY_ALLOW_SAME_SITE_PREVIEW=1 override it and are unsafe) and not a subdomain of it, with a wildcard record (*.preview.example-preview.test) pointing at the relay. Ask for it to be added to the Public Suffix List, or dedicate one domain per organization, so sibling previews do not share cookies (docs/relay/threat-model.md).
  • A wildcard TLS certificate for *.preview.example-preview.test. Either let Fly terminate it (fly certs add "*.preview.example-preview.test" prints the _acme-challenge record to create), or let the relay get one itself with tls.mode = "acme_dns01" and a Cloudflare API token limited to DNS edit on that zone (RELAY_ACME_DNS_TOKEN).
  • A Valkey or Redis only if you run more than one relay replica ([directory] kind = "redis", RELAY_REDIS_URL). The URL must be rediss:// and carry a password or ACL user; the relay refuses to start otherwise. The shared ACME bundle is stored encrypted under a key derived from RELAY_PEER_SECRET. Replicas dial only addresses inside listen.internal_cidrs (private IPv4 and Fly 6PN by default).

In your identity provider

Create an API resource for the relay and one for the hub’s relay endpoints, and four scopes:

ScopeResourceGranted to
relay:connectthe relay (https://relay.example.test)each worker’s client
relay:syncthe hub’s relay endpoints (https://relay.example.test/hub)the relay’s own client
relay:auditthe same hub endpoints resourcethe relay’s own client
relay:openREMOTE_API_RESOURCEscripted callers of the bearer ticket route

Hub settings

  • RELAY_TICKET_SIGNING_KEY — an Ed25519 private key in PKCS#8 PEM (generate with openssl genpkey -algorithm ed25519). Setting it turns the relay on; the three variables below are then required, or the hub refuses to start.
  • RELAY_PREVIEW_DOMAIN — the preview domain, for example preview.example-preview.test.
  • RELAY_PUBLIC_URL — the relay’s public URL, for example https://relay.example.test.
  • RELAY_HUB_RESOURCE — the resource/audience of the relay’s calls to the hub (the relay:sync and relay:audit tokens). Must differ from the other resources.
  • RELAY_TICKET_SIGNING_KEY_PREVIOUS — optional: the key being rotated out. The key document lists both, so tickets signed with it keep verifying.
  • RELAY_SHARE_MAX_TTL — optional, default 24h (also 3600, 90m, 2d): the longest a share link may live.
  • RELAY_ANONYMOUS_SHARES — optional, default off: set to 1 to allow anyone shares that need no hub login.
  • RELAY_API_ALLOWED_SUBJECTS — optional, comma-separated client-… subjects allowed to mint tickets on POST /api/remote/v1/relay/tickets. Unset, that route admits no one. It also needs REMOTE_API_RESOURCE.

The ticket key is the one secret that can forge a preview session; keep it out of the image and the repository like every other secret here.

Run the relay

deploy/relay/ holds a Dockerfile, a fly.toml, a commented relay.example.toml and a compose.yaml that runs the relay with a password-protected, TLS-only Valkey. The relay reads its TOML from RELAY_CONFIG and its secrets from RELAY_COOKIE_KEY, RELAY_PEER_SECRET, RELAY_HUB_CLIENT_SECRET, RELAY_ACME_DNS_TOKEN and RELAY_REDIS_URL. On Fly, keep tls.mode = "off" and expose the replica forwarding port only on the private network. With several replicas, set the cookie key and peer secret to the same value on every one.

Point workers at it with the helper’s [relay] section (Preview relay).

The worker

aya-helper loads one TOML file and refuses to start without it — no settings UI to fall back on. The smallest useful config points it at a single execution source rather than a full local index. Two DIFFERENT tokens are required here, not one: a top-level token_env/token_file for the worker’s OWN local HTTP server (it refuses to start with no resolvable token at all, even bound to loopback-only — this is unrelated to talking to any execution source), and each [[execution_sources]] entry’s own token_file/token_env for authenticating outbound to that source:

token_file = "/run/secrets/worker-local-token"

[[execution_sources]]
id = "primary"
base_url = "https://hub.example.test"
token_file = "/run/secrets/agent-token"

data_dir is optional and omitted here on purpose: leave it unset and the worker stores its data in a sibling data/ directory next to the config file itself (e.g. ~/.config/aya-helper/config.toml gets ~/.config/aya-helper/data/) — set it explicitly only when you want that data somewhere else entirely.

This is a plain source: the worker executes whatever provider, model, and root each claimed turn’s own body specifies, exactly as Integrate as a worker describes. Add an [execution_sources.worker_profile] block only for a worker-routed source — one that supplies prepared text and an opaque affinity instead of a provider/root choice — and only once you’re ready for the worker to enforce (and reject a claim that violates) the guarantee that such a source never sets provider, model, root_id, or any other local-policy field itself (see Integrate as a worker).