Runbook

Skim this once on a quiet day. The first three checks per incident usually localise the problem in under five minutes.

API returns 502 from nginx

1. docker ps --filter name=infra-api | grep healthy — is the container up and healthy?
2. docker logs --tail 30 infra-api-1 — any traceback at startup?
3. curl http://127.0.0.1:8000/healthz from the host — bypasses nginx; if this works the problem is in nginx, not the API.

Bot stopped responding

1. docker logs --tail 30 infra-bot-1 — anything since the last bot.boot log line?
2. From inside the container: docker exec infra-bot-1 sh -c 'curl -sS "https://api.telegram.org/bot${BOT_TOKEN}/getMe"' — does Telegram still recognise the bot?
3. Check the rate-limit page from Telegram — botpark blocks abuse: if getMe returns 429, wait.

Postgres auth failure on app startup

1. Did you regenerate .env after the first boot? The helios_app role was created with the original password. Either:
   • revert the password in .env to what was used at first boot, or
   • docker compose -f infra/docker-compose.yml down -v to drop the volume and re-init with the new credentials (destroys local data).
2. docker logs infra-postgres-1 | tail -20 — look for FATAL: password authentication failed.
3. From inside the postgres container, \du lists roles — confirm helios_app exists with Login attribute.

Payment failure spike

HeliosPaymentFailuresSpike alert in Prometheus.

1. Is Telegram Stars itself down? t.me/durov for the public status.
2. Audit log: GET /v1/owner/audit?event_name_prefix=payment. — what's the error breakdown?
3. helios_payments_total{status="paid"} in Prometheus — flat = funnel broken upstream; with paid steady = checkout-stage failure.

A single bad SKU configuration explains most real-world spikes. Verify billing.tier_price.pro_stars_month etc. match the Mini App's offer page.

Userbot pool understaffed

HeliosUserbotPoolUnderstaffed alert.

1. Owner panel → Userbots — how many are active vs quarantined?
2. Quarantined: read the most-recent audit_log row with event_name="userbot.quarantined" — what was the trigger (AUTH_KEY_UNREGISTERED, USER_DEACTIVATED, etc.)?
3. Re-import a fresh session via Owner panel → Userbots → Import. The resilience dispatcher hot-swaps active tracks once the new session is healthy.

Mini App initData verification fails

Symptom: every /v1/me call returns 401.

1. curl https://api.your-domain.tld/v1/public/config — does the API itself respond? If not, this is an API problem, not an auth one.
2. Open the Mini App → DevTools → Network — the X-Telegram-Init-Data header should be present on every request.
3. Check the bot's webhook + Mini App URL configuration in @BotFather — setdomain must match MINIAPP_PUBLIC_URL.

TLS expiring within 30 days

certbot renewal already runs nightly via systemd-timer. Verify:

sudo systemctl list-timers | grep certbot
sudo certbot certificates

If Expires within shows a problem:

sudo certbot renew --dry-run
sudo certbot renew
sudo systemctl reload nginx

"Stealth violation" alert

This is a P0. The runtime guard raised StealthViolation for a call that wasn't on the whitelist.

1. audit_log row with event_name="stealth.violation_attempted" lists the method + caller stack.
2. Look at the most recent userbot-touching code commit — is there a new RPC that snuck past CI?
3. Do not whitelist the method to make the alert go away. Fix the caller. The whitelist is a safety net, not a knob.