Observability

Three pillars: metrics (Prometheus), logs (Loki via structlog) and traces (OpenTelemetry, optional). The Grafana stack ships pre-provisioned in docker-compose.yml.

Metrics

The FastAPI service exposes /metrics; long-running workers each bind a daemon HTTP server (shared.metrics_server.start_metrics_server) on per-worker ports:

Process	Port	Notes
FastAPI	8000	/metrics route via MetricsMiddleware
ingester	9091	start_http_server
aggregator	9092	—
health_watchdog	9093	—
synthetic_status	9094	dev only
userbot pool	9095	—

All metrics are registered on a single prometheus_client.CollectorRegistry in shared/metrics.py. Adding a metric is one entry there, then the Grafana panel + Prometheus alert that references it both get caught by their respective CI validators if you forget to update them.

Key series

Name	Type	Labels	Use
helios_api_requests_total	Counter	method,route,status	request rate, error share
helios_api_request_duration_seconds	Histogram	method,route	p50/p95/p99
helios_api_in_flight	Gauge	route	concurrency saturation
helios_status_events_ingested_total	Counter	action	end-to-end pipeline health
helios_payments_total	Counter	provider,sku,status	revenue + failures
helios_health_alerts_total	Counter	kind,severity	watchdog activity
helios_notifications_sent_total	Counter	kind	DM fan-out volume
helios_stealth_violations_total	Counter	userbot_id,method	must stay zero
helios_userbots_active	Gauge	bucket	pool capacity

Adding a metric

1. Define it in shared/metrics.py against the shared registry.
2. Tick it from the relevant call site (use a local import to avoid pulling the prometheus dep into pure modules).
3. Optionally reference it in a Grafana panel + Prometheus alert. CI validators in tests/unit/test_prometheus_rules.py and tests/unit/test_grafana_dashboards.py ensure renames don't silently break alerting.

Alerts

infra/prometheus/alerts.yml defines five groups:

Group	Alert	Trigger
api	HighErrorRate	5xx > 1 % for 5 min
api	P99LatencyHigh / Critical	p99 > 2.5 s / 5 s
stealth	StealthViolationAttempted	any non-zero rate — P0
pool	UserbotPoolEmpty	helios_userbots_active sums to 0 for 5 min
business	PaymentFailuresSpike	failed share > 10 % for 10 min
business	HealthAlertsBurst	mirrors critical watchdog alerts

Critical alerts page on-call; warnings post to the team channel.

Logs

All processes use structlog with JSON output. Routed to Loki via the compose-provisioned config (infra/loki/local-config.yaml).

Standard fields on every event:

• event — dotted name (e.g. billing.paid, userbot.failure)
• timestamp — ISO 8601 UTC
• level — info / warning / error
• One of client_id, target_id, userbot_id where applicable

Never logged: session strings, API tokens, payment payloads, message bodies, media binary, parameters flagged secret=True. Use shared.redact.redact() if a struct holds anything sensitive.

Tracing

Opt-in. Set OTEL_EXPORTER_OTLP_ENDPOINT to enable; shared/tracing.py configures the OTLP exporter at boot. The hot path (userbot → Redis → ingester → notifier) is the natural first thing to trace.

Watchdog cross-check

The health_watchdog worker runs the same business-logic checks as the Prometheus alerts and DMs the owner directly via the bot. This is the secondary signal path — Prometheus alerts go to the team channel / PagerDuty; the watchdog goes straight to the owner. Both should agree.