ADR-0010 — Hybrid analytics: tiered realtime + batch

Status. Accepted. Date. 2026-05-18.

Context

Analytics has to serve four very different audiences from one engine:

Free users — see something useful with minimal cost to us.
Pro users — recent rollups, mostly fresh, still cheap to compute.
Premium users — feel "live"; charts visibly update as events arrive.
Enterprise users — full real-time with sub-second lag for trading-floor-grade dashboards.

The owner asked for a hybrid that gives each tier a perceptibly better experience without us paying realtime cost for everyone.

Decision — four-band engine

Every analytics query goes through core/analytics/engine.py:resolve(query, scope_ctx) which routes by tier:

Tier	Window resolution	Refresh strategy	Latency budget
Free	5-min buckets	on-read with 60 s memoize	≤ 3 s on first hit, ≤ 100 ms cached
Pro	1-min buckets	background rollup every 5 min, on-demand refresh button	≤ 1 s
Premium	1-min buckets + last-5-min live	continuous aggregate + WS push	≤ 200 ms
Enterprise	1-min buckets + full live	TimescaleDB continuous aggregates + WS push, all windows	≤ 100 ms

All resolution and refresh values are parameters (analytics.window_seconds, analytics.refresh_seconds, analytics.live_push_enabled, analytics.live_window_seconds) scoped GLOBAL | TIER | CLIENT | TARGET.

Decision — data layers

       ingester  ─→  status_events / target_messages / business_messages  (hypertables)
                              │
                              ├─→ aggregator  ─→  helios_rollup_1m  (continuous aggregate)
                              │                       │
                              │                       └─→ helios_rollup_5m, _1h, _1d
                              │
                              ├─→ live_emitter ─→ Redis Stream  helios:analytics:live
                              │                       │
                              │                       └─→ WS gateway (Premium+ clients)
                              │
                              └─→ Meilisearch (text)

status_events is partitioned daily, compressed after 7 days. Continuous aggregates (Timescale CREATE MATERIALIZED VIEW ... WITH (timescaledb.continuous)) keep the rollups fresh in the background. We never SELECT raw events for a 30-day chart — we hit the appropriate rollup.

Decision — what we compute

Universal metrics, applicable to both tracking and business modes:

Online time: minutes online / hour, per day.
Session counts: sessions per day, distribution of session lengths.
Hour-of-day heatmap: aggregate online minutes per (dow, hour).
Sleep schedule: longest offline window per day; derived "typical bedtime / wake-up" with confidence.
Pattern detection: anomalies (rare offline window during typical online hours, etc.).
Word stats (business + chat tracking): top words, most-used stickers, sticker-pack distribution, message length distribution, reply-vs-original ratio.
Interlocutor graph: per chat/per target — who talks to whom, frequency, who initiates.
Edit / delete tallies: number of messages edited or deleted per day, distribution of "minutes-after-send" of edits.
Reactions: emoji distribution, reaction velocity.
Comparison view: any two targets / chats / time windows side by side.

Each metric maps to a continuous aggregate. Definitions live in infrastructure/db/migrations/versions/NNNN_analytics_views.py.

Decision — query engine

core/analytics/engine.py exposes:

async def resolve(
    query: AnalyticsQuery,
    *,
    scope_ctx: ScopeContext,
) -> AnalyticsResult: ...

AnalyticsQuery is a typed dataclass — metric name, dimensions, time window, filters. The engine picks the best rollup that contains the query window, falls back to raw events only when no rollup covers it.

Decision — caching

Every resolved result has a cache key derived from (metric, scope_ctx, window, filters) and stored in Redis with a per-tier TTL (parameter). Cache invalidation is parameter-key-bound: when a relevant override changes (e.g. retention.target_history_days), the affected cache entries are dropped.

Decision — real-time push (Premium+)

A WebSocket gateway at /v1/stream subscribes the client to a set of metric streams. The live emitter publishes deltas, never full snapshots — the client merges. Backpressure is enforced by dropping intermediate deltas (coalesce parameter controls the floor latency, default 200 ms).

Decision — fairness & cost control

Two parameters cap analytics cost:

analytics.max_concurrent_heavy_queries_per_client — defaults 2 / 4 / 8 / 16 by tier.
analytics.archive_window_max_days — defaults 90 / 365 / 1095 / 3650.

Heavy queries (≥ 30 days or "compare two targets") route to a priority queue. Owner can throttle the queue globally.

Decision — UI behaviour

Charts know their data source:

If we have rollups, they fetch them and show a "Updated N min ago" timestamp.
If we have live, they keep an open WS and animate updates; the timestamp says "live".
If the user is on Free, the refresh button manually drops the 60 s memoize for that scope.
All charts share <TimeSeriesChart> from apps/miniapp/src/generators/ — a single component covers every tier.

Decision — extensibility

New metrics are one record:

register_metric(
    AnalyticsMetric(
        key="messages.edits_per_day",
        category="messages",
        dimensions=("chat", "sender"),
        rollup="helios_rollup_1m_edits",
        ...
    )
)

The Mini App picks it up via the parameters/analytics manifest auto-generated by scripts/parameters_codegen.py.

Consequences

Free users get sensible analytics for free; Premium feels noticeably faster.
We never run an unbounded raw scan in the request path — the aggregator carries that weight in the background.
A new metric is a few lines plus a continuous-aggregate definition.