ADR-0009 — Media compression strategy

Status. Accepted. Date. 2026-05-18.

Context

Helios captures media at scale — photos, voice notes, videos, documents, animated stickers, custom emoji. Storage on the spec'd MinIO self-host tier becomes the dominant cost. Each media type has a correct modern codec; using a single codec for everything is wrong.

Goals: high compression ratio without visible quality loss, low CPU on the worker, byte-perfect retrieval (originals are recoverable bit-for-bit when a client exports their archive), content-addressable deduplication so the same Telegram file across 50 subscribers costs us once.

Decision — codec per content type

Class	Source format	Stored format	Codec	Parameters
Photo / image	JPEG, PNG, HEIC	AVIF	libaom-av1	`q=50`, `speed=6`, 8-bit; WebP fallback `q=85`
Animated sticker	TGS (Lottie gzip)	TGS preserved + zstd container	zstd-19	`--long=27`
Video sticker	WebM VP9	WebM VP9 preserved (no re-encode)	n/a	wrapped in zstd if not already compressed
Static sticker	WebP	WebP preserved	n/a
Custom emoji	TGS / WebP / WebM	preserved exactly	n/a	classified by Telegram type field
Voice note	OGG/Opus	OGG/Opus preserved	n/a	already optimal
Audio file	MP3, M4A, FLAC	Opus 48 kHz	libopus	`--bitrate 64 --vbr`, 96 kbps for music, 32 kbps for voice
Video	MP4, MOV	WebM AV1	libaom-av1	CRF 32, two-pass, 30 fps cap; original kept until retention TTL if client paid for "lossless archive"
Document text	any	original + zstd	zstd-19	`--long=27`
Document binary	any	original + zstd	zstd-19	only if compressible (skip if entropy > 7.5 bits/byte)

We treat the original as the canonical artefact for compliance and exports. The optimised derivative is what we serve to clients in normal browsing. Both share the same file_unique_id row in target_media / business_media with two storage keys:

{file_unique_id}/orig
{file_unique_id}/opt

Originals are retained until the global retention.original_media_days parameter expires or every subscriber's history retention has lapsed (whichever comes later). Derivatives live as long as the row exists.

Decision — content-addressable storage & dedupe

Storage key is derived from file_unique_id (Telegram's stable, content-addressed identifier). Two clients tracking the same target who both see the same photo store one copy. The target_chat_subscriptions and business_chats tables just reference the file_unique_id.

Per-file encryption key remains HKDF(master_key, file_unique_id) (ADR — security model). Compression happens before encryption.

Decision — pipeline

apps/workers/media_downloader.py (existing) becomes the producer. A new worker apps/workers/media_compactor.py consumes:

Pull next media job from Redis Stream helios:media:to_compact.
Probe content type with python-magic; classify per the table.
Run codec via ffmpeg (audio, video) or pillow-heif/imagecodecs/avif-python (images).
Verify the derivative decodes (sanity check).
Stream both orig and opt to EncryptedStorage.
Update target_media with original_size, stored_size, derivative_codec, derivative_quality_q.
Emit media.compacted event with byte savings — analytics dashboard rolls it up.

CPU budget per process is a parameter (compression.cpu_quota_percent). The worker honours it via nice and a slot pool.

Decision — quality presets

Three named presets in core/policies/compression.py:

default — the table above.
archival — q=35 AVIF, AV1 CRF 28 (larger files, near-perfect).
aggressive — q=60 AVIF, AV1 CRF 36 (smaller, slight texture loss).

Presets are parameter-tunable per client and per chat (clients on Premium can opt their chats into archival; the platform default stays default).

Decision — retrievability guarantee

When a client requests an archive export (archive_builder.py), the export contains the originals, not the derivatives. The derivative track is purely a serving optimisation; we never lose the original until policy says we may.

If we have purged the original (retention lapsed) and only the derivative remains, the export note explicitly says so — clients are told once at the export step.

Decision — sticker & animated emoji handling

TGS is gzip-compressed Lottie. We preserve it as-is and apply zstd around it for further savings (≈10-15 %).
Video stickers (WebM VP9) are already well compressed. We do not re-encode.
Static WebP stickers stay WebP.
Custom emoji (custom_emoji_id) are also media files; the same rules apply but they're orders of magnitude smaller, so we still dedupe by file_unique_id.

Display in the Mini App: animated TGS renders via lottie-web, video stickers via <video> tags with autoplay loop muted playsinline, custom emoji via Telegram's own tg-emoji markup when available, otherwise inline rendering.

Decision — bandwidth-aware delivery

Mini App fetches the optimised derivative by default. The export ZIP fetches the original. A ?orig=1 query string is honoured only after an authenticated owner / archive flow has confirmed the request — the URL is signed and short-lived (ADR — security model).

Decision — observability

Two Prometheus metrics:

helios_media_compression_bytes_saved_total{class} — counter.
helios_media_compression_cpu_seconds_total{class} — counter.

A Grafana panel displays the cumulative savings; a per-class breakdown shows where compression is paying off.

Consequences

Disk usage stays roughly flat as new clients arrive — dedupe handles them.
Per-class codec choice yields 60-80 % savings on photos vs the source JPEGs.
CPU cost is bounded by parameter — owner can throttle the compactor during peak load.

Alternatives considered

JPEG-XL: better than AVIF in some benchmarks; lack of mobile decoder support is the blocker (Safari/iOS).
Single zstd-everything: loses huge wins on photo/video.
Re-encode video stickers: they're already heavily optimised by Telegram; re-encoding wastes CPU.