ADR-0004 — Userbot resilience and failure-mode taxonomy

Status. Accepted. Date. 2026-05-18.

Context

A userbot account can fail in many distinct ways and the system must recover correctly in each. Conflating them leads to the wrong remediation: a 5-minute FLOOD_WAIT is not a permanent ban; an AUTH_KEY_UNREGISTERED is not recoverable without owner action. Each failure must trigger the right combination of quarantine, hot-swap, alerting and notice.

Equally important: the userbot pool must stay statistically diverse. We cannot let one chat hold five of our userbots while a sixth attempts to join — that is a detectable pattern.

Decision — failure taxonomy

We codify the closed set of failure modes a userbot can hit in core/policies/resilience.py. Each carries an enum value, a severity, a default remediation, and notes. The runtime catches Pyrogram exceptions, maps them to this taxonomy, and routes to a single dispatcher.

Failure	Severity	Auto-remediation	Owner alert
FLOOD_WAIT	low	quarantine for wait_seconds + 5 s jitter, then resume	only if wait_seconds > 1 h
SLOW_MODE_WAIT	low	re-schedule action by wait_seconds	no
AUTH_KEY_DUPLICATED	high	revoke session, quarantine bot, hot-swap	yes
AUTH_KEY_UNREGISTERED	critical	mark unauthorized, hot-swap, owner re-import required	yes
SESSION_PASSWORD_NEEDED	critical	same as above	yes
USER_DEACTIVATED	critical	mark deactivated, drop from pool, hot-swap	yes
USER_DEACTIVATED_BAN	critical	mark banned, drop, hot-swap	yes
PHONE_NUMBER_BANNED	critical	mark banned, drop, hot-swap	yes
PREMIUM_REVOKED	medium	downgrade userbot capability tag, allow but flag	yes
CHAT_WRITE_FORBIDDEN	n/a	irrelevant — read-only invariant means we never write	—
CHANNEL_PRIVATE	medium	drop chat-track, schedule retry, try other userbot	only on Nth retry
CHAT_ADMIN_REQUIRED	medium	same as above	no
USER_NOT_PARTICIPANT	medium	userbot was kicked from a chat — retry with another userbot after cool-off	no
MSG_ID_INVALID / MESSAGE_ID_INVALID	low	resync chat history	no
RPC_CALL_FAIL / INTERNAL_RPC_ERROR	low	retry with exponential backoff	no
NETWORK_MIGRATE_X / FILE_MIGRATE_X	low	DC switch, transparent	no
CONNECTION_DROPPED	low	reconnect with backoff	only after connection_alert_threshold consecutive drops
TIMEOUT	low	retry, then escalate	escalate after 3 consecutive
PROXY_FAILED	medium	rotate proxy, retry	yes if no remaining proxies
UNHANDLED	high	quarantine for 60 s, log full traceback, owner alert	yes

The dispatcher always:

1. Stops in-flight tasks for the affected userbot (pool.quarantine(userbot_id, reason)).
2. Hot-swaps subscriptions to the next best-fit userbot for active chats, respecting the anti-detection cap (ADR-0005).
3. Emits userbot.failure_detected and (for critical ones) userbot.requires_reauth.
4. Writes an audit log row with severity from the table.
5. Updates the userbot's health score (decay rate per severity).

Decision — health scoring

pool/health.py keeps a per-userbot float in [0, 1]:

• Decay per failure: low −0.05, medium −0.15, high −0.30, critical −1.0.
• Recovery: +0.01 per uninterrupted hour of successful operation.
• A bot with score < pool.health.quarantine_threshold is removed from rotation until it recovers above pool.health.rotation_threshold.

Both thresholds are parameters. Premium accounts get a separate, dedicated pool tier with its own thresholds.

Decision — owner re-import flow

When a userbot is marked unauthorized, deactivated or banned:

1. Existing chat tracks remain — they migrate to the hot-swap target.
2. The owner sees a notification + the userbot row in the Mini App badged re-import needed.
3. Re-import uses the existing importer pipeline (apps/userbot/auth/) and reattaches the same userbot_id, so persona, history, and assigned tracks survive.

Decision — premium revocation

A userbot losing Premium loses the ability to handle Premium-only flows (faster polling cadence is fine; access to certain premium emoji rendering on outgoing messages is not used — we are read-only). The bot is flagged but stays in rotation.

Consequences

• The taxonomy is the single mapping from Pyrogram exception → action. New exception classes get a one-line entry; no other code changes.
• Owner action is required only for the critical bucket; the rest is self-healing.
• The hot-swap path is exercised in tests/resilience/ for every taxonomy entry.

Alternatives considered

• Per-call try/except everywhere. Rejected — drift between sites, no audit.
• "Always retry, always quarantine". Rejected — wastes time on permanent failures, drains the pool.

See also

• ADR-0005 (anti-detection chat-join policy)
• core/policies/resilience.py
• apps/userbot/pool.py
• tests/resilience/