WEPPcloud

← Back to help

ADR-0109: Bound RQ webhook tracking and delivery

Status: Accepted 2026-10-28 19:56 UTC; local implementation acceptance passed Date: 2026-10-28

Context

Per-submission callbacks must not consume computation capacity or retain unlimited receiver-controlled data. These are accepted initial operational ceilings, not measured capacity claims. Existing behavior has no application webhook service.

Decision

Policy Accepted value Rationale
Total attempt deadline 10 seconds, including DNS/connect/TLS/headers Operator's suggested short deadline; killable isolated attempt process
Attempts 3 total, including interrupted attempts Limits duplicates and receiver amplification
Retry delays 30 seconds after attempt 1, 120 after attempt 2 Brief outage tolerance without tight loops
Retry-After cap 300 seconds; use max(normal delay, bounded Retry-After) Respect receiver pressure within bounded delivery age
Delivery age 1 hour after event commit Bound backlog; no fresh attempt at/after expiry
Attempt claim lease 30 seconds; no renewal while HTTP runs Longer than deadline, bounded crash recovery; fenced owner token
Tracking window 30 days from admission Accommodate long workflows; expiry means outcome_unavailable
Sanitized audit and dedup tombstones 30 days after terminal tracking/delivery state Incident investigation and request replay protection
Raw destination removal Immediately on terminal delivery/unavailable state Minimize capability retention; backups follow existing restricted policy
URL / event / response headers 2048 UTF-8 bytes / 4096 bytes / 16 KiB Bound untrusted input and payload; response body never read
Pending roots 100 per verified principal, 10000 global Prevent one principal monopolizing tracking
Tracking storage 64 MiB per principal, 512 MiB global logical serialized bytes Includes intent/member/event/audit records; count before allocation
Members / edges per root 100000 / 500000 Bounded graph work; over-limit tracking becomes unavailable, computation continues
Attempt audit At most 3 rows per event Match attempt cap
Dispatcher concurrency 4 attempts global, 1 per principal Fair round-robin due-root selection, computation worker isolation
Evaluator page 256 members/edges, 100 ms scheduling slice Yield across roots; traversal is not run in workers
Reconciliation interval 30 seconds for subscribed due roots Recover missed signals with bounded indexed reads
Computation hook budget 50 ms total incremental bookkeeping per lifecycle call, no internal retries Failure suppresses closure, not computation
Admission pressure Retry-After 60 seconds Avoid tight retry loops
Idempotency key 16–128 ASCII token characters Bound receipt indexing; avoids new receipt lookup API
Noninterference acceptance p95 completion/next-job latency increase <=5% and <=50 ms per transition Both limits required, at least 1000 transitions against disabled baseline

All storage counts include terminal retained records, not just active roots. Reserve space for the bounded terminal summary at admission. When capacity cannot retain new member evidence, stop recording growth for that root, set incomplete tracking when possible, and never stop computation. Missing producer closure prevents false events if even that marker fails. Global logical bytes are not RSS: actual Redis memory and latency must be measured before activation, with admission disabled if these bounds cannot preserve tasking. Do not evict RQ keys to free notification capacity.

Retain only minimal membership while evaluation remains active. On tracking closure, delete member/edge receipts and raw intent, retaining bounded sanitized outcome, event/attempt summary and allowlisted credential-free idempotency receipt. Never retain browse tokens or their expiry as a replay response; freshly authorized retries use the existing mint helper as specified in the contract. Quotas still count retained bytes. Do not delete the destination before pending delivery has ended. Admission-pending records share the 30-day tracking window; unavailable tombstones prevent same-key reenqueue. No new attempt may extend beyond the one-hour age: shorten its deadline to remaining age. Dispatch pause does not reset time or attempt counts.

Decision Provenance

Decision Venue: user-directed kickoff execution, 2026-10-28, UTC.
Participants Present: requesting operator and Codex (asynchronous conversation).
Decision Owner(s): requesting operator; exact matrix approved 2026-10-28 19:56 UTC.
Implementer(s): Codex; runtime implementation follows the approved checkpoint.

Accepted direction: short independent delivery, limited retries, and suggested 10-second budget. The operator explicitly approved all exact values with the RQ-WH-01 matrix. No model parameterization changes.

Rationale and alternatives

Unlimited retention/retries would convert receiver outages into shared-resource pressure. A single attempt would make transient faults unnecessarily lossy. Longer tracking increases Redis residency; shorter tracking can miss large batches. Start with explicit bounds and measure representative graphs before activation. A separate durable database would add a transaction/recovery boundary; existing Redis is the smallest candidate and explicitly does not promise durable delivery after Redis loss.

Evidence

Source and probe evidence shows the initial publication gaps and actual RQ dependency behavior. Final local validation records passing deadline, quota/recovery, actual workflow and verified-sender performance evidence under canonical worker counts. The contract and acceptance matrix define the required tests; values must be revised through this ADR if measured bounds fail.

Risk and rollback notes

Limits can deny new callbacks or leave long/large workflows without notifications; computation behavior is preserved. Disable admission first, drain accepted tracking, retain sanitized audit, and keep worker instrumentation through producer drain. Dispatch pause is an emergency control, not retention extension. Local implementation/integration is authorized by the checkpoint approval; production rollout remains excluded.

Implementation budget allocation (2026-10-28)

The total 50 ms hook limit is unchanged. Receipt construction has at most half that budget and streams JSON with deadline checks rather than recursively copying large dependency tuples. Notification connections use 5 ms connect/read timeouts and no retry. Exhaustion suppresses closure without changing computation.

Actual model acceptance found a short-lived adopted multiprocessing helper after workhorse exit. A subscribed supervisor may spend at most 25 ms of the same total budget waiting on pidfds, followed by positive child/protected-scheduler checks. It never kills helpers or treats elapsed time as exit. Closure carries one absolute deadline into receipt construction and reserves the remaining half for persistence; worker retirement behavior is preserved. This bounded observation avoids a new reaper service; longer-lived or uncertain producers remain unclosed.

Evaluator scheduling allocation

The existing 100 ms scheduler budget is shared across active evaluators rather than multiplied by the number of roots. Rotate after each bounded generator step. This avoids consecutive per-root bursts observed while investigating failed backlog measurements, preserves fairness and adds no new numeric policy. A running step may complete after the cooperative deadline; workers never traverse the graph.