ADR-0050: Opt-in shared GridMET Redis admission
Queue deadline and transient Redis failure decisions are superseded by ADR-0061.
Status: Accepted; implemented and Forest-validated 2026-09-07 Date: 2026-09-07
Context
The multiple-interpolated downloader has a four-process pool per build. Point requests have no shared admission. Independent workers therefore multiply GridMET traffic, and the existing three attempts with 5/10-second backoffs do not coordinate that traffic. A shared opt-in boundary must preserve direct library use and existing climate output contracts.
Decision
Use one Redis FIFO admission pool for point and gridded attempts, in the
existing RedisDB.LOCK, with explicit immutable configuration propagation.
Ratify the package's initial Forest values for implementation and live testing:
| Parameter | Previous behavior | Enabled initial value |
|---|---|---|
| Shared limit | No shared cap | 4 live permits |
| Admission deadline | No admission wait | 900 seconds across all retries |
| Lease duration | No distributed lease | 300 seconds with background renewal |
| Waiting-ticket TTL | No queue | 60 seconds since waiting heartbeat |
| Polling | No queue polling | 0.25 seconds with 0.5–1.5 jitter multiplier |
| Operational namespace | None | wepppy:gridmet:admission:v1 |
| Development Compose default | No admission | Disabled |
Redis connect/socket timeouts are two seconds each with automatic retries off.
Renewal runs every min(lease / 3, 5) seconds. Configuration requires
lease > 3 * (renewal_interval + 2) and
queue_ttl > 3 * (1.5 * poll_interval + 2). Enabled HTTP clients also require
lease > max(connect_timeout, read_timeout) + renewal_interval + 2.
The durable contract defines
state, public None semantics, policy agreement, and safety limits.
Decision Provenance
Decision Venue: Roger Lew/Codex task session, 2026-09-07 UTC; exact request time
was not captured.
Participants Present: Roger Lew and Codex.
Decision Owner(s): Roger Lew authorized execution of the GridMET Redis admission
plan and selected Redis/explicit opt-in; Codex ratifies these implementation
values within that scope. Roger Lew/WEPPcloud operators own subsequent tuning.
Implementer(s): Codex agents executing the work package.
Change Summary
When explicitly enabled by orchestration, all runtime GridMET HTTP attempts
share four live permits rather than separate uncoordinated process limits.
The existing local pool of four, point (10, 60) and grid (10, 120) connect/
read timeouts, three HTTP attempts, payload limits, and result contracts remain.
Admission adds bounded queue scheduling, lease renewal, and explicit failures.
Public client admission=None and absent/false enable values retain zero Redis
construction and I/O. Environment resolution occurs only at orchestration.
Rationale
Four preserves the earlier per-build ceiling while making that starting policy aggregate. It is a conservative operational starting value, not a measured upstream capacity or optimal throughput claim. A 900-second shared deadline allows queued climate work to progress without multiplying that budget by three attempts; retries and their backoff consume the same deadline.
A 300-second lease exceeds both 60/120-second read inactivity timeouts and allows frequent five-second renewal, including while a request blocks waiting for a chunk. Sixty-second ticket TTL removes dead waiters substantially sooner than a holder lease, while quarter-second polling makes admission responsive. The timing inequalities reject configurations unable to tolerate multiple bounded Redis refresh opportunities. The HTTP inequality is conservative about individual blocking calls and is not a total transfer-time bound.
Alternatives Considered
- Retain only the local pool: does not limit point requests or aggregate workers.
- Shared counter/unexpiring ownership: lacks FIFO/position and leaks capacity after process death.
- Enable Redis implicitly inside public functions: breaks literal
Noneand adds a deployment dependency for direct library consumers. - Renew only after each response chunk: expires healthy holders during a blocked read, so use background renewal plus validity checks.
- Claim expired leases fence upstream requests: GridMET does not honor tokens; document the limitation instead of asserting an unsupported socket ceiling.
- Add a transport daemon or dependency: unnecessary to establish the bounded cooperative admission contract in this package.
Consequences
FIFO improves fairness between attempts, but slow streams still occupy capacity while live. Redis availability becomes required only for enabled calls. Conflicting policy values in a live shared namespace fail explicitly rather than silently permitting inconsistent limits. Independent namespaces and configuration left disabled bypass the aggregate pool by design; deployments must propagate one policy to every participating worker.
Redis guarantees the count of live leases, not hard upstream fencing after process pauses, DNS stalls, or partition. Detected lease loss stops subsequent HTTP work and fails the result; closing an already blocked response can wait for control to return. Read inactivity bounds do not bound total trickle-stream wall time. This limitation must remain visible in operational evidence.
Evidence
- Work package and initial values
- Execution and Forest acceptance plan
- Earlier HTTP retry/concurrency decision
- Existing timeouts and retry constants:
wepppy/climates/gridmet/acquisition.py. - Completed automated and Forest evidence: runtime tests, independent reviews, cross-container and public-client probes, and rollback passed on 2026-09-07.
Risk and Rollback Notes
Observe queue depth, wait exhaustion, active occupancy, Redis failures, lease loss, upstream timeout/retry rates, and completed-request throughput before changing parameters. Operators own tuning; revise this ADR and the contract, configuration/runbook docs, tests, and package values before deploying changed policy. Drain/recreate all participating clients together for policy changes.
Rollback sets GRIDMET_REDIS_ADMISSION_ENABLED=false, recreates the same Forest
workers, and verifies new processes resolve None. This restores prior
uncoordinated acquisition and removes Redis availability from that path. Never
flush Redis or delete a namespace with live owners. Revert code if required.
Implementation Notes
Configuration is immutable and pickle-safe, Redis connections are process-local, and atomic scripts use server time and one cluster hash slot. Each attempt owns a unique ticket/token, releases before validation/backoff, and retries at the FIFO tail. Independent review of this contract checkpoint precedes runtime edits. Forest tests use isolated namespaces and never run a batch; subsequent registry build, openwepp.org deployment, and batch validation are separate.