ADR-0072: Double the single-OFE continuous hillslope timeout
Date: 2026-09-23
Status
Accepted, 2026-09-23.
Context and evidence
The production run indistinguishable-keep failed during its 1000-year
stochastic simulation. Parent job d280184e-0cb4-420b-bf60-10f227533ce6
reported failed child 3a895113-93b7-48f8-967e-1d65e2c3ae13 at
2026-09-23 20:34:56 UTC. Its traceback identifies subprocess.TimeoutExpired
after 60 seconds in run_hillslope, for wepp_id=55, using
wepp_dcc52a6_hill. All four attempts timed out; the last output of attempts
three and four reached 20 September of year 1000. This establishes a
per-attempt execution timeout, not a simulation-year limit or RQ job timeout.
Evidence: production job tree.
Read-only SSH checks confirmed host wepp1, checkout /workdir/wepppy,
the run directory in host and worker views, and running rq-worker,
rq-worker-batch, rq-engine, and weppcloud services. Production source
readback confirmed the 60-second service constant.
Decision
WeppRunService shall pass a 120-second per-attempt timeout for single-OFE
continuous hillslope execution, doubling _CONTINUOUS_HILLSLOPE_TIMEOUT_S
from 60 to 120. The existing 300-second MOFE timeout, runner retry count,
standalone runner default, and RQ job timeouts retain their current values.
The scope is the requested literal timeout adjustment. No model inputs, output schemas, persistence, permissions, or queue dependencies change.
Decision provenance
- Venue: operator request in Codex chat, 2026-09-23 (America/Los_Angeles).
- Participants: requesting operator and Codex; Jackson N.'s incident report was quoted by the operator.
- Decision owner: requesting operator, who explicitly requested identifying the triggered timeout and doubling it.
- Implementer: Codex.
- Rationale: give the long simulation twice the execution budget at the confirmed failing boundary with a single reversible value change.
Alternatives considered
- Increase the RQ timeout: rejected because the subprocess timeout fired.
- Change retries, MOFE limits, or standalone runner defaults: outside the confirmed failing setting and requested scope.
- Infer a 1000-year model limit: unsupported by the failure evidence.
Validation and rollout
Validate by source readback and Python syntax parsing, following the literal config-edit policy; no tests are added or rewritten to assert the value. This repository change does not deploy to production or rerun the project. A successful 1000-year simulation remains unverified until rerun after rollout.
Risks and rollback
An actually stalled single-OFE process can consume twice the execution budget per attempt. Revert the constant to 60 if the longer allowance causes unacceptable worker occupancy. A repeated timeout requires fresh diagnosis; 120 seconds is not a guarantee of completion.