WEPPcloud
UserSummary Docs

← Back to usersum index

WEPP Runner Traceability Hardening (Hillslope + Watershed)

Status: Done (2026-04-27, post-closure addendum updated 20:07 UTC) Timezone: UTC

Overview

This package hardens WEPP run traceability so operators can distinguish model faults from infrastructure faults quickly and with high confidence. It targets production pain observed on 2026-04-27, including watershed stderr close failures (OSError: [Errno 116] Stale file handle) and missing run-context provenance in non-hillslope execution logs.

The work is intentionally a single rollout: startup-context parity, binary identity logging, and prolonged D-state watchdog logging land together with one validation gate.

Objectives

  • Add startup traceability parity for run_watershed to match the practical context currently emitted by run_hillslope (cmd, run/error files, run directory, attempt context).
  • Add binary identity logging (resolved executable path + stable hash/provenance fields) so every run log can be tied to an exact binary artifact.
  • Add bounded D-state watchdog logging for long-running/stuck Linux process states in run_hillslope and run_watershed.
  • Harden close-path logging in watershed runs so log-close infrastructure errors are explicit and classified.
  • Add targeted regression tests for traceability output and watchdog behavior.
  • Update operator-facing docs for new log fields and expected interpretation.

Scope

Included

  • wepp_runner/wepp_runner.py instrumentation for:
    • run_hillslope (binary identity + D-state watchdog integration)
    • run_watershed (startup trace parity + binary identity + D-state watchdog + close-path diagnostics)
  • Shared helper(s) in wepp_runner/wepp_runner.py for binary identity and watchdog telemetry.
  • Targeted tests under tests/wepp_runner/ for:
    • startup context logging
    • binary hash/provenance logging
    • watchdog emission behavior and no-op behavior when disabled/inapplicable
  • Documentation updates for traceability operations and troubleshooting.
  • Post-closure operator-directed release addendum:
    • build wepp_260427 + wepp_260427_hill from /workdir/wepp-forest
    • keep src/makefile -g/-fbacktrace gfortran flags as permanent traceability enhancement
    • vendor release binaries + updated wepp-forest changelog into wepppy
    • document dirty-cycle guidance for capacity include files:
      • src/pmxelm.inc
      • src/pmxhil.inc
      • src/pmxpln.inc
      • src/pntype.inc

Explicitly Out of Scope

  • run_flowpath.
  • Single-storm paths (run_ss_batch_hillslope, run_ss_batch_watershed, and related single-storm templates/runners).
  • WEPP model/science behavior changes.
  • Additional WEPP binary/science changes beyond the operator-requested wepp_260427 release addendum above.
  • RQ route/schema contract changes outside existing runner exception propagation behavior.

Stakeholders

  • Primary: WEPPcloud operators and maintainers triaging production run failures.
  • Reviewers: wepp_runner maintainers and RQ/operations maintainers.
  • Security Reviewer: Optional (not required by default triage; invoke if implementation expands attack surface).
  • Informed: Incident responders, release managers, and ablation-campaign maintainers.

Success Criteria

  • run_watershed emits a startup trace line with command and run-path context comparable to run_hillslope.
  • run_hillslope and run_watershed logs include binary identity fields sufficient to tie execution to an exact artifact.
  • Prolonged D-state watchdog logging is available, bounded, and does not break normal runs.
  • Watershed close-path failures are logged with explicit infrastructure context (instead of opaque generic failure signatures).
  • Targeted regression tests cover the new traceability behavior and pass.
  • Docs explain new log fields, watchdog toggles, and triage interpretation.

Dependencies

Prerequisites

  • Current run_hillslope trace format in wepp_runner/wepp_runner.py as baseline behavior.
  • Incident context from 2026-04-27 production failures (NFS stale-handle close-path and long-run diagnostics).

Blocks

  • Faster incident disposition for ambiguous "model vs infrastructure" run failures.
  • Reliable binary provenance tracking in production run diagnostics.
  • Future automated failure classifiers that depend on canonical run-trace fields.

Timeline Estimate

  • Expected duration: 2-4 focused sessions
  • Complexity: Medium
  • Risk level: Medium (touches production execution/logging path)

Security Impact and Review Gate

  • Security impact triage: low
  • Dedicated security review required: no
  • Triage rationale: Scope is observability and error-traceability in existing runner subprocess flows. No new public routes, auth, secrets handling, upload surfaces, or expanded external egress are introduced.
  • Security review artifact: N/A

Hardening and Callus Softening (Required for incident/remediation packages)

  • Failure signature(s):
    • OSError: [Errno 116] Stale file handle surfaced during watershed stderr close (_log.close()), causing job failure classification ambiguity.
    • Watershed runs lacked startup trace parity (cmd and run-file context), slowing binary/input attribution during incidents.
    • No built-in D-state telemetry for long-running/stuck executions.
  • Related prior hardening efforts:
  • Health signals:
    • Incident triage can identify binary, command, and run-path context from a single *.err file.
    • Watershed close-path failures are explicitly classified with infrastructure context.
    • Watchdog log markers appear for prolonged D-state conditions and are absent on healthy runs.
  • Danger signals:
    • Excessive log noise or heavy per-line hashing overhead.
    • Non-deterministic hash fields that cannot be correlated across runs.
    • Watchdog behavior causing subprocess interference or false positives.
  • Observation window: 14 days after merge/deploy.
  • Temporary calluses introduced:
    • Optional watchdog polling via env configuration; keep default conservative and revisit after observation window.
  • Callus softening hypothesis (if applicable):
    • If watchdog produces low-value noise with no operational signal during the observation window, reduce default frequency or disable by default while retaining opt-in diagnostics.

References

  • wepp_runner/wepp_runner.py - current runner implementations and logging behavior.
  • wepp_runner/AGENTS.md - binary lifecycle and observability expectations.
  • docs/binary-lifecycle.md - binary vendoring/provenance context.
  • tests/wepp_runner/test_run_hillslope_retries.py - existing hillslope traceability test baseline.
  • PROJECT_TRACKER.md - package lifecycle tracking.

Deliverables

  • Updated runner instrumentation in wepp_runner/wepp_runner.py.
  • Targeted regression tests in tests/wepp_runner/.
  • Updated operator/developer docs describing new traceability fields and watchdog behavior.
  • Package tracker and ExecPlan artifacts with validation notes.

Follow-up Work

  • Extend traceability parity to currently out-of-scope run methods (run_flowpath, single-storm runners) in a separate package, if needed.
  • Evaluate whether new fields should be mirrored into structured status-channel metrics.

Closure Notes

Closed 2026-04-27 19:27 UTC.

Implemented the single-rollout scope in wepp_runner/wepp_runner.py for continuous run_hillslope and run_watershed only. run_watershed now logs startup context parity (runs_dir, run_file, err_file, cmd, attempt=1/1), both in-scope runners log cached binary identity fields, and both use a Linux best-effort D-state watchdog with bounded env-configurable emissions. Watershed cleanup now classifies close-path I/O failures, including stale_file_handle for errno=116.

Targeted regression coverage landed in tests/wepp_runner/test_traceability_hardening.py for watershed startup parity, binary identity fields and fallback behavior, watchdog emit/no-emit behavior, and stale-handle close diagnostics.

Validation recorded at closure:

  • wctl run-pytest tests/wepp_runner --maxfail=1 - 17 passed, 2 warnings.
  • wctl run-pytest tests/wepp/test_wepp_runner_outputs.py tests/wepp_runner --maxfail=1 - 18 passed, 2 warnings.
  • Documentation lint and patch hygiene are recorded in tracker.md.

Post-closure addendum (2026-04-27 20:07 UTC):

  • Built /workdir/wepp-forest/release/wepp_260427 and /workdir/wepp-forest/release/wepp_260427_hill using tools/build_wepp_dated_release.sh with TARGET_TAG=260427 COMPILER=/usr/bin/gfortran.
  • Preserved src/makefile gfortran traceability flags -g -fbacktrace (with -O2 retained) as permanent default-lane behavior.
  • Updated /workdir/wepp-forest/change-log.md top entry for wepp_260427 with hashes, validation evidence, dirty include-file guidance, and explicit provenance marker 6bb872ca (dirty tree).
  • Vendored release artifacts into:
    • wepp_runner/bin/wepp_260427 (sha256=5ba0b48fbdaca702e05205daf4b9ab674ba65247949b887c8af74ed4eb8c1b62)
    • wepp_runner/bin/wepp_260427_hill (sha256=bf9e54686580598090840b6225f58d4f9361f7df20884c7aca440bcb8a954826)
  • Synced vendored changelog copy at wepppy/weppcloud/routes/usersum/vendor/wepp-forest/change-log.md.

Review/disposition addendum (2026-04-27 20:12 UTC):

  • Dispatched reviewer and qa_reviewer agents.
  • Resolved in-scope release-provenance finding via dirty-tree changelog provenance annotation.
  • Concurrent RQ-route/schema findings reported by subagents were classified as outside this package addendum scope and deferred to their owning package stream.

Post-closure review/disposition update (2026-04-27 19:41 UTC):

  • reviewer and qa_reviewer reported medium risks for binary identity cache staleness and watershed close-path exception precedence.
  • Both findings were dispositioned in code with added regression coverage in tests/wepp_runner/test_traceability_hardening.py.
  • Revalidation after dispositions:
    • wctl run-pytest tests/wepp_runner --maxfail=1 - 20 passed, 2 warnings.
    • wctl run-pytest tests/wepp/test_wepp_runner_outputs.py tests/wepp_runner --maxfail=1 - 21 passed, 2 warnings.