WEPPcloud
UserSummary Docs

← Back to usersum index

WEPP Interchange Dependency Race Guard

Status: Closed (2026-04-29) Timezone: UTC

Overview

This package addresses an intermittent WEPP pipeline failure on wepp1 where _build_hillslope_interchange_rq failed while committing H.wat.parquet.tmp because _post_watershed_interchange_rq started concurrently and touched the same interchange directory. The objective is to remove this race by tightening RQ dependencies instead of adding timing delays.

Objectives

  • Serialize _post_watershed_interchange_rq after _build_hillslope_interchange_rq in WEPP pipelines that can run both paths.
  • Preserve existing pipeline behavior and contracts outside the dependency ordering fix.
  • Add regression tests proving the dependency edge in all affected enqueue helpers.
  • Execute independent sub-agent implementation and QA/correctness reviews with recorded artifacts.

Scope

This package is scoped to queue dependency wiring and required regression/docs updates for the confirmed race condition.

Included

  • Queue wiring in wepppy/rq/wepp_rq_pipeline.py for:
    • enqueue_wepp_pipeline
    • enqueue_wepp_noprep_pipeline
    • Dependency-identity verification in watershed-only helpers:
      • enqueue_watershed_pipeline
      • enqueue_watershed_noprep_pipeline
  • Regression coverage updates in tests/rq/test_wepp_rq_pipeline.py.
  • Queue dependency contract updates in wepppy/rq/job-dependencies-catalog.md and graph regeneration if needed.
  • Sub-agent execution/review prompts and artifacts for code review + QA review.
  • Package lifecycle docs (package.md, tracker.md, active ExecPlan, PROJECT_TRACKER.md).

Explicitly Out of Scope

  • Changes to interchange parser/writer internals (wepppy/wepp/interchange/*) beyond ordering safeguards.
  • Changes to operational host tooling or runbook automation.
  • Broader NoDb/RQ architecture redesign beyond this specific dependency race.

Stakeholders

  • Primary: RQ/WEPP pipeline maintainers and production operators.
  • Reviewers: reviewer and qa_reviewer sub-agents (independent passes).
  • Security Reviewer: required (queue-wiring attack-surface class).
  • Informed: wepp1 operators tracking run reliability for short watershed runs.

Success Criteria

  • _post_watershed_interchange_rq no longer starts before _build_hillslope_interchange_rq in affected pipelines.
  • Regression tests cover the new dependency edge(s) and pass.
  • wctl check-rq-graph passes and dependency docs are synchronized.
  • Sub-agent code review and QA review artifacts are completed with no unresolved medium/high findings.
  • Security review artifact is completed with gate status pass.

Dependencies

Prerequisites

  • Existing queue helper patterns in wepppy/rq/wepp_rq_pipeline.py.
  • Existing pipeline tests in tests/rq/test_wepp_rq_pipeline.py.
  • Queue graph tooling (wctl check-rq-graph, tools/check_rq_dependency_graph.py).

Blocks

  • None currently identified.

Timeline Estimate

  • Expected duration: 1-2 focused sessions.
  • Complexity: Medium.
  • Risk level: Medium-High (production queue topology change).

Security Impact and Review Gate

  • Security impact triage: high
  • Dedicated security review required: yes
  • Triage rationale: queue dependency wiring changes can alter execution ordering and failure boundaries in production worker paths.
  • Security review artifact: docs/work-packages/20260428_wepp_interchange_dependency_race_guard/artifacts/2026-04-28_security_review.md

Hardening and Callus Softening (Required for incident/remediation packages)

  • Failure signature(s):
    • FileNotFoundError: ... H.wat.parquet.tmp -> H.wat.parquet
    • Failing job observed: fd1cb34d-c6fc-4bea-a785-b2a094ab24bc (runid=predictive-refectory, 2026-04-28 21:42 UTC).
  • Related prior hardening efforts:
    • docs/standards/hardening-lifecycle-standard.md
    • docs/work-packages/20260428_build_soils_rq_stale_cache_guard/package.md
  • Health signals:
    • No recurrence of interchange tmp-file commit failures for overlapping hillslope/watershed stages.
    • Dependency graph shows explicit _post_watershed_interchange_rq dependency on hillslope interchange.
  • Danger signals:
    • New stalled/deferred jobs caused by incorrect dependency fan-in.
    • Regression in legacy pathways that expect watershed-only execution.
  • Observation window: first production reruns on short/small watersheds after deployment.
  • Temporary calluses introduced: none planned (ordering fix intended to be durable).
  • Callus softening hypothesis (if applicable): not applicable.

References

  • wepppy/rq/wepp_rq_pipeline.py - queue dependency definitions.
  • wepppy/wepp/interchange/versioning.py - interchange directory refresh behavior.
  • tests/rq/test_wepp_rq_pipeline.py - pipeline dependency regression tests.
  • wepppy/rq/job-dependencies-catalog.md - documented enqueue dependency contract.
  • docs/prompt_templates/codex_exec_plans.md - ExecPlan authoring standard.

Deliverables

  • Queue dependency fix + regression tests.
  • Updated queue dependency catalog/graph artifacts.
  • Sub-agent review artifacts:
    • artifacts/2026-04-28_code_review.md
    • artifacts/2026-04-28_qa_review.md
    • artifacts/2026-04-28_security_review.md
  • Updated package lifecycle docs and tracker states.

Closure Notes (2026-04-29 UTC)

Delivered

  • Added explicit dependency fan-in so _post_watershed_interchange_rq waits for both:
    • _post_run_cleanup_out_rq
    • _build_hillslope_interchange_rq
  • Applied in:
    • enqueue_wepp_pipeline
    • enqueue_wepp_noprep_pipeline
  • Added regression tests that lock dependency identity for all four helpers that enqueue _post_watershed_interchange_rq, including watershed-only helpers that correctly remain cleanup-only.
  • Synchronized queue dependency artifacts:
    • wepppy/rq/job-dependency-graph.static.json
    • wepppy/rq/job-dependencies-catalog.md
  • Completed independent review artifacts and dedicated security gate artifact.

Validation Evidence

  • wctl run-pytest tests/rq/test_wepp_rq_pipeline.py --maxfail=1 -> 9 passed, 2 warnings
  • wctl check-rq-graph -> RQ dependency graph artifacts are up to date
  • wctl doc-lint --path wepppy/rq/job-dependencies-catalog.md --path docs/work-packages/20260428_wepp_interchange_dependency_race_guard --path PROJECT_TRACKER.md -> 12 files validated, 0 errors, 0 warnings
  • git diff --check -> pass

Review and Security Gates

  • Code review artifact: artifacts/2026-04-28_code_review.md (pass, no findings)
  • QA review artifact: artifacts/2026-04-28_qa_review.md (closure-ready, one medium finding resolved in-package)
  • Security review artifact: artifacts/2026-04-28_security_review.md (pass, no unresolved medium/high findings)

Residual Risk

  • Accepted low residual risk: no integration-level concurrency replay in this package scope.

Follow-up Work

  • If additional races are observed in other enqueue paths, spin off a dedicated queue-serialization audit package rather than broadening this one.

Kickoff Prompt

  • Main execution prompt: docs/work-packages/20260428_wepp_interchange_dependency_race_guard/prompts/active/execute_wepp_interchange_dependency_race_guard_prompt.md
  • Archived ExecPlan: docs/work-packages/20260428_wepp_interchange_dependency_race_guard/prompts/completed/wepp_interchange_dependency_race_guard_execplan.md
  • Sub-agent prompts:
    • prompts/active/subagent_worker_patch_prompt.md
    • prompts/active/subagent_reviewer_prompt.md
    • prompts/active/subagent_qa_reviewer_prompt.md