# RQ Job Dependencies Catalog
> Catalog of RQ entrypoints that enqueue child jobs and the dependency graph between them.
> Includes an auto-generated dependency graph section plus analyst-maintained narrative and audit notes.

## Scope
- Modules: `wepppy/rq/*.py` (entrypoints that enqueue other jobs).
- rq-engine routes that enqueue RQ jobs directly: `wepppy/microservices/rq_engine/batch_routes.py`, `wepppy/microservices/rq_engine/bootstrap_routes.py`, `wepppy/microservices/rq_engine/geneva_routes.py`, and `wepppy/microservices/rq_engine/run_sync_routes.py`.
- rq-initiated Flask handlers that enqueue RQ jobs: `wepppy/weppcloud/routes/**/*.py` and `wepppy/weppcloud/bootstrap/*.py`.

## Uploaded land-use builds

The normal `build-landuse` endpoint uses the existing default
`project_rq.build_landuse_rq` job family. An uploaded raster adds a
preallocation check that stages the raw and aligned inputs and a job-bound
receipt. This submission adds no edge or worker queue. The worker checks the
receipt and both directory leases before applying the upload. Previously
enqueued jobs without upload arguments continue to build their existing
aligned inputs. Setting batch inputs remains synchronous. Batch leaves
validate and align retained raw inputs under the existing directory leases
when their own watershed becomes available.

## Declarative DEM slice

The private `wepppy.declarative.adapter.submit` enqueues `execute_dem` on one
`declarative-dem-<submission UUID hex>` queue. This adapter calls shared
`project_rq.fetch_dem_rq` synchronously and never waits for queued children.
After science settlement the operator invokes `adapter.finalize`, which enqueues
one `finalize_dem` job on the same queue; it calls shared `set_run_readonly_rq`.
No RQ dependency edge, scientific retry or ordinary queue consumer is added.
The engine waits outside the single worker. Durable claims prevent replay after
uncertain execution. These enqueue sites are outside the generator's existing
`wepppy/rq`/route scan; this narrative explicitly inventories them.

## Query-engine asset catalog maintenance

`wepppy.rq.query_catalog_rq.enqueue_catalog_refresh` admits one
`refresh_query_catalog_rq` job per exact scope/incarnation on the existing default
queue. Producer requests persist filesystem receipts before admission. Active
queued/started/deferred/scheduled work coalesces; reservations remain through
RQ terminal transition so late invalidations cannot create a second active job.
Auth-actor and correlation metadata use the same Redis transaction as admission;
a WATCH conflict must not leave a metadata-only job hash.
The shipped `query_catalog_refresh` scheduler task selects `default` and passes
that queue explicitly. The scheduler invokes `reconcile_query_catalogs` to admit one active
`inventory_query_catalog_rq` per configured inventory cycle on default and drain
its durable Redis discoveries. The inventory worker streams configured roots,
checkpoints its frontier and fences every publication by current job/cycle.
Scheduler ticks acknowledge discovered scopes atomically with their pending
cursor. Restart replays the current directory prefix with deduplication.
Neither scheduler nor inventory worker executes native catalog scans; inventory
writes no project files. No model dependency edges are added.

Maintenance IDs use `query_catalog_refresh_<canonical UUID>` and generic
metadata. Polling classification precedes deserialization. Every live eligible
default consumer must advertise matching protocol/storage/root/coordination and
`inventory_protocol: 1`, `admission_protocol: 1`, `pilot_protocol: 1`, and
`queue: default` in
`query_catalog_configuration`; missing/incompatible/suspended capacity leaves
durable requests pending. Execution revalidates configuration and incarnation.

The global Redis admission ledger caps outstanding catalog work at seven across
configurations and pilots; reservations and uncertain job records retain capacity.
The explicit operator pilot requires all twelve compatible default consumers, exact owned
scopes, a live observation lease, and immutable retry-inclusive budgets (25 refresh,
two inventory). Ordinary waiting work pauses admission. Pilot inventory enumerates
only approved locators; admitted jobs may drain after pause or expiry.

`QUERY_CATALOG_REFRESH_ENABLED` defaults false for coordinated rollout. Disabled
producer migration preserves synchronous compatibility; enabled Redis/worker
failure must not fall back to inline scanning. Scheduler interval defaults 30 s;
inventory defaults to 100 entry records per streaming checkpoint and 100 scope
records drained per scheduler tick; admission remains 25 scopes per tick.
`QUERY_CATALOG_CYCLE_SECONDS` defaults 86400 as an unproven capacity target;
retry delay starts at 30 s and caps at 3600 s. These are configurable operational
limits, not a freshness guarantee during unavailable storage/workers. The
inventory cursor and cycle errors are operator-visible Redis state; loss resets
bounded discovery against configured roots, while project receipt/attempt files
retain durable intent and evidence. Full cycles include scopes without receipts
so artifact-publication crashes cannot leave metadata stale forever. Discovery
completion is separate from cycle completion: `completed_at` is recorded only
after all discovered scopes acknowledge their pending receipts. Current and
previous cycle errors remain visible without accumulating older cycles. The
scheduler lease is 60 seconds; expired owners cannot overwrite newer cursors.

Enabled mode requires explicit `QUERY_CATALOG_ROOTS_JSON`; missing configuration
fails preflight rather than guessing which legacy namespaces are active. Set the
same configuration in producers, scheduler, and every eligible default consumer.
For a deployment using the three canonical `/wc1` namespaces, the JSON value is:

```json
[
  {"path":"/wc1/runs","namespace":"wc1:runs","coordination":"/wc1/query-engine-coordination/locks","layout":"sharded"},
  {"path":"/wc1/batch","namespace":"wc1:batch","coordination":"/wc1/query-engine-coordination/locks","layout":"batch"},
  {"path":"/wc1/culverts","namespace":"wc1:culverts","coordination":"/wc1/query-engine-coordination/locks","layout":"culverts"}
]
```

Add legacy roots only when that deployment actively serves them. Verify each
configured namespace and the external coordination directory under the actual
runtime identity before enabling. Once configured, missing storage remains an
outage: inventory reports degraded health and retries on the next tick. It never
silently removes an unavailable namespace from the configured inventory.

Use `suspend_catalog_refresh(connection, scope_id)` for intentional operator
suspension and `resume_catalog_refresh` to allow future ticks; ordinary canceled
or failed jobs otherwise remain recoverable. Never purge pending receipt files
or external lifecycle lock inodes as queue cleanup. See the canonical
[query catalog contract](../../docs/schemas/query-engine-catalog-refresh-contract.md)
and [execution package](../../docs/work-packages/20261023_query_catalog_async_refresh/package.md).

## Path Legend
| Token | Meaning |
| --- | --- |
| `wd` | Run working directory (e.g., `/wc1/runs/<prefix>/<runid>`). |
| `runs_dir` | `Wepp.runs_dir` (typically `wd/wepp/runs`). |
| `output_dir` | `Wepp.output_dir` (typically `wd/wepp/output`). |
| `plot_dir` | `Wepp.plot_dir` (typically `wd/wepp/plots`). |
| `export_dir` | `Wepp.export_dir` (typically `wd/export`). |
| `wat_dir` | `Watershed.wat_dir` (watershed workspace; varies by backend). |
| `cli_dir` | `Wepp.cli_dir` (climate .cli directory). |
| `soils_dir` | `Soils.soils_dir` (source soils). |
| `lc_dir` | `Landuse.lc_dir` (landuse/management templates). |
| `omni_dir` | `wd/omni` (OMNI_REL_DIR). |
| `swat_txtinout_dir` | `Swat.swat_txtinout_dir` (SWAT TxtInOut). |
| `dem_dir` | `Ron.dem_dir` (DEM staging). |

## Conventions
- Parent jobs record child ids in `job.meta["jobs:<order>,..."]` for `job_info.py` and `cancel_job.py`.
- `depends_on` uses a single job or a list; when the list is empty, it is passed as `None` (no dependency).
- Required-output dependency edges are strict. A failed prerequisite leaves
  executable downstream work never-started/deferred; ordinary controller
  resubmission cancels and detaches that obsolete graph before replacement.
- Failure tolerance is limited to the finalizers and independent AgFields,
  Omni-contrast, and WBT request-serialization edges enumerated in the SURF-20A
  dependency matrix.
  Those sites use `failure_tolerant_depends_on` (or an explicit
  `Dependency(allow_failure=True)`) plus `release_deferred_job_if_ready` for the
  already-failed-before-registration race.
- WBT controlled policy failures retain their owning exception: they explicitly
  cancel the never-started abstraction receipt rather than leaving it deferred.
- Prereq files list explicit file/dir inputs referenced by the job or immediate controller calls. Optional inputs are marked.
- Paths are shown relative to `wd` unless noted; placeholders like `<topaz>` and `<wepp_id>` indicate per-hillslope files.

## Batch result settlement

Enrolled ordinary/culvert roots freeze selected run IDs and exact source job IDs
before dispatch. Culvert leaves depend on the root dispatch barrier; its finalizer
depends on root and leaves with failure tolerance. Ordinary hillslope/watershed
stages retain the existing root barrier and linked Omni completion edges.
All enrolled execution identities use non-expiring RQ retention.

Child creation/linkage and cancellation share the existing root resource lease.
Cancellation refreshes the tree after acquiring that lease. Publication validates
and builds outside the lease, then rechecks current attempt, execution states and
cancellation before the short manifest/ZIP/summary commit. Polling never builds
artifacts. Exhausted failed Omni dependencies permit only verified blocked deferred
Omni descendants to be canceled and the failure-tolerant batch finalizer released;
foreign or changed graphs remain untouched. See the
[batch outcome contract](../../docs/schemas/batch-outcome-contract.md).

## Scheduled WMesque Synchronization

`docker/scheduled-tasks.yml` routes the opt-in `wmesque_sync` task through
`wepppy.tools.scheduler._enqueue_task` to `wepppy.rq.wmesque_rq.enqueue_sync` on
`batch`. The generic queue parameter appears as `default/untracked` in static
extraction below; the scheduled binding is `batch`. It enqueues one standalone
`sync_wmesque_rq` job with no dependency edges, coalescing queued/running work by
destination. Source and destination are snapshotted into job arguments. Live
forest acceptance job IDs and terminal states are retained in the WMesque work
package. Task remains disabled by default.

<!-- AUTO-GENERATED: RQ DEPENDENCY GRAPH START -->
## Automated Dependency Graph

This section is generated by `tools/render_rq_dependency_graph_docs.py` from
`wepppy/rq/job-dependency-graph.static.json`. Do not edit this block manually.

- Extracted enqueue edges: `154`
- Unique enqueue targets: `98`
- Source globs: `wepppy/rq/*.py`, `wepppy/microservices/rq_engine/*.py`, `wepppy/weppcloud/routes/**/*.py`, `wepppy/weppcloud/bootstrap/*.py`

### `wepppy/microservices/rq_engine/ag_fields_routes.py::_enqueue_job`
- line `229` queue `default` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/microservices/rq_engine/ag_fields_routes.py::_enqueue_watershed_jobs`
- line `296` queue `default` stage `untracked` enqueue `run_ag_fields_watershed_rq` depends_on *(none)*
- line `322` queue `default` stage `untracked` enqueue `run_ag_fields_watershed_suite_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/ash_routes.py::run_ash`
- line `376` queue `default` stage `untracked` enqueue `run_ash_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/batch_routes.py::delete_batch`
- line `194` queue `batch` stage `untracked` enqueue `delete_batch_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/batch_routes.py::run_batch`
- line `116` queue `batch` stage `untracked` enqueue `run_batch_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/bootstrap_routes.py::_enqueue_no_prep_job`
- line `220` queue `default` stage `untracked` enqueue `job_fn` depends_on *(none)*

### `wepppy/microservices/rq_engine/climate_routes.py::build_climate`
- line `743` queue `default` stage `untracked` enqueue `build_climate_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/culvert_routes.py::_enqueue_culvert_job`
- line `218` queue `default` stage `untracked` enqueue `queue` depends_on *(none)*
- line `221` queue `batch` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/microservices/rq_engine/debris_flow_routes.py::run_debris_flow`
- line `148` queue `default` stage `untracked` enqueue `run_debris_flow_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/debug_routes.py::hello_world`
- line `54` queue `m4` stage `untracked` enqueue `hello_world_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/dss_export_routes.py::post_dss_export`
- line `199` queue `default` stage `untracked` enqueue `post_dss_export_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/export_routes.py::_enqueue_ermit_export_job`
- line `280` queue `default` stage `untracked` enqueue `run_ermit_export_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/export_routes.py::_enqueue_features_export_job`
- line `378` queue `default` stage `untracked` enqueue `target_func` depends_on *(none)*

### `wepppy/microservices/rq_engine/fork_archive_routes.py::archive_run`
- line `1740` queue `fork-archive` stage `untracked` enqueue `archive_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/fork_archive_routes.py::fork_project`
- line `1607` queue `fork-archive` stage `untracked` enqueue `fork_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/fork_archive_routes.py::restore_archive`
- line `1848` queue `fork-archive` stage `untracked` enqueue `restore_archive_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/geneva_routes.py::_enqueue_geneva_job`
- line `306` queue `default` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/microservices/rq_engine/geneva_routes.py::_enqueue_geneva_workflow_jobs`
- line `358` queue `default` stage `untracked` enqueue `run_geneva_prepare_hrus_rq` depends_on *(none)*
- line `365` queue `default` stage `untracked` enqueue `run_geneva_build_frequency_panel_rq` depends_on `run_geneva_prepare_hrus_rq`
- line `374` queue `default` stage `untracked` enqueue `run_geneva_run_batch_rq` depends_on `run_geneva_build_frequency_panel_rq`

### `wepppy/microservices/rq_engine/landuse_routes.py::build_landuse`
- line `885` queue `unknown` stage `untracked` enqueue `build_landuse_rq` depends_on *(none)*
- line `949` queue `default` stage `untracked` enqueue `build_landuse_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/landuse_routes.py::modify_landuse_mapping`
- line `1266` queue `default` stage `untracked` enqueue `modify_landuse_mapping_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/landuse_soils_routes.py::build_landuse_and_soils`
- line `64` queue `default` stage `untracked` enqueue `land_and_soil_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/migration_routes.py::_enqueue_migration_job`
- line `179` queue `default` stage `untracked` enqueue `migrations_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/omni_routes.py::_delete_omni_contrasts`
- line `1323` queue `batch` stage `untracked` enqueue `delete_omni_contrasts_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/omni_routes.py::_run_omni`
- line `969` queue `batch` stage `untracked` enqueue `run_omni_scenarios_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/omni_routes.py::_run_omni_contrasts`
- line `1141` queue `batch` stage `untracked` enqueue `run_omni_contrasts_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/openet_ts_routes.py::acquire_openet_ts`
- line `88` queue `default` stage `untracked` enqueue `fetch_and_analyze_openet_ts_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/polaris_routes.py::acquire_polaris`
- line `91` queue `default` stage `untracked` enqueue `fetch_and_align_polaris_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/postfire_debris_flow_routes.py::enqueue`
- line `151` queue `default` stage `untracked` enqueue `upload_dnbr_rq if kind == 'upload_attempt' else run_m3_rq if record.get('model') == 'M3' else run_m1_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/project_config_update_routes.py::update_apply`
- line `703` queue `default` stage `untracked` enqueue `<unknown>` depends_on *(none)*

### `wepppy/microservices/rq_engine/rap_ts_routes.py::acquire_rap_ts`
- line `160` queue `default` stage `untracked` enqueue `fetch_and_analyze_rap_ts_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/rhem_routes.py::run_rhem`
- line `85` queue `default` stage `untracked` enqueue `run_rhem_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/roads_routes.py::_enqueue_roads_job`
- line `60` queue `default` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/microservices/rq_engine/run_sync_routes.py::run_sync`
- line `280` queue `default` stage `untracked` enqueue `run_sync_rq` depends_on *(none)*
- line `288` queue `default` stage `untracked` enqueue `run_sync_rq` depends_on *(none)*
- line `319` queue `default` stage `untracked` enqueue `migrations_rq` depends_on `run_sync_rq`

### `wepppy/microservices/rq_engine/rusle_routes.py::build_rusle`
- line `100` queue `default` stage `untracked` enqueue `build_rusle_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/soils_routes.py::build_soils`
- line `334` queue `default` stage `untracked` enqueue `build_soils_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/swat_routes.py::run_swat`
- line `111` queue `default` stage `untracked` enqueue `run_swat_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/treatments_routes.py::build_treatments`
- line `211` queue `default` stage `untracked` enqueue `build_treatments_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/upload_climate_routes.py::upload_cli`
- line `212` queue `default` stage `untracked` enqueue `upload_cli_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/watershed_routes.py::build_subcatchments_and_abstract_watershed`
- line `1164` queue `default` stage `untracked` enqueue `build_subcatchments_and_abstract_watershed_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/watershed_routes.py::fetch_dem_and_build_channels`
- line `809` queue `default` stage `untracked` enqueue `fetch_dem_and_build_channels_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/watershed_routes.py::set_outlet`
- line `935` queue `default` stage `untracked` enqueue `set_outlet_rq` depends_on *(none)*

### `wepppy/microservices/rq_engine/webhooks.py::enqueue`
- line `214` queue `default` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/microservices/rq_engine/wepp_routes.py::_handle_run_wepp_request`
- line `237` queue `default` stage `untracked` enqueue `q` depends_on *(none)*
- line `240` queue `default` stage `untracked` enqueue `job_fn` depends_on *(none)*

### `wepppy/rq/ag_fields_rq.py::run_ag_fields_watershed_suite_rq`
- line `494` queue `default` stage `untracked` enqueue `run_ag_fields_watershed_rq` depends_on `Dependency(jobs=[previous_job_id], allow_failure=True)`
- line `520` queue `default` stage `untracked` enqueue `finalize_ag_fields_watershed_suite_rq` depends_on `Dependency(jobs=[child_job.id for child_job in scheme_jobs], allow_failure=True)`

### `wepppy/rq/batch_rq.py::run_batch_rq`
- line `744` queue `batch` stage `jobs:0` enqueue `run_batch_hillslopes_rq` depends_on `job_id`
- line `755` queue `batch` stage `jobs:0` enqueue `run_batch_watershed_rq` depends_on `Dependency(jobs=[hillslope_job_id], allow_failure=True)`
- line `780` queue `batch` stage `jobs:1` enqueue `_final_batch_complete_rq` depends_on `Dependency(jobs=[watershed_job.id for watershed_job in watershed_jobs], allow_failure=True)`

### `wepppy/rq/culvert_rq_pipeline.py::enqueue_culvert_batch_jobs`
- line `87` queue `batch` stage `jobs:0` enqueue `tasks.run_culvert_run_rq` depends_on `parent_job.id`
- line `106` queue `batch` stage `jobs:1` enqueue `tasks._final_culvert_batch_complete_rq` depends_on `*child_jobs`, `parent_job`, `tasks.run_culvert_run_rq`

### `wepppy/rq/omni_rq.py::run_omni_contrasts_rq`
- line `928` queue `batch` stage `jobs:contrast` enqueue `run_omni_contrast_rq` depends_on `_failure_tolerant_depends_on(batch_depends)`
- line `943` queue `batch` stage `jobs:finalize` enqueue `_finalize_omni_contrasts_rq` depends_on `_failure_tolerant_depends_on(batch_depends)`

### `wepppy/rq/omni_rq.py::run_omni_scenarios_rq`
- line `733` queue `batch` stage `jobs:0` enqueue `run_omni_scenario_rq` depends_on *(none)*
- line `752` queue `batch` stage `jobs:1` enqueue `run_omni_scenario_rq` depends_on `run_omni_scenario_rq`
- line `771` queue `batch` stage `jobs:2` enqueue `_compile_hillslope_summaries_rq` depends_on `run_omni_scenario_rq`
- line `783` queue `batch` stage `jobs:3` enqueue `_finalize_omni_scenarios_rq` depends_on `_compile_hillslope_summaries_rq`

### `wepppy/rq/project_rq.py::_enqueue_serial_subcatchment_tree`
- line `787` queue `default` stage `untracked` enqueue `build_subcatchments_rq` depends_on `Dependency(jobs=[previous_tail_id], allow_failure=True)`
- line `795` queue `default` stage `untracked` enqueue `abstract_watershed_rq` depends_on `build_subcatchments_rq`

### `wepppy/rq/project_rq.py::fetch_dem_and_build_channels_rq`
- line `1652` queue `default` stage `jobs:0` enqueue `build_channels_rq` depends_on *(none)*
- line `1668` queue `default` stage `jobs:0` enqueue `fetch_dem_rq` depends_on *(none)*
- line `1676` queue `default` stage `jobs:1` enqueue `build_channels_rq` depends_on `fetch_dem_rq`

### `wepppy/rq/project_rq.py::fork_rq`
- line `3133` queue `default` stage `jobs:1` enqueue `_finish_fork_rq` depends_on `Dependency(jobs=[final_wepp_job.id], allow_failure=True)`

### `wepppy/rq/query_catalog_rq.py::_enqueue_inventory`
- line `364` queue `default` stage `untracked` enqueue `inventory_query_catalog_rq` depends_on *(none)*

### `wepppy/rq/query_catalog_rq.py::enqueue_catalog_refresh`
- line `216` queue `default` stage `untracked` enqueue `refresh_query_catalog_rq` depends_on *(none)*

### `wepppy/rq/run_catalog_rq.py::enqueue_sweep`
- line `124` queue `default` stage `untracked` enqueue `sweep_run_catalog` depends_on *(none)*

### `wepppy/rq/swat_rq.py::run_swat_noprep_rq`
- line `118` queue `default` stage `jobs:0` enqueue `_run_swat_rq` depends_on *(none)*

### `wepppy/rq/swat_rq.py::run_swat_rq`
- line `71` queue `default` stage `jobs:0` enqueue `_build_swat_inputs_rq` depends_on *(none)*
- line `75` queue `default` stage `jobs:1` enqueue `_run_swat_rq` depends_on `_build_swat_inputs_rq`

### `wepppy/rq/wepp_rq.py::run_wepp_watershed_noprep_rq`
- line `1113` queue `default` stage `jobs:6` enqueue `sys.modules[__name__]._log_complete_rq` depends_on *(none)*

### `wepppy/rq/wepp_rq.py::run_wepp_watershed_rq`
- line `968` queue `default` stage `jobs:6` enqueue `sys.modules[__name__]._log_complete_rq` depends_on *(none)*

### `wepppy/rq/wepp_rq_pipeline.py::enqueue_log_prep_complete`
- line `90` queue `default` stage `jobs:6` enqueue `tasks._log_prep_complete_rq` depends_on `depends_on`

### `wepppy/rq/wepp_rq_pipeline.py::enqueue_watershed_noprep_pipeline`
- line `926` queue `default` stage `jobs:3` enqueue `tasks.run_ss_batch_watershed_rq` depends_on *(none)*
- line `938` queue `default` stage `jobs:3` enqueue `tasks.run_watershed_rq` depends_on *(none)*
- line `951` queue `default` stage `jobs:4` enqueue `tasks._post_run_cleanup_out_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`
- line `964` queue `default` stage `jobs:4` enqueue `tasks._post_prep_details_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`
- line `978` queue `default` stage `jobs:4` enqueue `tasks._post_make_loss_grid_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`
- line `991` queue `default` stage `jobs:4` enqueue `tasks._post_watershed_interchange_rq` depends_on `tasks._post_run_cleanup_out_rq`
- line `1005` queue `default` stage `jobs:5` enqueue `tasks._post_legacy_arc_export_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`
- line `1017` queue `default` stage `jobs:6` enqueue `tasks._log_complete_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._post_legacy_arc_export_rq`

### `wepppy/rq/wepp_rq_pipeline.py::enqueue_watershed_pipeline`
- line `785` queue `default` stage `jobs:2` enqueue `tasks._prep_watershed_rq` depends_on *(none)*
- line `799` queue `default` stage `jobs:3` enqueue `tasks.run_ss_batch_watershed_rq` depends_on `tasks._prep_watershed_rq`
- line `812` queue `default` stage `jobs:3` enqueue `tasks.run_watershed_rq` depends_on `tasks._prep_watershed_rq`
- line `826` queue `default` stage `jobs:4` enqueue `tasks._post_run_cleanup_out_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`
- line `839` queue `default` stage `jobs:4` enqueue `tasks._post_prep_details_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`
- line `853` queue `default` stage `jobs:4` enqueue `tasks._post_make_loss_grid_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`
- line `866` queue `default` stage `jobs:4` enqueue `tasks._post_watershed_interchange_rq` depends_on `tasks._post_run_cleanup_out_rq`
- line `880` queue `default` stage `jobs:5` enqueue `tasks._post_legacy_arc_export_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`
- line `892` queue `default` stage `jobs:6` enqueue `tasks._log_complete_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._post_legacy_arc_export_rq`

### `wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_noprep_pipeline`
- line `571` queue `default` stage `jobs:1` enqueue `tasks._run_hillslopes_rq` depends_on *(none)*
- line `580` queue `default` stage `jobs:2` enqueue `tasks._build_hillslope_interchange_rq` depends_on `tasks._run_hillslopes_rq`
- line `592` queue `default` stage `jobs:2` enqueue `tasks._build_totalwatsed3_rq` depends_on `tasks._build_hillslope_interchange_rq`
- line `613` queue `default` stage `jobs:3` enqueue `tasks.run_ss_batch_watershed_rq` depends_on `tasks._run_hillslopes_rq`
- line `626` queue `default` stage `jobs:3` enqueue `tasks.run_watershed_rq` depends_on `tasks._run_hillslopes_rq`
- line `639` queue `default` stage `jobs:4` enqueue `tasks._post_run_cleanup_out_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._run_hillslopes_rq`
- line `652` queue `default` stage `jobs:4` enqueue `tasks._post_prep_details_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._run_hillslopes_rq`
- line `667` queue `default` stage `jobs:4` enqueue `tasks._run_hillslope_watbal_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._run_hillslopes_rq`, `tasks._build_hillslope_interchange_rq`
- line `680` queue `default` stage `jobs:4` enqueue `tasks._post_make_loss_grid_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._run_hillslopes_rq`
- line `691` queue `default` stage `jobs:4` enqueue `tasks._post_watershed_interchange_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._build_hillslope_interchange_rq`
- line `704` queue `default` stage `jobs:4` enqueue `tasks._analyze_return_periods_rq` depends_on `tasks._post_watershed_interchange_rq`, `tasks._build_totalwatsed3_rq`
- line `719` queue `default` stage `jobs:2` enqueue `tasks.post_dss_export_rq` depends_on `tasks._build_hillslope_interchange_rq`, `tasks._post_run_cleanup_out_rq`
- line `732` queue `default` stage `jobs:5` enqueue `tasks._post_legacy_arc_export_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._run_hillslope_watbal_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._analyze_return_periods_rq`
- line `745` queue `default` stage `jobs:5` enqueue `tasks._post_gpkg_export_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._run_hillslope_watbal_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._analyze_return_periods_rq`
- line `763` queue `default` stage `jobs:6` enqueue `tasks._log_complete_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._run_hillslope_watbal_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._analyze_return_periods_rq`, `tasks._post_legacy_arc_export_rq`, `tasks._post_gpkg_export_rq`, `tasks._build_hillslope_interchange_rq`, `tasks._build_totalwatsed3_rq`, `tasks.post_dss_export_rq`

### `wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_pipeline`
- line `116` queue `default` stage `jobs:0` enqueue `tasks._prep_multi_ofe_rq` depends_on *(none)*
- line `127` queue `default` stage `jobs:0` enqueue `tasks._prep_slopes_rq` depends_on *(none)*
- line `137` queue `default` stage `jobs:0` enqueue `tasks._prep_managements_rq` depends_on *(none)*
- line `146` queue `default` stage `jobs:0` enqueue `tasks._prep_soils_rq` depends_on *(none)*
- line `157` queue `default` stage `jobs:0` enqueue `tasks._prep_climates_rq` depends_on *(none)*
- line `167` queue `default` stage `jobs:0` enqueue `tasks._prep_remaining_rq` depends_on `tasks._prep_multi_ofe_rq`, `tasks._prep_slopes_rq`, `tasks._prep_managements_rq`, `tasks._prep_soils_rq`, `tasks._prep_climates_rq`
- line `177` queue `default` stage `jobs:1` enqueue `tasks._run_hillslopes_rq` depends_on `tasks._prep_remaining_rq`
- line `190` queue `default` stage `jobs:2` enqueue `tasks._prep_watershed_rq` depends_on `tasks._run_hillslopes_rq`
- line `217` queue `default` stage `jobs:2` enqueue `tasks._build_swat_inputs_rq` depends_on `tasks._run_hillslopes_rq`, `tasks._prep_watershed_rq`
- line `226` queue `default` stage `jobs:3` enqueue `tasks._run_swat_rq` depends_on `tasks._build_swat_inputs_rq`
- line `239` queue `default` stage `jobs:2` enqueue `tasks._build_hillslope_interchange_rq` depends_on `tasks._run_hillslopes_rq`, `tasks._build_swat_inputs_rq`
- line `250` queue `default` stage `jobs:2` enqueue `tasks._build_totalwatsed3_rq` depends_on `tasks._build_hillslope_interchange_rq`
- line `264` queue `default` stage `jobs:2` enqueue `tasks._build_swat_inputs_rq` depends_on `tasks._build_hillslope_interchange_rq`, `tasks._prep_watershed_rq`
- line `273` queue `default` stage `jobs:3` enqueue `tasks._run_swat_rq` depends_on `tasks._build_swat_inputs_rq`
- line `290` queue `default` stage `jobs:3` enqueue `tasks.run_ss_batch_watershed_rq` depends_on `tasks._prep_watershed_rq`
- line `303` queue `default` stage `jobs:3` enqueue `tasks.run_watershed_rq` depends_on `tasks._prep_watershed_rq`
- line `316` queue `default` stage `jobs:4` enqueue `tasks._post_run_cleanup_out_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`
- line `329` queue `default` stage `jobs:4` enqueue `tasks._post_prep_details_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`
- line `345` queue `default` stage `jobs:4` enqueue `tasks._run_hillslope_watbal_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`, `tasks._build_hillslope_interchange_rq`
- line `358` queue `default` stage `jobs:4` enqueue `tasks._post_make_loss_grid_rq` depends_on `tasks.run_ss_batch_watershed_rq`, `tasks.run_watershed_rq`, `tasks._prep_watershed_rq`
- line `369` queue `default` stage `jobs:4` enqueue `tasks._post_watershed_interchange_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._build_hillslope_interchange_rq`
- line `382` queue `default` stage `jobs:4` enqueue `tasks._analyze_return_periods_rq` depends_on `tasks._post_watershed_interchange_rq`, `tasks._build_totalwatsed3_rq`
- line `397` queue `default` stage `jobs:2` enqueue `tasks.post_dss_export_rq` depends_on `tasks._build_hillslope_interchange_rq`, `tasks._post_run_cleanup_out_rq`
- line `410` queue `default` stage `jobs:5` enqueue `tasks._post_legacy_arc_export_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._run_hillslope_watbal_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._analyze_return_periods_rq`
- line `423` queue `default` stage `jobs:5` enqueue `tasks._post_gpkg_export_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._run_hillslope_watbal_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._analyze_return_periods_rq`
- line `444` queue `default` stage `jobs:6` enqueue `tasks._log_complete_rq` depends_on `tasks._post_run_cleanup_out_rq`, `tasks._post_prep_details_rq`, `tasks._run_hillslope_watbal_rq`, `tasks._post_make_loss_grid_rq`, `tasks._post_watershed_interchange_rq`, `tasks._analyze_return_periods_rq`, `tasks._post_legacy_arc_export_rq`, `tasks._post_gpkg_export_rq`, `tasks._build_hillslope_interchange_rq`, `tasks._build_totalwatsed3_rq`, `tasks.post_dss_export_rq`, `tasks._run_swat_rq`

### `wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_prep_only_pipeline`
- line `470` queue `default` stage `jobs:0` enqueue `tasks._prep_multi_ofe_rq` depends_on *(none)*
- line `481` queue `default` stage `jobs:0` enqueue `tasks._prep_slopes_rq` depends_on *(none)*
- line `491` queue `default` stage `jobs:0` enqueue `tasks._prep_managements_rq` depends_on *(none)*
- line `501` queue `default` stage `jobs:0` enqueue `tasks._prep_soils_rq` depends_on *(none)*
- line `512` queue `default` stage `jobs:0` enqueue `tasks._prep_climates_rq` depends_on *(none)*
- line `522` queue `default` stage `jobs:0` enqueue `tasks._prep_remaining_rq` depends_on `tasks._prep_multi_ofe_rq`, `tasks._prep_slopes_rq`, `tasks._prep_managements_rq`, `tasks._prep_soils_rq`, `tasks._prep_climates_rq`
- line `532` queue `default` stage `jobs:2` enqueue `tasks._prep_watershed_rq` depends_on `tasks._prep_remaining_rq`

### `wepppy/rq/wmesque_rq.py::enqueue_sync`
- line `48` queue `default` stage `untracked` enqueue `sync_wmesque_rq` depends_on *(none)*

### `wepppy/weppcloud/bootstrap/enable_jobs.py::enqueue_bootstrap_enable`
- line `122` queue `default` stage `untracked` enqueue `bootstrap_enable_rq` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/geneva_bp.py::_enqueue_geneva_job`
- line `252` queue `default` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/interchange_bp.py::_enqueue_interchange_job`
- line `58` queue `default` stage `untracked` enqueue `run_interchange_migration` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/path_ce_bp.py::run_path_cost_effective`
- line `276` queue `default` stage `untracked` enqueue `run_path_cost_effective_rq` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/project_bp.py::delete_run`
- line `376` queue `default` stage `untracked` enqueue `delete_run_rq` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/project_bp.py::task_set_readonly`
- line `748` queue `default` stage `untracked` enqueue `set_run_readonly_rq` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/rangeland_bp.py::task_build_rangeland_cover`
- line `254` queue `default` stage `untracked` enqueue `build_rangeland_cover_rq` depends_on *(none)*

### `wepppy/weppcloud/routes/nodb_api/roads_bp.py::_enqueue_roads_job`
- line `470` queue `default` stage `untracked` enqueue `func` depends_on *(none)*

### `wepppy/weppcloud/routes/weppcloudr.py::_enqueue_deval_job`
- line `642` queue `default` stage `untracked` enqueue `render_deval_details_rq` depends_on *(none)*
<!-- AUTO-GENERATED: RQ DEPENDENCY GRAPH END -->

## Single-Storm Deprecation
Valid saved individual single-storm inputs can run with a deprecation advisory. The event pipeline skips continuous-only stages, including annual hillslope loss, water balance, return periods, and DSS export. Saved storm batches require per-event postprocessing outside the active RQ pipeline. See the [WEPP input contract](../../docs/schemas/wepp-run-input-contract.md#deprecated-single-storm-execution).

## Wait Strategy And Inventory
- Prefer explicit `depends_on` edges. File waits are a last-resort guard for filesystem propagation, not a substitute for missing dependencies.
- Waits must be time-bounded and run only inside background RQ workers or offline tooling; never add waits to UI routes or sync handlers where a run might not exist yet.
- Use `wepppy.io_wait.wait_for_path` / `wait_for_paths` when possible for stable-size checks. The interchange helper `wepppy.wepp.interchange._utils._wait_for_path` now delegates to `io_wait` with stable-size checks.
- When outputs might not exist because a run never executed, fail fast rather than waiting.

### Wait Inventory (File Guards)
| Area | Function | Guard | Target(s) | Timeout / poll | Notes |
| --- | --- | --- | --- | --- | --- |
| `wepppy/rq/wepp_rq.py` | `_post_watershed_interchange_rq` | `wait_for_path` via `_wait_for_output` | `wepp/output/pass_pw0.txt` (or `.gz`), `ebe_pw0.txt`, `loss_pw0.txt`, `chan.out`, `chanwb.out`, `chnwb.txt`, `soil_pw0.txt` (or `.gz`) | `timeout_s=60`, `poll_s=0.5` | RQ worker only; stable-size check before watershed interchange. Legacy single-storm conditionals skip some waits; single-storm is deprecated. |
| `wepppy/rq/wepp_rq.py` | `_run_hillslope_watbal_rq` | `wait_for_path` | `wepp/output/interchange/H.wat.parquet` | `timeout_s=60` (poll via `PERIDOT_INPUT_POLL_S`) | RQ worker only; stable-size check. |
| `wepppy/rq/wepp_rq.py` | `_build_totalwatsed3_rq` | `wait_for_paths` | `wepp/output/interchange/H.pass.parquet`, `wepp/output/interchange/H.wat.parquet` | `timeout_s=60` (poll via `PERIDOT_INPUT_POLL_S`) | RQ worker only; stable-size check before totalwatsed3 aggregation. |
| `wepppy/rq/wepp_rq.py` | `_analyze_return_periods_rq` | `wait_for_paths` | `wepp/output/interchange/ebe_pw0.parquet` (fallback `wepp/output/ebe_pw0.parquet`), `wepp/output/interchange/totalwatsed3.parquet` | `timeout_s=60` | RQ worker only; stable-size check before return-period export. |
| `wepppy/rq/wepp_rq.py` | `post_dss_export_rq` | `wait_for_path` / `wait_for_paths` | `export/dss/dss_channels.geojson` (if written), `export/dss/totalwatsed3_chan_*.dss` (if written), `export/dss/peak_chan_*.dss` (if written), `export/dss/README.dss_export.md` (if written), `export/dss.zip` | `timeout_s=60` | RQ worker only; stable-size checks between DSS export stages. |
| `wepppy/rq/project_rq.py` | `build_subcatchments_rq` | `wait_for_path` | `watershed.subwta` (`SUBWTA.ARC` or `subwta.tif`) | defaults (`PERIDOT_INPUT_WAIT_S` / `PERIDOT_INPUT_POLL_S`) | Guard before publishing completion trigger. |
| `wepppy/rq/project_rq.py` | `abstract_watershed_rq` | `wait_for_path` | `watershed.subwta` (`SUBWTA.ARC` or `subwta.tif`) | defaults (`PERIDOT_INPUT_WAIT_S` / `PERIDOT_INPUT_POLL_S`) | Guard before abstraction. |
| `wepppy/rq/project_rq.py` | `fork_rq` (`undisturbify=True`) | `wait_for_paths` | `ron.nodb`, `wepp.nodb`, `landuse.nodb`, `soils.nodb`, `disturbed.nodb` | `timeout_s=60` (poll via `PERIDOT_INPUT_POLL_S`) | Ensures forked NoDb files are readable before undisturbify. |
| `wepppy/topo/peridot/peridot_runner.py` | `run_peridot_abstract_watershed` | `_wait_for_file` | `dem/topaz/SUBWTA.ARC` | `PERIDOT_INPUT_WAIT_S` / `PERIDOT_INPUT_POLL_S` | Defensive guard before Peridot. |
| `wepppy/topo/peridot/peridot_runner.py` | `run_peridot_wbt_abstract_watershed` | `_wait_for_file` | `dem/wbt/subwta.tif` | `PERIDOT_INPUT_WAIT_S` / `PERIDOT_INPUT_POLL_S` | Defensive guard before Peridot (WBT). |
| `wepppy/wepp/interchange/watershed_ebe_interchange.py` | `run_wepp_watershed_ebe_interchange` | `_utils._wait_for_path` | `wepp/output/ebe_pw0.txt` | `timeout=60`, `poll=0.5` | Stable-size wait inside interchange. |
| `wepppy/wepp/interchange/watershed_chanwb_interchange.py` | `run_wepp_watershed_chanwb_interchange` | `_utils._wait_for_path` | `wepp/output/chanwb.out` | `timeout=60`, `poll=0.5` | Stable-size wait inside interchange. |
| `wepppy/wepp/interchange/watershed_chan_peak_interchange.py` | `run_wepp_watershed_chan_peak_interchange` | `_utils._wait_for_path` | `wepp/output/chan.out` | `timeout=60`, `poll=0.5` | Stable-size wait inside interchange. |
| `wepppy/wepp/interchange/watershed_chnwb_interchange.py` | `run_wepp_watershed_chnwb_interchange` | `_utils._wait_for_path` | `wepp/output/chnwb.txt` | `timeout=60`, `poll=0.5` | Stable-size wait inside interchange. |
| `wepppy/wepp/reports/return_periods.py` | `refresh_return_period_events` | `wait_for_path` | `interchange/ebe_pw0.parquet` (or `wepp/output/ebe_pw0.parquet`), `interchange/totalwatsed3.parquet` | defaults | Report builder waits for parquet assets unless `wait_for_inputs=False` (UI routes). |
## Catalog

**Run catalog maintenance**

The opt-in scheduler entry calls `run_catalog_rq.enqueue_sweep` on the existing
`batch` queue. Admission requires a live idle compatible consumer and no global
RQ suspension, using passive consistent registration/TTL observations. No
qualifying worker means no job or admission-pointer mutation; dirty SQL state
waits for the next tick. Observation does not reserve the worker. Redis admission
still coalesces queued/started/deferred/scheduled sweeps; a PostgreSQL
transaction advisory lock serializes scheduled and CLI scans. There are no
child jobs or per-save enqueues. The generated scanner's generic queue label
does not override this scheduler-selected queue. IDs retain the exact
`run_catalog_sweep_<UUID>` form and public polling never exposes task arguments,
results or private exception metadata. All eligible consumers require the
canonical run-catalog compatibility/origin/mount proof before admission.
Real isolated-queue validation executes the callable and checks its terminal
job tree plus source-derived SQL values; live host-tree evidence remains a
staged deployment gate.

**AgFields rq-engine enqueue surface**

The helper `wepppy.microservices.rq_engine.ag_fields_routes._enqueue_job` is represented as one dynamic `func` edge in the generated graph. Its three concrete call sites have no RQ `depends_on` edges; prerequisite checks happen synchronously before enqueue. A single-scheme `_enqueue_watershed_jobs` call still enqueues the scheme worker directly. Run All instead enqueues `run_ag_fields_watershed_suite_rq` as one top-level parent. The route atomically stores all planned children on that parent, and dispatch shares a lock with cancellation. Concept 1, Concept 2, and hybrid remain ordered `jobs:*` children; each scheme after the first depends on its predecessor through `Dependency(..., allow_failure=True)`. A registered finalizer depends on all three scheme IDs through `Dependency(..., allow_failure=True)`. Every failure-tolerant dependent applies the Batch Runner already-terminal release guard, so a predecessor that fails before registration cannot strand the rest of the suite. This prevents overlapping watershed memory peaks while permitting complete comparison attempts and one deterministic terminal suite signal from the finalizer.

| Job key | RQ entrypoint | Preconditions / file inputs |
| --- | --- | --- |
| `agfields_build_subfields` | `wepppy.rq.ag_fields_rq.build_ag_fields_subfields_rq` | Confirmed schema; `dem/wbt/flovec.tif`; canonical boundary GeoJSON; runs rasterize → abstract → polygonize inside one job |
| `agfields_plantdb` | `wepppy.rq.ag_fields_rq.process_ag_fields_plant_db_rq` | Server-validated, uniquely named ZIP under `ag_fields/`; staged archive is removed after terminal success/failure |
| `agfields_run_wepp` | `wepppy.rq.ag_fields_rq.run_ag_fields_wepp_rq` | Current sub-fields; complete valid mapping; parent `wepp/runs/p<wepp_id>.sol` and `.cli` pairs; after WEPP, synchronously converts all six `H<sub_field_id>` report families into the specialized six-file bundle and stamps completion only after publication |
| `agfields_run_watershed_suite` | `wepppy.rq.ag_fields_rq.run_ag_fields_watershed_suite_rq` | Run All only; parent dispatches three serial scheme children and the always-released finalizer; response `job_id` names this parent |
| `agfields_run_watershed_concept_1` | `wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq` | Current successful Stage 4; observed climate; complete parent prepared inputs; writes only `wepp/ag_fields/watershed/concept-1/` |
| `agfields_run_watershed_concept_2` | `wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq` | Same preconditions; writes only `wepp/ag_fields/watershed/concept-2/`; the historical `agfields_run_watershed` key remains a Concept 2 job-id alias |
| `agfields_run_watershed_hybrid` | `wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq` | Same preconditions plus the explicit Peridot connectivity resources; writes only `wepp/ag_fields/watershed/hybrid/` |
| suite finalizer | `wepppy.rq.ag_fields_rq.finalize_ag_fields_watershed_suite_rq` | Depends on all three scheme jobs with `allow_failure=True`; publishes terminal suite status after every scheme is terminal |

**`wepppy/rq/wepp_rq.py`**

**`run_wepp_rq(runid)`**

Graph (stage order is the `jobs:<n>` prefix):
```text
jobs:0 prep inputs
  -> _prep_remaining_rq
  -> _run_hillslopes_rq
  -> (_prep_watershed_rq if run_watershed)
  -> _build_hillslope_interchange_rq (after SWAT build when delete_after_interchange)
  -> _build_totalwatsed3_rq
  -> (post_dss_export_rq if enabled and run_watershed; after `_post_run_cleanup_out_rq`)
  -> (SWAT build/run if enabled; before interchange when delete_after_interchange)
  -> watershed runs (if run_watershed)
  -> post-run jobs (cleanup, prep details, watbal, loss grid, watershed interchange, return periods)
  -> export jobs (legacy arc, gpkg)
  -> _log_complete_rq
```

Child jobs, dependencies, and file prerequisites:

Note: Parquet prerequisites are canonical directory paths (for example `watershed/hillslopes.parquet`). WD-root sidecars (for example `watershed.hillslopes.parquet`) are retired and treated as migration-required boundaries.

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:0 | `_prep_multi_ofe_rq` | none | `wat_dir/slope_files/hillslopes/hill_<topaz>.mofe.slp`; `soils_dir/hill_<topaz>.mofe.sol`; `lc_dir/*.man`; optional `wepp.kslast_map` raster |
| jobs:0 | `_prep_slopes_rq` | none | `wat_dir/hill_<topaz>.slp` or `wat_dir/slope_files/hillslopes/hill_<topaz>.slp` |
| jobs:0 | `_prep_managements_rq` | none | `landuse/landuse.parquet`; `lc_dir/*.man`; `soils/soils.parquet`; optional `disturbed.nodb` + disturbed maps; optional `rap/rap_ts.parquet` |
| jobs:0 | `_prep_soils_rq` | none | `soils_dir/*.sol`; optional `wepp.kslast_map` raster; `soils/soils.parquet` for soil metadata |
| jobs:0 | `_prep_climates_rq` | none | `cli_dir/<cli_fn>`; for SS batch writes `runs_dir/p<wepp_id>.<ss_batch_id>.cli` and `runs_dir/pw0.<ss_batch_id>.cli` |
| jobs:0 | `_prep_remaining_rq` | `jobs0_hillslopes_prep` list | `runs_dir/p*.slp`; `runs_dir/p*.sol`; `runs_dir/p*.man`; `runs_dir/p*.cli` |
| jobs:1 | `_run_hillslopes_rq` | `_prep_remaining_rq` | `runs_dir/*.run`; `runs_dir/p*.slp`; `runs_dir/p*.sol`; `runs_dir/p*.man`; `runs_dir/p*.cli` |
| jobs:2 | `_prep_watershed_rq` | `_run_hillslopes_rq` | `wat_dir/structure.json` or `wat_dir/network.txt`; `watershed/channels.parquet`; `watershed/hillslopes.parquet`; `wat_dir/slope_files/channels.slp` or `wat_dir/channels.slp`; `soils/soils.parquet`; `cli_dir/<cli_fn>` |
| jobs:2 | `_build_hillslope_interchange_rq` | `_run_hillslopes_rq` (+ `_build_swat_inputs_rq` if SWAT enabled + delete_after_interchange) | `output_dir/H*.pass.dat`; `output_dir/H*.ebe.dat`; `output_dir/H*.element.dat`; `output_dir/H*.loss.dat`; `output_dir/H*.soil.dat`; `output_dir/H*.wat.dat` |
| jobs:2 | `_build_totalwatsed3_rq` | `_build_hillslope_interchange_rq` | `output_dir/interchange/H.pass.parquet`; `output_dir/interchange/H.wat.parquet`; optional `ash/H<wepp_id>_ash.parquet` or `ash/H<wepp_id>.parquet`; optional `ash.nodb`; `watershed/hillslopes.parquet` (area lookup when ash is present) |
| jobs:2 | `post_dss_export_rq` | `_build_hillslope_interchange_rq` (+ `_post_run_cleanup_out_rq` if run_watershed) | `output_dir/interchange/H.pass.parquet`; `output_dir/interchange/H.wat.parquet`; optional `ash/*.parquet`; `output_dir/chan.out`; `watershed.channels_shp` (GeoJSON); `wat_dir/network.txt` (downstream mapping); `export/dss` dir (created) |
| jobs:2 | `_build_swat_inputs_rq` | `_build_hillslope_interchange_rq` (+ `_prep_watershed_rq` if present); if delete_after_interchange then `_run_hillslopes_rq` (+ `_prep_watershed_rq` if present) | `wepp/output/*` (hillslope outputs); `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `swat_template_dir/*` (config `swat_template_dir`) |
| jobs:3 | `_run_swat_rq` | `_build_swat_inputs_rq` | `swat_txtinout_dir/*` |
| jobs:3 | `run_watershed_rq` / `run_ss_batch_watershed_rq` | `_prep_watershed_rq` | `runs_dir/pw0.str`; `runs_dir/pw0.slp`; `runs_dir/pw0.chn`; `runs_dir/pw0.sol`; `runs_dir/pw0.cli`; `runs_dir/pw0.man`; `runs_dir/chan.inp`; for SS batch `runs_dir/pw0.<ss_batch_id>.cli` |
| jobs:4 | `_post_run_cleanup_out_rq` | `jobs3_watersheds` or `_prep_watershed_rq` | `runs_dir/*.out`; optional `runs_dir/tc_out.txt` |
| jobs:4 | `_post_prep_details_rq` | `post_dependencies` | `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `landuse/landuse.parquet`; `soils/soils.parquet` |
| jobs:4 | `_run_hillslope_watbal_rq` | `post_dependencies` + `_build_hillslope_interchange_rq` | `output_dir/interchange/H.wat.parquet` |
| jobs:4 | `_post_make_loss_grid_rq` | `post_dependencies` | `watershed.subwta` (`wat_dir/SUBWTA.ARC` or `wbt_wd/subwta.tif`); `watershed.discha` (`wat_dir/DISCHA.ARC` or `wbt_wd/discha.tif`); `output_dir/H*` |
| jobs:4 | `_post_watershed_interchange_rq` | `_post_run_cleanup_out_rq` | `output_dir/pass_pw0.txt`; `output_dir/ebe_pw0.txt`; `output_dir/chan.out`; `output_dir/chanwb.out`; `output_dir/chnwb.txt`; `output_dir/soil_pw0.txt` or `output_dir/soil_pw0.txt.gz`; `output_dir/loss_pw0.txt` |
| jobs:4 | `_analyze_return_periods_rq` | `_post_watershed_interchange_rq` + `_build_totalwatsed3_rq` | `output_dir/interchange/ebe_pw0.parquet` (fallback `output_dir/ebe_pw0.parquet`); `output_dir/interchange/totalwatsed3.parquet`; optional cached `output_dir/return_periods*.json` |
| jobs:5 | `_post_legacy_arc_export_rq` | `jobs4_post` | `topaz_wd/*.ARC`; `topaz_wd/SUBCATCHMENTS.JSON`; `topaz_wd/CHANNELS.JSON`; `output_dir/loss_pw0.txt`; `watershed/hillslopes.parquet`; `watershed/channels.parquet`; optional `ash` metadata |
| jobs:5 | `_post_gpkg_export_rq` | `jobs4_post` | `watershed.subwta_shp` (SUBCATCHMENTS.WGS JSON); `watershed.channels_shp` (CHANNELS.WGS JSON); `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `landuse/landuse.parquet`; `soils/soils.parquet`; `output_dir/interchange/loss_pw0.hill.parquet`; `output_dir/interchange/loss_pw0.chn.parquet` |
| jobs:6 | `_log_complete_rq` | `jobs4_post + jobs5_post` | `ron.nodb` |

Audit notes:
1. `_run_hillslope_watbal_rq` depends on `_build_hillslope_interchange_rq` and still waits for `output_dir/interchange/H.wat.parquet` internally.
2. `post_dss_export_rq` only enqueues when `run_watershed` is true (guarded in `run_wepp_rq`).
3. `post_dss_export_rq` calls `totalwatsed_partitioned_dss_export`, which runs `run_totalwatsed3` per channel; it does not require `_build_totalwatsed3_rq` unless the export is refactored to reuse a prebuilt `totalwatsed3.parquet`.
4. `_build_totalwatsed3_rq` depends on hillslope interchange. If watershed outputs are required, depend on `jobs3_watersheds` or `_post_watershed_interchange_rq`.
5. When `delete_after_interchange` is enabled, SWAT inputs are built before interchange and interchange waits on SWAT to avoid deleting raw `wepp/output` files.
6. `_run_hillslope_watbal_rq` and `post_dss_export_rq` target continuous runs; legacy single-storm conditionals may skip them, but single-storm is deprecated.
7. `_analyze_return_periods_rq` depends on `totalwatsed3`; legacy single-storm runs skip this pipeline and are deprecated.

**`run_wepp_noprep_rq(runid)`**

Graph (prep stages skipped; uses existing inputs under `wepp/runs/`):
```text
jobs:1 _run_hillslopes_rq
  -> _build_hillslope_interchange_rq
  -> _build_totalwatsed3_rq
  -> watershed runs (if run_watershed)
  -> post-run jobs (cleanup, prep details, watbal, loss grid, watershed interchange, return periods)
  -> exports (legacy arc, gpkg)
  -> (post_dss_export_rq if enabled and run_watershed)
  -> _log_complete_rq
```

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:1 | `_run_hillslopes_rq` | none | `runs_dir/*.run`; `runs_dir/p*.slp`; `runs_dir/p*.sol`; `runs_dir/p*.man`; `runs_dir/p*.cli` |
| jobs:2 | `_build_hillslope_interchange_rq` | `_run_hillslopes_rq` | `output_dir/H*.pass.dat`; `output_dir/H*.ebe.dat`; `output_dir/H*.element.dat`; `output_dir/H*.loss.dat`; `output_dir/H*.soil.dat`; `output_dir/H*.wat.dat` |
| jobs:2 | `_build_totalwatsed3_rq` | `_build_hillslope_interchange_rq` | `output_dir/interchange/H.pass.parquet`; `output_dir/interchange/H.wat.parquet`; optional `ash` parquet; `watershed/hillslopes.parquet` |
| jobs:3 | `run_watershed_rq` / `run_ss_batch_watershed_rq` | `_run_hillslopes_rq` | `runs_dir/pw0.*` as above; SS batch `runs_dir/pw0.<ss_batch_id>.cli` |
| jobs:4 | `_post_run_cleanup_out_rq` | `jobs3_watersheds` or `_run_hillslopes_rq` | `runs_dir/*.out`; optional `runs_dir/tc_out.txt` |
| jobs:4 | `_post_prep_details_rq` | `post_dependencies` | `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `landuse/landuse.parquet`; `soils/soils.parquet` |
| jobs:4 | `_run_hillslope_watbal_rq` | `post_dependencies` + `_build_hillslope_interchange_rq` | `output_dir/interchange/H.wat.parquet` |
| jobs:4 | `_post_make_loss_grid_rq` | `post_dependencies` | `watershed.subwta`; `watershed.discha`; `output_dir/H*` |
| jobs:4 | `_post_watershed_interchange_rq` | `_post_run_cleanup_out_rq` | `output_dir/pass_pw0.txt`; `output_dir/ebe_pw0.txt`; `output_dir/chan.out`; `output_dir/chanwb.out`; `output_dir/chnwb.txt`; `output_dir/soil_pw0.txt` or `output_dir/soil_pw0.txt.gz`; `output_dir/loss_pw0.txt` |
| jobs:4 | `_analyze_return_periods_rq` | `_post_watershed_interchange_rq` + `_build_totalwatsed3_rq` | `output_dir/interchange/ebe_pw0.parquet`; `output_dir/interchange/totalwatsed3.parquet` |
| jobs:2 | `post_dss_export_rq` | `_build_hillslope_interchange_rq` (+ `_post_run_cleanup_out_rq` if run_watershed) | `output_dir/interchange/H.pass.parquet`; `output_dir/interchange/H.wat.parquet`; optional `ash` parquet; `output_dir/chan.out`; `watershed.channels_shp`; `wat_dir/network.txt`; `export/dss` dir |
| jobs:5 | `_post_legacy_arc_export_rq` | `jobs4_post` | `topaz_wd/*.ARC`; `topaz_wd/SUBCATCHMENTS.JSON`; `topaz_wd/CHANNELS.JSON`; `output_dir/loss_pw0.txt`; `watershed/hillslopes.parquet`; `watershed/channels.parquet`; optional `ash` metadata |
| jobs:5 | `_post_gpkg_export_rq` | `jobs4_post` | `watershed.subwta_shp`; `watershed.channels_shp`; `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `landuse/landuse.parquet`; `soils/soils.parquet`; `output_dir/interchange/loss_pw0.hill.parquet`; `output_dir/interchange/loss_pw0.chn.parquet` |
| jobs:6 | `_log_complete_rq` | `jobs4_post + jobs5_post` | `ron.nodb` |

**`run_wepp_watershed_rq(runid)`**

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:2 | `_prep_watershed_rq` | none | `wat_dir/structure.json` or `wat_dir/network.txt`; `watershed/channels.parquet`; `watershed/hillslopes.parquet`; `wat_dir/slope_files/channels.slp` or `wat_dir/channels.slp`; `soils/soils.parquet`; `cli_dir/<cli_fn>` |
| jobs:3 | `run_watershed_rq` / `run_ss_batch_watershed_rq` | `_prep_watershed_rq` | `runs_dir/pw0.*` as above; SS batch `runs_dir/pw0.<ss_batch_id>.cli` |
| jobs:4 | `_post_run_cleanup_out_rq` | `jobs3_watersheds` or `_prep_watershed_rq` | `runs_dir/*.out`; optional `runs_dir/tc_out.txt` |
| jobs:4 | `_post_prep_details_rq` | `post_dependencies` | `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `landuse/landuse.parquet`; `soils/soils.parquet` |
| jobs:4 | `_post_make_loss_grid_rq` | `post_dependencies` | `watershed.subwta`; `watershed.discha`; `output_dir/H*` (job skipped when hillslope outputs are missing) |
| jobs:4 | `_post_watershed_interchange_rq` | `_post_run_cleanup_out_rq` | `output_dir/pass_pw0.txt`; `output_dir/ebe_pw0.txt`; `output_dir/chan.out`; `output_dir/chanwb.out`; `output_dir/chnwb.txt`; `output_dir/soil_pw0.txt` or `output_dir/soil_pw0.txt.gz`; `output_dir/loss_pw0.txt` |
| jobs:5 | `_post_legacy_arc_export_rq` | `jobs4_post` | `topaz_wd/*.ARC`; `topaz_wd/SUBCATCHMENTS.JSON`; `topaz_wd/CHANNELS.JSON`; `output_dir/loss_pw0.txt`; `watershed/hillslopes.parquet`; `watershed/channels.parquet`; optional `ash` metadata |
| jobs:6 | `_log_complete_rq` | `jobs4_post + jobs5_post` | `ron.nodb` |

Audit notes:
1. This entrypoint bypasses hillslope prep/run; ensure downstream jobs only require watershed outputs.
2. Return-period exports are not enqueued here because `totalwatsed3` is not scheduled.

**`run_wepp_watershed_noprep_rq(runid)`**

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:3 | `run_watershed_rq` / `run_ss_batch_watershed_rq` | none | `runs_dir/pw0.*` as above; SS batch `runs_dir/pw0.<ss_batch_id>.cli` |
| jobs:4 | `_post_run_cleanup_out_rq` | `jobs3_watersheds` | `runs_dir/*.out`; optional `runs_dir/tc_out.txt` |
| jobs:4 | `_post_prep_details_rq` | `post_dependencies` | `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `landuse/landuse.parquet`; `soils/soils.parquet` |
| jobs:4 | `_post_make_loss_grid_rq` | `post_dependencies` | `watershed.subwta`; `watershed.discha`; `output_dir/H*` (job skipped when hillslope outputs are missing) |
| jobs:4 | `_post_watershed_interchange_rq` | `_post_run_cleanup_out_rq` | `output_dir/pass_pw0.txt`; `output_dir/ebe_pw0.txt`; `output_dir/chan.out`; `output_dir/chanwb.out`; `output_dir/chnwb.txt`; `output_dir/soil_pw0.txt` or `output_dir/soil_pw0.txt.gz`; `output_dir/loss_pw0.txt` |
| jobs:5 | `_post_legacy_arc_export_rq` | `jobs4_post` | `topaz_wd/*.ARC`; `topaz_wd/SUBCATCHMENTS.JSON`; `topaz_wd/CHANNELS.JSON`; `output_dir/loss_pw0.txt`; `watershed/hillslopes.parquet`; `watershed/channels.parquet`; optional `ash` metadata |
| jobs:6 | `_log_complete_rq` | `jobs4_post + jobs5_post` | `ron.nodb` |

**`wepppy/rq/swat_rq.py`**

**`run_swat_rq(runid)`**

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:0 | `_build_swat_inputs_rq` | none | `wepp/output/*` (hillslope outputs); `watershed/hillslopes.parquet`; `watershed/channels.parquet`; `swat_template_dir/*` |
| jobs:1 | `_run_swat_rq` | `_build_swat_inputs_rq` | `swat_txtinout_dir/*` |

Audit notes:
1. This assumes WEPP outputs already exist. The caller is responsible for enqueueing only after WEPP hillslope/watershed outputs are ready.

**`run_swat_noprep_rq(runid)`**

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:0 | `_run_swat_rq` | none | `swat_txtinout_dir/*` |

**`wepppy/rq/batch_rq.py`**

**`run_batch_rq(batch_name)`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| jobs:0,hillslopes | `run_batch_hillslopes_rq` (per watershed feature) | root dispatch, strict | Batch workspace, validated watershed feature and base project |
| jobs:0 | `run_batch_watershed_rq` (per watershed feature) | corresponding hillslope job, failure-tolerant terminal observer | Finished upstream and matching visible `batch_handoff/<hillslope-job-id>.json` required before science |
| jobs:1 | `_final_batch_complete_rq` | all `jobs:0` via failure-tolerant RQ `Dependency` (+ Omni final jobs when `run_omni_scenarios_rq` enqueues work) | `batch_runner.nodb` (for status updates) |

Audit notes:
1. `run_batch_watershed_rq` verifies the attempt-bound handoff, freshly hydrates leaf state, and owns terminal metadata and the completion trigger. See `docs/schemas/batch-task-boundary-contract.md`.
2. It calls `run_omni_scenarios_rq(runid)` inline. Omni finalizer attachment is serialized under a finalizer-specific Redis lock and uses root RQ metadata, so parallel leaves cannot lose each other's dependencies.
3. The completion job allows failed dependencies so it still runs after leaves reach terminal `failed` status and can publish the failed-run summary instead of remaining deferred.

**`wepppy/rq/culvert_rq.py`**

**`run_culvert_batch_rq(culvert_batch_uuid)`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| jobs:0 | `run_culvert_run_rq` (per run_id) | none | `batch_root/metadata.json`; `batch_root/model-parameters.json`; `batch_root/topo/breached_filled_DEM_UTM.tif` (or payload override); `batch_root/topo/streams.tif` (or payload override); `batch_root/culverts/watersheds.geojson`; `batch_root/topo/flovec.tif`; `batch_root/topo/netful.tif`; `batch_root/topo/chnjnt.tif`; `batch_root/topo/chnjnt.streams.tif` |
| jobs:1 | `_final_culvert_batch_complete_rq` | all `jobs:0` | `batch_root/culverts/*.json` run metadata (written by child jobs) |

Audit notes:
1. Batch-level topo generation happens before enqueueing children. If that becomes async, ensure child jobs depend on it.

**`run_culvert_run_rq(runid, culvert_batch_uuid, run_id)`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| (inline) | `_process_culvert_run` (internal) | none | `batch_root/metadata.json`; `batch_root/model-parameters.json`; `batch_root/culverts/watersheds.geojson`; `batch_root/culverts/culvert_points.geojson`; `batch_root/topo/breached_filled_DEM_UTM.tif`; `batch_root/topo/flovec.tif`; `batch_root/topo/netful.tif`; `batch_root/topo/chnjnt.tif`; per-run directory `batch_root/runs/<run_id>` |

**`wepppy/rq/omni_rq.py`**

**`run_omni_scenarios_rq(runid)`**

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:0 | `run_omni_scenario_rq` (stage 1 scenarios) | none | Base scenario inputs under `wepp/output/*`; dependency hash uses `wepp/output/loss_pw0.txt` |
| jobs:1 | `run_omni_scenario_rq` (stage 2 scenarios) | all stage 1 jobs, strict | Stage 1 scenario outputs under `omni/scenarios/<scenario>/wepp/output/loss_pw0.txt` |
| jobs:2 | `_compile_hillslope_summaries_rq` | stage 2 jobs (or stage 1 if no stage 2), strict | `wepp/output/loss_pw0.txt`; `omni/scenarios/<scenario>/wepp/output/loss_pw0.txt` |
| jobs:3 | `_finalize_omni_scenarios_rq` | `_compile_hillslope_summaries_rq`, strict | `omni.nodb` |

Audit notes:
1. Stage 2 depends on the full set of stage 1 jobs, not just the specific scenario’s dependency. This is conservative but can increase critical path length.
2. Every scenario edge consumes required upstream outputs and is strict. A
   failure stops all transitive scenario, compile, and finalization work; the
   controller's ordinary retry replaces the blocked deferred graph.

**`run_omni_contrasts_rq(runid)`**

| Stage | Child job | `depends_on` | File prerequisites (relative to `wd`) |
| --- | --- | --- | --- |
| jobs:contrast:* | `run_omni_contrast_rq` | previous batch (if any), failure tolerant | `omni/contrasts/contrast_<id>.tsv`; scenario outputs referenced by sidecar paths under `omni/contrasts/<id>/wepp/output/*` |
| jobs:finalize | `_finalize_omni_contrasts_rq` | last batch, failure tolerant | `omni.nodb` |

Audit notes:
1. Batches are chained (`batch_depends`), so the final dependency on the last batch implies all prior batches.
2. Because the batches are chained, a bare `depends_on` list makes one failed contrast strand every later batch and the finalizer. Both edges use `_failure_tolerant_depends_on` plus `_release_deferred_job_if_ready` so `_finalize_omni_contrasts_rq` always reaches a terminal state and emits `END_BROADCAST`.

**`wepppy/rq/project_rq.py`**

**`fetch_dem_and_build_channels_rq(...)`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| jobs:0 | `fetch_dem_rq` | none | `ron.nodb`; writes `dem_dir/dem.tif` or `dem_dir/dem.vrt` |
| jobs:1 | `build_channels_rq` | `fetch_dem_rq` | `dem_dir/dem.tif` or `dem_dir/dem.vrt` |
| jobs:0 | `build_channels_rq` | none | (upload DEM mode) `dem_dir/dem.tif` or `dem_dir/dem.vrt` already present |

Audit notes:
1. No file-based waits here; ensure `fetch_dem_rq` fully writes `dem_dir/dem.tif` before returning.

**`build_subcatchments_and_abstract_watershed_rq(runid, updates)`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| jobs:0 | `build_subcatchments_rq` | prior same-run request tail, when present (failure-tolerant serialization) | `dem_dir/dem.tif` or `dem_dir/dem.vrt`; channel outputs under `wat_dir/*` from `build_channels_rq` |
| jobs:1 | `abstract_watershed_rq` (nonmutating completion receipt) | `build_subcatchments_rq` (strict) | none; build and abstraction already completed inside jobs:0 |

Audit notes:
1. `build_subcatchments_rq` uses `wait_for_path` on `watershed.subwta`, then
   performs abstraction under the same watershed directory-root lock before
   publishing completion.
2. One optimistic Redis transaction registers both jobs, both dependency
   directions, both root metadata links, the per-run tail, and jobs:0 queue or
   deferred membership. The prior-tail edge serializes independent request
   mutation ownership and may cross root lineage; retry cleanup treats it as an
   external dependency boundary and admission splices a canceled tail back to
   its still-active predecessor. A failed jobs:0 cancels only its matching
   receipt for controlled policy failures; other failures leave the strict
   receipt deferred.

**`fork_rq(..., undisturbify=True)`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| (inline) | `run_wepp_rq(new_runid)` | none | Forked run directory with `.nodb` files (`ron.nodb`, `wepp.nodb`, `landuse.nodb`, `soils.nodb`, `disturbed.nodb`) |
| (inline) | `_finish_fork_rq` | final job from `run_wepp_rq` | relies on `run_wepp_rq` completion metadata |

Audit notes:
1. `_finish_fork_rq` depends on the final `_log_complete_rq` job produced by `run_wepp_rq`, so it should run after all WEPP tasks finish.
2. Every WEPP pipeline child created for an undisturbify fork carries server-generated `fork_failure` lineage and `report_fork_failure` as its RQ failure callback. Root child registration is saved before enqueue. The callback atomically verifies the current destination receipt and records failure plus source status; strict dependency edges are unchanged.
3. `WepppyRqWorker.handle_job_failure` repeats guarded reporting after abrupt work-horse death, when RQ skips the job callback. Duplicate/stale reporting is suppressed. Neither path releases claims, cancels siblings, nor unblocks model stages.
4. Initial Wepp/Watershed controller reads in preparation jobs have a shared five-second ENOENT/ESTALE retry budget. This does not retry the job or any mutation. See [ADR-0049](../../docs/adrs/ADR-0049-fork-preparation-read-retry.md).

**`wepppy/microservices/rq_engine/run_sync_routes.py`**

**`POST /rq-engine/run-sync`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `run_sync_rq` | none | Target root writable (default `/wc1/runs`); remote source reachable; optional `source_run_token` with source-host scope `runs:read` for private runs |
| root | `migrations_rq` | `run_sync_rq` | `target_root/<prefix>/<runid>` run directory with `.nodb` files and run outputs |

Audit notes:
1. `migrations_rq` depends on the sync job, ensuring that run files exist before migrations execute.
2. `run_sync_rq` writes `RunMigration` rows by importing `wepppy.weppcloud.app`; workers executing it need `SECRET_KEY(_FILE)`, `SECURITY_PASSWORD_SALT(_FILE)`, and `POSTGRES_PASSWORD(_FILE)` configured.
3. When `source_run_token` is supplied, `run_sync_rq` uses `Authorization: Bearer <token>` for `aria2c.spec` and aria2 fetch operations; token is not written to provenance records.

**`wepppy/microservices/rq_engine/batch_routes.py`**

**`POST /api/batch/_/{batch_name}/run-batch`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `run_batch_rq` | none | Batch workspace `batch_root/<batch_name>`; `batch_runner.nodb`; uploaded watershed GeoJSON + validated template |

**`POST /api/batch/_/{batch_name}/delete-batch`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `delete_batch_rq` | none | Batch workspace `batch_root/<batch_name>` (including `_base` and `runs/*`) |

Audit notes:
1. `delete_batch_rq` is destructive and removes the batch workspace recursively after clearing batch run lock/cache metadata.

**`wepppy/microservices/rq_engine/bootstrap_routes.py`**

**`POST /api/runs/{runid}/{config}/bootstrap/enable`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `bootstrap_enable_rq` | none | `wd` (run directory from `get_wd(runid)`); writes `.git/` and commits tracked bootstrap inputs under `wepp/runs` and/or `swat/TxtInOut` |

Audit notes:
1. Route-level dedupe key `bootstrap:enable:job:<runid>` prevents duplicate enable jobs while one is active.
2. Route acquires run-scoped Redis git lock `bootstrap:git-lock:<runid>` before enqueueing; the worker releases it in `bootstrap_enable_rq` cleanup.

**`POST /api/runs/{runid}/{config}/run-wepp-npprep`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `run_wepp_noprep_rq` | none | Existing WEPP input state under `wepp/runs`; Bootstrap must already be enabled for the run |

Audit notes:
1. Route resets Redis prep timestamps for WEPP tasks before enqueueing the no-prep run.
2. This endpoint intentionally skips prep rebuilds and runs against the currently checked-out git state.

**`POST /api/runs/{runid}/{config}/run-wepp-watershed-no-prep`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `run_wepp_watershed_noprep_rq` | none | Existing watershed inputs under `wepp/runs`; Bootstrap must already be enabled for the run |

Audit notes:
1. Route resets the watershed prep timestamp before enqueueing.
2. No branch checkout occurs here; execution uses current repository checkout.

**`POST /api/runs/{runid}/{config}/run-swat-noprep`**

| Stage | Child job | `depends_on` | File prerequisites |
| --- | --- | --- | --- |
| root | `run_swat_noprep_rq` | none | Existing SWAT+ inputs under `swat/TxtInOut`; Bootstrap must already be enabled and SWAT mod present |

Audit notes:
1. Route enqueues SWAT no-prep execution without rebuilding TxtInOut.
2. Endpoint depends on existing SWAT inputs and the active checkout state.

## NFS Consistency Headaches

### Background (Feb 6, 2026)

Production WEPP runs experienced rare, intermittent failures (on the order of 1 in 10⁶ to 10⁹ events) where RQ worker jobs could not read files that upstream jobs had already written. Symptoms included `FileNotFoundError` on `.parquet` interchange files, `wait_for_path` timeouts on watershed outputs (`pass_pw0.txt`, `ebe_pw0.txt`, etc.), and stale reads returning truncated or zero-length files. These failures were non-deterministic and consistent with NFS client cache staleness, though the root cause has not been definitively confirmed.

Notably, the same `soft` mount configuration works reliably on the dev and test-production NFS servers, which use ZFS as the backing filesystem. The failures have only been observed on the production NAS, suggesting the issue may involve an interaction between the `soft` mount options and the production storage backend rather than the mount options alone.

### Suspected Cause: NFS Mount Options

The `/geodata` mount was configured with `soft` mode and large read/write buffers with aggressive attribute caching. The following options are suspected contributors to the degraded behavior:

```
# BEFORE (degraded)
nas.rocket.net:/wepp  /geodata  nfs4  rw,noatime,proto=tcp,vers=4.2,soft,rsize=32768,wsize=32768,timeo=600,_netdev  0  0
```

| Option | Potential problem |
| --- | --- |
| `soft` | NFS operations return errors after `timeo` retries instead of blocking. Under load, transient server delays could cause I/O errors that surface as missing or corrupt files in downstream RQ jobs. |
| `rsize=32768,wsize=32768` | Fixed 32 KB transfer sizes bypass the kernel's auto-negotiation, potentially limiting throughput on large sequential writes (interchange parquet files) and adding round-trips for small metadata operations. |
| `timeo=600` | 60-second timeout (in deciseconds) combined with `soft` means jobs may receive `EIO` after a single retry window rather than waiting for the server to recover. |
| *(no `actimeo`)* | Default attribute cache lifetime (~3–60 s) can allow NFS clients to serve stale `stat()` results. A file written by one worker could appear absent or zero-length to another worker for up to 60 seconds. |

### Remediation: Updated Mount

```
# AFTER (remediated)
nas.rocket.net:/wepp  /geodata  nfs4  rw,noatime,proto=tcp,vers=4.2,hard,intr,actimeo=0,_netdev  0  0
```

| Option | Effect |
| --- | --- |
| `hard` | NFS operations block indefinitely until the server responds, preventing spurious `EIO` errors during transient load spikes. Workers wait rather than fail. |
| `intr` | Allows blocked hard-mount operations to be interrupted by signals, so jobs can still be canceled or timed out by RQ without requiring a full process kill. |
| `actimeo=0` | Disables attribute caching entirely. Every `stat()`, `open()`, and directory listing goes to the server, ensuring that when an upstream job writes a file, downstream jobs should see it immediately. This targets the suspected file-guard race conditions. |
| *(removed `rsize`/`wsize`)* | Lets the NFS client and server auto-negotiate optimal transfer sizes (typically 1 MB on NFSv4.2), improving throughput for large parquet writes and reducing round-trips. |
| *(removed `timeo`)* | With `hard` mount, timeout is irrelevant; operations retry until the server responds. |

### Impact on RQ Job Dependencies and File Guards

The catalog documents two layers of ordering between RQ jobs: explicit `depends_on` edges (job-level) and `wait_for_path` / `wait_for_paths` file guards (filesystem-level). If the NFS attribute cache hypothesis is correct, it would affect the second layer directly and the first layer indirectly.

**Explicit `depends_on` edges** (job-level ordering) are not themselves affected by NFS caching. RQ tracks job completion in Redis, so a `depends_on` edge fires when the upstream job's Python function returns, not when the filesystem flushes. However, a job returning successfully does not guarantee that its output files are visible to other NFS clients—this is the gap the file guards fill. The edge graph in this catalog was verified through several auditing and review passes during this investigation.

**File guards** (`wait_for_path` / `wait_for_paths` with stable-size checks) are the suspected primary failure point. Under a stale attribute cache scenario:

1. `stat()` calls inside `wait_for_path` could return `ENOENT` even after the file existed on the server.
2. When `stat()` did succeed, it could report a stale size (often 0), causing the stable-size check to loop until timeout.
3. Under a `soft` mount, the timeout could produce an `EIO` instead of a retry, crashing the worker.

The following call chains were traced end-to-end from rq-engine API entry through RQ job execution to file-guard resolution:

#### Trace: `POST /rq-engine/run-sync` → Full WEPP Run

```
rq-engine API: POST /rq-engine/run-sync
  └─ run_sync_rq                          [depends_on: none]
       └─ migrations_rq                   [depends_on: run_sync_rq]
            └─ run_wepp_rq(runid)         [depends_on: none]
                 ├─ jobs:0  _prep_*_rq                        [depends_on: none]
                 ├─ jobs:0  _prep_remaining_rq                [depends_on: jobs0 prep list]
                 ├─ jobs:1  _run_hillslopes_rq                [depends_on: _prep_remaining_rq]
                 ├─ jobs:2  _prep_watershed_rq                [depends_on: _run_hillslopes_rq]
                 ├─ jobs:2  _build_hillslope_interchange_rq   [depends_on: _run_hillslopes_rq]
                 │    └─ READS: output_dir/H*.pass.dat, H*.ebe.dat, H*.wat.dat, ...
                 │    └─ WRITES: output_dir/interchange/H.pass.parquet, H.wat.parquet, ...
                 ├─ jobs:2  _build_totalwatsed3_rq            [depends_on: _build_hillslope_interchange_rq]
                 │    └─ FILE GUARD: wait_for_paths → H.pass.parquet, H.wat.parquet  ← NFS-affected
                 │    └─ WRITES: output_dir/interchange/totalwatsed3.parquet
                 ├─ jobs:3  run_watershed_rq                  [depends_on: _prep_watershed_rq]
                 │    └─ WRITES: output_dir/pass_pw0.txt, ebe_pw0.txt, chan.out, ...
                 ├─ jobs:4  _post_run_cleanup_out_rq          [depends_on: jobs3_watersheds]
                 ├─ jobs:4  _post_watershed_interchange_rq    [depends_on: _post_run_cleanup_out_rq]
                 │    └─ FILE GUARD: wait_for_path → pass_pw0.txt, ebe_pw0.txt,
                 │       chan.out, chanwb.out, chnwb.txt, soil_pw0.txt  ← NFS-affected
                 │    └─ WRITES: output_dir/interchange/ebe_pw0.parquet, ...
                 ├─ jobs:4  _run_hillslope_watbal_rq          [depends_on: post_deps + interchange]
                 │    └─ FILE GUARD: wait_for_path → H.wat.parquet  ← NFS-affected
                 ├─ jobs:4  _analyze_return_periods_rq        [depends_on: _post_watershed_interchange + totalwatsed3]
                 │    └─ FILE GUARD: wait_for_paths → ebe_pw0.parquet,
                 │       totalwatsed3.parquet  ← NFS-affected
                 ├─ jobs:5  _post_gpkg_export_rq              [depends_on: jobs4_post]
                 └─ jobs:6  _log_complete_rq                  [depends_on: jobs4+5]
```

Every `← NFS-affected` guard above is a candidate site for stale-cache failures under the old mount. With `actimeo=0`, these guards should resolve on the first or second poll cycle (typically < 1 s) instead of timing out.

#### Trace: Subcatchment Build → Abstract Watershed

```
rq-engine API: POST /api/subcatchments/build
  └─ build_subcatchments_rq               [depends_on: prior same-run child, if any]
       ├─ FILE GUARD: wait_for_path → watershed.subwta  ← NFS-affected
       └─ abstract watershed under the same directory-root lock
  └─ abstract_watershed_rq               [nonmutating receipt; depends_on: build_subcatchments_rq]
```

#### Trace: Fork with Undisturbify

```
rq-engine API: POST /api/fork
  └─ fork_rq(undisturbify=True)
       └─ FILE GUARD: wait_for_paths → ron.nodb, wepp.nodb, landuse.nodb,
          soils.nodb, disturbed.nodb  ← NFS-affected
       └─ run_wepp_rq(new_runid)          [full chain as above]
       └─ _finish_fork_rq                 [depends_on: _log_complete_rq]
```

### Summary

| Before | After |
| --- | --- |
| `soft` mount with fixed buffer sizes; stale attribute cache | `hard,intr` mount with `actimeo=0`; auto-negotiated transfer sizes |
| File guards timed out or received `EIO` under load | File guards should resolve immediately; operations block until server responds |
| Non-deterministic run failures correlated with NFS client count | Expected deterministic file visibility across all NFS clients |

The `depends_on` job graph documented in this catalog was reviewed through several auditing passes as part of this investigation—no missing edges were identified. However, the job dependency graph is not readily testable in isolation: edges are constructed dynamically at enqueue time, depend on runtime configuration (e.g. `run_watershed`, `delete_after_interchange`, SWAT/Omni enablement), and span multiple modules. Validation relies primarily on cognitive review of the code paths rather than automated verification. The interplay of RQ job orchestration, NoDb state, Rust-based interchange, and Fortran WEPP subprocesses makes constructing meaningful automated tests for the dependency graph near impossible, and the regression surface for dependency changes is correspondingly difficult to characterize.

The failures are consistent with the NFS attribute cache allowing downstream file guards to poll stale metadata, though other contributing factors have not been ruled out. The mount remediation targets this class of failure without requiring changes to job dependency wiring or wait timeouts. Continued monitoring is needed to confirm effectiveness.

## Checklist For New Child Jobs

### Notification inbox population

`docker/scheduled-tasks.yml` admits `wepppy.rq.notifications_rq.populate_notifications`
on the existing `batch` queue only when `WEPPCLOUD_NOTIFICATIONS_SWEEP_ENABLED=true`.
It has no child jobs or GC/access-compilation dependency. Its database sweep lock
and episode uniqueness make retries safe. Trusted test calls may pass a bounded
registration-ID scope; browser routes never enqueue this task. See
`docs/schemas/user-notifications-contract.md` and `docs/dev-notes/user-notifications.md`.

### Child-job checklist
1. Add `depends_on` for any file or controller state produced by upstream jobs.
2. Record `job.meta["jobs:<order>,..."]` for every child job so `job_info.py` and `cancel_job.py` can traverse the tree.
3. If you rely on filesystem writes inside a single job, prefer explicit file waits over fixed sleeps.
4. Update this catalog when adding new enqueue sites or dependency edges.
