WEPPcloud

← Back to help

RQ Job Dependencies Catalog

Catalog of RQ entrypoints that enqueue child jobs and the dependency graph between them. Includes an auto-generated dependency graph section plus analyst-maintained narrative and audit notes.

Scope

  • Modules: wepppy/rq/*.py (entrypoints that enqueue other jobs).
  • rq-engine routes that enqueue RQ jobs directly: wepppy/microservices/rq_engine/batch_routes.py, wepppy/microservices/rq_engine/bootstrap_routes.py, wepppy/microservices/rq_engine/geneva_routes.py, and wepppy/microservices/rq_engine/run_sync_routes.py.
  • rq-initiated Flask handlers that enqueue RQ jobs: wepppy/weppcloud/routes/**/*.py and wepppy/weppcloud/bootstrap/*.py.

Uploaded land-use builds

The normal build-landuse endpoint uses the existing default project_rq.build_landuse_rq job family. An uploaded raster adds a preallocation check that stages the raw and aligned inputs and a job-bound receipt. This submission adds no edge or worker queue. The worker checks the receipt and both directory leases before applying the upload. Previously enqueued jobs without upload arguments continue to build their existing aligned inputs. Setting batch inputs remains synchronous. Batch leaves validate and align retained raw inputs under the existing directory leases when their own watershed becomes available.

Declarative DEM slice

The private wepppy.declarative.adapter.submit enqueues execute_dem on one declarative-dem-<submission UUID hex> queue. This adapter calls shared project_rq.fetch_dem_rq synchronously and never waits for queued children. After science settlement the operator invokes adapter.finalize, which enqueues one finalize_dem job on the same queue; it calls shared set_run_readonly_rq. No RQ dependency edge, scientific retry or ordinary queue consumer is added. The engine waits outside the single worker. Durable claims prevent replay after uncertain execution. These enqueue sites are outside the generator's existing wepppy/rq/route scan; this narrative explicitly inventories them.

Query-engine asset catalog maintenance

wepppy.rq.query_catalog_rq.enqueue_catalog_refresh admits one refresh_query_catalog_rq job per exact scope/incarnation on the existing default queue. Producer requests persist filesystem receipts before admission. Active queued/started/deferred/scheduled work coalesces; reservations remain through RQ terminal transition so late invalidations cannot create a second active job. Auth-actor and correlation metadata use the same Redis transaction as admission; a WATCH conflict must not leave a metadata-only job hash. The shipped query_catalog_refresh scheduler task selects default and passes that queue explicitly. The scheduler invokes reconcile_query_catalogs to admit one active inventory_query_catalog_rq per configured inventory cycle on default and drain its durable Redis discoveries. The inventory worker streams configured roots, checkpoints its frontier and fences every publication by current job/cycle. Scheduler ticks acknowledge discovered scopes atomically with their pending cursor. Restart replays the current directory prefix with deduplication. Neither scheduler nor inventory worker executes native catalog scans; inventory writes no project files. No model dependency edges are added.

Maintenance IDs use query_catalog_refresh_<canonical UUID> and generic metadata. Polling classification precedes deserialization. Every live eligible default consumer must advertise matching protocol/storage/root/coordination and inventory_protocol: 1, admission_protocol: 1, pilot_protocol: 1, and queue: default in query_catalog_configuration; missing/incompatible/suspended capacity leaves durable requests pending. Execution revalidates configuration and incarnation.

The global Redis admission ledger caps outstanding catalog work at seven across configurations and pilots; reservations and uncertain job records retain capacity. The explicit operator pilot requires all twelve compatible default consumers, exact owned scopes, a live observation lease, and immutable retry-inclusive budgets (25 refresh, two inventory). Ordinary waiting work pauses admission. Pilot inventory enumerates only approved locators; admitted jobs may drain after pause or expiry.

QUERY_CATALOG_REFRESH_ENABLED defaults false for coordinated rollout. Disabled producer migration preserves synchronous compatibility; enabled Redis/worker failure must not fall back to inline scanning. Scheduler interval defaults 30 s; inventory defaults to 100 entry records per streaming checkpoint and 100 scope records drained per scheduler tick; admission remains 25 scopes per tick. QUERY_CATALOG_CYCLE_SECONDS defaults 86400 as an unproven capacity target; retry delay starts at 30 s and caps at 3600 s. These are configurable operational limits, not a freshness guarantee during unavailable storage/workers. The inventory cursor and cycle errors are operator-visible Redis state; loss resets bounded discovery against configured roots, while project receipt/attempt files retain durable intent and evidence. Full cycles include scopes without receipts so artifact-publication crashes cannot leave metadata stale forever. Discovery completion is separate from cycle completion: completed_at is recorded only after all discovered scopes acknowledge their pending receipts. Current and previous cycle errors remain visible without accumulating older cycles. The scheduler lease is 60 seconds; expired owners cannot overwrite newer cursors.

Enabled mode requires explicit QUERY_CATALOG_ROOTS_JSON; missing configuration fails preflight rather than guessing which legacy namespaces are active. Set the same configuration in producers, scheduler, and every eligible default consumer. For a deployment using the three canonical /wc1 namespaces, the JSON value is:

[
  {"path":"/wc1/runs","namespace":"wc1:runs","coordination":"/wc1/query-engine-coordination/locks","layout":"sharded"},
  {"path":"/wc1/batch","namespace":"wc1:batch","coordination":"/wc1/query-engine-coordination/locks","layout":"batch"},
  {"path":"/wc1/culverts","namespace":"wc1:culverts","coordination":"/wc1/query-engine-coordination/locks","layout":"culverts"}
]

Add legacy roots only when that deployment actively serves them. Verify each configured namespace and the external coordination directory under the actual runtime identity before enabling. Once configured, missing storage remains an outage: inventory reports degraded health and retries on the next tick. It never silently removes an unavailable namespace from the configured inventory.

Use suspend_catalog_refresh(connection, scope_id) for intentional operator suspension and resume_catalog_refresh to allow future ticks; ordinary canceled or failed jobs otherwise remain recoverable. Never purge pending receipt files or external lifecycle lock inodes as queue cleanup. See the canonical query catalog contract and execution package.

Path Legend

Token Meaning
wd Run working directory (e.g., /wc1/runs/<prefix>/<runid>).
runs_dir Wepp.runs_dir (typically wd/wepp/runs).
output_dir Wepp.output_dir (typically wd/wepp/output).
plot_dir Wepp.plot_dir (typically wd/wepp/plots).
export_dir Wepp.export_dir (typically wd/export).
wat_dir Watershed.wat_dir (watershed workspace; varies by backend).
cli_dir Wepp.cli_dir (climate .cli directory).
soils_dir Soils.soils_dir (source soils).
lc_dir Landuse.lc_dir (landuse/management templates).
omni_dir wd/omni (OMNI_REL_DIR).
swat_txtinout_dir Swat.swat_txtinout_dir (SWAT TxtInOut).
dem_dir Ron.dem_dir (DEM staging).

Conventions

  • Parent jobs record child ids in job.meta["jobs:<order>,..."] for job_info.py and cancel_job.py.
  • depends_on uses a single job or a list; when the list is empty, it is passed as None (no dependency).
  • Required-output dependency edges are strict. A failed prerequisite leaves executable downstream work never-started/deferred; ordinary controller resubmission cancels and detaches that obsolete graph before replacement.
  • Failure tolerance is limited to the finalizers and independent AgFields, Omni-contrast, and WBT request-serialization edges enumerated in the SURF-20A dependency matrix. Those sites use failure_tolerant_depends_on (or an explicit Dependency(allow_failure=True)) plus release_deferred_job_if_ready for the already-failed-before-registration race.
  • WBT controlled policy failures retain their owning exception: they explicitly cancel the never-started abstraction receipt rather than leaving it deferred.
  • Prereq files list explicit file/dir inputs referenced by the job or immediate controller calls. Optional inputs are marked.
  • Paths are shown relative to wd unless noted; placeholders like <topaz> and <wepp_id> indicate per-hillslope files.

Batch result settlement

Enrolled ordinary/culvert roots freeze selected run IDs and exact source job IDs before dispatch. Culvert leaves depend on the root dispatch barrier; its finalizer depends on root and leaves with failure tolerance. Ordinary hillslope/watershed stages retain the existing root barrier and linked Omni completion edges. All enrolled execution identities use non-expiring RQ retention.

Child creation/linkage and cancellation share the existing root resource lease. Cancellation refreshes the tree after acquiring that lease. Publication validates and builds outside the lease, then rechecks current attempt, execution states and cancellation before the short manifest/ZIP/summary commit. Polling never builds artifacts. Exhausted failed Omni dependencies permit only verified blocked deferred Omni descendants to be canceled and the failure-tolerant batch finalizer released; foreign or changed graphs remain untouched. See the batch outcome contract.

Scheduled WMesque Synchronization

docker/scheduled-tasks.yml routes the opt-in wmesque_sync task through wepppy.tools.scheduler._enqueue_task to wepppy.rq.wmesque_rq.enqueue_sync on batch. The generic queue parameter appears as default/untracked in static extraction below; the scheduled binding is batch. It enqueues one standalone sync_wmesque_rq job with no dependency edges, coalescing queued/running work by destination. Source and destination are snapshotted into job arguments. Live forest acceptance job IDs and terminal states are retained in the WMesque work package. Task remains disabled by default.

Automated Dependency Graph

This section is generated by tools/render_rq_dependency_graph_docs.py from wepppy/rq/job-dependency-graph.static.json. Do not edit this block manually.

  • Extracted enqueue edges: 154
  • Unique enqueue targets: 98
  • Source globs: wepppy/rq/*.py, wepppy/microservices/rq_engine/*.py, wepppy/weppcloud/routes/**/*.py, wepppy/weppcloud/bootstrap/*.py

wepppy/microservices/rq_engine/ag_fields_routes.py::_enqueue_job

  • line 229 queue default stage untracked enqueue func depends_on (none)

wepppy/microservices/rq_engine/ag_fields_routes.py::_enqueue_watershed_jobs

  • line 296 queue default stage untracked enqueue run_ag_fields_watershed_rq depends_on (none)
  • line 322 queue default stage untracked enqueue run_ag_fields_watershed_suite_rq depends_on (none)

wepppy/microservices/rq_engine/ash_routes.py::run_ash

  • line 376 queue default stage untracked enqueue run_ash_rq depends_on (none)

wepppy/microservices/rq_engine/batch_routes.py::delete_batch

  • line 194 queue batch stage untracked enqueue delete_batch_rq depends_on (none)

wepppy/microservices/rq_engine/batch_routes.py::run_batch

  • line 116 queue batch stage untracked enqueue run_batch_rq depends_on (none)

wepppy/microservices/rq_engine/bootstrap_routes.py::_enqueue_no_prep_job

  • line 220 queue default stage untracked enqueue job_fn depends_on (none)

wepppy/microservices/rq_engine/climate_routes.py::build_climate

  • line 743 queue default stage untracked enqueue build_climate_rq depends_on (none)

wepppy/microservices/rq_engine/culvert_routes.py::_enqueue_culvert_job

  • line 218 queue default stage untracked enqueue queue depends_on (none)
  • line 221 queue batch stage untracked enqueue func depends_on (none)

wepppy/microservices/rq_engine/debris_flow_routes.py::run_debris_flow

  • line 148 queue default stage untracked enqueue run_debris_flow_rq depends_on (none)

wepppy/microservices/rq_engine/debug_routes.py::hello_world

  • line 54 queue m4 stage untracked enqueue hello_world_rq depends_on (none)

wepppy/microservices/rq_engine/dss_export_routes.py::post_dss_export

  • line 199 queue default stage untracked enqueue post_dss_export_rq depends_on (none)

wepppy/microservices/rq_engine/export_routes.py::_enqueue_ermit_export_job

  • line 280 queue default stage untracked enqueue run_ermit_export_rq depends_on (none)

wepppy/microservices/rq_engine/export_routes.py::_enqueue_features_export_job

  • line 378 queue default stage untracked enqueue target_func depends_on (none)

wepppy/microservices/rq_engine/fork_archive_routes.py::archive_run

  • line 1740 queue fork-archive stage untracked enqueue archive_rq depends_on (none)

wepppy/microservices/rq_engine/fork_archive_routes.py::fork_project

  • line 1607 queue fork-archive stage untracked enqueue fork_rq depends_on (none)

wepppy/microservices/rq_engine/fork_archive_routes.py::restore_archive

  • line 1848 queue fork-archive stage untracked enqueue restore_archive_rq depends_on (none)

wepppy/microservices/rq_engine/geneva_routes.py::_enqueue_geneva_job

  • line 306 queue default stage untracked enqueue func depends_on (none)

wepppy/microservices/rq_engine/geneva_routes.py::_enqueue_geneva_workflow_jobs

  • line 358 queue default stage untracked enqueue run_geneva_prepare_hrus_rq depends_on (none)
  • line 365 queue default stage untracked enqueue run_geneva_build_frequency_panel_rq depends_on run_geneva_prepare_hrus_rq
  • line 374 queue default stage untracked enqueue run_geneva_run_batch_rq depends_on run_geneva_build_frequency_panel_rq

wepppy/microservices/rq_engine/landuse_routes.py::build_landuse

  • line 885 queue unknown stage untracked enqueue build_landuse_rq depends_on (none)
  • line 949 queue default stage untracked enqueue build_landuse_rq depends_on (none)

wepppy/microservices/rq_engine/landuse_routes.py::modify_landuse_mapping

  • line 1266 queue default stage untracked enqueue modify_landuse_mapping_rq depends_on (none)

wepppy/microservices/rq_engine/landuse_soils_routes.py::build_landuse_and_soils

  • line 64 queue default stage untracked enqueue land_and_soil_rq depends_on (none)

wepppy/microservices/rq_engine/migration_routes.py::_enqueue_migration_job

  • line 179 queue default stage untracked enqueue migrations_rq depends_on (none)

wepppy/microservices/rq_engine/omni_routes.py::_delete_omni_contrasts

  • line 1323 queue batch stage untracked enqueue delete_omni_contrasts_rq depends_on (none)

wepppy/microservices/rq_engine/omni_routes.py::_run_omni

  • line 969 queue batch stage untracked enqueue run_omni_scenarios_rq depends_on (none)

wepppy/microservices/rq_engine/omni_routes.py::_run_omni_contrasts

  • line 1141 queue batch stage untracked enqueue run_omni_contrasts_rq depends_on (none)

wepppy/microservices/rq_engine/openet_ts_routes.py::acquire_openet_ts

  • line 88 queue default stage untracked enqueue fetch_and_analyze_openet_ts_rq depends_on (none)

wepppy/microservices/rq_engine/polaris_routes.py::acquire_polaris

  • line 91 queue default stage untracked enqueue fetch_and_align_polaris_rq depends_on (none)

wepppy/microservices/rq_engine/postfire_debris_flow_routes.py::enqueue

  • line 151 queue default stage untracked enqueue upload_dnbr_rq if kind == 'upload_attempt' else run_m3_rq if record.get('model') == 'M3' else run_m1_rq depends_on (none)

wepppy/microservices/rq_engine/project_config_update_routes.py::update_apply

  • line 703 queue default stage untracked enqueue <unknown> depends_on (none)

wepppy/microservices/rq_engine/rap_ts_routes.py::acquire_rap_ts

  • line 160 queue default stage untracked enqueue fetch_and_analyze_rap_ts_rq depends_on (none)

wepppy/microservices/rq_engine/rhem_routes.py::run_rhem

  • line 85 queue default stage untracked enqueue run_rhem_rq depends_on (none)

wepppy/microservices/rq_engine/roads_routes.py::_enqueue_roads_job

  • line 60 queue default stage untracked enqueue func depends_on (none)

wepppy/microservices/rq_engine/run_sync_routes.py::run_sync

  • line 280 queue default stage untracked enqueue run_sync_rq depends_on (none)
  • line 288 queue default stage untracked enqueue run_sync_rq depends_on (none)
  • line 319 queue default stage untracked enqueue migrations_rq depends_on run_sync_rq

wepppy/microservices/rq_engine/rusle_routes.py::build_rusle

  • line 100 queue default stage untracked enqueue build_rusle_rq depends_on (none)

wepppy/microservices/rq_engine/soils_routes.py::build_soils

  • line 334 queue default stage untracked enqueue build_soils_rq depends_on (none)

wepppy/microservices/rq_engine/swat_routes.py::run_swat

  • line 111 queue default stage untracked enqueue run_swat_rq depends_on (none)

wepppy/microservices/rq_engine/treatments_routes.py::build_treatments

  • line 211 queue default stage untracked enqueue build_treatments_rq depends_on (none)

wepppy/microservices/rq_engine/upload_climate_routes.py::upload_cli

  • line 212 queue default stage untracked enqueue upload_cli_rq depends_on (none)

wepppy/microservices/rq_engine/watershed_routes.py::build_subcatchments_and_abstract_watershed

  • line 1164 queue default stage untracked enqueue build_subcatchments_and_abstract_watershed_rq depends_on (none)

wepppy/microservices/rq_engine/watershed_routes.py::fetch_dem_and_build_channels

  • line 809 queue default stage untracked enqueue fetch_dem_and_build_channels_rq depends_on (none)

wepppy/microservices/rq_engine/watershed_routes.py::set_outlet

  • line 935 queue default stage untracked enqueue set_outlet_rq depends_on (none)

wepppy/microservices/rq_engine/webhooks.py::enqueue

  • line 214 queue default stage untracked enqueue func depends_on (none)

wepppy/microservices/rq_engine/wepp_routes.py::_handle_run_wepp_request

  • line 237 queue default stage untracked enqueue q depends_on (none)
  • line 240 queue default stage untracked enqueue job_fn depends_on (none)

wepppy/rq/ag_fields_rq.py::run_ag_fields_watershed_suite_rq

  • line 494 queue default stage untracked enqueue run_ag_fields_watershed_rq depends_on Dependency(jobs=[previous_job_id], allow_failure=True)
  • line 520 queue default stage untracked enqueue finalize_ag_fields_watershed_suite_rq depends_on Dependency(jobs=[child_job.id for child_job in scheme_jobs], allow_failure=True)

wepppy/rq/batch_rq.py::run_batch_rq

  • line 744 queue batch stage jobs:0 enqueue run_batch_hillslopes_rq depends_on job_id
  • line 755 queue batch stage jobs:0 enqueue run_batch_watershed_rq depends_on Dependency(jobs=[hillslope_job_id], allow_failure=True)
  • line 780 queue batch stage jobs:1 enqueue _final_batch_complete_rq depends_on Dependency(jobs=[watershed_job.id for watershed_job in watershed_jobs], allow_failure=True)

wepppy/rq/culvert_rq_pipeline.py::enqueue_culvert_batch_jobs

  • line 87 queue batch stage jobs:0 enqueue tasks.run_culvert_run_rq depends_on parent_job.id
  • line 106 queue batch stage jobs:1 enqueue tasks._final_culvert_batch_complete_rq depends_on *child_jobs, parent_job, tasks.run_culvert_run_rq

wepppy/rq/omni_rq.py::run_omni_contrasts_rq

  • line 928 queue batch stage jobs:contrast enqueue run_omni_contrast_rq depends_on _failure_tolerant_depends_on(batch_depends)
  • line 943 queue batch stage jobs:finalize enqueue _finalize_omni_contrasts_rq depends_on _failure_tolerant_depends_on(batch_depends)

wepppy/rq/omni_rq.py::run_omni_scenarios_rq

  • line 733 queue batch stage jobs:0 enqueue run_omni_scenario_rq depends_on (none)
  • line 752 queue batch stage jobs:1 enqueue run_omni_scenario_rq depends_on run_omni_scenario_rq
  • line 771 queue batch stage jobs:2 enqueue _compile_hillslope_summaries_rq depends_on run_omni_scenario_rq
  • line 783 queue batch stage jobs:3 enqueue _finalize_omni_scenarios_rq depends_on _compile_hillslope_summaries_rq

wepppy/rq/project_rq.py::_enqueue_serial_subcatchment_tree

  • line 787 queue default stage untracked enqueue build_subcatchments_rq depends_on Dependency(jobs=[previous_tail_id], allow_failure=True)
  • line 795 queue default stage untracked enqueue abstract_watershed_rq depends_on build_subcatchments_rq

wepppy/rq/project_rq.py::fetch_dem_and_build_channels_rq

  • line 1652 queue default stage jobs:0 enqueue build_channels_rq depends_on (none)
  • line 1668 queue default stage jobs:0 enqueue fetch_dem_rq depends_on (none)
  • line 1676 queue default stage jobs:1 enqueue build_channels_rq depends_on fetch_dem_rq

wepppy/rq/project_rq.py::fork_rq

  • line 3133 queue default stage jobs:1 enqueue _finish_fork_rq depends_on Dependency(jobs=[final_wepp_job.id], allow_failure=True)

wepppy/rq/query_catalog_rq.py::_enqueue_inventory

  • line 364 queue default stage untracked enqueue inventory_query_catalog_rq depends_on (none)

wepppy/rq/query_catalog_rq.py::enqueue_catalog_refresh

  • line 216 queue default stage untracked enqueue refresh_query_catalog_rq depends_on (none)

wepppy/rq/run_catalog_rq.py::enqueue_sweep

  • line 124 queue default stage untracked enqueue sweep_run_catalog depends_on (none)

wepppy/rq/swat_rq.py::run_swat_noprep_rq

  • line 118 queue default stage jobs:0 enqueue _run_swat_rq depends_on (none)

wepppy/rq/swat_rq.py::run_swat_rq

  • line 71 queue default stage jobs:0 enqueue _build_swat_inputs_rq depends_on (none)
  • line 75 queue default stage jobs:1 enqueue _run_swat_rq depends_on _build_swat_inputs_rq

wepppy/rq/wepp_rq.py::run_wepp_watershed_noprep_rq

  • line 1113 queue default stage jobs:6 enqueue sys.modules[__name__]._log_complete_rq depends_on (none)

wepppy/rq/wepp_rq.py::run_wepp_watershed_rq

  • line 968 queue default stage jobs:6 enqueue sys.modules[__name__]._log_complete_rq depends_on (none)

wepppy/rq/wepp_rq_pipeline.py::enqueue_log_prep_complete

  • line 90 queue default stage jobs:6 enqueue tasks._log_prep_complete_rq depends_on depends_on

wepppy/rq/wepp_rq_pipeline.py::enqueue_watershed_noprep_pipeline

  • line 926 queue default stage jobs:3 enqueue tasks.run_ss_batch_watershed_rq depends_on (none)
  • line 938 queue default stage jobs:3 enqueue tasks.run_watershed_rq depends_on (none)
  • line 951 queue default stage jobs:4 enqueue tasks._post_run_cleanup_out_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq
  • line 964 queue default stage jobs:4 enqueue tasks._post_prep_details_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq
  • line 978 queue default stage jobs:4 enqueue tasks._post_make_loss_grid_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq
  • line 991 queue default stage jobs:4 enqueue tasks._post_watershed_interchange_rq depends_on tasks._post_run_cleanup_out_rq
  • line 1005 queue default stage jobs:5 enqueue tasks._post_legacy_arc_export_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq
  • line 1017 queue default stage jobs:6 enqueue tasks._log_complete_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._post_legacy_arc_export_rq

wepppy/rq/wepp_rq_pipeline.py::enqueue_watershed_pipeline

  • line 785 queue default stage jobs:2 enqueue tasks._prep_watershed_rq depends_on (none)
  • line 799 queue default stage jobs:3 enqueue tasks.run_ss_batch_watershed_rq depends_on tasks._prep_watershed_rq
  • line 812 queue default stage jobs:3 enqueue tasks.run_watershed_rq depends_on tasks._prep_watershed_rq
  • line 826 queue default stage jobs:4 enqueue tasks._post_run_cleanup_out_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq
  • line 839 queue default stage jobs:4 enqueue tasks._post_prep_details_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq
  • line 853 queue default stage jobs:4 enqueue tasks._post_make_loss_grid_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq
  • line 866 queue default stage jobs:4 enqueue tasks._post_watershed_interchange_rq depends_on tasks._post_run_cleanup_out_rq
  • line 880 queue default stage jobs:5 enqueue tasks._post_legacy_arc_export_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq
  • line 892 queue default stage jobs:6 enqueue tasks._log_complete_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._post_legacy_arc_export_rq

wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_noprep_pipeline

  • line 571 queue default stage jobs:1 enqueue tasks._run_hillslopes_rq depends_on (none)
  • line 580 queue default stage jobs:2 enqueue tasks._build_hillslope_interchange_rq depends_on tasks._run_hillslopes_rq
  • line 592 queue default stage jobs:2 enqueue tasks._build_totalwatsed3_rq depends_on tasks._build_hillslope_interchange_rq
  • line 613 queue default stage jobs:3 enqueue tasks.run_ss_batch_watershed_rq depends_on tasks._run_hillslopes_rq
  • line 626 queue default stage jobs:3 enqueue tasks.run_watershed_rq depends_on tasks._run_hillslopes_rq
  • line 639 queue default stage jobs:4 enqueue tasks._post_run_cleanup_out_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._run_hillslopes_rq
  • line 652 queue default stage jobs:4 enqueue tasks._post_prep_details_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._run_hillslopes_rq
  • line 667 queue default stage jobs:4 enqueue tasks._run_hillslope_watbal_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._run_hillslopes_rq, tasks._build_hillslope_interchange_rq
  • line 680 queue default stage jobs:4 enqueue tasks._post_make_loss_grid_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._run_hillslopes_rq
  • line 691 queue default stage jobs:4 enqueue tasks._post_watershed_interchange_rq depends_on tasks._post_run_cleanup_out_rq, tasks._build_hillslope_interchange_rq
  • line 704 queue default stage jobs:4 enqueue tasks._analyze_return_periods_rq depends_on tasks._post_watershed_interchange_rq, tasks._build_totalwatsed3_rq
  • line 719 queue default stage jobs:2 enqueue tasks.post_dss_export_rq depends_on tasks._build_hillslope_interchange_rq, tasks._post_run_cleanup_out_rq
  • line 732 queue default stage jobs:5 enqueue tasks._post_legacy_arc_export_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._run_hillslope_watbal_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._analyze_return_periods_rq
  • line 745 queue default stage jobs:5 enqueue tasks._post_gpkg_export_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._run_hillslope_watbal_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._analyze_return_periods_rq
  • line 763 queue default stage jobs:6 enqueue tasks._log_complete_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._run_hillslope_watbal_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._analyze_return_periods_rq, tasks._post_legacy_arc_export_rq, tasks._post_gpkg_export_rq, tasks._build_hillslope_interchange_rq, tasks._build_totalwatsed3_rq, tasks.post_dss_export_rq

wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_pipeline

  • line 116 queue default stage jobs:0 enqueue tasks._prep_multi_ofe_rq depends_on (none)
  • line 127 queue default stage jobs:0 enqueue tasks._prep_slopes_rq depends_on (none)
  • line 137 queue default stage jobs:0 enqueue tasks._prep_managements_rq depends_on (none)
  • line 146 queue default stage jobs:0 enqueue tasks._prep_soils_rq depends_on (none)
  • line 157 queue default stage jobs:0 enqueue tasks._prep_climates_rq depends_on (none)
  • line 167 queue default stage jobs:0 enqueue tasks._prep_remaining_rq depends_on tasks._prep_multi_ofe_rq, tasks._prep_slopes_rq, tasks._prep_managements_rq, tasks._prep_soils_rq, tasks._prep_climates_rq
  • line 177 queue default stage jobs:1 enqueue tasks._run_hillslopes_rq depends_on tasks._prep_remaining_rq
  • line 190 queue default stage jobs:2 enqueue tasks._prep_watershed_rq depends_on tasks._run_hillslopes_rq
  • line 217 queue default stage jobs:2 enqueue tasks._build_swat_inputs_rq depends_on tasks._run_hillslopes_rq, tasks._prep_watershed_rq
  • line 226 queue default stage jobs:3 enqueue tasks._run_swat_rq depends_on tasks._build_swat_inputs_rq
  • line 239 queue default stage jobs:2 enqueue tasks._build_hillslope_interchange_rq depends_on tasks._run_hillslopes_rq, tasks._build_swat_inputs_rq
  • line 250 queue default stage jobs:2 enqueue tasks._build_totalwatsed3_rq depends_on tasks._build_hillslope_interchange_rq
  • line 264 queue default stage jobs:2 enqueue tasks._build_swat_inputs_rq depends_on tasks._build_hillslope_interchange_rq, tasks._prep_watershed_rq
  • line 273 queue default stage jobs:3 enqueue tasks._run_swat_rq depends_on tasks._build_swat_inputs_rq
  • line 290 queue default stage jobs:3 enqueue tasks.run_ss_batch_watershed_rq depends_on tasks._prep_watershed_rq
  • line 303 queue default stage jobs:3 enqueue tasks.run_watershed_rq depends_on tasks._prep_watershed_rq
  • line 316 queue default stage jobs:4 enqueue tasks._post_run_cleanup_out_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq
  • line 329 queue default stage jobs:4 enqueue tasks._post_prep_details_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq
  • line 345 queue default stage jobs:4 enqueue tasks._run_hillslope_watbal_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq, tasks._build_hillslope_interchange_rq
  • line 358 queue default stage jobs:4 enqueue tasks._post_make_loss_grid_rq depends_on tasks.run_ss_batch_watershed_rq, tasks.run_watershed_rq, tasks._prep_watershed_rq
  • line 369 queue default stage jobs:4 enqueue tasks._post_watershed_interchange_rq depends_on tasks._post_run_cleanup_out_rq, tasks._build_hillslope_interchange_rq
  • line 382 queue default stage jobs:4 enqueue tasks._analyze_return_periods_rq depends_on tasks._post_watershed_interchange_rq, tasks._build_totalwatsed3_rq
  • line 397 queue default stage jobs:2 enqueue tasks.post_dss_export_rq depends_on tasks._build_hillslope_interchange_rq, tasks._post_run_cleanup_out_rq
  • line 410 queue default stage jobs:5 enqueue tasks._post_legacy_arc_export_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._run_hillslope_watbal_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._analyze_return_periods_rq
  • line 423 queue default stage jobs:5 enqueue tasks._post_gpkg_export_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._run_hillslope_watbal_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._analyze_return_periods_rq
  • line 444 queue default stage jobs:6 enqueue tasks._log_complete_rq depends_on tasks._post_run_cleanup_out_rq, tasks._post_prep_details_rq, tasks._run_hillslope_watbal_rq, tasks._post_make_loss_grid_rq, tasks._post_watershed_interchange_rq, tasks._analyze_return_periods_rq, tasks._post_legacy_arc_export_rq, tasks._post_gpkg_export_rq, tasks._build_hillslope_interchange_rq, tasks._build_totalwatsed3_rq, tasks.post_dss_export_rq, tasks._run_swat_rq

wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_prep_only_pipeline

  • line 470 queue default stage jobs:0 enqueue tasks._prep_multi_ofe_rq depends_on (none)
  • line 481 queue default stage jobs:0 enqueue tasks._prep_slopes_rq depends_on (none)
  • line 491 queue default stage jobs:0 enqueue tasks._prep_managements_rq depends_on (none)
  • line 501 queue default stage jobs:0 enqueue tasks._prep_soils_rq depends_on (none)
  • line 512 queue default stage jobs:0 enqueue tasks._prep_climates_rq depends_on (none)
  • line 522 queue default stage jobs:0 enqueue tasks._prep_remaining_rq depends_on tasks._prep_multi_ofe_rq, tasks._prep_slopes_rq, tasks._prep_managements_rq, tasks._prep_soils_rq, tasks._prep_climates_rq
  • line 532 queue default stage jobs:2 enqueue tasks._prep_watershed_rq depends_on tasks._prep_remaining_rq

wepppy/rq/wmesque_rq.py::enqueue_sync

  • line 48 queue default stage untracked enqueue sync_wmesque_rq depends_on (none)

wepppy/weppcloud/bootstrap/enable_jobs.py::enqueue_bootstrap_enable

  • line 122 queue default stage untracked enqueue bootstrap_enable_rq depends_on (none)

wepppy/weppcloud/routes/nodb_api/geneva_bp.py::_enqueue_geneva_job

  • line 252 queue default stage untracked enqueue func depends_on (none)

wepppy/weppcloud/routes/nodb_api/interchange_bp.py::_enqueue_interchange_job

  • line 58 queue default stage untracked enqueue run_interchange_migration depends_on (none)

wepppy/weppcloud/routes/nodb_api/path_ce_bp.py::run_path_cost_effective

  • line 276 queue default stage untracked enqueue run_path_cost_effective_rq depends_on (none)

wepppy/weppcloud/routes/nodb_api/project_bp.py::delete_run

  • line 376 queue default stage untracked enqueue delete_run_rq depends_on (none)

wepppy/weppcloud/routes/nodb_api/project_bp.py::task_set_readonly

  • line 748 queue default stage untracked enqueue set_run_readonly_rq depends_on (none)

wepppy/weppcloud/routes/nodb_api/rangeland_bp.py::task_build_rangeland_cover

  • line 254 queue default stage untracked enqueue build_rangeland_cover_rq depends_on (none)

wepppy/weppcloud/routes/nodb_api/roads_bp.py::_enqueue_roads_job

  • line 470 queue default stage untracked enqueue func depends_on (none)

wepppy/weppcloud/routes/weppcloudr.py::_enqueue_deval_job

  • line 642 queue default stage untracked enqueue render_deval_details_rq depends_on (none)

Single-Storm Deprecation

Valid saved individual single-storm inputs can run with a deprecation advisory. The event pipeline skips continuous-only stages, including annual hillslope loss, water balance, return periods, and DSS export. Saved storm batches require per-event postprocessing outside the active RQ pipeline. See the WEPP input contract.

Wait Strategy And Inventory

  • Prefer explicit depends_on edges. File waits are a last-resort guard for filesystem propagation, not a substitute for missing dependencies.
  • Waits must be time-bounded and run only inside background RQ workers or offline tooling; never add waits to UI routes or sync handlers where a run might not exist yet.
  • Use wepppy.io_wait.wait_for_path / wait_for_paths when possible for stable-size checks. The interchange helper wepppy.wepp.interchange._utils._wait_for_path now delegates to io_wait with stable-size checks.
  • When outputs might not exist because a run never executed, fail fast rather than waiting.

Wait Inventory (File Guards)

Area Function Guard Target(s) Timeout / poll Notes
wepppy/rq/wepp_rq.py _post_watershed_interchange_rq wait_for_path via _wait_for_output wepp/output/pass_pw0.txt (or .gz), ebe_pw0.txt, loss_pw0.txt, chan.out, chanwb.out, chnwb.txt, soil_pw0.txt (or .gz) timeout_s=60, poll_s=0.5 RQ worker only; stable-size check before watershed interchange. Legacy single-storm conditionals skip some waits; single-storm is deprecated.
wepppy/rq/wepp_rq.py _run_hillslope_watbal_rq wait_for_path wepp/output/interchange/H.wat.parquet timeout_s=60 (poll via PERIDOT_INPUT_POLL_S) RQ worker only; stable-size check.
wepppy/rq/wepp_rq.py _build_totalwatsed3_rq wait_for_paths wepp/output/interchange/H.pass.parquet, wepp/output/interchange/H.wat.parquet timeout_s=60 (poll via PERIDOT_INPUT_POLL_S) RQ worker only; stable-size check before totalwatsed3 aggregation.
wepppy/rq/wepp_rq.py _analyze_return_periods_rq wait_for_paths wepp/output/interchange/ebe_pw0.parquet (fallback wepp/output/ebe_pw0.parquet), wepp/output/interchange/totalwatsed3.parquet timeout_s=60 RQ worker only; stable-size check before return-period export.
wepppy/rq/wepp_rq.py post_dss_export_rq wait_for_path / wait_for_paths export/dss/dss_channels.geojson (if written), export/dss/totalwatsed3_chan_*.dss (if written), export/dss/peak_chan_*.dss (if written), export/dss/README.dss_export.md (if written), export/dss.zip timeout_s=60 RQ worker only; stable-size checks between DSS export stages.
wepppy/rq/project_rq.py build_subcatchments_rq wait_for_path watershed.subwta (SUBWTA.ARC or subwta.tif) defaults (PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S) Guard before publishing completion trigger.
wepppy/rq/project_rq.py abstract_watershed_rq wait_for_path watershed.subwta (SUBWTA.ARC or subwta.tif) defaults (PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S) Guard before abstraction.
wepppy/rq/project_rq.py fork_rq (undisturbify=True) wait_for_paths ron.nodb, wepp.nodb, landuse.nodb, soils.nodb, disturbed.nodb timeout_s=60 (poll via PERIDOT_INPUT_POLL_S) Ensures forked NoDb files are readable before undisturbify.
wepppy/topo/peridot/peridot_runner.py run_peridot_abstract_watershed _wait_for_file dem/topaz/SUBWTA.ARC PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S Defensive guard before Peridot.
wepppy/topo/peridot/peridot_runner.py run_peridot_wbt_abstract_watershed _wait_for_file dem/wbt/subwta.tif PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S Defensive guard before Peridot (WBT).
wepppy/wepp/interchange/watershed_ebe_interchange.py run_wepp_watershed_ebe_interchange _utils._wait_for_path wepp/output/ebe_pw0.txt timeout=60, poll=0.5 Stable-size wait inside interchange.
wepppy/wepp/interchange/watershed_chanwb_interchange.py run_wepp_watershed_chanwb_interchange _utils._wait_for_path wepp/output/chanwb.out timeout=60, poll=0.5 Stable-size wait inside interchange.
wepppy/wepp/interchange/watershed_chan_peak_interchange.py run_wepp_watershed_chan_peak_interchange _utils._wait_for_path wepp/output/chan.out timeout=60, poll=0.5 Stable-size wait inside interchange.
wepppy/wepp/interchange/watershed_chnwb_interchange.py run_wepp_watershed_chnwb_interchange _utils._wait_for_path wepp/output/chnwb.txt timeout=60, poll=0.5 Stable-size wait inside interchange.
wepppy/wepp/reports/return_periods.py refresh_return_period_events wait_for_path interchange/ebe_pw0.parquet (or wepp/output/ebe_pw0.parquet), interchange/totalwatsed3.parquet defaults Report builder waits for parquet assets unless wait_for_inputs=False (UI routes).

Catalog

Run catalog maintenance

The opt-in scheduler entry calls run_catalog_rq.enqueue_sweep on the existing batch queue. Admission requires a live idle compatible consumer and no global RQ suspension, using passive consistent registration/TTL observations. No qualifying worker means no job or admission-pointer mutation; dirty SQL state waits for the next tick. Observation does not reserve the worker. Redis admission still coalesces queued/started/deferred/scheduled sweeps; a PostgreSQL transaction advisory lock serializes scheduled and CLI scans. There are no child jobs or per-save enqueues. The generated scanner's generic queue label does not override this scheduler-selected queue. IDs retain the exact run_catalog_sweep_<UUID> form and public polling never exposes task arguments, results or private exception metadata. All eligible consumers require the canonical run-catalog compatibility/origin/mount proof before admission. Real isolated-queue validation executes the callable and checks its terminal job tree plus source-derived SQL values; live host-tree evidence remains a staged deployment gate.

AgFields rq-engine enqueue surface

The helper wepppy.microservices.rq_engine.ag_fields_routes._enqueue_job is represented as one dynamic func edge in the generated graph. Its three concrete call sites have no RQ depends_on edges; prerequisite checks happen synchronously before enqueue. A single-scheme _enqueue_watershed_jobs call still enqueues the scheme worker directly. Run All instead enqueues run_ag_fields_watershed_suite_rq as one top-level parent. The route atomically stores all planned children on that parent, and dispatch shares a lock with cancellation. Concept 1, Concept 2, and hybrid remain ordered jobs:* children; each scheme after the first depends on its predecessor through Dependency(..., allow_failure=True). A registered finalizer depends on all three scheme IDs through Dependency(..., allow_failure=True). Every failure-tolerant dependent applies the Batch Runner already-terminal release guard, so a predecessor that fails before registration cannot strand the rest of the suite. This prevents overlapping watershed memory peaks while permitting complete comparison attempts and one deterministic terminal suite signal from the finalizer.

Job key RQ entrypoint Preconditions / file inputs
agfields_build_subfields wepppy.rq.ag_fields_rq.build_ag_fields_subfields_rq Confirmed schema; dem/wbt/flovec.tif; canonical boundary GeoJSON; runs rasterize → abstract → polygonize inside one job
agfields_plantdb wepppy.rq.ag_fields_rq.process_ag_fields_plant_db_rq Server-validated, uniquely named ZIP under ag_fields/; staged archive is removed after terminal success/failure
agfields_run_wepp wepppy.rq.ag_fields_rq.run_ag_fields_wepp_rq Current sub-fields; complete valid mapping; parent wepp/runs/p<wepp_id>.sol and .cli pairs; after WEPP, synchronously converts all six H<sub_field_id> report families into the specialized six-file bundle and stamps completion only after publication
agfields_run_watershed_suite wepppy.rq.ag_fields_rq.run_ag_fields_watershed_suite_rq Run All only; parent dispatches three serial scheme children and the always-released finalizer; response job_id names this parent
agfields_run_watershed_concept_1 wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq Current successful Stage 4; observed climate; complete parent prepared inputs; writes only wepp/ag_fields/watershed/concept-1/
agfields_run_watershed_concept_2 wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq Same preconditions; writes only wepp/ag_fields/watershed/concept-2/; the historical agfields_run_watershed key remains a Concept 2 job-id alias
agfields_run_watershed_hybrid wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq Same preconditions plus the explicit Peridot connectivity resources; writes only wepp/ag_fields/watershed/hybrid/
suite finalizer wepppy.rq.ag_fields_rq.finalize_ag_fields_watershed_suite_rq Depends on all three scheme jobs with allow_failure=True; publishes terminal suite status after every scheme is terminal

wepppy/rq/wepp_rq.py

run_wepp_rq(runid)

Graph (stage order is the jobs:<n> prefix):

jobs:0 prep inputs
  -> _prep_remaining_rq
  -> _run_hillslopes_rq
  -> (_prep_watershed_rq if run_watershed)
  -> _build_hillslope_interchange_rq (after SWAT build when delete_after_interchange)
  -> _build_totalwatsed3_rq
  -> (post_dss_export_rq if enabled and run_watershed; after `_post_run_cleanup_out_rq`)
  -> (SWAT build/run if enabled; before interchange when delete_after_interchange)
  -> watershed runs (if run_watershed)
  -> post-run jobs (cleanup, prep details, watbal, loss grid, watershed interchange, return periods)
  -> export jobs (legacy arc, gpkg)
  -> _log_complete_rq

Child jobs, dependencies, and file prerequisites:

Note: Parquet prerequisites are canonical directory paths (for example watershed/hillslopes.parquet). WD-root sidecars (for example watershed.hillslopes.parquet) are retired and treated as migration-required boundaries.

Stage Child job depends_on File prerequisites (relative to wd)
jobs:0 _prep_multi_ofe_rq none wat_dir/slope_files/hillslopes/hill_<topaz>.mofe.slp; soils_dir/hill_<topaz>.mofe.sol; lc_dir/*.man; optional wepp.kslast_map raster
jobs:0 _prep_slopes_rq none wat_dir/hill_<topaz>.slp or wat_dir/slope_files/hillslopes/hill_<topaz>.slp
jobs:0 _prep_managements_rq none landuse/landuse.parquet; lc_dir/*.man; soils/soils.parquet; optional disturbed.nodb + disturbed maps; optional rap/rap_ts.parquet
jobs:0 _prep_soils_rq none soils_dir/*.sol; optional wepp.kslast_map raster; soils/soils.parquet for soil metadata
jobs:0 _prep_climates_rq none cli_dir/<cli_fn>; for SS batch writes runs_dir/p<wepp_id>.<ss_batch_id>.cli and runs_dir/pw0.<ss_batch_id>.cli
jobs:0 _prep_remaining_rq jobs0_hillslopes_prep list runs_dir/p*.slp; runs_dir/p*.sol; runs_dir/p*.man; runs_dir/p*.cli
jobs:1 _run_hillslopes_rq _prep_remaining_rq runs_dir/*.run; runs_dir/p*.slp; runs_dir/p*.sol; runs_dir/p*.man; runs_dir/p*.cli
jobs:2 _prep_watershed_rq _run_hillslopes_rq wat_dir/structure.json or wat_dir/network.txt; watershed/channels.parquet; watershed/hillslopes.parquet; wat_dir/slope_files/channels.slp or wat_dir/channels.slp; soils/soils.parquet; cli_dir/<cli_fn>
jobs:2 _build_hillslope_interchange_rq _run_hillslopes_rq (+ _build_swat_inputs_rq if SWAT enabled + delete_after_interchange) output_dir/H*.pass.dat; output_dir/H*.ebe.dat; output_dir/H*.element.dat; output_dir/H*.loss.dat; output_dir/H*.soil.dat; output_dir/H*.wat.dat
jobs:2 _build_totalwatsed3_rq _build_hillslope_interchange_rq output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash/H<wepp_id>_ash.parquet or ash/H<wepp_id>.parquet; optional ash.nodb; watershed/hillslopes.parquet (area lookup when ash is present)
jobs:2 post_dss_export_rq _build_hillslope_interchange_rq (+ _post_run_cleanup_out_rq if run_watershed) output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash/*.parquet; output_dir/chan.out; watershed.channels_shp (GeoJSON); wat_dir/network.txt (downstream mapping); export/dss dir (created)
jobs:2 _build_swat_inputs_rq _build_hillslope_interchange_rq (+ _prep_watershed_rq if present); if delete_after_interchange then _run_hillslopes_rq (+ _prep_watershed_rq if present) wepp/output/* (hillslope outputs); watershed/hillslopes.parquet; watershed/channels.parquet; swat_template_dir/* (config swat_template_dir)
jobs:3 _run_swat_rq _build_swat_inputs_rq swat_txtinout_dir/*
jobs:3 run_watershed_rq / run_ss_batch_watershed_rq _prep_watershed_rq runs_dir/pw0.str; runs_dir/pw0.slp; runs_dir/pw0.chn; runs_dir/pw0.sol; runs_dir/pw0.cli; runs_dir/pw0.man; runs_dir/chan.inp; for SS batch runs_dir/pw0.<ss_batch_id>.cli
jobs:4 _post_run_cleanup_out_rq jobs3_watersheds or _prep_watershed_rq runs_dir/*.out; optional runs_dir/tc_out.txt
jobs:4 _post_prep_details_rq post_dependencies watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet
jobs:4 _run_hillslope_watbal_rq post_dependencies + _build_hillslope_interchange_rq output_dir/interchange/H.wat.parquet
jobs:4 _post_make_loss_grid_rq post_dependencies watershed.subwta (wat_dir/SUBWTA.ARC or wbt_wd/subwta.tif); watershed.discha (wat_dir/DISCHA.ARC or wbt_wd/discha.tif); output_dir/H*
jobs:4 _post_watershed_interchange_rq _post_run_cleanup_out_rq output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt
jobs:4 _analyze_return_periods_rq _post_watershed_interchange_rq + _build_totalwatsed3_rq output_dir/interchange/ebe_pw0.parquet (fallback output_dir/ebe_pw0.parquet); output_dir/interchange/totalwatsed3.parquet; optional cached output_dir/return_periods*.json
jobs:5 _post_legacy_arc_export_rq jobs4_post topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata
jobs:5 _post_gpkg_export_rq jobs4_post watershed.subwta_shp (SUBCATCHMENTS.WGS JSON); watershed.channels_shp (CHANNELS.WGS JSON); watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet; output_dir/interchange/loss_pw0.hill.parquet; output_dir/interchange/loss_pw0.chn.parquet
jobs:6 _log_complete_rq jobs4_post + jobs5_post ron.nodb

Audit notes:

  1. _run_hillslope_watbal_rq depends on _build_hillslope_interchange_rq and still waits for output_dir/interchange/H.wat.parquet internally.
  2. post_dss_export_rq only enqueues when run_watershed is true (guarded in run_wepp_rq).
  3. post_dss_export_rq calls totalwatsed_partitioned_dss_export, which runs run_totalwatsed3 per channel; it does not require _build_totalwatsed3_rq unless the export is refactored to reuse a prebuilt totalwatsed3.parquet.
  4. _build_totalwatsed3_rq depends on hillslope interchange. If watershed outputs are required, depend on jobs3_watersheds or _post_watershed_interchange_rq.
  5. When delete_after_interchange is enabled, SWAT inputs are built before interchange and interchange waits on SWAT to avoid deleting raw wepp/output files.
  6. _run_hillslope_watbal_rq and post_dss_export_rq target continuous runs; legacy single-storm conditionals may skip them, but single-storm is deprecated.
  7. _analyze_return_periods_rq depends on totalwatsed3; legacy single-storm runs skip this pipeline and are deprecated.

run_wepp_noprep_rq(runid)

Graph (prep stages skipped; uses existing inputs under wepp/runs/):

jobs:1 _run_hillslopes_rq
  -> _build_hillslope_interchange_rq
  -> _build_totalwatsed3_rq
  -> watershed runs (if run_watershed)
  -> post-run jobs (cleanup, prep details, watbal, loss grid, watershed interchange, return periods)
  -> exports (legacy arc, gpkg)
  -> (post_dss_export_rq if enabled and run_watershed)
  -> _log_complete_rq
Stage Child job depends_on File prerequisites (relative to wd)
jobs:1 _run_hillslopes_rq none runs_dir/*.run; runs_dir/p*.slp; runs_dir/p*.sol; runs_dir/p*.man; runs_dir/p*.cli
jobs:2 _build_hillslope_interchange_rq _run_hillslopes_rq output_dir/H*.pass.dat; output_dir/H*.ebe.dat; output_dir/H*.element.dat; output_dir/H*.loss.dat; output_dir/H*.soil.dat; output_dir/H*.wat.dat
jobs:2 _build_totalwatsed3_rq _build_hillslope_interchange_rq output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash parquet; watershed/hillslopes.parquet
jobs:3 run_watershed_rq / run_ss_batch_watershed_rq _run_hillslopes_rq runs_dir/pw0.* as above; SS batch runs_dir/pw0.<ss_batch_id>.cli
jobs:4 _post_run_cleanup_out_rq jobs3_watersheds or _run_hillslopes_rq runs_dir/*.out; optional runs_dir/tc_out.txt
jobs:4 _post_prep_details_rq post_dependencies watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet
jobs:4 _run_hillslope_watbal_rq post_dependencies + _build_hillslope_interchange_rq output_dir/interchange/H.wat.parquet
jobs:4 _post_make_loss_grid_rq post_dependencies watershed.subwta; watershed.discha; output_dir/H*
jobs:4 _post_watershed_interchange_rq _post_run_cleanup_out_rq output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt
jobs:4 _analyze_return_periods_rq _post_watershed_interchange_rq + _build_totalwatsed3_rq output_dir/interchange/ebe_pw0.parquet; output_dir/interchange/totalwatsed3.parquet
jobs:2 post_dss_export_rq _build_hillslope_interchange_rq (+ _post_run_cleanup_out_rq if run_watershed) output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash parquet; output_dir/chan.out; watershed.channels_shp; wat_dir/network.txt; export/dss dir
jobs:5 _post_legacy_arc_export_rq jobs4_post topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata
jobs:5 _post_gpkg_export_rq jobs4_post watershed.subwta_shp; watershed.channels_shp; watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet; output_dir/interchange/loss_pw0.hill.parquet; output_dir/interchange/loss_pw0.chn.parquet
jobs:6 _log_complete_rq jobs4_post + jobs5_post ron.nodb

run_wepp_watershed_rq(runid)

Stage Child job depends_on File prerequisites (relative to wd)
jobs:2 _prep_watershed_rq none wat_dir/structure.json or wat_dir/network.txt; watershed/channels.parquet; watershed/hillslopes.parquet; wat_dir/slope_files/channels.slp or wat_dir/channels.slp; soils/soils.parquet; cli_dir/<cli_fn>
jobs:3 run_watershed_rq / run_ss_batch_watershed_rq _prep_watershed_rq runs_dir/pw0.* as above; SS batch runs_dir/pw0.<ss_batch_id>.cli
jobs:4 _post_run_cleanup_out_rq jobs3_watersheds or _prep_watershed_rq runs_dir/*.out; optional runs_dir/tc_out.txt
jobs:4 _post_prep_details_rq post_dependencies watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet
jobs:4 _post_make_loss_grid_rq post_dependencies watershed.subwta; watershed.discha; output_dir/H* (job skipped when hillslope outputs are missing)
jobs:4 _post_watershed_interchange_rq _post_run_cleanup_out_rq output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt
jobs:5 _post_legacy_arc_export_rq jobs4_post topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata
jobs:6 _log_complete_rq jobs4_post + jobs5_post ron.nodb

Audit notes:

  1. This entrypoint bypasses hillslope prep/run; ensure downstream jobs only require watershed outputs.
  2. Return-period exports are not enqueued here because totalwatsed3 is not scheduled.

run_wepp_watershed_noprep_rq(runid)

Stage Child job depends_on File prerequisites (relative to wd)
jobs:3 run_watershed_rq / run_ss_batch_watershed_rq none runs_dir/pw0.* as above; SS batch runs_dir/pw0.<ss_batch_id>.cli
jobs:4 _post_run_cleanup_out_rq jobs3_watersheds runs_dir/*.out; optional runs_dir/tc_out.txt
jobs:4 _post_prep_details_rq post_dependencies watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet
jobs:4 _post_make_loss_grid_rq post_dependencies watershed.subwta; watershed.discha; output_dir/H* (job skipped when hillslope outputs are missing)
jobs:4 _post_watershed_interchange_rq _post_run_cleanup_out_rq output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt
jobs:5 _post_legacy_arc_export_rq jobs4_post topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata
jobs:6 _log_complete_rq jobs4_post + jobs5_post ron.nodb

wepppy/rq/swat_rq.py

run_swat_rq(runid)

Stage Child job depends_on File prerequisites (relative to wd)
jobs:0 _build_swat_inputs_rq none wepp/output/* (hillslope outputs); watershed/hillslopes.parquet; watershed/channels.parquet; swat_template_dir/*
jobs:1 _run_swat_rq _build_swat_inputs_rq swat_txtinout_dir/*

Audit notes:

  1. This assumes WEPP outputs already exist. The caller is responsible for enqueueing only after WEPP hillslope/watershed outputs are ready.

run_swat_noprep_rq(runid)

Stage Child job depends_on File prerequisites (relative to wd)
jobs:0 _run_swat_rq none swat_txtinout_dir/*

wepppy/rq/batch_rq.py

run_batch_rq(batch_name)

Stage Child job depends_on File prerequisites
jobs:0,hillslopes run_batch_hillslopes_rq (per watershed feature) root dispatch, strict Batch workspace, validated watershed feature and base project
jobs:0 run_batch_watershed_rq (per watershed feature) corresponding hillslope job, failure-tolerant terminal observer Finished upstream and matching visible batch_handoff/<hillslope-job-id>.json required before science
jobs:1 _final_batch_complete_rq all jobs:0 via failure-tolerant RQ Dependency (+ Omni final jobs when run_omni_scenarios_rq enqueues work) batch_runner.nodb (for status updates)

Audit notes:

  1. run_batch_watershed_rq verifies the attempt-bound handoff, freshly hydrates leaf state, and owns terminal metadata and the completion trigger. See docs/schemas/batch-task-boundary-contract.md.
  2. It calls run_omni_scenarios_rq(runid) inline. Omni finalizer attachment is serialized under a finalizer-specific Redis lock and uses root RQ metadata, so parallel leaves cannot lose each other's dependencies.
  3. The completion job allows failed dependencies so it still runs after leaves reach terminal failed status and can publish the failed-run summary instead of remaining deferred.

wepppy/rq/culvert_rq.py

run_culvert_batch_rq(culvert_batch_uuid)

Stage Child job depends_on File prerequisites
jobs:0 run_culvert_run_rq (per run_id) none batch_root/metadata.json; batch_root/model-parameters.json; batch_root/topo/breached_filled_DEM_UTM.tif (or payload override); batch_root/topo/streams.tif (or payload override); batch_root/culverts/watersheds.geojson; batch_root/topo/flovec.tif; batch_root/topo/netful.tif; batch_root/topo/chnjnt.tif; batch_root/topo/chnjnt.streams.tif
jobs:1 _final_culvert_batch_complete_rq all jobs:0 batch_root/culverts/*.json run metadata (written by child jobs)

Audit notes:

  1. Batch-level topo generation happens before enqueueing children. If that becomes async, ensure child jobs depend on it.

run_culvert_run_rq(runid, culvert_batch_uuid, run_id)

Stage Child job depends_on File prerequisites
(inline) _process_culvert_run (internal) none batch_root/metadata.json; batch_root/model-parameters.json; batch_root/culverts/watersheds.geojson; batch_root/culverts/culvert_points.geojson; batch_root/topo/breached_filled_DEM_UTM.tif; batch_root/topo/flovec.tif; batch_root/topo/netful.tif; batch_root/topo/chnjnt.tif; per-run directory batch_root/runs/<run_id>

wepppy/rq/omni_rq.py

run_omni_scenarios_rq(runid)

Stage Child job depends_on File prerequisites (relative to wd)
jobs:0 run_omni_scenario_rq (stage 1 scenarios) none Base scenario inputs under wepp/output/*; dependency hash uses wepp/output/loss_pw0.txt
jobs:1 run_omni_scenario_rq (stage 2 scenarios) all stage 1 jobs, strict Stage 1 scenario outputs under omni/scenarios/<scenario>/wepp/output/loss_pw0.txt
jobs:2 _compile_hillslope_summaries_rq stage 2 jobs (or stage 1 if no stage 2), strict wepp/output/loss_pw0.txt; omni/scenarios/<scenario>/wepp/output/loss_pw0.txt
jobs:3 _finalize_omni_scenarios_rq _compile_hillslope_summaries_rq, strict omni.nodb

Audit notes:

  1. Stage 2 depends on the full set of stage 1 jobs, not just the specific scenario’s dependency. This is conservative but can increase critical path length.
  2. Every scenario edge consumes required upstream outputs and is strict. A failure stops all transitive scenario, compile, and finalization work; the controller's ordinary retry replaces the blocked deferred graph.

run_omni_contrasts_rq(runid)

Stage Child job depends_on File prerequisites (relative to wd)
jobs:contrast:* run_omni_contrast_rq previous batch (if any), failure tolerant omni/contrasts/contrast_<id>.tsv; scenario outputs referenced by sidecar paths under omni/contrasts/<id>/wepp/output/*
jobs:finalize _finalize_omni_contrasts_rq last batch, failure tolerant omni.nodb

Audit notes:

  1. Batches are chained (batch_depends), so the final dependency on the last batch implies all prior batches.
  2. Because the batches are chained, a bare depends_on list makes one failed contrast strand every later batch and the finalizer. Both edges use _failure_tolerant_depends_on plus _release_deferred_job_if_ready so _finalize_omni_contrasts_rq always reaches a terminal state and emits END_BROADCAST.

wepppy/rq/project_rq.py

fetch_dem_and_build_channels_rq(...)

Stage Child job depends_on File prerequisites
jobs:0 fetch_dem_rq none ron.nodb; writes dem_dir/dem.tif or dem_dir/dem.vrt
jobs:1 build_channels_rq fetch_dem_rq dem_dir/dem.tif or dem_dir/dem.vrt
jobs:0 build_channels_rq none (upload DEM mode) dem_dir/dem.tif or dem_dir/dem.vrt already present

Audit notes:

  1. No file-based waits here; ensure fetch_dem_rq fully writes dem_dir/dem.tif before returning.

build_subcatchments_and_abstract_watershed_rq(runid, updates)

Stage Child job depends_on File prerequisites
jobs:0 build_subcatchments_rq prior same-run request tail, when present (failure-tolerant serialization) dem_dir/dem.tif or dem_dir/dem.vrt; channel outputs under wat_dir/* from build_channels_rq
jobs:1 abstract_watershed_rq (nonmutating completion receipt) build_subcatchments_rq (strict) none; build and abstraction already completed inside jobs:0

Audit notes:

  1. build_subcatchments_rq uses wait_for_path on watershed.subwta, then performs abstraction under the same watershed directory-root lock before publishing completion.
  2. One optimistic Redis transaction registers both jobs, both dependency directions, both root metadata links, the per-run tail, and jobs:0 queue or deferred membership. The prior-tail edge serializes independent request mutation ownership and may cross root lineage; retry cleanup treats it as an external dependency boundary and admission splices a canceled tail back to its still-active predecessor. A failed jobs:0 cancels only its matching receipt for controlled policy failures; other failures leave the strict receipt deferred.

fork_rq(..., undisturbify=True)

Stage Child job depends_on File prerequisites
(inline) run_wepp_rq(new_runid) none Forked run directory with .nodb files (ron.nodb, wepp.nodb, landuse.nodb, soils.nodb, disturbed.nodb)
(inline) _finish_fork_rq final job from run_wepp_rq relies on run_wepp_rq completion metadata

Audit notes:

  1. _finish_fork_rq depends on the final _log_complete_rq job produced by run_wepp_rq, so it should run after all WEPP tasks finish.
  2. Every WEPP pipeline child created for an undisturbify fork carries server-generated fork_failure lineage and report_fork_failure as its RQ failure callback. Root child registration is saved before enqueue. The callback atomically verifies the current destination receipt and records failure plus source status; strict dependency edges are unchanged.
  3. WepppyRqWorker.handle_job_failure repeats guarded reporting after abrupt work-horse death, when RQ skips the job callback. Duplicate/stale reporting is suppressed. Neither path releases claims, cancels siblings, nor unblocks model stages.
  4. Initial Wepp/Watershed controller reads in preparation jobs have a shared five-second ENOENT/ESTALE retry budget. This does not retry the job or any mutation. See ADR-0049.

wepppy/microservices/rq_engine/run_sync_routes.py

POST /rq-engine/run-sync

Stage Child job depends_on File prerequisites
root run_sync_rq none Target root writable (default /wc1/runs); remote source reachable; optional source_run_token with source-host scope runs:read for private runs
root migrations_rq run_sync_rq target_root/<prefix>/<runid> run directory with .nodb files and run outputs

Audit notes:

  1. migrations_rq depends on the sync job, ensuring that run files exist before migrations execute.
  2. run_sync_rq writes RunMigration rows by importing wepppy.weppcloud.app; workers executing it need SECRET_KEY(_FILE), SECURITY_PASSWORD_SALT(_FILE), and POSTGRES_PASSWORD(_FILE) configured.
  3. When source_run_token is supplied, run_sync_rq uses Authorization: Bearer <token> for aria2c.spec and aria2 fetch operations; token is not written to provenance records.

wepppy/microservices/rq_engine/batch_routes.py

POST /api/batch/_/{batch_name}/run-batch

Stage Child job depends_on File prerequisites
root run_batch_rq none Batch workspace batch_root/<batch_name>; batch_runner.nodb; uploaded watershed GeoJSON + validated template

POST /api/batch/_/{batch_name}/delete-batch

Stage Child job depends_on File prerequisites
root delete_batch_rq none Batch workspace batch_root/<batch_name> (including _base and runs/*)

Audit notes:

  1. delete_batch_rq is destructive and removes the batch workspace recursively after clearing batch run lock/cache metadata.

wepppy/microservices/rq_engine/bootstrap_routes.py

POST /api/runs/{runid}/{config}/bootstrap/enable

Stage Child job depends_on File prerequisites
root bootstrap_enable_rq none wd (run directory from get_wd(runid)); writes .git/ and commits tracked bootstrap inputs under wepp/runs and/or swat/TxtInOut

Audit notes:

  1. Route-level dedupe key bootstrap:enable:job:<runid> prevents duplicate enable jobs while one is active.
  2. Route acquires run-scoped Redis git lock bootstrap:git-lock:<runid> before enqueueing; the worker releases it in bootstrap_enable_rq cleanup.

POST /api/runs/{runid}/{config}/run-wepp-npprep

Stage Child job depends_on File prerequisites
root run_wepp_noprep_rq none Existing WEPP input state under wepp/runs; Bootstrap must already be enabled for the run

Audit notes:

  1. Route resets Redis prep timestamps for WEPP tasks before enqueueing the no-prep run.
  2. This endpoint intentionally skips prep rebuilds and runs against the currently checked-out git state.

POST /api/runs/{runid}/{config}/run-wepp-watershed-no-prep

Stage Child job depends_on File prerequisites
root run_wepp_watershed_noprep_rq none Existing watershed inputs under wepp/runs; Bootstrap must already be enabled for the run

Audit notes:

  1. Route resets the watershed prep timestamp before enqueueing.
  2. No branch checkout occurs here; execution uses current repository checkout.

POST /api/runs/{runid}/{config}/run-swat-noprep

Stage Child job depends_on File prerequisites
root run_swat_noprep_rq none Existing SWAT+ inputs under swat/TxtInOut; Bootstrap must already be enabled and SWAT mod present

Audit notes:

  1. Route enqueues SWAT no-prep execution without rebuilding TxtInOut.
  2. Endpoint depends on existing SWAT inputs and the active checkout state.

NFS Consistency Headaches

Background (Feb 6, 2026)

Production WEPP runs experienced rare, intermittent failures (on the order of 1 in 10⁶ to 10⁹ events) where RQ worker jobs could not read files that upstream jobs had already written. Symptoms included FileNotFoundError on .parquet interchange files, wait_for_path timeouts on watershed outputs (pass_pw0.txt, ebe_pw0.txt, etc.), and stale reads returning truncated or zero-length files. These failures were non-deterministic and consistent with NFS client cache staleness, though the root cause has not been definitively confirmed.

Notably, the same soft mount configuration works reliably on the dev and test-production NFS servers, which use ZFS as the backing filesystem. The failures have only been observed on the production NAS, suggesting the issue may involve an interaction between the soft mount options and the production storage backend rather than the mount options alone.

Suspected Cause: NFS Mount Options

The /geodata mount was configured with soft mode and large read/write buffers with aggressive attribute caching. The following options are suspected contributors to the degraded behavior:

# BEFORE (degraded)
nas.rocket.net:/wepp  /geodata  nfs4  rw,noatime,proto=tcp,vers=4.2,soft,rsize=32768,wsize=32768,timeo=600,_netdev  0  0
Option Potential problem
soft NFS operations return errors after timeo retries instead of blocking. Under load, transient server delays could cause I/O errors that surface as missing or corrupt files in downstream RQ jobs.
rsize=32768,wsize=32768 Fixed 32 KB transfer sizes bypass the kernel's auto-negotiation, potentially limiting throughput on large sequential writes (interchange parquet files) and adding round-trips for small metadata operations.
timeo=600 60-second timeout (in deciseconds) combined with soft means jobs may receive EIO after a single retry window rather than waiting for the server to recover.
(no actimeo) Default attribute cache lifetime (~3–60 s) can allow NFS clients to serve stale stat() results. A file written by one worker could appear absent or zero-length to another worker for up to 60 seconds.

Remediation: Updated Mount

# AFTER (remediated)
nas.rocket.net:/wepp  /geodata  nfs4  rw,noatime,proto=tcp,vers=4.2,hard,intr,actimeo=0,_netdev  0  0
Option Effect
hard NFS operations block indefinitely until the server responds, preventing spurious EIO errors during transient load spikes. Workers wait rather than fail.
intr Allows blocked hard-mount operations to be interrupted by signals, so jobs can still be canceled or timed out by RQ without requiring a full process kill.
actimeo=0 Disables attribute caching entirely. Every stat(), open(), and directory listing goes to the server, ensuring that when an upstream job writes a file, downstream jobs should see it immediately. This targets the suspected file-guard race conditions.
(removed rsize/wsize) Lets the NFS client and server auto-negotiate optimal transfer sizes (typically 1 MB on NFSv4.2), improving throughput for large parquet writes and reducing round-trips.
(removed timeo) With hard mount, timeout is irrelevant; operations retry until the server responds.

Impact on RQ Job Dependencies and File Guards

The catalog documents two layers of ordering between RQ jobs: explicit depends_on edges (job-level) and wait_for_path / wait_for_paths file guards (filesystem-level). If the NFS attribute cache hypothesis is correct, it would affect the second layer directly and the first layer indirectly.

Explicit depends_on edges (job-level ordering) are not themselves affected by NFS caching. RQ tracks job completion in Redis, so a depends_on edge fires when the upstream job's Python function returns, not when the filesystem flushes. However, a job returning successfully does not guarantee that its output files are visible to other NFS clients—this is the gap the file guards fill. The edge graph in this catalog was verified through several auditing and review passes during this investigation.

File guards (wait_for_path / wait_for_paths with stable-size checks) are the suspected primary failure point. Under a stale attribute cache scenario:

  1. stat() calls inside wait_for_path could return ENOENT even after the file existed on the server.
  2. When stat() did succeed, it could report a stale size (often 0), causing the stable-size check to loop until timeout.
  3. Under a soft mount, the timeout could produce an EIO instead of a retry, crashing the worker.

The following call chains were traced end-to-end from rq-engine API entry through RQ job execution to file-guard resolution:

Trace: POST /rq-engine/run-sync → Full WEPP Run

rq-engine API: POST /rq-engine/run-sync
  └─ run_sync_rq                          [depends_on: none]
       └─ migrations_rq                   [depends_on: run_sync_rq]
            └─ run_wepp_rq(runid)         [depends_on: none]
                 ├─ jobs:0  _prep_*_rq                        [depends_on: none]
                 ├─ jobs:0  _prep_remaining_rq                [depends_on: jobs0 prep list]
                 ├─ jobs:1  _run_hillslopes_rq                [depends_on: _prep_remaining_rq]
                 ├─ jobs:2  _prep_watershed_rq                [depends_on: _run_hillslopes_rq]
                 ├─ jobs:2  _build_hillslope_interchange_rq   [depends_on: _run_hillslopes_rq]
                 │    └─ READS: output_dir/H*.pass.dat, H*.ebe.dat, H*.wat.dat, ...
                 │    └─ WRITES: output_dir/interchange/H.pass.parquet, H.wat.parquet, ...
                 ├─ jobs:2  _build_totalwatsed3_rq            [depends_on: _build_hillslope_interchange_rq]
                 │    └─ FILE GUARD: wait_for_paths → H.pass.parquet, H.wat.parquet  ← NFS-affected
                 │    └─ WRITES: output_dir/interchange/totalwatsed3.parquet
                 ├─ jobs:3  run_watershed_rq                  [depends_on: _prep_watershed_rq]
                 │    └─ WRITES: output_dir/pass_pw0.txt, ebe_pw0.txt, chan.out, ...
                 ├─ jobs:4  _post_run_cleanup_out_rq          [depends_on: jobs3_watersheds]
                 ├─ jobs:4  _post_watershed_interchange_rq    [depends_on: _post_run_cleanup_out_rq]
                 │    └─ FILE GUARD: wait_for_path → pass_pw0.txt, ebe_pw0.txt,
                 │       chan.out, chanwb.out, chnwb.txt, soil_pw0.txt  ← NFS-affected
                 │    └─ WRITES: output_dir/interchange/ebe_pw0.parquet, ...
                 ├─ jobs:4  _run_hillslope_watbal_rq          [depends_on: post_deps + interchange]
                 │    └─ FILE GUARD: wait_for_path → H.wat.parquet  ← NFS-affected
                 ├─ jobs:4  _analyze_return_periods_rq        [depends_on: _post_watershed_interchange + totalwatsed3]
                 │    └─ FILE GUARD: wait_for_paths → ebe_pw0.parquet,
                 │       totalwatsed3.parquet  ← NFS-affected
                 ├─ jobs:5  _post_gpkg_export_rq              [depends_on: jobs4_post]
                 └─ jobs:6  _log_complete_rq                  [depends_on: jobs4+5]

Every ← NFS-affected guard above is a candidate site for stale-cache failures under the old mount. With actimeo=0, these guards should resolve on the first or second poll cycle (typically < 1 s) instead of timing out.

Trace: Subcatchment Build → Abstract Watershed

rq-engine API: POST /api/subcatchments/build
  └─ build_subcatchments_rq               [depends_on: prior same-run child, if any]
       ├─ FILE GUARD: wait_for_path → watershed.subwta  ← NFS-affected
       └─ abstract watershed under the same directory-root lock
  └─ abstract_watershed_rq               [nonmutating receipt; depends_on: build_subcatchments_rq]

Trace: Fork with Undisturbify

rq-engine API: POST /api/fork
  └─ fork_rq(undisturbify=True)
       └─ FILE GUARD: wait_for_paths → ron.nodb, wepp.nodb, landuse.nodb,
          soils.nodb, disturbed.nodb  ← NFS-affected
       └─ run_wepp_rq(new_runid)          [full chain as above]
       └─ _finish_fork_rq                 [depends_on: _log_complete_rq]

Summary

Before After
soft mount with fixed buffer sizes; stale attribute cache hard,intr mount with actimeo=0; auto-negotiated transfer sizes
File guards timed out or received EIO under load File guards should resolve immediately; operations block until server responds
Non-deterministic run failures correlated with NFS client count Expected deterministic file visibility across all NFS clients

The depends_on job graph documented in this catalog was reviewed through several auditing passes as part of this investigation—no missing edges were identified. However, the job dependency graph is not readily testable in isolation: edges are constructed dynamically at enqueue time, depend on runtime configuration (e.g. run_watershed, delete_after_interchange, SWAT/Omni enablement), and span multiple modules. Validation relies primarily on cognitive review of the code paths rather than automated verification. The interplay of RQ job orchestration, NoDb state, Rust-based interchange, and Fortran WEPP subprocesses makes constructing meaningful automated tests for the dependency graph near impossible, and the regression surface for dependency changes is correspondingly difficult to characterize.

The failures are consistent with the NFS attribute cache allowing downstream file guards to poll stale metadata, though other contributing factors have not been ruled out. The mount remediation targets this class of failure without requiring changes to job dependency wiring or wait timeouts. Continued monitoring is needed to confirm effectiveness.

Checklist For New Child Jobs

Notification inbox population

docker/scheduled-tasks.yml admits wepppy.rq.notifications_rq.populate_notifications on the existing batch queue only when WEPPCLOUD_NOTIFICATIONS_SWEEP_ENABLED=true. It has no child jobs or GC/access-compilation dependency. Its database sweep lock and episode uniqueness make retries safe. Trusted test calls may pass a bounded registration-ID scope; browser routes never enqueue this task. See docs/schemas/user-notifications-contract.md and docs/dev-notes/user-notifications.md.

Child-job checklist

  1. Add depends_on for any file or controller state produced by upstream jobs.
  2. Record job.meta["jobs:<order>,..."] for every child job so job_info.py and cancel_job.py can traverse the tree.
  3. If you rely on filesystem writes inside a single job, prefer explicit file waits over fixed sleeps.
  4. Update this catalog when adding new enqueue sites or dependency edges.