RQ Job Dependencies Catalog
Catalog of RQ entrypoints that enqueue child jobs and the dependency graph between them. Includes an auto-generated dependency graph section plus analyst-maintained narrative and audit notes.
Scope
- Modules:
wepppy/rq/*.py(entrypoints that enqueue other jobs). - rq-engine routes that enqueue RQ jobs directly:
wepppy/microservices/rq_engine/batch_routes.py,wepppy/microservices/rq_engine/bootstrap_routes.py,wepppy/microservices/rq_engine/geneva_routes.py, andwepppy/microservices/rq_engine/run_sync_routes.py. - rq-initiated Flask handlers that enqueue RQ jobs:
wepppy/weppcloud/routes/**/*.pyandwepppy/weppcloud/bootstrap/*.py.
Uploaded land-use builds
The normal build-landuse endpoint uses the existing default
project_rq.build_landuse_rq job family. An uploaded raster adds a
preallocation check that stages the raw and aligned inputs and a job-bound
receipt. This submission adds no edge or worker queue. The worker checks the
receipt and both directory leases before applying the upload. Previously
enqueued jobs without upload arguments continue to build their existing
aligned inputs. Setting batch inputs remains synchronous. Batch leaves
validate and align retained raw inputs under the existing directory leases
when their own watershed becomes available.
Declarative DEM slice
The private wepppy.declarative.adapter.submit enqueues execute_dem on one
declarative-dem-<submission UUID hex> queue. This adapter calls shared
project_rq.fetch_dem_rq synchronously and never waits for queued children.
After science settlement the operator invokes adapter.finalize, which enqueues
one finalize_dem job on the same queue; it calls shared set_run_readonly_rq.
No RQ dependency edge, scientific retry or ordinary queue consumer is added.
The engine waits outside the single worker. Durable claims prevent replay after
uncertain execution. These enqueue sites are outside the generator's existing
wepppy/rq/route scan; this narrative explicitly inventories them.
Query-engine asset catalog maintenance
wepppy.rq.query_catalog_rq.enqueue_catalog_refresh admits one
refresh_query_catalog_rq job per exact scope/incarnation on the existing default
queue. Producer requests persist filesystem receipts before admission. Active
queued/started/deferred/scheduled work coalesces; reservations remain through
RQ terminal transition so late invalidations cannot create a second active job.
Auth-actor and correlation metadata use the same Redis transaction as admission;
a WATCH conflict must not leave a metadata-only job hash.
The shipped query_catalog_refresh scheduler task selects default and passes
that queue explicitly. The scheduler invokes reconcile_query_catalogs to admit one active
inventory_query_catalog_rq per configured inventory cycle on default and drain
its durable Redis discoveries. The inventory worker streams configured roots,
checkpoints its frontier and fences every publication by current job/cycle.
Scheduler ticks acknowledge discovered scopes atomically with their pending
cursor. Restart replays the current directory prefix with deduplication.
Neither scheduler nor inventory worker executes native catalog scans; inventory
writes no project files. No model dependency edges are added.
Maintenance IDs use query_catalog_refresh_<canonical UUID> and generic
metadata. Polling classification precedes deserialization. Every live eligible
default consumer must advertise matching protocol/storage/root/coordination and
inventory_protocol: 1, admission_protocol: 1, pilot_protocol: 1, and
queue: default in
query_catalog_configuration; missing/incompatible/suspended capacity leaves
durable requests pending. Execution revalidates configuration and incarnation.
The global Redis admission ledger caps outstanding catalog work at seven across configurations and pilots; reservations and uncertain job records retain capacity. The explicit operator pilot requires all twelve compatible default consumers, exact owned scopes, a live observation lease, and immutable retry-inclusive budgets (25 refresh, two inventory). Ordinary waiting work pauses admission. Pilot inventory enumerates only approved locators; admitted jobs may drain after pause or expiry.
QUERY_CATALOG_REFRESH_ENABLED defaults false for coordinated rollout. Disabled
producer migration preserves synchronous compatibility; enabled Redis/worker
failure must not fall back to inline scanning. Scheduler interval defaults 30 s;
inventory defaults to 100 entry records per streaming checkpoint and 100 scope
records drained per scheduler tick; admission remains 25 scopes per tick.
QUERY_CATALOG_CYCLE_SECONDS defaults 86400 as an unproven capacity target;
retry delay starts at 30 s and caps at 3600 s. These are configurable operational
limits, not a freshness guarantee during unavailable storage/workers. The
inventory cursor and cycle errors are operator-visible Redis state; loss resets
bounded discovery against configured roots, while project receipt/attempt files
retain durable intent and evidence. Full cycles include scopes without receipts
so artifact-publication crashes cannot leave metadata stale forever. Discovery
completion is separate from cycle completion: completed_at is recorded only
after all discovered scopes acknowledge their pending receipts. Current and
previous cycle errors remain visible without accumulating older cycles. The
scheduler lease is 60 seconds; expired owners cannot overwrite newer cursors.
Enabled mode requires explicit QUERY_CATALOG_ROOTS_JSON; missing configuration
fails preflight rather than guessing which legacy namespaces are active. Set the
same configuration in producers, scheduler, and every eligible default consumer.
For a deployment using the three canonical /wc1 namespaces, the JSON value is:
[
{"path":"/wc1/runs","namespace":"wc1:runs","coordination":"/wc1/query-engine-coordination/locks","layout":"sharded"},
{"path":"/wc1/batch","namespace":"wc1:batch","coordination":"/wc1/query-engine-coordination/locks","layout":"batch"},
{"path":"/wc1/culverts","namespace":"wc1:culverts","coordination":"/wc1/query-engine-coordination/locks","layout":"culverts"}
]
Add legacy roots only when that deployment actively serves them. Verify each configured namespace and the external coordination directory under the actual runtime identity before enabling. Once configured, missing storage remains an outage: inventory reports degraded health and retries on the next tick. It never silently removes an unavailable namespace from the configured inventory.
Use suspend_catalog_refresh(connection, scope_id) for intentional operator
suspension and resume_catalog_refresh to allow future ticks; ordinary canceled
or failed jobs otherwise remain recoverable. Never purge pending receipt files
or external lifecycle lock inodes as queue cleanup. See the canonical
query catalog contract
and execution package.
Path Legend
| Token | Meaning |
|---|---|
wd |
Run working directory (e.g., /wc1/runs/<prefix>/<runid>). |
runs_dir |
Wepp.runs_dir (typically wd/wepp/runs). |
output_dir |
Wepp.output_dir (typically wd/wepp/output). |
plot_dir |
Wepp.plot_dir (typically wd/wepp/plots). |
export_dir |
Wepp.export_dir (typically wd/export). |
wat_dir |
Watershed.wat_dir (watershed workspace; varies by backend). |
cli_dir |
Wepp.cli_dir (climate .cli directory). |
soils_dir |
Soils.soils_dir (source soils). |
lc_dir |
Landuse.lc_dir (landuse/management templates). |
omni_dir |
wd/omni (OMNI_REL_DIR). |
swat_txtinout_dir |
Swat.swat_txtinout_dir (SWAT TxtInOut). |
dem_dir |
Ron.dem_dir (DEM staging). |
Conventions
- Parent jobs record child ids in
job.meta["jobs:<order>,..."]forjob_info.pyandcancel_job.py. depends_onuses a single job or a list; when the list is empty, it is passed asNone(no dependency).- Required-output dependency edges are strict. A failed prerequisite leaves executable downstream work never-started/deferred; ordinary controller resubmission cancels and detaches that obsolete graph before replacement.
- Failure tolerance is limited to the finalizers and independent AgFields,
Omni-contrast, and WBT request-serialization edges enumerated in the SURF-20A
dependency matrix.
Those sites use
failure_tolerant_depends_on(or an explicitDependency(allow_failure=True)) plusrelease_deferred_job_if_readyfor the already-failed-before-registration race. - WBT controlled policy failures retain their owning exception: they explicitly cancel the never-started abstraction receipt rather than leaving it deferred.
- Prereq files list explicit file/dir inputs referenced by the job or immediate controller calls. Optional inputs are marked.
- Paths are shown relative to
wdunless noted; placeholders like<topaz>and<wepp_id>indicate per-hillslope files.
Batch result settlement
Enrolled ordinary/culvert roots freeze selected run IDs and exact source job IDs before dispatch. Culvert leaves depend on the root dispatch barrier; its finalizer depends on root and leaves with failure tolerance. Ordinary hillslope/watershed stages retain the existing root barrier and linked Omni completion edges. All enrolled execution identities use non-expiring RQ retention.
Child creation/linkage and cancellation share the existing root resource lease. Cancellation refreshes the tree after acquiring that lease. Publication validates and builds outside the lease, then rechecks current attempt, execution states and cancellation before the short manifest/ZIP/summary commit. Polling never builds artifacts. Exhausted failed Omni dependencies permit only verified blocked deferred Omni descendants to be canceled and the failure-tolerant batch finalizer released; foreign or changed graphs remain untouched. See the batch outcome contract.
Scheduled WMesque Synchronization
docker/scheduled-tasks.yml routes the opt-in wmesque_sync task through
wepppy.tools.scheduler._enqueue_task to wepppy.rq.wmesque_rq.enqueue_sync on
batch. The generic queue parameter appears as default/untracked in static
extraction below; the scheduled binding is batch. It enqueues one standalone
sync_wmesque_rq job with no dependency edges, coalescing queued/running work by
destination. Source and destination are snapshotted into job arguments. Live
forest acceptance job IDs and terminal states are retained in the WMesque work
package. Task remains disabled by default.
Automated Dependency Graph
This section is generated by tools/render_rq_dependency_graph_docs.py from
wepppy/rq/job-dependency-graph.static.json. Do not edit this block manually.
- Extracted enqueue edges:
154 - Unique enqueue targets:
98 - Source globs:
wepppy/rq/*.py,wepppy/microservices/rq_engine/*.py,wepppy/weppcloud/routes/**/*.py,wepppy/weppcloud/bootstrap/*.py
wepppy/microservices/rq_engine/ag_fields_routes.py::_enqueue_job
- line
229queuedefaultstageuntrackedenqueuefuncdepends_on (none)
wepppy/microservices/rq_engine/ag_fields_routes.py::_enqueue_watershed_jobs
- line
296queuedefaultstageuntrackedenqueuerun_ag_fields_watershed_rqdepends_on (none) - line
322queuedefaultstageuntrackedenqueuerun_ag_fields_watershed_suite_rqdepends_on (none)
wepppy/microservices/rq_engine/ash_routes.py::run_ash
- line
376queuedefaultstageuntrackedenqueuerun_ash_rqdepends_on (none)
wepppy/microservices/rq_engine/batch_routes.py::delete_batch
- line
194queuebatchstageuntrackedenqueuedelete_batch_rqdepends_on (none)
wepppy/microservices/rq_engine/batch_routes.py::run_batch
- line
116queuebatchstageuntrackedenqueuerun_batch_rqdepends_on (none)
wepppy/microservices/rq_engine/bootstrap_routes.py::_enqueue_no_prep_job
- line
220queuedefaultstageuntrackedenqueuejob_fndepends_on (none)
wepppy/microservices/rq_engine/climate_routes.py::build_climate
- line
743queuedefaultstageuntrackedenqueuebuild_climate_rqdepends_on (none)
wepppy/microservices/rq_engine/culvert_routes.py::_enqueue_culvert_job
- line
218queuedefaultstageuntrackedenqueuequeuedepends_on (none) - line
221queuebatchstageuntrackedenqueuefuncdepends_on (none)
wepppy/microservices/rq_engine/debris_flow_routes.py::run_debris_flow
- line
148queuedefaultstageuntrackedenqueuerun_debris_flow_rqdepends_on (none)
wepppy/microservices/rq_engine/debug_routes.py::hello_world
- line
54queuem4stageuntrackedenqueuehello_world_rqdepends_on (none)
wepppy/microservices/rq_engine/dss_export_routes.py::post_dss_export
- line
199queuedefaultstageuntrackedenqueuepost_dss_export_rqdepends_on (none)
wepppy/microservices/rq_engine/export_routes.py::_enqueue_ermit_export_job
- line
280queuedefaultstageuntrackedenqueuerun_ermit_export_rqdepends_on (none)
wepppy/microservices/rq_engine/export_routes.py::_enqueue_features_export_job
- line
378queuedefaultstageuntrackedenqueuetarget_funcdepends_on (none)
wepppy/microservices/rq_engine/fork_archive_routes.py::archive_run
- line
1740queuefork-archivestageuntrackedenqueuearchive_rqdepends_on (none)
wepppy/microservices/rq_engine/fork_archive_routes.py::fork_project
- line
1607queuefork-archivestageuntrackedenqueuefork_rqdepends_on (none)
wepppy/microservices/rq_engine/fork_archive_routes.py::restore_archive
- line
1848queuefork-archivestageuntrackedenqueuerestore_archive_rqdepends_on (none)
wepppy/microservices/rq_engine/geneva_routes.py::_enqueue_geneva_job
- line
306queuedefaultstageuntrackedenqueuefuncdepends_on (none)
wepppy/microservices/rq_engine/geneva_routes.py::_enqueue_geneva_workflow_jobs
- line
358queuedefaultstageuntrackedenqueuerun_geneva_prepare_hrus_rqdepends_on (none) - line
365queuedefaultstageuntrackedenqueuerun_geneva_build_frequency_panel_rqdepends_onrun_geneva_prepare_hrus_rq - line
374queuedefaultstageuntrackedenqueuerun_geneva_run_batch_rqdepends_onrun_geneva_build_frequency_panel_rq
wepppy/microservices/rq_engine/landuse_routes.py::build_landuse
- line
885queueunknownstageuntrackedenqueuebuild_landuse_rqdepends_on (none) - line
949queuedefaultstageuntrackedenqueuebuild_landuse_rqdepends_on (none)
wepppy/microservices/rq_engine/landuse_routes.py::modify_landuse_mapping
- line
1266queuedefaultstageuntrackedenqueuemodify_landuse_mapping_rqdepends_on (none)
wepppy/microservices/rq_engine/landuse_soils_routes.py::build_landuse_and_soils
- line
64queuedefaultstageuntrackedenqueueland_and_soil_rqdepends_on (none)
wepppy/microservices/rq_engine/migration_routes.py::_enqueue_migration_job
- line
179queuedefaultstageuntrackedenqueuemigrations_rqdepends_on (none)
wepppy/microservices/rq_engine/omni_routes.py::_delete_omni_contrasts
- line
1323queuebatchstageuntrackedenqueuedelete_omni_contrasts_rqdepends_on (none)
wepppy/microservices/rq_engine/omni_routes.py::_run_omni
- line
969queuebatchstageuntrackedenqueuerun_omni_scenarios_rqdepends_on (none)
wepppy/microservices/rq_engine/omni_routes.py::_run_omni_contrasts
- line
1141queuebatchstageuntrackedenqueuerun_omni_contrasts_rqdepends_on (none)
wepppy/microservices/rq_engine/openet_ts_routes.py::acquire_openet_ts
- line
88queuedefaultstageuntrackedenqueuefetch_and_analyze_openet_ts_rqdepends_on (none)
wepppy/microservices/rq_engine/polaris_routes.py::acquire_polaris
- line
91queuedefaultstageuntrackedenqueuefetch_and_align_polaris_rqdepends_on (none)
wepppy/microservices/rq_engine/postfire_debris_flow_routes.py::enqueue
- line
151queuedefaultstageuntrackedenqueueupload_dnbr_rq if kind == 'upload_attempt' else run_m3_rq if record.get('model') == 'M3' else run_m1_rqdepends_on (none)
wepppy/microservices/rq_engine/project_config_update_routes.py::update_apply
- line
703queuedefaultstageuntrackedenqueue<unknown>depends_on (none)
wepppy/microservices/rq_engine/rap_ts_routes.py::acquire_rap_ts
- line
160queuedefaultstageuntrackedenqueuefetch_and_analyze_rap_ts_rqdepends_on (none)
wepppy/microservices/rq_engine/rhem_routes.py::run_rhem
- line
85queuedefaultstageuntrackedenqueuerun_rhem_rqdepends_on (none)
wepppy/microservices/rq_engine/roads_routes.py::_enqueue_roads_job
- line
60queuedefaultstageuntrackedenqueuefuncdepends_on (none)
wepppy/microservices/rq_engine/run_sync_routes.py::run_sync
- line
280queuedefaultstageuntrackedenqueuerun_sync_rqdepends_on (none) - line
288queuedefaultstageuntrackedenqueuerun_sync_rqdepends_on (none) - line
319queuedefaultstageuntrackedenqueuemigrations_rqdepends_onrun_sync_rq
wepppy/microservices/rq_engine/rusle_routes.py::build_rusle
- line
100queuedefaultstageuntrackedenqueuebuild_rusle_rqdepends_on (none)
wepppy/microservices/rq_engine/soils_routes.py::build_soils
- line
334queuedefaultstageuntrackedenqueuebuild_soils_rqdepends_on (none)
wepppy/microservices/rq_engine/swat_routes.py::run_swat
- line
111queuedefaultstageuntrackedenqueuerun_swat_rqdepends_on (none)
wepppy/microservices/rq_engine/treatments_routes.py::build_treatments
- line
211queuedefaultstageuntrackedenqueuebuild_treatments_rqdepends_on (none)
wepppy/microservices/rq_engine/upload_climate_routes.py::upload_cli
- line
212queuedefaultstageuntrackedenqueueupload_cli_rqdepends_on (none)
wepppy/microservices/rq_engine/watershed_routes.py::build_subcatchments_and_abstract_watershed
- line
1164queuedefaultstageuntrackedenqueuebuild_subcatchments_and_abstract_watershed_rqdepends_on (none)
wepppy/microservices/rq_engine/watershed_routes.py::fetch_dem_and_build_channels
- line
809queuedefaultstageuntrackedenqueuefetch_dem_and_build_channels_rqdepends_on (none)
wepppy/microservices/rq_engine/watershed_routes.py::set_outlet
- line
935queuedefaultstageuntrackedenqueueset_outlet_rqdepends_on (none)
wepppy/microservices/rq_engine/webhooks.py::enqueue
- line
214queuedefaultstageuntrackedenqueuefuncdepends_on (none)
wepppy/microservices/rq_engine/wepp_routes.py::_handle_run_wepp_request
- line
237queuedefaultstageuntrackedenqueueqdepends_on (none) - line
240queuedefaultstageuntrackedenqueuejob_fndepends_on (none)
wepppy/rq/ag_fields_rq.py::run_ag_fields_watershed_suite_rq
- line
494queuedefaultstageuntrackedenqueuerun_ag_fields_watershed_rqdepends_onDependency(jobs=[previous_job_id], allow_failure=True) - line
520queuedefaultstageuntrackedenqueuefinalize_ag_fields_watershed_suite_rqdepends_onDependency(jobs=[child_job.id for child_job in scheme_jobs], allow_failure=True)
wepppy/rq/batch_rq.py::run_batch_rq
- line
744queuebatchstagejobs:0enqueuerun_batch_hillslopes_rqdepends_onjob_id - line
755queuebatchstagejobs:0enqueuerun_batch_watershed_rqdepends_onDependency(jobs=[hillslope_job_id], allow_failure=True) - line
780queuebatchstagejobs:1enqueue_final_batch_complete_rqdepends_onDependency(jobs=[watershed_job.id for watershed_job in watershed_jobs], allow_failure=True)
wepppy/rq/culvert_rq_pipeline.py::enqueue_culvert_batch_jobs
- line
87queuebatchstagejobs:0enqueuetasks.run_culvert_run_rqdepends_onparent_job.id - line
106queuebatchstagejobs:1enqueuetasks._final_culvert_batch_complete_rqdepends_on*child_jobs,parent_job,tasks.run_culvert_run_rq
wepppy/rq/omni_rq.py::run_omni_contrasts_rq
- line
928queuebatchstagejobs:contrastenqueuerun_omni_contrast_rqdepends_on_failure_tolerant_depends_on(batch_depends) - line
943queuebatchstagejobs:finalizeenqueue_finalize_omni_contrasts_rqdepends_on_failure_tolerant_depends_on(batch_depends)
wepppy/rq/omni_rq.py::run_omni_scenarios_rq
- line
733queuebatchstagejobs:0enqueuerun_omni_scenario_rqdepends_on (none) - line
752queuebatchstagejobs:1enqueuerun_omni_scenario_rqdepends_onrun_omni_scenario_rq - line
771queuebatchstagejobs:2enqueue_compile_hillslope_summaries_rqdepends_onrun_omni_scenario_rq - line
783queuebatchstagejobs:3enqueue_finalize_omni_scenarios_rqdepends_on_compile_hillslope_summaries_rq
wepppy/rq/project_rq.py::_enqueue_serial_subcatchment_tree
- line
787queuedefaultstageuntrackedenqueuebuild_subcatchments_rqdepends_onDependency(jobs=[previous_tail_id], allow_failure=True) - line
795queuedefaultstageuntrackedenqueueabstract_watershed_rqdepends_onbuild_subcatchments_rq
wepppy/rq/project_rq.py::fetch_dem_and_build_channels_rq
- line
1652queuedefaultstagejobs:0enqueuebuild_channels_rqdepends_on (none) - line
1668queuedefaultstagejobs:0enqueuefetch_dem_rqdepends_on (none) - line
1676queuedefaultstagejobs:1enqueuebuild_channels_rqdepends_onfetch_dem_rq
wepppy/rq/project_rq.py::fork_rq
- line
3133queuedefaultstagejobs:1enqueue_finish_fork_rqdepends_onDependency(jobs=[final_wepp_job.id], allow_failure=True)
wepppy/rq/query_catalog_rq.py::_enqueue_inventory
- line
364queuedefaultstageuntrackedenqueueinventory_query_catalog_rqdepends_on (none)
wepppy/rq/query_catalog_rq.py::enqueue_catalog_refresh
- line
216queuedefaultstageuntrackedenqueuerefresh_query_catalog_rqdepends_on (none)
wepppy/rq/run_catalog_rq.py::enqueue_sweep
- line
124queuedefaultstageuntrackedenqueuesweep_run_catalogdepends_on (none)
wepppy/rq/swat_rq.py::run_swat_noprep_rq
- line
118queuedefaultstagejobs:0enqueue_run_swat_rqdepends_on (none)
wepppy/rq/swat_rq.py::run_swat_rq
- line
71queuedefaultstagejobs:0enqueue_build_swat_inputs_rqdepends_on (none) - line
75queuedefaultstagejobs:1enqueue_run_swat_rqdepends_on_build_swat_inputs_rq
wepppy/rq/wepp_rq.py::run_wepp_watershed_noprep_rq
- line
1113queuedefaultstagejobs:6enqueuesys.modules[__name__]._log_complete_rqdepends_on (none)
wepppy/rq/wepp_rq.py::run_wepp_watershed_rq
- line
968queuedefaultstagejobs:6enqueuesys.modules[__name__]._log_complete_rqdepends_on (none)
wepppy/rq/wepp_rq_pipeline.py::enqueue_log_prep_complete
- line
90queuedefaultstagejobs:6enqueuetasks._log_prep_complete_rqdepends_ondepends_on
wepppy/rq/wepp_rq_pipeline.py::enqueue_watershed_noprep_pipeline
- line
926queuedefaultstagejobs:3enqueuetasks.run_ss_batch_watershed_rqdepends_on (none) - line
938queuedefaultstagejobs:3enqueuetasks.run_watershed_rqdepends_on (none) - line
951queuedefaultstagejobs:4enqueuetasks._post_run_cleanup_out_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq - line
964queuedefaultstagejobs:4enqueuetasks._post_prep_details_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq - line
978queuedefaultstagejobs:4enqueuetasks._post_make_loss_grid_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq - line
991queuedefaultstagejobs:4enqueuetasks._post_watershed_interchange_rqdepends_ontasks._post_run_cleanup_out_rq - line
1005queuedefaultstagejobs:5enqueuetasks._post_legacy_arc_export_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq - line
1017queuedefaultstagejobs:6enqueuetasks._log_complete_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._post_legacy_arc_export_rq
wepppy/rq/wepp_rq_pipeline.py::enqueue_watershed_pipeline
- line
785queuedefaultstagejobs:2enqueuetasks._prep_watershed_rqdepends_on (none) - line
799queuedefaultstagejobs:3enqueuetasks.run_ss_batch_watershed_rqdepends_ontasks._prep_watershed_rq - line
812queuedefaultstagejobs:3enqueuetasks.run_watershed_rqdepends_ontasks._prep_watershed_rq - line
826queuedefaultstagejobs:4enqueuetasks._post_run_cleanup_out_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq - line
839queuedefaultstagejobs:4enqueuetasks._post_prep_details_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq - line
853queuedefaultstagejobs:4enqueuetasks._post_make_loss_grid_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq - line
866queuedefaultstagejobs:4enqueuetasks._post_watershed_interchange_rqdepends_ontasks._post_run_cleanup_out_rq - line
880queuedefaultstagejobs:5enqueuetasks._post_legacy_arc_export_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq - line
892queuedefaultstagejobs:6enqueuetasks._log_complete_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._post_legacy_arc_export_rq
wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_noprep_pipeline
- line
571queuedefaultstagejobs:1enqueuetasks._run_hillslopes_rqdepends_on (none) - line
580queuedefaultstagejobs:2enqueuetasks._build_hillslope_interchange_rqdepends_ontasks._run_hillslopes_rq - line
592queuedefaultstagejobs:2enqueuetasks._build_totalwatsed3_rqdepends_ontasks._build_hillslope_interchange_rq - line
613queuedefaultstagejobs:3enqueuetasks.run_ss_batch_watershed_rqdepends_ontasks._run_hillslopes_rq - line
626queuedefaultstagejobs:3enqueuetasks.run_watershed_rqdepends_ontasks._run_hillslopes_rq - line
639queuedefaultstagejobs:4enqueuetasks._post_run_cleanup_out_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._run_hillslopes_rq - line
652queuedefaultstagejobs:4enqueuetasks._post_prep_details_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._run_hillslopes_rq - line
667queuedefaultstagejobs:4enqueuetasks._run_hillslope_watbal_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._run_hillslopes_rq,tasks._build_hillslope_interchange_rq - line
680queuedefaultstagejobs:4enqueuetasks._post_make_loss_grid_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._run_hillslopes_rq - line
691queuedefaultstagejobs:4enqueuetasks._post_watershed_interchange_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._build_hillslope_interchange_rq - line
704queuedefaultstagejobs:4enqueuetasks._analyze_return_periods_rqdepends_ontasks._post_watershed_interchange_rq,tasks._build_totalwatsed3_rq - line
719queuedefaultstagejobs:2enqueuetasks.post_dss_export_rqdepends_ontasks._build_hillslope_interchange_rq,tasks._post_run_cleanup_out_rq - line
732queuedefaultstagejobs:5enqueuetasks._post_legacy_arc_export_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._run_hillslope_watbal_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._analyze_return_periods_rq - line
745queuedefaultstagejobs:5enqueuetasks._post_gpkg_export_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._run_hillslope_watbal_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._analyze_return_periods_rq - line
763queuedefaultstagejobs:6enqueuetasks._log_complete_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._run_hillslope_watbal_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._analyze_return_periods_rq,tasks._post_legacy_arc_export_rq,tasks._post_gpkg_export_rq,tasks._build_hillslope_interchange_rq,tasks._build_totalwatsed3_rq,tasks.post_dss_export_rq
wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_pipeline
- line
116queuedefaultstagejobs:0enqueuetasks._prep_multi_ofe_rqdepends_on (none) - line
127queuedefaultstagejobs:0enqueuetasks._prep_slopes_rqdepends_on (none) - line
137queuedefaultstagejobs:0enqueuetasks._prep_managements_rqdepends_on (none) - line
146queuedefaultstagejobs:0enqueuetasks._prep_soils_rqdepends_on (none) - line
157queuedefaultstagejobs:0enqueuetasks._prep_climates_rqdepends_on (none) - line
167queuedefaultstagejobs:0enqueuetasks._prep_remaining_rqdepends_ontasks._prep_multi_ofe_rq,tasks._prep_slopes_rq,tasks._prep_managements_rq,tasks._prep_soils_rq,tasks._prep_climates_rq - line
177queuedefaultstagejobs:1enqueuetasks._run_hillslopes_rqdepends_ontasks._prep_remaining_rq - line
190queuedefaultstagejobs:2enqueuetasks._prep_watershed_rqdepends_ontasks._run_hillslopes_rq - line
217queuedefaultstagejobs:2enqueuetasks._build_swat_inputs_rqdepends_ontasks._run_hillslopes_rq,tasks._prep_watershed_rq - line
226queuedefaultstagejobs:3enqueuetasks._run_swat_rqdepends_ontasks._build_swat_inputs_rq - line
239queuedefaultstagejobs:2enqueuetasks._build_hillslope_interchange_rqdepends_ontasks._run_hillslopes_rq,tasks._build_swat_inputs_rq - line
250queuedefaultstagejobs:2enqueuetasks._build_totalwatsed3_rqdepends_ontasks._build_hillslope_interchange_rq - line
264queuedefaultstagejobs:2enqueuetasks._build_swat_inputs_rqdepends_ontasks._build_hillslope_interchange_rq,tasks._prep_watershed_rq - line
273queuedefaultstagejobs:3enqueuetasks._run_swat_rqdepends_ontasks._build_swat_inputs_rq - line
290queuedefaultstagejobs:3enqueuetasks.run_ss_batch_watershed_rqdepends_ontasks._prep_watershed_rq - line
303queuedefaultstagejobs:3enqueuetasks.run_watershed_rqdepends_ontasks._prep_watershed_rq - line
316queuedefaultstagejobs:4enqueuetasks._post_run_cleanup_out_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq - line
329queuedefaultstagejobs:4enqueuetasks._post_prep_details_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq - line
345queuedefaultstagejobs:4enqueuetasks._run_hillslope_watbal_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq,tasks._build_hillslope_interchange_rq - line
358queuedefaultstagejobs:4enqueuetasks._post_make_loss_grid_rqdepends_ontasks.run_ss_batch_watershed_rq,tasks.run_watershed_rq,tasks._prep_watershed_rq - line
369queuedefaultstagejobs:4enqueuetasks._post_watershed_interchange_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._build_hillslope_interchange_rq - line
382queuedefaultstagejobs:4enqueuetasks._analyze_return_periods_rqdepends_ontasks._post_watershed_interchange_rq,tasks._build_totalwatsed3_rq - line
397queuedefaultstagejobs:2enqueuetasks.post_dss_export_rqdepends_ontasks._build_hillslope_interchange_rq,tasks._post_run_cleanup_out_rq - line
410queuedefaultstagejobs:5enqueuetasks._post_legacy_arc_export_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._run_hillslope_watbal_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._analyze_return_periods_rq - line
423queuedefaultstagejobs:5enqueuetasks._post_gpkg_export_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._run_hillslope_watbal_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._analyze_return_periods_rq - line
444queuedefaultstagejobs:6enqueuetasks._log_complete_rqdepends_ontasks._post_run_cleanup_out_rq,tasks._post_prep_details_rq,tasks._run_hillslope_watbal_rq,tasks._post_make_loss_grid_rq,tasks._post_watershed_interchange_rq,tasks._analyze_return_periods_rq,tasks._post_legacy_arc_export_rq,tasks._post_gpkg_export_rq,tasks._build_hillslope_interchange_rq,tasks._build_totalwatsed3_rq,tasks.post_dss_export_rq,tasks._run_swat_rq
wepppy/rq/wepp_rq_pipeline.py::enqueue_wepp_prep_only_pipeline
- line
470queuedefaultstagejobs:0enqueuetasks._prep_multi_ofe_rqdepends_on (none) - line
481queuedefaultstagejobs:0enqueuetasks._prep_slopes_rqdepends_on (none) - line
491queuedefaultstagejobs:0enqueuetasks._prep_managements_rqdepends_on (none) - line
501queuedefaultstagejobs:0enqueuetasks._prep_soils_rqdepends_on (none) - line
512queuedefaultstagejobs:0enqueuetasks._prep_climates_rqdepends_on (none) - line
522queuedefaultstagejobs:0enqueuetasks._prep_remaining_rqdepends_ontasks._prep_multi_ofe_rq,tasks._prep_slopes_rq,tasks._prep_managements_rq,tasks._prep_soils_rq,tasks._prep_climates_rq - line
532queuedefaultstagejobs:2enqueuetasks._prep_watershed_rqdepends_ontasks._prep_remaining_rq
wepppy/rq/wmesque_rq.py::enqueue_sync
- line
48queuedefaultstageuntrackedenqueuesync_wmesque_rqdepends_on (none)
wepppy/weppcloud/bootstrap/enable_jobs.py::enqueue_bootstrap_enable
- line
122queuedefaultstageuntrackedenqueuebootstrap_enable_rqdepends_on (none)
wepppy/weppcloud/routes/nodb_api/geneva_bp.py::_enqueue_geneva_job
- line
252queuedefaultstageuntrackedenqueuefuncdepends_on (none)
wepppy/weppcloud/routes/nodb_api/interchange_bp.py::_enqueue_interchange_job
- line
58queuedefaultstageuntrackedenqueuerun_interchange_migrationdepends_on (none)
wepppy/weppcloud/routes/nodb_api/path_ce_bp.py::run_path_cost_effective
- line
276queuedefaultstageuntrackedenqueuerun_path_cost_effective_rqdepends_on (none)
wepppy/weppcloud/routes/nodb_api/project_bp.py::delete_run
- line
376queuedefaultstageuntrackedenqueuedelete_run_rqdepends_on (none)
wepppy/weppcloud/routes/nodb_api/project_bp.py::task_set_readonly
- line
748queuedefaultstageuntrackedenqueueset_run_readonly_rqdepends_on (none)
wepppy/weppcloud/routes/nodb_api/rangeland_bp.py::task_build_rangeland_cover
- line
254queuedefaultstageuntrackedenqueuebuild_rangeland_cover_rqdepends_on (none)
wepppy/weppcloud/routes/nodb_api/roads_bp.py::_enqueue_roads_job
- line
470queuedefaultstageuntrackedenqueuefuncdepends_on (none)
wepppy/weppcloud/routes/weppcloudr.py::_enqueue_deval_job
- line
642queuedefaultstageuntrackedenqueuerender_deval_details_rqdepends_on (none)
Single-Storm Deprecation
Valid saved individual single-storm inputs can run with a deprecation advisory. The event pipeline skips continuous-only stages, including annual hillslope loss, water balance, return periods, and DSS export. Saved storm batches require per-event postprocessing outside the active RQ pipeline. See the WEPP input contract.
Wait Strategy And Inventory
- Prefer explicit
depends_onedges. File waits are a last-resort guard for filesystem propagation, not a substitute for missing dependencies. - Waits must be time-bounded and run only inside background RQ workers or offline tooling; never add waits to UI routes or sync handlers where a run might not exist yet.
- Use
wepppy.io_wait.wait_for_path/wait_for_pathswhen possible for stable-size checks. The interchange helperwepppy.wepp.interchange._utils._wait_for_pathnow delegates toio_waitwith stable-size checks. - When outputs might not exist because a run never executed, fail fast rather than waiting.
Wait Inventory (File Guards)
| Area | Function | Guard | Target(s) | Timeout / poll | Notes |
|---|---|---|---|---|---|
wepppy/rq/wepp_rq.py |
_post_watershed_interchange_rq |
wait_for_path via _wait_for_output |
wepp/output/pass_pw0.txt (or .gz), ebe_pw0.txt, loss_pw0.txt, chan.out, chanwb.out, chnwb.txt, soil_pw0.txt (or .gz) |
timeout_s=60, poll_s=0.5 |
RQ worker only; stable-size check before watershed interchange. Legacy single-storm conditionals skip some waits; single-storm is deprecated. |
wepppy/rq/wepp_rq.py |
_run_hillslope_watbal_rq |
wait_for_path |
wepp/output/interchange/H.wat.parquet |
timeout_s=60 (poll via PERIDOT_INPUT_POLL_S) |
RQ worker only; stable-size check. |
wepppy/rq/wepp_rq.py |
_build_totalwatsed3_rq |
wait_for_paths |
wepp/output/interchange/H.pass.parquet, wepp/output/interchange/H.wat.parquet |
timeout_s=60 (poll via PERIDOT_INPUT_POLL_S) |
RQ worker only; stable-size check before totalwatsed3 aggregation. |
wepppy/rq/wepp_rq.py |
_analyze_return_periods_rq |
wait_for_paths |
wepp/output/interchange/ebe_pw0.parquet (fallback wepp/output/ebe_pw0.parquet), wepp/output/interchange/totalwatsed3.parquet |
timeout_s=60 |
RQ worker only; stable-size check before return-period export. |
wepppy/rq/wepp_rq.py |
post_dss_export_rq |
wait_for_path / wait_for_paths |
export/dss/dss_channels.geojson (if written), export/dss/totalwatsed3_chan_*.dss (if written), export/dss/peak_chan_*.dss (if written), export/dss/README.dss_export.md (if written), export/dss.zip |
timeout_s=60 |
RQ worker only; stable-size checks between DSS export stages. |
wepppy/rq/project_rq.py |
build_subcatchments_rq |
wait_for_path |
watershed.subwta (SUBWTA.ARC or subwta.tif) |
defaults (PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S) |
Guard before publishing completion trigger. |
wepppy/rq/project_rq.py |
abstract_watershed_rq |
wait_for_path |
watershed.subwta (SUBWTA.ARC or subwta.tif) |
defaults (PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S) |
Guard before abstraction. |
wepppy/rq/project_rq.py |
fork_rq (undisturbify=True) |
wait_for_paths |
ron.nodb, wepp.nodb, landuse.nodb, soils.nodb, disturbed.nodb |
timeout_s=60 (poll via PERIDOT_INPUT_POLL_S) |
Ensures forked NoDb files are readable before undisturbify. |
wepppy/topo/peridot/peridot_runner.py |
run_peridot_abstract_watershed |
_wait_for_file |
dem/topaz/SUBWTA.ARC |
PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S |
Defensive guard before Peridot. |
wepppy/topo/peridot/peridot_runner.py |
run_peridot_wbt_abstract_watershed |
_wait_for_file |
dem/wbt/subwta.tif |
PERIDOT_INPUT_WAIT_S / PERIDOT_INPUT_POLL_S |
Defensive guard before Peridot (WBT). |
wepppy/wepp/interchange/watershed_ebe_interchange.py |
run_wepp_watershed_ebe_interchange |
_utils._wait_for_path |
wepp/output/ebe_pw0.txt |
timeout=60, poll=0.5 |
Stable-size wait inside interchange. |
wepppy/wepp/interchange/watershed_chanwb_interchange.py |
run_wepp_watershed_chanwb_interchange |
_utils._wait_for_path |
wepp/output/chanwb.out |
timeout=60, poll=0.5 |
Stable-size wait inside interchange. |
wepppy/wepp/interchange/watershed_chan_peak_interchange.py |
run_wepp_watershed_chan_peak_interchange |
_utils._wait_for_path |
wepp/output/chan.out |
timeout=60, poll=0.5 |
Stable-size wait inside interchange. |
wepppy/wepp/interchange/watershed_chnwb_interchange.py |
run_wepp_watershed_chnwb_interchange |
_utils._wait_for_path |
wepp/output/chnwb.txt |
timeout=60, poll=0.5 |
Stable-size wait inside interchange. |
wepppy/wepp/reports/return_periods.py |
refresh_return_period_events |
wait_for_path |
interchange/ebe_pw0.parquet (or wepp/output/ebe_pw0.parquet), interchange/totalwatsed3.parquet |
defaults | Report builder waits for parquet assets unless wait_for_inputs=False (UI routes). |
Catalog
Run catalog maintenance
The opt-in scheduler entry calls run_catalog_rq.enqueue_sweep on the existing
batch queue. Admission requires a live idle compatible consumer and no global
RQ suspension, using passive consistent registration/TTL observations. No
qualifying worker means no job or admission-pointer mutation; dirty SQL state
waits for the next tick. Observation does not reserve the worker. Redis admission
still coalesces queued/started/deferred/scheduled sweeps; a PostgreSQL
transaction advisory lock serializes scheduled and CLI scans. There are no
child jobs or per-save enqueues. The generated scanner's generic queue label
does not override this scheduler-selected queue. IDs retain the exact
run_catalog_sweep_<UUID> form and public polling never exposes task arguments,
results or private exception metadata. All eligible consumers require the
canonical run-catalog compatibility/origin/mount proof before admission.
Real isolated-queue validation executes the callable and checks its terminal
job tree plus source-derived SQL values; live host-tree evidence remains a
staged deployment gate.
AgFields rq-engine enqueue surface
The helper wepppy.microservices.rq_engine.ag_fields_routes._enqueue_job is represented as one dynamic func edge in the generated graph. Its three concrete call sites have no RQ depends_on edges; prerequisite checks happen synchronously before enqueue. A single-scheme _enqueue_watershed_jobs call still enqueues the scheme worker directly. Run All instead enqueues run_ag_fields_watershed_suite_rq as one top-level parent. The route atomically stores all planned children on that parent, and dispatch shares a lock with cancellation. Concept 1, Concept 2, and hybrid remain ordered jobs:* children; each scheme after the first depends on its predecessor through Dependency(..., allow_failure=True). A registered finalizer depends on all three scheme IDs through Dependency(..., allow_failure=True). Every failure-tolerant dependent applies the Batch Runner already-terminal release guard, so a predecessor that fails before registration cannot strand the rest of the suite. This prevents overlapping watershed memory peaks while permitting complete comparison attempts and one deterministic terminal suite signal from the finalizer.
| Job key | RQ entrypoint | Preconditions / file inputs |
|---|---|---|
agfields_build_subfields |
wepppy.rq.ag_fields_rq.build_ag_fields_subfields_rq |
Confirmed schema; dem/wbt/flovec.tif; canonical boundary GeoJSON; runs rasterize → abstract → polygonize inside one job |
agfields_plantdb |
wepppy.rq.ag_fields_rq.process_ag_fields_plant_db_rq |
Server-validated, uniquely named ZIP under ag_fields/; staged archive is removed after terminal success/failure |
agfields_run_wepp |
wepppy.rq.ag_fields_rq.run_ag_fields_wepp_rq |
Current sub-fields; complete valid mapping; parent wepp/runs/p<wepp_id>.sol and .cli pairs; after WEPP, synchronously converts all six H<sub_field_id> report families into the specialized six-file bundle and stamps completion only after publication |
agfields_run_watershed_suite |
wepppy.rq.ag_fields_rq.run_ag_fields_watershed_suite_rq |
Run All only; parent dispatches three serial scheme children and the always-released finalizer; response job_id names this parent |
agfields_run_watershed_concept_1 |
wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq |
Current successful Stage 4; observed climate; complete parent prepared inputs; writes only wepp/ag_fields/watershed/concept-1/ |
agfields_run_watershed_concept_2 |
wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq |
Same preconditions; writes only wepp/ag_fields/watershed/concept-2/; the historical agfields_run_watershed key remains a Concept 2 job-id alias |
agfields_run_watershed_hybrid |
wepppy.rq.ag_fields_rq.run_ag_fields_watershed_rq |
Same preconditions plus the explicit Peridot connectivity resources; writes only wepp/ag_fields/watershed/hybrid/ |
| suite finalizer | wepppy.rq.ag_fields_rq.finalize_ag_fields_watershed_suite_rq |
Depends on all three scheme jobs with allow_failure=True; publishes terminal suite status after every scheme is terminal |
wepppy/rq/wepp_rq.py
run_wepp_rq(runid)
Graph (stage order is the jobs:<n> prefix):
jobs:0 prep inputs
-> _prep_remaining_rq
-> _run_hillslopes_rq
-> (_prep_watershed_rq if run_watershed)
-> _build_hillslope_interchange_rq (after SWAT build when delete_after_interchange)
-> _build_totalwatsed3_rq
-> (post_dss_export_rq if enabled and run_watershed; after `_post_run_cleanup_out_rq`)
-> (SWAT build/run if enabled; before interchange when delete_after_interchange)
-> watershed runs (if run_watershed)
-> post-run jobs (cleanup, prep details, watbal, loss grid, watershed interchange, return periods)
-> export jobs (legacy arc, gpkg)
-> _log_complete_rq
Child jobs, dependencies, and file prerequisites:
Note: Parquet prerequisites are canonical directory paths (for example watershed/hillslopes.parquet). WD-root sidecars (for example watershed.hillslopes.parquet) are retired and treated as migration-required boundaries.
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:0 | _prep_multi_ofe_rq |
none | wat_dir/slope_files/hillslopes/hill_<topaz>.mofe.slp; soils_dir/hill_<topaz>.mofe.sol; lc_dir/*.man; optional wepp.kslast_map raster |
| jobs:0 | _prep_slopes_rq |
none | wat_dir/hill_<topaz>.slp or wat_dir/slope_files/hillslopes/hill_<topaz>.slp |
| jobs:0 | _prep_managements_rq |
none | landuse/landuse.parquet; lc_dir/*.man; soils/soils.parquet; optional disturbed.nodb + disturbed maps; optional rap/rap_ts.parquet |
| jobs:0 | _prep_soils_rq |
none | soils_dir/*.sol; optional wepp.kslast_map raster; soils/soils.parquet for soil metadata |
| jobs:0 | _prep_climates_rq |
none | cli_dir/<cli_fn>; for SS batch writes runs_dir/p<wepp_id>.<ss_batch_id>.cli and runs_dir/pw0.<ss_batch_id>.cli |
| jobs:0 | _prep_remaining_rq |
jobs0_hillslopes_prep list |
runs_dir/p*.slp; runs_dir/p*.sol; runs_dir/p*.man; runs_dir/p*.cli |
| jobs:1 | _run_hillslopes_rq |
_prep_remaining_rq |
runs_dir/*.run; runs_dir/p*.slp; runs_dir/p*.sol; runs_dir/p*.man; runs_dir/p*.cli |
| jobs:2 | _prep_watershed_rq |
_run_hillslopes_rq |
wat_dir/structure.json or wat_dir/network.txt; watershed/channels.parquet; watershed/hillslopes.parquet; wat_dir/slope_files/channels.slp or wat_dir/channels.slp; soils/soils.parquet; cli_dir/<cli_fn> |
| jobs:2 | _build_hillslope_interchange_rq |
_run_hillslopes_rq (+ _build_swat_inputs_rq if SWAT enabled + delete_after_interchange) |
output_dir/H*.pass.dat; output_dir/H*.ebe.dat; output_dir/H*.element.dat; output_dir/H*.loss.dat; output_dir/H*.soil.dat; output_dir/H*.wat.dat |
| jobs:2 | _build_totalwatsed3_rq |
_build_hillslope_interchange_rq |
output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash/H<wepp_id>_ash.parquet or ash/H<wepp_id>.parquet; optional ash.nodb; watershed/hillslopes.parquet (area lookup when ash is present) |
| jobs:2 | post_dss_export_rq |
_build_hillslope_interchange_rq (+ _post_run_cleanup_out_rq if run_watershed) |
output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash/*.parquet; output_dir/chan.out; watershed.channels_shp (GeoJSON); wat_dir/network.txt (downstream mapping); export/dss dir (created) |
| jobs:2 | _build_swat_inputs_rq |
_build_hillslope_interchange_rq (+ _prep_watershed_rq if present); if delete_after_interchange then _run_hillslopes_rq (+ _prep_watershed_rq if present) |
wepp/output/* (hillslope outputs); watershed/hillslopes.parquet; watershed/channels.parquet; swat_template_dir/* (config swat_template_dir) |
| jobs:3 | _run_swat_rq |
_build_swat_inputs_rq |
swat_txtinout_dir/* |
| jobs:3 | run_watershed_rq / run_ss_batch_watershed_rq |
_prep_watershed_rq |
runs_dir/pw0.str; runs_dir/pw0.slp; runs_dir/pw0.chn; runs_dir/pw0.sol; runs_dir/pw0.cli; runs_dir/pw0.man; runs_dir/chan.inp; for SS batch runs_dir/pw0.<ss_batch_id>.cli |
| jobs:4 | _post_run_cleanup_out_rq |
jobs3_watersheds or _prep_watershed_rq |
runs_dir/*.out; optional runs_dir/tc_out.txt |
| jobs:4 | _post_prep_details_rq |
post_dependencies |
watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet |
| jobs:4 | _run_hillslope_watbal_rq |
post_dependencies + _build_hillslope_interchange_rq |
output_dir/interchange/H.wat.parquet |
| jobs:4 | _post_make_loss_grid_rq |
post_dependencies |
watershed.subwta (wat_dir/SUBWTA.ARC or wbt_wd/subwta.tif); watershed.discha (wat_dir/DISCHA.ARC or wbt_wd/discha.tif); output_dir/H* |
| jobs:4 | _post_watershed_interchange_rq |
_post_run_cleanup_out_rq |
output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt |
| jobs:4 | _analyze_return_periods_rq |
_post_watershed_interchange_rq + _build_totalwatsed3_rq |
output_dir/interchange/ebe_pw0.parquet (fallback output_dir/ebe_pw0.parquet); output_dir/interchange/totalwatsed3.parquet; optional cached output_dir/return_periods*.json |
| jobs:5 | _post_legacy_arc_export_rq |
jobs4_post |
topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata |
| jobs:5 | _post_gpkg_export_rq |
jobs4_post |
watershed.subwta_shp (SUBCATCHMENTS.WGS JSON); watershed.channels_shp (CHANNELS.WGS JSON); watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet; output_dir/interchange/loss_pw0.hill.parquet; output_dir/interchange/loss_pw0.chn.parquet |
| jobs:6 | _log_complete_rq |
jobs4_post + jobs5_post |
ron.nodb |
Audit notes:
_run_hillslope_watbal_rqdepends on_build_hillslope_interchange_rqand still waits foroutput_dir/interchange/H.wat.parquetinternally.post_dss_export_rqonly enqueues whenrun_watershedis true (guarded inrun_wepp_rq).post_dss_export_rqcallstotalwatsed_partitioned_dss_export, which runsrun_totalwatsed3per channel; it does not require_build_totalwatsed3_rqunless the export is refactored to reuse a prebuilttotalwatsed3.parquet._build_totalwatsed3_rqdepends on hillslope interchange. If watershed outputs are required, depend onjobs3_watershedsor_post_watershed_interchange_rq.- When
delete_after_interchangeis enabled, SWAT inputs are built before interchange and interchange waits on SWAT to avoid deleting rawwepp/outputfiles. _run_hillslope_watbal_rqandpost_dss_export_rqtarget continuous runs; legacy single-storm conditionals may skip them, but single-storm is deprecated._analyze_return_periods_rqdepends ontotalwatsed3; legacy single-storm runs skip this pipeline and are deprecated.
run_wepp_noprep_rq(runid)
Graph (prep stages skipped; uses existing inputs under wepp/runs/):
jobs:1 _run_hillslopes_rq
-> _build_hillslope_interchange_rq
-> _build_totalwatsed3_rq
-> watershed runs (if run_watershed)
-> post-run jobs (cleanup, prep details, watbal, loss grid, watershed interchange, return periods)
-> exports (legacy arc, gpkg)
-> (post_dss_export_rq if enabled and run_watershed)
-> _log_complete_rq
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:1 | _run_hillslopes_rq |
none | runs_dir/*.run; runs_dir/p*.slp; runs_dir/p*.sol; runs_dir/p*.man; runs_dir/p*.cli |
| jobs:2 | _build_hillslope_interchange_rq |
_run_hillslopes_rq |
output_dir/H*.pass.dat; output_dir/H*.ebe.dat; output_dir/H*.element.dat; output_dir/H*.loss.dat; output_dir/H*.soil.dat; output_dir/H*.wat.dat |
| jobs:2 | _build_totalwatsed3_rq |
_build_hillslope_interchange_rq |
output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash parquet; watershed/hillslopes.parquet |
| jobs:3 | run_watershed_rq / run_ss_batch_watershed_rq |
_run_hillslopes_rq |
runs_dir/pw0.* as above; SS batch runs_dir/pw0.<ss_batch_id>.cli |
| jobs:4 | _post_run_cleanup_out_rq |
jobs3_watersheds or _run_hillslopes_rq |
runs_dir/*.out; optional runs_dir/tc_out.txt |
| jobs:4 | _post_prep_details_rq |
post_dependencies |
watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet |
| jobs:4 | _run_hillslope_watbal_rq |
post_dependencies + _build_hillslope_interchange_rq |
output_dir/interchange/H.wat.parquet |
| jobs:4 | _post_make_loss_grid_rq |
post_dependencies |
watershed.subwta; watershed.discha; output_dir/H* |
| jobs:4 | _post_watershed_interchange_rq |
_post_run_cleanup_out_rq |
output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt |
| jobs:4 | _analyze_return_periods_rq |
_post_watershed_interchange_rq + _build_totalwatsed3_rq |
output_dir/interchange/ebe_pw0.parquet; output_dir/interchange/totalwatsed3.parquet |
| jobs:2 | post_dss_export_rq |
_build_hillslope_interchange_rq (+ _post_run_cleanup_out_rq if run_watershed) |
output_dir/interchange/H.pass.parquet; output_dir/interchange/H.wat.parquet; optional ash parquet; output_dir/chan.out; watershed.channels_shp; wat_dir/network.txt; export/dss dir |
| jobs:5 | _post_legacy_arc_export_rq |
jobs4_post |
topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata |
| jobs:5 | _post_gpkg_export_rq |
jobs4_post |
watershed.subwta_shp; watershed.channels_shp; watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet; output_dir/interchange/loss_pw0.hill.parquet; output_dir/interchange/loss_pw0.chn.parquet |
| jobs:6 | _log_complete_rq |
jobs4_post + jobs5_post |
ron.nodb |
run_wepp_watershed_rq(runid)
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:2 | _prep_watershed_rq |
none | wat_dir/structure.json or wat_dir/network.txt; watershed/channels.parquet; watershed/hillslopes.parquet; wat_dir/slope_files/channels.slp or wat_dir/channels.slp; soils/soils.parquet; cli_dir/<cli_fn> |
| jobs:3 | run_watershed_rq / run_ss_batch_watershed_rq |
_prep_watershed_rq |
runs_dir/pw0.* as above; SS batch runs_dir/pw0.<ss_batch_id>.cli |
| jobs:4 | _post_run_cleanup_out_rq |
jobs3_watersheds or _prep_watershed_rq |
runs_dir/*.out; optional runs_dir/tc_out.txt |
| jobs:4 | _post_prep_details_rq |
post_dependencies |
watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet |
| jobs:4 | _post_make_loss_grid_rq |
post_dependencies |
watershed.subwta; watershed.discha; output_dir/H* (job skipped when hillslope outputs are missing) |
| jobs:4 | _post_watershed_interchange_rq |
_post_run_cleanup_out_rq |
output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt |
| jobs:5 | _post_legacy_arc_export_rq |
jobs4_post |
topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata |
| jobs:6 | _log_complete_rq |
jobs4_post + jobs5_post |
ron.nodb |
Audit notes:
- This entrypoint bypasses hillslope prep/run; ensure downstream jobs only require watershed outputs.
- Return-period exports are not enqueued here because
totalwatsed3is not scheduled.
run_wepp_watershed_noprep_rq(runid)
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:3 | run_watershed_rq / run_ss_batch_watershed_rq |
none | runs_dir/pw0.* as above; SS batch runs_dir/pw0.<ss_batch_id>.cli |
| jobs:4 | _post_run_cleanup_out_rq |
jobs3_watersheds |
runs_dir/*.out; optional runs_dir/tc_out.txt |
| jobs:4 | _post_prep_details_rq |
post_dependencies |
watershed/hillslopes.parquet; watershed/channels.parquet; landuse/landuse.parquet; soils/soils.parquet |
| jobs:4 | _post_make_loss_grid_rq |
post_dependencies |
watershed.subwta; watershed.discha; output_dir/H* (job skipped when hillslope outputs are missing) |
| jobs:4 | _post_watershed_interchange_rq |
_post_run_cleanup_out_rq |
output_dir/pass_pw0.txt; output_dir/ebe_pw0.txt; output_dir/chan.out; output_dir/chanwb.out; output_dir/chnwb.txt; output_dir/soil_pw0.txt or output_dir/soil_pw0.txt.gz; output_dir/loss_pw0.txt |
| jobs:5 | _post_legacy_arc_export_rq |
jobs4_post |
topaz_wd/*.ARC; topaz_wd/SUBCATCHMENTS.JSON; topaz_wd/CHANNELS.JSON; output_dir/loss_pw0.txt; watershed/hillslopes.parquet; watershed/channels.parquet; optional ash metadata |
| jobs:6 | _log_complete_rq |
jobs4_post + jobs5_post |
ron.nodb |
wepppy/rq/swat_rq.py
run_swat_rq(runid)
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:0 | _build_swat_inputs_rq |
none | wepp/output/* (hillslope outputs); watershed/hillslopes.parquet; watershed/channels.parquet; swat_template_dir/* |
| jobs:1 | _run_swat_rq |
_build_swat_inputs_rq |
swat_txtinout_dir/* |
Audit notes:
- This assumes WEPP outputs already exist. The caller is responsible for enqueueing only after WEPP hillslope/watershed outputs are ready.
run_swat_noprep_rq(runid)
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:0 | _run_swat_rq |
none | swat_txtinout_dir/* |
wepppy/rq/batch_rq.py
run_batch_rq(batch_name)
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| jobs:0,hillslopes | run_batch_hillslopes_rq (per watershed feature) |
root dispatch, strict | Batch workspace, validated watershed feature and base project |
| jobs:0 | run_batch_watershed_rq (per watershed feature) |
corresponding hillslope job, failure-tolerant terminal observer | Finished upstream and matching visible batch_handoff/<hillslope-job-id>.json required before science |
| jobs:1 | _final_batch_complete_rq |
all jobs:0 via failure-tolerant RQ Dependency (+ Omni final jobs when run_omni_scenarios_rq enqueues work) |
batch_runner.nodb (for status updates) |
Audit notes:
run_batch_watershed_rqverifies the attempt-bound handoff, freshly hydrates leaf state, and owns terminal metadata and the completion trigger. Seedocs/schemas/batch-task-boundary-contract.md.- It calls
run_omni_scenarios_rq(runid)inline. Omni finalizer attachment is serialized under a finalizer-specific Redis lock and uses root RQ metadata, so parallel leaves cannot lose each other's dependencies. - The completion job allows failed dependencies so it still runs after leaves reach terminal
failedstatus and can publish the failed-run summary instead of remaining deferred.
wepppy/rq/culvert_rq.py
run_culvert_batch_rq(culvert_batch_uuid)
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| jobs:0 | run_culvert_run_rq (per run_id) |
none | batch_root/metadata.json; batch_root/model-parameters.json; batch_root/topo/breached_filled_DEM_UTM.tif (or payload override); batch_root/topo/streams.tif (or payload override); batch_root/culverts/watersheds.geojson; batch_root/topo/flovec.tif; batch_root/topo/netful.tif; batch_root/topo/chnjnt.tif; batch_root/topo/chnjnt.streams.tif |
| jobs:1 | _final_culvert_batch_complete_rq |
all jobs:0 |
batch_root/culverts/*.json run metadata (written by child jobs) |
Audit notes:
- Batch-level topo generation happens before enqueueing children. If that becomes async, ensure child jobs depend on it.
run_culvert_run_rq(runid, culvert_batch_uuid, run_id)
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| (inline) | _process_culvert_run (internal) |
none | batch_root/metadata.json; batch_root/model-parameters.json; batch_root/culverts/watersheds.geojson; batch_root/culverts/culvert_points.geojson; batch_root/topo/breached_filled_DEM_UTM.tif; batch_root/topo/flovec.tif; batch_root/topo/netful.tif; batch_root/topo/chnjnt.tif; per-run directory batch_root/runs/<run_id> |
wepppy/rq/omni_rq.py
run_omni_scenarios_rq(runid)
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:0 | run_omni_scenario_rq (stage 1 scenarios) |
none | Base scenario inputs under wepp/output/*; dependency hash uses wepp/output/loss_pw0.txt |
| jobs:1 | run_omni_scenario_rq (stage 2 scenarios) |
all stage 1 jobs, strict | Stage 1 scenario outputs under omni/scenarios/<scenario>/wepp/output/loss_pw0.txt |
| jobs:2 | _compile_hillslope_summaries_rq |
stage 2 jobs (or stage 1 if no stage 2), strict | wepp/output/loss_pw0.txt; omni/scenarios/<scenario>/wepp/output/loss_pw0.txt |
| jobs:3 | _finalize_omni_scenarios_rq |
_compile_hillslope_summaries_rq, strict |
omni.nodb |
Audit notes:
- Stage 2 depends on the full set of stage 1 jobs, not just the specific scenario’s dependency. This is conservative but can increase critical path length.
- Every scenario edge consumes required upstream outputs and is strict. A failure stops all transitive scenario, compile, and finalization work; the controller's ordinary retry replaces the blocked deferred graph.
run_omni_contrasts_rq(runid)
| Stage | Child job | depends_on |
File prerequisites (relative to wd) |
|---|---|---|---|
| jobs:contrast:* | run_omni_contrast_rq |
previous batch (if any), failure tolerant | omni/contrasts/contrast_<id>.tsv; scenario outputs referenced by sidecar paths under omni/contrasts/<id>/wepp/output/* |
| jobs:finalize | _finalize_omni_contrasts_rq |
last batch, failure tolerant | omni.nodb |
Audit notes:
- Batches are chained (
batch_depends), so the final dependency on the last batch implies all prior batches. - Because the batches are chained, a bare
depends_onlist makes one failed contrast strand every later batch and the finalizer. Both edges use_failure_tolerant_depends_onplus_release_deferred_job_if_readyso_finalize_omni_contrasts_rqalways reaches a terminal state and emitsEND_BROADCAST.
wepppy/rq/project_rq.py
fetch_dem_and_build_channels_rq(...)
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| jobs:0 | fetch_dem_rq |
none | ron.nodb; writes dem_dir/dem.tif or dem_dir/dem.vrt |
| jobs:1 | build_channels_rq |
fetch_dem_rq |
dem_dir/dem.tif or dem_dir/dem.vrt |
| jobs:0 | build_channels_rq |
none | (upload DEM mode) dem_dir/dem.tif or dem_dir/dem.vrt already present |
Audit notes:
- No file-based waits here; ensure
fetch_dem_rqfully writesdem_dir/dem.tifbefore returning.
build_subcatchments_and_abstract_watershed_rq(runid, updates)
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| jobs:0 | build_subcatchments_rq |
prior same-run request tail, when present (failure-tolerant serialization) | dem_dir/dem.tif or dem_dir/dem.vrt; channel outputs under wat_dir/* from build_channels_rq |
| jobs:1 | abstract_watershed_rq (nonmutating completion receipt) |
build_subcatchments_rq (strict) |
none; build and abstraction already completed inside jobs:0 |
Audit notes:
build_subcatchments_rquseswait_for_pathonwatershed.subwta, then performs abstraction under the same watershed directory-root lock before publishing completion.- One optimistic Redis transaction registers both jobs, both dependency directions, both root metadata links, the per-run tail, and jobs:0 queue or deferred membership. The prior-tail edge serializes independent request mutation ownership and may cross root lineage; retry cleanup treats it as an external dependency boundary and admission splices a canceled tail back to its still-active predecessor. A failed jobs:0 cancels only its matching receipt for controlled policy failures; other failures leave the strict receipt deferred.
fork_rq(..., undisturbify=True)
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| (inline) | run_wepp_rq(new_runid) |
none | Forked run directory with .nodb files (ron.nodb, wepp.nodb, landuse.nodb, soils.nodb, disturbed.nodb) |
| (inline) | _finish_fork_rq |
final job from run_wepp_rq |
relies on run_wepp_rq completion metadata |
Audit notes:
_finish_fork_rqdepends on the final_log_complete_rqjob produced byrun_wepp_rq, so it should run after all WEPP tasks finish.- Every WEPP pipeline child created for an undisturbify fork carries server-generated
fork_failurelineage andreport_fork_failureas its RQ failure callback. Root child registration is saved before enqueue. The callback atomically verifies the current destination receipt and records failure plus source status; strict dependency edges are unchanged. WepppyRqWorker.handle_job_failurerepeats guarded reporting after abrupt work-horse death, when RQ skips the job callback. Duplicate/stale reporting is suppressed. Neither path releases claims, cancels siblings, nor unblocks model stages.- Initial Wepp/Watershed controller reads in preparation jobs have a shared five-second ENOENT/ESTALE retry budget. This does not retry the job or any mutation. See ADR-0049.
wepppy/microservices/rq_engine/run_sync_routes.py
POST /rq-engine/run-sync
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | run_sync_rq |
none | Target root writable (default /wc1/runs); remote source reachable; optional source_run_token with source-host scope runs:read for private runs |
| root | migrations_rq |
run_sync_rq |
target_root/<prefix>/<runid> run directory with .nodb files and run outputs |
Audit notes:
migrations_rqdepends on the sync job, ensuring that run files exist before migrations execute.run_sync_rqwritesRunMigrationrows by importingwepppy.weppcloud.app; workers executing it needSECRET_KEY(_FILE),SECURITY_PASSWORD_SALT(_FILE), andPOSTGRES_PASSWORD(_FILE)configured.- When
source_run_tokenis supplied,run_sync_rqusesAuthorization: Bearer <token>foraria2c.specand aria2 fetch operations; token is not written to provenance records.
wepppy/microservices/rq_engine/batch_routes.py
POST /api/batch/_/{batch_name}/run-batch
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | run_batch_rq |
none | Batch workspace batch_root/<batch_name>; batch_runner.nodb; uploaded watershed GeoJSON + validated template |
POST /api/batch/_/{batch_name}/delete-batch
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | delete_batch_rq |
none | Batch workspace batch_root/<batch_name> (including _base and runs/*) |
Audit notes:
delete_batch_rqis destructive and removes the batch workspace recursively after clearing batch run lock/cache metadata.
wepppy/microservices/rq_engine/bootstrap_routes.py
POST /api/runs/{runid}/{config}/bootstrap/enable
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | bootstrap_enable_rq |
none | wd (run directory from get_wd(runid)); writes .git/ and commits tracked bootstrap inputs under wepp/runs and/or swat/TxtInOut |
Audit notes:
- Route-level dedupe key
bootstrap:enable:job:<runid>prevents duplicate enable jobs while one is active. - Route acquires run-scoped Redis git lock
bootstrap:git-lock:<runid>before enqueueing; the worker releases it inbootstrap_enable_rqcleanup.
POST /api/runs/{runid}/{config}/run-wepp-npprep
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | run_wepp_noprep_rq |
none | Existing WEPP input state under wepp/runs; Bootstrap must already be enabled for the run |
Audit notes:
- Route resets Redis prep timestamps for WEPP tasks before enqueueing the no-prep run.
- This endpoint intentionally skips prep rebuilds and runs against the currently checked-out git state.
POST /api/runs/{runid}/{config}/run-wepp-watershed-no-prep
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | run_wepp_watershed_noprep_rq |
none | Existing watershed inputs under wepp/runs; Bootstrap must already be enabled for the run |
Audit notes:
- Route resets the watershed prep timestamp before enqueueing.
- No branch checkout occurs here; execution uses current repository checkout.
POST /api/runs/{runid}/{config}/run-swat-noprep
| Stage | Child job | depends_on |
File prerequisites |
|---|---|---|---|
| root | run_swat_noprep_rq |
none | Existing SWAT+ inputs under swat/TxtInOut; Bootstrap must already be enabled and SWAT mod present |
Audit notes:
- Route enqueues SWAT no-prep execution without rebuilding TxtInOut.
- Endpoint depends on existing SWAT inputs and the active checkout state.
NFS Consistency Headaches
Background (Feb 6, 2026)
Production WEPP runs experienced rare, intermittent failures (on the order of 1 in 10⁶ to 10⁹ events) where RQ worker jobs could not read files that upstream jobs had already written. Symptoms included FileNotFoundError on .parquet interchange files, wait_for_path timeouts on watershed outputs (pass_pw0.txt, ebe_pw0.txt, etc.), and stale reads returning truncated or zero-length files. These failures were non-deterministic and consistent with NFS client cache staleness, though the root cause has not been definitively confirmed.
Notably, the same soft mount configuration works reliably on the dev and test-production NFS servers, which use ZFS as the backing filesystem. The failures have only been observed on the production NAS, suggesting the issue may involve an interaction between the soft mount options and the production storage backend rather than the mount options alone.
Suspected Cause: NFS Mount Options
The /geodata mount was configured with soft mode and large read/write buffers with aggressive attribute caching. The following options are suspected contributors to the degraded behavior:
# BEFORE (degraded)
nas.rocket.net:/wepp /geodata nfs4 rw,noatime,proto=tcp,vers=4.2,soft,rsize=32768,wsize=32768,timeo=600,_netdev 0 0
| Option | Potential problem |
|---|---|
soft |
NFS operations return errors after timeo retries instead of blocking. Under load, transient server delays could cause I/O errors that surface as missing or corrupt files in downstream RQ jobs. |
rsize=32768,wsize=32768 |
Fixed 32 KB transfer sizes bypass the kernel's auto-negotiation, potentially limiting throughput on large sequential writes (interchange parquet files) and adding round-trips for small metadata operations. |
timeo=600 |
60-second timeout (in deciseconds) combined with soft means jobs may receive EIO after a single retry window rather than waiting for the server to recover. |
(no actimeo) |
Default attribute cache lifetime (~3–60 s) can allow NFS clients to serve stale stat() results. A file written by one worker could appear absent or zero-length to another worker for up to 60 seconds. |
Remediation: Updated Mount
# AFTER (remediated)
nas.rocket.net:/wepp /geodata nfs4 rw,noatime,proto=tcp,vers=4.2,hard,intr,actimeo=0,_netdev 0 0
| Option | Effect |
|---|---|
hard |
NFS operations block indefinitely until the server responds, preventing spurious EIO errors during transient load spikes. Workers wait rather than fail. |
intr |
Allows blocked hard-mount operations to be interrupted by signals, so jobs can still be canceled or timed out by RQ without requiring a full process kill. |
actimeo=0 |
Disables attribute caching entirely. Every stat(), open(), and directory listing goes to the server, ensuring that when an upstream job writes a file, downstream jobs should see it immediately. This targets the suspected file-guard race conditions. |
(removed rsize/wsize) |
Lets the NFS client and server auto-negotiate optimal transfer sizes (typically 1 MB on NFSv4.2), improving throughput for large parquet writes and reducing round-trips. |
(removed timeo) |
With hard mount, timeout is irrelevant; operations retry until the server responds. |
Impact on RQ Job Dependencies and File Guards
The catalog documents two layers of ordering between RQ jobs: explicit depends_on edges (job-level) and wait_for_path / wait_for_paths file guards (filesystem-level). If the NFS attribute cache hypothesis is correct, it would affect the second layer directly and the first layer indirectly.
Explicit depends_on edges (job-level ordering) are not themselves affected by NFS caching. RQ tracks job completion in Redis, so a depends_on edge fires when the upstream job's Python function returns, not when the filesystem flushes. However, a job returning successfully does not guarantee that its output files are visible to other NFS clients—this is the gap the file guards fill. The edge graph in this catalog was verified through several auditing and review passes during this investigation.
File guards (wait_for_path / wait_for_paths with stable-size checks) are the suspected primary failure point. Under a stale attribute cache scenario:
stat()calls insidewait_for_pathcould returnENOENTeven after the file existed on the server.- When
stat()did succeed, it could report a stale size (often 0), causing the stable-size check to loop until timeout. - Under a
softmount, the timeout could produce anEIOinstead of a retry, crashing the worker.
The following call chains were traced end-to-end from rq-engine API entry through RQ job execution to file-guard resolution:
Trace: POST /rq-engine/run-sync → Full WEPP Run
rq-engine API: POST /rq-engine/run-sync
└─ run_sync_rq [depends_on: none]
└─ migrations_rq [depends_on: run_sync_rq]
└─ run_wepp_rq(runid) [depends_on: none]
├─ jobs:0 _prep_*_rq [depends_on: none]
├─ jobs:0 _prep_remaining_rq [depends_on: jobs0 prep list]
├─ jobs:1 _run_hillslopes_rq [depends_on: _prep_remaining_rq]
├─ jobs:2 _prep_watershed_rq [depends_on: _run_hillslopes_rq]
├─ jobs:2 _build_hillslope_interchange_rq [depends_on: _run_hillslopes_rq]
│ └─ READS: output_dir/H*.pass.dat, H*.ebe.dat, H*.wat.dat, ...
│ └─ WRITES: output_dir/interchange/H.pass.parquet, H.wat.parquet, ...
├─ jobs:2 _build_totalwatsed3_rq [depends_on: _build_hillslope_interchange_rq]
│ └─ FILE GUARD: wait_for_paths → H.pass.parquet, H.wat.parquet ← NFS-affected
│ └─ WRITES: output_dir/interchange/totalwatsed3.parquet
├─ jobs:3 run_watershed_rq [depends_on: _prep_watershed_rq]
│ └─ WRITES: output_dir/pass_pw0.txt, ebe_pw0.txt, chan.out, ...
├─ jobs:4 _post_run_cleanup_out_rq [depends_on: jobs3_watersheds]
├─ jobs:4 _post_watershed_interchange_rq [depends_on: _post_run_cleanup_out_rq]
│ └─ FILE GUARD: wait_for_path → pass_pw0.txt, ebe_pw0.txt,
│ chan.out, chanwb.out, chnwb.txt, soil_pw0.txt ← NFS-affected
│ └─ WRITES: output_dir/interchange/ebe_pw0.parquet, ...
├─ jobs:4 _run_hillslope_watbal_rq [depends_on: post_deps + interchange]
│ └─ FILE GUARD: wait_for_path → H.wat.parquet ← NFS-affected
├─ jobs:4 _analyze_return_periods_rq [depends_on: _post_watershed_interchange + totalwatsed3]
│ └─ FILE GUARD: wait_for_paths → ebe_pw0.parquet,
│ totalwatsed3.parquet ← NFS-affected
├─ jobs:5 _post_gpkg_export_rq [depends_on: jobs4_post]
└─ jobs:6 _log_complete_rq [depends_on: jobs4+5]
Every ← NFS-affected guard above is a candidate site for stale-cache failures under the old mount. With actimeo=0, these guards should resolve on the first or second poll cycle (typically < 1 s) instead of timing out.
Trace: Subcatchment Build → Abstract Watershed
rq-engine API: POST /api/subcatchments/build
└─ build_subcatchments_rq [depends_on: prior same-run child, if any]
├─ FILE GUARD: wait_for_path → watershed.subwta ← NFS-affected
└─ abstract watershed under the same directory-root lock
└─ abstract_watershed_rq [nonmutating receipt; depends_on: build_subcatchments_rq]
Trace: Fork with Undisturbify
rq-engine API: POST /api/fork
└─ fork_rq(undisturbify=True)
└─ FILE GUARD: wait_for_paths → ron.nodb, wepp.nodb, landuse.nodb,
soils.nodb, disturbed.nodb ← NFS-affected
└─ run_wepp_rq(new_runid) [full chain as above]
└─ _finish_fork_rq [depends_on: _log_complete_rq]
Summary
| Before | After |
|---|---|
soft mount with fixed buffer sizes; stale attribute cache |
hard,intr mount with actimeo=0; auto-negotiated transfer sizes |
File guards timed out or received EIO under load |
File guards should resolve immediately; operations block until server responds |
| Non-deterministic run failures correlated with NFS client count | Expected deterministic file visibility across all NFS clients |
The depends_on job graph documented in this catalog was reviewed through several auditing passes as part of this investigation—no missing edges were identified. However, the job dependency graph is not readily testable in isolation: edges are constructed dynamically at enqueue time, depend on runtime configuration (e.g. run_watershed, delete_after_interchange, SWAT/Omni enablement), and span multiple modules. Validation relies primarily on cognitive review of the code paths rather than automated verification. The interplay of RQ job orchestration, NoDb state, Rust-based interchange, and Fortran WEPP subprocesses makes constructing meaningful automated tests for the dependency graph near impossible, and the regression surface for dependency changes is correspondingly difficult to characterize.
The failures are consistent with the NFS attribute cache allowing downstream file guards to poll stale metadata, though other contributing factors have not been ruled out. The mount remediation targets this class of failure without requiring changes to job dependency wiring or wait timeouts. Continued monitoring is needed to confirm effectiveness.
Checklist For New Child Jobs
Notification inbox population
docker/scheduled-tasks.yml admits wepppy.rq.notifications_rq.populate_notifications
on the existing batch queue only when WEPPCLOUD_NOTIFICATIONS_SWEEP_ENABLED=true.
It has no child jobs or GC/access-compilation dependency. Its database sweep lock
and episode uniqueness make retries safe. Trusted test calls may pass a bounded
registration-ID scope; browser routes never enqueue this task. See
docs/schemas/user-notifications-contract.md and docs/dev-notes/user-notifications.md.
Child-job checklist
- Add
depends_onfor any file or controller state produced by upstream jobs. - Record
job.meta["jobs:<order>,..."]for every child job sojob_info.pyandcancel_job.pycan traverse the tree. - If you rely on filesystem writes inside a single job, prefer explicit file waits over fixed sleeps.
- Update this catalog when adding new enqueue sites or dependency edges.