WEPPcloud

← Back to help

Production M1 upload and execution workflow

Status: accepted 2026-09-10; implemented and validated on the development stack. Scope explicitly requested: NoDb state, prerequisite/freshness checks, dNBR upload/publication, RQ execution, and a minimal control to upload and run M1. Enable Post-fire debris flow from Mods; its control appears immediately below RUSLE. Prepare soil erodibility in RUSLE before running M1. The linked job ID remains available after reload; failed-job diagnostics appear in Details. Use Upload dNBR again to retry the stored map without reselecting it. Reports and interactive dashboard are deferred. Canonical UI contract: Post-fire debris-flow control.

Authority and acceptance boundary

The 2026-09-14 scientific integration amendment accepts common valid support for both models and protects upstream soil building. Implementation is pending; existing numerical behavior described below remains the v1 baseline.

The implemented M1/M3 selection amendment records model-specific UI, task dispatch and prerequisites after checkpoint aa30e637e. M1 has no independent built WEPP Soils requirement; M3 uses that inventory and its dedicated task, with scientific composition still deferred.

Reuse existing predictor, rainfall, dNBR, slope/SBS and scalar contracts. Preserve legacy debris_flow.nodb and outputs. The new facade is PostfireDebrisFlow, file postfire_debris_flow.nodb. Existing project watershed/outlet is exclusive assessment scope. Both upload and run enforce authorization and CSRF at the proper session/token boundary through existing contracts.

Complete the contract-first checkpoint before any coupled runtime edits: explicit operator approval of UI/data/state matrix, two independent read-only reviews, all affected canonical amendments, and a standalone ancestor commit. Checkpoints 5c0a172ee and 595816476 precede runtime implementation. Security impact is high.

Prerequisites and source ownership

M1 eligibility is WBT plus canonical effective continental-us locale. Project-config authority resolves legacy us and cross-boundary behavior; never infer geography from template/config names. A 10 m project is the intended operator smoke test, not a new M1 10 m restriction. Respect accepted area warnings outside inclusive 0.2–8 km² without rejecting on area alone.

Run prerequisites: completed delineation with valid raw DEM/grid/boundary/outlet, prepared SBS, RUSLE Nomograph K with accepted provenance, accepted normalized dNBR, and valid project Climate event parquet. NOAA artifact is required only for requested NOAA design comparisons. Use K- artifact readiness, not a successful unrelated full RUSLE factor build (ADR-0059). No implicit upstream rebuild or NOAA acquisition. Readiness checks validate actual artifacts and source ownership, not simply controller/file presence.

Owner-required live readiness: publish prerequisite/freshness and NOAA availability through the existing Redis-backed preflight WebSocket flow. The control consumes preflight:update, including initial hydration and reconnect reconciliation. Integrate producer invalidations when upstream artifacts change; a historical task-completion flag alone cannot establish current K/dNBR/NOAA readiness. Freeze additive payload keys and producer ownership in the checkpoint. Keep raster decoding/full hashing out of the preflight service and heartbeat path; model submission and worker publication independently enforce current source checks. Disable unavailable/stale NOAA immediately; do not silently switch rainfall sources.

Upload needs eligible project/grid/domain plus write access; it does not require finished Soils, K or Climate. Read-only GET on a never-used module must show a usable empty state without writing NoDb. First authorized mutation creates state under normal lock/persistence rules; present-empty is not corruption.

Durable state and immutable artifacts

Version the new facade state. Freeze exact public/property/serialized names at checkpoint; record at minimum active dNBR artifact identity/metadata, current upload attempt, current run/job identity, selected rainfall source, last accepted run revision, dependency snapshot/freshness, and terminal failure details. Keep large rasters/tables in artifacts, not NoDb JSON. Bind attempt IDs to published artifacts so an old worker cannot replace a newer submission.

Layout under postfire_debris_flow/: visible attempts// contains immutable source/normalized artifacts and unfinished predictor/results bundles. Only accepted completed model files are published directly in postfire_debris_flow/ and exposed through the standard project browser and fixed result links. A completed accepted pointer in NoDb is the active authority; partial directories are never active. Retain previously accepted artifacts on upload/run failure. Specify retention/cleanup and orphan reconciliation before code; no destructive cleanup of prior successful runs is implicit. Generated permissions must work for web and worker identities on the actual target mounts.

Use short lock/refresh/allowlisted-state updates, not a NoDb lock across uploads, normalization, hashing or model execution. Workers snapshot inputs, compute in visible attempt directories and finalize only after reacquiring fresh state and checking attempt/dependency identities. Readbacks refresh cached NoDb using the canonical persistence contract. No whole-object merging or stale mutation retry.

Upload transport and publication

Freeze exact route/multipart fields, size limits and error contract in checkpoint. Accept the backend's format family and integer/float single bands. The browser offers Auto (default), standard scale presets and Custom with factor/offset fields. Resolve Auto to explicit factor/offset before calling the unchanged normalizer; distribution-based detection and ambiguity retry are specified in the UI contract. Record encoding evidence and prevent double scaling. Default GDAL identity metadata does not establish encoding. Auto examines robust valid-value distributions, with metadata as evidence; numerical selection rules require fixture evaluation under ADR-0063. Dtype or a single extreme is insufficient. An unresolved candidate is not published; an authorized bounded retry can reuse its staged files after the user selects a scale. Freeze candidate access/expiry and metadata validation in the checkpoint. No user-entered imagery dates or date prerequisite; retain available source metadata without inheriting dates from a replaced map. Preserve original filename as escaped display metadata across reloads; generate server-owned paths. Show accepted filename separately from a pending replacement. Publish upload details and applied scale in a persistent two-column table inside wc-control__panel-summary, using the exact UI-contract rows. Permit scale correction from retained source through the existing upload action; re-normalize and publish as a new attempt, invalidating results only on accepted replacement. Always display accepted raster formats and integer/floating-point datatype help. Require actual valid-data watershed overlap; partial dNBR is accepted with coverage.

Bound streamed upload size before decoder access; .vrt source references map only to explicitly uploaded companion file(s) within this attempt. No arbitrary server/network paths, traversal, symlinks, derived VRT execution, external HFA resources or archive extraction. Preserve strict dNBR backend behavior rather than exposing its trusted-path assumptions to browser input. Specify overall request cap as well as per-file cap; use SBS precedent without importing its categorical-value limits. Do not write final inputs into a shared public path.

The upload route stages validated transport, enqueues normalization and returns canonical job identity. Queued worker uses normalize_dnbr with current project grid/domain, verifies source hashes and publishes only on complete success with unchanged grid and current attempt. Failed replacement preserves previous accepted map and results. New accepted dNBR invalidates derived results. Grid/outlet changes invalidate normalized dNBR; re-normalization of stored source must be an explicit documented action, not an unnoticed source change.

RQ model execution and freshness

One model job prepares an immutable snapshot, invokes the completed build_m1_predictors and build_m1_results sequentially, then finalizes the active run pointer. Use existing rq-engine submission/auth/job/status conventions; freeze retry/timeout/idempotency and response states before code. Do not invent a separate scheduler or fan out tasks absent a measured need. Never copy a partially built predictor/result directory into active output.

The run route refuses missing/stale prerequisites and attaches to an already active matching attempt on duplicate clicks. Worker rechecks source association and hashes at start and before publication. Inputs changed during computation produce a superseded/not-current result, never a current pointer. Upload/run races and queued retries use explicit attempt ownership. No automatic M3 switch. Incomplete scientific support can produce a successful diagnostic result with null probabilities under accepted backend contracts; this is distinct from a failed worker or an invalid request.

M1 dependency snapshot includes DEM/grid/domain/resolved outlet, SBS artifact/class mapping/assessment, named K and its provenance, dNBR source/normalization/assessment, CLI parquet/mode/date semantics, selected frequency source/NOAA or CLI parity artifact, engine/tool identity and scientific parameters. Hashes establish content identity, not association with current controllers; record and revalidate relevant owner revision/config identities. Do not rerun terrain because only climate changed when validated predictor reuse is possible; reuse policy must verify all predictor dependencies, not timestamps. Unitizer changes never stale scientific results. Freeze exact invalidation matrix at checkpoint, including source deletion and legacy missing provenance.

UI parameters and output boundary

Accepted first-control defaults: frequency source project climate, all 15/30/60 minute event/design durations, 1/2/5/10-year design intervals and 50% inverse threshold. These UI/workflow defaults are accepted in ADR-0063; backend contracts remain explicit-argument APIs. No M3 selector or algorithm/coverage knobs. Encoding describes uploaded values, not model tuning; there are no date fields.

Show status, upload coverage, actionable errors and completion/download access only. Generated event/design/inverse files exist and can be inspected by the operator through existing authorized run-file access. No report tables/charts, new report renderer, public query API or dashboard. Prefer an existing protected file listing with allowlisted bundle links; do not add a general archive service.

Validation and deployment distinction

Exercise valid absent/empty/populated/read-only/legacy states separately from malformed/hostile requests. Require actual upload → normalization/publication → run → generated files → reload under production-equivalent web/worker identities, groups, mounts, umask and orchestration. Include failed replacement, stale inputs, source mutation mid-job, duplicate submit and competing upload/run finalizers. UI tests must prove user labels/status/action outcomes, not only payload shapes.

Update RQ dependency catalog/graph and verify actual job trees. Confirm installed WBT binary capability/hash through real worker execution before rollout. Local Rust build evidence alone does not prove worker availability. Build a concrete release/preflight/rollback plan through canonical deployment tooling; installation or deployment requires existing explicit target authority, not inference from this scaffold. Operator's later 10 m project is the final upload/run smoke case.

Execution contract (2026-09-10)

This section supersedes preceding provisional implementation details and pending choice language. Its exact decisions are normative for the checkpoint; the earlier sections describe scope/rationale. Validation is recorded in the production package.

The owner's instruction to execute this package authorizes implementation of the reviewed UI and the following bounded engineering choices, including its required standalone contract commits. Development conformance is recorded in the package; installation on other hosts remains a separate action.

Transport and admission

Use rq-engine /api/runs/{runid}/{config}/postfire-debris-flow/state (GET), upload-dnbr (POST multipart), retry-dnbr (POST JSON), and run-m1 (POST JSON) under that same postfire-debris-flow/ prefix. GET requires rq:status; mutations require rq:enqueue. All use canonical JWT and run authorization. Before any mutation, require normalized URL config (strip optional .cfg suffix) equals current Ron.config_stem and reject readonly projects. Feature must be enabled; runtime locale and WBT requirements are server checks. Session-token browser requests retain existing CSRF/token contracts.

Upload fields: file, optional companion only for VRT, scale_mode default auto (auto, scaled_1000, normalized, custom), and scale_factor/ add_offset only for custom. Custom requires finite positive factor and finite offset. Reject unknown/duplicate fields, multiple file values and empty files. At most two files and three text fields; cap actual streamed request bytes before multipart parsing at 201 MiB, each raster at 100 MiB, VRT at 64 KiB, each text field at 256 bytes. Enforce actual streaming limits even without Content-Length; close parser files on success/failure. Reject remote/absolute/traversal VRT sources; only one simple basename referring to the explicitly uploaded companion is allowed. Preserve validated basenames inside an exclusive server-owned attempt directory so the VRT identity is preserved. Basenames are not storage authority.

Retry fields: candidate_id, scale_mode, optional custom factor/offset; the ID is a server-issued 32-character lowercase hex UUID, bound to this run's recorded latest candidate or active accepted source. No browser paths. Pending candidates expire after 24 hours; accepted sources remain available for correction. Retrying creates a new attempt; never change accepted files in place. Reauthorize every retry. Concurrent upload normalization rejects a second distinct upload with 409; model execution may coexist with an upload, subject to publication freshness.

JSON retry/run bodies are capped at 4 KiB actual streamed bytes before decoding. Require object JSON and reject duplicate/unknown keys, nonfinite values and non-JSON bodies; Content-Length does not substitute for counted bytes.

Run fields: frequency_source (cli default or noaa), no other parameters. Always evaluate 15/30/60-minute wet events and design storms at 1/2/5/10 years, with inverse target 0.5. No optional CLI frequency CSV is supplied: calculate CLI frequencies directly from parquet, avoiding stale CSV parity artifacts. NOAA is required only when selected. Defaults are recorded with ADR-0063's workflow scope.

POST success returns job_id, result: {attempt_id}; matching active model submissions return the same job ID. Distinct active model requests return 409. Use existing tracked submission/lifecycle lease and enqueue-recovery helpers; record attempt receipt before enqueue and reconcile unknown enqueue outcomes before admitting another attempt. Enqueue failure must not strand a ready-looking job or overwrite accepted output. Terminal RQ state is authoritative for orphan reconciliation; GET may report reconciliation needs but does not persist state. Use the configured RQ timeout and existing status-stream job lifecycle. No unbounded automatic retries; explicit user retry starts a new attempt.

Errors follow canonical RQ envelope: invalid payload/format/encoding/overlap 400, auth 401/403, missing candidate 404, stale/busy/config mismatch 409, size 413, missing prerequisites/ineligible project 422, dependency/service failure 503. Worker expected errors persist a sanitized code/message; detailed exceptions remain in protected logs, not open preflight or job-result payloads.

State, sources and publication

NoDb _state is schema version 1 with active_dnbr, upload_attempt, run_attempt, frequency_source and last_successful_run, initially null except frequency source cli. Attempt records include ID, job key/ID, phase, timestamp, source snapshot and safe error. Successful dNBR record includes immutable attempt ID, relative artifact references, source display filename, summary and snapshot; run record includes ID, source snapshot and completed result bundle references. GET returns a sanitized projection: eligibility/readonly, required-data readiness, NOAA availability, selected source, upload/run status, accepted dNBR summary, previous/current result state and fixed file links. Never serialize raw controller objects or arbitrary manifests to the browser.

Store sources and unfinished computation under postfire_debris_flow/attempts/ with exclusive visible attempt directories. Sources, normalization, intermediate files and failed work are available through the normal project browser. Expose only verified completed model bundle files via fixed accepted-result links. All paths must remain within nonsymlink project-owned directories. Use group- compatible web/worker permissions and validate them under actual identities. No automatic deletion in this increment: reject expired retry candidates; cleanup is an operator task limited to verified unreferenced attempt directories.

RunCapabilityAuthority.locale_profile must resolve to continental-us; missing/invalid or other locale authority is unavailable. Existing authority handles legacy us. No new rectangular/geographic cutoff is inferred. Western US guidance remains visible; scientific area warnings do not block execution.

Read current Ron.dem_fn, Watershed.wbt_wd/bound.tif and outlet.geojson, current Disturbed.sbs_4class_path, rusle/k_polaris_nomograph.tif and rusle/manifest.json, Climate's climate/wepp_cli.parquet and selected climate/atlas14_intensity_pds_mean_metric.csv. Confirm artifact ownership and current controller selections, not arbitrary file presence. SBS is already 0–3; retain classes/masks and use accepted nearest alignment. Use raw DEM, never conditioned relief/fvslop. Initial raster admission retains the existing local M1 supported formats/limits; unsupported raw DEM representation produces an explicit prerequisite error, not an implicit terrain substitution. For dNBR preparation, materialize a self-contained reference DEM through the local M1 reader/writer, preserving samples, support and grid and recording it in accepted artifacts. Statistics-only PAM admission follows the local M1 contract; the uploaded dNBR decoder retains its strict self-contained-file boundary. Original DEM/domain/outlet sources and meaningful external masks are hashed before/after preparation and retained in accepted artifact signatures. External mask appearance/removal/content changes affect dependency freshness; inert statistics do not. The preparation repair is implemented under 7447e6243.

Snapshot relevant input selections and actual source signatures for cheap live freshness; SHA-256 all consumed inputs before/after computation. Exclude unrelated NoDb metadata, Unitizer and unselected NOAA/CLI CSV from scientific identity. Normalize source/grid into immutable attempt inputs while preserving recorded lineage. Model worker builds predictors then results, validates complete manifests, and publishes only after fresh locked rehydration confirms attempt and dependency identity. Changed input or newer attempt yields superseded, preserving prior accepted output. Avoid broad cache clearing; clear only postfire_debris_flow.nodb immediately before mutable RQ hydration and refresh while locked before mutation.

Live preflight and control

Keep the existing boolean preflight wire contract. On initial hydration and each preflight:update, fetch authenticated state; debounce/coalesce requests and ignore out-of-order responses. Upstream RedisPrep changes trigger refresh even when checklist booleans remain the same; ensure the stream emits these events. Publish a domain revision notification on upload/run transitions. Reconcile on socket reconnect and expose connection state using the existing preflight status; do not create another socket or an expensive raster scan in the Go service. Controllers initialized after socket connection read window.preflightConnected before listening for subsequent connection events. Live state inspects owner selections and file signatures, while workers perform full validation. A prerequisite row means input available for model validation, not a guarantee that every watershed cell has valid scientific support.

Use the exact control fields/table/copy in the UI contract. Register postfire_debris_flow as a preview feature, user role, WBT, requires disturbed, enable dependencies polaris and rusle (the registry enables direct dependencies only); enabling features never builds owner data. Render eligible absent-controller state. Feature metadata cannot define locale; run capability authority controls locale availability.

Upload summary remains visible through reload/failed replacement; scale correction uses the existing upload action. Completed model files are status/download access, not a new report. No probability table or dashboard is included.

Auto decoding must reuse dnbr._identity_vrt and bounded detached dnbr._raster validation before reading values. Never open an uploaded pathname directly with GDAL outside that boundary, including during distribution detection.

State serialization and notification details

The authenticated GET result has exactly these top-level keys: schema_version (1), eligible (bool), readonly (bool), unavailable_reason (string or null), required (records below), noaa_available (bool), frequency_source (cli or noaa), upload_ready/run_ready (bool), upload/run (attempt or null), dnbr (accepted summary or null), results (accepted result or null), freshness (absent, current, stale). Readiness records have key (watershed, soils, sbs, k, climate, dnbr), ready (bool), reason (safe code or null), message and control (fixed anchor). The five upstream rows are visible; dNBR readiness belongs to its upload section.

Public attempts have id, job_id (nullable until reconciled), phase, created_at (UTC ISO), error (null or {code,message}), retryable (bool). Phases: staged, queued, running, needs_scale, complete, failed, superseded, enqueue_unknown. needs_scale is successful job processing with unresolved encoding, not accepted publication. Terminal outcomes survive reload. Public dNBR has id (also valid for correction), filename, format, dtype, cell_size_m (two numbers), coverage_fraction, source_range, prepared_range (two numbers each), scale_mode, scale_factor, add_offset, scale_method (distribution, metadata, selected), completed_at, current (bool). Result has id, completed_at, current (bool), partial (bool), files (list of {name,url} for events.parquet, design.parquet, inverse.parquet and manifest.json only). Never include source paths or candidate tokens in preflight/job results. UUID is an identifier, not authorization.

Attempt records additionally store snapshot, source_id, encoding, filename and job_key; schema allows absent optional values only in a freshly created/empty state. Reject malformed populated records; do not interpret them as a ready/empty controller. Public projections explicitly select fields.

Preflight notification choice: emit on every Redis keyspace update, even if checklist/locks/whole-second last_modified compare equal. Do not invent a new revision payload. Existing RedisPrep timestamp/clear/config methods notify the run hash; add an opaque postfire_debris_flow:revision value on domain changes. Managed artifact mutations must notify on publication and invalidation: watershed DEM/domain/outlet, Soils builds, Disturbed SBS validation, RUSLE K publication, Climate export and postfire upload/run transitions. Add notification at any such owner path not already touching the run preflight hash. The browser emits a preflight:connection event with connected boolean on open/close/error; this control rehydrates on open and marks cached readiness non-live on close/error. Existing per-run authorization remains on the detailed state endpoint. No filesystem watcher is introduced: unmanaged operator file edits are checked at next state request/submission and are outside realtime producer notifications.

Preflight completion task

TaskEnum.run_postfire_debris_flow has label Run Post-fire Debris Flow and emoji 🌋. The boolean checklist key postfire_debris_flow maps to #postfire-debris-flow, after RUSLE; the legacy run_debris task stays separate.

Pending M1/M3 amendment (2026-09-11; implementation checkpoint required): 🌋 represents the latest accepted publication evaluated against its own model, frequency and inputs. It does not represent current radio selection. A different model/frequency selection or failed retry does not clear an otherwise current accepted result. No historical search for another current result is introduced.

Project the accepted run's original UTC completion second into timestamps:run_postfire_debris_flow and its immutable model into postfire_debris_flow:model. Legacy missing model means M1; malformed model means incomplete. Complete and scientifically partial publications both count. Absent publication produces no completion timestamp. An accepted dNBR replacement clears an M1 marker, not an M3 marker. The dedicated pending-integration M3 task never sets a completion marker. Python result freshness evaluates the accepted record's model and frequency, independently of selected UI settings.

Serialize each fresh durable read and Redis projection with the existing short run-scoped Redis lock, checking ownership before writing. Project timestamp and model together and persist through RedisPrep recovery dump. Redis/dump failures are logged and do not reclassify durable success. Public notifications contain no filenames, paths or scientific metadata. Marker freshness remains advisory.

The Go checklist requires a positive completion timestamp later than present shared upstream timestamps for DEM, channels, outlet, subcatchments, abstraction, SBS and climate. M1 additionally checks POLARIS/RUSLE markers; it does not check WEPP soils or landuse/rangeland. M3 additionally checks soils and its landuse/ rangeland prerequisites; it does not check POLARIS/RUSLE. Missing upstream markers alone do not reject an accepted publication; malformed present markers do. Same-second ties remain conservatively incomplete. No WEPP run or successful full RUSLE build is required. These coarse checks can conservatively lag exact artifact readiness; authenticated model-specific source freshness remains otherwise authoritative. Source revision notifications still refresh the UI.

The wiring scope includes module preflight.py, production.py result freshness, services/preflight2/internal/checklist/checklist.go and their Python/Go tests. Verify selected M3 with a current M1 result, selected NOAA with accepted CLI, failed M3 after M1 success, and both models' relevant/irrelevant upstream changes.

Existing completed runs may be reconciled after verifying authenticated state reports current results, using the original completion time rather than the migration time. No model rerun is required. This additive task changes neither NoDb nor model-output schemas. The adapter lives in preflight.py, outside the numerical engine fingerprint, so task-only changes preserve scientific freshness.

Fixed completed-file access (publication review clarification)

Completed M1 outputs MUST be published directly as postfire_debris_flow/events.parquet, design.parquet, inverse.parquet, and manifest.json, following the ordinary RUSLE module-directory pattern. They must appear in the existing project file browser and use its existing project access rules. No new UI, nested result directory, symlink, or custom browser route is required. Scientifically partial results are published too.

After the accepted NoDb commit, materialize the four fixed files from that accepted bundle, verifying their recorded hashes. Prepare all files in visible publication_work directories, then replace each destination atomically using ordinary generated-file permissions, with manifest.json replaced last. Validate every source/destination path and all working-copy hashes before the first replacement. Serialize the latest durable-state read and installation with the existing postfire NoDb lock so an older callback cannot restore an older bundle. Each file is atomic; the four-file browser view is not a multi-file transaction. A publication I/O failure is explicit and retryable by republishing accepted files, without rerunning the model. No prior outputs are deleted first. A copy failure preserves accepted NoDb/originals but may leave mixed public files; repair with wepppy.nodb.mods.postfire_debris_flow.publication.publish_outputs(wd).

The accepted NoDb pointer and immutable attempt artifacts continue to establish scientific provenance. Existing authenticated fixed-file download links remain compatible: authenticated rq-engine GET postfire-debris-flow/files/{attempt_id}/{name} requires rq:export, run/config checks, accepted-ID equality, and the four-name allowlist. Resolve the recorded attempt bundle, open once, validate its published signature and SHA-256, then stream that same handle. Changed/missing files return 409; stale previous results remain downloadable if their recorded files match. Uploaded sources and unfinished computations remain visible under attempts/. A failed/unaccepted new run does not replace visible prior outputs; visible files represent the last accepted run and may be scientifically stale. The control continues to show authoritative freshness.

Existing accepted projects can be republished after checking their recorded artifact hashes, without changing their NoDb schema, source snapshots, numerical engine fingerprint, or model files. Publication lives outside fingerprinted engine modules. Absent/empty state is a no-op. Malformed records, changed source files, and symlinked paths fail explicitly before output replacement. This supersedes the former hidden-only output policy at the operator's direction.

Owner completion and recovery conformance

File presence alone never establishes readiness. Require the existing RedisPrep completion receipts for watershed, soils, SBS, climate and POLARIS. Cleared owner receipts make their prerequisite unavailable even when old files remain. K uses its own manifest.json.k publication (mean POLARIS nomograph, expected artifact), not full RUSLE completion. Its required near-surface source layers must exist and must not be newer than that publication; a newer POLARIS completion also requires K preparation again. Legacy K without sufficient publication provenance is shown as needing preparation in RUSLE. This reuses existing owner provenance without changing K values or the RUSLE output schema.

An attempt persists its exact allocated canonical RQ job ID before any enqueue receipt or queue write. Reconciliation uses that ID, function, queue and run/attempt arguments, never a previous attempt's Redis marker. Reconcile under the existing submission lease and re-read durable state before projecting a terminal failure. No enqueue response is required to recover a successfully queued operation. The producer completes the exact-ID queued receipt before queue admission, then hands NoDb mutation ownership to the worker. A successful enqueue response must not reacquire the controller lock or rewrite the association: an immediately starting worker may already own that lock. RQ remains the queue-status authority.

Uploaded source and companion hashes are recorded while streaming, checked across Auto inspection and normalization, and retained through accepted publication. Uploaded source companions participate in accepted-artifact freshness. Validated statistics-only caches on trusted local project rasters are excluded from scientific freshness and expected hashes under the local M1 contract. Parser and download lifetimes close files on cancellation/disconnect, including before response headers. Admission rechecks config/read-only state under the lease before staging or NoDb mutation. Rejected model requests do not change rainfall selection.

A failed candidate's filename identifies the map a scale correction will retry; the accepted summary remains visible separately. Empty upload controls and cached run readiness after a state-fetch or preflight connection failure are disabled.

Publication verification and reuse

Climate readiness requires the active CLI and a successful event-Parquet producer lineage matching its selection and content, as specified by Climate Parquet lineage. This supersedes mtime ordering: an old or proofless event table cannot establish new execution readiness after a failed export. Historical report/calendar reads remain compatible; state polling never creates provenance or regenerates climate. Climate receipt ordering follows the shared preflight contract relative to watershed abstraction. M3 Soils/landuse/rangeland checks are specified in model_selection.md; they are not independent M1 prerequisites. Upload requires only the delineated grid; model execution requires completed watershed abstraction.

Use stat identities for live readiness and locked finalizers. Retain full hashes for accepted dNBR/results/predictors and verify them at admission, execution, reuse and opened-file download. Engine/tool digests are cached by device, inode, size and nanosecond modification/change timestamps. Managed owner invalidations also emit preflight updates; unmanaged file edits are conclusively checked at the execution boundaries rather than by rereading every raster on every UI event.

Rainfall-only reruns may reuse the latest published predictor bundle only after verifying its exact recorded inventory, all predictor-source hashes, tool hash and engine identity. Copy only that inventory; extra files or links are never followed or published. Verify predictor and result signatures before final publication. Model files retain canonical units; UI numeric range formatting does not change stored values.

Completed results include an area_warning boolean propagated directly from the accepted predictor's area_outside_study_range warning. Display it inline with the 0.2–8 km² study range through Unitizer, without disabling execution. General Western US intended-use guidance is visible before upload.

State reconciliation preserves the entire atomic worker publication revision when an attempt completes during a read; it never combines a completed attempt with previously accepted inputs/results. This keeps completion and download readiness consistent during concurrent polling.

Operator installation and acceptance

The current package proves the development Compose workflow only. For another host, inspect its installed wctl preset and obtain the canonical deployment plan with scripts/deploy-production.sh --print-plan --no-flush-rq-db. This change spans web, rq-engine, workers and preflight2; a targeted web-only restart does not install the complete workflow. Use that entry point and its existing queue cutover protections; do not create a parallel rollout or flush RQ.

Follow WBT release cutover and its linked owned runbook: locked release build, atomic installation into /workdir/weppcloud-wbt/WBT/whitebox_tools, matching wrapper surfaces, recorded source/lock/previous/built/installed hashes. Verify StaleySlopeSbs discovery and a disposable execution under every relevant worker's actual identity and mounts. Keep the previous executable and application deployment revision for rollback.

Before enabling wider use, exercise the browser's normal upload, queued normalization, M1 run, authenticated download and reload using real run data. Check both display-unit choices, failed replacement preservation and live prerequisite changes. Record exact attempt/job IDs and generated artifacts; a wrapper import or successful upload response is insufficient. Block rollout when the worker cannot execute the tool or web cannot read its accepted outputs.

If cutover fails, use the deployment entry point's existing recovery procedure and restore the recorded matching application/WBT revision. Stop admitting new postfire jobs while restoring compatibility; preserve accepted NoDb/artifact data and exact job receipts. Do not delete state or substitute an old completed job. After restoration, repeat actual runtime discovery and the disposable workflow before resuming. For hidden legacy artifact storage, use the separate migration below.

Observable intermediate artifacts and migration

The artifact-observability standard overrides the former hidden-attempt policy. Use postfire_debris_flow/attempts// for uploaded source files, normalized dNBR, prepared predictor maps, WBT outputs/diagnostics, manifests and result tables. Both successful and failed attempts remain inspectable and archivable through the existing browser; incomplete.json and durable job state distinguish validity. The four latest accepted files remain directly in postfire_debris_flow/.

dNBR normalization writes visible reserved output paths and retains incomplete artifacts on failure. Publication uses visible publication_work// with status metadata, retains failed copies, and moves successful copies to the module root. Empty coordination files may be removed after successful work; meaningful data and diagnostics may not be deleted merely because an operation failed.

Legacy migration requires a quiescent project and rejects a simultaneous legacy and destination tree. Rename .staging to attempts without changing binary data. Before mutation, retain exact original NoDb/JSON metadata and a file inventory in visible migrations//. Rebase path strings and dependent JSON hash references in dependency order, then refresh affected NoDb artifact signatures. Record every metadata/hash transition; preserve numerical arrays, tables and predictor values. Republish the four accepted root files. No automatic GET mutation or model rerun.

Storage-only engine fingerprint migration is permitted only for the exact known pre-change engine revision recorded by the migration. Record old/new fingerprints and the nonnumerical changes. Unknown prior fingerprints remain unchanged and therefore stale; do not silently promote different science. Preserve metadata for all failed/historical attempts, including incomplete or malformed diagnostic files; malformed JSON is retained verbatim rather than interpreted as lineage. An interrupted migration must retain an explicit visible status and original metadata; recovery is explicit and cannot overwrite a concurrent tree.

Acceptance requires real writer/failure tests, a legacy data migration test with unchanged scientific payloads, canonical archive/restore byte equality, and live browser/download inspection of input, predictor and failed-attempt files.

Attempt receipts and migration exclusion

Every upload/model attempt retains attempts//status.json with its kind and full durable attempt record (ID, job ID, phase, timestamps, input snapshot and existing sanitized error). Persist the old receipt before replacing current NoDb attempt state. A later retry and Redis expiry must not erase failure identity. Retain per-attempt worker error.log diagnostics using the existing run exception logging convention; do not include credentials or session tokens. Historical attempts lacking a recoverable receipt are labeled historical_unknown, never invented as successful. Receipts are ordinary project records in archives.

Migration acquires the existing run-scoped postfire-admission submission/lifecycle lease before refreshing NoDb and taking its mutation lock. Reject ACTIVE phases, including enqueue_unknown, and verify recorded jobs and active default-queue postfire jobs for the run are terminal before mutation. Missing expired jobs are permitted only for already terminal attempt records. NoDb locking alone is not sufficient to exclude a worker that is writing raster files.

Operator migration and recovery

Run in the normal worker container with the project's service identity:

from wepppy.nodb.mods.postfire_debris_flow.migration import migrate_attempts
migrate_attempts("/wc1/runs/ad/addicted-reservist")

Inspect postfire_debris_flow/migrations/<id>/migration.json, then verify current NoDb freshness and the ordinary browser/download paths. Recovery after relocation requires migrate_attempts(wd, resume="<id>"); it revalidates accepted original signatures, independently derives JSON rebasing, and rejects altered evidence. A status.json in preparing with no migration.json means preparation stopped before any attempt-tree mutation; retain that audit and start a fresh migration. Do not merge competing trees or edit checksums to make a failed migration pass. New uploads/state changes explicitly reject an unmigrated .staging tree.

The raw NoDb backup must contain the same plain JSON _state captured in the migration report. An unsupported serialization requires operator inspection; recovery never executes arbitrary serialized backup objects. Only the exact LEGACY_ENGINE to TARGET_ENGINE pair in migration.py permits storage-only freshness promotion. Future numerical code changes cannot inherit that approval.

An upload receives its visible staged receipt before project file copying. A failed transfer retains partial source bytes, an error log and failed receipt without replacing the prior accepted NoDb state. If storage itself prevents the receipt update, the staged receipt remains and the service logs that failure.

Scientific integration amendment — 2026-09-14

For new production execution, the runtime integration contract and accepted ADR-0067 govern model-specific common support, recorded-depth soil policy, version-2 predictors and mask publication. Existing local/offline v1 semantics below remain reproducible. Runtime implementation conformance is pending; the amendment does not authorize source acquisition or production deployment.

Kf source amendment — 2026-09-16

Status: accepted 2026-09-16 after two independent reviews; implementation conformance verified on forest, 2026-09-16. The Kf source/runtime contract is the controlling amendment for new production M1.

Run M1 prepares absent Kf through the existing worker; new-run readiness and freshness no longer depend on RUSLE or POLARIS. Earlier conflicting requirements remain the historical/legacy contract only once this checkpoint is accepted; do not reinterpret old accepted artifacts.

File-content currentness refinement (implemented)

The file dependency freshness contract refines ordinary source/artifact currentness: new snapshots add content hashes, metadata-only changes revalidate bytes without alone staling accepted results, and legacy hashless snapshots retain strict metadata comparison. It supersedes stat-only accepted-currentness wording above for this explicit boundary. Strict worker publication, source association, engine identity and SQLite/soil coherent snapshot obligations remain unchanged. The rationale is that WEPP preparation creates a hard link to an unchanged CLI and thereby changes its ctime. Hash revalidation must also detect equal-size writes whose mtime was restored.

Implementation pending checkpoint. The active CLI amendment in the file dependency freshness contract supersedes strict worker source equality only for unchanged active-CLI ctime drift proven by a fresh coherent uncached hash. It applies to M1/M3 admission, source preparation and pre/locked publication. All other source identities, owner/selection checks, SQLite coherence and result-artifact guards stay strict. This permits ordinary concurrent WEPP hard-link preparation without accepting changed climate bytes or inventing a hash for legacy records.

Failed migration mutation state

Migration's manual NoDb mutation scope follows the shared failed-mutation contract. On failure, held controller state must reflect the actual durable generation or refuse reuse until recovery. An explicit intermediate NoDb commit remains committed if later receipt/publication work fails. Migration evidence, admission leases, explicit resume, retained originals and artifact publication rules above remain unchanged. This prevents a later unrelated save from publishing failed migration state; it does not roll back migrated files.