← Back to usersum index

Tracker - NoDb Atomicity, RQ Graph Baseline, and Observability Follow-Ups

Living document tracking progress, decisions, risks, and verification for this work package.

Quick Status

Timezone: UTC
Started: 2026-04-25 23:06 UTC
Current phase: Closed (post-close regression addendum documented)
Last updated: 2026-04-26 09:19 UTC
Next milestone: None (package closed).
Security impact: low
Dedicated security review: no
Security artifact: N/A

Task Board

Ready / Backlog

• None.

In Progress

• None.

Blocked

• None.

Done

• Created package scaffold (package.md, tracker.md, prompt directories, active ExecPlan placeholder) (2026-04-25 23:06 UTC).
• Authored active ExecPlan and synced package into PROJECT_TRACKER.md Backlog (2026-04-25 23:09 UTC).
• Ran docs lint for package docs + PROJECT_TRACKER.md (2026-04-25 23:09 UTC).
• Milestone 1: Implemented scoped cross-controller failure-atomicity strategy for grouped rq-engine mutation flows with required review closure (2026-04-26 00:36 UTC).
• Milestone 2: Resolved wctl check-rq-graph drift baseline; artifacts regenerated and check clean with review closure (2026-04-26 00:43 UTC).
• Milestone 3: Hardened WEPP hint persistence boundary behavior (lock-contention + non-RuntimeError), re-reviewed, and closed with no remaining High/Medium findings (2026-04-26 01:08 UTC).
• Milestone 4: Added lock/dump-efficiency observability guard (test_rq_engine_lock_dump_efficiency_guard.py) plus remediation for landuse grouped-update failure-atomicity; closed with no remaining High/Medium findings (2026-04-26 01:34 UTC).
• Milestone 5: Completed scoped test maintainability cleanup (shared grouped test doubles + less brittle logger assertions) and closed with no remaining High/Medium findings (2026-04-26 01:50 UTC).
• Milestone 6: Ran package-wide validation and closure gates; docs and tracker updated; ExecPlan archived to completed (2026-04-26 01:50 UTC).
• Captured post-close _wepp_bin regression mechanics and fix rationale in package closure docs (tracker.md, completed ExecPlan, PROJECT_TRACKER.md) (2026-04-26 09:19 UTC).

Timeline

• 2026-04-25 23:06 UTC - Package created as follow-up to 20260425_nodb_lock_dump_efficiency_refactor to implement atomicity, queue-graph baseline cleanup, failure-path hardening, observability guards, and test maintainability cleanup.
• 2026-04-25 23:09 UTC - Package planning artifacts completed (active ExecPlan + tracker + project tracker backlog entry) and docs lint passed (0 errors, 0 warnings).
• 2026-04-25 23:16 UTC - Execution started; Milestone 1 implementation scope mapped to wepp_run_payload.py + scoped NoDb helper interfaces/tests and worker delegation launched.
• 2026-04-25 23:18 UTC - Milestone 1 worker dispatched with bounded write scope (atomic soils/watershed update strategy + targeted regressions + milestone validation command).
• 2026-04-25 23:26 UTC - Milestone 1 targeted validation passed for scoped suites (82 passed).
• 2026-04-25 23:27 UTC - Milestone 1 independent review triad launched in parallel (reviewer, qa_reviewer, security_reviewer) for findings/disposition before Milestone 2.
• 2026-04-25 23:36 UTC - Review triad completed. Findings include 2 High and multiple Medium items; remediation required before Milestone 1 closure.
• 2026-04-25 23:57 UTC - Milestone 1 remediation patch validated (99 passed) and closure re-review triad launched.
• 2026-04-26 00:00 UTC - Closure re-review surfaced remaining Medium findings (single-controller over-locking, grouped post-dump parity, conflict message sanitization, and cascading observability assertion depth); final remediation required before Milestone 1 close.
• 2026-04-26 00:10 UTC - Remediation patch validated (105 passed) with lock-scope narrowing, post-finalize parity, conflict-message sanitization, and expanded observability assertions.
• 2026-04-26 00:18 UTC - Closure re-review indicates one remaining Medium: TOCTOU gap between grouped lock preflight and grouped commit lock acquisition can still allow partial WEPP persistence on conflict path.
• 2026-04-26 00:30 UTC - TOCTOU remediation landed: grouped locks now held across parse + grouped commit path; partial-lock acquisition cleanup regression added.
• 2026-04-26 00:36 UTC - Milestone 1 closure triad confirms no remaining High/Medium findings (reviewer, qa_reviewer, security_reviewer); Milestone 2 started.
• 2026-04-26 00:39 UTC - Final post-TOCTOU spot-check triad reconfirmed no remaining High/Medium findings; Milestone 1 status frozen as complete.
• 2026-04-26 00:42 UTC - Ran wctl check-rq-graph; drift detected, regenerated canonical graph/catalog artifacts via python tools/check_rq_dependency_graph.py --write, then revalidated clean check.
• 2026-04-26 00:43 UTC - Milestone 2 review triad reported no High/Medium findings; root-cause documented as line-number metadata drift only (no dependency-edge change).
• 2026-04-26 00:53 UTC - Milestone 3 targeted validation run passed (wctl run-pytest tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1: 89 passed).
• 2026-04-26 01:02 UTC - Milestone 3 first review triad found Medium findings (lock-conflict log redaction, brittle log assertions, 409 metadata drift); remediation worker dispatched and integrated.
• 2026-04-26 01:08 UTC - Milestone 3 closure triad returned no High/Medium findings; Milestone 3 marked complete.
• 2026-04-26 01:14 UTC - Milestone 4 worker added tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py and targeted guard validation passed (4 passed).
• 2026-04-26 01:18 UTC - Milestone 4 validation sweep passed (102 passed) across guard + watershed + landuse + upload-batch microservice suites.
• 2026-04-26 01:24 UTC - Milestone 4 review found Medium findings (landuse grouped-update failure atomicity gap, AST setattr bypass). Remediation landed in wepppy/nodb/core/landuse.py + guard tests and passed targeted validation (66 passed).
• 2026-04-26 01:34 UTC - Milestone 4 closure triad returned no High/Medium findings; Milestone 4 marked complete.
• 2026-04-26 01:40 UTC - Milestone 5 worker extracted shared grouped doubles (tests/microservices/_wepp_payload_doubles.py) and reduced brittle logger assertions in scoped WEPP/bootstrap route tests.
• 2026-04-26 01:44 UTC - Milestone 5 scoped validation passed (103 passed) across WEPP/bootstrap/guard/landuse grouped suites.
• 2026-04-26 01:47 UTC - Milestone 5 QA review flagged Medium changed-file broad-exception gate failure; resolved by documenting deliberate boundaries in docs/standards/broad-exception-boundary-allowlist.md.
• 2026-04-26 01:48 UTC - python3 tools/check_broad_exceptions.py --enforce-changed --base-ref origin/master re-run passed (Net delta +0).
• 2026-04-26 01:49 UTC - Milestone 6 package-wide validation passed: wctl run-pytest ... across 9 scoped suites (228 passed).
• 2026-04-26 01:50 UTC - Queue graph drift reappeared as line-number churn after route edits; regenerated via python tools/check_rq_dependency_graph.py --write, then wctl check-rq-graph passed clean.
• 2026-04-26 01:55 UTC - Closure docs finalized: package moved to Done in PROJECT_TRACKER.md with WIP metadata refresh; final doc-lint passed (5 files validated, 0 errors, 0 warnings).
• 2026-04-26 02:14 UTC - Post-closure follow-up review integration: simplified grouped-update handoff branch in apply_wepp_run_payload to reduce dead-branch confusion while preserving lock-transfer/fallback semantics; revalidation queued.
• 2026-04-26 07:28 UTC - Same-signature NoDb rewrite hardening landed in wepppy/nodb/base.py (ef22c188c); later incident showed Redis mirror payload/signature behavior still allowed stale cache state to overwrite selected WEPP exec.
• 2026-04-26 08:55 UTC - Regression fix landed in wepppy/nodb/base.py (ada260d79): cache/file signature parity validation for Redis cache hits (getInstance + load_detached) and post-write cache mirror re-encode after final signature updates.
• 2026-04-26 09:16 UTC - Post-close regression mechanics documented in this tracker, completed ExecPlan, and PROJECT_TRACKER.md.
• 2026-04-26 09:19 UTC - Package doc-lint revalidated after addendum updates (4 files validated, 0 errors, 0 warnings).

Decisions Log

2026-04-25 23:06 UTC: Create a new follow-up package instead of reopening the closed package

Context: Prior package is already closed with complete closure evidence; new work was identified as follow-up scope.

Options considered:

1. Reopen/extend 20260425_nodb_lock_dump_efficiency_refactor.
2. Create a new package with explicit dependency on the closed package.

Decision: Option 2.

Impact: Preserves clean closure history for the completed package and keeps new implementation/review evidence isolated and auditable.

2026-04-25 23:06 UTC: Execute follow-up items in risk-first order

Context: Requested follow-ups include transaction semantics, boundary hardening, baseline hygiene, guardrails, and maintainability work.

Options considered:

1. Start with low-risk maintainability cleanup.
2. Start with atomicity and boundary correctness first, then observability and maintainability.

Decision: Option 2.

Impact: Reduces correctness/regression risk early and lets guardrails/cleanup reflect the final behavior contract.

2026-04-26 00:30 UTC: Remove grouped lock preflight/reacquire TOCTOU gap

Context: Closure re-review found grouped lock preflight could pass, then lock be reacquired by another actor before grouped commit lock acquisition, allowing WEPP-side persistence before 409 conflict.

Options considered:

1. Keep preflight and accept residual risk.
2. Move grouped updates ahead of WEPP parse.
3. Acquire grouped locks once and hold through parse + grouped commit path.

Decision: Option 3.

Impact: Prevents grouped lock-conflict responses from occurring after WEPP parse/persist due to preflight release/reacquire race while preserving grouped rollback semantics.

2026-04-26 00:30 UTC: Sanitize lock-conflict client messages

Context: Route handlers were returning raw str(NoDbAlreadyLockedError) in 409 responses, which can expose lock owner/token details.

Options considered:

1. Keep raw lock error text.
2. Return fixed sanitized message and log detailed exception server-side.

Decision: Option 2.

Impact: Preserves operator diagnostics in logs while removing internal lock metadata from client-visible payloads.

2026-04-26 01:14 UTC: Use AST guard tests for lock/dump-efficiency regression detection

Context: Milestone 4 required an explicit observability guard for scoped routes/helpers to prevent reintroduction of direct setter churn.

Options considered:

1. Rely only on existing runtime route tests.
2. Add a lightweight static AST guard focused on grouped-helper usage and forbidden direct setter writes.

Decision: Option 2.

Impact: Adds a fast, deterministic guardrail that fails quickly on structural regression patterns; runtime suites still cover behavioral semantics.

2026-04-26 01:47 UTC: Resolve changed-file broad-exception gate via allowlist boundary records

Context: check_broad_exceptions --enforce-changed failed on deliberate boundary catches introduced/retained in scoped rq-engine and nodb hardening paths.

Options considered:

1. Narrow/remove boundary catches and risk contract drift for cleanup/rollback paths.
2. Keep deliberate boundary catches and register exact file/line allowlist records with rationale + expiry.

Decision: Option 2.

Impact: Preserves explicit boundary behavior while restoring enforcement-gate cleanliness (Net delta +0) and auditability.

2026-04-26 08:55 UTC: Treat Redis NoDb cache as a strict signature mirror of on-disk state

Context: Production regression showed wepp.nodb could briefly hold the selected _wepp_bin value and then revert to an older value before hillslope execution.

Options considered:

1. Keep Redis cache acceptance permissive and rely on lock discipline alone.
2. Accept Redis cache entries only when cached _nodb_mtime/_nodb_size match current file signature and ensure cache writes occur after final post-write signature updates.

Decision: Option 2.

Impact: Prevents stale cache payloads from being rehydrated and re-dumped over newer disk state (observed as _wepp_bin rollback), while preserving cache best-effort behavior on Redis failures.

Post-Close Regression Mechanics Addendum (_wepp_bin reversion)

Symptom

• During Run WEPP, wepp.nodb persisted the selected WEPP binary (for example wepp_260425) and then reverted to a prior value (wepp_260421b) before worker-stage hillslope execution.

Root Cause Chain

1. ef22c188c introduced forced mtime advancement for same-size NoDb rewrites.
2. dump() still mirrored Redis cache from a pre-advance serialized payload (js), so cached _nodb_mtime/_nodb_size could lag the final file signature.
3. Redis cache hydration paths could accept that stale payload and skip disk rehydrate.
4. A later dump from the stale hydrated instance overwrote newer on-disk fields, including _wepp_bin.

Remediation Mechanics

1. Added _cache_instance_matches_file_signature(...) and applied it to Redis cache reads in both getInstance and load_detached.
2. Reordered disk-hydrate cache population so cached signatures always reflect stat-backed post-load state.
3. Changed dump() cache mirror writes to re-encode from the final in-memory object after mtime/size updates.
4. Added focused regressions in tests/nodb/test_base_boundary_characterization.py for stale-cache rejection and cache-signature parity after forced mtime-advance rewrites.

Residual Risk

• Low: signature equality depends on filesystem timestamp precision; the new epsilon-based signature check and same-size rewrite regressions cover the known boundary path.

Risks and Issues

Hardening Signal Log (Required for incident/remediation packages)

• Applicability: Preventive hardening + post-close callus-softening follow-up.
• Baseline health signals:
  • prior package left low residual risks for cross-controller atomicity and hint-persist boundary behavior.
  • wctl check-rq-graph currently reports drift.
• Post-change health signals:
  • grouped update failure paths restore state and avoid partial persisted side effects in scoped flows.
  • lock-conflict and hint-persist post-enqueue boundaries preserve response contracts while redacting client-visible lock metadata.
  • observability guard test and scoped runtime suites remain green across package closure validation.
• Danger signals observed:
  • queue-graph line-number drift can reappear after unrelated route line movement; canonical regeneration remains required at closure.
• Temporary callus register: none yet.
• Softening experiments: to be captured if exception-boundary behavior changes.

Verification Checklist

Code Quality

• Targeted rq-engine and NoDb suites pass for each milestone.
• wctl check-rq-graph is green after queue-graph baseline fix.
• No broad regressions in touched modules.

Security

• Security impact triage recorded (low) with rationale.
• Dedicated security artifact not required.
• Residual security-sensitive changes documented at closure.

Documentation

• Package docs remain current during execution.
• PROJECT_TRACKER.md status remains synchronized.
• ExecPlan moved from prompts/active/ to prompts/completed/ at closure.

Testing

• Atomicity/failure-path regressions added and passing.
• Observability guard validation added and passing.
• Maintainability cleanup keeps route/helper behavior assertions intact.

Deployment

• No deployment change required (or explicitly documented if needed).

Progress Notes

2026-04-25 23:06 UTC: Package creation and planning initialization

Agent/Contributor: Codex

Work completed:

• Created follow-up package scaffold and wrote initial package.md scope for requested items 1, 2, 3, 4, 6.
• Initialized tracker with ordered milestones and initial risk register.
• Prepared to author active ExecPlan and update root PROJECT_TRACKER.md.

Blockers encountered:

• None.

Next steps:

• Author active ExecPlan with milestone-level implementation/validation plan.
• Add package entry to PROJECT_TRACKER.md Backlog.
• Run doc lint on newly created package docs.

Test results: N/A (docs-only setup).

2026-04-25 23:09 UTC: Package preparation complete

Agent/Contributor: Codex

Work completed:

• Authored active ExecPlan with milestone order for requested items 1,2,3,4,6.
• Added package entry under PROJECT_TRACKER.md Backlog with dependencies and next steps.
• Linted docs:
  • package.md
  • tracker.md
  • active ExecPlan
  • PROJECT_TRACKER.md

Blockers encountered:

• None.

Next steps:

• Begin Milestone 1 execution (cross-controller atomicity strategy and implementation).

Test results:

• wctl doc-lint ... -> 4 files validated, 0 errors, 0 warnings.

2026-04-25 23:18 UTC: Milestone 1 delegation kickoff

Agent/Contributor: Codex (orchestrator) + worker subagent

Work completed:

• Completed Milestone 1 code-path analysis in:
  • wepppy/microservices/rq_engine/wepp_run_payload.py
  • wepppy/nodb/core/soils.py
  • wepppy/nodb/core/watershed.py
• Identified scoped failure-atomicity gap: sequential grouped helper persistence can still leave partial state when later controller update/commit fails.
• Dispatched bounded worker implementation for atomic multi-controller update flow and targeted regression coverage.

Worker write scope:

• wepppy/microservices/rq_engine/wepp_run_payload.py
• wepppy/nodb/core/soils.py
• wepppy/nodb/core/watershed.py
• tests/microservices/test_rq_engine_wepp_routes.py
• tests/microservices/test_rq_engine_bootstrap_routes.py
• tests/nodb/test_wepp_run_payload_grouped_updates.py

Blockers encountered:

• None.

Next steps:

• Integrate Milestone 1 patch.
• Run orchestrator re-validation.
• Dispatch milestone review triad (reviewer, qa_reviewer, security_reviewer) and disposition findings.

Test results: Pending worker completion.

2026-04-25 23:27 UTC: Milestone 1 targeted validation + review kickoff

Agent/Contributor: Codex (orchestrator)

Work completed:

• Ran milestone-targeted validation after worker integration:
  • wctl run-pytest tests/nodb/test_wepp_run_payload_grouped_updates.py tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1
• Dispatched required independent review triad in parallel:
  • reviewer (correctness/regression),
  • qa_reviewer (test quality/coverage/readability),
  • security_reviewer (boundary/error-safety).

Blockers encountered:

• None.

Next steps:

• Disposition review findings with severity.
• Fix any Medium/High findings before Milestone 1 closure.
• Update task board and proceed to Milestone 2.

Test results:

• Milestone-targeted suite: 82 passed, 0 failed, 6 warnings.

2026-04-25 23:36 UTC: Milestone 1 review findings disposition started

Agent/Contributor: Codex (orchestrator) + independent review agents

Review findings summary (pre-remediation):

• reviewer:
  • High: unlock failures swallowed on success path in grouped helper can hide lock integrity failure.
  • Medium: grouped path bypasses dump_and_unlock(validate=True) follow-up semantics (_post_dump_and_unlock/validation).
  • Medium: rollback controller tracking may miss late-failing dump().
• qa_reviewer:
  • Medium: route tests do not assert grouped commit persistence (dump) behavior.
  • Medium: rollback/unlock observability branches are untested.
  • Medium: real Soils/Watershed snapshot-stage-restore contracts not directly unit-tested.
• security_reviewer:
  • High: snapshot captured before lock acquisition allows stale rollback overwrite under interleaving concurrency.
  • Medium: unlock catch narrowed to RuntimeError can mask primary failure on other exception types.
  • Medium: lock contention (NoDbAlreadyLockedError) not mapped to canonical rq-engine payload in run-wepp setup path.

Disposition decision:

• All High/Medium findings are in-scope and will be fixed before Milestone 1 closure.
• Low findings (dummy helper duplication, assertion strictness refinements) will be addressed in Milestone 5 maintainability cleanup unless naturally resolved during remediation.

Next steps:

• Apply remediation patch for grouped helper correctness/boundary behavior + route contract mapping.
• Add targeted tests for identified boundary paths.
• Re-run Milestone 1 targeted validation and re-review.

2026-04-26 00:00 UTC: Milestone 1 closure re-review disposition (round 2)

Agent/Contributor: Codex (orchestrator) + review triad

Work completed:

• Integrated remediation patch and reran targeted suite:
  • wctl run-pytest tests/nodb/test_wepp_run_payload_grouped_updates.py tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1
  • Result: 101 passed, 0 failed.
• Ran closure re-review triad after remediation.

Remaining findings to fix before Milestone 1 closure:

• Medium: grouped helper currently locks both controllers even for single-controller updates (over-broad lock contention surface).
• Medium: grouped commit path still needs explicit dump_and_unlock post-step parity (_post_dump_and_unlock/validation semantics) for committed controllers.
• Medium: route lock-conflict responses return raw str(exc) and may leak internal lock-owner metadata; sanitize client-facing message while preserving logs.
• Medium (QA): cascading rollback+unlock failure test should assert observability log emissions for critical multi-failure path.

Disposition decision:

• All remaining Medium findings are in-scope and will be fixed before moving to Milestone 2.

2026-04-26 00:18 UTC: Milestone 1 closure re-review disposition (round 3)

Agent/Contributor: Codex (orchestrator) + review triad

Work completed:

• Validated latest remediation:
  • wctl run-pytest tests/nodb/test_wepp_run_payload_grouped_updates.py tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1
  • Result: 105 passed, 0 failed.
• Ran closure triad and dispositioned findings.

Remaining finding to fix before Milestone 1 closure:

• Medium: TOCTOU race remains between grouped lock preflight and grouped commit lock acquisition (wepp.parse_inputs can persist before second lock attempt), leaving potential partial persistence on 409 conflict paths.
• Related QA Medium: no direct test for preflight lock-conflict path.

Disposition decision:

• Remaining Medium findings are in-scope and will be fixed before Milestone 2.

2026-04-26 00:36 UTC: Milestone 1 closure complete

Agent/Contributor: Codex (orchestrator) + worker + review triad

Work completed:

• Implemented final TOCTOU remediation by removing preflight-only lock release/reacquire and holding grouped controller locks across:
  • WEPP parse + SWAT/reveg side effects,
  • grouped stage/dump/rollback,
  • grouped unlock + post-finalize hooks.
• Added direct regression for partial lock-acquisition cleanup (first lock acquired, second lock conflict, acquired lock released, no parse/persist side effects).
• Sanitized NoDb lock-conflict responses for WEPP/bootstrap payload-apply paths while preserving server-side logging context.
• Revalidated milestone-targeted scope:
  • wctl run-pytest tests/nodb/test_wepp_run_payload_grouped_updates.py tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1
  • Result: 107 passed, 0 failed, 6 warnings.

Review closure disposition:

• Final triad (reviewer, qa_reviewer, security_reviewer) reported no remaining High/Medium findings.
• Remaining notes are Low/non-blocking (test-double duplication, optional additional edge coverage).

Next steps:

• Begin Milestone 2 queue-graph drift root-cause analysis and baseline cleanup.

2026-04-26 00:43 UTC: Milestone 2 queue-graph baseline cleanup complete

Agent/Contributor: Codex (orchestrator) + review triad

Work completed:

• Ran queue-graph drift check:
  • wctl check-rq-graph -> drift detected for:
    • wepppy/rq/job-dependency-graph.static.json
    • wepppy/rq/job-dependencies-catalog.md
• Regenerated canonical artifacts:
  • python tools/check_rq_dependency_graph.py --write
• Revalidated clean baseline:
  • wctl check-rq-graph -> RQ dependency graph artifacts are up to date
• Reviewed artifact diff/root-cause:
  • line-number metadata churn only (no dependency-edge, queue, or enqueue-target contract changes).

Review closure disposition:

• Milestone 2 triad (reviewer, qa_reviewer, security_reviewer) reported no High/Medium findings.
• Non-blocking follow-up notes captured for maintainability only.

Next steps:

• Start Milestone 3 WEPP hint persistence boundary hardening.

2026-04-26 01:08 UTC: Milestone 3 boundary hardening complete

Agent/Contributor: Codex (orchestrator) + worker + review triad

Work completed:

• Hardened post-enqueue WEPP hint persistence paths:
  • explicit NoDbAlreadyLockedError boundary handling,
  • explicit unexpected-exception fail-open boundary logging,
  • preserved enqueue success response contract (200 + job_id) on post-enqueue hint persistence failures.
• Added sanitized payload-apply lock-conflict response helper for WEPP/bootstrap rq-engine endpoints (409 conflict with fixed client-safe message).
• Updated route response metadata text for affected 409 scenarios to include both single-flight and payload-input lock contention.
• Reduced brittle lock-contention log assertions and added redaction assertions for owner= / token= in warning args.

Files changed in milestone scope:

• wepppy/microservices/rq_engine/wepp_routes.py
• wepppy/microservices/rq_engine/bootstrap_routes.py
• tests/microservices/test_rq_engine_wepp_routes.py
• tests/microservices/test_rq_engine_bootstrap_routes.py

Validation results:

• wctl run-pytest tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1 -> 89 passed, 0 failed.

Review findings disposition:

• Fixed all reported Medium findings (log redaction, brittle assertions, 409 metadata drift).
• Remaining findings were Low and recorded as residual/non-blocking.

2026-04-26 01:34 UTC: Milestone 4 observability guard complete

Agent/Contributor: Codex (orchestrator) + worker + review triad

Work completed:

• Added scoped lock/dump-efficiency observability guard:
  • new AST guard suite tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py
  • checks grouped-helper usage and forbids direct setter churn in scoped rq-engine route/helper functions.
• Remediated guard-review findings:
  • added setattr(obj, "field", ...) detection to guard checks,
  • fixed landuse grouped update failure-atomicity (apply_set_landuse_mode_updates) with rollback snapshot/restore.
• Added landuse grouped-update regression tests for selection-failure rollback.

Files changed in milestone scope:

• tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py
• wepppy/nodb/core/landuse.py
• tests/nodb/test_landuse_grouped_updates.py

Validation results:

• wctl run-pytest tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py tests/microservices/test_rq_engine_watershed_routes.py tests/microservices/test_rq_engine_landuse_routes.py tests/microservices/test_rq_engine_upload_batch_runner_routes.py --maxfail=1 -> 102 passed, 0 failed.
• wctl run-pytest tests/nodb/test_landuse_grouped_updates.py tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py tests/microservices/test_rq_engine_landuse_routes.py --maxfail=1 -> 66 passed, 0 failed.

Review findings disposition:

• Fixed all reported Medium findings (landuse rollback atomicity gap and AST setattr bypass).
• Remaining findings were Low and captured as residual follow-up notes.

2026-04-26 01:50 UTC: Milestone 5 maintainability cleanup complete

Agent/Contributor: Codex (orchestrator) + worker + review triad

Work completed:

• Extracted shared grouped test doubles to reduce duplicated collaborator scaffolding:
  • new helper module tests/microservices/_wepp_payload_doubles.py.
• Updated WEPP/bootstrap route suites to use shared doubles.
• Replaced remaining brittle exact logger assertions with intent-level assertions.
• Strengthened landuse rollback snapshot handling for partially hydrated legacy state (getattr + missing-attribute rollback behavior).

Files changed in milestone scope:

• tests/microservices/_wepp_payload_doubles.py
• tests/microservices/test_rq_engine_wepp_routes.py
• tests/microservices/test_rq_engine_bootstrap_routes.py
• wepppy/nodb/core/landuse.py
• tests/nodb/test_landuse_grouped_updates.py

Validation results:

• wctl run-pytest tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py tests/nodb/test_landuse_grouped_updates.py --maxfail=1 -> 103 passed, 0 failed.
• python3 tools/check_broad_exceptions.py --enforce-changed --base-ref origin/master -> PASS (Net delta +0) after boundary allowlist updates.

Review findings disposition:

• Fixed all reported Medium findings (legacy missing-attribute snapshot path and broad-exception enforcement failure).
• Remaining findings were Low and documented as residual follow-up notes.

2026-04-26 01:50 UTC: Milestone 6 package-wide validation and closure readiness

Agent/Contributor: Codex (orchestrator)

Work completed:

• Ran package-wide scoped regression sweep covering all touched route/helper suites.
• Re-ran queue-graph and broad-exception enforcement gates.
• Queue graph drift reappeared as line-number churn after later route edits; regenerated artifacts and revalidated clean.

Validation results:

• wctl run-pytest tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py tests/microservices/test_rq_engine_lock_dump_efficiency_guard.py tests/microservices/test_rq_engine_watershed_routes.py tests/microservices/test_rq_engine_landuse_routes.py tests/microservices/test_rq_engine_upload_batch_runner_routes.py tests/nodb/test_wepp_run_payload_grouped_updates.py tests/nodb/test_wepp_job_hint_grouped_updates.py tests/nodb/test_landuse_grouped_updates.py --maxfail=1 -> 228 passed, 0 failed.
• python tools/check_rq_dependency_graph.py --write -> updated graph/catalog artifacts after line-churn drift.
• wctl check-rq-graph -> RQ dependency graph artifacts are up to date.
• python3 tools/check_broad_exceptions.py --enforce-changed --base-ref origin/master -> PASS.

2026-04-26 01:55 UTC: Closure docs synchronization finalized

Agent/Contributor: Codex (orchestrator)

Work completed:

• Updated PROJECT_TRACKER.md:
  • package moved from In Progress to Done,
  • top-level metadata refreshed (Last Updated, Active Packages, and WIP count),
  • added done-summary with closure validation + review disposition highlights.
• Confirmed ExecPlan archival path references remain aligned to prompts/completed/.

Validation results:

• wctl doc-lint --path docs/work-packages/20260425_nodb_atomicity_observability_followups_a/package.md --path docs/work-packages/20260425_nodb_atomicity_observability_followups_a/tracker.md --path docs/work-packages/20260425_nodb_atomicity_observability_followups_a/prompts/completed/nodb_atomicity_observability_followups_execplan.md --path PROJECT_TRACKER.md --path docs/standards/broad-exception-boundary-allowlist.md -> 5 files validated, 0 errors, 0 warnings.

2026-04-26 02:14 UTC: Post-review cleanup integration

Agent/Contributor: Codex (orchestrator)

Work completed:

• Incorporated follow-up external review feedback at low risk by simplifying the grouped-update call path in:
  • wepppy/microservices/rq_engine/wepp_run_payload.py
• Removed the nested branch that appeared unreachable in normal grouped-update flows and replaced it with a single call that:
  • transfers lock ownership when grouped locks were pre-acquired, and
  • preserves defensive fallback by allowing helper-side lock-order recompute when no pre-acquired grouped locks are present.

Validation results:

• wctl run-pytest tests/nodb/test_wepp_run_payload_grouped_updates.py tests/microservices/test_rq_engine_wepp_routes.py tests/microservices/test_rq_engine_bootstrap_routes.py --maxfail=1 -> 111 passed, 0 failed.

2026-04-26 09:19 UTC: Post-close regression mechanics documentation sync

Agent/Contributor: Codex (orchestrator)

Work completed:

• Added explicit post-close _wepp_bin rollback mechanics to package closure docs:
  • tracker.md (timeline, decision log, risk register, and dedicated mechanics addendum),
  • completed ExecPlan outcomes/surprises,
  • PROJECT_TRACKER.md done-summary addendum,
  • package.md closure-note pointer to tracker addendum.
• Captured root-cause chain and remediation mechanics from the production regression:
  • stale cache payload acceptance after signature drift,
  • post-write cache mirror serialization ordering,
  • signature parity enforcement and post-write re-encode fix.

Validation results:

• wctl doc-lint --path docs/work-packages/20260425_nodb_atomicity_observability_followups_a/package.md --path docs/work-packages/20260425_nodb_atomicity_observability_followups_a/tracker.md --path docs/work-packages/20260425_nodb_atomicity_observability_followups_a/prompts/completed/nodb_atomicity_observability_followups_execplan.md --path PROJECT_TRACKER.md -> 4 files validated, 0 errors, 0 warnings.

Communication Log

2026-04-25 23:06 UTC: Follow-up package request

Participants: User, Codex
Question/Topic: Prepare package for follow-ups 1, 2, 3, 4, 6 from prior closure recommendations.
Outcome: New follow-up package created with explicit scope and milestone plan initialized.

2026-04-26 09:19 UTC: Documentation mechanics request

Participants: User, Codex
Question/Topic: Ensure regression/fix mechanics are explicitly captured in package documentation.
Outcome: Added tracker addendum, ExecPlan outcome note, package closure pointer, and PROJECT_TRACKER.md done-summary update.