WEPPcloud
UserSummary Docs

← Back to usersum index

NoDb Lock/Dump Efficiency Refactor (RQ Engine)

Status: Closed (2026-04-25 21:55 UTC) Timezone: UTC

Overview

Wepp.parse_inputs() and the climate parser already use a single lock context per parse call, but several rq-engine mutation paths still perform sequential @nodb_setter writes. Because @nodb_setter wraps each write in with self.locked(), those paths create avoidable lock/dump churn, increase contention windows, and add unnecessary Redis/file persistence overhead.

This package refactors the highest-impact and secondary rq-engine candidates into single-lock mutation flows while preserving existing request contracts and behavior. Legacy Flask routes are explicitly excluded.

Objectives

  • Reduce lock/dump churn in the highest-impact rq-engine mutation paths identified during lock-efficiency discovery.
  • Refactor secondary rq-engine candidates where multi-field updates are currently persisted as multiple independent lock/dump cycles.
  • Preserve payload contracts, response contracts, and controller invariants while changing mutation orchestration.
  • Add regression coverage that demonstrates behavioral parity and validates single-lock mutation semantics for touched paths.
  • Document the lock/dump efficiency rule and applied patterns for future NoDb work.

Scope

This package targets rq-engine and NoDb facade mutation orchestration only.

Included

  • Highest-impact refactors:
    • wepppy/microservices/rq_engine/wepp_run_payload.py
    • wepppy/microservices/rq_engine/watershed_routes.py
    • wepppy/microservices/rq_engine/landuse_routes.py (build-landuse disturbed toggles + set-landuse-mode dual-field set)
  • Secondary refactors:
    • wepppy/microservices/rq_engine/upload_batch_runner_routes.py
    • wepppy/microservices/rq_engine/wepp_routes.py
    • wepppy/microservices/rq_engine/bootstrap_routes.py
  • Supporting controller/facade helper updates under wepppy/nodb/** as needed to enable single-lock mutation entrypoints.
  • Tests and docs updates required to prove parity and enforce the new mutation pattern.

Explicitly Out of Scope

  • Legacy Flask route refactors under wepppy/weppcloud/routes/nodb_api/**.
  • Business-rule changes to payload validation or run workflow semantics.
  • Queue topology changes, new job dependency edges, or RQ graph rewiring.
  • Redis connection strategy changes (handled in prior/parallel packages).

Stakeholders

  • Primary: NoDb maintainers, rq-engine maintainers, RQ operators.
  • Reviewers: code reviewer + QA reviewer for behavior/regression parity.
  • Security Reviewer: optional (security triage is low).
  • Informed: maintainers of packages touching adjacent lock/cache hardening work.

Success Criteria

  • Highest-impact paths are converted to single-lock mutation orchestration with no behavior regressions.
  • Secondary candidates are converted to single-lock mutation orchestration with no behavior regressions.
  • No legacy Flask files are changed.
  • Targeted regression tests are added/updated and passing.
  • wctl check-rq-graph result is documented with no queue-wiring edits in this package.
  • Documentation updated to capture the lock/dump efficiency rule and the preferred facade mutation pattern.

Dependencies

Prerequisites

  • Existing NoDb lock ownership and stale-write hardening baseline from:
    • docs/work-packages/20260424_rq_worker_nodb_cache_hardening/package.md
  • Current rq-engine route behavior and payload contracts in:
    • docs/schemas/rq-response-contract.md

Blocks

  • None.

Timeline Estimate

  • Expected duration: 2-4 focused sessions.
  • Complexity: Medium.
  • Risk level: Medium.

Security Impact and Review Gate

  • Security impact triage: low
  • Dedicated security review required: no
  • Triage rationale: changes are internal mutation-orchestration refactors that should not alter auth boundaries, route exposure, or privilege semantics.
  • Security review artifact: N/A

Hardening and Callus Softening (Required for incident/remediation packages)

  • Applicability: Not an incident-response package; this is preventive lock/dump efficiency and reliability hardening.
  • Failure signature(s): N/A (no single triggering incident).
  • Related prior hardening efforts: 20260424_rq_worker_nodb_cache_hardening.
  • Health signals: lower mutation-path lock churn, reduced persistence overhead, unchanged route behavior.
  • Danger signals: hidden contract changes, direct underscore writes outside lock discipline, coverage gaps.
  • Observation window: during package execution and post-merge CI/regression runs.
  • Temporary calluses introduced: none planned.
  • Callus softening hypothesis: N/A.

References

  • wepppy/nodb/base.py
  • wepppy/nodb/core/wepp.py
  • wepppy/nodb/core/climate_input_parser.py
  • wepppy/microservices/rq_engine/wepp_run_payload.py
  • wepppy/microservices/rq_engine/watershed_routes.py
  • wepppy/microservices/rq_engine/landuse_routes.py
  • wepppy/microservices/rq_engine/upload_batch_runner_routes.py
  • wepppy/microservices/rq_engine/wepp_routes.py
  • wepppy/microservices/rq_engine/bootstrap_routes.py
  • docs/standards/nodb-facade-collaborator-pattern.md
  • docs/work-packages/README.md
  • docs/prompt_templates/codex_exec_plans.md

Deliverables

  • Grouped single-lock mutation helper flows delivered for all scoped highest-impact and secondary candidates:
    • wepp_run_payload.py + Soils/Watershed grouped apply helpers.
    • watershed_routes.py + grouped build-subcatchment update helper.
    • landuse_routes.py + grouped Landuse and Disturbed update helpers.
    • upload_batch_runner_routes.py + grouped BatchRunner SBS helper.
    • wepp_routes.py + bootstrap_routes.py + grouped Wepp.persist_job_hint(...).
  • New/updated parity regressions in scoped microservice and NoDb suites, including direct grouped-helper branch coverage tests under tests/nodb/.
  • Package closure evidence in:
    • tracker.md (timeline, review dispositions, validation logs).
    • Completed ExecPlan under prompts/completed/.

Follow-up Work

  • Optional hardening follow-up: evaluate whether post-enqueue WEPP job-hint persistence should catch additional NoDb lock exceptions beyond RuntimeError while preserving response-contract stability.
  • Optional architectural follow-up: design broader cross-controller transaction semantics for failure-atomicity across multi-controller mutation sequences.

Closure Notes

  • All scoped backlog items were completed in required execution order with per-milestone review triads and Medium/High finding closure before milestone progression.
  • Final targeted package validation passed:
    • wctl run-pytest ... across 10 scoped suites (198 passed).
  • wctl check-rq-graph was executed; it reported drift in existing graph/catalog artifacts, but no queue-wiring files were edited in this package.
  • Scope discipline held: no changes under legacy Flask wepppy/weppcloud/routes/nodb_api/**.
  • Lock/dump efficiency rule captured and applied consistently: route-level multi-field updates now flow through explicit grouped helper entrypoints that execute one lock/dump cycle per controller mutation sequence.

Kickoff Prompt

  • Completed ExecPlan: docs/work-packages/20260425_nodb_lock_dump_efficiency_refactor/prompts/completed/nodb_lock_dump_efficiency_refactor_execplan.md