← Back to usersum index

Mini Work Package: rq/api migration to rq-engine (staged deprecation)

Status: Complete (Phase 5 validation) Last Updated: 2026-01-12 Primary Areas: wepppy/microservices/rq_engine/*, wepppy/weppcloud/controllers_js/*, wepppy/weppcloud/controllers_js/utils.js, wepppy/weppcloud/controllers_js/http.js, wepppy/weppcloud/routes/rq/job_dashboard/*

Objective

Move all run-scoped RQ endpoints from Flask (/weppcloud/rq/api/*) to rq-engine (/rq-engine/api/*) and remove the Flask routes after dev validation. Complete the migration in dev before prod.

Current State Review

• rq-engine JWT auth is in place (Phase 6 JWT mini work package complete).
• Flask /rq/api/* and /upload/* routes are removed in dev.
• rq-engine owns the run-scoped enqueue routes, culvert batch routes, and cancel job endpoint.
• Controllers now target /rq-engine/api/* for queue triggers and polling.

Scope

• Confirm rq-engine parity for all run-scoped enqueue endpoints.
• Verify controller calls use the rq-engine base path.
• Keep /rq/api/* and /upload/* removed after dev validation.
• Ensure rq-engine routes stay kebab-case and callers align.
• Verify jobstatus/jobinfo remain read-only and open (as agreed).

Non-goals

• Changing payload schemas or response contracts beyond endpoint location.
• Reworking auth beyond existing JWT/session behavior.

Plan

Phase 1 - Inventory + parity map (completed)

• Enumerate all Flask /rq/api/* routes and map to rq-engine equivalents.
• Identify endpoints still missing in rq-engine (likely run-scoped job enqueues).
• Decide which endpoints must remain Flask-only (if any).

Flask /rq/api/* inventory (legacy; removed in dev)

Flask /upload/* inventory (legacy; removed in dev)

rq-engine routes already present

Phase 2 - rq-engine endpoint parity (completed)

• Implement missing rq-engine endpoints with identical request/response payloads.
• Reuse existing RQ job enqueuing helpers where possible.
• Ensure auth aligns with the JWT model (scopes, run claims, denylist).

Parity plan (proposed rq-engine route modules)

Implementation notes

• Prefer moving shared logic into wepppy/rq/* helpers or new shared modules to avoid importing Flask handlers into rq-engine.
• Keep routers in separate files and include them in wepppy/microservices/rq_engine/__init__.py.
• Use kebab-case for all new rq-engine route paths and update controller call sites (no new underscores).
• Batch Runner run submissions now require an admin JWT; the manage page injects a rqEngineToken bootstrap value for the controller.

Phase 2.5 - Upload proxy migration (completed)

• Port upload routes to rq-engine under /rq-engine/api/* with identical payloads.
• Update controller calls to use /rq-engine/api/* upload endpoints (drop /upload usage).
• Drop the /upload/* proxy entirely (Caddy rule removed).

Upload parity plan (rq-engine route groups)

Upload implementation notes

• Extract request-agnostic helpers from Flask upload handlers (avoid importing Flask view functions into rq-engine).
• Replace Flask request.files usage with FastAPI UploadFile handling.
• Keep timeout behavior on /rq-engine* (already long in Caddy); /upload* routing is removed.

Phase 3 - Controller migration (completed)

• Add a run-scoped helper that targets rq-engine (e.g., url_for_rq_engine_run()).
• Update controllers that call rq/api/* to use the new helper.

Phase 4 - Remove Flask /rq/api/* and /upload/* (completed)

• Delete /rq/api/* and /upload/* routes once rq-engine parity + controller migration are validated in dev.
• Remove any route imports/registrations so hot reload startup stays clean.
• Update any docs/tests that still reference /rq/api/* or /upload/*.

Phase 5 - Validation + cleanup plan (completed)

• Verify UI workflows: enqueue + polling + cancel + results display.
• Confirm jobstatus/jobinfo remain reachable without auth tokens.
• Add tests for rq-engine parity (including upload routes).
• Draft a follow-on removal checklist once prod is migrated.
• Manual smoke test completed in dev; schedule a fresh agent review before removing legacy routes in prod.
• Jobstatus/jobinfo spot check (2026-01-13): /rq-engine/api/jobstatus/4e8a87af-49c4-4eac-8385-dfc2a6843406 and /rq-engine/api/jobinfo/4e8a87af-49c4-4eac-8385-dfc2a6843406 returned status=finished for run brocaded-confection with no exc_info in children.

Follow-on removal checklist (post-prod migration)

• Confirm there are still no external clients relying on /rq/api/* or /upload/* before removing any compatibility shims.
• Remove legacy Flask upload endpoints once confirmed unused:
  • wepppy/weppcloud/routes/nodb_api/climate_bp.py (/runs/<runid>/<config>/tasks/upload_cli/)
  • wepppy/weppcloud/routes/nodb_api/disturbed_bp.py (/runs/<runid>/<config>/tasks/upload_sbs/, /runs/<runid>/<config>/tasks/upload_cover_transform)
  • wepppy/weppcloud/routes/huc_fire.py (/huc-fire/tasks/upload_sbs/)
  • wepppy/weppcloud/routes/batch_runner/batch_runner_bp.py (/batch/_/<batch_name>/upload-geojson, /batch/_/<batch_name>/upload-sbs-map)
• Retire legacy /rq/api replay fallbacks once old captures are archived (wepppy/profile_recorder/*).
• Sweep archival docs for /rq/api or /upload mentions if the historical notes are no longer needed.

Exit Criteria

• All UI controller calls use /rq-engine/api/* for rq endpoints.
• rq-engine covers the full /rq/api/* surface with matching payloads.
• Flask /rq/api/* routes removed and no longer imported.
• Upload routes are handled under /rq-engine/api/* and clients no longer rely on /upload.
• Dev environment validated; ready for prod deployment.

Risks / Notes

• Removing Flask endpoints is a hard cut; dev validation must cover all controllers.
• Ensure any new helper preserves url_for_run() semantics so run-scoped URLs stay correct.
• Upload endpoints must continue to honor long timeouts (covered by /rq-engine reverse proxy).
• Keep rq-engine routes organized in separate modules (avoid a single monolithic router file).
• Set SESSION_COOKIE_PATH=/ so rq-engine endpoints can read Flask session cookies.

Open Questions

• Confirm rq-engine base path for browser use: same-origin /rq-engine vs direct :8042.
• Should rq-engine be exposed outside the default /rq-engine reverse proxy path?