WEPPcloud
UserSummary Docs

← Back to usersum index

Observed Controller

Observed NoDb controller for model-fit statistics between observed CSV time series and WEPP outputs. See also: ../../../../AGENTS.md and ../../../../wepppy/weppcloud/controllers_js/README.md#observed-controller-reference-2025-helper-migration

Overview

  • Parses observed CSV text (Date + measures) into a normalized observed.csv with year/month/day/julian fields.
  • Loads WEPP interchange outputs to compute daily/yearly model-fit statistics for hillslopes and channels.
  • Persists results in observed.nodb, writes comparison CSVs under <run>/observed/, and exposes report + CSV routes.

Data Flow

  1. parse_textdata(textdata) reads CSV, parses dates, and writes <run>/observed/observed.csv.
  2. calc_model_fit() loads hillslope + channel simulations and runs metrics:
    • Hillslopes: wepp/output/interchange/totalwatsed3.parquet (reused if present).
    • Channels: wepp/output/ebe_pw0.txt + wepp/output/chanwb.out.
  3. Outputs are written to <run>/observed/ and summarized in observed.nodb.

Outputs

  • observed/observed.csv (normalized input with Year, Month, Day, Julian).
  • observed/Hillslopes-<Measure>-Daily.csv, observed/Channels-<Measure>-Yearly.csv, etc.
  • observed.log (timing logs, status stream).
  • observed.nodb (results payload).

Observed Report + Graph (Specification)

  • Route: /runs/<runid>/<config>/report/observed/ with optional ?selected=<series>.
  • Default graph selection: Hillslopes-Streamflow_(mm)-Daily when available, otherwise the first CSV in <run>/observed/.
  • Report content order: summary tables first, graph second.
  • Graph data source: GET /runs/<runid>/<config>/resources/observed/<file> (client-side CSV fetch).
  • Graph CSV format: date,Modeled,Observed (date is YYYY-MM-DD for Daily, YYYY for Yearly).
  • Graph behavior: two line series (Simulated/Observed), main plot + brush context, no precipitation bars.
  • Error handling: if no comparison files exist, the report shows a message and skips the chart.

UI Wiring

  • Control template: wepppy/weppcloud/templates/controls/observed_pure.htm.
  • Controller: wepppy/weppcloud/controllers_js/observed.js.
    • On page load, the summary pane shows “View Model Fit Results” when observed.hasResults is true.
    • Status stream uses channel observed (controlBase.attach_status_stream).
  • Report: wepppy/weppcloud/templates/reports/wepp/observed.htm (tables + graph).

Endpoints

Route Method Purpose
/runs/<runid>/<config>/tasks/run_model_fit POST Parse CSV + run model fit (sync)
/runs/<runid>/<config>/report/observed/ GET Render observed summary report
/runs/<runid>/<config>/resources/observed/<file> GET Download CSV artifacts

Profiling Notes (Dec 2025)

Measured on test run /wc1/runs/un/unpresidential-shabbiness with tests/data/observed/CedarRv_WA.csv:

  • Before refactor (rebuilding interchange): parse_textdata ~5s, calc_model_fit ~75s (hillslope load ~72s).
  • After reuse of totalwatsed3.parquet + parallel stats: parse_textdata ~5s, calc_model_fit ~2.2s. Times depend on cache state and machine I/O; use observed.log for per-step timings.

Test Data & Reference Run

  • Observed CSV fixture: tests/data/observed/CedarRv_WA.csv.
  • Reference run: https://wc.bearhive.duckdns.org/weppcloud/runs/unpresidential-shabbiness/disturbed9002/
    • Run directory: /wc1/runs/un/unpresidential-shabbiness
    • Config: disturbed9002
  • Note: running the observed regression test overwrites <run>/observed/ outputs and updates observed.nodb.

Tests

wctl run-pytest tests/nodb/mods/test_observed_processing.py
wctl run-pytest tests/weppcloud/routes/test_observed_bp.py
wctl run-npm test -- observed

Development Guidance

  • Report payload assembly lives in wepppy/weppcloud/routes/nodb_api/observed_bp.py and should stay lightweight; chart data is fetched client-side.
  • Graph rendering lives in wepppy/weppcloud/templates/reports/wepp/observed.htm and uses the local static/js/d3.js bundle.
  • If you add new measures, ensure Observed._write_measure outputs Modeled + Observed columns with date so the graph can render without backend changes.
  • The selected query param should always map to a CSV in <run>/observed/; guard against missing files and fall back to the first available series.

Implementation Notes

  • parse_textdata uses pandas with the pyarrow CSV engine and in-reader date parsing.
  • calc_model_fit runs hillslope stats in parallel with channel load + stats.
  • The observed model fit remains synchronous (not RQ); large runs can still block the request thread.