totalwatsed3 Aggregation
Fuses WEPP hillslope water, sediment, and ash-transport outputs into a single daily table served as
totalwatsed3.parquet.
Overview
totalwatsed3.parquet is produced by wepppy.wepp.interchange.totalwatsed3.run_totalwatsed3. The helper reads the per-hillslope interchange artifacts emitted by WEPP (H.pass.parquet, H.wat.parquet) and, when present, optional soil/element diagnostics (H.soil.parquet, H.element.parquet) plus WEPP–Ash Transport summaries under <run>/ash. The rows are grouped by simulation date (year, julian, sim_day_index, etc.) so that each record represents summed hillslope contributions for a given day, not routed watershed outlet flow:
- Most hydrologic volumes (m³) are summed from
H.wat, then converted back to depths (mm) via watershed area. Runoffdepth is computed fromH.passrunoff volume (runvol) over aggregated watershed area.- Sediment masses are derived from the pass file (
H.pass) and converted to both kg (seddep_*) and a dimensionless volumetric concentration (sed_vol_conc). - Optional ash metrics are joined by date and scaled using either supplied area hints or the Watershed controller (per-ash-type mass totals plus volumetric concentration/black fraction when bulk densities are known).
The resulting table is written to <run>/wepp/output/interchange/totalwatsed3.parquet using the schema defined in totalwatsed3.py.
Source Artifacts
| Artifact | Path | Purpose |
|---|---|---|
| Hillslope PASS | H.pass.parquet |
Event-scale runoff, detachment, and sediment concentration (sedcon_*) for every WEPP hillslope. |
| Hillslope WAT | H.wat.parquet |
Daily hydrology terms (Area, Q, Ep, …) per hillslope/OFE combination. |
| Hillslope SOIL (optional) | H.soil.parquet |
Daily soil profile state; TSMF is used for full-profile true moisture fraction when present. |
| Hillslope ELEMENT (optional) | H.element.parquet |
Daily event partitioning; QRain and QSnow are area-weighted into watershed-scale depths when present. |
| Ash Transport (optional) | <run>/ash/H{wepp_id}_ash.parquet |
Daily ash water/wind transport statistics per hillslope, merged when present. |
Only the PASS file includes per-particle sediment concentrations (sedcon_1-sedcon_5, kg m⁻³). Those values are multiplied by the per-event runoff volume (runvol, m³) to recover per-class sediment mass in kilograms. The water-balance terms come from H.wat where inputs are expressed as depths (mm); the aggregator multiplies each depth by the contributing area to recover volumes, performs any needed sums, and finally divides by area again to restore depths.
The required wepppyo3.wepp_interchange.totalwatsed3_to_parquet producer streams
projected batches of at most 8,192 rows. It retains date aggregates and joins for
one hillslope per worker, with at most twelve workers. Compact physical chunk
metadata replaces full column statistics after indexing; readers reconstruct at
most sixteen row groups at a time. Python resolves paths, options, and ash
controller metadata, and retains the existing Path return value. Missing native
support fails explicitly; there is no Python/DuckDB/pandas producer fallback.
The legacy pandas schema metadata is constructed from field types and installed
package versions without importing pandas or creating a DataFrame.
After publication, interchange README generation reads only the schema and three preview rows from each input. Reading entire tables here can exceed a 12 GiB worker even when native aggregation itself succeeds.
Output Snapshots
Key output columns (see SCHEMA in totalwatsed3.py for the complete list):
-
runvol,sbrunv,tdet,tdep: watershed totals (m³ or kg). -
seddep_1–seddep_5: total mass per sediment class (kg) computed asΣ sedcon_i * runvol. -
sed_del: total daily sediment delivery (kg), computed asΣ seddep_i. -
sed_vol_conc: watershed-wide volumetric sediment concentration (m³ of solids per m³ of runoff). -
Area,P,RM,Q,Dp,latqcc, … : volumetric sums fromH.wat, later converted to depths (Precipitation,Lateral Flow, etc.). -
SoilWaterTotal,ProfileDepth,ProfilePorosityCap,ProfileFCStore,ProfileWPStore: optional producer-authoritative storage/capacity terms fromH.watin mm. They are null for legacy WEPP executables that do not emit the additive columns. -
TSMF: area-weighted true soil moisture fraction fromH.soilwhen available (null when absent). -
QRain,QSnow: area-weighted runoff-partition depths fromH.elementwhen available (null when absent). -
MOFE aggregation rules (multi-OFE hillslopes): per-OFE columns in
H.watare not all physically summable across OFEs without producing a non-canonical total.totalwatsed3handles this by:latqcc: uses only the outlet-facing (last) OFE per hillslope/day to avoid counting internal lateral-routing transfers multiple times. The native producer tracks the maximum OFE for one hillslope at a time and contributes only that OFE to the daily lateral-flow sum.Runoff(the user-facing column): computed fromH.pass.runvolvolume divided by aggregated watershed area, not from summedQorQOFE.runvolis threshold-filtered delivery volume, not all physical runoff.QandQOFEaggregate columns (the literalSUM(col * 0.001 * Area)over OFEs): retained for diagnostic continuity but not the canonical hillslope totals. QOFE normalization is producer-version-dependent: older local normalization and current Q aliases both exist. The discrepancy depends on geometry, not a universal OFE-count factor. See Water and PASS variables and the current aggregation contract.
-
Ash transport columns when the ash directory is available:
- Totals and per-area masses:
wind_transport,water_transport,ash_transport,transportable_ash(+_per_ha). - Per-ash-type masses:
{wind,water,ash}_transport_{black,white}(+_per_havia per-type area). - Ash volumetrics:
ash_vol_conc(ash solids volume / runoff),sed+ash_vol_conc(sediment + ash solids volume / runoff),ash_black_pct_by_vol(% of ash solids volume that is black ash).
- Totals and per-area masses:
Nulls from missing PASS or ash rows are filled with zeros before the final Arrow table is materialized.
Storage Terms
Total-Soil Water in H.wat is WEPP's unfrozen profile-water term. It is not interchangeable with TSW in H.soil, which is a top-layer diagnostic. When available, prefer SoilWaterTotal for full-profile storage closure because WEPP emits it as watcon + frozwt.
The optional H.wat capacity terms are direct WEPP outputs in mm: ProfileDepth = solthk(nsl), ProfilePorosityCap = sum(por * dg), ProfileFCStore = sum(thetfc * dg), and ProfileWPStore = sum(thetdr * dg). totalwatsed3 area-weights these fields when present and preserves nulls when parsing legacy layouts.
Volumetric Concentration Details
WEPP Provenance
Within WEPP/WEPP-Forest the per-class sediment concentrations are derived directly from erosion physics:
sloss.for:305-316computessedcon(ipart,nplane)on the last OFE of every hillslope as:
sedcon = avsole / (runoff * efflen) * frcflw, resulting in kg m⁻³.wshimp.for:235-244performs the same calculation for impoundment routed flows: class-specific sediment masses (clout,slout, …) are divided byrunvolto yield kg m⁻³.
The WEPP particle system sets explicit specific gravities (unitless) for the five canonical classes inside prtcmp.for:122-126:
| Class | Description | Specific Gravity | Density (kg m⁻³) |
|---|---|---|---|
| 1 | Primary clay | 2.60 | 2 600 |
| 2 | Primary silt | 2.65 | 2 650 |
| 3 | Small aggregates | 1.80 | 1 800 |
| 4 | Large aggregates | 1.60 | 1 600 |
| 5 | Primary sand | 2.65 | 2 650 |
Specific gravity × 1 000 kg m⁻³ (density of water) yields the absolute particle density used to convert mass to volume.
Implementation in totalwatsed3
The required native producer implements the following steps for every grouped day:
- Aggregate class masses per day:
seddep_i = Σ sedcon_i * runvol. - Compute total sediment delivery:
sed_del = Σ_i seddep_i. - Convert each class to solids volume:
V_i = seddep_i / ρ_iwhereρ_iis the density from the table above. - Sum the solids volume:
V_total = Σ_i V_i. - Divide by total runoff:
sed_vol_conc = V_total / runvol_total(guarded so zero runoff yields zero concentration). - When ash is present, compute
ash_vol_concusing per-hillslope ash masses and bulk densities fromAsh.meta(defaulting to baked-in black/white values);sed+ash_vol_concadds sediment solids;ash_black_pct_by_volreports the black-ash share of ash solids volume.
This mirrors the physics in WEPP: the numerator reconstructs the actual cubic meters of sediment solids mobilized that day, while the denominator is the water volume those solids traveled with. The resulting value is dimensionless (m³ m⁻³) and represents the watershed-average volumetric sediment concentration for that day. totalwatsed3 stores it immediately after the seddep_* columns so future stakeholders can append per-class volumetric fractions alongside it.
Proof Sketch
- WEPP emits
sedcon_iin kg m⁻³ (sloss.for:305-316,wshimp.for:235-244). totalwatsed3multipliessedcon_ibyrunvol(m³) to obtain kg (seddep_i).- Dividing each
seddep_iby the class density (kg m⁻³) produces a solids volume (m³). - Summing those volumes and dividing by the aggregated
runvol(m³) cancels the units, giving m³ m⁻³, matching the physical definition of volumetric concentration.
Consequently sed_vol_conc is a direct, unit-consistent translation of the WEPP physics into the aggregated daily dataset, with specific-gravity metadata baked in so per-class volumetric columns can be added without re-litigating the derivation.
Physical runoff versus diagnostics
Named Runoff retains threshold-filtered PASS volume over watershed area.
Raw Q and QOFE columns retain their all-OFE local-area volume sums and are
explicitly labeled internal/version-sensitive diagnostics; neither certifies
physical hillslope outlet runoff or whole-system closure. Physical WAT consumers
use terminal Q separately. Sparse-day TotalWatSed lateral behavior remains a
legacy limitation. AshPost subtracts the PASS contribution already present in
this baseline before adding modeled ash runoff; it verifies numerical agreement
and source/model provenance before publishing corrected results.