WEPPcloud

← Back to help

ADR-0149: Parquet CSV integration resource envelope

Status: Accepted; local runtime acceptance passed, deployment pending Date: 2026-11-18

Context

The standalone envelope did not account for HTTP proxy first-byte deadlines or worker death. wepp1 has eight Uvicorn workers, three-minute culvert/batch Caddy read timeouts, and a Gunicorn heartbeat timeout of 120 seconds. The real corpus fits existing byte/structural ceilings; standalone Rust conversion measured less than 20 seconds on its largest evaluated input.

Decision

Use code-owned staging 30 seconds, child wall 120 seconds, CPU 100 seconds and serving 600 seconds. Keep two shared slots, 512 MiB input, 2 GiB output, 1 GiB child address space and all standalone structural ceilings. Filtered CSV matches are capped at 2,000,000 rows, preserving the existing default. These are resource budgets, not scientific parameter changes. No environment override is available.

Inherit the flock descriptor into the child; install a child-owned wall alarm and race-checked parent-death SIGKILL. Reclaim fixed per-slot directories only while holding their persistent lock. Keep slots through serving completion.

Decision provenance

Venue: operator/Codex session, 2026-11-18 UTC. Operator Roger Lew authorized execution, immediate cutover, fixed configuration and no fallback. Codex selected these bounded integration parameters from the observed proxy budget and accepted standalone envelope; independent review and exact-image acceptance are required.

Rationale and alternatives

The staging/conversion total of 150 seconds leaves 30 seconds of proxy margin. A heartbeat timeout alone cannot bound a single async request. An inherited lock plus fixed directories resolves parent-death accounting without a daemon or PID scavenging. Two slots bound staged/output payload at 5 GiB per shared root and child address space at 2 GiB. Separate replicas add separate budgets. A finite serving deadline prevents slow readers retaining both slots indefinitely.

Consequences and validation

Requests exceeding these budgets fail explicitly. Prove actual request/worker responsiveness, cross-process admission, parent death, cancellation and slow-reader cleanup with the real image. Do not claim a CPU limit bounds an I/O-blocked orphan; its inherited lock remains held until exit. The retained compatibility inventory and focused resource tests must establish valid corpus conformance before cutover. Whole-release rollback uses the canonical deployment workflow; no runtime fallback.

Evidence

Integration contract and active package.

Dictionary compatibility amendment (2026-11-18)

Implementation corpus readback found legitimate flowpath and road-event exports rejected by the standalone100,000 dictionary-entry ceiling. A bounded scan of all inventoried dictionary headers found a maximum125,855 entries; see dictionary evidence. The integrated limit becomes131,072 entries. This is the smallest conventional power-of-two bound above observed use, not an unbounded decoder setting.

All8 MiB page limits,4 GiB cumulative decode,1 GiB address space, CPU/wall limits and two-slot accounting remain unchanged. The additional32,000-ish entries add bounded dictionary bookkeeping within those same process/page ceilings. Require exact-boundary and limit-plus-one tests plus real flowpath/event byte readback. Do not silently route these valid files back to Python. Independent contract amendment review and a documentation ancestor precede changing the source limit.

Observed column-width amendment (2026-11-18)

Three inventoried totalwatsed3.parquet files contain 79 physical columns; the standalone limit of 64 rejects existing exports. Permit 128 physical columns and 129 flat schema elements (including the root). Preserve every other structural, page, cumulative decode, process and service limit. Inputs are at most 1,127,082 bytes and 5,844 rows, with footer sizes below 53 KiB.

This is a bounded compatibility correction, not an open schema budget. Retain exact-cap/plus-one tests and real-file byte parity. Both independent reviewers verify the inventory and approve before the source threshold edit.