WEPPcloud
UserSummary Docs

← Back to usersum index

Dedicated Download Service for Critical Run Artifacts

Status: Open (2026-06-19) - implementation complete; production rollout pending Timezone: UTC

Overview

Critical WEPPcloud archive downloads historically shared the browse service with interactive directory browsing, previews, parquet export, D-Tale handoff, and crawler-facing route families. This package separates exact archive delivery into a dedicated download service so large downloads can be isolated, range/resume friendly, and instrumented with request-level evidence. Production hosts still need cutover/smoke evidence before the remediation is considered fully closed.

The split does not remove shared NFS/run-root risk. It reduces other common-cause vectors so download reliability can be measured and hardened separately from browse UI pressure.

Objectives

  • Create a dedicated download microservice for exact run artifact delivery, starting with archive ZIP downloads at /weppcloud/runs/{runid}/{config}/download/archives/*.zip.
  • Preserve canonical browse authorization, public-run behavior, root-only path restrictions, and path traversal protections.
  • Support range/resume-friendly HTTP behavior for archive downloads, including HEAD, Accept-Ranges, 206 Partial Content, 416 Range Not Satisfiable, Content-Length, and Content-Range.
  • Add structured download logs with request id, run id, config, route family, sanitized path category, basename, file size, range start/end, bytes sent or best available byte count, duration, status, client abort/error reason when known, client address, and user agent.
  • Wire Docker and Caddy so the exact archive route can be cut over independently of browse, with its own health check, worker/process model, timeouts, and concurrency controls.
  • Add focused automated tests and an operator validation checklist that prove full, range, missing-file, unauthorized, traversal, and proxy-routed archive downloads behave correctly.

Scope

Included

  • New or extracted download service code under wepppy/microservices/, using the existing Starlette/gunicorn/uvicorn style already used by browse.
  • Shared auth/path resolution helpers only where needed to preserve current behavior without broad browse refactoring.
  • Caddy and Docker Compose changes in docker/docker-compose.dev.yml, docker/docker-compose.prod.yml, docker/docker-compose.prod.wepp1.yml, docker/caddy/Caddyfile, and docker/caddy/Caddyfile.wepp1.
  • Focused unit/integration tests for service behavior and route wiring.
  • Documentation updates for wepppy/microservices/browse/README.md, any new download service README, and package closeout artifacts.
  • Production rollout notes for wepp1, including a rollback path that returns archive routes to browse.

Explicitly Out of Scope

  • Removing NFS as a shared dependency or moving archives to object storage.
  • Rewriting the browse UI, D-Tale bridge, parquet preview/export implementation, or manifest listing system.
  • Migrating all /download/* families in one step. Culvert, batch, parquet-to-CSV, filtered CSV, and aria2c flows remain on browse until separately scoped.
  • Changing user-visible download URLs.
  • Adding new third-party serving frameworks unless the dependency evaluation standard is completed first.

Implementation Fidelity and Evidence

  • Fidelity target: faithful extraction for archive delivery semantics.
  • Authoritative source path(s): wepppy/microservices/browse/_download.py, wepppy/microservices/browse/auth.py, wepppy/microservices/browse/security.py, and docs/schemas/weppcloud-browse-auth-contract.md.
  • Cutover proof required: Docker/Caddy route evidence showing /weppcloud/runs/{runid}/{config}/download/archives/*.zip reaches the new service while other browse route families continue to reach browse:9009.
  • Acceptance evidence type: both generated-output and fixture-only evidence. Fixture tests prove edge cases; generated evidence must include local docker/Caddy curl probes for HEAD, full GET, and ranged GET.

Stakeholders

  • Primary: WEPPcloud operators and users downloading completed run archives.
  • Reviewers: WEPPcloud maintainers responsible for browse/download and production routing.
  • Security Reviewer: Required because the package changes externally reachable artifact delivery and proxy routing.
  • Informed: Incident responders tracking the June 2026 wepp1 browse/download slowdown.

Success Criteria

  • Archive ZIP downloads can be served by the dedicated service without changing the public URL.
  • HEAD, full GET, valid range GET, and invalid range GET return correct HTTP status and headers.
  • Auth, public-run allowance, traversal denial, missing-file, and non-archive route behavior are covered by tests.
  • Caddy routes exact archive download traffic to the new service in dev and production configuration, while browse/schema/dtale/files/gdalinfo and non-migrated downloads still route correctly.
  • Download logs include bytes, duration, status, range metadata, sanitized path category, client identity fields, and abort/error classification without logging secrets or raw tokens.
  • The service has independent health check, worker/process configuration, timeout posture, and documented rollback.
  • Work package security review has no unresolved high or medium findings before production rollout.
  • Work package QA review has no unresolved high, medium, or low findings before production rollout.
  • Local Caddy HEAD, full GET, range, sparse-resume, and service-log smoke evidence are captured for a 2.5 GB archive.
  • wepp1 production cutover and live archive HEAD/full/range smoke evidence are captured.

Parameterization ADR Gate

  • Parameterization change present: no
  • ADR required: no
  • ADR link(s): N/A
  • Decision provenance captured: yes

Reference: docs/standards/parameterization-adr-standard.md

Dependencies

Prerequisites

  • Current browse auth and path-boundary contracts remain canonical.
  • Docker/Caddy development stack can be used for route validation.
  • Operators can provide or confirm a representative archive ZIP under /wc1/runs/ for production smoke testing.

Blocks

  • Production cutover of critical run archive downloads to an isolated service.
  • Follow-up work to decide whether all non-transforming downloads should move out of browse.
  • Any later object-storage or non-NFS download architecture evaluation.
  • Depends on: docs/work-packages/20260616_browse_arrow_pandas_elimination/ for browse RSS remediation context.
  • Related: docs/work-packages/20260616_dtale_lazy_parquet_backend/ for D-Tale pandas memory isolation context.
  • Related: docs/infrastructure/incident-2026-06-16-wepp1-browse-download-slowdown.md for the incident record motivating this package.
  • Follow-up: A separate package may evaluate object storage or non-NFS archive publication if evidence shows NFS remains the dominant reliability limit.

Timeline Estimate

  • Expected duration: 2-4 focused sessions.
  • Complexity: Medium-High.
  • Risk level: High, because this changes production download routing and authorization-sensitive file delivery.

Security Impact and Review Gate

  • Security impact triage: high
  • Dedicated security review required: yes
  • Triage rationale: The package adds an externally reachable artifact-serving service, changes proxy routing, and reuses sensitive run-scoped authorization/path logic.
  • Security review artifact: docs/work-packages/20260619_dedicated_download_service/artifacts/20260619_security_review.md
  • QA review artifact: docs/work-packages/20260619_dedicated_download_service/artifacts/20260619_qa_review.md

Hardening and Callus Softening

  • Failure signature(s): Slow or timed-out archive downloads on wepp1; browse worker RSS growth previously observed around 45-49 GiB; Caddy/browser downloads lacking service-level byte/duration/abort evidence; mixed browse UI and long download load on the same worker pool.
  • Related prior hardening efforts: docs/work-packages/20260616_browse_arrow_pandas_elimination/, docs/work-packages/20260616_dtale_lazy_parquet_backend/, and docs/infrastructure/incident-2026-06-16-wepp1-browse-download-slowdown.md.
  • Health signals: Stable archive download throughput under browse UI/crawler load, low download worker RSS, clear 2xx/206/4xx/5xx/abort logs, and successful browser resume/range probes.
  • Danger signals: Duplicate auth logic drifting from browse, route shadowing that breaks non-archive downloads, logs containing tokens or sensitive full paths, or a new service that still lacks byte/duration evidence.
  • Observation window: 14 days after production cutover.
  • Temporary calluses introduced: Route-specific Caddy matcher and independent service knobs, owner WEPPcloud operators, review after the observation window.
  • Callus softening hypothesis: If the dedicated service proves stable and route matching is clean, archive-specific Caddy exceptions can be simplified or generalized only after tests and security review cover the broader route family.

References

  • wepppy/microservices/browse/README.md - Current browse service contract and dedicated download service boundary.
  • wepppy/microservices/browse/_download.py - Current download route implementation and archive serving behavior.
  • wepppy/microservices/browse/auth.py - Browse authorization helpers that must remain authoritative or be shared safely.
  • wepppy/microservices/browse/security.py - Path-boundary helpers for run-scoped files.
  • docs/schemas/weppcloud-browse-auth-contract.md - Canonical browse/download authorization contract.
  • docker/docker-compose.dev.yml - Development service wiring.
  • docker/docker-compose.prod.yml - Production service wiring.
  • docker/docker-compose.prod.wepp1.yml - wepp1 production override.
  • docker/caddy/Caddyfile - Default Caddy route wiring.
  • docker/caddy/Caddyfile.wepp1 - wepp1 Caddy route wiring.
  • docs/infrastructure/incident-2026-06-16-wepp1-browse-download-slowdown.md - Incident timeline and operational evidence.

Deliverables

  • Dedicated download service code and tests in wepppy/microservices/download/ and tests/microservices/test_dedicated_download_service.py.
  • Docker Compose and Caddy route updates in dev, prod, and wepp1 configuration files.
  • New service README at wepppy/microservices/download/README.md plus updated browse and microservices READMEs.
  • QA and security review artifacts with implementation findings resolved.
  • Local config validation for Compose and Caddy.
  • Production validation notes with full/range/HEAD smoke evidence remain pending until wepp1 cutover.

Follow-up Work

  • Decide whether other exact non-transforming downloads should migrate after archive cutover evidence.
  • Evaluate object storage, local archive cache, or other non-NFS approaches if NFS remains a dominant bottleneck.
  • Revisit bot/rate-limit policies only after critical download service telemetry shows request classes clearly.