WEPPcloud

← Back to help

Sandboxed Shapefile Conversion Security Contract

Status: Rust CLI implementation locally validated; independent review and rollout pending Last Updated: 2026-11-14 Primary Areas: docker/docker-compose.dev.yml, docker/caddy/Caddyfile, wepppy/microservices/*, wepppy/all_your_base/geo/geo.py, wepppy/weppcloud/*

Contract Status and Conformance

This document is the normative security and behavior contract for the public shape-converter service. Closed work packages record implementation history but do not prove continuing conformance. A control is implemented only when the checked-in runtime path and regression evidence enforce it.

Resolution is tracked in docs/work-packages/20261113_shape_converter_security_conformance/ and GitHub issues #655 and #656. The operator selected a service-local Rust CLI to replace production GDAL/Fiona parsing on 2026-11-14. The previously qualified dedicated GDAL image remains a pre-cutover parity oracle only. The common WEPPpy image remains unchanged.

The 2026-11-14 independent remediation slice has the following source/local-test state; this table does not assert deployed behavior:

Boundary Current evidence Status
ZIP admission and extraction Size/member quotas, path containment, extension policy, NFC/case-fold duplicate detection, special-file rejection, and request-scoped extraction are implemented in archive_validation.py. Implemented. Absolute compressed, uncompressed, member, scratch, and parser-input quotas are the expansion policy; no ratio threshold is adopted.
Dataset selection select_single_shapefile_dataset(...) requires one complete shared-prefix .shp/.shx/.dbf set and rejects additional complete or incomplete prefixes. Implemented.
Format/parser identity Production conversion invokes only the service-local shape_parser_cli; the legacy Fiona worker is removed and the final candidate image contains no Fiona/GDAL files. Locally implemented and exact-image validated. The retained GDAL image is test/oracle evidence only.
Parser process boundary Convert launches shape_parser_cli in a new process group with a minimal environment, fixed exits, strict JSON IPC, timeout/kill, and OS resource limits. Implemented and locally validated for input, output, feature, vertex, file-descriptor, memory, and CPU exhaustion.
Runtime containment Compose enforces non-root, read-only root, dropped capabilities, resource limits, hardened tmpfs, and an internal network. Partial: the selected CLI shares the container UID, mounts, and network authority. It is a bounded failure boundary, not arbitrary-code containment; final acceptance of this residual is pending.
Semantic and adversarial validation Point, line, polygon, reversed ring, Z/null, MultiPatch, DBF primitive/null/date, malformed input, strict IPC, cleanup, timeout, and both output formats have direct tests/readback. Locally validated. Broader fuzz/property expansion remains defense-in-depth follow-up, not a production fallback.
Dependency qualification shapefile 0.9.0 and its 30-package locked graph have dependency, license, unsafe, RustSec, benchmark, and parity evidence. Locally accepted. Exact release image still requires independent security signoff and normal rollout authorization.

Until the nonconformant and partial items are closed or explicitly risk-accepted, the service must not be represented as fully conformant with this contract.

GeoParquet input normalization amendment (VUN-01)

Accepted contract intent 2026-11-16; implementation and exact-image conformance are pending. WEPPcloud's Roads, AgFields, Omni contrasts and Batch Runner may submit one admitted local GeoParquet file to this service for normalization to canonical WGS84 GeoJSON. The public converter UI remains ZIP/Shapefile-oriented. Culvert and D-Tale paths are excluded.

GeoParquet uses a separate service-local geoparquet_parser_cli; it is never a mode of shape_parser_cli. The exact accepted graph is Parquet 60.1.0 with default features disabled and Snappy only, plus serde_json 1.0.145, bound to the exact Cargo.lock SHA-256 ce923427941fabe153c8b282dcfddeb1168b60670055ab34f47911d039748f13. Direct use of an upstream Parquet decoder on unpreflighted hostile bytes is forbidden. Any graph, feature, version or lock-hash change reopens dependency review.

Before any upstream decoder runs, owned code must open one regular nonsymlink immutable local file and perform the complete bounded Compact-Thrift/footer, schema, row-group, column, page, dictionary, encoding, codec, offset and range preflight required by ADR-0145. Preflight and decode use that same descriptor, with identity and size rechecked before decode. The parser accepts no pathname selected by output metadata, URL, directory, dataset, external column file, encryption, plugin or network resource.

The admitted profile and canonical output are defined by the vector-upload normalization contract. The CLI emits strict bounded data-only JSON. The service revalidates its schema, feature count, finite coordinates, geometry types, observed bounds and hashes, resolves/reprojects CRS offline without ballpark fallback, serializes canonical GeoJSON, reads that artifact back and reconciles its declared metadata before success. The service computes and returns source_size and source_sha256 from its own complete staged request snapshot; receiving workflows must reconcile both against their descriptor-bound retained original before receiving validation or accepted-state mutation. Every receiving workflow then revalidates canonical GeoJSON before any accepted-resource or workflow-state mutation.

Reuse the accepted shape_parser_cli lifecycle: private request scratch, minimal environment, fixed argv without shell, process-group ownership, direct resource limits, wall timeout, cancellation shielding/drainage, bounded stdout/stderr/IPC, adjacent binary provenance, runtime canary and cleanup after every outcome. A missing, changed, unready, timed-out or invalid parser fails closed with the stable normalization errors; no Python, GDAL/Fiona, PyArrow, alternate CLI or in-process fallback is allowed. Exact-image qualification, SBOM/advisory scan and rollout remain separate gates.

Receiving workflows retain their original project uploads and provenance under the canonical normalization contract. Shape-converter receives no user account, ban authority or project-artifact pathname. Its request copy remains private ephemeral scratch and is removed after complete process-group drainage. A project-retained quarantined original is never a converter retry source; a retry must create a new receiving attempt and repeat all admission controls.

Objective

Provide a sandboxed upload/conversion service where users can upload a ZIP containing a shapefile and convert it to GeoJSON or GeoParquet, with explicit CRS handling options:

  • Same as shapefile
  • WGS84
  • UTM (zone determined the same way WEPPpy currently determines UTM from the upper-left extent)

The service must report projection and attribute schema in the UI, and must delete uploaded ZIPs, extracted shapefile sidecars, and generated artifacts after each request completes.

Why This Is Needed

Users currently struggle with local shapefile conversion workflows. A dedicated conversion service removes desktop GIS friction while centralizing security controls for high-risk ZIP and geospatial parser inputs.

Scope

In scope

  • New standalone containerized microservice for conversion.
  • Browser UI for upload, CRS option selection, and metadata reporting.
  • Output formats: GeoJSON, GeoParquet.
  • Strict upload validation, extraction controls, parser hardening, and cleanup lifecycle.
  • Comprehensive negative/security test coverage for ZIP and shapefile risks.

Out of scope

  • General-purpose GIS editing.
  • Multi-format archive support beyond ZIP.
  • Persistent dataset storage.
  • Long-running async job queue (v1 should be synchronous with strict timeouts).
  • WEPPcloud route/controller updates to consume new relay flow (separate scope).

User Experience

  1. User opens converter page and sees one Upload control panel.
  2. User selects .zip (containing exactly one logical shapefile dataset) in a single archive input.
  3. User can choose output format (GeoJSON/GeoParquet) and target CRS (same_as_shapefile/wgs84/utm_wepppy_upper_left) in that same panel.
  4. User clicks Inspect Upload (secondary action) to populate metadata panels:
    • Projection status (known / unknown / invalid) with detected CRS details when present.
    • Geometry summary (feature count, geometry types, bounds).
    • Attribute schema table (field name, type, width/precision, nullability note).
    • Warnings (lossy/null/geometry/CRS caveats).
  5. User clicks Convert And Download (primary action) to run conversion and download the artifact.
  6. inspect and convert remain independent request-scoped uploads by backend contract; the browser may reuse the same selected local file, but each action sends its own multipart request and no server-side cross-request staging is allowed.
  7. Service deletes all artifacts created by each request.

Functional Requirements

Upload and input requirements

  • Accept only multipart/form-data with one file field: archive.
  • Accept only .zip extension and ZIP signature/magic-byte validation.
  • Reject password-protected/encrypted ZIP entries.
  • Reject nested archives by policy in v1.
  • Require shapefile core sidecars by shared prefix: .shp, .shx, .dbf.
  • Require exactly one logical dataset: core sidecars must share one normalized directory and prefix; reject extra complete or incomplete shapefile prefixes.
  • Normalize path separators and Unicode before duplicate/case-collision checks.
  • Permit optional sidecars: .prj, .cpg, .sbn, .sbx, .qix.
  • Allow .shp.xml metadata sidecars as non-fatal input but immediately unlink them after extraction.
  • Allow .qmd metadata sidecars as non-fatal input but immediately unlink them after extraction.
  • Emit an explicit warning when .shp.xml is removed that it is generally not advisable to pack .shp.xml in shapefile ZIPs because metadata may leak sensitive details.
  • Reject other non-required XML sidecars (.xml, .gml) in v1.
  • Reject archives containing multiple shapefile prefixes unless user explicitly selects one (v1 default: reject).
  • Treat extensions and magic bytes as admission hints, not parser authorization. The parser must enable only the ESRI Shapefile driver, verify the opened driver is exactly ESRI Shapefile, and reject renamed VRT, HDF, or other formats.
  • inspect and convert are independent uploads by design; no server-side file staging across requests.

Access model (explicit)

  • This is a public unauthenticated service.
  • inspect and convert endpoints require no login, no JWT, and no session cookie.
  • Service must remain stateless with no per-user persistence.
  • Because there is no auth, abuse controls are mandatory:
    • Per-IP request rate limits and concurrent conversion caps.
    • Strict payload size, expansion, and runtime limits.
    • Fast-fail 429/503 behavior under saturation.
    • Trusted client identity rules:
      • Only proxy-set forwarding headers are trusted.
      • Edge strips inbound X-Forwarded-* from clients and sets canonical values.
      • Service trusts only configured proxy hops.
      • IPv6 rate-limit keys are aggregated to /64.

CRS behavior

  • same_as_shapefile:
    • Preserve source CRS if present.
    • If source CRS missing/unknown, keep coordinates as-is and mark projection as unknown in response/UI.
  • wgs84:
    • Reproject to EPSG:4326 (longitude, latitude axis order in outputs).
    • If source CRS missing/unknown, fail with explicit error requiring known CRS.
    • If source CRS is present but invalid (.prj malformed), fail with explicit invalid_source_crs.
  • utm_wepppy_upper_left:
    • Determine target UTM zone from upper-left extent in WGS84 using the same approach as wepppy/all_your_base/geo/geo.py (utm.from_latlon(north, west) using bbox top-left).
    • Use hemisphere from north coordinate sign (north > 0 => northern hemisphere, else southern), consistent with current WEPPpy behavior.
    • If source CRS missing/unknown, fail with explicit error requiring known CRS.
    • If source CRS is present but invalid (.prj malformed), fail with explicit invalid_source_crs.
    • Fail with explicit utm_not_supported_for_extent when extent is outside supported UTM latitude domain.

Output requirements

  • GeoJSON output:
    • When target_crs=wgs84, emit RFC 7946-compatible GeoJSON.
    • When target_crs is projected (for example UTM), emit non-RFC-7946 GeoJSON with coordinates in the selected CRS.
    • For projected GeoJSON, include explicit CRS details in API metadata and compatibility warning in warnings.
    • For projected GeoJSON, use application/json content type (not application/geo+json) to avoid claiming RFC 7946 compliance.
  • GeoParquet output:
    • Write valid GeoParquet metadata (geo metadata key, primary_column, columns, crs where known).
  • UI/API metadata response must include:
    • detected_crs (authority/WKT when available)
    • projection_status (known, unknown, invalid)
    • attribute_schema array (name, type, width, precision, nullability_note)
    • geometry_types
    • feature_count
    • bbox
    • warnings array (for example: lossy-field mapping, null-value caveats, CRS/extent caveats)

WEPPcloud browser relay mode (json_body)

  • Convert supports:
    • response_mode=download (default; unchanged behavior).
    • response_mode=json_body only when output_format=geojson.
  • Relay success behavior:
    • 200 application/json response containing request_id, geojson, and metadata.
  • Relay validation behavior:
    • Unsupported relay combinations (for example output_format=geoparquet + response_mode=json_body) fail with canonical 400 invalid_request and explicit error.details.
    • Unknown response_mode values fail with canonical 400 invalid_request.
  • Human-facing converter UI remains download-oriented (response_mode=download) and does not expose a relay-mode selector.
  • json_body is an API-level mode for programmatic relay clients.
  • Browser client forwards resulting relay payload to WEPPcloud endpoint over authenticated WEPPcloud session/API.
  • Relay CORS policy is allowlist-driven via SHAPE_CONVERTER_RELAY_CORS_ALLOWED_ORIGINS and scoped to /v1/convert.
  • Relay mode keeps .zip and shapefile sidecars processed/deleted inside shape-converter request scope.
  • Shape-converter does not require WEPPcloud credentials and does not call WEPPcloud directly.

Independent remediation contract (2026-11-14)

The operator authorized IPC, exact-driver, and archive fixes to proceed before the then-required sandbox feasibility decision. This historical sequencing removed independently confirmed weaknesses without waiving the controls then in force. The later Rust parser CLI amendment below supersedes that sandbox objective only after its cutover gates pass. Compression policy, semantic, and deployment gates remain open.

Parser IPC uses strict UTF-8 JSON with exactly properties, geometries, source_bounds, and source_crs_wkt. Properties are objects with string keys and only string, finite number, boolean, or null values. Geometry is null or a Point, MultiPoint, LineString, MultiLineString, Polygon, or MultiPolygon object containing exactly type and coordinates; positions have two or three finite numbers. Preserve MultiPatch GeometryCollections recursively as exactly type and geometries, aggregating the vertex cap across their members. During worker normalization remove Fiona-only null geometries from coordinate geometries and null coordinates from GeometryCollections. Bounds are four finite numbers in min-x/min-y/max-x/max-y order, with ordered extrema. CRS is null or a nonempty WKT string, parsed only after structural checks. Property/geometry lengths must match and obey the existing feature limit; each geometry obeys the existing vertex limit in both worker and parent.

The parser payload byte cap (default 16 MiB) remains authoritative; read at most cap + 1 bytes and reject oversize/non-regular/symlink payloads. Before JSON object allocation, scan for container nesting beyond 16 and more than 2,000,000 JSON values. Reject duplicate keys, trailing data, invalid UTF-8, non-finite numbers, and extra/missing fields. A duplicate-key detector may return plain JSON dictionaries only; no extensible object construction hooks are permitted. Worker output is atomically renamed within request scratch. Parser decode/schema errors use fixed messages without uploaded values or filesystem paths. The parent discards worker stdout/stderr rather than buffering untrusted diagnostics; exit codes 4 and 5 select the existing feature/vertex quota errors, and other nonzero exits select a fixed parse error. Standalone worker stderr stays generic. See ADR-0142.

The retained Fiona parity oracle asserts ESRI Shapefile immediately after its allowlisted open and disables plugin discovery. This transitional rule is not a production fallback; raw production Shapefile bytes reach only the Rust CLI.

For archive paths, normalize separators and Unicode NFC before case-folded collision comparison. Preserve the normalized spelling on disk. Dataset selection requires one logical NFC/case-folded prefix across all admitted shapefile sidecars, including optional and incomplete prefixes; stripped .shp.xml/.qmd files remain excluded. Reject ambiguity with invalid_shapefile. The expansion policy uses absolute compressed-upload, total-uncompressed, member-count, scratch, and parser-input caps plus streamed counters. A ratio-only threshold is intentionally not adopted because it rejects small legitimate highly compressible datasets without reducing the absolute resource ceilings; therefore no new numeric parameter or ADR is introduced. For users: archives must contain one dataset, and duplicate filename aliases or additional sidecar prefixes are rejected. Very large valid conversions can now reach the JSON value budget before the older byte/feature caps; errors are explicit and no partial output is returned. Operators must ship parent and worker together and must not interpret matching readiness mode strings as containment.

Compatibility and regression plan: this is request-internal IPC, with no persisted run-data migration. Parent and worker must ship together; legacy pickle is rejected without fallback. Public output keys/formats remain unchanged. Validate generated GeoJSON and GeoParquet through the actual worker and existing endpoint tests, plus hostile IPC/driver/archive tests. Broader shapefile semantics remain a later gate.

Dedicated image amendment (2026-11-14)

The operator authorizes a shape-converter-only image with an upgraded, coherent GDAL/Fiona stack. This supersedes the dependency-version freeze only for this service image; the common WEPPpy image remains unchanged. The split must preserve the public API and Compose service identity while removing inherited application secrets, datastore configuration, host data mounts, and unrelated runtime packages. It is defense in depth and did not close issue #656 or authorize deployment. Pin the base by digest and install Python dependencies from a resolved hash-locked input. A Dockerfile- specific allowlist must keep repository secrets and unrelated source out of the build context. Qualification must inspect the resolved Compose service, final image, and running container. Matching container/container readiness strings remain non-evidence for sandbox containment. The later Rust parser CLI amendment defines the final production readiness and #656 closure basis.

Starlette 1.3.1 is required in the dedicated image to close the form-limit advisory affecting earlier releases. Register HTTP middleware explicitly through add_middleware(BaseHTTPMiddleware, dispatch=...); the removed decorator API is not a compatibility reason to retain a vulnerable Starlette release. Preserve the existing abuse-control and relay-CORS middleware order and public behavior.

Rust parser CLI amendment (2026-11-14)

The production goal is replacement of GDAL/Fiona hostile Shapefile parsing, not construction of a nested GDAL sandbox. The shape-converter owns one lightweight Rust binary crate at wepppy/microservices/shape_converter/rust/shape_parser_cli. Parser code remains private to that crate; no separate shared Rust library is required. The shape-converter Python process must launch the CLI as a short-lived subprocess; it must never import the production parser through PyO3. The existing service, container, public API, request scratch layout, and strict JSON payload contract remain unchanged. This amendment does not authorize a parser service, socket, queue, datastore, daemon, new container, privilege increase, or network protocol.

The launcher must create a process group, enforce hard wall time, terminate the whole group on timeout or cancellation, and apply independently tested memory, CPU, open-file, input-file, output-file, IPC-payload, feature, and vertex limits. Those limits apply before any attacker-controlled file traversal or record scan. The parser independently validates exact SHP/SHX/DBF record cardinality and SHX offset/length identity, and rejects multipart records unless the first part starts at zero and every later part offset is strictly increasing and within the declared point count. Feature admission stops at the ceiling rather than first building an oversize record list. Parser output remains untrusted and must pass the existing UTF-8 JSON byte, depth, value-count, duplicate-key, primitive, finite-number, and exact-schema validation before use. Panic, abort, malformed output, partial output, limit exhaustion, timeout, and cancellation must map to bounded canonical errors and leave no request residue. No production fallback to Fiona/GDAL or in-process PyO3 is allowed.

For conversion requests, archive extraction, DBF/CRS inspection, payload decoding, geometry construction/reprojection, serialization, and result encoding that can consume meaningful CPU or block on files runs outside the sole ASGI event-loop thread. Client cancellation does not release global admission or remove request scratch until any already-started blocking phase has ended. The process therefore keeps the one-request resource invariant even while draining a canceled bounded thread. Health remains responsive during a held conversion, and a competing conversion fails closed as saturated.

Startup validates the complete effective resource envelope, not only defaults. Production permits one active request globally and per client and rejects an environment override above the reviewed maxima for features, vertices, parser payload/input/memory/CPU/open files/time, scratch/precheck, or serialized output. Changing those maxima or concurrency requires a revised resource ADR and new exact-image near-ceiling evidence.

This CLI process boundary limits failure propagation and permits deterministic termination; it is not a claim of containment from arbitrary code execution. The CLI inherits the shape-converter container's UID, mounts, and network authority. That residual is acceptable only after dependency and unsafe-code review, malformed-input testing, oracle parity, exact-image validation, and independent security/operator acceptance. Until those gates pass, issue #656 remains open. After cutover, #656 may close as superseded by removal of the GDAL/Fiona failure class, not as evidence that a second sandbox was implemented.

The qualified GDAL 3.12.4/Fiona 1.10.1 image is a transitional parity oracle. The final production image must contain shape_parser_cli, must attest the binary and a real bounded parse during readiness, and must not contain or expose Fiona/GDAL parser libraries or a PyO3 parser entry point. The executable ships with adjacent /usr/local/bin/shape_parser_cli.json provenance using schema shape-parser-binary-provenance-v1. The sidecar binds the binary hash, size, version, ELF identity, source manifest/revision, Cargo lock, compiler, command, and build time. Build validation and readiness fail closed if the pair is missing, malformed, or mismatched. This metadata is not a publisher signature; the immutable image digest remains the authenticity boundary.

Non-Functional Requirements

  • Dedicated container/service (no embedding conversion in existing web app process).
  • Sandboxed runtime:
    • Non-root user.
    • Read-only root filesystem.
    • no-new-privileges.
    • Drop all Linux capabilities except explicit minimum.
    • Seccomp/AppArmor (or SELinux) enabled.
    • Attach the service only to its dedicated internal Compose network so neither the API nor its Rust child has outbound routing. The child shares the container network namespace; this is not a separate network sandbox.
    • Parse hostile Shapefile bytes only through the standalone Rust CLI with hard timeout, process-group termination, and explicit per-process resource limits.
    • Treat parser-worker output as untrusted. Exchange data only through a non-executable serialization with an explicit schema, byte limit, nesting limit, and primitive-type allowlist. pickle, marshal, YAML object tags, and equivalent executable deserialization are forbidden across this boundary.
    • During the parity transition only, disable GDAL driver/plugin autoload from attacker-writable paths and keep an explicit driver allowlist at every oracle open.
  • Resource limits (initial defaults, tune with data):
    • Request timeout: 30s metadata scan, 120s conversion hard timeout.
    • Max ZIP upload size: 100 MB.
    • Edge proxy body cap must include multipart overhead headroom above the ZIP limit so canonical app quota errors remain observable (current WEPP1 policy: 120 MB edge cap for 100 MB ZIP quota).
    • Max total uncompressed bytes: 128 MiB.
    • Max member count: 200.
    • Max features: 1,000,000.
    • Max vertices per feature: configurable guardrail (start 250,000).
    • Multipart guardrails: max part count and max text-field size.
    • Scratch-space guardrails: extraction/write free-space preflight and per-request scratch quota.
  • Observability:
    • Request ID, rejection reason codes, structured parse_duration_ms, structured convert_duration_ms, cleanup success/failure.
    • Do not log raw uploaded content.

Service Design

Deployment shape

  • Service: shape-converter.
  • Caddy utility namespace route: /utils/shape-converter.
  • Caddy matcher is tightened to avoid prefix bleed (no /utils/shape-converter* glob matching).
  • Implemented Caddy route shape:
@shape_converter_exact path /utils/shape-converter
redir @shape_converter_exact /utils/shape-converter/ 308

handle_path /utils/shape-converter/* {
    request_body {
        max_size 100MB
    }

    reverse_proxy shape-converter:8060 {
        header_up X-Forwarded-Prefix /utils/shape-converter
        header_up -Forwarded
        header_up -X-Forwarded-For
        header_up -X-Forwarded-Host
        header_up -X-Forwarded-Proto
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Host {host}
        header_up X-Forwarded-Proto {scheme}
        header_up Host {host}
        transport http {
            read_timeout 130s
            write_timeout 30s
            response_header_timeout 130s
        }
    }
}
  • WEPP1-specific runtime keeps the same header-sanitization/timeouts policy with a 120MB edge body cap to preserve canonical app-level 100MB ZIP quota behavior under multipart overhead.
  • UI implementation is a minimal static page served by this service; WEPPcloud relay integration remains separate and consumes API contracts (response_mode=json_body) rather than replacing the local UI flow.

Container specification (detailed)

Image and runtime

  • Multi-stage image build:
    • Stage 1: build the lockfile-pinned shape_parser_cli release binary and app dependencies.
    • Stage 2: minimal runtime image with the CLI and app code, without Fiona/GDAL parser dependencies.
  • Run as non-root fixed UID/GID (for example 10001:10001).
  • Root filesystem read-only.
  • Writable paths only via tmpfs mounts:
    • /tmp for request scratch (archive, extraction, generated output).
    • /run/shape-converter for process/runtime state.
  • Production compose runs without host path mounts; dev compose mounts the repo read-only for local iteration only.

Compose-level hardening contract

  • read_only: true
  • user: "10001:10001"
  • security_opt includes no-new-privileges:true
  • cap_drop: [ALL]
  • pids_limit (for example 256)
  • mem_limit and cpus must be set (no unlimited defaults)
  • tmpfs mounts use noexec,nosuid,nodev options
  • Service is attached only to a dedicated internal network segment and proxied by Caddy
  • East-west access to other app services is denied by network policy
  • Outbound egress is deny-all (including DNS) unless explicitly allowlisted
  • No Docker socket mount, no privileged mode, no host PID/IPC namespaces
  • Runtime/image digest pinning is tracked as an operational hardening follow-up (not yet enforced directly in current compose manifests)

Health and lifecycle

  • GET /utils/shape-converter/health/live for liveness.
  • Startup checks scratch writability, the exact shape_parser_cli identity, configured resource limits, a real bounded fixture parse, strict JSON validation, and cleanup. Startup fails if any check fails.
  • GET /utils/shape-converter/health/ready reports the cached successful startup parser check plus a cheap current scratch check. Public readiness requests do not launch attacker-triggerable parser processes.
  • The transitional SHAPE_CONVERTER_SANDBOX_MODE and SHAPE_CONVERTER_REQUIRED_SANDBOX_MODE equality check is not evidence of containment and must not be the final production readiness basis.
  • Graceful shutdown:
    • Stop accepting new work.
    • Wait for in-flight conversions up to shutdown timeout.
    • SIGKILL any remaining conversion subprocess group after timeout.
  • Startup self-check:
    • Validate the adjacent executable provenance sidecar, binary SHA-256 and size, and shape_parser_cli --version agreement.
    • Validate writable scratch root and cleanup permissions.
    • Run and clean up a bounded parser fixture under the production launcher and reject readiness when any required limit or JSON validation is inactive.

Performance and concurrency contract

  • Production runs one Uvicorn application worker. Abuse-control counters and conversion metadata are intentionally process-local, so multiple workers would multiply configured limits and make metadata lookup nondeterministic. Scale-up requires a reviewed shared-state contract before increasing the worker count.
  • Bound active requests with middleware abuse controls:
    • SHAPE_CONVERTER_MAX_INFLIGHT_GLOBAL (default 1) with immediate 503 service_saturated when the single resource envelope is occupied.
    • SHAPE_CONVERTER_MAX_INFLIGHT_PER_IP (default 1) with immediate 429 rate_limited when that client already has an active request.
    • The single-flight defaults are part of the aggregate memory invariant. Do not raise either value without exact-image concurrent near-ceiling evidence and a revised cgroup/resource ADR.
    • SHAPE_CONVERTER_RATE_LIMIT_COUNT / SHAPE_CONVERTER_RATE_LIMIT_WINDOW_SECONDS (defaults 120 / 60) for sliding-window request throttling.
  • Per-request timeout:
    • Metadata inspect timeout: 30 seconds.
    • Conversion timeout: 120 seconds.
    • Multipart body read timeout is enforced via SHAPE_CONVERTER_BODY_READ_TIMEOUT_SECONDS (default 15).
    • Edge proxy read/write/response-header timeouts enforce bounded request/response lifetime.
  • Backpressure policy:
    • No unbounded in-memory queue.
    • Reject quickly with retry guidance once capacity is reached.
    • Multipart guardrails: max part count and max field size.

Storage and quota contract

  • tmpfs scratch size is explicitly configured and must exceed max uncompressed archive bytes plus conversion headroom.
  • Service performs free-space preflight checks before extraction and before conversion write.
  • Per-request scratch quota is enforced independently of global tmpfs size.

Worker lock-up prevention contract

  • Never run heavy geospatial conversion on event loop threads.
  • Execute hostile parsing through shape_parser_cli and keep other heavy conversion steps out of the event loop, with:
    • Process-group creation.
    • Hard wall-clock timeout.
    • Kill entire process group on timeout/cancel.
    • Direct memory, CPU, open-file, input/output-file, payload, feature, and vertex limits.
  • Keep ASGI handlers async and lightweight:
    • Upload validation.
    • Job orchestration.
    • Streaming response.
  • Ensure all cleanup runs in finally blocks independent of client disconnect.
  • In production, Python must invoke the standalone Rust CLI and must not import the parser through PyO3. The CLI is a reliability and resource boundary within the existing hardened container, not a second arbitrary-code containment boundary.

Framework/runtime decision

Option comparison

Option Pros Cons Worker lock-up risk profile Fit for this service
Starlette (Python) Minimal ASGI layer, low overhead, already aligned with WEPPpy query-engine patterns, few dependencies More manual request validation/OpenAPI wiring than FastAPI Low if conversion is subprocess-bound and concurrency is capped; high if blocking work leaks into event loop Strong
FastAPI (Python) Fast development, built-in validation/docs, based on Starlette/Pydantic Slightly higher framework/validation overhead, extra abstraction for a very small API Similar to Starlette for lock-up risk (risk is from blocking work, not framework choice) Good but not necessary
Go (net/http or chi/gin) Very strong concurrency model, low runtime overhead, each request served in goroutines Additional service/runtime stack while parsing is already owned by the selected Rust CLI Very low runtime lock risk, but operational complexity is higher in this codebase Not justified for this control plane

Final decision

  • Choose Starlette for v1, with a strict subprocess execution model for geospatial work.
  • Rationale:
    • Throughput here is dominated by parser/IO and archive processing, not JSON framework overhead.
    • Starlette keeps the control plane lean and matches existing repo ASGI patterns.
    • Worker lock-ups are prevented by design through bounded subprocess execution and hard timeouts.
    • This avoids introducing a new language/runtime stack prematurely.
  • Revisit Go only if production data shows the ASGI control plane itself is a measurable bottleneck after subprocess isolation and concurrency tuning.

Processing pipeline

  1. inspect creates a request temp directory under isolated tmpfs mount.
  2. inspect saves uploaded ZIP to temp dir.
  3. Pre-scan ZIP central directory and validate all entries before extraction.
  4. Extract with canonical-path enforcement into temp dir only.
  5. Validate shapefile set and parse metadata.
  6. If this is convert, perform requested reprojection/serialization and stream artifact.
  7. In finally, cleanup deletes:
    • Uploaded ZIP
    • Extracted members
    • Generated output artifact
    • Request temp directory
  8. Janitor removes stale temp directories left only by abnormal process termination.

Cleanup Contract (Required)

  • All request artifacts must be ephemeral and request-scoped.
  • Deletion attempted on every code path (success, validation failure, conversion failure, timeout, client disconnect).
  • Cleanup failures are logged as structured security events and retried by janitor.
  • No artifact persistence after request terminal response.

API Contract (Implemented)

POST /utils/shape-converter/v1/inspect

Request:

  • archive (ZIP)
  • No auth credentials required.

Response 200:

  • request_id
  • detected_crs
  • projection_status
  • feature_count
  • geometry_types
  • bbox
  • attribute_schema
  • warnings

Error behavior:

  • Canonical error payload with error.code, error.message, and error.details.
  • request_id is included in error payloads.

POST /utils/shape-converter/v1/convert

Request:

  • archive (ZIP)
  • output_format = geojson|geoparquet
  • target_crs = same_as_shapefile|wgs84|utm_wepppy_upper_left
  • response_mode supports download|json_body (default download)
  • response_mode=json_body is valid only with output_format=geojson
  • No auth credentials required.

Response 200:

  • response_mode=download: streamed artifact download + metadata sidecar JSON endpoint keyed by request_id
  • response_mode=json_body: JSON payload with:
    • request_id
    • geojson
    • metadata

Download headers:

  • X-Shape-Converter-Request-Id
  • X-Shape-Converter-Metadata-Path
  • Content-Disposition attachment filename (RFC 5987-compatible encoding)

GET /utils/shape-converter/v1/convert/metadata/{request_id}

Response 200:

  • Convert metadata for download-mode conversions (output_format, target_crs, projection_status, detected_crs, attribute_schema, geometry summary, warnings, output CRS details, and download metadata fields).

Response 404:

  • Canonical metadata_not_found when request metadata has expired or is unknown.

Error response contract:

  • Follow canonical WEPPpy error payload style (error.code, error.message, error.details required).
  • Use explicit codes like:
    • metadata_not_found
    • rate_limited
    • request_timeout
    • invalid_request
    • invalid_archive
    • archive_path_traversal
    • archive_quota_exceeded
    • missing_required_sidecar
    • invalid_shapefile
    • invalid_source_crs
    • unknown_source_crs
    • reprojection_failed
    • service_saturated
    • utm_not_supported_for_extent
  • Error status mapping (minimum):
    • 400: validation/input errors.
    • 404: metadata sidecar request_id not found.
    • 408: request body/processing timeout.
    • 413: upload/body/expansion quota exceeded.
    • 429: rate-limit or concurrency saturation.
    • 503: service saturation and scratch-capacity exhaustion.
    • 500: unexpected internal failure.

ZIP Upload Risk Matrix (Comprehensive)

Risk Why it matters Required mitigations Required tests
Path traversal (../, absolute paths, drive paths) Write outside extraction root Canonicalize + commonpath enforcement per member; reject violators Archive with ../../etc/passwd, /tmp/x, C:\\... rejected
Zip Slip variants via encoded names Bypass naive prefix checks Normalize Unicode + separators before path checks Mixed separator + encoded traversal payloads rejected
Symlink entries Indirect traversal/overwrite Reject symlink members and special file types ZIP containing symlink entry rejected
Duplicate member names Overwrite confusion Reject duplicate normalized extraction paths ZIP with duplicate filenames rejected deterministically
Case-collision names Cross-platform overwrite ambiguity Normalize policy to lowercase compare for collision detection Road.shp + road.shp rejected
ZIP bombs / high compression ratio Disk and memory exhaustion Enforce absolute compressed, total-uncompressed, member, scratch, and parser-input caps while streaming with counters; do not rely on a ratio threshold High-expansion fixtures crossing an absolute ceiling are rejected before parser execution; valid compressed controls remain accepted
Excessive member count Inode/CPU exhaustion Max member cap ZIP with > cap rejected
Nested archives Recursive bomb/smuggling Reject nested archives in v1 ZIP containing .zip members rejected
Encrypted/password ZIPs Uninspectable content, operational failures Reject encrypted flag entries Encrypted ZIP rejected with explicit error
Unsupported compression methods Parser exceptions/DoS Allowlist supported compression methods only Unsupported compression method returns validation error
Corrupt CRC/central directory Crash/undefined behavior Validate archive integrity before extraction Truncated/corrupt ZIP rejected
Oversize filenames/path depth FS/path handling failures Enforce max filename length and max path depth Very deep path archive rejected
Hidden/system files in archive Ambiguous behavior Allowlist required extensions and optional sidecars only; sanitize known metadata sidecars (.shp.xml, .qmd) by immediate unlink .exe, .dll, .py members rejected
Null-byte or control-char filenames Sanitization bypass Reject non-printable/control chars and null bytes Crafted control-char names rejected
ZIP metadata trust issues Spoofed types Validate extension + signature + entry policy; exact-driver restricted open remains authoritative Renamed VRT/HDF/other format with .shp extension is rejected before data use
Archive parser dependency CVEs Parser compromise risk Keep libraries patched, run in sandbox, pin versions Dependency audit gate + CVE regression check

Shapefile Risk Matrix (Comprehensive)

Risk Why it matters Required mitigations Required tests
Missing required sidecars (.shp/.shx/.dbf) Cannot reliably read dataset Explicit required-file validation Omit each required sidecar -> explicit failure
Multiple shapefile prefixes in one ZIP Ambiguous conversion target Reject unless explicit dataset selection implemented ZIP with two prefixes rejected
Missing .prj Unknown CRS; reprojection unsafe Mark unknown for inspect; block CRS transforms requiring known source Missing .prj + wgs84/utm request fails
Invalid .prj WKT Bad reprojection Parse CRS strictly; return invalid_source_crs Corrupted .prj rejected
DBF encoding ambiguity (.cpg/LDID issues) Attribute corruption Detect/report source encoding; allow explicit override in future Non-UTF8 DBF fixture preserved/flagged correctly
10-char field name truncation collisions Schema ambiguity/data loss Report original + normalized names when possible; fail on unresolved duplicates Collision fixture handled deterministically
Null-value semantics in DBF/shapefile Silent value substitution Surface null/substitution behavior in metadata warnings Dataset with sentinel null-like values flagged
Attribute width/precision limits Rounding/truncation Preserve types carefully; emit warnings for lossy mappings Large numeric precision fixture warning/assertion
Mixed/unsupported geometry content Conversion failure or wrong output Validate geometry types and null geometries before write Mixed geometry fixture produces explicit policy error
Invalid geometries (self-intersections, ring issues) Output unusable or crashes Run geometry validity checks; optionally repair under explicit flag (off by default) Invalid polygon fixture returns explicit error
Multipart/multipatch edge cases Type coercion surprises Normalize and report resulting geometry types Multipatch/multipart fixtures validated and reported
Massive vertex counts CPU/memory exhaustion Vertex and complexity guardrails High-vertex polygon fixture rejected by threshold
Record count mismatch (.shp vs .dbf) Corrupt feature/attribute mapping Cross-check record counts and fail if inconsistent Tampered sidecar counts rejected
Malformed binary structures Native parser crash risk Parse in hardened sandbox process and fail safely Fuzz corpus malformed files do not crash host
Out-of-range coordinates Projection errors Validate coordinate ranges by CRS before transform Extreme-coordinate fixtures fail clearly
Anti-meridian/multi-zone extents UTM ambiguity Document UL-corner UTM behavior and emit warning if bbox spans zones Multi-zone fixture warns, still deterministic
Shapefile size/component limits Operational failures Enforce file-size and feature-count caps before full load Over-limit fixtures rejected early
XML sidecar entity expansion (.xml, .gml, .shp.xml) CPU/memory DoS via parser Reject generic XML sidecars; if .shp.xml is present, delete it immediately after extraction and enforce parser timeout + memory caps Generic XML sidecar rejected; .shp.xml stripped with warning
Parser-triggered remote fetch (/vsicurl/, remote references) Unwanted network egress/data exfil risk Keep deny-all egress and disable remote parser sources by policy/config Fixture attempting remote reference fails without outbound access
Metadata PII leak from metadata sidecars (.shp.xml, .qmd) Exposure of usernames, contact details, host paths, org/process history Never parse or return sidecar metadata; sanitize by immediate unlink; keep warnings/log fields free of raw sidecar content Fixture containing PII in sidecar yields no PII in API payloads/logs
Executable parser-worker IPC Native parser compromise crosses back into the parent process Use schema-validated, data-only serialization; never deserialize pickle or object-capable formats from the worker Malicious worker payload is rejected without code execution or filesystem side effects

CRS Rules and UTM Determination Details

  • Source of truth for UTM mode should mirror existing WEPPpy logic in wepppy/all_your_base/geo/geo.py:
    • Determine UTM zone from upper-left bbox corner (north, west).
    • Force lower-right corner into the same zone.
    • Set hemisphere from sign of north latitude.
  • Practical implementation for converter:
    1. Compute source bbox.
    2. Transform bbox to WGS84 if needed.
    3. Use upper-left (max_lat, min_lon) to compute UTM zone.
    4. Build target CRS EPSG: 326xx for north, 327xx for south.
    5. Reproject all features.

UI Requirements

  • Layout and control contract:
    • One controls panel titled Upload with:
      • one archive file input
      • output format selector
      • target CRS selector
      • primary Convert And Download button
      • secondary Inspect Upload button
    • Warnings panel appears immediately after the Upload panel.
    • Projection, Geometry Summary, and Attribute Schema panels follow warning/error content.
    • All user-facing panels are the same width in the default layout.
    • Static asset URLs must resolve correctly behind proxied namespace /utils/shape-converter/ (no root-relative asset assumptions).
  • Visibility contract:
    • Initial load:
      • show Upload panel
      • hide Warnings, Projection, Geometry Summary, and Attribute Schema
    • Show Projection, Geometry Summary, and Attribute Schema only after a successful inspect payload provides content.
    • Keep Warnings hidden until there is warning/error/advisory content to show.
  • Metadata rendering contract:
    • Projection panel:
      • Detected CRS (authority or unwrapped WKT when authority is unavailable)
      • Projection status badge (known, unknown, invalid)
      • Output CRS (after convert metadata is available)
    • Geometry summary:
      • Feature count
      • Geometry types
      • BBox
    • Attribute schema table:
      • Column name
      • Source type
      • Width/precision
      • Nullability note (if inferable)
  • Interaction contract:
    • UI may reuse one selected local archive file for both actions, but each Inspect/Convert click sends a separate request.
    • Convert UI is download-oriented and submits response_mode=download.
    • Relay response_mode=json_body is available via API for programmatic clients and is not presented as a default end-user control.
  • Error UX contract:
    • All API/network errors must be observable by users (no silent failures).
    • inspect-status and convert-status messages must use plain language with specific next-step guidance.
    • 429/503 abuse-control states must include clear retry guidance and display Retry-After when present.
  • Keep messages explicit; no silent fallback behavior.

Security Controls Baseline

  • Implemented baseline controls:
    • OWASP-style upload defense-in-depth: extension allowlist, signature checks, strict size/runtime limits, sandboxing, and abuse controls for anonymous access.
    • Container/runtime hardening in compose: non-root user, read_only, no-new-privileges, cap_drop=ALL, explicit pid/memory/cpu limits, hardened tmpfs mounts.
    • No Docker socket mount, no privileged mode, and isolated internal network segment.
    • Public-service edge controls:
    • Edge request body size cap.
    • Edge upload/read/write/response-header timeout controls.
    • Edge forwarding-header sanitization and trusted-proxy configuration.
    • CORS policy for browser relay use-cases is required for json_body mode.
    • Application-level controls (authoritative when edge lacks native rate-limit module support):
    • Per-IP rate/concurrency controls in service middleware.
    • Per-request scratch quotas and timeout cancellation hooks.
    • Metadata sanitization and length bounds before serialization/logging.
  • Program/release controls retained for future hardening:
    • Runtime image digest pinning.
    • SBOM generation for release builds.
    • Vulnerability scan gate that blocks unresolved High/Critical issues in reachable runtime components.
  • Parser dependency controls:
    • Track and triage the Rust parser's direct and transitive dependencies, unsafe-code surface, advisories, licenses, and malformed-input behavior before each release. Record the exact Cargo lock and binary build provenance.
    • Maintain explicit timeout/kill and resource-limit behavior for parser panic, non-termination, allocation, file, and output exhaustion classes.
    • Treat archive extraction and parser process separation as implementation details, not security boundaries; archive and parser inputs still require full validation and resource controls.
    • Maintain metadata-privacy policy that forbids exposing raw .shp.xml content in API payloads, logs, or warnings (only advisory removal notices are allowed).
    • Retain the dedicated low-privilege identity, restricted filesystem, bounded CPU/memory/files/output, and internal network for the container. The Rust CLI inherits those authorities and is not a second sandbox. GDAL-specific driver/plugin restrictions apply only to the transitional parity oracle.

Test Plan

Unit tests

  • ZIP entry path canonicalization and collision handling.
  • Absolute compressed/uncompressed/member/scratch quota enforcement, including high-expansion rejection and valid compressed controls.
  • CRS option validation matrix.
  • UTM zone function parity with WEPPpy upper-left rule.

Integration tests

  • Valid uploads for point/line/polygon shapefiles.
  • Missing sidecars and invalid .prj flows.
  • GeoJSON and GeoParquet output metadata correctness.
  • Browser relay flow correctness (response_mode=json_body) including success payload shape and invalid-combination failures.
  • Cleanup verification: no residual files after request terminal state.

Security tests

  • Zip-Slip fixtures.
  • Zip bomb fixtures.
  • Encrypted ZIP fixtures.
  • Malformed shapefile fuzz corpus.
  • Parser timeout and cancellation behavior.
  • Disguised VRT/HDF/other non-shapefile content renamed .shp is rejected by the exact-driver boundary.
  • Malicious or malformed parser-worker output cannot trigger executable deserialization in the parent.
  • XML sidecar bomb fixtures (.shp.xml/.xml entity expansion class payloads).
  • Parser non-termination fixtures (malformed WKB parser-loop class).
  • Metadata privacy fixtures: .shp.xml and .qmd containing usernames/paths/contact fields are not exposed in responses/logging.
  • Runtime hardening verification:
    • read_only, cap_drop=ALL, no-new-privileges, seccomp/AppArmor/SELinux, pids/mem/cpu limits, deny-all egress.
  • Slowloris and pinned-download resilience:
    • Slow upload/body timeout enforcement.
    • Slow download/write timeout cleanup behavior.
  • Anonymous abuse controls:
    • Endpoints reachable without auth.
    • Proxy-validated IP rate limits return expected 429.
    • Concurrent flood beyond caps does not starve healthy traffic.

Performance tests

  • Large-but-valid shapefile throughput.
  • Concurrent conversion requests under CPU/memory limits.

Acceptance Criteria

  • Users can upload ZIP shapefiles and download GeoJSON/GeoParquet reliably.
  • Service is publicly accessible without authentication by design.
  • UI implements the single Upload controls panel (one archive input + output/CRS selectors + primary convert/secondary inspect actions).
  • Warnings panel is positioned directly after Upload and is hidden until warning/error/advisory content exists.
  • Projection, Geometry Summary, and Attribute Schema remain hidden until successful inspect metadata is available.
  • Convert API supports both response_mode=download and relay response_mode=json_body (GeoJSON-only), with explicit canonical 4xx errors for unsupported combinations; the public UI remains download-oriented.
  • Attribute schema rendering includes nullability notes with explicit fallback text when nullability is not inferable from DBF metadata.
  • CRS options behave exactly as specified (including WEPPpy UL-corner UTM mode and explicit out-of-domain failures).
  • Invalid .prj reprojection paths return explicit invalid_source_crs while missing CRS remains unknown_source_crs.
  • GeoJSON behavior is explicit: RFC 7946 in WGS84 mode and clearly labeled non-RFC projected mode for UTM/same-CRS output.
  • ZIP and shapefile risk tests are implemented and passing.
  • Native parser opens are exact-driver restricted and parser-worker IPC is schema-validated and non-executable.
  • XML sidecar and parser-loop abuse tests are implemented and passing.
  • No sidecar-derived PII (for example .shp.xml/.qmd usernames, contacts, file paths) is exposed by API metadata or logs.
  • When .shp.xml is included, API warnings explicitly report removal and advise that packing .shp.xml in shapefile ZIPs is generally not advisable.
  • When .qmd is included, it is sanitized via immediate unlink and never parsed or surfaced in API metadata.
  • No uploaded or generated artifacts persist after each request terminal response (verified by tests).
  • Service runs in its own hardened container and is reachable through Caddy.
  • Production contains no GDAL/Fiona or PyO3 parser fallback. It uses only the bounded Rust CLI and fails startup when executable provenance or its real parser canary fails. If deployment cannot establish that state, deployment stops shape-converter and verifies that it is not running; it does not restore an older parser implementation.
  • CI/security validation proves container hardening and runtime abuse controls are active.

Source Notes

  • Existing WEPPpy UTM upper-left behavior: wepppy/all_your_base/geo/geo.py lines 808-823.
  • Internal prior risk evidence for ZIP/shapefile flow: docs/culvert-at-risk-integration/audits/culvert-web-app-codebase-audit-2026-02-20.md lines 116-127 and 152-167.

External References