Sandboxed Shapefile Conversion Security Contract
Status: Rust CLI implementation locally validated; independent review and rollout pending
Last Updated: 2026-11-14
Primary Areas: docker/docker-compose.dev.yml, docker/caddy/Caddyfile, wepppy/microservices/*, wepppy/all_your_base/geo/geo.py, wepppy/weppcloud/*
Contract Status and Conformance
This document is the normative security and behavior contract for the public shape-converter service. Closed work packages record implementation history but do not prove continuing conformance. A control is implemented only when the checked-in runtime path and regression evidence enforce it.
Resolution is tracked in
docs/work-packages/20261113_shape_converter_security_conformance/ and GitHub
issues #655 and #656. The operator selected a service-local Rust CLI to replace
production GDAL/Fiona parsing on 2026-11-14. The previously qualified dedicated
GDAL image remains a pre-cutover parity oracle only. The common WEPPpy image
remains unchanged.
The 2026-11-14 independent remediation slice has the following source/local-test state; this table does not assert deployed behavior:
| Boundary | Current evidence | Status |
|---|---|---|
| ZIP admission and extraction | Size/member quotas, path containment, extension policy, NFC/case-fold duplicate detection, special-file rejection, and request-scoped extraction are implemented in archive_validation.py. |
Implemented. Absolute compressed, uncompressed, member, scratch, and parser-input quotas are the expansion policy; no ratio threshold is adopted. |
| Dataset selection | select_single_shapefile_dataset(...) requires one complete shared-prefix .shp/.shx/.dbf set and rejects additional complete or incomplete prefixes. |
Implemented. |
| Format/parser identity | Production conversion invokes only the service-local shape_parser_cli; the legacy Fiona worker is removed and the final candidate image contains no Fiona/GDAL files. |
Locally implemented and exact-image validated. The retained GDAL image is test/oracle evidence only. |
| Parser process boundary | Convert launches shape_parser_cli in a new process group with a minimal environment, fixed exits, strict JSON IPC, timeout/kill, and OS resource limits. |
Implemented and locally validated for input, output, feature, vertex, file-descriptor, memory, and CPU exhaustion. |
| Runtime containment | Compose enforces non-root, read-only root, dropped capabilities, resource limits, hardened tmpfs, and an internal network. | Partial: the selected CLI shares the container UID, mounts, and network authority. It is a bounded failure boundary, not arbitrary-code containment; final acceptance of this residual is pending. |
| Semantic and adversarial validation | Point, line, polygon, reversed ring, Z/null, MultiPatch, DBF primitive/null/date, malformed input, strict IPC, cleanup, timeout, and both output formats have direct tests/readback. | Locally validated. Broader fuzz/property expansion remains defense-in-depth follow-up, not a production fallback. |
| Dependency qualification | shapefile 0.9.0 and its 30-package locked graph have dependency, license, unsafe, RustSec, benchmark, and parity evidence. |
Locally accepted. Exact release image still requires independent security signoff and normal rollout authorization. |
Until the nonconformant and partial items are closed or explicitly risk-accepted, the service must not be represented as fully conformant with this contract.
GeoParquet input normalization amendment (VUN-01)
Accepted contract intent 2026-11-16; implementation and exact-image conformance are pending. WEPPcloud's Roads, AgFields, Omni contrasts and Batch Runner may submit one admitted local GeoParquet file to this service for normalization to canonical WGS84 GeoJSON. The public converter UI remains ZIP/Shapefile-oriented. Culvert and D-Tale paths are excluded.
GeoParquet uses a separate service-local geoparquet_parser_cli; it is never a
mode of shape_parser_cli. The exact accepted graph is Parquet 60.1.0 with
default features disabled and Snappy only, plus serde_json 1.0.145, bound to the
exact Cargo.lock SHA-256
ce923427941fabe153c8b282dcfddeb1168b60670055ab34f47911d039748f13.
Direct use of an upstream Parquet decoder on unpreflighted hostile bytes is
forbidden. Any graph, feature, version or lock-hash change reopens dependency
review.
Before any upstream decoder runs, owned code must open one regular nonsymlink immutable local file and perform the complete bounded Compact-Thrift/footer, schema, row-group, column, page, dictionary, encoding, codec, offset and range preflight required by ADR-0145. Preflight and decode use that same descriptor, with identity and size rechecked before decode. The parser accepts no pathname selected by output metadata, URL, directory, dataset, external column file, encryption, plugin or network resource.
The admitted profile and canonical output are defined by the
vector-upload normalization contract.
The CLI emits strict bounded data-only JSON. The service revalidates its schema,
feature count, finite coordinates, geometry types, observed bounds and hashes,
resolves/reprojects CRS offline without ballpark fallback, serializes canonical
GeoJSON, reads that artifact back and reconciles its declared metadata before
success. The service computes and returns source_size and source_sha256 from
its own complete staged request snapshot; receiving workflows must reconcile both
against their descriptor-bound retained original before receiving validation or
accepted-state mutation. Every receiving workflow then revalidates canonical
GeoJSON before any accepted-resource or workflow-state mutation.
Reuse the accepted shape_parser_cli lifecycle: private request scratch, minimal
environment, fixed argv without shell, process-group ownership, direct resource
limits, wall timeout, cancellation shielding/drainage, bounded stdout/stderr/IPC,
adjacent binary provenance, runtime canary and cleanup after every outcome. A
missing, changed, unready, timed-out or invalid parser fails closed with the
stable normalization errors; no Python, GDAL/Fiona, PyArrow, alternate CLI or
in-process fallback is allowed. Exact-image qualification, SBOM/advisory scan and
rollout remain separate gates.
Receiving workflows retain their original project uploads and provenance under the canonical normalization contract. Shape-converter receives no user account, ban authority or project-artifact pathname. Its request copy remains private ephemeral scratch and is removed after complete process-group drainage. A project-retained quarantined original is never a converter retry source; a retry must create a new receiving attempt and repeat all admission controls.
Objective
Provide a sandboxed upload/conversion service where users can upload a ZIP containing a shapefile and convert it to GeoJSON or GeoParquet, with explicit CRS handling options:
- Same as shapefile
- WGS84
- UTM (zone determined the same way WEPPpy currently determines UTM from the upper-left extent)
The service must report projection and attribute schema in the UI, and must delete uploaded ZIPs, extracted shapefile sidecars, and generated artifacts after each request completes.
Why This Is Needed
Users currently struggle with local shapefile conversion workflows. A dedicated conversion service removes desktop GIS friction while centralizing security controls for high-risk ZIP and geospatial parser inputs.
Scope
In scope
- New standalone containerized microservice for conversion.
- Browser UI for upload, CRS option selection, and metadata reporting.
- Output formats: GeoJSON, GeoParquet.
- Strict upload validation, extraction controls, parser hardening, and cleanup lifecycle.
- Comprehensive negative/security test coverage for ZIP and shapefile risks.
Out of scope
- General-purpose GIS editing.
- Multi-format archive support beyond ZIP.
- Persistent dataset storage.
- Long-running async job queue (v1 should be synchronous with strict timeouts).
- WEPPcloud route/controller updates to consume new relay flow (separate scope).
User Experience
- User opens converter page and sees one
Uploadcontrol panel. - User selects
.zip(containing exactly one logical shapefile dataset) in a single archive input. - User can choose output format (
GeoJSON/GeoParquet) and target CRS (same_as_shapefile/wgs84/utm_wepppy_upper_left) in that same panel. - User clicks
Inspect Upload(secondary action) to populate metadata panels:- Projection status (
known/unknown/invalid) with detected CRS details when present. - Geometry summary (feature count, geometry types, bounds).
- Attribute schema table (field name, type, width/precision, nullability note).
- Warnings (lossy/null/geometry/CRS caveats).
- Projection status (
- User clicks
Convert And Download(primary action) to run conversion and download the artifact. inspectandconvertremain independent request-scoped uploads by backend contract; the browser may reuse the same selected local file, but each action sends its own multipart request and no server-side cross-request staging is allowed.- Service deletes all artifacts created by each request.
Functional Requirements
Upload and input requirements
- Accept only
multipart/form-datawith one file field:archive. - Accept only
.zipextension and ZIP signature/magic-byte validation. - Reject password-protected/encrypted ZIP entries.
- Reject nested archives by policy in v1.
- Require shapefile core sidecars by shared prefix:
.shp,.shx,.dbf. - Require exactly one logical dataset: core sidecars must share one normalized directory and prefix; reject extra complete or incomplete shapefile prefixes.
- Normalize path separators and Unicode before duplicate/case-collision checks.
- Permit optional sidecars:
.prj,.cpg,.sbn,.sbx,.qix. - Allow
.shp.xmlmetadata sidecars as non-fatal input but immediately unlink them after extraction. - Allow
.qmdmetadata sidecars as non-fatal input but immediately unlink them after extraction. - Emit an explicit warning when
.shp.xmlis removed that it is generally not advisable to pack.shp.xmlin shapefile ZIPs because metadata may leak sensitive details. - Reject other non-required XML sidecars (
.xml,.gml) in v1. - Reject archives containing multiple shapefile prefixes unless user explicitly selects one (v1 default: reject).
- Treat extensions and magic bytes as admission hints, not parser authorization.
The parser must enable only the ESRI Shapefile driver, verify the opened driver
is exactly
ESRI Shapefile, and reject renamed VRT, HDF, or other formats. inspectandconvertare independent uploads by design; no server-side file staging across requests.
Access model (explicit)
- This is a public unauthenticated service.
inspectandconvertendpoints require no login, no JWT, and no session cookie.- Service must remain stateless with no per-user persistence.
- Because there is no auth, abuse controls are mandatory:
- Per-IP request rate limits and concurrent conversion caps.
- Strict payload size, expansion, and runtime limits.
- Fast-fail
429/503behavior under saturation. - Trusted client identity rules:
- Only proxy-set forwarding headers are trusted.
- Edge strips inbound
X-Forwarded-*from clients and sets canonical values. - Service trusts only configured proxy hops.
- IPv6 rate-limit keys are aggregated to
/64.
CRS behavior
same_as_shapefile:- Preserve source CRS if present.
- If source CRS missing/unknown, keep coordinates as-is and mark projection as unknown in response/UI.
wgs84:- Reproject to EPSG:4326 (longitude, latitude axis order in outputs).
- If source CRS missing/unknown, fail with explicit error requiring known CRS.
- If source CRS is present but invalid (
.prjmalformed), fail with explicitinvalid_source_crs.
utm_wepppy_upper_left:- Determine target UTM zone from upper-left extent in WGS84 using the same approach as
wepppy/all_your_base/geo/geo.py(utm.from_latlon(north, west)using bbox top-left). - Use hemisphere from north coordinate sign (
north > 0=> northern hemisphere, else southern), consistent with current WEPPpy behavior. - If source CRS missing/unknown, fail with explicit error requiring known CRS.
- If source CRS is present but invalid (
.prjmalformed), fail with explicitinvalid_source_crs. - Fail with explicit
utm_not_supported_for_extentwhen extent is outside supported UTM latitude domain.
- Determine target UTM zone from upper-left extent in WGS84 using the same approach as
Output requirements
- GeoJSON output:
- When
target_crs=wgs84, emit RFC 7946-compatible GeoJSON. - When
target_crsis projected (for example UTM), emit non-RFC-7946 GeoJSON with coordinates in the selected CRS. - For projected GeoJSON, include explicit CRS details in API metadata and compatibility warning in
warnings. - For projected GeoJSON, use
application/jsoncontent type (notapplication/geo+json) to avoid claiming RFC 7946 compliance.
- When
- GeoParquet output:
- Write valid GeoParquet metadata (
geometadata key,primary_column,columns,crswhere known).
- Write valid GeoParquet metadata (
- UI/API metadata response must include:
detected_crs(authority/WKT when available)projection_status(known,unknown,invalid)attribute_schemaarray (name,type,width,precision,nullability_note)geometry_typesfeature_countbboxwarningsarray (for example: lossy-field mapping, null-value caveats, CRS/extent caveats)
WEPPcloud browser relay mode (json_body)
- Convert supports:
response_mode=download(default; unchanged behavior).response_mode=json_bodyonly whenoutput_format=geojson.
- Relay success behavior:
200 application/jsonresponse containingrequest_id,geojson, andmetadata.
- Relay validation behavior:
- Unsupported relay combinations (for example
output_format=geoparquet+response_mode=json_body) fail with canonical400 invalid_requestand expliciterror.details. - Unknown
response_modevalues fail with canonical400 invalid_request.
- Unsupported relay combinations (for example
- Human-facing converter UI remains download-oriented (
response_mode=download) and does not expose a relay-mode selector. json_bodyis an API-level mode for programmatic relay clients.- Browser client forwards resulting relay payload to WEPPcloud endpoint over authenticated WEPPcloud session/API.
- Relay CORS policy is allowlist-driven via
SHAPE_CONVERTER_RELAY_CORS_ALLOWED_ORIGINSand scoped to/v1/convert. - Relay mode keeps
.zipand shapefile sidecars processed/deleted inside shape-converter request scope. - Shape-converter does not require WEPPcloud credentials and does not call WEPPcloud directly.
Independent remediation contract (2026-11-14)
The operator authorized IPC, exact-driver, and archive fixes to proceed before the then-required sandbox feasibility decision. This historical sequencing removed independently confirmed weaknesses without waiving the controls then in force. The later Rust parser CLI amendment below supersedes that sandbox objective only after its cutover gates pass. Compression policy, semantic, and deployment gates remain open.
Parser IPC uses strict UTF-8 JSON with exactly properties, geometries,
source_bounds, and source_crs_wkt. Properties are objects with string keys and
only string, finite number, boolean, or null values. Geometry is null or a Point,
MultiPoint, LineString, MultiLineString, Polygon, or MultiPolygon object containing
exactly type and coordinates; positions have two or three finite numbers.
Preserve MultiPatch GeometryCollections recursively as exactly type and
geometries, aggregating the vertex cap across their members. During worker
normalization remove Fiona-only null geometries from coordinate geometries
and null coordinates from GeometryCollections.
Bounds are four finite numbers in min-x/min-y/max-x/max-y order, with ordered
extrema. CRS is null or a nonempty WKT string, parsed only after structural checks.
Property/geometry lengths must match and obey the existing feature limit; each
geometry obeys the existing vertex limit in both worker and parent.
The parser payload byte cap (default 16 MiB) remains authoritative; read at most cap + 1 bytes and reject oversize/non-regular/symlink payloads. Before JSON object allocation, scan for container nesting beyond 16 and more than 2,000,000 JSON values. Reject duplicate keys, trailing data, invalid UTF-8, non-finite numbers, and extra/missing fields. A duplicate-key detector may return plain JSON dictionaries only; no extensible object construction hooks are permitted. Worker output is atomically renamed within request scratch. Parser decode/schema errors use fixed messages without uploaded values or filesystem paths. The parent discards worker stdout/stderr rather than buffering untrusted diagnostics; exit codes 4 and 5 select the existing feature/vertex quota errors, and other nonzero exits select a fixed parse error. Standalone worker stderr stays generic. See ADR-0142.
The retained Fiona parity oracle asserts ESRI Shapefile immediately after its
allowlisted open and disables plugin discovery. This transitional rule is not a
production fallback; raw production Shapefile bytes reach only the Rust CLI.
For archive paths, normalize separators and Unicode NFC before case-folded
collision comparison. Preserve the normalized spelling on disk. Dataset selection
requires one logical NFC/case-folded prefix across all admitted shapefile sidecars,
including optional and incomplete prefixes; stripped .shp.xml/.qmd files
remain excluded. Reject ambiguity with invalid_shapefile. The expansion policy
uses absolute compressed-upload, total-uncompressed, member-count, scratch, and
parser-input caps plus streamed counters. A ratio-only threshold is intentionally
not adopted because it rejects small legitimate highly compressible datasets
without reducing the absolute resource ceilings; therefore no new numeric
parameter or ADR is introduced.
For users: archives must contain one dataset, and duplicate filename aliases or
additional sidecar prefixes are rejected. Very large valid conversions can now
reach the JSON value budget before the older byte/feature caps; errors are
explicit and no partial output is returned. Operators must ship parent and worker
together and must not interpret matching readiness mode strings as containment.
Compatibility and regression plan: this is request-internal IPC, with no persisted run-data migration. Parent and worker must ship together; legacy pickle is rejected without fallback. Public output keys/formats remain unchanged. Validate generated GeoJSON and GeoParquet through the actual worker and existing endpoint tests, plus hostile IPC/driver/archive tests. Broader shapefile semantics remain a later gate.
Dedicated image amendment (2026-11-14)
The operator authorizes a shape-converter-only image with an upgraded, coherent
GDAL/Fiona stack. This supersedes the dependency-version freeze only for this
service image; the common WEPPpy image remains unchanged. The split must preserve
the public API and Compose service identity while removing inherited application
secrets, datastore configuration, host data mounts, and unrelated runtime
packages. It is defense in depth and did not close issue #656 or authorize
deployment. Pin the base by digest and
install Python dependencies from a resolved hash-locked input. A Dockerfile-
specific allowlist must keep repository secrets and unrelated source out of the
build context. Qualification must inspect the resolved Compose service, final
image, and running container. Matching container/container readiness strings
remain non-evidence for sandbox containment. The later Rust parser CLI amendment
defines the final production readiness and #656 closure basis.
Starlette 1.3.1 is required in the dedicated image to close the form-limit
advisory affecting earlier releases. Register HTTP middleware explicitly through
add_middleware(BaseHTTPMiddleware, dispatch=...); the removed decorator API is
not a compatibility reason to retain a vulnerable Starlette release. Preserve
the existing abuse-control and relay-CORS middleware order and public behavior.
Rust parser CLI amendment (2026-11-14)
The production goal is replacement of GDAL/Fiona hostile Shapefile parsing, not
construction of a nested GDAL sandbox. The shape-converter owns one lightweight
Rust binary crate at
wepppy/microservices/shape_converter/rust/shape_parser_cli. Parser code remains
private to that crate; no separate shared Rust library is required. The
shape-converter Python process must launch the CLI as a short-lived subprocess;
it must never import the production parser through PyO3. The existing service,
container, public API, request scratch layout, and strict JSON payload contract
remain unchanged. This amendment does not authorize a parser service, socket,
queue, datastore, daemon, new container, privilege increase, or network protocol.
The launcher must create a process group, enforce hard wall time, terminate the whole group on timeout or cancellation, and apply independently tested memory, CPU, open-file, input-file, output-file, IPC-payload, feature, and vertex limits. Those limits apply before any attacker-controlled file traversal or record scan. The parser independently validates exact SHP/SHX/DBF record cardinality and SHX offset/length identity, and rejects multipart records unless the first part starts at zero and every later part offset is strictly increasing and within the declared point count. Feature admission stops at the ceiling rather than first building an oversize record list. Parser output remains untrusted and must pass the existing UTF-8 JSON byte, depth, value-count, duplicate-key, primitive, finite-number, and exact-schema validation before use. Panic, abort, malformed output, partial output, limit exhaustion, timeout, and cancellation must map to bounded canonical errors and leave no request residue. No production fallback to Fiona/GDAL or in-process PyO3 is allowed.
For conversion requests, archive extraction, DBF/CRS inspection, payload decoding, geometry construction/reprojection, serialization, and result encoding that can consume meaningful CPU or block on files runs outside the sole ASGI event-loop thread. Client cancellation does not release global admission or remove request scratch until any already-started blocking phase has ended. The process therefore keeps the one-request resource invariant even while draining a canceled bounded thread. Health remains responsive during a held conversion, and a competing conversion fails closed as saturated.
Startup validates the complete effective resource envelope, not only defaults. Production permits one active request globally and per client and rejects an environment override above the reviewed maxima for features, vertices, parser payload/input/memory/CPU/open files/time, scratch/precheck, or serialized output. Changing those maxima or concurrency requires a revised resource ADR and new exact-image near-ceiling evidence.
This CLI process boundary limits failure propagation and permits deterministic termination; it is not a claim of containment from arbitrary code execution. The CLI inherits the shape-converter container's UID, mounts, and network authority. That residual is acceptable only after dependency and unsafe-code review, malformed-input testing, oracle parity, exact-image validation, and independent security/operator acceptance. Until those gates pass, issue #656 remains open. After cutover, #656 may close as superseded by removal of the GDAL/Fiona failure class, not as evidence that a second sandbox was implemented.
The qualified GDAL 3.12.4/Fiona 1.10.1 image is a transitional parity oracle.
The final production image must contain shape_parser_cli, must attest the
binary and a real bounded parse during readiness, and must not contain or expose
Fiona/GDAL parser libraries or a PyO3 parser entry point. The executable ships
with adjacent /usr/local/bin/shape_parser_cli.json provenance using schema
shape-parser-binary-provenance-v1. The sidecar binds the binary hash, size,
version, ELF identity, source manifest/revision, Cargo lock, compiler, command,
and build time. Build validation and readiness fail closed if the pair is
missing, malformed, or mismatched. This metadata is not a publisher signature;
the immutable image digest remains the authenticity boundary.
Non-Functional Requirements
- Dedicated container/service (no embedding conversion in existing web app process).
- Sandboxed runtime:
- Non-root user.
- Read-only root filesystem.
no-new-privileges.- Drop all Linux capabilities except explicit minimum.
- Seccomp/AppArmor (or SELinux) enabled.
- Attach the service only to its dedicated internal Compose network so neither the API nor its Rust child has outbound routing. The child shares the container network namespace; this is not a separate network sandbox.
- Parse hostile Shapefile bytes only through the standalone Rust CLI with hard timeout, process-group termination, and explicit per-process resource limits.
- Treat parser-worker output as untrusted. Exchange data only through a
non-executable serialization with an explicit schema, byte limit, nesting
limit, and primitive-type allowlist.
pickle,marshal, YAML object tags, and equivalent executable deserialization are forbidden across this boundary. - During the parity transition only, disable GDAL driver/plugin autoload from attacker-writable paths and keep an explicit driver allowlist at every oracle open.
- Resource limits (initial defaults, tune with data):
- Request timeout: 30s metadata scan, 120s conversion hard timeout.
- Max ZIP upload size: 100 MB.
- Edge proxy body cap must include multipart overhead headroom above the ZIP limit so canonical app quota errors remain observable (current WEPP1 policy: 120 MB edge cap for 100 MB ZIP quota).
- Max total uncompressed bytes: 128 MiB.
- Max member count: 200.
- Max features: 1,000,000.
- Max vertices per feature: configurable guardrail (start 250,000).
- Multipart guardrails: max part count and max text-field size.
- Scratch-space guardrails: extraction/write free-space preflight and per-request scratch quota.
- Observability:
- Request ID, rejection reason codes, structured
parse_duration_ms, structuredconvert_duration_ms, cleanup success/failure. - Do not log raw uploaded content.
- Request ID, rejection reason codes, structured
Service Design
Deployment shape
- Service:
shape-converter. - Caddy utility namespace route:
/utils/shape-converter. - Caddy matcher is tightened to avoid prefix bleed (no
/utils/shape-converter*glob matching). - Implemented Caddy route shape:
@shape_converter_exact path /utils/shape-converter
redir @shape_converter_exact /utils/shape-converter/ 308
handle_path /utils/shape-converter/* {
request_body {
max_size 100MB
}
reverse_proxy shape-converter:8060 {
header_up X-Forwarded-Prefix /utils/shape-converter
header_up -Forwarded
header_up -X-Forwarded-For
header_up -X-Forwarded-Host
header_up -X-Forwarded-Proto
header_up X-Forwarded-For {remote_host}
header_up X-Forwarded-Host {host}
header_up X-Forwarded-Proto {scheme}
header_up Host {host}
transport http {
read_timeout 130s
write_timeout 30s
response_header_timeout 130s
}
}
}
- WEPP1-specific runtime keeps the same header-sanitization/timeouts policy with a
120MBedge body cap to preserve canonical app-level100MBZIP quota behavior under multipart overhead. - UI implementation is a minimal static page served by this service; WEPPcloud relay integration remains separate and consumes API contracts (
response_mode=json_body) rather than replacing the local UI flow.
Container specification (detailed)
Image and runtime
- Multi-stage image build:
- Stage 1: build the lockfile-pinned
shape_parser_clirelease binary and app dependencies. - Stage 2: minimal runtime image with the CLI and app code, without Fiona/GDAL parser dependencies.
- Stage 1: build the lockfile-pinned
- Run as non-root fixed UID/GID (for example
10001:10001). - Root filesystem read-only.
- Writable paths only via
tmpfsmounts:/tmpfor request scratch (archive, extraction, generated output)./run/shape-converterfor process/runtime state.
- Production compose runs without host path mounts; dev compose mounts the repo read-only for local iteration only.
Compose-level hardening contract
read_only: trueuser: "10001:10001"security_optincludesno-new-privileges:truecap_drop: [ALL]pids_limit(for example256)mem_limitandcpusmust be set (no unlimited defaults)tmpfsmounts usenoexec,nosuid,nodevoptions- Service is attached only to a dedicated internal network segment and proxied by Caddy
- East-west access to other app services is denied by network policy
- Outbound egress is deny-all (including DNS) unless explicitly allowlisted
- No Docker socket mount, no privileged mode, no host PID/IPC namespaces
- Runtime/image digest pinning is tracked as an operational hardening follow-up (not yet enforced directly in current compose manifests)
Health and lifecycle
GET /utils/shape-converter/health/livefor liveness.- Startup checks scratch writability, the exact
shape_parser_cliidentity, configured resource limits, a real bounded fixture parse, strict JSON validation, and cleanup. Startup fails if any check fails. GET /utils/shape-converter/health/readyreports the cached successful startup parser check plus a cheap current scratch check. Public readiness requests do not launch attacker-triggerable parser processes.- The transitional
SHAPE_CONVERTER_SANDBOX_MODEandSHAPE_CONVERTER_REQUIRED_SANDBOX_MODEequality check is not evidence of containment and must not be the final production readiness basis. - Graceful shutdown:
- Stop accepting new work.
- Wait for in-flight conversions up to shutdown timeout.
- SIGKILL any remaining conversion subprocess group after timeout.
- Startup self-check:
- Validate the adjacent executable provenance sidecar, binary SHA-256 and
size, and
shape_parser_cli --versionagreement. - Validate writable scratch root and cleanup permissions.
- Run and clean up a bounded parser fixture under the production launcher and reject readiness when any required limit or JSON validation is inactive.
- Validate the adjacent executable provenance sidecar, binary SHA-256 and
size, and
Performance and concurrency contract
- Production runs one Uvicorn application worker. Abuse-control counters and conversion metadata are intentionally process-local, so multiple workers would multiply configured limits and make metadata lookup nondeterministic. Scale-up requires a reviewed shared-state contract before increasing the worker count.
- Bound active requests with middleware abuse controls:
SHAPE_CONVERTER_MAX_INFLIGHT_GLOBAL(default1) with immediate503 service_saturatedwhen the single resource envelope is occupied.SHAPE_CONVERTER_MAX_INFLIGHT_PER_IP(default1) with immediate429 rate_limitedwhen that client already has an active request.- The single-flight defaults are part of the aggregate memory invariant. Do not raise either value without exact-image concurrent near-ceiling evidence and a revised cgroup/resource ADR.
SHAPE_CONVERTER_RATE_LIMIT_COUNT/SHAPE_CONVERTER_RATE_LIMIT_WINDOW_SECONDS(defaults120/60) for sliding-window request throttling.
- Per-request timeout:
- Metadata inspect timeout: 30 seconds.
- Conversion timeout: 120 seconds.
- Multipart body read timeout is enforced via
SHAPE_CONVERTER_BODY_READ_TIMEOUT_SECONDS(default15). - Edge proxy read/write/response-header timeouts enforce bounded request/response lifetime.
- Backpressure policy:
- No unbounded in-memory queue.
- Reject quickly with retry guidance once capacity is reached.
- Multipart guardrails: max part count and max field size.
Storage and quota contract
tmpfsscratch size is explicitly configured and must exceed max uncompressed archive bytes plus conversion headroom.- Service performs free-space preflight checks before extraction and before conversion write.
- Per-request scratch quota is enforced independently of global tmpfs size.
Worker lock-up prevention contract
- Never run heavy geospatial conversion on event loop threads.
- Execute hostile parsing through
shape_parser_cliand keep other heavy conversion steps out of the event loop, with:- Process-group creation.
- Hard wall-clock timeout.
- Kill entire process group on timeout/cancel.
- Direct memory, CPU, open-file, input/output-file, payload, feature, and vertex limits.
- Keep ASGI handlers async and lightweight:
- Upload validation.
- Job orchestration.
- Streaming response.
- Ensure all cleanup runs in
finallyblocks independent of client disconnect. - In production, Python must invoke the standalone Rust CLI and must not import the parser through PyO3. The CLI is a reliability and resource boundary within the existing hardened container, not a second arbitrary-code containment boundary.
Framework/runtime decision
Option comparison
| Option | Pros | Cons | Worker lock-up risk profile | Fit for this service |
|---|---|---|---|---|
| Starlette (Python) | Minimal ASGI layer, low overhead, already aligned with WEPPpy query-engine patterns, few dependencies |
More manual request validation/OpenAPI wiring than FastAPI | Low if conversion is subprocess-bound and concurrency is capped; high if blocking work leaks into event loop | Strong |
| FastAPI (Python) | Fast development, built-in validation/docs, based on Starlette/Pydantic | Slightly higher framework/validation overhead, extra abstraction for a very small API | Similar to Starlette for lock-up risk (risk is from blocking work, not framework choice) | Good but not necessary |
Go (net/http or chi/gin) |
Very strong concurrency model, low runtime overhead, each request served in goroutines | Additional service/runtime stack while parsing is already owned by the selected Rust CLI | Very low runtime lock risk, but operational complexity is higher in this codebase | Not justified for this control plane |
Final decision
- Choose Starlette for v1, with a strict subprocess execution model for geospatial work.
- Rationale:
- Throughput here is dominated by parser/IO and archive processing, not JSON framework overhead.
- Starlette keeps the control plane lean and matches existing repo ASGI patterns.
- Worker lock-ups are prevented by design through bounded subprocess execution and hard timeouts.
- This avoids introducing a new language/runtime stack prematurely.
- Revisit Go only if production data shows the ASGI control plane itself is a measurable bottleneck after subprocess isolation and concurrency tuning.
Processing pipeline
inspectcreates a request temp directory under isolated tmpfs mount.inspectsaves uploaded ZIP to temp dir.- Pre-scan ZIP central directory and validate all entries before extraction.
- Extract with canonical-path enforcement into temp dir only.
- Validate shapefile set and parse metadata.
- If this is
convert, perform requested reprojection/serialization and stream artifact. - In
finally, cleanup deletes:- Uploaded ZIP
- Extracted members
- Generated output artifact
- Request temp directory
- Janitor removes stale temp directories left only by abnormal process termination.
Cleanup Contract (Required)
- All request artifacts must be ephemeral and request-scoped.
- Deletion attempted on every code path (success, validation failure, conversion failure, timeout, client disconnect).
- Cleanup failures are logged as structured security events and retried by janitor.
- No artifact persistence after request terminal response.
API Contract (Implemented)
POST /utils/shape-converter/v1/inspect
Request:
archive(ZIP)- No auth credentials required.
Response 200:
request_iddetected_crsprojection_statusfeature_countgeometry_typesbboxattribute_schemawarnings
Error behavior:
- Canonical error payload with
error.code,error.message, anderror.details. request_idis included in error payloads.
POST /utils/shape-converter/v1/convert
Request:
archive(ZIP)output_format=geojson|geoparquettarget_crs=same_as_shapefile|wgs84|utm_wepppy_upper_leftresponse_modesupportsdownload|json_body(defaultdownload)response_mode=json_bodyis valid only withoutput_format=geojson- No auth credentials required.
Response 200:
response_mode=download: streamed artifact download + metadata sidecar JSON endpoint keyed byrequest_idresponse_mode=json_body: JSON payload with:request_idgeojsonmetadata
Download headers:
X-Shape-Converter-Request-IdX-Shape-Converter-Metadata-PathContent-Dispositionattachment filename (RFC 5987-compatible encoding)
GET /utils/shape-converter/v1/convert/metadata/{request_id}
Response 200:
- Convert metadata for download-mode conversions (
output_format,target_crs,projection_status,detected_crs,attribute_schema, geometry summary, warnings, output CRS details, and download metadata fields).
Response 404:
- Canonical
metadata_not_foundwhen request metadata has expired or is unknown.
Error response contract:
- Follow canonical WEPPpy error payload style (
error.code,error.message,error.detailsrequired). - Use explicit codes like:
metadata_not_foundrate_limitedrequest_timeoutinvalid_requestinvalid_archivearchive_path_traversalarchive_quota_exceededmissing_required_sidecarinvalid_shapefileinvalid_source_crsunknown_source_crsreprojection_failedservice_saturatedutm_not_supported_for_extent
- Error status mapping (minimum):
400: validation/input errors.404: metadata sidecarrequest_idnot found.408: request body/processing timeout.413: upload/body/expansion quota exceeded.429: rate-limit or concurrency saturation.503: service saturation and scratch-capacity exhaustion.500: unexpected internal failure.
ZIP Upload Risk Matrix (Comprehensive)
| Risk | Why it matters | Required mitigations | Required tests |
|---|---|---|---|
Path traversal (../, absolute paths, drive paths) |
Write outside extraction root | Canonicalize + commonpath enforcement per member; reject violators |
Archive with ../../etc/passwd, /tmp/x, C:\\... rejected |
| Zip Slip variants via encoded names | Bypass naive prefix checks | Normalize Unicode + separators before path checks | Mixed separator + encoded traversal payloads rejected |
| Symlink entries | Indirect traversal/overwrite | Reject symlink members and special file types | ZIP containing symlink entry rejected |
| Duplicate member names | Overwrite confusion | Reject duplicate normalized extraction paths | ZIP with duplicate filenames rejected deterministically |
| Case-collision names | Cross-platform overwrite ambiguity | Normalize policy to lowercase compare for collision detection | Road.shp + road.shp rejected |
| ZIP bombs / high compression ratio | Disk and memory exhaustion | Enforce absolute compressed, total-uncompressed, member, scratch, and parser-input caps while streaming with counters; do not rely on a ratio threshold | High-expansion fixtures crossing an absolute ceiling are rejected before parser execution; valid compressed controls remain accepted |
| Excessive member count | Inode/CPU exhaustion | Max member cap | ZIP with > cap rejected |
| Nested archives | Recursive bomb/smuggling | Reject nested archives in v1 | ZIP containing .zip members rejected |
| Encrypted/password ZIPs | Uninspectable content, operational failures | Reject encrypted flag entries | Encrypted ZIP rejected with explicit error |
| Unsupported compression methods | Parser exceptions/DoS | Allowlist supported compression methods only | Unsupported compression method returns validation error |
| Corrupt CRC/central directory | Crash/undefined behavior | Validate archive integrity before extraction | Truncated/corrupt ZIP rejected |
| Oversize filenames/path depth | FS/path handling failures | Enforce max filename length and max path depth | Very deep path archive rejected |
| Hidden/system files in archive | Ambiguous behavior | Allowlist required extensions and optional sidecars only; sanitize known metadata sidecars (.shp.xml, .qmd) by immediate unlink |
.exe, .dll, .py members rejected |
| Null-byte or control-char filenames | Sanitization bypass | Reject non-printable/control chars and null bytes | Crafted control-char names rejected |
| ZIP metadata trust issues | Spoofed types | Validate extension + signature + entry policy; exact-driver restricted open remains authoritative | Renamed VRT/HDF/other format with .shp extension is rejected before data use |
| Archive parser dependency CVEs | Parser compromise risk | Keep libraries patched, run in sandbox, pin versions | Dependency audit gate + CVE regression check |
Shapefile Risk Matrix (Comprehensive)
| Risk | Why it matters | Required mitigations | Required tests |
|---|---|---|---|
Missing required sidecars (.shp/.shx/.dbf) |
Cannot reliably read dataset | Explicit required-file validation | Omit each required sidecar -> explicit failure |
| Multiple shapefile prefixes in one ZIP | Ambiguous conversion target | Reject unless explicit dataset selection implemented | ZIP with two prefixes rejected |
Missing .prj |
Unknown CRS; reprojection unsafe | Mark unknown for inspect; block CRS transforms requiring known source | Missing .prj + wgs84/utm request fails |
Invalid .prj WKT |
Bad reprojection | Parse CRS strictly; return invalid_source_crs |
Corrupted .prj rejected |
DBF encoding ambiguity (.cpg/LDID issues) |
Attribute corruption | Detect/report source encoding; allow explicit override in future | Non-UTF8 DBF fixture preserved/flagged correctly |
| 10-char field name truncation collisions | Schema ambiguity/data loss | Report original + normalized names when possible; fail on unresolved duplicates | Collision fixture handled deterministically |
| Null-value semantics in DBF/shapefile | Silent value substitution | Surface null/substitution behavior in metadata warnings | Dataset with sentinel null-like values flagged |
| Attribute width/precision limits | Rounding/truncation | Preserve types carefully; emit warnings for lossy mappings | Large numeric precision fixture warning/assertion |
| Mixed/unsupported geometry content | Conversion failure or wrong output | Validate geometry types and null geometries before write | Mixed geometry fixture produces explicit policy error |
| Invalid geometries (self-intersections, ring issues) | Output unusable or crashes | Run geometry validity checks; optionally repair under explicit flag (off by default) | Invalid polygon fixture returns explicit error |
| Multipart/multipatch edge cases | Type coercion surprises | Normalize and report resulting geometry types | Multipatch/multipart fixtures validated and reported |
| Massive vertex counts | CPU/memory exhaustion | Vertex and complexity guardrails | High-vertex polygon fixture rejected by threshold |
Record count mismatch (.shp vs .dbf) |
Corrupt feature/attribute mapping | Cross-check record counts and fail if inconsistent | Tampered sidecar counts rejected |
| Malformed binary structures | Native parser crash risk | Parse in hardened sandbox process and fail safely | Fuzz corpus malformed files do not crash host |
| Out-of-range coordinates | Projection errors | Validate coordinate ranges by CRS before transform | Extreme-coordinate fixtures fail clearly |
| Anti-meridian/multi-zone extents | UTM ambiguity | Document UL-corner UTM behavior and emit warning if bbox spans zones | Multi-zone fixture warns, still deterministic |
| Shapefile size/component limits | Operational failures | Enforce file-size and feature-count caps before full load | Over-limit fixtures rejected early |
XML sidecar entity expansion (.xml, .gml, .shp.xml) |
CPU/memory DoS via parser | Reject generic XML sidecars; if .shp.xml is present, delete it immediately after extraction and enforce parser timeout + memory caps |
Generic XML sidecar rejected; .shp.xml stripped with warning |
Parser-triggered remote fetch (/vsicurl/, remote references) |
Unwanted network egress/data exfil risk | Keep deny-all egress and disable remote parser sources by policy/config | Fixture attempting remote reference fails without outbound access |
Metadata PII leak from metadata sidecars (.shp.xml, .qmd) |
Exposure of usernames, contact details, host paths, org/process history | Never parse or return sidecar metadata; sanitize by immediate unlink; keep warnings/log fields free of raw sidecar content | Fixture containing PII in sidecar yields no PII in API payloads/logs |
| Executable parser-worker IPC | Native parser compromise crosses back into the parent process | Use schema-validated, data-only serialization; never deserialize pickle or object-capable formats from the worker | Malicious worker payload is rejected without code execution or filesystem side effects |
CRS Rules and UTM Determination Details
- Source of truth for UTM mode should mirror existing WEPPpy logic in
wepppy/all_your_base/geo/geo.py:- Determine UTM zone from upper-left bbox corner (north, west).
- Force lower-right corner into the same zone.
- Set hemisphere from sign of north latitude.
- Practical implementation for converter:
- Compute source bbox.
- Transform bbox to WGS84 if needed.
- Use upper-left (
max_lat,min_lon) to compute UTM zone. - Build target CRS EPSG:
326xxfor north,327xxfor south. - Reproject all features.
UI Requirements
- Layout and control contract:
- One controls panel titled
Uploadwith:- one archive file input
- output format selector
- target CRS selector
- primary
Convert And Downloadbutton - secondary
Inspect Uploadbutton
Warningspanel appears immediately after theUploadpanel.Projection,Geometry Summary, andAttribute Schemapanels follow warning/error content.- All user-facing panels are the same width in the default layout.
- Static asset URLs must resolve correctly behind proxied namespace
/utils/shape-converter/(no root-relative asset assumptions).
- One controls panel titled
- Visibility contract:
- Initial load:
- show
Uploadpanel - hide
Warnings,Projection,Geometry Summary, andAttribute Schema
- show
- Show
Projection,Geometry Summary, andAttribute Schemaonly after a successful inspect payload provides content. - Keep
Warningshidden until there is warning/error/advisory content to show.
- Initial load:
- Metadata rendering contract:
- Projection panel:
- Detected CRS (authority or unwrapped WKT when authority is unavailable)
- Projection status badge (
known,unknown,invalid) - Output CRS (after convert metadata is available)
- Geometry summary:
- Feature count
- Geometry types
- BBox
- Attribute schema table:
- Column name
- Source type
- Width/precision
- Nullability note (if inferable)
- Projection panel:
- Interaction contract:
- UI may reuse one selected local archive file for both actions, but each
Inspect/Convertclick sends a separate request. - Convert UI is download-oriented and submits
response_mode=download. - Relay
response_mode=json_bodyis available via API for programmatic clients and is not presented as a default end-user control.
- UI may reuse one selected local archive file for both actions, but each
- Error UX contract:
- All API/network errors must be observable by users (no silent failures).
inspect-statusandconvert-statusmessages must use plain language with specific next-step guidance.429/503abuse-control states must include clear retry guidance and displayRetry-Afterwhen present.
- Keep messages explicit; no silent fallback behavior.
Security Controls Baseline
- Implemented baseline controls:
- OWASP-style upload defense-in-depth: extension allowlist, signature checks, strict size/runtime limits, sandboxing, and abuse controls for anonymous access.
- Container/runtime hardening in compose: non-root user,
read_only,no-new-privileges,cap_drop=ALL, explicit pid/memory/cpu limits, hardened tmpfs mounts. - No Docker socket mount, no privileged mode, and isolated internal network segment.
- Public-service edge controls:
- Edge request body size cap.
- Edge upload/read/write/response-header timeout controls.
- Edge forwarding-header sanitization and trusted-proxy configuration.
- CORS policy for browser relay use-cases is required for
json_bodymode. - Application-level controls (authoritative when edge lacks native rate-limit module support):
- Per-IP rate/concurrency controls in service middleware.
- Per-request scratch quotas and timeout cancellation hooks.
- Metadata sanitization and length bounds before serialization/logging.
- Program/release controls retained for future hardening:
- Runtime image digest pinning.
- SBOM generation for release builds.
- Vulnerability scan gate that blocks unresolved High/Critical issues in reachable runtime components.
- Parser dependency controls:
- Track and triage the Rust parser's direct and transitive dependencies, unsafe-code surface, advisories, licenses, and malformed-input behavior before each release. Record the exact Cargo lock and binary build provenance.
- Maintain explicit timeout/kill and resource-limit behavior for parser panic, non-termination, allocation, file, and output exhaustion classes.
- Treat archive extraction and parser process separation as implementation details, not security boundaries; archive and parser inputs still require full validation and resource controls.
- Maintain metadata-privacy policy that forbids exposing raw
.shp.xmlcontent in API payloads, logs, or warnings (only advisory removal notices are allowed). - Retain the dedicated low-privilege identity, restricted filesystem, bounded CPU/memory/files/output, and internal network for the container. The Rust CLI inherits those authorities and is not a second sandbox. GDAL-specific driver/plugin restrictions apply only to the transitional parity oracle.
Test Plan
Unit tests
- ZIP entry path canonicalization and collision handling.
- Absolute compressed/uncompressed/member/scratch quota enforcement, including high-expansion rejection and valid compressed controls.
- CRS option validation matrix.
- UTM zone function parity with WEPPpy upper-left rule.
Integration tests
- Valid uploads for point/line/polygon shapefiles.
- Missing sidecars and invalid
.prjflows. - GeoJSON and GeoParquet output metadata correctness.
- Browser relay flow correctness (
response_mode=json_body) including success payload shape and invalid-combination failures. - Cleanup verification: no residual files after request terminal state.
Security tests
- Zip-Slip fixtures.
- Zip bomb fixtures.
- Encrypted ZIP fixtures.
- Malformed shapefile fuzz corpus.
- Parser timeout and cancellation behavior.
- Disguised VRT/HDF/other non-shapefile content renamed
.shpis rejected by the exact-driver boundary. - Malicious or malformed parser-worker output cannot trigger executable deserialization in the parent.
- XML sidecar bomb fixtures (
.shp.xml/.xmlentity expansion class payloads). - Parser non-termination fixtures (malformed WKB parser-loop class).
- Metadata privacy fixtures:
.shp.xmland.qmdcontaining usernames/paths/contact fields are not exposed in responses/logging. - Runtime hardening verification:
read_only,cap_drop=ALL,no-new-privileges, seccomp/AppArmor/SELinux, pids/mem/cpu limits, deny-all egress.
- Slowloris and pinned-download resilience:
- Slow upload/body timeout enforcement.
- Slow download/write timeout cleanup behavior.
- Anonymous abuse controls:
- Endpoints reachable without auth.
- Proxy-validated IP rate limits return expected
429. - Concurrent flood beyond caps does not starve healthy traffic.
Performance tests
- Large-but-valid shapefile throughput.
- Concurrent conversion requests under CPU/memory limits.
Acceptance Criteria
- Users can upload ZIP shapefiles and download GeoJSON/GeoParquet reliably.
- Service is publicly accessible without authentication by design.
- UI implements the single
Uploadcontrols panel (one archive input + output/CRS selectors + primary convert/secondary inspect actions). Warningspanel is positioned directly afterUploadand is hidden until warning/error/advisory content exists.Projection,Geometry Summary, andAttribute Schemaremain hidden until successful inspect metadata is available.- Convert API supports both
response_mode=downloadand relayresponse_mode=json_body(GeoJSON-only), with explicit canonical 4xx errors for unsupported combinations; the public UI remains download-oriented. - Attribute schema rendering includes nullability notes with explicit fallback text when nullability is not inferable from DBF metadata.
- CRS options behave exactly as specified (including WEPPpy UL-corner UTM mode and explicit out-of-domain failures).
- Invalid
.prjreprojection paths return explicitinvalid_source_crswhile missing CRS remainsunknown_source_crs. - GeoJSON behavior is explicit: RFC 7946 in WGS84 mode and clearly labeled non-RFC projected mode for UTM/same-CRS output.
- ZIP and shapefile risk tests are implemented and passing.
- Native parser opens are exact-driver restricted and parser-worker IPC is schema-validated and non-executable.
- XML sidecar and parser-loop abuse tests are implemented and passing.
- No sidecar-derived PII (for example
.shp.xml/.qmdusernames, contacts, file paths) is exposed by API metadata or logs. - When
.shp.xmlis included, API warnings explicitly report removal and advise that packing.shp.xmlin shapefile ZIPs is generally not advisable. - When
.qmdis included, it is sanitized via immediate unlink and never parsed or surfaced in API metadata. - No uploaded or generated artifacts persist after each request terminal response (verified by tests).
- Service runs in its own hardened container and is reachable through Caddy.
- Production contains no GDAL/Fiona or PyO3 parser fallback. It uses only the bounded Rust CLI and fails startup when executable provenance or its real parser canary fails. If deployment cannot establish that state, deployment stops shape-converter and verifies that it is not running; it does not restore an older parser implementation.
- CI/security validation proves container hardening and runtime abuse controls are active.
Source Notes
- Existing WEPPpy UTM upper-left behavior:
wepppy/all_your_base/geo/geo.pylines 808-823. - Internal prior risk evidence for ZIP/shapefile flow:
docs/culvert-at-risk-integration/audits/culvert-web-app-codebase-audit-2026-02-20.mdlines 116-127 and 152-167.
External References
- OWASP File Upload Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
- Python
zipfiledocs (extract warnings, traversal notes, decompression pitfalls): https://docs.python.org/3/library/zipfile.html - NVD CVE-2024-55587 (archive extraction traversal class example): https://nvd.nist.gov/vuln/detail/CVE-2024-55587
- GDAL security model for untrusted input: https://gdal.org/en/stable/user/security.html
- GDAL ESRI Shapefile driver docs: https://gdal.org/en/stable/drivers/vector/shapefile.html
- GDAL releases: https://github.com/OSGeo/gdal/releases
- GDAL issue #14039 (WKB parser infinite-loop class): https://github.com/OSGeo/gdal/issues/14039
- ESRI Shapefile Technical Description: https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf
- ArcGIS shapefile output limitations: https://pro.arcgis.com/en/pro-app/latest/tool-reference/appendices/geoprocessing-considerations-for-shapefile-output.htm
- GeoParquet 1.1.0 specification: https://geoparquet.org/releases/v1.1.0/
- OWASP Docker Security Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html
- gVisor architecture/security intro: https://gvisor.dev/docs/architecture_guide/intro/
- Starlette homepage/docs: https://www.starlette.io/
- FastAPI features: https://fastapi.tiangolo.com/features/
- Go
net/httppackage docs: https://pkg.go.dev/net/http