feat(server,server-client): a streaming cache hit by hash, without re-uploading the file (#3907)

Closes #3901.

`POST /parse/parquet-stream` accepts an optional `sha256` query parameter (the same `ParseQuery` mechanism as `parquet_layout`, so it composes with layout and quality in the cache identity). With no body and a hash, the route validates the shape (400 otherwise), builds the key and delegates to the same `try_cached_replay` the upload path uses, so "the entries a hit needs" is one list; a hash-only miss answers 404 like `GET /cache/check`; a body present ignores the hash entirely, so a client hash can only select entries that already exist. `parseParquetStream` in `@ifc-lite/server-client` hashes locally, probes, and uploads on any answer that is not a replay (a pre-change server's 400 included); `skipCacheProbe` opts out. The SSE reader moved to its own module so probe and upload share one loop, and `ParquetStreamEvent` moved to `stream_event.rs` to keep the route under the ratchet without an allowlist row.

Mutation checks: disabling the probe hit path fails the no-upload test; narrowing the fallback set to 404 fails all five degradation cases; removing the non-404 passthrough fails the 500 test. Not measured against a real server or a 40 MB file; the saving is argued from the code path.

Gates on the head: `cargo test -p ifc-lite-server` 269 pass, clippy -D warnings 0, module_size_ratchet 0, typecheck 0, server-client 91 pass, check-module-size 0, check-changesets 0, docs:check-samples 0, api-surface regenerated. Changesets: `@ifc-lite/server-bin` minor, `@ifc-lite/server-client` minor.

Author: louistrue

.changeset/stream-hit-by-hash.md

@@ -0,0 +1,21 @@
+---
+'@ifc-lite/server-bin': minor
+'@ifc-lite/server-client': minor
+---
+
+Streaming cache hits no longer pay for the upload.
+
+`POST /api/v1/parse/parquet-stream` keys on the SHA-256 of the bytes it
+receives, so the whole file had to arrive before the cache could be consulted.
+On a 40 MB model that upload was the entire cost of a hit. The route now also
+accepts `?sha256={hex}` with no request body: if everything the replay needs is
+already cached it streams it back, and otherwise answers 404 meaning "send the
+file". A hash that arrives alongside a body is ignored, so the received bytes
+still decide which entry is read and written.
+
+`parseParquetStream` in `@ifc-lite/server-client` hashes the file locally and
+probes before uploading. The probe degrades to the upload whenever it is not
+answered, so pointing an upgraded client at a server that predates this change
+still works. This also makes a hit progressive: it used to fetch the
+whole model through `/cache/geometry` and hand it over as one batch. Pass
+`{ skipCacheProbe: true }` to upload straight away.

apps/server/src/routes/parse/cache_keys.rs

@@ -37,6 +37,24 @@ pub(crate) fn cache_key_from_parts(
     )
 }
 
+/// Whether `hash` has the shape [`DiskCache::generate_key`] produces: 64
+/// lowercase hex characters.
+///
+/// Lives beside [`cache_key_from_parts`] because that is what it protects. The
+/// hash a client supplies is concatenated into `{hash}-{filter}{quality}` and
+/// the namespace suffix (`-parquet-v5`, `-datamodel-v6`, ...) is appended after
+/// it, so a caller-shaped string is a caller-shaped cache key. Checking the
+/// shape keeps the value to the one job it has, naming a file.
+///
+/// Applied by the hash-only stream probe (#3901). The two older hash-taking
+/// endpoints, `check_cache` and `get_cached_geometry`, do NOT call it yet: a
+/// malformed hash there names a key nobody wrote and gets a 404, which is a
+/// correct answer by a different route. Tightening them is a behaviour change
+/// to a published surface and is deliberately not part of #3901.
+pub(crate) fn is_file_digest(hash: &str) -> bool {
+    hash.len() == 64 && hash.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f'))
+}
+
 /// Request-level cache key: file hash + opening-filter suffix + quality suffix.
 pub(crate) fn request_cache_key(data: &[u8], query: &ParseQuery, quality: TessellationQuality) -> String {
     cache_key_from_parts(
@@ -187,6 +205,22 @@ pub(crate) async fn has_current_data_model(cache: &DiskCache, cache_key: &str) -
     has_entry(cache, &data_model_cache_key(cache_key)).await
 }
 
+/// Whether the Parquet metadata header is cached for `cache_key`.
+///
+/// The cheap half of "is this replayable": the header is a few hundred bytes,
+/// where the geometry blob is the whole model. The hash-only stream probe
+/// (#3901) asks this, plus [`has_current_data_model`], BEFORE it takes an
+/// admission slot, so the common miss (a file the server has never seen) is
+/// answered by two small reads rather than by charging a parse slot for a disk
+/// lookup. It is a pre-filter, never the decision: [`try_cached_replay`] still
+/// makes that, and a metadata entry present here with no geometry beside it
+/// falls through to the same 404.
+///
+/// [`try_cached_replay`]: super::cached_replay::try_cached_replay
+pub(crate) async fn has_parquet_metadata(cache: &DiskCache, cache_key: &str) -> bool {
+    has_entry(cache, &parquet_metadata_key(cache_key)).await
+}
+
 /// Whether `key` has a readable entry.
 ///
 /// Reads the value rather than asking `DiskCache::has`, which is an index

apps/server/src/routes/parse/cache_keys_tests.rs

@@ -267,9 +267,18 @@ fn request_cache_key_separates_content_filter_and_quality() {
                 opening_filter: mode,
                 tessellation_quality: None,
                 parquet_layout: ParquetLayout::Flat,
+                // A client-supplied hash is a SELECTOR for the hash-only
+                // stream probe (#3901); it is not part of the cache identity
+                // a body-carrying request builds. Set to a value that would
+                // be visible if it leaked in.
+                sha256: Some("f".repeat(64)),
             };
             let key = request_cache_key(data, &query, quality);
             assert!(key.starts_with(&hash), "the file hash must lead the key: {key}");
+            assert!(
+                !key.contains(&"f".repeat(64)),
+                "request_cache_key must key off the bytes, never the query hash: {key}"
+            );
             assert!(
                 seen.insert(key.clone()),
                 "collision: {mode:?}/{quality:?} reuses an existing key {key}"

apps/server/src/routes/parse/cached_replay.rs

@@ -18,11 +18,14 @@
 //! still monotonic and still ends at its own stated total.
 
 use super::cache_keys::{
-    has_current_data_model, load_cached_symbolic, parquet_geometry_key, parquet_metadata_key,
+    cache_key_from_parts, has_current_data_model, has_parquet_metadata, is_file_digest,
+    load_cached_symbolic, parquet_geometry_key, parquet_metadata_key,
 };
+use super::ParseQuery;
+use ifc_lite_processing::TessellationQuality;
 use crate::services::ParquetLayout;
 use super::parquet::ParquetMetadataHeader;
-use super::parquet_stream::ParquetStreamEvent;
+use super::stream_event::ParquetStreamEvent;
 use super::stream_progress::load_stream_progress;
 use crate::error::ApiError;
 use crate::services::parquet_replay_batches::split_into_batches;
@@ -32,6 +35,83 @@ use axum::response::IntoResponse;
 use base64::{engine::general_purpose::STANDARD, Engine};
 use std::convert::Infallible;
 
+/// Serve `POST /api/v1/parse/parquet-stream` from a client-supplied file hash,
+/// with no request body at all (issue #3901).
+///
+/// The upload used to be unavoidable on a hit: the cache key is the SHA-256 of
+/// the RECEIVED bytes, so `extract_file` had to finish before the cache could
+/// be consulted, and on a 40 MB model over a real connection that upload was
+/// the whole cost of the hit. A client that hashes locally can name the entry
+/// instead.
+///
+/// The hash SELECTS; it never asserts. Everything the replay needs has to be on
+/// disk under that key already, which is exactly what [`try_cached_replay`]
+/// checks (geometry body, metadata header, a data model at the current payload
+/// version, plus whatever it later adds). Delegating to it rather than
+/// re-listing those entries here is what makes "a hash-only hit replays the
+/// same events as an upload hit" true by construction instead of by two lists
+/// agreeing. A miss answers `404`, the status
+/// `GET /api/v1/cache/check/{hash}` already uses for "upload it", and the
+/// client uploads.
+///
+/// A MISS costs no admission slot. The probe answers it from two small reads
+/// (the metadata header and the data-model marker) before touching the gate,
+/// because the common miss is a file the server has never seen and charging a
+/// parse slot for a disk lookup would mean every cold-cache client wins
+/// admission twice, probe then upload, and could be shed on the probe rather
+/// than queueing for the upload that would have succeeded.
+///
+/// A HIT does take one, around the replay BUILD, dropped before the response is
+/// returned. That is what the body-carrying hit path does, and for the same
+/// reason: `try_cached_replay` reads the whole geometry blob into memory,
+/// base64-encodes every batch, and materializes the full event vector before a
+/// byte is sent, which is several times the model's geometry resident at once.
+/// Leaving that window unbounded would make the cheapest request a client can
+/// send the cheapest way to exhaust the server. Draining the finished stream
+/// needs no slot.
+pub(super) async fn replay_by_client_hash(
+    state: &AppState,
+    query: &ParseQuery,
+    quality: TessellationQuality,
+    sha256: &str,
+) -> Result<axum::response::Response, ApiError> {
+    if !is_file_digest(sha256) {
+        return Err(ApiError::BadRequest(
+            "sha256 must be a 64-character lowercase hex SHA-256 digest".to_string(),
+        ));
+    }
+    let cache_key = cache_key_from_parts(sha256, query.opening_filter, quality);
+
+    let replay = if has_parquet_metadata(&state.cache, &cache_key).await
+        && has_current_data_model(&state.cache, &cache_key).await
+    {
+        let admission_guard = state
+            .admission
+            .acquire(state.config.max_file_size_mb as u64 * 1024 * 1024)
+            .await?;
+        let replay = try_cached_replay(state, &cache_key, query.parquet_layout).await;
+        drop(admission_guard);
+        replay?
+    } else {
+        None
+    };
+
+    if let Some(response) = replay {
+        tracing::info!(
+            cache_key = %cache_key,
+            "Streaming cache HIT by client-supplied hash - no upload"
+        );
+        return Ok(response);
+    }
+    tracing::debug!(
+        cache_key = %cache_key,
+        "Hash-only stream request has nothing cached; asking the client to upload"
+    );
+    Err(ApiError::NotFound(format!(
+        "Nothing cached for sha256 {sha256} under this opening_filter / tessellation_quality / parquet_layout. Resend the request with the multipart file body."
+    )))
+}
+
 /// Return the geometry slice from a cached combined-Parquet blob, framed as
 /// `[geometry_len: u32-LE][geometry_data][data_model_len: u32]...`. Returns
 /// `None` (rather than panicking) when the blob is too short to hold the length

apps/server/src/routes/parse/mod.rs

@@ -6,6 +6,7 @@
 
 mod cache_keys;
 mod cached_replay;
+mod stream_event;
 mod stream_progress;
 mod fetch;
 mod json;
@@ -45,6 +46,19 @@ pub struct ParseQuery {
     /// takes this struct, so the signal travels with the cache identity.
     #[serde(default)]
     pub parquet_layout: ParquetLayout,
+    /// SHA-256 of the file the client is asking about, hex, lowercase (#3901).
+    ///
+    /// Read only by `POST /api/v1/parse/parquet-stream`, and only when the
+    /// request carries no multipart body: see
+    /// [`cached_replay::replay_by_client_hash`] for what it does and what it
+    /// is not allowed to do. It lives on this struct rather than in a header
+    /// so it travels with the rest of the cache identity (`opening_filter`,
+    /// `tessellation_quality`, `parquet_layout`) through the one place every
+    /// parse route already parses. A hash paired with the wrong layout names a
+    /// different entry, and splitting one identity across two transports is how
+    /// such pairings drift apart.
+    #[serde(default)]
+    pub sha256: Option<String>,
 }
 
 impl ParseQuery {

apps/server/src/routes/parse/parquet_stream.rs

@@ -9,56 +9,19 @@ use super::cache_keys::{
     request_cache_key,
 };
 use super::parquet::ParquetMetadataHeader;
+use super::stream_event::ParquetStreamEvent;
 use super::stream_progress::{cache_stream_progress, StreamProgressRecorder};
 use super::{extract_file, ParseQuery};
 use crate::error::ApiError;
 use crate::services::{extract_data_model, process_streaming, serialize_data_model_to_parquet};
-use crate::types::{ModelMetadata, ProcessingStats, StreamEvent};
+use crate::types::StreamEvent;
 use crate::AppState;
 use axum::{
     extract::{Multipart, Query, State},
     response::sse::{Event, KeepAlive, Sse},
 };
-use ifc_lite_processing::SymbolicData;
-use serde::Serialize;
 use std::convert::Infallible;
 
-/// SSE event types for Parquet streaming.
-// Variant sizes differ because the payload events carry buffers; boxing them
-// would complicate the SSE serialization path for no runtime benefit here.
-#[allow(clippy::large_enum_variant)]
-#[derive(Debug, Clone, Serialize)]
-#[serde(tag = "type", rename_all = "lowercase")]
-pub enum ParquetStreamEvent {
-    /// Initial event with estimated totals.
-    Start {
-        total_estimate: usize,
-        cache_key: String,
-    },
-    /// Progress update.
-    Progress { processed: usize, total: usize },
-    /// Batch of geometry data as base64-encoded Parquet.
-    Batch {
-        /// Base64-encoded Parquet data containing this batch's meshes.
-        data: String,
-        /// Number of meshes in this batch.
-        mesh_count: usize,
-        /// Batch sequence number (1-indexed).
-        batch_number: usize,
-    },
-    /// Processing complete.
-    Complete {
-        stats: ProcessingStats,
-        metadata: ModelMetadata,
-        /// 2D symbol data extracted from `IfcAnnotation` and `IfcGrid`
-        /// entities — parity with `POST /api/v1/parse` (issue #900).
-        #[serde(default, skip_serializing_if = "SymbolicData::is_empty")]
-        symbolic_data: SymbolicData,
-    },
-    /// Error occurred.
-    Error { message: String },
-}
-
 /// POST /api/v1/parse/parquet-stream - Streaming parse with Parquet batches.
 ///
 /// Returns SSE events with Parquet-encoded geometry batches for progressive rendering.
@@ -72,17 +35,53 @@ pub enum ParquetStreamEvent {
 /// - `error`: Error event with `message`
 ///
 /// After `complete`, client should fetch data model via `/api/v1/data-model/{cache_key}`.
+///
+/// ## Two ways to ask
+///
+/// With a multipart `file` body: the normal path. The cache key is the SHA-256
+/// of the RECEIVED bytes, so the upload always completes first, and a `sha256`
+/// query parameter sent alongside a body is IGNORED - the bytes on the wire
+/// decide which entry is read and written, never the client's claim about them.
+///
+/// With `?sha256={hex}` and no body: a probe (issue #3901). It replays the
+/// cached stream if, and only if, every entry the replay needs already exists
+/// under that key, and otherwise answers `404` meaning "upload it". See
+/// [`cached_replay::replay_by_client_hash`]. This is what lets a 40 MB cache
+/// hit cost no upload.
 pub async fn parse_parquet_stream(
     State(state): State<AppState>,
     Query(query): Query<ParseQuery>,
-    mut multipart: Multipart,
+    multipart: Option<Multipart>,
 ) -> Result<axum::response::Response, ApiError> {
     use crate::services::{serialize_batch_with_layout, StreamingParquetCacheWriter};
     use axum::response::IntoResponse;
     use base64::{engine::general_purpose::STANDARD, Engine};
     use futures::StreamExt;
     use std::sync::{Arc, Mutex};
 
+    let tessellation_quality = query.resolved_tessellation_quality()?;
+
+    // Hash-only probe: no body was sent, so there is nothing to extract and
+    // nothing to parse. It is checked before the gate below only because that
+    // gate reserves an upload that does not exist. The probe takes admission
+    // itself, on the hit path where it has real work to bound -- see
+    // `replay_by_client_hash`.
+    let Some(mut multipart) = multipart else {
+        let Some(sha256) = query.sha256.as_deref() else {
+            // No body and no hash: there is nothing to identify a file with.
+            // `MissingFile` (400) is what a body with no `file` field already
+            // answers, and it says the same thing here.
+            return Err(ApiError::MissingFile);
+        };
+        return super::cached_replay::replay_by_client_hash(
+            &state,
+            &query,
+            tessellation_quality,
+            sha256,
+        )
+        .await;
+    };
+
     // Extract file
     // Admission gate (bounded concurrency + byte budget): acquired BEFORE the
     // upload is buffered, reserving the max upload size since multipart rarely
@@ -94,8 +93,10 @@ pub async fn parse_parquet_stream(
         .await?;
     let data = extract_file(&mut multipart, state.config.max_file_size_mb).await?;
 
-    // Generate cache key before processing (include opening filter + quality)
-    let tessellation_quality = query.resolved_tessellation_quality()?;
+    // Generate cache key before processing (include opening filter + quality).
+    // From the RECEIVED BYTES, always: a `sha256` parameter that arrived
+    // alongside a body has no say here, so a client whose claimed hash does not
+    // describe what it uploaded still reads and writes the entry its bytes name.
     let cache_key = request_cache_key(&data, &query, tessellation_quality);
     let cache_key_clone = cache_key.clone();
     // This route SHARES nothing -- a per-batch writer cannot see across a batch