feat(server): cache the optimized parquet route like the flat one (#3894)

Closes #3889.

`POST /api/v1/parse/parquet/optimized` re-parsed on every request because the response shape had no cache key. It now has two, `{seed}-parquet-optimized-v1` for the body and `{seed}-parquet-optimized-metadata-v1` for the `X-IFC-Metadata` header stored verbatim, so a replay reports the same `optimization_stats` the live parse did. The namespace is separate from `-parquet-v5`, so a cached flat response never satisfies the optimized route and vice versa. The handler moves to its own module so `parquet.rs` stays under the ratchet.

Three design points on the record: the write is synchronous rather than backgrounded, because a background write races the very next request and this is the small payload; a hit also requires the symbolic sidecar, since the optimized parse writes it and replaying past a missing one would leave `GET /parse/symbolic/{key}` polling a key nobody writes; and a truncated or orphaned cached blob degrades to a miss with a warning instead of a 500 on every later request.

Tests: a second identical request replays without parsing (a sentinel written over the cached body comes back), flat-cached does not satisfy optimized and vice versa with anti-vacuity checks, a missing sidecar re-parses and writes one, a body without metadata re-parses, a different file does not read the first file's entry, and key uniqueness across all suffixes. Both keys documented in docs/guide/server.md and cache_keys.rs.

Not verified against a live server or the 57.9 MB file from the issue; what is proven is that the second request does not parse. Gates on the head: `cargo test -p ifc-lite-server` 0 (246 passed), clippy -D warnings 0, module_size_ratchet 6/6, check-module-size 0, check-changesets 0. Changeset: `@ifc-lite/server-bin` minor.

Author: louistrue

.changeset/cache-optimized-parquet-route.md

@@ -0,0 +1,9 @@
+---
+'@ifc-lite/server-bin': minor
+---
+
+`POST /api/v1/parse/parquet/optimized` is now cached, so a repeat request is a disk read instead of a full re-parse (#3889). The route was added without a cache key of its own, which left the two Parquet endpoints with opposite properties: the flat route stored its large payload and replayed it, while the optimized route rebuilt its small payload on every request, and got no benefit from a flat response cached seconds earlier either. On a 57.9 MB file the flat route roughly halved on its second call and the optimized route did not improve at all.
+
+The optimized response now has its own key pair, `{sha256}-{filter}-parquet-optimized-v1` for the body and `{sha256}-{filter}-parquet-optimized-metadata-v1` for the `X-IFC-Metadata` header, `optimization_stats` included, so a replay reports what the live parse reported. They are a separate namespace from the flat route's `-parquet-v5` / `-parquet-metadata-v4` on purpose: the optimized payload is quantized and deduplicated, so a cached flat response must never satisfy the optimized route or the other way round. Bump the suffix on any change to the optimized payload's columns.
+
+Two details of the write. It happens before the response goes out rather than in a background task, because this payload is the small one and a background write races the very next request, which is the request the cache exists to serve. And a hit requires the symbolic sidecar to still be present alongside the body and metadata: the optimized parse is what writes that sidecar, so replaying past a missing one would leave `GET /api/v1/parse/symbolic/{cache_key}` polling a key nobody writes.

apps/server/src/routes/parse/cache_keys.rs

@@ -100,6 +100,40 @@ pub(crate) fn parquet_metadata_cache_key(
     )
 }
 
+/// Build the optimized-Parquet body cache key for a given file cache key.
+///
+/// `POST /api/v1/parse/parquet/optimized` was added without a key of its own,
+/// so its response had nowhere to be stored and every request re-parsed the
+/// file while the flat route beside it replayed from disk (issue #3889). This
+/// is that key.
+///
+/// Deliberately a DIFFERENT namespace from `-parquet-v5`: the two routes emit
+/// different payloads (quantized vertices, deduplicated shapes, byte colours),
+/// so a hit on one must never satisfy the other.
+///
+/// `v1` is the ara3d BOS payload as it stands after #3595 (rotation-aware
+/// instancing). Bump on EVERY change to the optimized payload's columns, to
+/// what one of them means, OR to the geometry pipeline behind them -- this key
+/// covers `process_geometry_filtered_with_quality` output just as
+/// [`parquet_cache_key`] does, and that key's own `v3` -> `v4` bump was a
+/// pipeline change with no column change at all. In practice: a bump of
+/// `-parquet-v5` almost always needs a bump here too. Otherwise a warm cache
+/// replays a pre-change blob that the decoder reads cleanly, and the change is
+/// silently absent.
+pub(crate) fn parquet_optimized_cache_key(cache_key: &str) -> String {
+    format!("{cache_key}-parquet-optimized-v1")
+}
+
+/// Build the optimized-Parquet metadata cache key for a given file cache key.
+///
+/// Holds the serialized `X-IFC-Metadata` header, `optimization_stats` included,
+/// so a replay carries the same stats the live parse reported. Versioned in
+/// lockstep with [`parquet_optimized_cache_key`], and distinct from the flat
+/// route's `-parquet-metadata-v4` for the same reason the bodies are.
+pub(crate) fn parquet_optimized_metadata_cache_key(cache_key: &str) -> String {
+    format!("{cache_key}-parquet-optimized-metadata-v1")
+}
+
 /// Build the data-model cache key for a given file cache key.
 ///
 /// One definition for the writers (`parse_parquet`, `parse_parquet_stream`) and
@@ -129,10 +163,19 @@ pub(crate) fn data_model_cache_key(cache_key: &str) -> String {
 ///
 /// A cache read error answers `false`: re-parsing is the safe direction.
 pub(crate) async fn has_current_data_model(cache: &DiskCache, cache_key: &str) -> bool {
-    matches!(
-        cache.get_bytes(&data_model_cache_key(cache_key)).await,
-        Ok(Some(_))
-    )
+    has_entry(cache, &data_model_cache_key(cache_key)).await
+}
+
+/// Whether `key` has a readable entry.
+///
+/// Reads the value rather than asking `DiskCache::has`, which is an index
+/// lookup only: an index row whose content is gone would answer `true` here
+/// while every real reader still gets nothing, and a gate that reports present
+/// for an entry nobody can read is worse than no gate.
+///
+/// A read error answers `false`: re-parsing is the safe direction.
+async fn has_entry(cache: &DiskCache, key: &str) -> bool {
+    matches!(cache.get_bytes(key).await, Ok(Some(_)))
 }
 
 /// Build the symbolic-data cache key for a given file cache key.
@@ -167,6 +210,21 @@ pub(crate) async fn cache_symbolic_data(cache: &DiskCache, cache_key: &str, symb
     }
 }
 
+/// Whether symbolic data is cached for `cache_key`.
+///
+/// The optimized-Parquet route's parse is what writes the symbolic sidecar, so
+/// a replay that skips the parse must first check the sidecar is there. Without
+/// this, a body entry that outlived its symbolic entry replays forever and
+/// `GET /api/v1/parse/symbolic/{cache_key}` answers `202` to a key nobody
+/// writes -- the same shape as the geometry/data-model trap in #3869.
+///
+/// [`load_cached_symbolic`] cannot stand in: it answers `SymbolicData::default()`
+/// for an absent entry and for a model with no 2D symbols alike, so absence
+/// there is indistinguishable from success.
+pub(crate) async fn has_cached_symbolic(cache: &DiskCache, cache_key: &str) -> bool {
+    has_entry(cache, &symbolic_cache_key(cache_key)).await
+}
+
 /// Load cached symbolic data for `cache_key`, defaulting to empty when the
 /// entry is absent or unreadable.
 pub(crate) async fn load_cached_symbolic(cache: &DiskCache, cache_key: &str) -> SymbolicData {

apps/server/src/routes/parse/cache_keys_tests.rs

@@ -242,6 +242,8 @@ fn derived_keys_never_collide_with_each_other() {
         data_model_cache_key(seed),
         parquet_cache_key("0ab20f4e4014", OpeningFilterMode::Default, TessellationQuality::Medium),
         parquet_metadata_cache_key("0ab20f4e4014", OpeningFilterMode::Default, TessellationQuality::Medium),
+        parquet_optimized_cache_key(seed),
+        parquet_optimized_metadata_cache_key(seed),
     ];
     let unique: std::collections::HashSet<&String> = derived.iter().collect();
     assert_eq!(unique.len(), derived.len(), "derived keys collide: {derived:?}");
@@ -281,3 +283,39 @@ ENDSEC;";
 
     assert_eq!(detect_schema_version(content), "IFC2X3");
 }
+
+/// The optimized-Parquet route got a key of its own with #3889. Its whole
+/// point is that it is a DIFFERENT namespace from the flat route's: the two
+/// emit different payloads, so a hit on one must never satisfy the other.
+/// Deriving the optimized key from the flat one (or reusing `-parquet-v5`)
+/// would put a quantized, deduplicated payload where a client expecting flat
+/// meshes reads it.
+#[test]
+fn optimized_parquet_keys_are_a_distinct_namespace_from_the_flat_route() {
+    let hash = "0ab20f4e4014";
+    let seed = format!("{hash}-default");
+
+    assert_eq!(
+        parquet_optimized_cache_key(&seed),
+        format!("{seed}-parquet-optimized-v1")
+    );
+    assert_eq!(
+        parquet_optimized_metadata_cache_key(&seed),
+        format!("{seed}-parquet-optimized-metadata-v1")
+    );
+
+    // Neither optimized key may equal, or be a prefix-shadow of, the flat pair.
+    let flat = parquet_cache_key(hash, OpeningFilterMode::Default, TessellationQuality::Medium);
+    let flat_metadata =
+        parquet_metadata_cache_key(hash, OpeningFilterMode::Default, TessellationQuality::Medium);
+    for optimized in [
+        parquet_optimized_cache_key(&seed),
+        parquet_optimized_metadata_cache_key(&seed),
+    ] {
+        assert_ne!(optimized, flat);
+        assert_ne!(optimized, flat_metadata);
+        assert_ne!(optimized, symbolic_cache_key(&seed));
+        assert_ne!(optimized, data_model_cache_key(&seed));
+        assert_ne!(optimized, json_response_cache_key(&seed));
+    }
+}

apps/server/src/routes/parse/mod.rs

@@ -9,11 +9,13 @@ mod cached_replay;
 mod fetch;
 mod json;
 mod parquet;
+mod parquet_optimized;
 mod parquet_stream;
 
 pub use fetch::{check_cache, get_cached_geometry, get_data_model, get_symbolic};
 pub use json::{parse_full, parse_metadata, parse_stream};
-pub use parquet::{parse_parquet, parse_parquet_optimized};
+pub use parquet::parse_parquet;
+pub use parquet_optimized::parse_parquet_optimized;
 pub use parquet_stream::parse_parquet_stream;
 
 use crate::error::ApiError;
@@ -258,6 +260,9 @@ mod ifczip_tests;
 #[cfg(test)]
 mod parquet_tests;
 
+#[cfg(test)]
+mod parquet_optimized_tests;
+
 #[cfg(test)]
 mod json_tests;
 

apps/server/src/routes/parse/parquet.rs

@@ -9,10 +9,7 @@ use super::cache_keys::{
 };
 use super::{extract_file, ParseQuery};
 use crate::error::ApiError;
-use crate::services::{
-    extract_data_model, serialize_data_model_to_parquet, serialize_to_parquet,
-    serialize_to_parquet_optimized_with_stats, OptimizedStats, VERTEX_MULTIPLIER,
-};
+use crate::services::{extract_data_model, serialize_data_model_to_parquet, serialize_to_parquet};
 use crate::types::{ModelMetadata, ProcessingStats};
 use crate::AppState;
 use axum::{
@@ -237,7 +234,11 @@ pub async fn parse_parquet(
     let metadata_json_clone = metadata_json.clone();
     let cache = state.cache.clone();
 
-    // Cache in background (don't block response)
+    // Cache in background (don't block response). Deliberately NOT the
+    // optimized route's synchronous write (#3889): this payload is the large
+    // one, so blocking the response on it costs the client real time. The
+    // trade is a window where an immediate repeat request re-parses because
+    // the write has not landed yet.
     tokio::spawn(async move {
         if let Err(e) = cache
             .set_bytes(&parquet_cache_key, &combined_parquet_clone)
@@ -269,124 +270,3 @@ pub async fn parse_parquet(
 
     Ok(response)
 }
-
-/// Response header containing metadata for optimized Parquet response.
-#[derive(Debug, Clone, Serialize)]
-pub struct OptimizedParquetMetadataHeader {
-    pub cache_key: String,
-    pub metadata: ModelMetadata,
-    pub stats: ProcessingStats,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub mesh_coordinate_space: Option<String>,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub site_transform: Option<Vec<f64>>,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub building_transform: Option<Vec<f64>>,
-    pub optimization_stats: OptimizedStats,
-    /// Vertex multiplier for dequantization (10,000 = 0.1mm precision)
-    pub vertex_multiplier: f32,
-}
-
-/// POST /api/v1/parse/parquet/optimized - Full parse with ara3d BOS-optimized Parquet format.
-///
-/// Returns highly optimized binary Parquet data with:
-/// - Integer quantized vertices (0.1mm precision)
-/// - Mesh deduplication (instancing)
-/// - Byte colors instead of floats
-/// - Optional normals
-///
-/// Query params:
-/// - `normals=true` - Include normals (default: false, compute on client)
-///
-/// Typical compression: 3-5x smaller than basic Parquet, 50-75x smaller than JSON.
-pub async fn parse_parquet_optimized(
-    State(state): State<AppState>,
-    Query(query): Query<ParseQuery>,
-    mut multipart: Multipart,
-) -> Result<Response, ApiError> {
-    // Extract file from multipart
-    // Admission gate (bounded concurrency + byte budget): acquired BEFORE the
-    // upload is buffered, reserving the max upload size since multipart rarely
-    // declares a length up front. Held for the request's whole lifetime so a
-    // disconnected-but-still-running job keeps its memory slot.
-    let admission_guard = state
-        .admission
-        .acquire(state.config.max_file_size_mb as u64 * 1024 * 1024)
-        .await?;
-    let data = extract_file(&mut multipart, state.config.max_file_size_mb).await?;
-
-    // Generate cache key (include opening filter so different modes get different cache entries)
-    let tessellation_quality = query.resolved_tessellation_quality()?;
-    let cache_key = request_cache_key(&data, &query, tessellation_quality);
-
-    tracing::info!(
-        cache_key = %cache_key,
-        size = data.len(),
-        "Processing with optimized Parquet output (ara3d BOS format)"
-    );
-
-    // Parse content
-    let content = data;
-    let opening_filter = query.opening_filter;
-
-    // Process on blocking thread pool (CPU-intensive). Extract the 2D symbol
-    // stream (IfcAnnotation + IfcGrid) alongside geometry for endpoint parity
-    // (issue #900) — it's cached and served via the symbolic fetch endpoint.
-    // Guard rides the blocking task (see parse_full).
-    let ((result, symbolic_data), _admission) = tokio::task::spawn_blocking(move || {
-        (
-            rayon::join(
-                || process_geometry_filtered_with_quality(&content, opening_filter, tessellation_quality),
-                || extract_symbolic_data(&content),
-            ),
-            admission_guard,
-        )
-    })
-    .await?;
-
-    // Cache the symbolic stream so the client can fetch it via
-    // `GET /api/v1/parse/symbolic/{cache_key}`.
-    cache_symbolic_data(&state.cache, &cache_key, &symbolic_data).await;
-
-    // Serialize to optimized Parquet (with deduplication, quantization, etc.)
-    // Don't include normals by default - client can compute them
-    let (parquet_data, opt_stats) =
-        serialize_to_parquet_optimized_with_stats(&result.meshes, false)?;
-
-    tracing::info!(
-        input_meshes = opt_stats.input_meshes,
-        unique_meshes = opt_stats.unique_meshes,
-        unique_materials = opt_stats.unique_materials,
-        mesh_reuse_ratio = opt_stats.mesh_reuse_ratio,
-        payload_size = parquet_data.len(),
-        "Optimized Parquet serialization complete"
-    );
-
-    // Create metadata header
-    let metadata_header = OptimizedParquetMetadataHeader {
-        cache_key,
-        metadata: result.metadata,
-        stats: result.stats,
-        mesh_coordinate_space: result.mesh_coordinate_space,
-        site_transform: result.site_transform,
-        building_transform: result.building_transform,
-        optimization_stats: opt_stats,
-        vertex_multiplier: VERTEX_MULTIPLIER,
-    };
-
-    let metadata_json = serde_json::to_string(&metadata_header)?;
-
-    // Build response with binary body and metadata header
-    let response = Response::builder()
-        .status(StatusCode::OK)
-        .header(
-            header::CONTENT_TYPE,
-            "application/x-parquet-geometry-optimized",
-        )
-        .header("X-IFC-Metadata", metadata_json)
-        .header(header::CONTENT_LENGTH, parquet_data.len())
-        .body(Body::from(parquet_data))
-        .map_err(|e| ApiError::Internal(e.to_string()))?;
-
-    Ok(response)
-}