LTplus-AG
diff --git a/‎.changeset/header-comment-hides-the-data-marker.md‎
Lines changed: 7 additions & 0 deletions b/‎.changeset/header-comment-hides-the-data-marker.md‎
Lines changed: 7 additions & 0 deletions
diff --git a/‎.changeset/report-rust-scanner-malformed-record.md‎
Lines changed: 13 additions & 0 deletions b/‎.changeset/report-rust-scanner-malformed-record.md‎
Lines changed: 13 additions & 0 deletions
diff --git a/‎rust/core/src/columnar_index.rs‎
Lines changed: 4 additions & 1 deletion b/‎rust/core/src/columnar_index.rs‎
Lines changed: 4 additions & 1 deletion
diff --git a/‎rust/core/src/decoder.rs‎
Lines changed: 2 additions & 2 deletions b/‎rust/core/src/decoder.rs‎
Lines changed: 2 additions & 2 deletions
diff --git a/‎rust/core/src/lib.rs‎
Lines changed: 3 additions & 2 deletions b/‎rust/core/src/lib.rs‎
Lines changed: 3 additions & 2 deletions
diff --git a/‎rust/core/src/parser/malformed_records.rs‎
Lines changed: 61 additions & 0 deletions b/‎rust/core/src/parser/malformed_records.rs‎
Lines changed: 61 additions & 0 deletions
diff --git a/‎rust/core/src/parser/mod.rs‎
Lines changed: 11 additions & 0 deletions b/‎rust/core/src/parser/mod.rs‎
Lines changed: 11 additions & 0 deletions
diff --git a/‎rust/core/src/parser/oversized_ids.rs‎
Lines changed: 5 additions & 11 deletions b/‎rust/core/src/parser/oversized_ids.rs‎
Lines changed: 5 additions & 11 deletions
diff --git a/‎rust/core/src/parser/report_sink.rs‎
Lines changed: 30 additions & 0 deletions b/‎rust/core/src/parser/report_sink.rs‎
Lines changed: 30 additions & 0 deletions
diff --git a/‎rust/core/src/parser/scanner.rs‎
Lines changed: 47 additions & 61 deletions b/‎rust/core/src/parser/scanner.rs‎
Lines changed: 47 additions & 61 deletions
@@ -0,0 +1,7 @@
+---
+'@ifc-lite/wasm': patch
+---
+
+`EntityScanner`'s HEADER skip (`rust/core/src/parser/scanner_header.rs`) matched the `DATA;` section marker inside a STEP `/* ... */` comment. ISO 10303-21 allows a comment wherever whitespace is allowed, the HEADER included, so `HEADER; /* DATA; #99=IFCWALL($); */ ENDSEC; DATA; #1=IFCWALL($);` ended the marker search inside the comment and started the entity scan there: `#99`, a record the file does not declare, came back alongside the real `#1`. The search already skipped quoted strings for exactly this reason; it now skips complete comments the same way, through the shared `skip_step_comment`.
+
+An unterminated `/*` in the HEADER gets the same answer as a missing marker, so the scan starts at the top and `next_entity` meets the same comment, which reports it through the `malformed_record_start` channel rather than silently returning nothing. A headerless partial file whose records precede a bad comment still scans those records.
@@ -0,0 +1,13 @@
+---
+'@ifc-lite/wasm': patch
+---
+
+The Rust twin of #3695's TS fix, landing alongside it: `EntityScanner::find_entity_end` (`rust/core`) has no byte to resume from when a record opens a `'` string or a `/* ... */` comment that never closes, so the scan stops there, correctly, but until now with no trace of why. A truncated download, a failed export, or a lossy round-trip through another tool could silently drop every entity after the break on the WASM/native load path, so which path a user hit decided whether they knew their model was incomplete.
+
+`EntityScanner` now exposes `malformed_record_start()`, and every index-building whole-file scan reports it through the existing `report_oversized_ids` sink: `columnar_index.rs`, `decoder.rs`, the streaming processor (`rust/processing/src/processor/mod.rs`), the wasm sharded-prepass path (`rust/wasm-bindings/src/api/gpu_meshes/prepass.rs`), and the wasm parsing entry points (`rust/wasm-bindings/src/api/parsing.rs`), via `ifc_lite_core::report_malformed_records` and the combined `report_scan_diagnostics` convenience both call. The behaviour is unchanged on purpose: an unterminated string leaves no reliable resume point, so the scan still stops there rather than guessing past it, only the silence is fixed.
+
+The native sharded parallel scan (`rust/processing::build_entity_index_parallel`, files 8MB+) now also attributes a malformed stop to the shard whose real (resynchronised) region actually contains it, distinguishing it from a false stop a shard's speculative mid-record start can produce, and reports it once, stitched, exactly like the existing #3395 oversized-id refusal count, byte-identical to the serial scanner's truncation point.
+
+The wasm-bindings columns event (`buildPrePassOnce`'s sharded/columns path) now also carries `malformedRecordFound` alongside the existing `oversizedIdCount`. This is the wasm half only: the TS side does not read this field yet, and wiring the browser host to surface it is a separate, following change.
+
+Two gaps are explicitly NOT covered here, and are deferred to follow-up issues: the browser's sharded pre-pass (`scan_shard_classified_with_refusals` / `stitchShards`) discards a shard's malformed-record offset entirely, so a malformed stop on that path stays silent; and `parse_stream` (`rust/core/src/streaming.rs`) plus the server's `/parse/json` handler run whole-file scans that report neither the #3395 oversized-id count nor a malformed stop.
@@ -178,7 +178,10 @@ impl ColumnarEntityIndex {
             starts.push(start as u32);
             lengths.push((end - start) as u32);
         }
-        crate::parser::report_oversized_ids(scanner.skipped_oversized_ids());
+        crate::parser::report_scan_diagnostics(
+            scanner.skipped_oversized_ids(),
+            scanner.malformed_record_start().is_some(),
+        );
         Self::from_unsorted(ids, starts, lengths)
     }
 
 
@@ -8,7 +8,7 @@
 
 use crate::columnar_index::EntityIndexStore;
 use crate::error::{Error, Result};
-use crate::parser::{parse_entity, report_oversized_ids, EntityScanner};
+use crate::parser::{parse_entity, report_scan_diagnostics, EntityScanner};
 use crate::schema_gen::{AttributeValue, DecodedEntity};
 use rustc_hash::FxHashMap;
 use std::sync::Arc;
@@ -36,7 +36,7 @@ where
     while let Some((id, _type_name, start, end)) = scanner.next_entity() {
         index.insert(id, (start, end));
     }
-    report_oversized_ids(scanner.skipped_oversized_ids());
+    report_scan_diagnostics(scanner.skipped_oversized_ids(), scanner.malformed_record_start().is_some());
     index
 }
 
 
@@ -98,8 +98,9 @@ pub use legacy_entities::{
 };
 pub use model_bounds::{scan_model_bounds, scan_placement_bounds, ModelBounds};
 pub use parser::{
-    entity_count, oversized_id_report, parse_entity, report_oversized_ids, set_report_sink,
-    skip_step_comment, EntityScanner, Token,
+    entity_count, oversized_id_report, parse_entity, report_malformed_records,
+    report_oversized_ids, report_scan_diagnostics, set_report_sink, skip_step_comment,
+    EntityScanner, Token,
 };
 pub use project_units::{
     measure::{measure_unit, MeasureUnit},
 
@@ -0,0 +1,61 @@
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+//! The one place Rust says that a scan stopped early because a record had no
+//! terminator - the Rust twin of the TS fix on `EntityScanResult.malformedRecordCount`
+//! (`packages/parser/src/entity-scanner.ts`, #3695).
+//!
+//! [`EntityScanner`](super::EntityScanner) stops the whole scan the moment a
+//! record opens a `'` string, a `/* ... */` comment, or simply runs to end of
+//! input with none of them ever closing - [`EntityScanner::find_entity_end`]
+//! has no byte to resume from once that happens (unlike the oversized-id
+//! refusal in [`super::oversized_ids`], which SKIPS the one record and keeps
+//! scanning). A model that comes back with its tail silently missing reads
+//! exactly like a complete one, so this module is the other half: the caller
+//! reports it rather than staying quiet.
+//!
+//! Modelled directly on [`super::oversized_ids`] - same "one home, not one
+//! call site" reasoning, same host-installed sink (shared with it via
+//! [`super::report_sink`], not a second `OnceLock`) - but the count this
+//! reports is always 0 or 1: once a scan stops here it never resumes, so
+//! there is nothing left to accumulate a second refusal from.
+//!
+//! One constant message, not a builder like [`super::oversized_id_report`]:
+//! the oversized-id report names a count, so it has to allocate a formatted
+//! string; this report names nothing that varies, so a `const &str` is the
+//! whole of it and there is no `Option<String>`-returning sibling to keep in
+//! sync with it.
+
+/// The one-line report for a scan that stopped because of a malformed
+/// record.
+const MALFORMED_RECORD_MESSAGE: &str =
+    "scan: stopped early - a record had no terminating ';' before end of input \
+     (an unterminated quoted string, comment, or truncated file); the entities \
+     returned may be an incomplete view of this file (#3695)";
+
+/// Emit [`MALFORMED_RECORD_MESSAGE`] to the installed sink (stderr by
+/// default; see [`super::set_report_sink`]).
+///
+/// A no-op at `stopped == false`, so a caller can hand it
+/// `scanner.malformed_record_start().is_some()` unconditionally, exactly
+/// like [`super::report_oversized_ids`] takes `skipped_oversized_ids()`.
+pub fn report_malformed_records(stopped: bool) {
+    if !stopped {
+        return;
+    }
+    super::report_sink::emit(MALFORMED_RECORD_MESSAGE);
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn message_names_the_shape_of_the_problem() {
+        assert!(
+            MALFORMED_RECORD_MESSAGE.contains("incomplete"),
+            "report must warn the caller the model may be short: {MALFORMED_RECORD_MESSAGE}"
+        );
+    }
+}
@@ -17,11 +17,22 @@
 //! unterminated comment mean" differently gives the same answer (#3303).
 
 mod lexical;
+mod malformed_records;
 mod oversized_ids;
+mod report_sink;
 mod scanner;
 mod tokenizer;
 
 pub use lexical::skip_step_comment;
+pub use malformed_records::report_malformed_records;
 pub use oversized_ids::{oversized_id_report, report_oversized_ids, set_report_sink};
 pub use scanner::{entity_count, EntityScanner};
 pub use tokenizer::{parse_entity, Token};
+
+/// [`report_oversized_ids`] + [`report_malformed_records`] in one call — the
+/// two diagnostics every whole-file `EntityScanner` walk in this workspace
+/// emits after a scan, so every call site needs only one line, not two.
+pub fn report_scan_diagnostics(skipped_oversized_ids: usize, malformed_record_found: bool) {
+    report_oversized_ids(skipped_oversized_ids);
+    report_malformed_records(malformed_record_found);
+}
@@ -40,14 +40,11 @@
 //! scan with the sink still unset. Anything embedding this crate with its own
 //! log pipeline can do the same.
 
-use std::sync::OnceLock;
+use super::report_sink::REPORT_SINK;
 
-/// Host-installed destination for [`report_oversized_ids`]. Set once, because
-/// a swappable sink invites a reset race between two loads on different
-/// threads and nothing here needs one.
-static REPORT_SINK: OnceLock<fn(&str)> = OnceLock::new();
-
-/// Route [`report_oversized_ids`] to `sink` instead of stderr.
+/// Route every scan diagnostic in this crate — [`report_oversized_ids`] and
+/// [`super::malformed_records::report_malformed_records`] alike — to `sink`
+/// instead of stderr.
 ///
 /// Returns `true` when this call installed the sink, `false` when one was
 /// already installed (the first wins). Callers on `wasm32` MUST install one:
@@ -81,10 +78,7 @@ pub fn report_oversized_ids(skipped: usize) {
     let Some(message) = oversized_id_report(skipped) else {
         return;
     };
-    match REPORT_SINK.get() {
-        Some(sink) => sink(&format!("[ifc-lite] {message}")),
-        None => eprintln!("[ifc-lite] {message}"),
-    }
+    super::report_sink::emit(&message);
 }
 
 #[cfg(test)]
 
@@ -0,0 +1,30 @@
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+//! The one host-installed destination every scan diagnostic in this crate
+//! writes to (issue #3395's oversized-id refusals, and the malformed-record
+//! stop this module now shares the channel with).
+//!
+//! One sink, not one per diagnostic kind: [`super::oversized_ids`] and
+//! [`super::malformed_records`] both report a scan that came back short of
+//! what the file declares, for a different reason each, but a host that
+//! wants to see one wants to see the other the same way — installing two
+//! sinks (or forgetting to install the second) would silently drop half the
+//! warnings on whichever target adds a diagnostic later. `set_report_sink`
+//! stays exported from [`super::oversized_ids`] (its original, still public
+//! name) so no caller-visible API changes.
+
+use std::sync::OnceLock;
+
+/// Set once, because a swappable sink invites a reset race between two loads
+/// on different threads and nothing here needs one.
+pub(super) static REPORT_SINK: OnceLock<fn(&str)> = OnceLock::new();
+
+/// Route a diagnostic `message` to the installed sink, or stderr by default.
+pub(super) fn emit(message: &str) {
+    match REPORT_SINK.get() {
+        Some(sink) => sink(&format!("[ifc-lite] {message}")),
+        None => eprintln!("[ifc-lite] {message}"),
+    }
+}
@@ -7,6 +7,10 @@
 //! Independent of the nom [`tokenizer`](super::tokenizer): does its own
 //! hand-rolled, quote- and comment-aware parsing without building [`Token`]s.
 
+#[path = "scanner_header.rs"]
+mod scanner_header;
+use scanner_header::data_section_start;
+
 /// Fast entity scanner over raw IFC bytes without full parsing.
 /// O(n) performance for finding entities by type
 /// Uses memchr for SIMD-accelerated byte searching
@@ -20,6 +24,8 @@ pub struct EntityScanner<'a> {
     /// [`Self::skipped_oversized_id_starts`]. It never allocates on a file
     /// with nothing to refuse, which is every real file.
     skipped_oversized_id_starts: Vec<usize>,
+    /// See [`Self::malformed_record_start`] for what this points at.
+    malformed_record_start: Option<usize>,
 }
 
 impl<'a> EntityScanner<'a> {
@@ -38,6 +44,7 @@ impl<'a> EntityScanner<'a> {
             bytes,
             position: data_section_start(bytes),
             skipped_oversized_id_starts: Vec::new(),
+            malformed_record_start: None,
         }
     }
 
@@ -61,6 +68,7 @@ impl<'a> EntityScanner<'a> {
             bytes,
             position: clamped,
             skipped_oversized_id_starts: Vec::new(),
+            malformed_record_start: None,
         }
     }
 
@@ -99,6 +107,23 @@ impl<'a> EntityScanner<'a> {
         &self.skipped_oversized_id_starts
     }
 
+    /// The byte offset that stopped this scan because no terminator was
+    /// found, or `None` otherwise: a record's `line_start` (its `#`) when
+    /// [`find_entity_end`](Self::find_entity_end) fails, or the `/` of an
+    /// unterminated comment found BETWEEN records (no record to name yet).
+    /// A whole-file scan needs only `is_some()`; a SHARDED scan needs the
+    /// offset, for the reason [`Self::skipped_oversized_id_starts`] does.
+    pub fn malformed_record_start(&self) -> Option<usize> {
+        self.malformed_record_start
+    }
+
+    /// Record `at` as this scan's stop point, the first time only.
+    fn mark_malformed(&mut self, at: usize) {
+        if self.malformed_record_start.is_none() {
+            self.malformed_record_start = Some(at);
+        }
+    }
+
     /// Scan for the next entity
     /// Returns (entity_id, type_name, line_start, line_end)
     #[inline]
@@ -152,13 +177,18 @@ impl<'a> EntityScanner<'a> {
                     // past `*/`; if not, it's a STEP arithmetic '/' inside a
                     // value list (rare; just step past it).
                     if candidate + 1 < len && bytes[candidate + 1] == b'*' {
-                        // An unterminated `/*` here means corrupt input.
-                        // `skip_step_comment` refuses (returns `None`) rather
-                        // than silently consuming the rest of the file — see
-                        // its doc comment for why that's the right call for a
-                        // scanner (issue #3303).
-                        self.position = super::lexical::skip_step_comment(bytes, candidate)?;
-                        continue;
+                        // An unterminated `/*` means corrupt input (#3303) —
+                        // same "no resume point" shape as `find_entity_end`.
+                        match super::lexical::skip_step_comment(bytes, candidate) {
+                            Some(next_pos) => {
+                                self.position = next_pos;
+                                continue;
+                            }
+                            None => {
+                                self.mark_malformed(candidate);
+                                return None;
+                            }
+                        }
                     }
                     // Lone '/' — not a comment. Skip past.
                     self.position = candidate + 1;
@@ -194,7 +224,15 @@ impl<'a> EntityScanner<'a> {
             // Find the end of the entity (semicolon) while respecting quoted strings
             // IFC strings use single quotes and can contain semicolons
             let line_content = &bytes[line_start..];
-            let end_offset = self.find_entity_end(line_content)?;
+            let end_offset = match self.find_entity_end(line_content) {
+                Some(o) => o,
+                None => {
+                    // No terminator found (see `find_entity_end`'s doc
+                    // comment) — record and stop, per `tokenizer.ts`.
+                    self.mark_malformed(line_start);
+                    return None;
+                }
+            };
             let line_end = line_start + end_offset + 1;
 
             // Parse entity ID — digit range already validated in the candidate loop.
@@ -384,6 +422,7 @@ impl<'a> EntityScanner<'a> {
     pub fn reset(&mut self) {
         self.position = data_section_start(self.bytes);
         self.skipped_oversized_id_starts.clear();
+        self.malformed_record_start = None;
     }
 
     /// Fast check if attribute at given index is non-null (not '$')
@@ -513,59 +552,6 @@ where
     EntityScanner::new(content).count()
 }
 
-/// Locate the byte offset of the first character after `DATA;` (skipping the
-/// STEP HEADER section). Returns 0 if the marker isn't found — partial files
-/// without a HEADER still scan from the top.
-///
-/// Scanning the HEADER for entities is unsafe: the HEADER is a free-form
-/// STEP record that legally contains arbitrary characters inside quoted
-/// strings (filenames, descriptions). CATIA emits `FILE_NAME('…\X0\2#.ifc'…)`,
-/// and a tokenizer that anchors on `#` will latch onto the in-string `#`,
-/// flip `find_entity_end`'s quote parity, and drop the rest of the file.
-/// See issue #654.
-///
-/// Quote-aware: the marker is only matched outside `'…'` strings, since a
-/// HEADER field could legally contain the literal text `DATA;` in a
-/// description or filename. Escaped single quotes (`''`) are treated as a
-/// pair of in-string characters per ISO 10303-21.
-fn data_section_start(bytes: &[u8]) -> usize {
-    const MARKER: &[u8] = b"DATA;";
-    let len = bytes.len();
-    if len < MARKER.len() {
-        return 0;
-    }
-    // Cap the header scan. Real-world headers are <2 KB; an unbounded scan
-    // here would defeat the point of an O(1)-up-front fix on giant files
-    // that legitimately lack a HEADER section.
-    let limit = len.min(1 << 18); // 256 KB
-    let mut pos = 0;
-    let mut in_string = false;
-    while pos < limit {
-        let b = bytes[pos];
-        if in_string {
-            if b == b'\'' {
-                if pos + 1 < limit && bytes[pos + 1] == b'\'' {
-                    pos += 2; // escaped quote
-                    continue;
-                }
-                in_string = false;
-            }
-            pos += 1;
-            continue;
-        }
-        if b == b'\'' {
-            in_string = true;
-            pos += 1;
-            continue;
-        }
-        if b == b'D' && pos + MARKER.len() <= len && &bytes[pos..pos + MARKER.len()] == MARKER {
-            return pos + MARKER.len();
-        }
-        pos += 1;
-    }
-    0
-}
-
 #[cfg(test)]
 #[path = "scanner_tests.rs"]
 mod scanner_tests;
Original file line number	Diff line number	Diff line change
`@@ -178,7 +178,10 @@ impl ColumnarEntityIndex {`
`178`	`178`	`starts.push(start as u32);`
`179`	`179`	`lengths.push((end - start) as u32);`
`180`	`180`	`}`
`181`		`- crate::parser::report_oversized_ids(scanner.skipped_oversized_ids());`
	`181`	`+ crate::parser::report_scan_diagnostics(`
	`182`	`+ scanner.skipped_oversized_ids(),`
	`183`	`+ scanner.malformed_record_start().is_some(),`
	`184`	`+ );`
`182`	`185`	`Self::from_unsorted(ids, starts, lengths)`
`183`	`186`	`}`
`184`	`187`