Replace frame_sync with SynchronizedTimestamps, make timestamps.csv optional

.gitignore

@@ -44,3 +44,4 @@ progress.txt
 tasks.json
 tests/tmp
 specs/
+progress

docs/extrinsic_calibration.md

@@ -17,7 +17,7 @@ See [Calibration Targets](calibration_targets.md) for detailed information on ea
 
 ### 1. Recording and File Setup
 
-Save synchronized videos to `project_root/calibration/extrinsic/` according to the naming convention outlined in [Project Setup](project_setup.md#stage-2-extrinsic-calibration). Ensure videos were synchronized during recording, or provide a [`timestamps.csv`](project_setup.md#timestampscsv-format) file for post-hoc synchronization.
+Save synchronized videos to `project_root/calibration/extrinsic/` according to the naming convention outlined in [Project Setup](project_setup.md#stage-2-extrinsic-calibration). If you have per-frame timestamps, include a [`timestamps.csv`](project_setup.md#timestampscsv-format) file. Otherwise, Caliscope infers timing from the video files (see [Frame Synchronization](project_setup.md#frame-synchronization)).
 
 ### 2. Extraction
 

docs/project_setup.md

@@ -19,7 +19,8 @@ The application monitors these directories and automatically updates when files
 
 ## Camera Identification
 
-Cameras are identified by integer IDs assigned through your video file naming. Video files must follow the naming convention `cam_N.mp4`, where N is the camera ID (e.g., `cam_0.mp4`, `cam_1.mp4`, `cam_2.mp4`).
+Cameras are identified by integer IDs assigned through your video file naming.
+Video files must follow the naming convention `cam_N.mp4`, where N is the camera ID (e.g., `cam_0.mp4`, `cam_1.mp4`, `cam_2.mp4`).
 
 - Camera IDs can be any non-negative integer
 - Camera IDs do not need to be contiguous (e.g., `cam_0.mp4`, `cam_3.mp4`, `cam_7.mp4` is valid)
@@ -30,7 +31,9 @@ Cameras are identified by integer IDs assigned through your video file naming. V
 
 Intrinsic calibration determines each camera's internal properties (focal length, principal point, lens distortion).
 
-Place one video per camera in `calibration/intrinsic/`. These videos **do not need to be synchronized**. Each video should show a calibration target (Charuco board, chessboard, or ArUco grid) being moved throughout the camera's field of view.
+Place one video per camera in `calibration/intrinsic/`.
+These videos **do not need to be synchronized**.
+Each video should show a calibration target (Charuco board, chessboard, or ArUco grid) being moved throughout the camera's field of view.
 
 ```
 workspace/
@@ -47,7 +50,8 @@ After calibration, each camera's intrinsic parameters are stored internally for
 
 Extrinsic calibration determines the spatial relationship between cameras (their positions and orientations in 3D space).
 
-Place synchronized videos in `calibration/extrinsic/`. All cameras must observe the same physical space during the same time period.
+Place synchronized videos in `calibration/extrinsic/`.
+All cameras must observe the same physical space during the same time period.
 
 ```
 workspace/
@@ -56,25 +60,24 @@ workspace/
         ├── cam_0.mp4
         ├── cam_1.mp4
         ├── cam_2.mp4
-        └── timestamps.csv      # Optional: only needed for software synchronization
+        └── timestamps.csv      # Optional: per-frame timing data
 ```
 
-### Synchronization Methods
+### Frame Synchronization
 
-Caliscope supports two approaches to synchronization:
+Caliscope needs to know which frames across cameras correspond to the same moment in time.
 
-**1. Hardware Synchronization (Preferred)**
+**If you have per-frame timestamps**, place a `timestamps.csv` file in the recording directory.
+This handles cameras with different frame rates, dropped frames, and different start/stop times.
 
-Record all videos with a common external trigger so each frame captures the same moment in time. All video files should:
-- Start and stop at the same time
-- Have the same number of frames
-- Have matching frame timestamps
+**If you don't have per-frame timestamps**, leave out `timestamps.csv` and Caliscope will infer timing from the video files.
+It reads each video's frame count and frame rate, then spaces each camera's frames evenly across a shared average duration.
+The inferred frame times are saved to `inferred_timestamps.csv` for inspection.
+Hardware synchronization will result in the same number of frames in each file which Caliscope synchronizes exactly.
+In the absence of hardware synchronization, ensure that the files start and stop at the same moment in time.
+The inferred timestamp approach will attempt to time align these files even in the presence of mild drift in frame rate.
 
-When using hardware synchronization, no `timestamps.csv` file is needed.
-
-**2. Software Synchronization**
-
-If cameras record independently without a common trigger, provide a `timestamps.csv` file containing the timestamp for each captured frame.
+Misalignment of the start or stop frames along with drift in the actual recording will impact the time alignment.
 
 ### `timestamps.csv` Format
 
@@ -93,14 +96,12 @@ cam_id,frame_time
 ...
 ```
 
-Requirements:
 - **cam_id**: Must match the camera IDs from your video filenames
-- **frame_time**: Numerical timestamp showing relative time (e.g., from Python's `time.perf_counter()`)
+- **frame_time**: Numerical timestamp for when the frame was captured (e.g., from Python's `time.perf_counter()`)
 - Rows can be in any order
-- Files do not need the same number of frames
-- Cameras do not need to start on the same frame
+- Cameras do not need the same number of frames or the same start time
 
-Caliscope automatically synchronizes the videos during processing, inserting blank frames when necessary to maintain temporal alignment.
+Caliscope automatically aligns the videos during processing, inserting blank frames where necessary to maintain temporal correspondence.
 
 ### Calibration Output
 
@@ -113,7 +114,8 @@ workspace/
         ├── cam_0.mp4
         ├── cam_1.mp4
         ├── cam_2.mp4
-        ├── timestamps.csv           # If using software sync
+        ├── timestamps.csv           # If per-frame timing was provided
+        ├── inferred_timestamps.csv  # Caliscope's timing assumptions when no timestamps.csv provided (not read back)
         ├── CHARUCO/                 # Extraction output (tracker name varies)
         │   └── image_points.csv
         └── capture_volume/          # Calibration result
@@ -126,7 +128,8 @@ The `capture_volume/` directory contains the complete calibrated camera system a
 
 ## Stage 3: Recording and Reconstruction
 
-For each motion capture session, create a subfolder within `recordings/` and populate it with synchronized videos following the same requirements as extrinsic calibration (hardware sync preferred, software sync via `timestamps.csv` if needed).
+For each motion capture session, create a subfolder within `recordings/` and populate it with synchronized videos.
+The same synchronization rules apply: provide a `timestamps.csv` if you have per-frame timing, or Caliscope infers from the video files (see [Frame Synchronization](#frame-synchronization)).
 
 ```
 workspace/
@@ -138,7 +141,7 @@ workspace/
         └── timestamps.csv          # Optional: same format as extrinsic
 ```
 
-After processing with a motion tracking system (e.g., POSE, HAND, HOLISTIC), output files are created in a tracker-named subdirectory:
+After processing with a motion tracking system, output files are created in a tracker-named subdirectory:
 
 ```
 workspace/
@@ -147,7 +150,7 @@ workspace/
         ├── cam_0.mp4
         ├── cam_1.mp4
         ├── cam_2.mp4
-        ├── timestamps.csv
+        ├── timestamps.csv                  # If provided
         └── POSE/                           # Output subdirectory (tracker name)
             ├── camera_array.toml           # Snapshot of calibration used
             ├── xy_POSE.csv                 # 2D tracked points per camera

src/caliscope/core/process_synchronized_recording.py

@@ -1,7 +1,7 @@
 """Batch processing of synchronized multi-camera video.
 
 Pure function that extracts 2D landmarks from synchronized video streams.
-Uses batch synchronization from timestamps.csv — no real-time streaming.
+Uses batch synchronization from SynchronizedTimestamps -- no real-time streaming.
 """
 
 import logging
@@ -17,7 +17,7 @@
 from caliscope.core.point_data import ImagePoints
 from caliscope.packets import PointPacket
 from caliscope.recording.frame_source import FrameSource
-from caliscope.recording.frame_sync import compute_sync_indices
+from caliscope.recording.synchronized_timestamps import SynchronizedTimestamps
 from caliscope.task_manager.cancellation import CancellationToken
 from caliscope.tracker import Tracker
 
@@ -37,6 +37,7 @@ def process_synchronized_recording(
     recording_dir: Path,
     cameras: dict[int, CameraData],
     tracker: Tracker,
+    synced_timestamps: SynchronizedTimestamps,
     *,
     subsample: int = 1,
     on_progress: Callable[[int, int], None] | None = None,
@@ -45,13 +46,14 @@ def process_synchronized_recording(
 ) -> ImagePoints:
     """Process synchronized video recordings to extract 2D landmarks.
 
-    Reads timestamps.csv to determine frame alignment, then processes
-    each sync index by seeking directly to aligned frames.
+    Uses SynchronizedTimestamps for frame alignment, then processes each
+    sync index by seeking directly to aligned frames.
 
     Args:
-        recording_dir: Directory containing cam_N.mp4 and timestamps.csv
+        recording_dir: Directory containing cam_N.mp4 files
         cameras: Camera data by cam_id (provides rotation_count for frame orientation)
         tracker: Tracker for 2D point extraction (handles per-cam_id state internally)
+        synced_timestamps: Pre-constructed timestamp alignment object
         subsample: Process every Nth sync index (1 = all, 10 = every 10th)
         on_progress: Called with (current, total) during processing
         on_frame_data: Called with (sync_index, {cam_id: FrameData}) for live display
@@ -60,45 +62,33 @@ def process_synchronized_recording(
     Returns:
         ImagePoints containing all tracked 2D observations
     """
-    # Load frame timestamps and compute sync assignments
-    timestamps_csv = recording_dir / "timestamps.csv"
-    sync_map = compute_sync_indices(timestamps_csv)
-
-    # Load frame_time data for enriching output
-    timestamps_df = pd.read_csv(timestamps_csv)
-    frame_times = _build_frame_time_lookup(timestamps_df)
-
-    # Get sync indices to process (with subsampling)
-    all_sync_indices = sorted(sync_map.keys())
-    sync_indices_to_process = all_sync_indices[::subsample]
-    total = len(sync_indices_to_process)
+    all_sync_indices = synced_timestamps.sync_indices[::subsample]
+    total = len(all_sync_indices)
 
-    logger.info(f"Processing {total} sync indices (subsample={subsample}, total available={len(all_sync_indices)})")
+    logger.info(
+        f"Processing {total} sync indices "
+        f"(subsample={subsample}, total available={len(synced_timestamps.sync_indices)})"
+    )
 
-    # Create frame sources (one per camera, for seeking)
     frame_sources = _create_frame_sources(recording_dir, cameras)
-
-    # Point accumulation
     point_rows: list[dict] = []
 
     try:
-        for i, sync_index in enumerate(sync_indices_to_process):
-            # Check cancellation
+        for i, sync_index in enumerate(all_sync_indices):
             if token is not None and token.is_cancelled:
                 logger.info("Processing cancelled")
                 break
 
-            # Read and track frames for this sync index
             frame_data: dict[int, FrameData] = {}
-            cam_id_assignments = sync_map[sync_index]
 
-            for cam_id, frame_index in cam_id_assignments.items():
+            for cam_id in synced_timestamps.cam_ids:
+                frame_index = synced_timestamps.frame_for(sync_index, cam_id)
+
                 if frame_index is None:
                     logger.debug(f"Dropped frame: sync={sync_index}, cam_id={cam_id}")
                     continue
 
                 if cam_id not in frame_sources:
-                    # Camera in sync_map but not in cameras dict (shouldn't happen)
                     logger.warning(f"cam_id {cam_id} not in cameras dict, skipping")
                     continue
 
@@ -111,15 +101,12 @@ def process_synchronized_recording(
                     )
                     continue
 
-                # Tracker handles per-cam_id state internally via cam_id parameter
                 points = tracker.get_points(frame, cam_id, camera.rotation_count)
                 frame_data[cam_id] = FrameData(frame, points, frame_index)
 
-                # Accumulate points
-                frame_time = frame_times.get((cam_id, frame_index), 0.0)
+                frame_time = synced_timestamps.time_for(cam_id, frame_index)
                 _accumulate_points(point_rows, sync_index, cam_id, frame_index, frame_time, points)
 
-            # Invoke callbacks
             if on_frame_data is not None:
                 on_frame_data(sync_index, frame_data)
             if on_progress is not None:
@@ -182,22 +169,6 @@ def _create_frame_sources(recording_dir: Path, cameras: dict[int, CameraData]) -
     return sources
 
 
-def _build_frame_time_lookup(timestamps_df: pd.DataFrame) -> dict[tuple[int, int], float]:
-    """Build lookup table: (cam_id, frame_index) -> frame_time.
-
-    Frame index is the row number within each cam_id's sequence.
-    """
-    lookup: dict[tuple[int, int], float] = {}
-
-    for cam_id, group in timestamps_df.groupby("cam_id"):
-        sorted_group = group.sort_values("frame_time").reset_index(drop=True)
-        for frame_index, row in sorted_group.iterrows():
-            # frame_index here is actually the integer index from iterrows
-            lookup[(int(cam_id), int(frame_index))] = float(row["frame_time"])  # type: ignore[arg-type]
-
-    return lookup
-
-
 def _accumulate_points(
     point_rows: list[dict],
     sync_index: int,
@@ -214,7 +185,6 @@ def _accumulate_points(
     if point_count == 0:
         return
 
-    # Get obj_loc columns (may be None)
     obj_loc_x, obj_loc_y, obj_loc_z = points.obj_loc_list
 
     for i in range(point_count):
@@ -237,7 +207,6 @@ def _accumulate_points(
 def _build_image_points(point_rows: list[dict]) -> ImagePoints:
     """Construct ImagePoints from accumulated point data."""
     if not point_rows:
-        # Return empty ImagePoints with correct schema
         df = pd.DataFrame(
             columns=[
                 "sync_index",

src/caliscope/gui/presenters/multi_camera_processing_presenter.py

@@ -27,6 +27,7 @@
     get_initial_thumbnails,
     process_synchronized_recording,
 )
+from caliscope.recording.synchronized_timestamps import SynchronizedTimestamps
 from caliscope.task_manager.cancellation import CancellationToken
 from caliscope.task_manager.task_handle import TaskHandle
 from caliscope.task_manager.task_manager import TaskManager
@@ -260,7 +261,10 @@ def set_rotation(self, cam_id: int, rotation_count: int) -> None:
     def start_processing(self, subsample: int = 1) -> None:
         """Start background processing.
 
-        Submits process_synchronized_recording to TaskManager.
+        Constructs SynchronizedTimestamps synchronously at the boundary before
+        submitting the worker task. A ValueError from timestamp construction
+        (e.g., incompatible frame counts) is emitted via processing_failed
+        immediately -- no state transition to PROCESSING.
 
         Args:
             subsample: Process every Nth sync index (1 = all)
@@ -275,19 +279,30 @@ def start_processing(self, subsample: int = 1) -> None:
 
         logger.info(f"Starting multi-camera processing: {self._recording_dir}")
 
-        # Clear previous results
-        self._reset_results()
-
         # Capture values for closure
         recording_dir = self._recording_dir
         cameras = dict(self._cameras)
         tracker = self._tracker
+        cam_ids = list(cameras.keys())
+
+        # Construct timestamps synchronously at the boundary (Amendment 6).
+        # ValueError means incompatible videos -- emit failure and stay READY.
+        try:
+            synced_timestamps = SynchronizedTimestamps.load(recording_dir, cam_ids)
+        except ValueError as e:
+            logger.error(f"Cannot load timestamps: {e}")
+            self.processing_failed.emit(str(e))
+            return
+
+        # Clear previous results only after successful timestamp construction
+        self._reset_results()
 
         def worker(token: CancellationToken, handle: TaskHandle) -> ImagePoints:
             return process_synchronized_recording(
                 recording_dir=recording_dir,
                 cameras=cameras,
                 tracker=tracker,
+                synced_timestamps=synced_timestamps,
                 subsample=subsample,
                 on_progress=self._on_progress,
                 on_frame_data=self._on_frame_data,

src/caliscope/recording/__init__.py

@@ -5,17 +5,16 @@
     create_streamer,
 )
 from caliscope.recording.frame_source import FrameSource
-from caliscope.recording.frame_sync import SyncMap, compute_sync_indices
 from caliscope.recording.frame_timestamps import FrameTimestamps
+from caliscope.recording.synchronized_timestamps import SynchronizedTimestamps
 from caliscope.recording.video_utils import VideoProperties, read_video_properties
 
 __all__ = [
     "FramePacketStreamer",
     "FrameSource",
     "FrameTimestamps",
-    "SyncMap",
+    "SynchronizedTimestamps",
     "VideoProperties",
-    "compute_sync_indices",
     "create_streamer",
     "read_video_properties",
 ]