Equirectangular HDRI background image - known issues / follow up

[kmeirlaen]

When fixing #1354 / #1367, some remaining rendering flows are still problematic - see Item 4 for details.

HDRI Rendering Flow Analysis

Scope: how an equirectangular HDRI background is mixed with gaussian splats across
(a) the live GUI viewport and (b) every export route, for three content scenarios:

1. plain .ply
2. plain .ply with PPISP manually activated in the GUI
3. .ply + .ppisp sidecar

Diagrams (generated, same directory):

• analysis_hdri_flow_1_live_viewport.png — live viewport frame flow
• analysis_hdri_flow_2_export_paths.png — the four export lanes
• analysis_hdri_flow_3_duplication_bugs.png — duplication and issue map

State analyzed: branch fix-hdrirendering at e87e3148 ("apply ppisp post processing to
custom export"); the custom-resolution HDRI export path landed in b76ac885 ("fix 4K export
for ppisp", incl. the toU8Hwc double-permute fix) and e87e3148 (PPISP pass).

---

1. The two HDRI compositors

There are exactly two implementations of "HDRI behind splats", and every flow uses one of them:

	GPU compositor	CPU compositor
Code	resources/viewport/environment.frag + vksplat_compose.comp + vulkan_environment_pass.cpp	renderSoftwareVideoComposite / renderEnvironmentBackground in src/rendering/raster_rendering_engine.cpp:265,1494
Math	equirect UV, rotate-Y, exp2 exposure, ACES tonemap, gamma 1/2.2	identical math re-implemented in C++
Used by	live viewport display only	all captures and exports (viewport-res export, custom-res export, video export)
Splat input	straight-alpha RGBA from vksplat_compose.comp:107 (empty pixels → (0,0,0,0), ZERO_COVERAGE_ALPHA early-out)	straight-alpha RGBA tensor, alpha-blended per pixel with nearest-neighbor rescale

Consequence: what you see is never the pipeline you export through. The viewport is a GPU
composite; every file on disk is a CPU re-composite (the HDRI is re-loaded — cached by path —
and re-rendered per export). The math currently matches, but nothing enforces it (finding D1).

2. PPISP activation matrix

applyViewportAppearanceCorrection (viewport_appearance_correction.cpp:101) picks one of three branches:

Scenario	Branch taken	Effect
training session with PPISP	trainer path (:147) — trainer->applyPPISPForViewport, MANUAL mode forwards ppisp_overrides	corrected
.ply + .ppisp sidecar	standalone path (:175) — SceneManager::appearance_ppisp_; AUTO → controller pool, MANUAL → overrides; free camera (uid −1) → any_camera_id() anchor (:86-94)	corrected
plain .ply, correction manually enabled	no-op (:175-177 returns input unchanged)	not corrected

Sidecar loading auto-enables apply_appearance_correction and picks AUTO/MANUAL
(scene_manager.cpp:828,843 → ui::AppearanceModelLoaded → visualizer_impl.cpp:830).

Correction is always applied to the splat layer only, before HDRI compositing: alpha is
split off, PPISP runs on RGB, alpha is restored (viewport_appearance_correction.cpp:112-145).
That contract is honored by the live viewport and (in the current worktree) the custom-res
export; it is violated by video export (B2 below).

3. Flow summary per surface

Live viewport (diagram 1)

environmentBackgroundUsesTransparentViewerCompositing (rendering_types.hpp:330) gates everything:

• split view off + HDRI on → VkSplat renders straight-alpha RGBA → optional PPISP round-trip
  (full-frame GPU→CPU readback readOutputImageRgba, correct, re-upload,
  rendering_manager_vulkan.cpp:2185-2213) → VulkanViewportPass draws environment quad first,
  splat RGBA alpha-blended on top (vulkan_viewport_pass.cpp:305). Each frame also registers a
  lazy capture closure that does the CPU composite on demand (:2215 corrected variant,
  :2278 uncorrected variant).
• split view on → transparent compositing disabled; solid background baked in-shader; no HDRI at all.

Export lanes (diagram 2)

Lane	Entry	Splat source	PPISP	HDRI mix	Output
A: viewport-res	export_viewport_image(w=0,h=0) → resolveLazyCapture	live frame (already corrected if enabled)	✓ inherited	CPU composite (request builder copy #1)	JPG/PNG @ viewport size
B: custom-res	renderCurrentViewHdrComposite8 (py_rendering.cpp:1283)	fresh Preview-slot render at target res	✓ (:1329)	CPU composite (builder copy #2)	JPG/PNG @ custom res
C: transparent PNG	renderCurrentViewRgba8	fresh Preview-slot render	✗	none (by design)	true-alpha PNG
D: video export	renderVideoExportFrame (async_task_manager.cpp:445, bg thread)	fresh Preview-slot render	✗ (bug B2)	CPU composite (builder copy #3)	encoded frames

4. Findings

Bugs / risks

• B1 — FIXED & VERIFIED (commit b76ac885; tensor-lib root cause still open). 4K/custom-res
  export produced a scrambled image: the CHW→HWC permute in toU8Hwc was applied twice by the
  tensor-lib lazy chain (permute → to(CPU) → clamp*255 → to(UInt8) → contiguous). Fix:
  materialize with .permute(...).contiguous() immediately. Verified 2026-07-05 on a fresh
  2910×2160 export (D:\temp\viewport_export.jpg): the period-3 stripe signature is gone
  (adjacent-pixel diff 0.52 vs 19.8 before, smooth autocorrelation decay, zero inter-channel
  shifts) and channel statistics match the expected scene (blue sky: B mean 172 > R 137, vs the
  corrupted R 176 > B 156). The underlying tensor-lib stale-stride behavior still deserves a
  dedicated unit test so the next caller of that op sequence doesn't rediscover it.
• B2. Video export skips PPISP. makeGaussianPreviewVideoFrame (async_task_manager.cpp:409)
  converts the preview but never calls applyViewportAppearanceCorrection → exported video ≠
  viewport for scenarios 2/3.
• B3. No-op correction still pays full price. With apply_appearance_correction on but no
  trainer/appearance model (scenario 2 on a plain .ply), the live path still does a full-frame
  GPU→CPU RGBA readback + re-upload every frame (rendering_manager_vulkan.cpp:2185) for an
  identity transform. Gate should be apply_appearance_correction && (trainer PPISP || hasAppearanceModel()).
• B4. Split-view inconsistency. Split view silently disables HDRI in the viewport and in
  viewport-res export, but custom-res export (environmentBackgroundEnabled, ignores split) and
  video export still bake HDRI. Three different answers to "is HDRI on?" depending on route.
• B5. Equirect viewport projection + HDRI export mismatch. Preview renders force
  equirectangular = false (rendering_manager_viewport.cpp:885), but the composite request passes
  settings.equirectangular through — the CPU compositor then samples the environment with an
  equirect view while splats were rendered perspective. Misaligned background in lanes B/D when
  the viewport is in 360° projection mode.
• B6. Shared single-slot frame cache race. materializeGpuFrame + renderVideoCompositeFrame
  is a two-call protocol over one cached tensor (raster_rendering_engine.cpp:1888). Lanes A/B run
  on the viewer thread, lane D on a background thread; interleaving clobbers
  cached_tensor_frame_id_ (fails the id check → export error at best). Make the frame handle
  carry its own payload instead of a singleton slot.
• B7. Standalone PPISP controller buffers sized 1920×1080 (scene_manager.cpp:880,
  DEFAULT_MAX_H/W). Verify AUTO-mode correction at 4K export doesn't clamp or reallocate per frame.

Duplicate code (diagram 3)

• D1. HDRI sampling math exists twice (GLSL + C++). Add a golden test rendering both against
  the same map/exposure/rotation, or generate one from the other.
• D2. VideoCompositeFrameRequest construction ×3 (rendering_manager_vulkan.cpp:2159,
  py_rendering.cpp:1381, async_task_manager.cpp:591) → extract
  makeEnvironmentCompositeRequest(settings, frame_view, meshes).
• D3. The materialize→composite→HWC pipeline ×4 (two ~30-line near-identical capture lambdas
  in rendering_manager_vulkan.cpp:2215/:2278, py export, video export) → extract
  compositeSplatOverEnvironment(engine, image, metadata, size, request); also merges the two
  lazy-capture closures.
• D4. CHW/HWC normalization helpers ×5 (toU8Hwc, viewportRenderImageHwc,
  makeViewportCaptureImageHwc, toCpuChwFloatTensor, save_image_u8's inline heuristic) with
  different materialization orders — the direct cause of B1. → toContiguousHwc()/toContiguousChw()
  in rendering/image_layout.hpp, used everywhere.
• D5. renderPreviewImage*: six overloads in rendering_manager_viewport.cpp:359-775
  differing only in readback enum and scene-state source, each repeating the tiling branch →
  single function with an options struct.
• D6. u8→float→CHW→CUDA frame prep duplicated between makeGaussianPreviewVideoFrame and the
  inline block in renderCurrentViewHdrComposite8 → shared prepPreviewFrame(image, apply_ppisp)
  (fixing B2 in the same stroke).

5. Suggested refactor (one target architecture)

One module, e.g. src/visualizer/rendering/environment_composite.{hpp,cpp}:

struct EnvironmentCompositeInput {  // built from RenderSettings once
    FrameView frame_view; ViewportData viewport;
    EnvironmentSettings environment; std::vector<MeshFrameItem> meshes;
};
EnvironmentCompositeInput makeEnvironmentCompositeInput(const RenderSettings&, const FrameView&, ...);

// splat RGBA (any layout/dtype/device) + PPISP + CPU HDRI composite -> HWC u8/float
Result<Tensor> compositeSplatOverEnvironment(RenderingEngine&, SceneManager*,
                                             std::shared_ptr<Tensor> splat_rgba,
                                             const FrameMetadata&, glm::ivec2 size,
                                             const EnvironmentCompositeInput&,
                                             bool apply_ppisp, int camera_uid);

Then: lane A closures, lane B, and lane D all become one-liners over this helper; PPISP parity
(B2), split-view/equirect gating (B4/B5), and request drift (D2/D3) are solved in one place; the
layout helpers collapse into image_layout.hpp (D4); renderPreviewImage* collapses behind an
options struct (D5). The engine frame-cache protocol (B6) can then be retired for exports by
passing the tensor directly into the composite call instead of round-tripping through
materializeGpuFrame/readbackGpuFrameColor.

[kmeirlaen]

This was largely tackled in 926b551 - needs a validation if this is still relevant.