Grid-to-World Gaussian Prototyper

This project creates multi-world experiences by stitching collections of “mini-worlds.” Each mini-world is derived from Gaussian splatting techniques, enabling rapid conversion of 2D reference material into explorable 3D space. The goal is to provide a fast, intuitive storyboard or cinematic prototyping tool: lay out regions on a 2D grid, attach splat-based scenes, and instantly walk through the combined world to test composition, scale, and narrative flow.

---

Why This Exists

Concept artists, world-builders, and previs teams often need to assemble “what if” worlds fast. Traditional 3D modeling pipelines (meshes, UVs, PBR) are slow when you just want believable spatial composition. Gaussian splatting flipped the script: feed a single image (or a handful of photos) into an image→3D model, and you get a photorealistic splat cloud in minutes. The missing piece was a layout tool to drop multiple splat “islands” on a grid and instantly walk through the stitched world.

This project is that missing piece:

• A browser-based 2D editor for carving the ground into regions.
• A one-click 3D viewer powered by @mkkellogg/gaussian-splats-3d.
• A Flask backend that can call into Hugging Face / local checkpoints to turn reference images (or text prompts) into GN splats.

---

Key Features

• Frictionless 2D → 3D: Draw rectangles, upload images, attach splats, and enter the world view with no export/import hoops.
• Real sky & ground presets: Photo panoramas for blue-sky and sunset domes, plus tiled grass. No GPU shading tricks required.
• Splat asset staging: Upload existing .splat, .ksplat, or .ply files, or let the backend synthesise them via an inference provider.
• Stateful editing: Save the entire project to JSON / localStorage, export/import thumbnails, and recover from previous sessions automatically.
• Full-screen overlay viewer: Smooth arrow-key navigation and immediate feedback on sky/ground/environment tweaks.
• Pluggable providers: Swap between DreamGaussian, LGM, or any Hugging Face endpoint via a lightweight provider interface.

Quick Start

# 1. Clone and step inside
git clone https://github.com/your-handle/Grid-to-World-Gaussian-Prototyper
cd Grid-to-World-Gaussian-Prototyper/mini-worlds

# 2. Launch everything (Windows)
start-mini-worlds.bat

#  (macOS/Linux)
python tools/fetch_demo_splats.py
python tools/fetch_realistic_env.py
cd server
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python server.py

# 3. Open the UI
http://localhost:5000

Within the UI:

1. Click Load Demo to populate the grid with three sample splats.
2. Change Sky to sunset or blue_sky, Ground to grass or grey.
3. Hit Enter Grid (3D) and explore with arrow keys (Shift for a boost, Esc to exit).
4. Draw new rectangles in the 2D view, attach your own .splat files, and save the project state.

---

Detailed Setup

Component	Notes
Python	3.10+. Used for the Flask API and tooling scripts.
Frontend	Plain ES modules served from public/. No bundler required.
Gaussian viewer	@mkkellogg/gaussian-splats-3d loaded dynamically.
Textures	tools/fetch_realistic_env.py downloads a blue-sky and sunset panorama + CC0 grass tile.
Demo splats	tools/fetch_demo_splats.py creates a seed nike.splat and tinted variants to showcase the UI.

Run loop (Windows one-click)

start-mini-worlds.bat
├─ creates/activates server/.venv
├─ installs requirements (Flask, requests)
├─ fetches demo splats + sky/ground textures
└─ launches Flask + opens http://localhost:5000

macOS / Linux equivalent

python tools/fetch_demo_splats.py
python tools/fetch_realistic_env.py
cd server
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python server.py

---

Integrating Image→Gaussian Splats

1. Pick a checkpoint – e.g. LGM (Large Gaussian Model) for fast single-image conversion, or DreamGaussian for higher-fidelity multi-view results.
2. Decide where to run it:
   • Local GPU: install PyTorch + CUDA, clone the repo, expose a CLI command the provider can call.
   • Hugging Face Inference Endpoint / Space: wrap the REST endpoint in our provider.
   • Replicate / Banana.dev: same idea; just fill in the URL + token.
3. Implement the provider under server/providers/ by extending the stub interface:

# server/providers/lgm_provider.py
class LGMProvider:
    def __init__(self, output_dir: Path, exe: Path):
        self.output_dir = output_dir
        self.exe = exe

    def image_to_3d(self, image_path: Path) -> Path:
        job_dir = self.output_dir / f"lgm_{int(time.time())}"
        job_dir.mkdir(parents=True, exist_ok=True)
        subprocess.run([
            self.exe,
            '--input', str(image_path),
            '--output', str(job_dir / 'scene.splat')
        ], check=True)
        return job_dir / 'scene.splat'

4. Configure the API (e.g. server/config.env):

PROVIDER=lgm
LGM_EXE=C:\tools\lgm\infer.exe

5. Call it from the UI: once the provider returns a .splat/.ksplat, the front-end automatically stages it under public/assets/generated/<job>/... and attaches it to the selected region.

---

Project Layout

mini-worlds/
├─ public/
│  ├─ index.html, app.js, viewer.js, env.js, api.js
│  ├─ assets/
│  │  ├─ generated/ (runtime jobs)
│  │  ├─ ground/grass.jpg
│  │  └─ skybox/day_clouds.jpg, sunset_clouds.jpg
│  └─ project.demo.json
├─ server/
│  ├─ server.py (Flask REST)
│  ├─ providers/ (pluggable inference backends)
│  └─ postprocess/ (optional .ply → .ksplat tools)
├─ tools/
│  ├─ fetch_demo_splats.py
│  ├─ fetch_realistic_env.py
│  └─ (add your own converters/scripts)
└─ README.md (this file)

---

Workflow Walkthrough

1. Draw regions in 2D – hold Space + drag to pan, Shift + drag edges to resize.
2. Attach content – either:
   • Click Attach Splat (.ply/.ksplat) and choose a file, or
   • Click Insert Splat (image→3D) once a provider is wired in.
3. Tweak environment – pick sky, ground, tile scale, and sky intensity from the sidebar, then hit Enter Grid (3D).
4. Explore – arrow keys to strafe, Shift for speed, Esc to return to 2D.
5. Save or export:
   • Save writes to localStorage (auto-loaded the next time you visit).
   • Export JSON for version control, shareable projects, or reproducing test cases.

---

Recommended Hugging Face Checkpoints

Model	Task	Pros	Notes
TencentARC/LGM	Single-image → 3D Gaussian	Very fast (seconds); good shape fidelity	Requires CUDA GPU (≥12 GB). Use the official CLI to output .ply / .splat.
dreamgaussian/dreamgaussian	Multi-view text/image → 3D	Higher fidelity, supports text prompts	Slower (minutes). Exports .ply; run tools/postprocess/convert_to_ksplat.py afterwards.
dyl1019/gaussian-splatting-samples	Sample splats	Ready-made .splat files for testing	CC-BY-4.0 dataset; great for regression tests.

If you rely on a Hugging Face endpoint, add your token to server/config.env and set PROVIDER=remote with the appropriate URLs.

---

Hardware Requirements

Role	Minimum	Recommended
Frontend	Any WebGL2-capable GPU (integrated Intel works)	Dedicated GPU for smoother splat sorting (NVIDIA GTX 1060+).
Backend (inference)	CPU-only if you use remote providers	Local GPU (RTX 3060 12 GB / RTX 4070 12 GB) for LGM or DreamGaussian.
RAM / Disk	8 GB RAM, ~2 GB disk for demo assets	16 GB RAM, SSD preferred for tmp splat files.

Sorting millions of splats is CPU-heavy. On low-end hardware toggle to blue-sky + grey ground, or reduce splat sizes via the provider.

---

Testing Matrix

Scenario	Steps	Expected
Smoke test	Run start-mini-worlds.bat, click Load Demo, Enter Grid	Viewer opens with three splat islands, blue sky, grass ground
Empty world	Clear localStorage (Reset), Enter Grid	Viewer opens with environment only, (empty) tag shown
Attach custom splat	Select rectangle, Attach Splat → choose .splat	Asset loads, camera recenters when you Enter World
Provider round-trip	Configure LGM provider, Insert Splat (image)	Job folder created under public/assets/generated/, asset attached
Persistence	Save, refresh page	Project auto-restored from localStorage
Export/import	Export JSON, Reset, Import	Layout restored from exported file

Document your own regression suite in docs/testing.md if you expand the project.

---

Roadmap

• Shared-memory / WASM sorting when COOP/COEP headers are available.
• Region-level annotations (notes, splat metadata, prompts).
• Camera bookmarks + cinematic fly-through mode.
• WebRTC peer session for collaborative layout.
• Procedural ground library (sand, snow, rock) with CC0 textures bundled locally.
• Cloud build recipe (Render/Gunicorn) for one-click deploy.

Contributions welcome—file an issue labelled “proposal” before opening large PRs.

---

Credits

• Gaussian rendering – Mark Kellogg and contributors of GaussianSplats3D.
• Splat research – DreamGaussian, LGM, 3D Gaussian Splatting teams.
• Texture photography – Unsplash contributors (linked via image URLs).
• CC0 assets – ambientCG (for optional ground textures).

If you showcase the tool, tag it #GaussianWorlds so we can see what you build!

_{Built with ❤️ for artists who prototype worlds faster than they can model meshes.}