Lift4D stages 1+2 on ZeroGPU: splat viewer + orbit video

.gitattributes

@@ -42,3 +42,6 @@ static/orbits/rhino.mp4 filter=lfs diff=lfs merge=lfs -text
 static/orbits/robot.mp4 filter=lfs diff=lfs merge=lfs -text
 static/orbits/yarn.mp4 filter=lfs diff=lfs merge=lfs -text
 static/videos/rhino_sam3d_orbit.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/horse_running.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/poodle_running.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/squirrel_eating.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,45 +1,61 @@
 ---
-title: Lift4D 4D Reconstruction
-emoji: ⚡️
+title: Lift4D
+emoji: 🐐
 colorFrom: gray
 colorTo: yellow
 sdk: gradio
 sdk_version: 6.22.0
-app_file: app.py
 python_version: "3.12"
-short_description: Interactive 4D Gaussian-splat gallery for Lift4D
+app_file: app.py
 pinned: false
+license: apache-2.0
+short_description: Text-prompted 4D reconstruction from a casual video
+startup_duration_timeout: 1h
 ---
 
-# Lift4D — 4D Reconstruction In-the-Wild
-
-Interactive gallery for [**Lift4D**](https://github.com/yehonathanlitman/Lift4D)
-(Litman et al., CMU), which recovers complete, temporally-consistent **4D
-Gaussian-splat assets** from single in-the-wild videos.
-
-Each scene is shown in **two viewers**:
-
-- **🟢 Gaussian-splat viewer** — the authors' own client-side WebGL 4D splat
-  player, rendering the genuine per-scene deformable Gaussians (canonical splats
-  + per-frame deformation nodes). Fully interactive: drag to orbit, scroll to
-  zoom, scrub the timestep.
-- **🎞️ Orbit-video viewer** — a turntable render of the deformed 4D Gaussians
-  (4 azimuths), baked offline from the same reconstruction (mirroring the
-  pipeline's own `save_orbit_video`). The `rhino` scene additionally exposes the
-  authors' official multi-stage orbit videos.
-
-The featured scene is **`goat`** — the in-the-wild custom clip documented in the
-repo README's custom-dataset path (`data/custom/goat`, mask `goat_1`).
-
-## Why a gallery, not live inference?
-
-Lift4D's full pipeline is a three-stage optimization — SAM 3 segmentation →
-Causal SAM3D per-frame 3D → Lift4D SC-GS training with Zero123 SDS guidance,
-thousands of iterations, tested on an A100 40GB — that needs gated weights and
-takes minutes-to-hours per asset. That cannot run inside a per-request GPU
-budget, so this Space serves the authors' **actual released 4D reconstructions**
-instead of running the pipeline live.
-
-Assets and viewer code are the authors' own (project page `lift4d.github.io`);
-the orbit videos are baked from the released splat params with a faithful port
-of the viewer's two-skinning deformation math.
+# Lift4D
+
+Front half of [**Lift4D**](https://github.com/yehonathanlitman/Lift4D) (CMU),
+run end to end on ZeroGPU. Give it a monocular clip and a text prompt naming
+the object, and it produces a **posed 3D Gaussian splat per video frame**
+plus a rendered **orbit video** of the reconstructed sequence.
+
+## What runs
+
+The Space follows the *custom dataset* workflow from the Lift4D README, and
+materialises exactly that on-disk layout for each request:
+
+```
+data/custom/<clip>/frames/frame_%06d.jpg
+data/custom/<clip>/masks/frame_%06d_<prompt>_1.png
+sam3d_output/custom_<clip>_<prompt>_1/<frame>/result.ply
+sam3d_output/custom_<clip>_<prompt>_1/<frame>/obj_transform.json
+```
+
+| Stage | Upstream entry point | Here |
+|---|---|---|
+| 1 | `segment_video.py` | frames extracted with OpenCV, SAM 3 text-prompt segmentation, masks written as `<stem>_<prompt>_1.png` |
+| 2 | `sam3d/run_inference.py --dataset custom` | resolved through the repo's own `lift4d_datasets` registry, then `InferencePipelinePointMap.run_video_inference(...)` with bi-directional SDEdit warm-starting |
+| 3 | `lift4d_scgs/train_lift4d_scgs.py` | **not run** — 10k–30k SDS optimisation steps, hours on an A100 |
+
+## Deviations from the reference implementation
+
+* **Stage 3 is out of scope.** The Lift4D SC-GS optimisation (Zero123 SDS,
+  10k + 20k iterations) cannot fit in a ZeroGPU call. What you see is the
+  per-frame causal reconstruction that stage 3 consumes.
+* **SAM 3 segmentation is per-frame** (`transformers` `Sam3Model` concept
+  segmentation with greedy IoU matching between frames) instead of
+  `segment_video.py`'s SAM 3 video tracker. Same result for the
+  single-object clips this demo targets.
+* **Depth Anything 3 pointmap alignment** and **silhouette pose refinement**
+  — both optional flags of `run_inference.py` — are off.
+* Decoding is `["gaussian"]` only; mesh post-processing, texture baking and
+  layout optimisation are disabled (they need `nvdiffrast` / FlexiCubes).
+* The orbit video is rendered with the upstream Inria 3DGS rasterizer
+  instead of `gsplat`, which has no sm_120 kernels. The object→camera
+  placement is taken verbatim from `run_inference.py:render_gs_frame`.
+
+## Notes
+
+Both weight repos are gated: this Space needs an `HF_TOKEN` secret with
+access to `facebook/sam-3d-objects` and `facebook/sam3`.

app.py

@@ -1,182 +1,308 @@
-"""
-Lift4D — interactive 4D reconstruction gallery.
-
-Lift4D (Litman et al., CMU) recovers complete 4D asset reconstructions from
-in-the-wild monocular videos through a 3-stage pipeline:
-
-    video ─▶ 1. Segment (SAM 3) ─▶ 2. Causal SAM3D (per-frame 3D)
-          ─▶ 3. Lift4D SC-GS training (fuse into a deformable Gaussian model) ─▶ 4D asset
-
-The full pipeline is a multi-stage optimization (SAM3 + SAM3D + SC-GS with SDS
-guidance, thousands of iterations, tested on an A100 40GB) that takes minutes-to-
-hours per asset and needs gated weights — it cannot run inside a per-request GPU
-budget. This Space instead showcases the authors' *actual* released 4D
-reconstructions in two dedicated viewers:
-
-  • Gaussian-splat viewer — the authors' own client-side WebGL 4D splat player
-    rendering the genuine per-scene deformable Gaussians (canonical splats +
-    per-frame deformation nodes), fully interactive (orbit / zoom / scrub time).
-  • Orbit-video viewer — turntable renders of the deformed 4D Gaussians, one
-    orbit per scene (mirroring the pipeline's own `save_orbit_video`), plus the
-    authors' official multi-stage orbit videos for the DAVIS `rhino` scene.
-
-The featured scene is `goat` — the in-the-wild custom clip documented in the
-repo README's custom-dataset path (data/custom/goat, mask goat_1).
-
-Serving: no GPU is used. Static assets (viewer HTML/JS, per-scene splat params,
-orbit + official videos) live under ./static and are served through Gradio 6's
-native static file endpoint (/gradio_api/file=...); the WebGL viewer is embedded
-in an iframe and fetches its assets relative to its own served URL.
+"""Lift4D — text-prompted 4D reconstruction from a monocular video.
+
+Runs the first two stages of the Lift4D pipeline
+(https://github.com/yehonathanlitman/Lift4D) through the *custom dataset*
+workflow documented in the repo README:
+
+    stage 1  segment_video.py        SAM 3 text prompt -> per-frame masks
+    stage 2  sam3d/run_inference.py  Causal SAM 3D Objects -> per-frame 3DGS
 """
 import os
-import gradio as gr
-
-HERE = os.path.dirname(os.path.abspath(__file__))
-STATIC = os.path.join(HERE, "static")
-
-# Expose the static asset tree to Gradio's file endpoint.
-gr.set_static_paths(paths=[STATIC])
-
-FILE = "/gradio_api/file="  # Gradio static-serve prefix
-VIEWER_URL = FILE + "static/viewer4d.html"
-PARAMS_URL = FILE + "static/params"
-
-# Scene catalogue: (name, dataset tag). goat first — the README custom-dataset path.
-SCENES = [
-    ("goat", "custom (README path)"),
-    ("rhino", "DAVIS"),
-    ("horsejump-low", "DAVIS"),
-    ("ball", "in-the-wild"),
-    ("bicycle", "in-the-wild"),
-    ("bird", "in-the-wild"),
-    ("robot", "in-the-wild"),
-    ("yarn", "in-the-wild"),
-]
-SCENE_NAMES = [s for s, _ in SCENES]
-SCENE_LABELS = {s: f"{s}  ·  {tag}" for s, tag in SCENES}
-DEFAULT_SCENE = "goat"
-
-# Authors' official multi-stage orbit videos (DAVIS rhino) — bundled locally.
-RHINO_OFFICIAL = [
-    ("Complete 4D reconstruction", "static/videos/rhino_complete.mp4"),
-    ("Per-frame SAM3D orbit", "static/videos/rhino_sam3d_orbit.mp4"),
-    ("Canonical Gaussians", "static/videos/rhino_canonical.mp4"),
-    ("Deformation nodes", "static/videos/rhino_nodes.mp4"),
-    ("Fine appearance nodes (delta)", "static/videos/rhino_delta.mp4"),
-]
-
-
-def _viewer_iframe(scene: str) -> str:
-    """HTML embed of the client-side WebGL 4D Gaussian-splat player for a scene."""
-    src = f"{VIEWER_URL}?embed=1&v=3&scene={scene}&params={PARAMS_URL}"
-    return (
-        f'<iframe src="{src}" title="Lift4D 4D Gaussian-splat viewer" '
-        'style="width:100%;height:560px;border:0;border-radius:12px;'
-        'background:#0d0f12;" allow="fullscreen"></iframe>'
+import shutil
+import subprocess
+import sys
+import time
+import uuid
+from pathlib import Path
+
+os.environ.setdefault("CUDA_HOME", "/usr/local/cuda")
+os.environ.setdefault("CONDA_PREFIX", "/usr/local")
+os.environ["LIDRA_SKIP_INIT"] = "true"
+os.environ["ATTN_BACKEND"] = "sdpa"
+os.environ["SPARSE_ATTN_BACKEND"] = "sdpa"
+os.environ["SPARSE_BACKEND"] = "spconv"
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+
+import spaces  # noqa: E402  (must precede torch)
+import gradio as gr  # noqa: E402
+import numpy as np  # noqa: E402
+from huggingface_hub import login, snapshot_download  # noqa: E402
+
+if os.environ.get("HF_TOKEN"):
+    login(token=os.environ["HF_TOKEN"])
+
+APP_ROOT = Path(__file__).parent
+sys.path.insert(0, str(APP_ROOT / "kaolin_stub"))
+
+LIFT4D_COMMIT = "aa295d66d1baabc6f9954b619f8b1bdb557bd862"
+WORK = Path("/tmp/lift4d")
+DATA_ROOT = WORK / "data"
+SAM3D_OUT = WORK / "sam3d_output"
+for d in (DATA_ROOT / "custom", SAM3D_OUT):
+    d.mkdir(parents=True, exist_ok=True)
+os.environ["LIFT4D_DATA_ROOT"] = str(DATA_ROOT)
+os.environ["LIFT4D_SAM3D_OUT"] = str(SAM3D_OUT)
+
+
+def _pip(*args):
+    r = subprocess.run([sys.executable, "-m", "pip", "install", "--no-cache-dir", *args],
+                       capture_output=True, text=True, timeout=1800)
+    print(f"  pip {'OK' if r.returncode == 0 else 'FAIL'}: {args[-1][:80]}", flush=True)
+    if r.returncode != 0:
+        print(r.stderr[-1500:], flush=True)
+    return r.returncode == 0
+
+
+print("=== runtime installs (need torch present) ===", flush=True)
+# utils3d pinned to the commit MoGe expects; MoGe supplies the geometry helpers
+# sam3d_objects.pipeline.utils.pointmap imports directly.
+_pip("--no-deps", "git+https://github.com/EasternJournalist/utils3d.git@3913c65d81e05e47b9f367250cf8c0f7462a0900")
+_pip("--no-deps", "git+https://github.com/microsoft/MoGe.git@a8c37341bc0325ca99b9d57981cc3bb2bd3e255b")
+# gaussian_render.py imports gsplat at module scope; only its python side is
+# touched (rendering goes through the Inria rasterizer / pytorch3d).
+_pip("--no-deps", "gsplat")
+
+LIFT4D_DIR = APP_ROOT / "Lift4D"
+if not LIFT4D_DIR.exists():
+    print("Cloning Lift4D...", flush=True)
+    subprocess.run(["git", "clone", "https://github.com/yehonathanlitman/Lift4D.git",
+                    str(LIFT4D_DIR)], check=True)
+    subprocess.run(["git", "checkout", "-q", LIFT4D_COMMIT], cwd=str(LIFT4D_DIR), check=True)
+
+SAM3D_DIR = LIFT4D_DIR / "sam3d"
+sys.path.insert(0, str(LIFT4D_DIR))     # lift4d_datasets
+sys.path.insert(0, str(SAM3D_DIR))      # sam3d_objects
+sys.path.insert(0, str(APP_ROOT))
+
+print("Downloading SAM 3D Objects checkpoints (gated: facebook/sam-3d-objects)...", flush=True)
+CKPT = Path(snapshot_download("facebook/sam-3d-objects", token=os.environ.get("HF_TOKEN")))
+hf_ckpt = CKPT / "checkpoints"
+local_ckpt = SAM3D_DIR / "checkpoints" / "hf"
+if hf_ckpt.exists() and not local_ckpt.exists():
+    local_ckpt.parent.mkdir(parents=True, exist_ok=True)
+    local_ckpt.symlink_to(hf_ckpt)
+CONFIG_PATH = str(local_ckpt / "pipeline.yaml")
+
+print("Downloading SAM 3 (gated: facebook/sam3)...", flush=True)
+try:
+    snapshot_download("facebook/sam3", token=os.environ.get("HF_TOKEN"),
+                      allow_patterns=["*.json", "*.txt", "*.safetensors", "*.model", "*.bin"])
+except Exception as exc:  # pragma: no cover
+    print(f"  sam3 prefetch failed ({exc}); will download on first run", flush=True)
+
+print("=== startup complete ===", flush=True)
+
+# Model construction needs a real GPU (the SAM3D constructors run device-mixing
+# tensor ops that ZeroGPU's startup CUDA shim rejects), so both models are
+# built inside @spaces.GPU and cached in the worker.
+_SAM3 = None
+_PIPE = None
+
+
+def _get_sam3():
+    global _SAM3
+    if _SAM3 is None:
+        from transformers import Sam3Model, Sam3Processor
+        model = Sam3Model.from_pretrained("facebook/sam3").to("cuda").eval()
+        processor = Sam3Processor.from_pretrained("facebook/sam3")
+        _SAM3 = (model, processor)
+    return _SAM3
+
+
+def _get_pipeline():
+    global _PIPE
+    if _PIPE is None:
+        import lift4d_stage12
+        _PIPE = lift4d_stage12.build_pipeline(CONFIG_PATH)
+        if getattr(_PIPE, "rendering_engine", "pytorch3d") != "pytorch3d":
+            _PIPE.rendering_engine = "pytorch3d"
+    return _PIPE
+
+
+MAX_FRAMES = 16
+
+
+def _estimate(*args, **kwargs):
+    n = 8
+    if len(args) > 2:
+        try:
+            n = int(args[2])
+        except (TypeError, ValueError):
+            pass
+    return int(min(420, 150 + 30 * n))
+
+
+@spaces.GPU(duration=_estimate)
+def reconstruct(video, prompt="", num_frames=8, consistency=0.2, seed=42,
+                stage1_steps=50, stage2_steps=25, orbit_steps=48,
+                progress=gr.Progress(track_tqdm=True)):
+    import torch
+    import lift4d_datasets as ds_registry
+    import lift4d_stage12 as stage12
+    import orbit_render
+    from PIL import Image
+
+    if video is None:
+        raise gr.Error("Please upload a video.")
+    if not (prompt or "").strip():
+        raise gr.Error("Please give a text prompt naming the object to reconstruct.")
+
+    t0 = time.time()
+    uid = f"clip_{uuid.uuid4().hex[:8]}"
+    video_dir = DATA_ROOT / "custom" / uid
+    frames_dir = video_dir / "frames"
+    masks_dir = video_dir / "masks"
+
+    # ---- stage 1: frames + SAM 3 masks in the README's custom layout --------
+    progress(0.02, desc="Extracting frames")
+    stems = stage12.extract_frames(video, frames_dir, int(num_frames))
+
+    model, processor = _get_sam3()
+    mask_name, masks = stage12.segment_frames(
+        frames_dir, masks_dir, stems, prompt, model, processor,
+        progress_cb=lambda f, d: progress(0.02 + 0.13 * f, desc=d))
+
+    preview_dir = video_dir / "preview"
+    preview_dir.mkdir(exist_ok=True)
+    gallery = []
+    for stem, m in zip(stems, masks):
+        rgb = np.asarray(Image.open(frames_dir / f"{stem}.jpg").convert("RGB"))
+        cut = (rgb * m[..., None] + 255 * (~m[..., None])).astype(np.uint8)
+        p = preview_dir / f"{stem}.jpg"
+        Image.fromarray(cut).save(p, quality=90)
+        gallery.append(str(p))
+
+    # ---- stage 2: causal SAM 3D Objects reconstruction ----------------------
+    progress(0.18, desc="Loading SAM 3D Objects")
+    pipe = _get_pipeline()
+    spec = ds_registry.resolve("custom", uid, mask_name)
+
+    progress(0.22, desc=f"Reconstructing {len(stems)} frames (causal SAM 3D)")
+    saved, frame_images, out_base = stage12.run_stage2(
+        pipe, spec, stems, mask_name,
+        seed=int(seed), stage1_steps=int(stage1_steps), stage2_steps=int(stage2_steps),
+        initial_frame_index=0, consistency_strength=float(consistency))
+    t_recon = time.time() - t0
+
+    # ---- placement, splat export, orbit render ------------------------------
+    progress(0.85, desc="Exporting Gaussian splats")
+    cam_frames, ply_paths = [], []
+    centers = []
+    for rec in saved:
+        gs = rec["gs"]
+        if gs is None:
+            continue
+        packed = orbit_render.to_camera_space(
+            gs, rec["scale"], rec["translation"], rec["rotation"], device="cuda")
+        cam_frames.append(packed)
+        centers.append(packed[0].median(dim=0).values)
+
+    if not cam_frames:
+        raise gr.Error("SAM 3D Objects returned no Gaussians for this clip.")
+
+    center = torch.stack(centers).mean(dim=0).cpu().numpy()
+    splat_dir = out_base / "splats"
+    splat_dir.mkdir(exist_ok=True)
+    for rec, packed in zip(saved, cam_frames):
+        p = splat_dir / f"{rec['label']}.ply"
+        orbit_render.write_viewer_ply(p, *packed, center=center)
+        ply_paths.append(str(p))
+
+    progress(0.92, desc="Rendering orbit video")
+    imgs = orbit_render.render_orbit(cam_frames, size=512,
+                                     orbit_steps=int(orbit_steps), device="cuda")
+    orbit_path = out_base / "orbit.mp4"
+    orbit_render.save_video(imgs, orbit_path, fps=12)
+
+    total = time.time() - t0
+    status = (
+        f"**Done in {total:.0f}s** — {len(saved)} frames reconstructed "
+        f"({t_recon:.0f}s in SAM 3D).\n\n"
+        f"Custom dataset: `data/custom/{uid}/frames` + "
+        f"`data/custom/{uid}/masks/<frame>_{mask_name}.png` → "
+        f"stage-1 output tag `{spec.tag}`."
     )
+    return str(orbit_path), ply_paths[0], gallery, status, str(splat_dir)
 
 
-def _orbit_path(scene: str):
-    p = os.path.join(HERE, "static", "orbits", f"{scene}.mp4")
-    return p if os.path.exists(p) else None
-
-
-def load_scene(scene: str):
-    """Load a scene into both viewers (Gaussian-splat player + orbit video).
-
-    Args:
-        scene: scene name, one of the catalogue keys (e.g. "goat", "rhino").
-
-    Returns:
-        A tuple of (splat_viewer_html, orbit_video_path).
-    """
-    if scene not in SCENE_NAMES:
-        scene = DEFAULT_SCENE
-    return _viewer_iframe(scene), _orbit_path(scene)
+def show_frame(splat_dir, idx):
+    """Swap the Gaussian-splat viewer to another reconstructed frame (CPU)."""
+    if not splat_dir:
+        return gr.update()
+    plys = sorted(Path(splat_dir).glob("*.ply"))
+    if not plys:
+        return gr.update()
+    i = max(0, min(int(idx), len(plys) - 1))
+    return gr.update(value=str(plys[i]))
 
 
 CSS = """
-#col-container { max-width: 1180px; margin: 0 auto; }
-.dark .gradio-container { color: var(--body-text-color); }
+.dark .gradio-container { background: #0b0b0d; }
+#viewers .wrap { min-height: 420px; }
 """
 
-with gr.Blocks(theme=gr.themes.Citrus(), css=CSS, title="Lift4D 4D Reconstruction") as demo:
-    with gr.Column(elem_id="col-container"):
-        gr.Markdown(
-            """
-            # ⚡️ Lift4D — 4D Reconstruction In-the-Wild
-            **Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild** ·
-            Litman, Ma, Shah, Ugrinovic, Kitani, De la Torre, Tulsiani (CMU)
-
-            Lift4D recovers a complete, temporally-consistent **4D Gaussian-splat asset**
-            from a single in-the-wild video: segment (SAM 3) → per-frame 3D (Causal SAM3D)
-            → fuse into canonical Gaussians + learned deformation nodes (SC-GS + SDS).
-
-            Pick a scene to explore its released reconstruction in **two viewers**: an
-            interactive **Gaussian-splat player** and a rendered **orbit video**.
-
-            [Project page](https://lift4d.github.io) ·
-            [arXiv](https://arxiv.org/abs/2606.23688) ·
-            [Code](https://github.com/yehonathanlitman/Lift4D)
-
-            > ℹ️ The full pipeline (gated SAM3/SAM3D/Zero123 weights, thousands of SC-GS
-            > optimization iterations on an A100) can't run per-request, so this Space
-            > serves the authors' **actual released 4D reconstructions**, not live inference.
-            """
-        )
-
-        scene = gr.Radio(
-            choices=[(SCENE_LABELS[s], s) for s in SCENE_NAMES],
-            value=DEFAULT_SCENE,
-            label="Scene",
-            info="goat is the in-the-wild custom-dataset clip from the repo README.",
-        )
-
-        with gr.Row(equal_height=False):
-            with gr.Column():
-                gr.Markdown(
-                    "### 🟢 Gaussian-splat viewer\n"
-                    "Interactive 4D splats — drag to orbit, scroll to zoom, scrub time."
-                )
-                splat_viewer = gr.HTML(_viewer_iframe(DEFAULT_SCENE))
-            with gr.Column():
-                gr.Markdown(
-                    "### 🎞️ Orbit-video viewer\n"
-                    "Turntable render of the deformed 4D Gaussians (4 azimuths)."
-                )
-                orbit_video = gr.Video(
-                    value=_orbit_path(DEFAULT_SCENE),
-                    label="Orbit render",
-                    autoplay=True,
-                    loop=True,
-                    height=360,
-                )
-
-        with gr.Accordion("Official multi-stage orbit videos (DAVIS rhino)", open=False):
-            gr.Markdown(
-                "The authors' released renders showing each pipeline stage of the "
-                "`rhino` reconstruction."
-            )
+with gr.Blocks(theme=gr.themes.Citrus(), css=CSS, title="Lift4D") as demo:
+    gr.Markdown(
+        "# Lift4D — 4D reconstruction from a casual video\n"
+        "Name an object in your clip and Lift4D's front half runs: **SAM 3** "
+        "segments it in every sampled frame, then **causal SAM 3D Objects** "
+        "reconstructs a posed Gaussian splat per frame with SDEdit temporal "
+        "warm-starting. Two viewers below: the per-frame **Gaussian splats** "
+        "and a rendered **orbit video** of the reconstructed sequence."
+    )
+
+    with gr.Row():
+        with gr.Column(scale=1):
+            video = gr.Video(label="Input video", height=280)
+            prompt = gr.Textbox(label="Object prompt", placeholder="e.g. goat",
+                                value="")
+            num_frames = gr.Slider(3, MAX_FRAMES, value=8, step=1,
+                                   label="Frames to reconstruct",
+                                   info="Sampled evenly across the clip. More frames = smoother 4D, longer run.")
+            run = gr.Button("Reconstruct", variant="primary")
+            with gr.Accordion("Advanced", open=False):
+                consistency = gr.Slider(0.0, 1.0, value=0.2, step=0.05,
+                                        label="Temporal consistency (SDEdit strength)",
+                                        info="Higher preserves more structure between frames; lower allows more deformation.")
+                seed = gr.Number(value=42, precision=0, label="Seed")
+                stage1_steps = gr.Slider(10, 50, value=50, step=5,
+                                         label="Sparse-structure steps")
+                stage2_steps = gr.Slider(10, 50, value=25, step=5,
+                                         label="Structured-latent steps")
+                orbit_steps = gr.Slider(16, 96, value=48, step=8,
+                                        label="Orbit video frames")
+
+        with gr.Column(scale=2, elem_id="viewers"):
             with gr.Row():
-                for _label, _path in RHINO_OFFICIAL:
-                    gr.Video(value=_path, label=_label, autoplay=False, loop=True, height=220)
-
-        gr.Examples(
-            examples=[[s] for s in SCENE_NAMES],
-            inputs=[scene],
-            outputs=[splat_viewer, orbit_video],
-            fn=load_scene,
-            cache_examples=False,
-            run_on_click=True,
-            label="Scenes",
-        )
-
-        scene.change(
-            fn=load_scene,
-            inputs=[scene],
-            outputs=[splat_viewer, orbit_video],
-            api_name="load_scene",
-        )
-
-if __name__ == "__main__":
-    demo.launch(mcp_server=True)
+                splat = gr.Model3D(label="Gaussian splats (per frame)",
+                                   height=400, clear_color=[0.05, 0.05, 0.06, 1.0])
+                orbit = gr.Video(label="Orbit video", height=400,
+                                 autoplay=True, loop=True)
+            frame_idx = gr.Slider(0, MAX_FRAMES - 1, value=0, step=1,
+                                  label="Splat viewer: frame")
+            status = gr.Markdown()
+            gallery = gr.Gallery(label="SAM 3 segmentation (stage 1)", columns=8,
+                                 height=130, object_fit="contain")
+            splat_dir = gr.Textbox(visible=False)
+
+    inputs = [video, prompt, num_frames, consistency, seed,
+              stage1_steps, stage2_steps, orbit_steps]
+    outputs = [orbit, splat, gallery, status, splat_dir]
+
+    run.click(reconstruct, inputs=inputs, outputs=outputs)
+    frame_idx.change(show_frame, inputs=[splat_dir, frame_idx], outputs=[splat])
+
+    gr.Examples(
+        examples=[
+            ["examples/horse_running.mp4", "horse"],
+            ["examples/poodle_running.mp4", "dog"],
+            ["examples/squirrel_eating.mp4", "squirrel"],
+        ],
+        inputs=[video, prompt],
+        outputs=outputs,
+        fn=reconstruct,
+        cache_examples=True,
+        cache_mode="lazy",
+    )
+
+demo.queue(max_size=8).launch(mcp_server=True, show_error=True)

examples/horse_running.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2ad29b7a01bb89d0a7255d5eccb389e049a4889fb39a05d6e5212c6f77dbd68
+size 4755442

examples/poodle_running.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:586b3e8f9b2a775d51f199d132dabbe3d106968601e49bd066c41786898d23c3
+size 4173175

examples/squirrel_eating.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d75addbe53e3780b202dd733a9fc0113b79d5cd10e7b33d42ca29c0ed46d2fb
+size 2120607

kaolin_stub/kaolin/__init__.py

@@ -0,0 +1 @@
+"""Kaolin stub — flexicubes only needs utils.testing.check_tensor."""

kaolin_stub/kaolin/utils/__init__.py

@@ -0,0 +1,2 @@
+"""kaolin.utils stub."""
+from . import testing

kaolin_stub/kaolin/utils/testing.py

@@ -0,0 +1,4 @@
+"""kaolin.utils.testing stub."""
+def check_tensor(tensor, shape=None, dtype=None, device=None, throw=True):
+    """No-op tensor check."""
+    return True

lift4d_stage12.py

@@ -0,0 +1,284 @@
+"""Lift4D stages 1 + 2, ported to a single ZeroGPU process.
+
+This is a faithful port of the two CLI entry points documented in the Lift4D
+README's *custom dataset* workflow:
+
+    stage 1   segment_video.py       frames + text prompt -> per-frame masks
+    stage 2   sam3d/run_inference.py masks -> per-frame Gaussian reconstruction
+
+Both stages write into (and read from) exactly the on-disk layout the README
+specifies, via the repo's own ``lift4d_datasets`` registry:
+
+    $LIFT4D_DATA_ROOT/custom/<video>/frames/frame_%06d.jpg
+    $LIFT4D_DATA_ROOT/custom/<video>/masks/frame_%06d_<prompt>_<inst>.png
+    $LIFT4D_SAM3D_OUT/custom_<video>_<prompt>_<inst>/<frame_stem>/result.ply
+    $LIFT4D_SAM3D_OUT/custom_<video>_<prompt>_<inst>/<frame_stem>/obj_transform.json
+
+Deviations from the upstream CLIs are listed in the Space README.
+"""
+from __future__ import annotations
+
+import json
+import os
+import re
+from pathlib import Path
+from typing import List, Optional, Tuple
+
+import numpy as np
+
+
+# --------------------------------------------------------------------------
+# stage 1 -- SAM 3 text-prompted segmentation
+# --------------------------------------------------------------------------
+
+def clean_prompt(prompt: str) -> str:
+    """Match segment_video.py: spaces become underscores."""
+    cleaned = re.sub(r"\s+", "_", prompt.strip())
+    cleaned = re.sub(r"[^0-9A-Za-z_\-]", "", cleaned)
+    return cleaned or "object"
+
+
+def extract_frames(video_path: str, frames_dir: Path, num_frames: int,
+                   max_side: int = 720) -> List[str]:
+    """Evenly sample ``num_frames`` frames and write frame_%06d.jpg.
+
+    Mirrors segment_video.py's extract_frames_from_video (cv2, JPEG q95) but
+    samples evenly across the whole clip instead of taking a prefix, so a
+    short demo run still covers the full motion.
+    """
+    import cv2
+
+    frames_dir.mkdir(parents=True, exist_ok=True)
+    cap = cv2.VideoCapture(str(video_path))
+    if not cap.isOpened():
+        raise RuntimeError(f"Could not open video: {video_path}")
+
+    total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+    if total <= 0:  # some containers do not report a frame count
+        total = 0
+        while cap.grab():
+            total += 1
+        cap.release()
+        cap = cv2.VideoCapture(str(video_path))
+    if total <= 0:
+        raise RuntimeError("Video appears to contain no frames.")
+
+    num_frames = max(2, min(int(num_frames), total))
+    wanted = sorted({int(round(i * (total - 1) / (num_frames - 1)))
+                     for i in range(num_frames)})
+
+    stems: List[str] = []
+    idx = 0
+    want_ptr = 0
+    while want_ptr < len(wanted):
+        ok, frame = cap.read()
+        if not ok:
+            break
+        if idx == wanted[want_ptr]:
+            h, w = frame.shape[:2]
+            scale = max_side / float(max(h, w))
+            if scale < 1.0:
+                frame = cv2.resize(frame, (int(round(w * scale)), int(round(h * scale))),
+                                   interpolation=cv2.INTER_AREA)
+            stem = f"frame_{len(stems):06d}"
+            cv2.imwrite(str(frames_dir / f"{stem}.jpg"), frame,
+                        [cv2.IMWRITE_JPEG_QUALITY, 95])
+            stems.append(stem)
+            want_ptr += 1
+        idx += 1
+    cap.release()
+
+    if len(stems) < 2:
+        raise RuntimeError("Could not decode at least two frames from the video.")
+    return stems
+
+
+def _iou(a: np.ndarray, b: np.ndarray) -> float:
+    inter = np.logical_and(a, b).sum()
+    union = np.logical_or(a, b).sum()
+    return float(inter) / float(union) if union else 0.0
+
+
+def segment_frames(frames_dir: Path, masks_dir: Path, stems: List[str],
+                   prompt: str, model, processor, threshold: float = 0.4,
+                   progress_cb=None) -> Tuple[str, List[np.ndarray]]:
+    """Stage 1: write `<stem>_<prompt>_1.png` masks for every frame.
+
+    segment_video.py runs SAM 3's video tracker to propagate the first frame's
+    instance.  Here we run SAM 3 concept segmentation per frame (the API
+    exposed by `transformers`) and keep temporal identity by greedily matching
+    each frame's masks against the previous frame's mask by IoU -- the same
+    single-instance result for the single-object clips this demo targets.
+    """
+    import torch
+    from PIL import Image
+
+    masks_dir.mkdir(parents=True, exist_ok=True)
+    name = clean_prompt(prompt)
+    prev: Optional[np.ndarray] = None
+    masks: List[np.ndarray] = []
+
+    for i, stem in enumerate(stems):
+        if progress_cb is not None:
+            progress_cb(i / max(1, len(stems)), f"SAM 3 segmentation {i + 1}/{len(stems)}")
+        image = Image.open(frames_dir / f"{stem}.jpg").convert("RGB")
+        inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
+        with torch.inference_mode():
+            outputs = model(**inputs)
+        results = processor.post_process_instance_segmentation(
+            outputs,
+            threshold=threshold,
+            mask_threshold=0.5,
+            target_sizes=inputs.get("original_sizes").tolist(),
+        )[0]
+        cand = [np.asarray(m, dtype=bool) for m in
+                np.asarray(results["masks"].cpu().numpy() if hasattr(results["masks"], "cpu")
+                           else results["masks"])] if len(results["masks"]) else []
+        scores = [float(s) for s in torch.as_tensor(results["scores"]).cpu().numpy()] if cand else []
+
+        chosen: Optional[np.ndarray] = None
+        if cand:
+            if prev is None:
+                chosen = cand[int(np.argmax(scores))]
+            else:
+                ious = [_iou(prev, m) for m in cand]
+                if max(ious) > 0.05:
+                    chosen = cand[int(np.argmax(ious))]
+                else:
+                    chosen = cand[int(np.argmax(scores))]
+        if chosen is None:
+            if prev is None:
+                raise RuntimeError(
+                    f"SAM 3 found no '{prompt}' in the first sampled frame. "
+                    "Try a different prompt or a different clip."
+                )
+            chosen = prev  # keep the last good mask rather than dropping the frame
+
+        prev = chosen
+        masks.append(chosen)
+        Image.fromarray((chosen * 255).astype(np.uint8)).save(
+            masks_dir / f"{stem}_{name}_1.png")
+
+    return f"{name}_1", masks
+
+
+# --------------------------------------------------------------------------
+# stage 2 -- Causal SAM 3D Objects per-frame reconstruction
+# --------------------------------------------------------------------------
+
+def build_pipeline(config_path: str):
+    """Instantiate SAM3D's InferencePipelinePointMap.
+
+    Equivalent to Lift4D's ``sam3d/notebook/inference.py:Inference.__init__``
+    (same three config overrides) minus its module-level kaolin.visualize /
+    plotly / matplotlib imports, which are only used by notebook widgets.
+    """
+    from omegaconf import OmegaConf
+    from hydra.utils import instantiate
+
+    import sam3d_objects  # noqa: F401  (guarded by LIDRA_SKIP_INIT)
+    # The attention modules snapshot ATTN_BACKEND/SPARSE_ATTN_BACKEND from the
+    # environment at import time, and inference_pipeline.set_attention_backend()
+    # rewrites them at *its* import time. Import the attention modules first so
+    # they stay on sdpa.
+    import sam3d_objects.model.backbone.tdfy_dit.modules.attention  # noqa: F401
+    import sam3d_objects.model.backbone.tdfy_dit.modules.sparse  # noqa: F401
+    from sam3d_objects.pipeline.inference_pipeline_pointmap import (  # noqa: F401
+        InferencePipelinePointMap,
+    )
+
+    config = OmegaConf.load(config_path)
+    config.rendering_engine = "pytorch3d"   # disable nvdiffrast
+    config.compile_model = False
+    config.workspace_dir = os.path.dirname(config_path)
+    return instantiate(config)
+
+
+def run_stage2(pipeline, spec, stems: List[str], mask_name: str,
+               seed: int = 42, stage1_steps: int = 50, stage2_steps: int = 25,
+               initial_frame_index: int = 0, consistency_strength: float = 0.2,
+               output_dir: Optional[str] = None):
+    """Port of ``run_video_inference_cli`` for ``--dataset custom``.
+
+    Same frame/mask resolution (MASK_SUFFIX -> ``<stem>_<mask_name>``), same
+    ``run_video_inference`` call, same per-frame ``result.ply`` /
+    ``obj_transform.json`` outputs under the registry's output dir.
+    DA3 pointmap alignment and silhouette pose refinement (both optional in
+    the CLI) are off.
+    """
+    from sam3d_objects.pipeline.video_utils import load_video_frames_and_masks
+
+    mask_names = [f"{s}_{mask_name}" for s in stems]
+    frame_images, frame_masks = load_video_frames_and_masks(
+        frames_dir=Path(spec.frames_dir),
+        masks_dir=Path(spec.masks_dir),
+        image_names=stems,
+        mask_names=mask_names,
+    )
+
+    output_base = Path(output_dir or spec.sam3d_output_dir())
+    output_base.mkdir(parents=True, exist_ok=True)
+
+    results = pipeline.run_video_inference(
+        frame_images=frame_images,
+        frame_masks=frame_masks,
+        seed=seed,
+        stage1_inference_steps=stage1_steps,
+        stage2_inference_steps=stage2_steps,
+        decode_formats=["gaussian"],
+        with_mesh_postprocess=False,
+        with_texture_baking=False,
+        use_vertex_color=True,
+        initial_frame_index=initial_frame_index,
+        consistency_strength=consistency_strength,
+        pointmaps=None,
+        pointmap_intrinsics=None,
+    )
+
+    def frame_focal(result):
+        fi = result["frame_idx"]
+        gt_h, gt_w = frame_images[fi].shape[:2]
+        intr = result.get("intrinsics", None)
+        if intr is not None:
+            f_px = float(np.asarray(intr.detach().cpu()).reshape(3, 3)[1, 1] * gt_h)
+            if f_px > 0:
+                return f_px
+        return max(gt_h, gt_w) * 1.2
+
+    saved = []
+    for result in results:
+        fi = result["frame_idx"]
+        frame_label = stems[fi] if 0 <= fi < len(stems) else f"frame_{fi:04d}"
+        frame_dir = output_base / frame_label
+        frame_dir.mkdir(parents=True, exist_ok=True)
+
+        gs = result.get("gs")
+        if gs is None:
+            gaussians = result.get("gaussian")
+            if isinstance(gaussians, list) and gaussians:
+                gs = gaussians[0]
+        if gs is not None:
+            gs.save_ply(str(frame_dir / "result.ply"))
+
+        obj_transforms = {}
+        for key in ("scale", "translation", "rotation"):
+            if key in result:
+                obj_transforms[key] = result[key].cpu().numpy().tolist()
+        obj_transforms["focal"] = frame_focal(result)
+        with open(frame_dir / "obj_transform.json", "w") as fh:
+            json.dump(obj_transforms, fh, indent=4)
+
+        saved.append({
+            "frame_idx": fi,
+            "label": frame_label,
+            "dir": str(frame_dir),
+            "gs": gs,
+            "scale": result.get("scale"),
+            "translation": result.get("translation"),
+            "rotation": result.get("rotation"),
+            "focal": obj_transforms["focal"],
+            "size": frame_images[fi].shape[:2],
+        })
+
+    saved.sort(key=lambda r: r["frame_idx"])
+    return saved, frame_images, output_base

orbit_render.py

@@ -0,0 +1,240 @@
+"""Camera-space placement, 3DGS PLY export and orbit rendering for Lift4D.
+
+The object->camera convention is taken verbatim from Lift4D's
+``sam3d/run_inference.py:render_gs_frame``:
+
+    p_p3d = compose_transform(scale, quaternion_to_matrix(q_l2c), trans)
+                .transform_points(p_local)                # PyTorch3D row-vector
+    p_cam = p_p3d @ diag(-1, -1, 1)
+
+PyTorch3D's ``Transform3d.rotate(R)`` applies ``p_row @ R``, i.e. R^T in the
+usual column-vector sense -- which is why run_inference composes the Gaussian
+orientations with ``quaternion_invert(q_l2c)``.  Written as one rigid map:
+
+    R_o2c = diag(-1,-1,1) @ R(q_l2c)^T
+    t_o2c = diag(-1,-1,1) @ t
+    p_cam = R_o2c @ (scale * p_local) + t_o2c
+
+Camera space is the usual OpenCV/3DGS frame (+x right, +y down, +z forward).
+"""
+from __future__ import annotations
+
+import math
+from typing import List
+
+import numpy as np
+import torch
+
+SH_C0 = 0.28209479177387814
+
+
+def _flip(device):
+    """diag(-1, -1, 1): PyTorch3D -> camera, as in run_inference.py."""
+    return torch.tensor([[-1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, 1.0]],
+                        dtype=torch.float32, device=device)
+
+
+def _mat_to_quat(R: torch.Tensor) -> torch.Tensor:
+    from pytorch3d.transforms import matrix_to_quaternion
+    return matrix_to_quaternion(R)
+
+
+def to_camera_space(gs, scale_t, trans_t, rot_t, device="cuda"):
+    """Return (means, quats, scales, colors, opacities) in camera space."""
+    from pytorch3d.transforms import quaternion_to_matrix, quaternion_multiply
+
+    xyz = gs.get_xyz.to(device).float()
+    quats = gs.get_rotation.to(device).float()
+    scales = gs.get_scaling.to(device).float()
+    opac = gs.get_opacity.to(device).float().reshape(-1)
+    dc = gs._features_dc.to(device).float()
+    if dc.dim() == 3:
+        dc = dc.squeeze(1)
+    colors = torch.clamp(dc * SH_C0 + 0.5, 0.0, 1.0)
+
+    s = scale_t.to(device).float().reshape(-1)
+    if s.numel() == 1:
+        s = s.repeat(3)
+    t = trans_t.to(device).float().reshape(3)
+    q = rot_t.to(device).float().reshape(4)
+
+    M = _flip(device)
+    R_o2c = M @ quaternion_to_matrix(q).transpose(0, 1)
+    t_o2c = M @ t
+    q_o2c = _mat_to_quat(R_o2c)
+
+    means = (R_o2c @ (xyz * s).transpose(0, 1)).transpose(0, 1) + t_o2c
+    quats = quaternion_multiply(q_o2c.unsqueeze(0).expand(quats.shape[0], -1), quats)
+    scales = scales * s
+    return means, quats, scales, colors, opac
+
+
+# --------------------------------------------------------------------------
+# 3DGS PLY export for gr.Model3D
+# --------------------------------------------------------------------------
+
+# camera space (y down, z forward) -> viewer space (y up, z toward viewer)
+_VIEW_FLIP = np.array([[1.0, 0.0, 0.0], [0.0, -1.0, 0.0], [0.0, 0.0, -1.0]], dtype=np.float32)
+_VIEW_QUAT = np.array([0.0, 1.0, 0.0, 0.0], dtype=np.float32)  # wxyz, Rx(180 deg)
+
+
+def _quat_mul_np(a, b):
+    aw, ax, ay, az = a
+    bw, bx, by, bz = b[:, 0], b[:, 1], b[:, 2], b[:, 3]
+    return np.stack([
+        aw * bw - ax * bx - ay * by - az * bz,
+        aw * bx + ax * bw + ay * bz - az * by,
+        aw * by - ax * bz + ay * bw + az * bx,
+        aw * bz + ax * by - ay * bx + az * bw,
+    ], axis=-1)
+
+
+def write_viewer_ply(path, means, quats, scales, colors, opacities, center=None):
+    """Write a standard 3DGS .ply that gr.Model3D / gsplat.js can display.
+
+    Gaussians come in camera space; they are rotated into the viewer's y-up
+    frame and recentred so the object appears upright and framed.
+    """
+    m = means.detach().cpu().numpy().astype(np.float32)
+    q = quats.detach().cpu().numpy().astype(np.float32)
+    s = scales.detach().cpu().numpy().astype(np.float32)
+    c = colors.detach().cpu().numpy().astype(np.float32)
+    o = opacities.detach().cpu().numpy().astype(np.float32).reshape(-1)
+
+    if center is None:
+        center = np.median(m, axis=0)
+    m = (m - center) @ _VIEW_FLIP.T
+    q = _quat_mul_np(_VIEW_QUAT, q)
+
+    eps = 1e-6
+    f_dc = (c - 0.5) / SH_C0
+    opacity = np.log(np.clip(o, eps, 1 - eps) / (1 - np.clip(o, eps, 1 - eps)))
+    scale_log = np.log(np.clip(s, eps, None))
+    normals = np.zeros_like(m)
+
+    data = np.concatenate([m, normals, f_dc, opacity[:, None], scale_log, q], axis=1).astype(np.float32)
+
+    header = (
+        "ply\nformat binary_little_endian 1.0\n"
+        f"element vertex {data.shape[0]}\n"
+        "property float x\nproperty float y\nproperty float z\n"
+        "property float nx\nproperty float ny\nproperty float nz\n"
+        "property float f_dc_0\nproperty float f_dc_1\nproperty float f_dc_2\n"
+        "property float opacity\n"
+        "property float scale_0\nproperty float scale_1\nproperty float scale_2\n"
+        "property float rot_0\nproperty float rot_1\nproperty float rot_2\nproperty float rot_3\n"
+        "end_header\n"
+    )
+    with open(path, "wb") as fh:
+        fh.write(header.encode("ascii"))
+        fh.write(data.tobytes())
+    return str(path)
+
+
+# --------------------------------------------------------------------------
+# orbit rendering with the Inria 3DGS rasterizer
+# --------------------------------------------------------------------------
+
+def _proj_matrix(znear, zfar, fovx, fovy, device):
+    tx, ty = math.tan(fovx / 2), math.tan(fovy / 2)
+    P = torch.zeros(4, 4, dtype=torch.float32, device=device)
+    P[0, 0] = 1.0 / tx
+    P[1, 1] = 1.0 / ty
+    P[2, 2] = zfar / (zfar - znear)
+    P[2, 3] = -(zfar * znear) / (zfar - znear)
+    P[3, 2] = 1.0
+    return P
+
+
+def _look_at(cam_pos, target, device):
+    """World->camera rotation/translation in the OpenCV (y-down) convention."""
+    f = target - cam_pos
+    f = f / torch.norm(f).clamp_min(1e-8)
+    up_hint = torch.tensor([0.0, -1.0, 0.0], device=device)
+    if torch.abs(torch.dot(f, up_hint)) > 0.999:
+        up_hint = torch.tensor([0.0, 0.0, 1.0], device=device)
+    r = torch.cross(f, up_hint, dim=0)
+    r = r / torch.norm(r).clamp_min(1e-8)
+    d = torch.cross(f, r, dim=0)
+    R_wc = torch.stack([r, d, f], dim=0)
+    t_wc = -R_wc @ cam_pos
+    return R_wc, t_wc
+
+
+def render_orbit(frames, size=512, orbit_steps=48, fov_deg=42.0,
+                 elevation_deg=12.0, bg=(1.0, 1.0, 1.0), device="cuda"):
+    """Render a 360-degree orbit that also plays the reconstructed sequence.
+
+    ``frames`` is a list of (means, quats, scales, colors, opacities) tuples in
+    camera space, one per reconstructed video frame.  Step ``t`` of the orbit
+    shows the frame at ``t / orbit_steps`` of the sequence, so the video shows
+    both the object's motion and every side of the geometry.
+    """
+    from diff_gaussian_rasterization import (GaussianRasterizationSettings,
+                                             GaussianRasterizer)
+    from pytorch3d.transforms import matrix_to_quaternion, quaternion_multiply
+
+    all_means = torch.cat([f[0] for f in frames], dim=0)
+    center = all_means.median(dim=0).values
+    rad = torch.quantile((all_means - center).norm(dim=1), 0.97).clamp_min(1e-3)
+
+    fov = math.radians(fov_deg)
+    dist = float(rad) / math.tan(fov / 2.0) * 1.25
+    el = math.radians(elevation_deg)
+    bg_t = torch.tensor(bg, dtype=torch.float32, device=device)
+    P = _proj_matrix(0.01, 100.0, fov, fov, device)
+    tan_half = math.tan(fov / 2.0)
+
+    out = []
+    n = len(frames)
+    with torch.no_grad():
+        for t in range(orbit_steps):
+            az = 2.0 * math.pi * t / orbit_steps
+            direction = torch.tensor([
+                math.sin(az) * math.cos(el),
+                -math.sin(el),
+                -math.cos(az) * math.cos(el),
+            ], dtype=torch.float32, device=device)
+            cam_pos = center + dist * direction
+            R_wc, t_wc = _look_at(cam_pos, center, device)
+
+            # Gaussians are moved into the orbit camera's frame here, so the
+            # rasterizer gets an identity view matrix and a pure camera->clip
+            # projection.
+            means, quats, scales, colors, opac = frames[min(int(t * n / orbit_steps), n - 1)]
+            q_wc = matrix_to_quaternion(R_wc)
+            quats_v = quaternion_multiply(q_wc.unsqueeze(0).expand(quats.shape[0], -1), quats)
+            means_v = (R_wc @ means.transpose(0, 1)).transpose(0, 1) + t_wc
+
+            settings = GaussianRasterizationSettings(
+                image_height=size, image_width=size,
+                tanfovx=tan_half, tanfovy=tan_half,
+                bg=bg_t, scale_modifier=1.0,
+                viewmatrix=torch.eye(4, dtype=torch.float32, device=device),
+                projmatrix=P.transpose(0, 1).contiguous(),
+                sh_degree=0,
+                campos=torch.zeros(3, dtype=torch.float32, device=device),
+                prefiltered=False, debug=False,
+            )
+            rasterizer = GaussianRasterizer(raster_settings=settings)
+            screen = torch.zeros_like(means_v)
+            image, _ = rasterizer(
+                means3D=means_v.contiguous(),
+                means2D=screen,
+                shs=None,
+                colors_precomp=colors.contiguous(),
+                opacities=opac.reshape(-1, 1).contiguous(),
+                scales=scales.contiguous(),
+                rotations=quats_v.contiguous(),
+                cov3D_precomp=None,
+            )
+            frame = (image.clamp(0, 1).permute(1, 2, 0) * 255).to(torch.uint8).cpu().numpy()
+            out.append(frame)
+    return out
+
+
+def save_video(frames: List[np.ndarray], path, fps=12):
+    import imageio.v2 as imageio
+    imageio.mimwrite(str(path), frames, fps=fps, quality=8, macro_block_size=1,
+                     codec="libx264", output_params=["-pix_fmt", "yuv420p"])
+    return str(path)

requirements.txt

@@ -1,3 +1,43 @@
-# No extra runtime deps: the app only uses gradio (preinstalled by the SDK) and
-# the Python stdlib. Orbit videos are pre-baked offline (see render_orbits.py,
-# not shipped) so numpy/imageio are not needed at runtime.
+torchvision
+
+# --- prebuilt sm_120 (Blackwell) CUDA wheels, cell pt211-cu130-cp312 ---
+# real pytorch3d (SAM3D's pose/transform math + its "pytorch3d" rendering engine)
+https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt211-cu130-cp312/pytorch3d-0.7.9-cp312-cp312-linux_x86_64.whl
+# upstream Inria 3DGS rasterizer, used for the orbit video
+https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt211-cu130-cp312/diff_gaussian_rasterization-0.0.0-cp312-cp312-linux_x86_64.whl
+
+# pytorch3d runtime deps
+numpy
+iopath
+fvcore
+
+# sparse backbone for SAM 3D Objects
+spconv-cu124==2.3.8
+
+hydra-core==1.3.2
+omegaconf
+loguru
+tqdm
+Pillow
+opencv-python-headless
+scipy
+matplotlib
+easydict
+optree
+lightning
+safetensors
+timm
+astor
+transformers
+accelerate
+einops
+trimesh
+plyfile
+xatlas
+igraph
+open3d
+pyvista
+pymeshfix
+seaborn
+imageio
+imageio-ffmpeg