Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,76 @@
+---
+license: other
+license_name: fair-noncommercial-research-derivative
+license_link: https://github.com/facebookresearch/vggt-omega/blob/main/LICENSE
+tags: [3d-gaussian-splatting, novel-view-synthesis, autonomous-driving, vggt, feed-forward]
+---
+
+# MapVGGT — Map-grounded feed-forward 3DGS on a VGGT-Omega backbone (PRIVATE)
+
+> **PRIVATE research artifact.** Non-commercial, research-only. See **License** below before any use.
+
+MapVGGT is a feed-forward novel-view-synthesis model for driving scenes:
+**VGGT-Omega (1B)** predicts per-pixel **metric depth**; each input pixel is lifted to a
+world-space 3D Gaussian (positions from depth + **known** camera poses); a per-pixel head
+predicts opacity/scale/rotation; the union is rendered with `gsplat`; a small 2D **UNet
+refines** the rendered image. MapGS components (HD-map–anchored tokens, scene-graph
+dynamics, map-depth / free-space losses) are included but — see results — found neutral.
+
+## Honest results (held-out-SCENE, segment-disjoint Waymo, 40 distinct scenes, 256×448, n_in=8)
+
+| model | PSNR | SSIM | notes |
+|---|---|---|---|
+| VGGT-Omega + gentle finetune (backbone) | 21.7 | 0.66 | `abl_base_best` |
+| + MAGT map tokens + scene-graph dynamics | 21.7 | 0.66 | `abl_full_best` — **neutral** (ablation) |
+| **+ UNet render-refine** | **22.67** | **0.689** | `mapvggt_refine_best` — **headline** |
+
+**Be candid about scope.** This is a **research/system artifact, not SOTA**: ~22.7 dB is
+**~5 dB below** published feed-forward driving NVS (DGGT 27.4, PointForward 28.5, on
+different protocols). Established by clean ablation: the entire gain over a generic
+backbone is **VGGT-Omega + gentle backbone finetuning**; the single extra lever that
+moved the metric is the **UNet refine (+0.85 dB)**. HD-map tokens, scene-graph dynamics,
+higher resolution, multi-view color fusion, uncertainty-shaped covariance, and a skybox
+were all **measured neutral** on this metric (the image-space UNet subsumes them). The
+binding constraint is data scale (1/3 Waymo, ~1157 clips; overfits ~step 1000). Per-clip
+PSNR anti-correlates with view-extrapolation distance (r=-0.57): the model is strong on
+slow/overlapping scenes, weak on fast ego-motion / disocclusion.
+
+## Contents
+- `mapvggt/` — model (`model.py`), heads (`heads.py`: MAGT map tokens, scene-graph dynamics),
+  `refine.py` (RefineUNet). `crosscolor.py` / `uncertainty.py` are **experimental, validated
+  negative** (kept for the record; not used in training).
+- `mapgs/` — data pipeline (unified clip format, Waymo/AV2 converters), HD-map, losses, metrics.
+- `scripts/` — `train_mapvggt_refine.py` (main trainer), `train_mapvggt_full.py` (map+dyn),
+  `eval_mapvggt.py` (canonical loader + held-out eval), data-restore utilities.
+- `checkpoints/` — `mapvggt_refine_best.safetensors` (headline 22.67), `abl_base_best`,
+  `abl_full_best`. **Each ~4.6 GB and embeds the finetuned VGGT-Omega 1B backbone** (keys
+  `model.vggt.*`, `model.head.*`, `unet.*` for the refine ckpt).
+
+## NOT included (by design)
+- **Base VGGT-Omega weights** (`vggt_omega_1b_512.pt`) — obtain from its FAIR-licensed source;
+  set `MAPVGGT_VGGT_CKPT`. (Our refine ckpt already contains a finetuned copy of these weights.)
+- **Training data** — Waymo Open clips (its license **forbids redistribution**) and AV2 clips
+  (regenerate with `mapgs/data/convert/*` from your own licensed copies).
+- Vendored clones (`_vggt_omega_repo`, `_tokengs_repo`); clone yourself and set `VGGT_OMEGA_REPO`.
+
+## Usage
+```bash
+export VGGT_OMEGA_REPO=/path/to/vggt-omega           # facebookresearch/vggt-omega clone
+export MAPVGGT_VGGT_CKPT=/path/to/vggt_omega_1b_512.pt  # base weights (FAIR-licensed)
+# eval the released checkpoint on a segment-disjoint Waymo val split:
+python -m scripts.eval_mapvggt --ckpt checkpoints/mapvggt_refine_best.safetensors \
+       --roots /path/to/data/unified/waymo
+```
+The refine checkpoint round-trips to 22.67±3.76 / 0.689 via `scripts/eval_mapvggt.py`.
+
+## License & provenance (read before use)
+- **Derivative of VGGT-Omega (Meta FAIR), under the FAIR Noncommercial Research License.**
+  The checkpoints contain finetuned VGGT-Omega weights → they inherit FAIR terms:
+  **non-commercial, research-only; do not redistribute.** This repo is **PRIVATE** for that reason.
+  ⚠️ Commercial use (incl. by a commercial org) is **not permitted** under FAIR terms.
+- Lineage: **TokenGS** (NVIDIA, research-only) — earlier backbone, code under Apache-2.0;
+  **Depth-Anything-V2** (Apache-2.0); **PointForward** (scene-graph dynamics formulation).
+- Training data: **Waymo Open Dataset** (subject to Waymo terms, no redistribution) and
+  **Argoverse 2** (CC BY-NC-SA 4.0). MapGS code is the authors' own.
+
+*Reproducibility note:* gsplat + bf16 make runs reproducible at the seed/config level, not bit-exact.