diff --git a/.gitignore b/.gitignore index 27cac26..40e96d1 100644 --- a/.gitignore +++ b/.gitignore @@ -8,10 +8,10 @@ renders videos edited **/__pycache__ -submodules/simple-knn -submodules/tiny-cuda-nn-fp32 -submodules/diff-surfel-rasterization -submodules/diff-surfel-rasterization_debug +submodules/ +*.egg-info/ +uv.lock +.venv/ internal/dataparsers/colmap_cluster_dataparser.py internal/utils/data_sampler.py tools/merge_new_img.py diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9c7b9b6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,150 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +CityGaussian is a large-scale scene reconstruction system using 3D/2D Gaussian Splatting. Built on [Gaussian Lightning](https://github.com/yzslab/gaussian-splatting-lightning) with PyTorch Lightning. + +## Build & Installation + +**Important**: This project requires CUDA-compiled submodules. For CUDA 12.x (GTX 40-series and newer), the submodules need `` includes in header files. + +**Always use `uv` for package management. Never use `pip` or `pip install` directly - always use `uv add` or `uv pip install`.** + +```bash +# Create environment +uv venv --python 3.9 +source .venv/bin/activate + +# Install PyTorch matching your CUDA version +# For CUDA 11.8: +uv add torch==2.0.1 torchvision torchaudio --index https://download.pytorch.org/whl/cu118 +# For CUDA 12.4: +uv add torch torchvision torchaudio --index https://download.pytorch.org/whl/cu124 + +# Install base requirements +uv add -r requirements.txt + +# Install CityGaussian specific packages +uv add -r requirements/CityGS.txt + +# Build CUDA submodules (requires --no-build-isolation for torch access during build) +uv pip install -e ./submodules/diff-gaussian-rasterization --no-build-isolation +uv pip install -e ./submodules/diff-surfel-rasterization-city --no-build-isolation +uv pip install -e ./submodules/diff-trim-gaussian-rasterization --no-build-isolation +uv pip install -e ./submodules/simple-knn --no-build-isolation + +# Fix setuptools version (newer versions removed pkg_resources needed by lightning) +uv pip install "setuptools<70" +``` + +## Common Commands + +### Training Pipeline +```bash +# 1. Train coarse model +python main.py fit --config configs/citygsv2_lfls_coarse_sh2.yaml -n experiment_name + +# 2. Partition scene and assign data +python utils/partition_citygs.py --config_path configs/citygsv2_lfls_sh2_trim.yaml --force + +# 3. Parallel fine-tuning +python utils/train_citygs_partitions.py -n experiment_name + +# 4. Merge checkpoints +python utils/merge_citygs_ckpts.py outputs/experiment_name +``` + +### Evaluation & Rendering +```bash +# Test/evaluate model +python main.py test --config outputs/experiment_name/config.yaml --save_val --test_speed + +# Mesh extraction +python utils/gs2d_mesh_extraction.py outputs/experiment_name --voxel_size 0.05 + +# Render video +python render.py model.pth --camera-path-filename camera.json --output-path output.mp4 + +# Web viewer +gs-viewer model.pth --port 8080 +``` + +### Testing +```bash +python -m unittest discover tests -v +python -m unittest tests.vanilla_gaussian_model_test -v +``` + +### Quick Installation Test +```bash +# Download test dataset (Tanks & Temples truck scene, ~180MB) +wget "https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/input/tandt_db.zip" -O data/tandt.zip +unzip data/tandt.zip -d data/ +ln -s images data/tandt/truck/images_2 + +# Train for 3000 steps (~2 min on RTX 4080) +python main.py fit --data.path data/tandt/truck \ + --data.parser.class_path Colmap \ + --data.parser.init_args.down_sample_factor 2 \ + --data.parser.init_args.down_sample_rounding_mode ceil \ + --trainer.max_steps 3000 -n test_truck + +# View result +gs-viewer outputs/test_truck/checkpoints/*.ckpt --port 8080 +``` + +## Architecture + +### Entry Points +- `main.py` → `internal/entrypoints/gspl.py` - Main CLI (fit/val/test/predict) +- `render.py` - Video rendering with camera paths +- `viewer.py` - Web-based model viewer + +### Core Modules (internal/) +- `gaussian_splatting.py` - Main LightningModule for training +- `models/` - Gaussian model variants (vanilla, 2D, mip, deformable, appearance) +- `renderers/` - Pluggable rendering backends (vanilla, gsplat, partition_lod) +- `dataset.py` - DataModule with async caching for large datasets +- `dataparsers/` - Dataset loaders (Colmap, Blender, NSVF, Nerfies) +- `density_controllers/` - Gaussian densification strategies +- `callbacks.py` - Training callbacks (SaveGaussian, etc.) + +### Submodules +- `diff-gaussian-rasterization` → `diff_gaussian_rasterization` - Standard 3DGS rasterizer +- `diff-surfel-rasterization-city` → `diff_trim_surfel_rasterization` - Modified surfel rasterizer for street views +- `diff-trim-gaussian-rasterization` → `diff_trim_gaussian_rasterization` - Trimmed Gaussian variant +- `simple-knn` → `simple_knn` - KNN implementation + +## Configuration + +YAML configs in `/configs/` define model, data, renderer, and optimizer settings. Override via CLI: +```bash +python main.py fit --config configs/base.yaml --trainer.max_steps 30000 --data.parser.image_scale 2 +``` + +Output structure: `outputs//config.yaml`, `checkpoints/`, `cameras.json` + +## Key Patterns + +- **Lightning-based**: Training uses LightningModule/DataModule pattern +- **Modular renderers**: Swap renderers via config (vanilla, gsplat, feature_3dgs) +- **Scene partitioning**: Large scenes split into blocks for parallel training +- **Caching**: Async image caching to manage memory on large datasets + +## Allowed Commands + +The following commands are pre-approved and can be run without confirmation: + +- `python -c "import ..."` - Python import tests +- `python main.py --help` - CLI help +- `python main.py fit/validate/test/predict ...` - Training and evaluation +- `python -m unittest ...` - Running tests +- `uv add ...` - Installing packages +- `uv pip install ... --no-build-isolation` - Building CUDA extensions +- `gs-viewer ...` - Web viewer +- `wget ...` - Download files +- `curl ...` - Download files +- `gdown ...` - Download from Google Drive +- `unzip ...` - Extract archives diff --git a/doc/installation.md b/doc/installation.md index dbcaedb..b563f6a 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -9,6 +9,19 @@ cd CityGaussian ### B. Create virtual environment +#### Option 1: Using uv (Recommended) + +```bash +# install uv if not already installed +curl -LsSf https://astral.sh/uv/install.sh | sh + +# create virtual environment +uv venv --python 3.9 +source .venv/bin/activate +``` + +#### Option 2: Using conda + ```bash # create virtual environment conda create -yn gspl python=3.9 pip @@ -16,24 +29,79 @@ conda activate gspl ``` ### C. Install PyTorch -* Tested on `PyTorch==2.0.1` -* You must install the one match to the version of your nvcc (nvcc --version) -* For CUDA 11.8 +* Tested on `PyTorch==2.0.1` and `PyTorch==2.5.1` +* You must install the one matching your CUDA version (check with `nvcc --version`) + +#### For CUDA 11.8 - ```bash - pip install -r requirements/pyt201_cu118.txt - ``` +```bash +# Using uv +uv pip install torch==2.0.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 + +# Using conda/pip +pip install -r requirements/pyt201_cu118.txt +``` + +#### For CUDA 12.4+ (RTX 40-series and newer) + +```bash +# Using uv +uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 + +# Using conda/pip +pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 +``` ### D. Install requirements ```bash +# Using uv +uv pip install -r requirements.txt + +# Using conda/pip pip install -r requirements.txt ``` + For pose and 3DGS joint optimization, please directly jump to this [document](../doc/vggt_x.md). ### E. Install additional package for CityGaussian ```bash +# Using uv +uv pip install -r requirements/CityGS.txt + +# Using conda/pip pip install -r requirements/CityGS.txt ``` -Note that here we use modified version of Trim2DGS rasterizer, so as to resolve [impulse noise problem](https://github.com/hbb1/2d-gaussian-splatting/issues/174) under street views. This version also avoids interference from out-of-view surfels. \ No newline at end of file + +Note that here we use modified version of Trim2DGS rasterizer, so as to resolve [impulse noise problem](https://github.com/hbb1/2d-gaussian-splatting/issues/174) under street views. This version also avoids interference from out-of-view surfels. + +### F. Build CUDA submodules + +For CUDA 12.x, you need to build submodules with `--no-build-isolation` to allow torch access during compilation: + +```bash +# Using uv (recommended for CUDA 12.x) +uv pip install -e ./submodules/diff-gaussian-rasterization --no-build-isolation +uv pip install -e ./submodules/diff-surfel-rasterization-city --no-build-isolation +uv pip install -e ./submodules/diff-trim-gaussian-rasterization --no-build-isolation +uv pip install -e ./submodules/simple-knn --no-build-isolation + +# Using conda/pip +pip install -e ./submodules/diff-gaussian-rasterization +pip install -e ./submodules/diff-surfel-rasterization-city +pip install -e ./submodules/diff-trim-gaussian-rasterization +pip install -e ./submodules/simple-knn +``` + +### G. Fix setuptools (if using PyTorch Lightning) + +Newer setuptools versions (v70+) removed `pkg_resources` which PyTorch Lightning requires: + +```bash +# Using uv +uv pip install "setuptools>=61.0,<70" + +# Using conda/pip +pip install "setuptools>=61.0,<70" +``` diff --git a/pyproject.toml b/pyproject.toml index 4999dad..9f71605 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -9,6 +9,9 @@ include = ["internal*", "utils*"] name = "gaussian-splatting-lightning" dynamic = ["version"] requires-python = ">=3.8" +dependencies = [ + "setuptools>=61.0,<70", +] [project.scripts] gs-fit = "internal.entrypoints.gspl:cli_fit"