UFORecon (CVPR 2024)

This repository contains a official code of UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and Unfavorable Sets. by Youngju Na, Woo Jae Kim, Kyu Beom Han, Suhyeon Ha, and Sung-Eui Yoon.

Check out the project page and paper.

Project Page | arXiv

Updates

• release evaluation code
• release pretrained weights

Overview

VolRecon	Ours

Requirements

All experiments are tested on Ubuntu (20.04 or 22.04) with a single RTX 3090 or 4090.

• python 3.10
• CUDA 11.x

conda create --name UFORecon python=3.10
conda activate UFORecon
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

Run one of .vscode/launch.json to see whether your environment is built properly (Change the root_dir to your DTU dataset directory).

Reproducing Sparse View Reconstruction on DTU

• Download pre-processed DTU dataset following VolRecon. The dataset is organized as follows:

root_directory
├──cameras
    ├── 00000000_cam.txt
    ├── 00000001_cam.txt
    └── ...  
├──pair.txt
├──scan24
├──scan37
      ├── image               
      │   ├── 000000.png       
      │   ├── 000001.png       
      │   └── ...                
      └── mask                   
          ├── 000.png   
          ├── 001.png
          └── ...                

Camera file cam.txt stores the camera parameters, which include extrinsic, intrinsic, minimum depth, and depth range interval:

extrinsic
E00 E01 E02 E03
E10 E11 E12 E13
E20 E21 E22 E23
E30 E31 E32 E33

intrinsic
K00 K01 K02
K10 K11 K12
K20 K21 K22

DEPTH_MIN DEPTH_INTERVAL

pair.txt stores the view selection result. For each reference image, 10 best source views are stored in the file:

TOTAL_IMAGE_NUM
IMAGE_ID0                       # index of reference image 0 
10 ID0 SCORE0 ID1 SCORE1 ...    # 10 best source images for reference image 0 
IMAGE_ID1                       # index of reference image 1
10 ID0 SCORE0 ID1 SCORE1 ...    # 10 best source images for reference image 1 
...

Evaluation (unfavorable)

Set DATASET as the root directory of the dataset, set OUT_DIR as the directory to store the rendered depth maps. By default, 3 images (--test_n_view 3) of unfavorable camera configuration (cam id: 1, 16, 36) in the image set 0 (--set 0) are used for testing. For reproduction, use cam id: (23, 24, 33) for favorable set, and (1, 16, 36) for unfavorable set. All other combinations with different number of images are availble.

bash script/eval_dtu_unfavorable.sh
bash script/tsdf_fusion.sh
bash script/clean_mesh.sh
bash script/eval_dtu.sh

• For quantitative evaluation, download SampleSet and Points from DTU's website. Unzip them and place Points folder in SampleSet/MVS Data/. The structure looks like:

SampleSet
├──MVS Data
      └──Points

Training UFORecon on DTU

root_directory
├──Cameras
├──Rectified
└──Depths_raw

• Train the model by running bash train_dtu.sh on GPU.

bash script/train_dtu.sh

Set --view_selection_type to random for applying random set training (default is best for training following the common protocol.)

Citation

If you find this project useful for your research, please cite:

@inproceedings{na2024uforecon,
  title={UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and Unfavorable Sets},
  author={Na, Youngju and Kim, Woo Jae and Han, Kyu Beom and Ha, Suhyeon and Yoon, Sung-Eui},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5094--5104},
  year={2024}
}

Acknowledgement

This project is based on VolRecon, TransMVSNet, and MatchNeRF. Thanks for their amazing work.

License

This project is released under the MIT License.