Tensor Holography

June 22, 2023 · View on GitHub

Towards Real-time Photorealistic 3D Holography with Deep Neural Networks (TensorHolo V1)
Nature 2021
Project Page | Paper | Data
Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer, Wojciech Matusik

End-to-end Learning of 3D Phase-only Holograms for Holographic Display (TensorHolo V2)
Light: Science and Applications 2022 [Impact factor: 20.26 (in 2022)]
Project Page | Paper | Data
Liang Shi, Beichen Li, Wojciech Matusik

This repository contains the code to reproduce the results presented in the above papers. Please read the License before using the software.

Ergonomic-Centric Holography: Optimizing Realism, Immersion, and Comfort for Holographic Display
Arxiv 2023
Project Page (in preparation) | Paper | Supplement | Code (in preparation)
Liang Shi*, DongHun Ryu*, Wojciech Matusik (* indicates equal contribution)

Getting Started

8/9/2022 Update: TensorHolo V2 code/dataset released.

This code runs with Python 3.8 and Tensorflow 1.15 (NVIDIA-maintained version to support training on the latest NVIDIA GPUS). You can set up a conda environment with the required dependencies using:

conda env create -f environment.yml
conda activate tensorholo
pip install nvidia-pyindex
pip install nvidia-tensorflow[horovod]

Alternatively, set up the following enviroment if you plan to export the model for TensorRT accelerated inference. The following instructions are tested on Ubuntu 20.04 with Python=3.8 CUDA=11.6 and TensorRT=8.4.

# Install CUDA 11.6 (Change to the correct link based on your own system)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.6.1/local_installers/cuda-repo-ubuntu2004-11-6-local_11.6.1-510.47.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-6-local_11.6.1-510.47.03-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-6-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

# Install TensorRT
# Download deb package from NVIDIA
# Replace xx in the os and tag with your package name
os="ubuntuxx04"
tag="cudax.x-trt8.x.x.x-yyyymmdd"
sudo dpkg -i nv-tensorrt-repo-${os}-${tag}_1-1_amd64.deb
sudo apt-key add /var/nv-tensorrt-repo-${os}-${tag}/7fa2af80.pub
sudo apt-get update
sudo apt-get install tensorrt

# Install other relevant packages
sudo apt-get install python3-libnvinfer-dev
sudo apt-get install onnx-graphsurgeon

# Add tensorrt bin path to use trtexec
export PATH=/usr/src/tensorrt/bin:$PATH

# Create a tensorrt environment
conda env create -f environment_trt.yml
conda activate trt
pip install nvidia-pyindex
pip install nvidia-tensorflow[horovod]
pip install nvidia-tensorrt

High-level structure

The code is organized as follows:

main.py defines/trains/validates/evaluates/exports the CNN for TensorHolo V1.
main_v2.py defines/trains/validates/evaluates/exports the CNN for TensorHolo V2.
optics.py contains optics-related helper functions and various implementations of double phase encoding.
util.py contains several utility functions for network training.
tfrecord.py contains code to generate and parse tfrecord.
tensorrt_onnx.py contains code to generate a TensorRT engine for accelerated inference.

Reproducing the experiments of TensorHolo V1

Reproducing the experiments of TensorHolo V2

Citation

If you find our work useful in your research, please cite:

@article{Shi2021:TensorHolography,
    title   = "Towards real-time photorealistic {3D} holography with deep neural
                networks",
    author  = "Shi, Liang and Li, Beichen and Kim, Changil and Kellnhofer, Petr
                and Matusik, Wojciech",
    journal = "Nature",
    volume  =  591,
    number  =  7849,
    pages   = "234--239",
    year    =  2021
}

@article{Shi2022:TensorHolography-v2,
    title    = "End-to-end learning of {3D} phase-only holograms for holographic
                display",
    author   = "Shi, Liang and Li, Beichen and Matusik, Wojciech",
    journal  = "Light Sci Appl",
    volume   =  11,
    number   =  1,
    pages    = "247",
    month    =  aug,
    year     =  2022,
    language = "en"
}

@misc{Shi2023:EC-H,
    title={Ergonomic-Centric Holography: Optimizing Realism,Immersion, and Comfort for Holographic Display}, 
    author={Liang Shi and Donghun Ryu and Wojciech Matusik},
    year={2023},
    eprint={2306.08138},
    archivePrefix={arXiv},
    primaryClass={cs.GR}
}

License

Our dataset and code, with the exception of the files in "data/example_image", are licensed under a custom license provided by the MIT Technology Licensing Office. By downloading the software, you agree to the terms of this License.

(New) Related Works