Awesome Deep Optics/End-to-End Optical Design

This is a list of awesome deep optics papers, inspired by awesome-computer-vision. Deep optics/end-to-end optical design is different from the traditional optical design because of:

• Differentiable optical model enables more optimization power
• End-to-end co-design of optics and image processing

Please bring these questions when you read the papers:

• What is the imaging model? A differentiable model enables end-to-end optical design with a network.
• What information is encoded? Additional information is encoded for computational imaging applications.
• How to fabricate the prototype? After designing a deep optics system, verify it with a prototype!

News

• 01/2025: This paper list is simplified following rules. A full list is available in full list.

Background materials

(I am trying to start a new repo to collect papers for optical/imaging process simulation. How do you think about it?)

The following are some materials I think will help you learn the imaging process.

• [1996 Book] Introduction to Fourier Optics McGraw-Hill Series in Electrical and Computer Engineering. link
• [2007 Book] Modern Optical Engineering. link
• [2011 Book] Computational fourier optics : a MATLAB tutorial. link
• [2012 Siggraph course] Computational displays: combining optical fabrication, computational processing, and perceptual tricks to build the displays of the future. link
• [2019 PhD thesis] Ray-based methods for simulating aberrations and cascaded diffraction in imaging systems. link
• [2020 Siggraph course] Deep optics: joint design of optics and image recovery algorithms for domain specific cameras. link
• [2022 Siggraph course] Differentiable cameras and displays. link

Must-read papers

1. Wave propagation model

In the wave propagation model, each optical element (DOE, lens, aperture, et al.) is represented as a phase mask. This idealized optics is easy to simulate; however, it is not accurate enough and cannot model optical aberrations.

Single DOE or metasurface (to flat cameras)

• 2018 End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. paper, supp, project, code
• 2020 Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. paper, supp, project, video
• 2020 Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array. paper, code
• 2021 Learning Privacy-Preserving Optics for Human Pose Estimation. paper, supp, project, slides, video, code
• 2021 Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics. paper, supp, project, video
• 2021 Neural nano-optics for high-quality thin lens imaging. paper, supp, project, project, code

DOE + non-optimizable thin lens (to hybrid optics)

• 2019 PhaseCam3D — Learning Phase Masks for Passive Single View Depth Estimation. paper, supp, project, code
• 2020 Learning Rank-1 Diffractive Optics for Single-shot High Dynamic Range Imaging. paper, supp, project
• 2021 Depth from Defocus with Learned Optics for Imaging and Occlusion-aware Depth Estimation. paper, supp, project, code
• 2022 End-to-end snapshot compressed super-resolution imaging with deep optics. paper, supp
• 2022 Seeing Through Obstructions with Diffractive Cloaking. paper, project, code
• 2024 Split-aperture 2-in-1 computational cameras. paper, supp, project, code

DOE/Metasurface array/stack (to flat cameras)

• 2016 Encoded diffractive optics for full-spectrum computational imaging. paper, supp
• 2021 Shift-variant color-coded diffractive spectral imaging system. paper, video, code
• 2024 Learned Multi-aperture Color-coded Optics for Snapshot Hyperspectral Imaging. paper, supp, project, code
• 2024 Spatially varying nanophotonic neural networks. paper, project, code

2. Ray tracing model

Ray tracing is the most common technique in optical design (e.g., ZEMAX and CodeV). In the field of deep optics, people usually compute the point spread function (PSF) and convolve it with the input, or perform ray-tracing-based rendering to simulate sensor images. Most ray tracing works are incoherent, but there are also some works of coherent ray tracing.

Lens (to optical design)

• 2021 dO: A differentiable engine for Deep Lens design of computational imaging systems. paper, project, code
• 2022 Computational Optics for Mobile Terminals in Mass Production. paper
• 2022 The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection. paper, code
• 2023 Curriculum Learning for ab initio Deep Learned Refractive Optics. paper, video, code
• 2023 Large depth-of-field ultra-compact microscope by progressive optimization and deep learning. paper, code

3. Network Representation

Model a group of optical systems by a network. The network takes optical parameters (e.g., curvatures) as input and outputs the PSF. We can back-propagate gradients through the network to get the gradients to update input parameters.

Lens (to computational photography)

• 2021 Deep learning-enabled framework for automatic lens design starting point generation. paper, project
• 2021 Differentiable Compound Optics and Processing Pipeline Optimization for End-To-end Camera Design. paper, project
• 2023 Aberration-Aware Depth-from-Focus. paper, supp, project, code

Lithography (to computational photolithography)

• 2023 Close the Design-to-Manufacturing Gap in Computational Optics with a 'Real2Sim' Learned Two-Photon Neural Lithography Simulator. paper, supp, project, code

4. Hybrid model

Hybrid ray tracing and wave propagation model for refractive-diffractive optical systems.

Lens

• 2023 Metalens enhanced ray optics: an end-to-end wave-ray co-optimization framework. paper
• 2024 End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model. paper, code

Contribution

Please feel free to open pull requests or email ([email protected]) to contibute to this repo.

Licenses

License

To the extent possible under law, Xinge Yang has waived all copyright and related or neighboring rights to this work.