image-registration-resources

image registration related books, papers, videos, and toolboxes

Many thanks to yzhao062 Anomaly Detection Learning Resources. I followed his style to make this depository.

Image registration is the process of transforming different sets of data into one coordinate system. Data may be multiple photographs, and from different sensors, times, depths, or viewpoints.

It is used in computer vision, medical imaging, military automatic target recognition, compiling and analyzing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from different measurements.

This repository collects:

Books & Academic Papers
On-line Courses and Videos
Datasets
Open-source and Commercial Libraries/Toolkits
Key Conferences & Journals

[toc]

More items will be added to the repository. Please feel free to suggest other key resources by opening an issue report, submitting a pull request, or dropping me an email @ ([email protected]). Enjoy reading!

1. Books & Tutorials

1.1. Books

Natural image

Multiple view geometry in computer vision by Richard Hartley and Andrew Zisserman, 2004: Mathematic and geometric basis for 2D-2D and 2D-3D registration. A must-read for people in the field of registration. E-book

Computer Vision: A Modern Approach by David A. Forsyth, Jean Ponce: for upper-division undergraduate- and graduate-level courses in computer vision found in departments of Computer Science, Computer Engineering and Electrical Engineering.

Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Engineering by Jean Gallier and Jocelyn Quaintance. The latest book from upenn about the algebra and optimization theory.

Three-Dimensional Computer vision-A Geometric Viewpoint Classical 3D computer vision textbook.

An invitation to 3D vision a self-contained introduction to the geometry of three-dimensional (3-D) vision.

Medical Image

Zhenhuan Zhou, et.al: A software guide for medical image segmentation and registration algorithm. 医学图像分割与配准(ITK实现分册) Part Ⅱ introduces the most basic network and architecture of medical registration algorithms (Chinese Version).

2-D and 3-D Image Registration for Medical, Remote Sensing, and Industrial Applications by A. Ardeshir Goshtasby

医学图像配准技术与应用 by 吕晓琪

Intensity-based 2D-3D Medical Image Registration by Russakoff, Daniel

Biomedical Image Registration by Fischer, Dawant, Lorenz

Medical Image Registration by Hajnal, Joseph V.

Deep Learning for Medical Image Analysis (part IV)

Point Cloud

14 lectures on visual SLAM By Xiang Gao and Tao Zhang and Yi Liu and Qinrui Yan. 视觉SLAM十四讲 视觉配准方向较易懂的入门教材。通俗讲述视觉匹配的物理模型，数学几何基础，优化过程等。新手必读。 [github] [Videos]

点云数据配准及曲面细分技术 by 薛耀红, 赵建平, 蒋振刚, 等书籍内容比较过时，仅适合零基础读者阅读。推荐自行查找相关博客学习。

Remote Sensing

Image Registration for Remote Sensing

2-D and 3-D Image Registration: For Medical, Remote Sensing, and Industrial Applications by A. A. Goshtasby, 2005.

航空遥感图像配准技术

基于特征的光学与SAR遥感图像配准

基于特征的航空遥感图像配准及部件检测技术

Introduction to Remote Sensing

Remote Sensing and Image Interpretation

Remote Sensing: Models and Methods for Image Processing

1.2. Tutorials

Natural image

Medical Image

Medical Image Registration
[MICCAI2019] learn2reg PDF

Big thanks to Yipeng Hu organizing the excellent tutorial.

Description:

Medical image registration has been a cornerstone in the research fields of medical image computing and computer assisted intervention, responsible for many clinical applications. Whilst machine learning methods have long been important in developing pairwise algorithms, recently proposed deep-learning-based frameworks directly infer displacement fields without iterative optimization for unseen image pairs, using neural networks trained from large population data. These novel approaches promise to tackle several most challenging aspects previously faced by classical pairwise methods, such as high computational cost, robustness for generalization and lack of inter-modality similarity measures.

Output from several international research groups working in this area include award-winning conference presentations, high-impact journal publications, well-received open-source implementations and industrial-partnered translational projects, generating significant interests to all levels of world-wide researchers. Accessing to the experience and expertise in this inherently multidisciplinary topic can be beneficial to many in our community, especially for the next generation of young scientists, engineers and clinicians who often have only been exposed to a subset of these methodologies and applications.

We organize a tutorial including both theoretical and practical sessions, inviting expert lectures and tutoring coding for real-world examples. Three hands-on sessions guiding participants to understand and implement published algorithms using clinical imaging data. This aims to provide an opportunity for the participants to bridge the gap between expertises in medical image registration and deep learning, as well as to start a forum to discuss know-hows, challenges and future opportunities in this area.

[kaggle:2016] Image registration, the R way, (almost) from scratch

There are some packages in R for image manipulation and after some test I select “imager” , based on the CImg C++, fast and providing several image processing tools.

[kaggle:2018] X-Ray Patient Scan Registration

SimpleITK, ITK, scipy, OpenCV, Tensorflow and PyTorch all offer tools for registering images, we explore a few here to see how well they work when applied to the fairly tricky problem of registering from the same person at different time and disease points.

[MICCAI2019] Autograd Image Registration Laboratory
[MIT] HST.582J Biomedical Signal and Image Processing PDF

Remote Sensing

Point Cloud

1.3. Blogs

图像配准指北

图像配准综述

基于深度学习的医学图像配准综述

基于深度学习和图像引导的医学图像配准

图像配准：从SIFT到深度学习

点云配准综述

图像配准会议介绍@ MICCAI2019 / CVPR2019 / ICCV2019 / NeurIPS2019

Image Registration: From SIFT to Deep Learning

点云配准 (！该部分需要重新编辑)

2. Courses/Seminars/Videos

Courses

16-822: Geometry-based Methods in Vision

[VALSE 2018] Talk: 2017以来的2D to 3D by 吴毅红

Workshops

WBIR - International Workshop on Biomedical Image Registration

WBIR2018, Leiden, Netherlands WBIR2016, Las Vegas NV WBIR2014, London, UK

Seminars

Videos

Definition and Introduction to Image Registration Pre Processing Overview
仿射变换与图像配准（科普性视频，比较简陋）

Remote Sensing

Registration of images of different modalities in Remote Sensing

3. Toolbox

Natural image

[C++] [Python] OpenCV: OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.

[C++] PCL: Point Cloud Library. The Point Cloud Library (PCL) is a standalone, large scale, open project for 2D/3D image and point cloud processing.

[C++] Ceres Solver: Ceres Solver is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems.

[C++] Open3D: Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization.

Medical Image

[c++] ITK: Segmentation & Registration Toolkit

An open-source, cross-platform system that provides developers with an extensive suite of software tools for image analysis. Developed through extreme programming methodologies. ITK employs leading-edge algorithms for registering and segmenting multidimensional data.

[c++] [Python] [Java] SimpleITK: a simplified layer built on top of ITK.

[c++] ANTs: Advanced Normalization Tools.

Image registration with variable transformations (elastic, diffeomorphic, diffeomorphisms, unbiased) and similarity metrics (landmarks, cross-correlation, mutual information, etc.). Image segmentation with priors & nonparametric, multivariate models.

[c++] Elastix: open source software, based on the well-known ITK .

The software consists of a collection of algorithms that are commonly used to solve (medical) image registration problems. [manual]

[C++] [Python] [Java] [R] [Ruby] [Lua] [Tcl] [C#] SimpleElastix: a medical image registration library that makes state-of-the-art image registration really easy to do in languages like Python, Java and R.

3D slicer : an open source software platform for medical image informatics, image processing, and three-dimensional visualization. Built over two decades through support from the National Institutes of Health and a worldwide developer community, Slicer brings free, powerful cross-platform processing tools to physicians, researchers, and the general public.

Github repository for deep learning medical image registration:

[Keras] VoxelMorph

[Keras] FAIM

[Tensorflow] Weakly-supervised CNN

[Tensorflow] RegNet3D

[Tensorflow] Recursive-Cascaded-Networks

[Pytorch] Probabilistic Dense Displacement Network

[Pytorch] Linear and Deformable Image Registration

[Pytorch] Inverse-Consistent Deep Networks

[Pytorch] Non-parametric image registration

[Pytorch] One Shot Deformable Medical Image Registration

[Pytorch] Image-and-Spatial Transformer Networks

Remote Sensing

[C++] OTB: Orfeo ToolBox (OTB) is an open-source project for state-of-the-art remote sensing. Built on the shoulders of the open-source geospatial community, it can process high resolution optical, multispectral and radar images at the terabyte scale. A wide variety of applications are available: from ortho-rectification or pansharpening, all the way to classification, SAR processing, and much more!

[C++] [Python] OpenCV: OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.Being a BSD-licensed product, OpenCV makes it easy for businesses to utilize and modify the code.

[C++] ITK: Insight Toolkit (ITK) an open-source, cross-platform system that provides developers with an extensive suite of software tools for image analysis. Developed through extreme programming methodologies, ITK employs leading-edge algorithms for registering and segmenting multidimensional data.

[Python] Spectral Python (SPy): Spectral Python (SPy) is a pure Python module for processing hyperspectral image data (imaging spectroscopy data). It has functions for reading, displaying, manipulating, and classifying hyperspectral imagery.

Post Processing Tools

[C++] enblend: Enblend blends away the seams in a panoramic image mosaic using a multi-resolution spline. Enfuse merges different exposures of the same scene to produce an image that looks much like a tone-mapped image.

[C++] maxflow: An implementation of the maxflow algorithm which can be used to detect the optimal seamline.

[C++] [Matlab] gco-v3.0: Multi-label optimization library by Olga Veksler and Andrew Delong.

Source Code

Parallax-tolerant image stitching

Point Cloud

MeshLab

简介：是一款开源、可移植和可扩展的三维几何处理系统。主要用于处理和编辑3D三角网格，它提供了一组用于编辑、清理、修复、检查、渲染、纹理化和转换网格的工具。提供了处理由3D数字化工具/设备生成的原始数据以及3D打印功能，功能全面而且丰富。MeshLab支持多数市面上常见的操作系统，包括Windows、Linux及Mac OS X，支持输入/输出的文件格式有：STL 、OBJ 、 VRML2.0、U3D、X3D、COLLADA MeshLab可用于各种学术和研究环境，如微生物学、文化遗产及表面重建等。

ICP开源库

4. Datasets & Competitions

4.1. Datasets

Natural image

Indoor LiDAR-RGBD Scan Dataset

ETH3D SLAM & Stereo Benchmarks

EuRoC MAV Dataset

ViViD : Vision for Visibility Dataset

Apolloscape: Scene Parsing

KITTI Visual Odometry dataset

NCLT Dataset

Oxford Robotcar Dataset

Medical Image

Dataset	Number	Modality	Region	Format
DIRLAB	10	4D CT	Lung	.img
LPBA40	40	3D MRI	T1 Brain	.img+.hdr .nii
IBSR18	18	3D MRI	T1 Brain	.img+.hdr
EMPIRE	30	4D CT	Lung	.mhd+.raw
LiTS	131	3D CT	Liver	.nii
CT-scans-before-and-after
Openi		X-ray
POPI	6	4D CT
NLST		CT	Lung
ADNI		3D MRI	Brain
OASIS		3D MRI	Brain
ABIDE		3D MRI	Brain
ADHD200
CUMC12	12	3D MRI	Brain	.img+.hdr
MGH10	10	3D MRI	Brain	.img+.hdr
FIRE	134	2D fundus	Retina	.jpg
MSD		CT	Liver
BFH	92	CT	Liver
SLIVER	20	CT	Liver
LSPIG	17	CT	Liver

Remote Sensing

ISPRS Benchmarks

HPatches: The HPatches dataset was used as the basis for the local descriptor evaluation challenge that was presented in the Local Features: State of the Art, Open Problems and Performance Evaluation workshop during ECCV 2016.

The Zurich Urban Micro Aerial Vehicle Dataset

Zurich Summer Dataset

Inria Aerial Image Labeling DataSet

Point Cloud(！该部分需要重新编辑)

The Stanford 3D Scanning Repository（斯坦福大学的3d扫描存储库）

http://graphics.stanford.edu/data/3Dscanrep/

这应该是做点云数据最初大家用最多的数据集，其中包含最开始做配准的Bunny、Happy Buddha、Dragon等模型。

Shapenet

ShapeNet是一个丰富标注的大规模点云数据集，其中包含了55中常见的物品类别和513000个三维模型。

The KITTI Vision Benchmark Suite

链接：http://www.cvlibs.net/datasets/kitti/

这个数据集来自德国卡尔斯鲁厄理工学院的一个项目，其中包含了利用KIT的无人车平台采集的大量城市环境的点云数据集（KITTI），这个数据集不仅有雷达、图像、GPS、INS的数据，而且有经过人工标记的分割跟踪结果，可以用来客观的评价大范围三维建模和精细分类的效果和性能。

Robotic 3D Scan Repository

链接：http://kos.informatik.uni-osnabrueck.de/3Dscans/

这个数据集比较适合做SLAM研究，包含了大量的Riegl和Velodyne雷达数据

佐治亚理工大型几何模型数据集

链接：https://www.cc.gatech.edu/projects/large_models/

PASCAL3D+

链接：http://cvgl.stanford.edu/projects/pascal3d.html

包含了12中刚体分类，每一类超过了3000个实例。并且包含了对应的imageNet中每一类的图像。

其他总结

链接：https://github.com/timzhang642/3D-Machine-Learning

4.2. Competitions

All Challenges

2019

CuRIOUS:2019 | Official solution

1 Register pre-operative MRI to iUS before tumor resection
2 Register iUS after tumor resection to iUS before tumor resection

ANHIR:2019 | Official solution

IEEE International Symposium on Biomedical Imaging (ISBI) 2019
High-resolution (up to 40x magnification) whole-slide images of tissues (lesions, lung-lobes, mammary-glands) were acquired - the original size of our images is up to 100k x 200k pixels. The acquired images are organized in sets of consecutive sections where each slice was stained with a different dye and any two images within a set can be meaningfully registered.