Skip to content

googleearthengine/satellite-image-deep-learning-1

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This repository lists resources on the topic of deep learning applied to satellite and aerial imagery. To a lesser extent classical machine learning techniques are listed, as are topics such as cloud computing and model deployment. How to use this repository: if you know exactly what you are looking for (e.g. a dataset name) you can Control+F to search for it in the page. Otherwise refer to the table of contents below, and search within the relevant section. Material that is suitable for getting started with a topic is tagged with BEGINNER, which can also be searched.

Table of contents

Techniques

This section explores the different deep and machine learning (ML) techniques applied to common problems in satellite imagery analysis. Good background reading is Deep learning in remote sensing applications: A meta-analysis and review

Classification

The classic cats vs dogs image classification task, which in the remote sensing domain is used to assign a label to an image, e.g. this is an image of a forest. The more complex case is applying multiple labels to an image. This approach of image level classification is not to be confused with pixel-level classification which is called semantic segmentation. Read my blog post A brief introduction to satellite image classification with neural networks

Segmentation

Segmentation will assign a class label to each pixel in an image. Single class models are often trained for road or building segmentation, with multi class for land use/crop type classification. Note that many articles which refer to 'hyperspectral land classification' are actually describing semantic segmentation. Note that cloud detection can be addressed with semantic segmentation and has its own section Cloud detection & removal. Read my blog post A brief introduction to satellite image segmentation with neural networks

Segmentation - Land use & land cover

Segmentation - Vegetation, crops & crop boundaries

Segmentation - Water, coastlines & floods

Segmentation - Fire, smoke & burn areas

Segmentation - Landslides

Segmentation - Glaciers

  • HED-UNet -> a model for simultaneous semantic segmentation and edge detection, examples provided are glacier fronts and building footprints using the Inria Aerial Image Labeling dataset
  • glacier_mapping -> Mapping glaciers in the Hindu Kush Himalaya, Landsat 7 images, Shapefile labels of the glaciers, Unet with dropout
  • glacier-detect-ML -> a simple logistic regression model to identify a glacier in Landsat satellite imagery
  • GlacierSemanticSegmentation -> uses unet
  • Antarctic-fracture-detection -> uses UNet with the MODIS Mosaic of Antarctica to detect surface fractures (paper)

Segmentation - Other environmental

Segmentation - Roads

Extracting roads is challenging due to the occlusions caused by other objects and the complex traffic environment

Segmentation - Buildings & rooftops

Segmentation - Solar panels

Segmentation - Other manmade

Instance segmentation

In instance segmentation, each individual 'instance' of a segmented area is given a unique lable. For detection of very small objects this may a good approach, but it can struggle seperating individual objects that are closely spaced.

Panoptic segmentation

Object detection

Onject detection is the task of placing a box around the bounds of an object (i.e. creating a bounding box). Typical use cases are detecting vehicles, aircraft & ships. A good introduction to the challenge of performing object detection on aerial imagery is given in this paper. In summary, images are large and objects may comprise only a few pixels, easily confused with random features in background. For the same reason, object detection datasets are inherently imbalanced, since the area of background typically dominates over the area of the objects to be detected. In general object detection performs well on large objects, and gets increasingly difficult as the objects get smaller & more densely packed. Model accuracy falls off rapidly as image resolution degrades, so it is common for object detection to use very high resolution imagery, e.g. 30cm RGB. A particular characteristic of aerial images is that objects can be oriented in any direction, so using rotated bounding boxes which align with the object can be crucial for extracting measurements of the length and width of an object.

Object detection with rotated bounding boxes

Orinted bounding boxes (OBB) are polygons representing rotated rectangles

  • For datasets checkout DOTA & HRSC2016
  • mmrotate -> Rotated Object Detection Benchmark, with pretrained models and function for inferencing on very large images
  • OBBDetection -> an oriented object detection library, which is based on MMdetection
  • rotate-yolov3 -> Rotation object detection implemented with yolov3. Also see yolov3-polygon
  • DRBox -> for detection tasks where the objects are orientated arbitrarily, e.g. vehicles, ships and airplanes
  • s2anet -> Official code of the paper 'Align Deep Features for Oriented Object Detection'
  • CFC-Net -> Official implementation of "CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Detection in Remote Sensing Images"
  • ReDet -> Official code of the paper "ReDet: A Rotation-equivariant Detector for Aerial Object Detection"
  • BBAVectors-Oriented-Object-Detection -> Oriented Object Detection in Aerial Images with Box Boundary-Aware Vectors
  • CSL_RetinaNet_Tensorflow -> Code for ECCV 2020 paper: Arbitrary-Oriented Object Detection with Circular Smooth Label
  • r3det-on-mmdetection -> R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object
  • R-DFPN_FPN_Tensorflow -> Rotation Dense Feature Pyramid Networks (Tensorflow)
  • R2CNN_Faster-RCNN_Tensorflow -> Rotational region detection based on Faster-RCNN
  • Rotated-RetinaNet -> implemented in pytorch, it supports the following datasets: DOTA, HRSC2016, ICDAR2013, ICDAR2015, UCAS-AOD, NWPU VHR-10, VOC2007
  • OBBDet_Swin -> The sixth place winning solution in 2021 Gaofen Challenge
  • CG-Net -> Learning Calibrated-Guidance for Object Detection in Aerial Images. With paper
  • OrientedRepPoints_DOTA -> Oriented RepPoints + Swin Transformer/ReResNet
  • yolov5_obb -> yolov5 + Oriented Object Detection
  • How to Train YOLOv5 OBB -> YOLOv5 OBB tutorial and YOLOv5 OBB noteboook
  • OHDet_Tensorflow -> can be applied to rotation detection and object heading detection
  • Seodore -> framework maintaining recent updates of mmdetection
  • Rotation-RetinaNet-PyTorch -> oriented detector Rotation-RetinaNet implementation on Optical and SAR ship dataset
  • AIDet -> an open source object detection in aerial image toolbox based on MMDetection
  • rotation-yolov5 -> rotation detection based on yolov5
  • ShipDetection -> Ship Detection in HR Optical Remote Sensing Images via Rotated Bounding Box, based on Faster R-CNN and ORN, uses caffe
  • SLRDet -> project based on mmdetection to reimplement RRPN and use the model Faster R-CNN OBB
  • AxisLearning -> code for 2020 paper: Axis Learning for Orientated Objects Detection in Aerial Images
  • Detection_and_Recognition_in_Remote_Sensing_Image -> This work uses PaNet to realize Detection and Recognition in Remote Sensing Image by MXNet
  • DrBox-v2-tensorflow -> tensorflow implementation of DrBox-v2 which is an improved detector with rotatable boxes for target detection in remote sensing images
  • Rotation-EfficientDet-D0 -> A PyTorch Implementation Rotation Detector based EfficientDet Detector, applied to custom rotation vehicle datasets
  • DODet -> Dual alignment for oriented object detection, uses DOTA dataset. With paper
  • GF-CSL -> code for 2022 paper: Gaussian Focal Loss: Learning Distribution Polarized Angle Prediction for Rotated Object Detection in Aerial Images
  • simplified_rbox_cnn -> code for 2018 paper: RBox-CNN: rotated bounding box based CNN for ship detection in remote sensing image. Uses Tensorflow object detection API
  • Polar-Encodings -> code for 2021 [paper](Learning Polar Encodings for Arbitrary-Oriented Ship Detection in SAR Images)
  • R-CenterNet -> detector for rotated-object based on CenterNet
  • piou -> Orientated Object Detection; IoU Loss, applied to DOTA dataset
  • DAFNe -> code for 2021 paper: DAFNe: A One-Stage Anchor-Free Approach for Oriented Object Detection
  • AProNet -> code for 2021 paper: AProNet: Detecting objects with precise orientation from aerial images. Applied to datasets DOTA and HRSC2016
  • UCAS-AOD-benchmark -> A benchmark of UCAS-AOD dataset
  • RotateObjectDetection -> based on Ultralytics/yolov5, with adjustments to enable rotate prediction boxes. Also see PolygonObjectDetection
  • AD-Toolbox -> Aerial Detection Toolbox based on MMDetection and MMRotate, with support for more datasets
  • GGHL -> code for 2022 paper: A General Gaussian Heatmap Label Assignment for Arbitrary-Oriented Object Detection
  • NPMMR-Det -> code for 2021 paper: A Novel Nonlocal-Aware Pyramid and Multiscale Multitask Refinement Detector for Object Detection in Remote Sensing Images
  • AOPG -> code for 2022 paper: Anchor-Free Oriented Proposal Generator for Object Detection
  • SE2-Det -> code for 2022 paper: Semantic-Edge-Supervised Single-Stage Detector for Oriented Object Detection in Remote Sensing Imagery
  • OrientedRepPoints -> code for 2021 paper: Oriented RepPoints for Aerial Object Detection
  • TS-Conv -> code for 2022 paper: Task-wise Sampling Convolutions for Arbitrary-Oriented Object Detection in Aerial Images
  • FCOSR -> A Simple Anchor-free Rotated Detector for Aerial Object Detection. This implement is modified from mmdetection. See also TensorRT_Inference

Object detection enhanced by super resolution

Salient object detection

Detecting the most noticeable or important object in a scene

  • ACCoNet -> code for 2022 paper: Adjacent Context Coordination Network for Salient Object Detection in Optical Remote Sensing Images
  • MCCNet -> Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images
  • CorrNet -> Lightweight Salient Object Detection in Optical Remote Sensing Images via Feature Correlation. With paper
  • Reading list for deep learning based Salient Object Detection in Optical Remote Sensing Images
  • ORSSD-dataset -> salient object detection dataset
  • EORSSD-dataset -> Extended Optical Remote Sensing Saliency Detection (EORSSD) Dataset
  • DAFNet_TIP20 -> code for 2020 paper: Dense Attention Fluid Network for Salient Object Detection in Optical Remote Sensing Images
  • EMFINet -> code for 2021 paper: Edge-Aware Multiscale Feature Integration Network for Salient Object Detection in Optical Remote Sensing Images
  • ERPNet -> code for 2022 paper: Edge-guided Recurrent Positioning Network for Salient Object Detection in Optical Remote Sensing Images
  • FSMINet -> code for 2022 paper: Fully Squeezed Multi-Scale Inference Network for Fast and Accurate Saliency Detection in Optical Remote Sensing Images
  • AGNet -> code for 2022 paper: AGNet: Attention Guided Network for Salient Object Detection in Optical Remote Sensing Images
  • MSCNet -> code for 2022 paper: A lightweight multi-scale context network for salient object detection in optical remote sensing images
  • GPnet -> code for 2022 paper: Global Perception Network for Salient Object Detection in Remote Sensing Images

Object detection - Buildings, rooftops & solar panels

Object detection - Ships & boats

Object detection - Cars, vehicles & trains

Object detection - Planes & aircraft

Object detection - Infrastructure & utilities

Object detection - Oil storage tank detection

Oil is stored in tanks at many points between extraction and sale, and the volume of oil in storage is an important economic indicator.

Object detection - Animals

A variety of techniques can be used to count animals, including object detection and instance segmentation. For convenience they are all listed here:

Object tracking in videos

Object counting

When the object count, but not its shape is required, U-net can be used to treat this as an image-to-image translation problem.

  • centroid-unet -> Centroid-UNet is deep neural network model to detect centroids from satellite images, with paper BEGINNER
  • cownter_strike -> counting cows, located with point-annotations, two models: CSRNet (a density-based method) & LCFCN (a detection-based method)
  • DO-U-Net -> an effective approach for when the size of an object needs to be known, as well as the number of objects in the image, initially created to segment and count Internally Displaced People (IDP) camps in Afghanistan
  • Cassava Crop Counting
  • Counting from Sky -> A Large-scale Dataset for Remote Sensing Object Counting and A Benchmark Method
  • PSGCNet -> code for 2022 paper: PSGCNet: A Pyramidal Scale and Global Context Guided Network for Dense Object Counting in Remote Sensing Images
  • psgcnet -> code for 2022 paper: PSGCNet: A Pyramidal Scale and Global Context Guided Network for Dense Object Counting in Remote-Sensing Images

Regression

Cloud detection & removal

There are many algorithms that use band math to detect clouds, but the deep learning approach is to use semantic segmentation

Change detection

Generally speaking, change detection methods are applied to a pair of images to generate a mask of change, e.g. of buildings damaged in a disaster. Note, clouds & shadows change often too..!

  • awesome-remote-sensing-change-detection lists many datasets and publications
  • Change-Detection-Review -> A review of change detection methods, including code and open data sets for deep learning
  • Change Detection using Siamese Networks -> Medium article BEGINNER
  • STANet -> official implementation of the spatial-temporal attention neural network (STANet) for remote sensing image change detection BEGINNER
  • UNet-based-Unsupervised-Change-Detection -> A convolutional neural network (CNN) and semantic segmentation is implemented to detect the changes between the images, as well as classify the changes into the correct semantic class, with arxiv paper BEGINNER
  • BIT_CD -> Official Pytorch Implementation of Remote Sensing Image Change Detection with Transformers
  • Unstructured-change-detection-using-CNN
  • Siamese neural network to detect changes in aerial images -> uses Keras and VGG16 architecture
  • Change Detection in 3D: Generating Digital Elevation Models from Dove Imagery
  • QGIS plugin for applying change detection algorithms on high resolution satellite imagery
  • LamboiseNet -> Master thesis about change detection in satellite imagery using Deep Learning
  • Fully Convolutional Siamese Networks for Change Detection -> with paper
  • Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks -> with paper, used the Onera Satellite Change Detection (OSCD) dataset
  • IAug_CDNet -> Official Pytorch Implementation of Adversarial Instance Augmentation for Building Change Detection in Remote Sensing Images
  • dpm-rnn-public -> Code implementing a damage mapping method combining satellite data with deep learning
  • SenseEarth2020-ChangeDetection -> 1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime; predictions of five HRNet-based segmentation models are ensembled, serving as pseudo labels of unchanged areas
  • KPCAMNet -> Python implementation of the paper Unsupervised Change Detection in Multi-temporal VHR Images Based on Deep Kernel PCA Convolutional Mapping Network
  • CDLab -> benchmarking deep learning-based change detection methods.
  • Siam-NestedUNet -> The pytorch implementation for "SNUNet-CD: A Densely Connected Siamese Network for Change Detection of VHR Images"
  • SUNet-change_detection -> Implementation of paper SUNet: Change Detection for Heterogeneous Remote Sensing Images from Satellite and UAV Using a Dual-Channel Fully Convolution Network
  • Self-supervised Change Detection in Multi-view Remote Sensing Images
  • MFPNet -> Remote Sensing Change Detection Based on Multidirectional Adaptive Feature Fusion and Perceptual Similarity
  • GitHub for the DIUx xView Detection Challenge -> The xView2 Challenge focuses on automating the process of assessing building damage after a natural disaster
  • DASNet -> Dual attentive fully convolutional siamese networks for change detection of high-resolution satellite images
  • Self-Attention for Raw Optical Satellite Time Series Classification
  • planet-movement -> Find and process Planet image pairs to highlight object movement
  • temporal-cluster-matching -> detecting change in structure footprints from time series of remotely sensed imagery
  • autoRIFT -> fast and intelligent algorithm for finding the pixel displacement between two images
  • DSAMNet -> Code for “A Deeply Supervised Attention Metric-Based Network and an Open Aerial Image Dataset for Remote Sensing Change Detection”. The main types of changes in the dataset include: (a) newly built urban buildings; (b) suburban dilation; (c) groundwork before construction; (d) change of vegetation; (e) road expansion; (f) sea construction.
  • SRCDNet -> The pytorch implementation for "Super-resolution-based Change Detection Network with Stacked Attention Module for Images with Different Resolutions ". SRCDNet is designed to learn and predict change maps from bi-temporal images with different resolutions
  • Land-Cover-Analysis -> Land Cover Change Detection using Satellite Image Segmentation
  • A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images
  • Satellite-Image-Alignment-Differencing-and-Segmentation -> thesis on change detection
  • Change Detection in Multi-temporal Satellite Images -> uses Principal Component Analysis (PCA) and K-means clustering
  • Unsupervised Change Detection Algorithm using PCA and K-Means Clustering -> in Matlab but has paper
  • ChangeFormer -> A Transformer-Based Siamese Network for Change Detection. Uses transformer architecture to address the limitations of CNN in handling multi-scale long-range details. Demonstrates that ChangeFormer captures much finer details compared to the other SOTA methods, achieving better performance on benchmark datasets
  • Heterogeneous_CD -> Heterogeneous Change Detection in Remote Sensing Images. Accompanies Code-Aligned Autoencoders for Unsupervised Change Detection in Multimodal Remote Sensing Images
  • ChangeDetectionProject -> Trying out Active Learning in with deep CNNs for Change detection on remote sensing data
  • DSFANet -> Unsupervised Deep Slow Feature Analysis for Change Detection in Multi-Temporal Remote Sensing Images
  • siamese-change-detection -> Targeted synthesis of multi-temporal remote sensing images for change detection using siamese neural networks
  • Bi-SRNet -> code for 2022 paper: Bi-Temporal Semantic Reasoning for the Semantic Change Detection in HR Remote Sensing Images
  • SiROC -> Implementation of the paper Spatial Context Awareness for Unsupervised Change Detection in Optical Satellite Images. Applied to Sentinel-2 and high-resolution Planetscope imagery on four datasets
  • DSMSCN -> Tensorflow implementation for Change Detection in Multi-temporal VHR Images Based on Deep Siamese Multi-scale Convolutional Neural Networks
  • RaVAEn -> a lightweight, unsupervised approach for change detection in satellite data based on Variational Auto-Encoders (VAEs) with the specific purpose of on-board deployment. It flags changed areas to prioritise for downlink, shortening the response time
  • SemiCD -> Code for paper: Revisiting Consistency Regularization for Semi-supervised Change Detection in Remote Sensing Images. Achieves the performance of supervised CD even with access to as little as 10% of the annotated training data
  • FCCDN_pytorch -> code for paper: FCCDN: Feature Constraint Network for VHR Image Change Detection. Uses the LEVIR-CD building change detection dataset
  • INLPG_Python -> code for paper: Structure Consistency based Graph for Unsupervised Change Detection with Homogeneous and Heterogeneous Remote Sensing Images
  • NSPG_Python -> code for paper: Nonlocal patch similarity based heterogeneous remote sensing change detection
  • LGPNet-BCD -> code for 2021 paper: Building Change Detection for VHR Remote Sensing Images via Local-Global Pyramid Network and Cross-Task Transfer Learning Strategy
  • DS_UNet -> code for 2021 paper: Sentinel-1 and Sentinel-2 Data Fusion for Urban Change Detection using a Dual Stream U-Net, uses Onera Satellite Change Detection dataset
  • SiameseSSL -> code for 2022 paper: Urban change detection with a Dual-Task Siamese network and semi-supervised learning. Uses SpaceNet 7 dataset
  • CD-SOTA-methods -> Remote sensing change detection: State-of-the-art methods and available datasets
  • multimodalCD_ISPRS21 -> code for 2021 paper: Fusing Multi-modal Data for Supervised Change Detection
  • Unsupervised-CD-in-SITS-using-DL-and-Graphs -> code for article: Unsupervised Change Detection Analysis in Satellite Image Time Series using Deep Learning Combined with Graph-Based Approaches
  • LSNet -> code for 2022 paper: Extremely Light-Weight Siamese Network For Change Detection in Remote Sensing Image
  • Change-Detection-in-Remote-Sensing-Images -> using PCA & K-means
  • End-to-end-CD-for-VHR-satellite-image -> code for 2019 paper: End-to-End Change Detection for High Resolution Satellite Images Using Improved UNet++
  • Semantic-Change-Detection -> code for 2021 paper: SCDNET: A novel convolutional network for semantic change detection in high resolution optical remote sensing imagery
  • ERCNN-DRS_urban_change_monitoring -> code for 2021 paper: Neural Network-Based Urban Change Monitoring with Deep-Temporal Multispectral and SAR Remote Sensing Data
  • EGRCNN -> code for 2021 paper: Edge-guided Recurrent Convolutional Neural Network for Multi-temporal Remote Sensing Image Building Change Detection
  • Unsupervised-Remote-Sensing-Change-Detection -> code for 2021 paper: An Unsupervised Remote Sensing Change Detection Method Based on Multiscale Graph Convolutional Network and Metric Learning
  • CropLand-CD -> code for 2022 paper: A CNN-transformer Network with Multi-scale Context Aggregation for Fine-grained Cropland Change Detection
  • contrastive-surface-image-pretraining -> code for 2022 paper: Supervising Remote Sensing Change Detection Models with 3D Surface Semantics
  • dcvaVHROptical -> Deep Change Vector Analysis (DCVA) change detection. Code for 2019 paper: Unsupervised Deep Change Vector Analysis for Multiple-Change Detection in VHR Images
  • hyperdimensionalCD -> code for 2021 paper: Change Detection in Hyperdimensional Images Using Untrained Models
  • DSFANet -> code for 2018 paper: Unsupervised Deep Slow Feature Analysis for Change Detection in Multi-Temporal Remote Sensing Images
  • FCD-GAN-pytorch -> Fully Convolutional Change Detection Framework with Generative Adversarial Network (FCD-GAN) is a framework for change detection in multi-temporal remote sensing images
  • DARNet-CD -> code for 2022 paper: A Densely Attentive Refinement Network for Change Detection Based on Very-High-Resolution Bitemporal Remote Sensing Images
  • xView2_Vulcan -> Damage assessment using pre and post orthoimagery. Modified + productionized model based off the first-place model from the xView2 challenge.
  • ESCNet -> code for 2021 paper: An End-to-End Superpixel-Enhanced Change Detection Network for Very-High-Resolution Remote Sensing Images
  • ForestCoverChange -> Detecting and Predicting Forest Cover Change in Pakistani Areas Using Remote Sensing Imagery
  • deforestation-detection -> code for 2020 paper: DEEP LEARNING FOR HIGH-FREQUENCY CHANGE DETECTION IN UKRAINIAN FOREST ECOSYSTEM WITH SENTINEL-2
  • forest_change_detection -> forest change segmentation with time-dependent models, including Siamese, UNet-LSTM, UNet-diff, UNet3D models. Code for 2021 paper: Deep Learning for Regular Change Detection in Ukrainian Forest Ecosystem With Sentinel-2
  • SentinelClearcutDetection -> Scripts for deforestation detection on the Sentinel-2 Level-A images
  • clearcut_detection -> research & web-service for clearcut detection
  • CDRL -> code for 2022 paper: Unsupervised Change Detection Based on Image Reconstruction Loss
  • ddpm-cd -> code for 2022 paper: Remote Sensing Change Detection (Segmentation) using Denoising Diffusion Probabilistic Models
  • Remote-sensing-time-series-change-detection -> code for 2022 paper: Graph-based block-level urban change detection using Sentinel-2 time series
  • austin-ml-change-detection-demo -> A change detection demo for the Austin area using a pre-trained PyTorch model scaled with Dask on Planet imagery
  • dfc2021-msd-baseline -> A baseline for the "Multitemporal Semantic Change Detection" track of the 2021 IEEE GRSS Data Fusion Competition
  • CorrFusionNet -> code for 2020 paper: Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion
  • ChangeDetectionPCAKmeans -> MATLAB implementation for Unsupervised Change Detection in Satellite Images Using Principal Component Analysis and k-Means Clustering.
  • IRCNN -> code for 2022 paper: IRCNN: An Irregular-Time-Distanced Recurrent Convolutional Neural Network for Change Detection in Satellite Time Series
  • UTRNet -> An Unsupervised Time-Distance-Guided Convolutional Recurrent Network for Change Detection in Irregularly Collected Images
  • open-cd -> an open source change detection toolbox based on a series of open source general vision task tools
  • Tiny_model_4_CD -> code for 2022 paper: TINYCD: A (Not So) Deep Learning Model For Change Detection. Uses LEVIR-CD & WHU-CD datasets
  • FHD -> code for 2022 paper: Feature Hierarchical Differentiation for Remote Sensing Image Change Detection
  • Change detection with Raster Vision -> blog post with Colab notebook
  • building-expansion -> code for 2021 paper: Enhancing Environmental Enforcement with Near Real-Time Monitoring: Likelihood-Based Detection of Structural Expansion of Intensive Livestock Farms
  • SaDL_CD -> code for 2022 paper: Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection
  • EGCTNet_pytorch -> code for 2022 paper: Building Change Detection Based on an Edge-Guided Convolutional Neural Network Combined with a Transformer
  • S2-cGAN -> code for 2020 paper: S2-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images

Time series

More general than change detection, time series observations can be used for applications including improving the accuracy of crop classification, or predicting future patterns & events. Crop yield is very typically application and has its own section below

  • CropDetectionDL -> using GRU-net, First place solution for Crop Detection from Satellite Imagery competition organized by CV4A workshop at ICLR 2020
  • LANDSAT Time Series Analysis for Multi-temporal Land Cover Classification using Random Forest
  • temporalCNN -> Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series
  • pytorch-psetae -> PyTorch implementation of the model presented in Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention
  • satflow -> optical flow models for predicting future satellite images from current and past ones
  • esa-superresolution-forecasting -> Forecasting air pollution using ESA Sentinel-5p data, and an encoder-decoder convolutional LSTM neural network architecture, implemented in Pytorch
  • Radiant-Earth-Spot-the-Crop-Challenge -> The main objective of this challenge was to use time-series of Sentinel-2 multi-spectral data to classify crops in the Western Cape of South Africa. The challenge was to build a machine learning model to predict crop type classes for the test dataset
  • lightweight-temporal-attention-pytorch -> A PyTorch implementation of the Light Temporal Attention Encoder (L-TAE) for satellite image time series. classification
  • Crop-Classification -> crop classification using multi temporal satellite images
  • dtwSat -> Time-Weighted Dynamic Time Warping for satellite image time series analysis
  • DeepCropMapping -> A multi-temporal deep learning approach with improved spatial generalizability for dynamic corn and soybean mapping, uses LSTM
  • CropMappingInterpretation -> An interpretation pipeline towards understanding multi-temporal deep learning approaches for crop mapping
  • MTLCC -> code for paper: Multitemporal Land Cover Classification Network. A recurrent neural network approach to encode multi-temporal data for land cover classification
  • timematch -> code for 2022 paper: A method to perform unsupervised cross-region adaptation of crop classifiers trained with satellite image time series. We also introduce an open-access dataset for cross-region adaptation with SITS from four different regions in Europe
  • PWWB -> Code for the 2021 paper: Real-Time Spatiotemporal Air Pollution Prediction with Deep Convolutional LSTM through Satellite Image Analysis
  • Classification of Crop Fields through Satellite Image Time Series -> using a pytorch-psetae & Sentinel-2 data
  • spaceweather -> predicting geomagnetic storms from satellite measurements of the solar wind and solar corona, uses LSTMs
  • Forest_wildfire_spreading_convLSTM -> Modeling of the spreading of forest wildfire using a neural network with ConvLSTM cells. Prediction 3-days forward
  • ConvTimeLSTM -> Extension of ConvLSTM and Time-LSTM for irregularly spaced images, appropriate for Remote Sensing
  • dl-time-series -> Deep Learning algorithms applied to characterization of Remote Sensing time-series
  • tpe -> code for 2022 paper: Generalized Classification of Satellite Image Time Series With Thermal Positional Encoding
  • wildfire_forecasting -> code for 2021 paper: Deep Learning Methods for Daily Wildfire Danger Forecasting. Uses ConvLSTM
  • satellite_image_forecasting -> predict future satellite images from past ones using features such as precipitation and elevation maps. Entry for the EarthNet2021 challenge

Crop yield

Wealth and economic activity

The goal is to predict economic activity from satellite imagery rather than conducting labour intensive ground surveys

Disaster response

Also checkout the sections on change detection and water/fire/building segmentation

Super-resolution

Super-resolution attempts to enhance the resolution of an imaging system, and can be applied as a pre-processing step to improve the detection of small objects or boundaries. Its use is controversial since it can introduce artefacts at the same rate as real features. These techniques are generally grouped into single image super resolution (SISR) or a multi image super resolution (MISR)

Single image super-resolution (SISR)

Multi image super-resolution (MISR)

Note that nearly all the MISR publications resulted from the PROBA-V Super Resolution competition

  • deepsum -> Deep neural network for Super-resolution of Unregistered Multitemporal images (ESA PROBA-V challenge)
  • 3DWDSRNet -> code to reproduce Satellite Image Multi-Frame Super Resolution (MISR) Using 3D Wide-Activation Neural Networks
  • RAMS -> Official TensorFlow code for paper Multi-Image Super Resolution of Remotely Sensed Images Using Residual Attention Deep Neural Networks
  • TR-MISR -> Transformer-based MISR framework for the the PROBA-V super-resolution challenge. With paper
  • HighRes-net -> Pytorch implementation of HighRes-net, a neural network for multi-frame super-resolution, trained and tested on the European Space Agency’s Kelvin competition
  • ProbaVref -> Repurposing the Proba-V challenge for reference-aware super resolution
  • The missing ingredient in deep multi-temporal satellite image super-resolution -> Permutation invariance harnesses the power of ensembles in a single model, with repo piunet
  • MSTT-STVSR -> Space-time Super-resolution for Satellite Video: A Joint Framework Based on Multi-Scale Spatial-Temporal Transformer, JAG, 2022
  • Self-Supervised Super-Resolution for Multi-Exposure Push-Frame Satellites
  • DDRN -> Deep Distillation Recursive Network for Video Satellite Imagery Super-Resolution
  • worldstrat -> SISR and MISR implementations of SRCNN
  • MISR-GRU -> Pytorch implementation of MISR-GRU, a deep neural network for multi image super-resolution (MISR), for ProbaV Super Resolution Competition

Pansharpening

Image fusion of low res multispectral with high res pan band.

  • Several algorithms described in the ArcGIS docs, with the simplest being taking the mean of the pan and RGB pixel value.
  • For into to classical methods see this notebook and this kaggle kernel
  • rio-pansharpen -> pansharpening Landsat scenes
  • Simple-Pansharpening-Algorithms
  • Working-For-Pansharpening -> long list of pansharpening methods and update of Awesome-Pansharpening
  • PSGAN -> A Generative Adversarial Network for Remote Sensing Image Pan-sharpening, arxiv paper
  • Pansharpening-by-Convolutional-Neural-Network
  • PBR_filter -> {P}ansharpening by {B}ackground {R}emoval algorithm for sharpening RGB images
  • py_pansharpening -> multiple algorithms implemented in python
  • Deep-Learning-PanSharpening -> deep-learning based pan-sharpening code package, we reimplemented include PNN, MSDCNN, PanNet, TFNet, SRPPNN, and our purposed network DIPNet
  • HyperTransformer -> A Textural and Spectral Feature Fusion Transformer for Pansharpening
  • DIP-HyperKite -> Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction
  • D2TNet -> code for 2022 paper: A ConvLSTM Network with Dual-direction Transfer for Pan-sharpening
  • PanColorGAN-VHR-Satellite-Images -> code for 2020 paper: Rethinking CNN-Based Pansharpening: Guided Colorization of Panchromatic Images via GANs
  • MTL_PAN_SEG -> code for 2019 paper: Multi-task deep learning for satellite image pansharpening and segmentation
  • Z-PNN -> code for 2022 paper: Pansharpening by convolutional neural networks in the full resolution framework
  • GTP-PNet -> code for 2021 paper: GTP-PNet: A residual learning network based on gradient transformation prior for pansharpening
  • UDL -> code for 2021 paper: Dynamic Cross Feature Fusion for Remote Sensing Pansharpening
  • PSData -> A Large-Scale General Pan-sharpening DataSet, which contains PSData3 (QB, GF-2, WV-3) and PSData4 (QB, GF-1, GF-2, WV-2).
  • AFPN -> Adaptive Detail Injection-Based Feature Pyramid Network For Pan-sharpening
  • pan-sharpening -> multiple methods demonstrated for multispectral and panchromatic images
  • PSGan-Family -> code for 2020 paper: PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening
  • PanNet-Landsat -> code for 2017 paper: A Deep Network Architecture for Pan-Sharpening
  • DLPan-Toolbox -> code for 2022 paper: Machine Learning in Pansharpening: A Benchmark, from Shallow to Deep Networks
  • LPPN -> code for 2021 paper: Laplacian pyramid networks: A new approach for multispectral pansharpening
  • S2_SSC_CNN -> code for 2020 paper: Zero-shot Sentinel-2 Sharpening Using A Symmetric Skipped Connection Convolutional Neural Network
  • S2S_UCNN -> code for 2021 paper: Sentinel 2 sharpening using a single unsupervised convolutional neural network with MTF-Based degradation model
  • SSE-Net -> code for 2022 paper: Spatial and Spectral Extraction Network With Adaptive Feature Fusion for Pansharpening
  • UCGAN -> code for 2022 paper: Unsupervised Cycle-consistent Generative Adversarial Networks for Pan-sharpening
  • GCPNet -> code for 2022 paper: When Pansharpening Meets Graph Convolution Network and Knowledge Distillation
  • PanFormer -> code for 2022 paper: PanFormer: a Transformer Based Model for Pan-sharpening
  • Pansharpening -> code for 2021 paper: Pansformers: Transformer-Based Self-Attention Network for Pansharpening

Image-to-image translation

Translate images e.g. from SAR to RGB.

GANS

GANS are famously used for generating synthetic data, see the section Synthetic data

Transformers

Adversarial ML

Efforts to detect falsified images & deepfakes. Also checkout Synthetic data

  • UAE-RS -> dataset that provides black-box adversarial samples in the remote sensing field
  • PSGAN -> code for paper: Perturbation Seeking Generative Adversarial Networks: A Defense Framework for Remote Sensing Image Scene Classification
  • SACNet -> code for 2021 paper: Self-Attention Context Network: Addressing the Threat of Adversarial Attacks for Hyperspectral Image Classification

Autoencoders, dimensionality reduction, image embeddings & similarity search

Image retreival

  • Demo_AHCL_for_TGRS2022 -> code for 2022 paper: Asymmetric Hash Code Learning (AHCL) for remote sensing image retreival
  • GaLR -> code for 2022 paper: Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information
  • retrievalSystem -> cross-modal image retrieval system
  • AMFMN -> code for the 2021 paper: Exploring a Fine-grained Multiscale Method for Cross-modal Remote Sensing Image Retrieval
  • Active-Learning-for-Remote-Sensing-Image-Retrieval -> unofficial implementation of paper: A Novel Active Learning Method in Relevance Feedback for Content-Based Remote Sensing Image Retrieval
  • CMIR-NET -> code for 2020 paper: A deep learning based model for cross-modal retrieval in remote sensing
  • Deep-Hash-learning-for-Remote-Sensing-Image-Retrieval -> code for 2020 paper: Deep Hash Learning for Remote Sensing Image Retrieval
  • MHCLN -> code for 2018 paper: Deep Metric and Hash-Code Learning for Content-Based Retrieval of Remote Sensing Images
  • HydroViet_VOR -> Object Retrieval in satellite images with Triplet Network
  • AMFMN -> code for 2021 paper: Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

Image Captioning & Visual Question Answering

Mixed data learning

These techniques combine multiple data types, e.g. imagery and text data.

Few-shot learning

This is a class of techniques which attempt to make predictions for classes with few, one or even zero examples provided during training. In zero shot learning (ZSL) the model is assisted by the provision of auxiliary information which typically consists of descriptions/semantic attributes/word embeddings for both the seen and unseen classes at train time (ref). These approaches are particularly relevant to remote sensing, where there may be many examples of common classes, but few or even zero examples for other classes of interest.

Self-supervised, unsupervised & contrastive learning

These techniques use unlabelled datasets. Yann LeCun has described self/unsupervised learning as the 'base of the cake': If we think of our brain as a cake, then the cake base is unsupervised learning. The machine predicts any part of its input for any observed part, all without the use of labelled data. Supervised learning forms the icing on the cake, and reinforcement learning is the cherry on top.

Weakly & semi-supervised learning

These techniques use a partially annotated dataset

  • MARE -> self-supervised Multi-Attention REsu-net for semantic segmentation in remote sensing
  • SSGF-for-HRRS-scene-classification -> code for 2018 paper: A semi-supervised generative framework with deep learning features for high-resolution remote sensing image scene classification
  • SFGAN -> code for 2018 paper: Semantic-Fusion Gans for Semi-Supervised Satellite Image Classification
  • SSDAN -> code for 2021 paper: Multi-Source Semi-Supervised Domain Adaptation Network for Remote Sensing Scene Classification
  • HR-S2DML -> code for 2020 paper: High-Rankness Regularized Semi-Supervised Deep Metric Learning for Remote Sensing Imagery
  • Semantic Segmentation of Satellite Images Using Point Supervision
  • fcd -> code for 2021 paper: Fixed-Point GAN for Cloud Detection. A weakly-supervised approach, training with only image-level labels
  • weak-segmentation -> Weakly supervised semantic segmentation for aerial images in pytorch
  • TNNLS_2022_X-GPN -> Code for paper: Semisupervised Cross-scale Graph Prototypical Network for Hyperspectral Image Classification
  • weakly_supervised -> code for the paper Weakly Supervised Deep Learning for Segmentation of Remote Sensing Imagery. Demonstrates that segmentation can be performed using small datasets comprised of pixel or image labels
  • wan -> Weakly-Supervised Domain Adaptation for Built-up Region Segmentation in Aerial and Satellite Imagery, with arxiv paper
  • sourcerer -> A Bayesian-inspired deep learning method for semi-supervised domain adaptation designed for land cover mapping from satellite image time series (SITS). Paper
  • MSMatch -> Semi-Supervised Multispectral Scene Classification with Few Labels. Includes code to work with both the RGB and the multispectral (MS) versions of EuroSAT dataset and the UC Merced Land Use (UCM) dataset. Paper
  • Flood Segmentation on Sentinel-1 SAR Imagery with Semi-Supervised Learning with arxiv paper
  • Semi-supervised learning in satellite image classification -> experimenting with MixMatch and the EuroSAT data set
  • ScRoadExtractor -> code for 2020 paper: Scribble-based Weakly Supervised Deep Learning for Road Surface Extraction from Remote Sensing Images
  • ICSS -> code for 2022 paper: Weakly-supervised continual learning for class-incremental segmentation
  • es-CP -> code for 2022 paper: Semi-Supervised Hyperspectral Image Classification Using a Probabilistic Pseudo-Label Generation Framework

Active learning

Supervised deep learning techniques typically require a huge number of annotated/labelled examples to provide a training dataset. However labelling at scale take significant time, expertise and resources. Active learning techniques aim to reduce the total amount of annotation that needs to be performed by selecting the most useful images to label from a large pool of unlabelled images, thus reducing the time to generate useful training datasets. These processes may be referred to as Human-in-the-Loop Machine Learning

Federated learning

Federated learning is a process for training models in a distributed fashion without sharing of data

Image registration

Image registration is the process of registering one or more images onto another (typically well georeferenced) image. Traditionally this is performed manually by identifying control points (tie-points) in the images, for example using QGIS. This section lists approaches which mostly aim to automate this manual process. There is some overlap with the data fusion section but the distinction I make is that image registration is performed as a prerequisite to downstream processes which will use the registered data as an input.

Data fusion

Data fusion covers techniques which integrate multiple datasources, for example fusing SAR & optical to make predictions about crop type. It can also cover fusion with non imagery data such as IOT sensor data

  • Awesome-Data-Fusion-for-Remote-Sensing
  • UDALN_GRSL -> Deep Unsupervised Blind Hyperspectral and Multispectral Data Fusion
  • CropTypeMapping -> Crop type mapping from optical and radar (Sentinel-1&2) time series using attention-based deep learning
  • Multimodal-Remote-Sensing-Toolkit -> uses Hyperspectral and LiDAR Data
  • Aerial-Template-Matching -> development of an algorithm for template Matching on aerial imagery applied to UAV dataset
  • DS_UNet -> code for 2021 paper: Sentinel-1 and Sentinel-2 Data Fusion for Urban Change Detection using a Dual Stream U-Net, uses Onera Satellite Change Detection dataset
  • DDA_UrbanExtraction -> Unsupervised Domain Adaptation for Global Urban Extraction using Sentinel-1 and Sentinel-2 Data
  • swinstfm -> code for paper: Remote Sensing Spatiotemporal Fusion using Swin Transformer
  • LoveCS -> code for 2022 paper: Cross-sensor domain adaptation for high-spatial resolution urban land-cover mapping: from airborne to spaceborne imagery
  • comingdowntoearth -> code for 2021 paper: Implementation of 'Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization'
  • Matching between acoustic and satellite images
  • MapRepair -> Deep Cadastre Maps Alignment and Temporal Inconsistencies Fix in Satellite Images
  • Compressive-Sensing-and-Deep-Learning-Framework -> Compressive Sensing is used as an initial guess to combine data from multiple sources, with LSTM used to refine the result
  • DeepSim -> code for paper: DeepSIM: GPS Spoofing Detection on UAVs using Satellite Imagery Matching
  • MHF-net -> code for 2019 paper: Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net
  • Remote_Sensing_Image_Fusion -> code for 2021 paper: Semi-Supervised Remote Sensing Image Fusion Using Multi-Scale Conditional Generative Adversarial network with Siamese Structure
  • CNNs for Multi-Source Remote Sensing Data Fusion -> code for 2021 paper: Single-stream CNN with Learnable Architecture for Multi-source Remote Sensing Data
  • Deep Generative Reflectance Fusion -> Achieving Landsat-like reflectance at any date by fusing Landsat and MODIS surface reflectance with deep generative models
  • IEEE_TGRS_MDL-RS -> code for 2021 paper: More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification
  • SSRNET -> code for 2022 paper: SSR-NET: Spatial-Spectral Reconstruction Network for Hyperspectral and Multispectral Image Fusion
  • cross-view-image-matching -> code for 2019 paper: Bridging the Domain Gap for Ground-to-Aerial Image Matching
  • CoF-MSMG-PCNN -> code for 2020 paper: Remote Sensing Image Fusion via Boundary Measured Dual-Channel PCNN in Multi-Scale Morphological Gradient Domain
  • robust_matching_network_on_remote_sensing_imagery_pytorch -> code for 2019 paper: A Robust Matching Network for Gradually Estimating Geometric Transformation on Remote Sensing Imagery
  • edcstfn -> code for 2019 paper: An Enhanced Deep Convolutional Model for Spatiotemporal Image Fusion
  • ganstfm -> code for 2021 paper: A Flexible Reference-Insensitive Spatiotemporal Fusion Model for Remote Sensing Images Using Conditional Generative Adversarial Network
  • CMAFF -> code for 2021 paper: Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery
  • SOLC -> code for 2022 paper: MCANet: A joint semantic segmentation framework of optical and SAR images for land use classification. Uses WHU-OPT-SAR-dataset
  • MFT -> code for 2022 paper: Multimodal Fusion Transformer for Remote Sensing Image Classification
  • ISPRS_S2FL -> code for 2021 paper: Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model
  • HSHT-Satellite-Imagery-Synthesis -> code for thesis - Improving Flood Maps by Increasing the Temporal Resolution of Satellites Using Hybrid Sensor Fusion
  • MDC -> code for 2021 paper: Unsupervised Data Fusion With Deeper Perspective: A Novel Multisensor Deep Clustering Algorithm
  • FusAtNet -> code for 2020 paper: FusAtNet: Dual Attention based SpectroSpatial Multimodal Fusion Network for Hyperspectral and LiDAR Classification
  • AMM-FuseNet -> code for 2022 paper: AMM-FuseNet: Attention-Based Multi-Modal Image Fusion Network for Land Cover Mapping
  • S1-S2_Transformer -> Sentinel-1 SAR and Sentinel-2 optical timeseries based Transformer architecture for tropical dry forest disturbance mapping
  • MANet -> code for 2022 paper: MANet: A Network Architecture for Remote Sensing Spatiotemporal Fusion Based on Multiscale and Attention Mechanisms
  • DCSA-Net -> code for 2022 paper: Dynamic Convolution Self-Attention Network for Land-Cover Classification in VHR Remote-Sensing Images

Terrain mapping, Disparity Estimation, Lidar, DEMs & NeRF

Measure surface contours & locate 3D points in space from 2D images. NeRF stands for Neural Radiance Fields and is the term used in deep learning communities to describe a model that generates views of complex 3D scenes based on a partial set of 2D images

Thermal Infrared

SAR

See SAR

NDVI - vegetation index

General image quality

  • Convolutional autoencoder network can be employed to image denoising, read about this on the Keras blog
  • jitter-compensation -> Remote Sensing Image Jitter Detection and Compensation Using CNN
  • DeblurGANv2 -> Deblurring (Orders-of-Magnitude) Faster and Better
  • image-quality-assessment -> CNN to predict the aesthetic and technical quality of images
  • Convolutional autoencoder for image denoising -> keras guide
  • piq -> a collection of measures and metrics for image quality assessment
  • FFA-Net -> Feature Fusion Attention Network for Single Image Dehazing
  • DeepCalib -> A Deep Learning Approach for Automatic Intrinsic Calibration of Wide Field-of-View Cameras
  • PerceptualSimilarity -> LPIPS is a perceptual metric which aims to overcome the limitations of traditional metrics such as PSNR & SSIM, to better represent the features the human eye picks up on
  • Optical-RemoteSensing-Image-Resolution -> code for 2018 paper: Deep Memory Connected Neural Network for Optical Remote Sensing Image Restoration. Two applications: Gaussian image denoising and single image super-resolution
  • Hyperspectral-Deblurring-and-Destriping
  • HyDe -> Hyperspectral Denoising algorithm toolbox in Python, with paper
  • HLF-DIP -> code for 2022 paper: Unsupervised Hyperspectral Denoising Based on Deep Image Prior and Least Favorable Distribution
  • RQUNetVAE -> code for 2022 paper: Riesz-Quincunx-UNet Variational Auto-Encoder for Satellite Image Denoising
  • deep-hs-prior -> code for 2019 paper: Deep Hyperspectral Prior: Denoising, Inpainting, Super-Resolution
  • iquaflow -> from Satellogic, an image quality framework that aims at providing a set of tools to assess image quality by using the performance of AI models trained on the images as a proxy.

Synthetic data

Training data can be hard to acquire, particularly for rare events such as change detection after disasters, or imagery of rare classes of objects. In these situations, generating synthetic training data might be the only option. This has become quite sophisticated, with 3D models being use with open source games engines such as Unreal.

Image dataset creation

Many datasets on kaggle & elsewhere have been created by screen-clipping Google Maps or browsing web portals. The tools below are to create datasets programatically

  • MapTilesDownloader -> A super easy to use map tiles downloader built using Python
  • jimutmap -> get enormous amount of high resolution satellite images from apple / google maps quickly through multi-threading
  • google-maps-downloader -> A short python script that downloads satellite imagery from Google Maps
  • ExtractSatelliteImagesFromCSV -> extract satellite images using a CSV file that contains latitude and longitude, uses mapbox
  • sentinelsat -> Search and download Copernicus Sentinel satellite images
  • SentinelDownloader -> a high level wrapper to the SentinelSat that provides an object oriented interface, asynchronous downloading, quickview & simpler searching methods
  • GEES2Downloader -> Downloader for GEE S2 bands
  • Sentinel-2 satellite tiles images downloader from Copernicus -> Minimizes data download and combines multiple tiles to return a single area of interest
  • felicette -> Satellite imagery for dummies. Generate JPEG earth imagery from coordinates/location name with publicly available satellite data
  • Easy Landsat Download
  • A simple python scrapper to get satellite images of Africa, Europe and Oceania's weather using the Sat24 website
  • RGISTools -> Tools for Downloading, Customizing, and Processing Time Series of Satellite Images from Landsat, MODIS, and Sentinel
  • DeepSatData -> Automatically create machine learning datasets from satellite images
  • landsat_ingestor -> Scripts and other artifacts for landsat data ingestion into Amazon public hosting
  • satpy -> a python library for reading and manipulating meteorological remote sensing data and writing it to various image and data file formats
  • GIBS-Downloader -> a command-line tool which facilitates the downloading of NASA satellite imagery and offers different functionalities in order to prepare the images for training in a machine learning pipeline
  • eodag -> Earth Observation Data Access Gateway
  • pylandsat -> Search, download, and preprocess Landsat imagery
  • landsatxplore -> Search and download Landsat scenes from EarthExplorer
  • OpenSarToolkit -> High-level functionality for the inventory, download and pre-processing of Sentinel-1 data in the python language
  • lsru -> Query and Order Landsat Surface Reflectance data via ESPA
  • eoreader -> Remote-sensing opensource python library reading optical and SAR sensors, loading and stacking bands, clouds, DEM and index in a sensor-agnostic way
  • Export thumbnails from Earth Engine
  • deepsentinel-osm -> A repository to generate land cover labels from OpenStreetMap
  • img2dataset -> Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine
  • ohsome2label -> Historical OpenStreetMap (OSM) Objects to Machine Learning Training Samples
  • Label Maker -> a library for creating machine-learning ready data by pairing satellite images with OpenStreetMap (OSM) vector data
  • sentinel2tools -> downloading & basic processing of Sentinel 2 imagesry. Read Sentinel2tools: simple lib for downloading Sentinel-2 satellite images
  • Aerial-Satellite-Imagery-Retrieval -> A program using Bing maps tile system to automatically download Aerial / Satellite Imagery given a lat/lon bounding box and level of detail
  • google-maps-at-88-mph -> Google Maps keeps old satellite imagery around for a while – this tool collects what's available for a user-specified region in the form of a GIF
  • srtmDownloader -> Python library (multi-threaded) for retrieving SRTM elevation map of CGIAR-CSI
  • ImageDatasetViz -> create a mosaic of images in a dataset for previewing purposes
  • landsatlinks -> A simple CLI interface to generate download urls for Landsat Collection 2 Level 1 product bundles
  • pyeo -> a set of portable, extensible and modular Python scripts for machine learning in earth observation and GIS, including downloading, preprocessing, creation of base layers, classification and validation.
  • metaearth -> Download and access remote sensing data from any platform
  • geoget -> Download geodata for anywhere in Earth via ladsweb.modaps.eosdis.nasa.gov
  • geeml -> A python package to extract Google Earth Engine data for machine learning

Image chipping/tiling & merging

Since raw images can be very large, it is usually necessary to chip/tile them into smaller images before annotation & training

  • image_slicer -> Split images into tiles. Join the tiles back together
  • tiler by nuno-faria -> split images into tiles and merge tiles into a large image
  • tiler by the-lay -> N-dimensional NumPy array tiling and merging with overlapping, padding and tapering
  • xbatcher -> Xbatcher is a small library for iterating xarray DataArrays in batches. The goal is to make it easy to feed xarray datasets to machine learning libraries such as Keras
  • GeoTagged_ImageChip -> A simple script to create geo tagged image chips from high resolution RS iamges for training deep learning models such as Unet
  • geotiff-crop-dataset -> A Pytorch Dataloader for tif image files that dynamically crops the image
  • Train-Test-Validation-Dataset-Generation -> app to crop images and create small patches of a large image e.g. Satellite/Aerial Images, which will then be used for training and testing Deep Learning models specifically semantic segmentation models
  • satproc -> Python library and CLI tools for processing geospatial imagery for ML
  • Sliding Window -> break large images into a series of smaller chunks
  • patchify -> A library that helps you split image into small, overlappable patches, and merge patches into original image
  • split-rs-data -> Divide remote sensing images and their labels into data sets of specified size
  • image-reconstructor-patches -> Reconstruct Image from Patches with a Variable Stride
  • rpc_cropper -> A small standalone tool to crop satellite images and their RPC
  • geotile -> python library for tiling the geographic raster data
  • GeoPatch -> generating patches from remote sensing data
  • ImageTilingUtils -> Minimalistic set of image reader agnostic tools to easily iterate over large images
  • split_raster -> Creates a tiled output from an input raster dataset. pip installable
  • SAHI -> Utilties for slicing COCO formatted annotations and image files, performing sliced inference using MMDetection, Detectron2, YOLOv5, HuggingFace detectors and calculating AP over image slices.

Annotation

For supervised machine learning, you will require annotated images. For example if you are performing object detection you will need to annotate images with bounding boxes. Check that your annotation tool of choice supports large image (likely geotiff) files, as not all will. Note that GeoJSON is widely used by remote sensing researchers but this annotation format is not commonly supported in general computer vision frameworks, and in practice you may have to convert the annotation format to use the data with your chosen framework. There are both closed and open source tools for creating and converting annotation formats. Some of these tools are simply for performing annotation, whilst others add features such as dataset management and versioning. Note that self-supervised and active learning approaches might circumvent the need to perform a large scale annotation exercise. Note that tiffs/geotiffs cannot be displayed by most browsers (Chrome), but CAN render in Safari.

Annotation tools with GEO features

  • GroundWork is designed for annotating and labeling geospatial data like satellite imagery, from Azavea
  • labelbox.com -> free tier is quite generous, supports annotating Geotiffs & returning annotations with geospatial coordinates. Watch this webcast
  • diffgram describes itself as a complete training data platform for machine learning delivered as a single application, supports streaming data to pytorch & tensorflow. COGS can be annotated
  • iris -> Tool for manual image segmentation and classification of satellite imagery
  • If you are considering building an in house annotation platform read this article. Used PostGis database, GeoJson format and GIS standard in a stateless architecture
  • satellite-imagery-labeling-tool -> from Microsoft, this is a lightweight web-interface for creating and sharing vector annotations over satellite/aerial imagery scenes
  • RSLabel -> remote sensing (RS) image annotation tool for deep learning
  • encord -> supports annotatin SAR
  • Flood-Annotation-Tool -> Annotation tool to annotate flooded and non-flooded regions on satellite image, uses jupyter notebook

Open source annotation tools

  • awesome-data-labeling -> long list of annotation tools
  • awesome-open-data-annotation -> another long list of annotation tools
  • labelImg is the classic desktop tool, limited to bounding boxes for object detection. Also checkout roLabelImg which supports ROTATED rectangle regions, as often occurs in aerial imagery. labelImg_OBB is another fork supporting orinted bounding boxes (OBB)
  • Labelme is a very popular & simple dektop app for polygonal annotation suitable for object detection and semantic segmentation. Note it outputs annotations in a custom LabelMe JSON format which you will need to convert, e.g. using labelme2coco. Read Labelme Image Annotation for Geotiffs
  • Label Studio is a multi-type data labeling and annotation tool with standardized output format, syncing to buckets, and supports importing pre-annotations (create with a model). Checkout label-studio-converter for converting Label Studio annotations into common dataset formats
  • CVAT suports object detection, segmentation and classification via a local web app. This article on Roboflow gives a good intro to CVAT. Checkout CVAT images validator
  • VoTT -> an electron app for building end to end Object Detection Models from Images and Videos, by Microsoft
  • Create your own annotation tool using Bokeh Holoviews, tkinter, or see these dash examples for object detection and segmentation
  • Deeplabel is a cross-platform tool for annotating images with labelled bounding boxes. Deeplabel also supports running inference using state-of-the-art object detection models like Faster-RCNN and YOLOv4. With support out-of-the-box for CUDA, you can quickly label an entire dataset using an existing model.
  • Alturos.ImageAnnotation is a collaborative tool for labeling image data on S3 for yolo
  • pigeonXT -> create custom image classification annotators within Jupyter notebooks
  • ipyannotations -> Image annotations in python using Jupyter notebooks
  • Label-Detect -> is a graphical image annotation tool and using this tool a user can also train and test large satellite images, fork of the popular labelImg tool
  • Swipe-Labeler -> Swipe Labeler is a Graphical User Interface based tool that allows rapid labeling of image data
  • dash_doodler -> A web application built with plotly/dash for image segmentation with minimal supervision
  • remo -> A webapp and Python library that lets you explore and control your image datasets
  • TensorFlow Object Detection API provides a handy utility for object annotation within Google Colab notebooks. See usage here
  • coco-annotator -> Web-based image segmentation tool for object detection, localization, and keypoints
  • pylabel -> Python library for computer vision labeling tasks. The core functionality is to translate bounding box annotations between different formats-for example, from coco to yolo. PyLabel also includes an image labeling tool that runs in a Jupyter notebook that can annotate images manually or perform automatic labeling using a pre-trained model
  • BMW-Labeltool-Lite -> bounding box annotator
  • django-labeller -> An image labelling tool for creating segmentation data sets, for Django and Flask
  • scalabel -> supports 2D images and 3D point clouds
  • Detection-Label-Tool -> Change detection and object annotation, uses PyQt
  • image_sorter -> A quick interface for sorting a folder of images into two other folders
  • bulk -> A Simple Bulk Labelling Tool using embeddings

Cloud hosted & paid annotation tools & services

Several open source tools are also available on the cloud, including CVAT, label-studio & Diffgram. In general cloud solutions will provide a lot of infrastructure and storage for you, as well as integration with outsourced annotators.

  • GroundWork is designed for annotating and labeling geospatial data like satellite imagery, from Azavea
  • labelbox.com -> free tier is quite generous, supports annotating Geotiffs & returning annotations with geospatial coordinates. Watch this webcast
  • Roboflow -> in addition to annotation this platform makes it easy to convert between annotation formats & manage datasets, as well as train and deploy custom models to private API endpoints. Read How to Train Computer Vision Models on Aerial Imagery
  • supervise.ly is one of the more fully featured platforms, decent free tier
  • AWS supports image annotation via the Rekognition Custom Labels console
  • rectlabel is a desktop app for MacOS to annotate images for bounding box object detection and segmentation, paid and free (rectlabel-lite) versions
  • hasty.ai -> supports model assisted annotation & inferencing

Annotation formats

Note there are many annotation formats, although PASCAL VOC and coco-json are the most commonly used. I recommend using geojson for storing polygons, then converting these to the required format when needed.

  • PASCAL VOC format: XML files in the format used by ImageNet
  • coco-json format: JSON in the format used by the 2015 COCO dataset
  • YOLO Darknet TXT format: contains one text file per image, used by YOLO
  • Tensorflow TFRecord: a proprietary binary file format used by the Tensorflow Object Detection API
  • Many more formats listed here
  • OBB: orinted bounding boxes are polygons representing rotated rectangles

Annotation visualisation & conversion tools

Tools to visualise annotations & convert between formats. Note that most annotation software will allow you to visualise existing annotations

  • Dataset-Converters -> a conversion toolset between different object detection and instance segmentation annotation formats
  • FiftyOne -> open-source tool for building high quality datasets and computer vision models. Visualise labels, evaluate model predictions, explore scenarios of interest, identify failure modes, find annotation mistakes, and much more! Read Nearest Neighbor Embeddings Search with Qdrant and FiftyOne
  • rebox -> Easily convert between bounding box annotation formats
  • Pascal VOC BBox Viewer
  • COCO-Assistant -> Helper for dealing with MS-COCO annotations; Merge datasets, Remove specfic category from dataset, Generate annotations statistics - distribution of object areas and category distribution
  • pybboxes -> Light weight toolkit for bounding boxes providing conversion between bounding box types and simple computations
  • voc2coco -> Convert VOC format XMLs to COCO format json
  • ObjectDetectionEval -> Parse all kinds of object detection databases (ImageNet, COCO, YOLO, PascalVOC, OpenImage, CVAT, LabelMe, etc.) & save to other formats
  • LabelMeYoloConverter -> Convert LabelMe Annotation Tool JSON format to YOLO text file format
  • mask-to-polygons -> Routines for extracting and working with polygons from semantic segmentation masks
  • labelme2coco -> Converts LabelMe JSON format into COCO object detection and instance segmentation format.

Training models

This section discusses training machine learning models.

Metrics

A number of metrics are common to all model types (but can have slightly different meanings in contexts such as object detection), whilst other metrics are very specific to particular classes of model. The correct choice of metric is particularly critical for imbalanced dataset problems, e.g. object detection

  • TP = true positive, FP = false positive, TN = true negative, FN = false negative
  • Precision is the % of correct positive predictions, calculated as precision = TP/(TP+FP)
  • Recall or true positive rate (TPR), is the % of true positives captured by the model, calculated as recall = TP/(TP+FN). Note that FN is not possible in object detection, so recall is not appropriate.
  • The F1 score (also called the F-score or the F-measure) is the harmonic mean of precision and recall, calculated as F1 = 2*(precision * recall)/(precision + recall). It conveys the balance between the precision and the recall. Ref
  • The false positive rate (FPR), calculated as FPR = FP/(FP+TN) is often plotted against recall/TPR in an ROC curve which shows how the TPR/FPR tradeoff varies with classification threshold. Lowering the classification threshold returns more true positives, but also more false positives. Note that since FN is not possible in object detection, ROC curves are not appropriate.
  • Precision-vs-recall curves visualise the tradeoff between making false positives and false negatives
  • Accuracy is the most commonly used metric in 'real life' but can be a highly misleading metric for imbalanced data sets.
  • IoU is an object detection specific metric, being the average intersect over union of prediction and ground truth bounding boxes for a given confidence threshold
  • [email protected] is another object detection specific metric, being the mean value of the average precision for each class. @0.5 sets a threshold for how much of the predicted bounding box overlaps the ground truth bounding box, i.e. "minimum 50% overlap"
  • For more comprehensive definitions checkout Object-Detection-Metrics
  • Metrics to Evaluate your Semantic Segmentation Model

ML best practice

This section includes tips and ideas I have picked up from other practitioners including ai-fast-track, FraPochetti & the IceVision community

Free online compute

A GPU is required for training deep learning models (but not necessarily for inferencing), and this section lists a couple of free Jupyter environments with GPU available. There is a good overview of online Jupyter development environments on the fastai site. I personally use Colab Pro with data hosted on Google Drive, or Sagemaker if I have very long running training jobs.

Google Colab

  • Collaboratory notebooks with GPU as a backend for free for 12 hours at a time. Note that the GPU may be shared with other users, so if you aren't getting good performance try reloading.
  • Also a pro tier for $10 a month -> https://colab.research.google.com/signup
  • Tensorflow, pytorch & fastai available but you may need to update them
  • Colab Alive is a chrome extension that keeps Colab notebooks alive.
  • colab-ssh -> lets you ssh to a colab instance like it’s an EC2 machine and install packages that require full linux functionality

Kaggle - also Google!

  • Free to use
  • GPU Kernels - may run for 1 hour
  • Tensorflow, pytorch & fastai available but you may need to update them
  • Advantage that many datasets are already available

Others

Deep learning projects & frameworks

  • TorchGeo -> a PyTorch domain library providing datasets, samplers, transforms, and pre-trained models specific to geospatial data, supported by Microsoft. Read Geospatial deep learning with TorchGeo
  • rastervision -> An open source Python framework for building computer vision models on aerial, satellite, and other large imagery sets
  • geotorch -> A Deep Learning and Scalable Data Processing Framework for Raster and Spatio-Temporal Datasets
  • torchrs -> PyTorch implementation of popular datasets and models in remote sensing tasksenhance) -> Enhance PyTorch vision for semantic segmentation, multi-channel images and TIF file
  • DeepHyperX -> A Python/pytorch tool to perform deep learning experiments on various hyperspectral datasets
  • DELTA -> Deep Earth Learning, Tools, and Analysis, by NASA is a framework for deep learning on satellite imagery, based on Tensorflow & using MLflow for tracking experiments
  • Lightly is a computer vision framework for training deep learning models using self-supervised learning
  • Icevision offers a curated collection of hundreds of high-quality pre-trained models within an easy to use framework
  • pytorch_eo -> aims to make Deep Learning for Earth Observation data easy and accessible to real-world cases and research alike
  • NGVEO -> applying convolutional neural networks (CNN) to Earth Observation (EO) data from Sentinel 1 and 2 using python and PyTorch
  • chip-n-scale-queue-arranger by developmentseed -> an orchestration pipeline for running machine learning inference at scale. Supports fastai models
  • http://spaceml.org/ -> A Machine Learning toolbox and developer community building the next generation AI applications for space science and exploration
  • TorchSat is an open-source deep learning framework for satellite imagery analysis based on PyTorch (no activity since June 2020)
  • DeepNetsForEO -> Uses SegNET for working on remote sensing images using deep learning (no activity since 2019)
  • RoboSat -> semantic segmentation on aerial and satellite imagery. Extracts features such as: buildings, parking lots, roads, water, clouds (no longer maintained)
  • DeepOSM -> Train a deep learning net with OpenStreetMap features and satellite imagery (no activity since 2017)
  • mapwith.ai -> AI assisted mapping of roads with OpenStreetMap. Part of Open-Mapping-At-Facebook
  • SAHI -> Python library for slicing image datasets, performing sliced inference with MMDetection, Detectron2, YOLOv5, Torchvision detectors and generating error analysis plots. Read the arxiv paper and article SAHI: A vision library for large-scale object detection & instance segmentation
  • terragpu -> Python library to process and classify remote sensing imagery by means of GPUs and AI/ML
  • EOTorchLoader -> Pytorch dataloader and pytorch lightning datamodule for Earth Observation imagery
  • satellighte -> an image classification library that consist state-of-the-art deep learning methods, using PyTorch Lightning
  • aeronetlib -> Python library to work with geospatial raster and vector data for deep learning
  • rsi-semantic-segmentation -> A unified PyTorch framework for semantic segmentation from remote sensing imagery
  • AiTLAS -> implements state-of-the-art AI methods for exploratory and predictive analysis of satellite images
  • mmsegmentation -> Semantic Segmentation Toolbox with support for many remote sensing datasets including LoveDA , Potsdam, Vaihingen & iSAID
  • ODEON landcover -> a set of command-line tools performing semantic segmentation on remote sensing images (aerial and/or satellite) with as many layers as you wish
  • aitlas-arena -> An open-source benchmark framework for evaluating state-of-the-art deep learning approaches for image classification in Earth Observation (EO)
  • PaddleRS -> remote sensing image processing development kit
  • RocketML Deep Neural Networks -> read Satellite Image Classification using rmldnn and Sentinel 2 data
  • segmentation_gym -> A neural gym for training deep learning models to carry out geoscientific image segmentation
  • EarthNets -> includes a database of 400 baseline models, and tutorial examples of common deep learning tasks on satellite imagery. Accompanied by a paper: EarthNets: Empowering AI in Earth Observation
  • raster4ml -> A geospatial raster processing library for machine learning

Neural nets in space

Processing on board a satellite allows less data to be downlinked. e.g. super-resolution image might take 8 images to generate, then a single image is downlinked. Other applications include cloud detection and collision avoidance.

Courses

Deploying models

See Deployment

Software

See Software

Datasets

See Datasets

Archive

For archived material see Archive


About

Deep learning with satellite & aerial imagery

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published