Document TorchGeo alternatives

[adamjstewart]

This PR adds a list of alternatives to TorchGeo that users can explore. You can preview the docs here.

@calebrob6 any idea how to make tables wider or scrollable? It might be a limitation of our current Sphinx theme or it might be something we can override. Either way, I want to look into replacing our current sphinx theme with something that is properly maintained.

[adamjstewart]

Hello maintainers of the following libraries:

• eo-learn: @AleksMat @zigaLuksic
• Raster Vision: @lewfish @AdeelH
• PaddleRS: @Bobholamovic @geoyee
• DeepForest: @bw4sz @ethanwhite
• SITS: @rolfsimoes @OldLipe @gilbertocamara
• segment-geospatial: @giswqs
• GeoTorchAI: @kanchanchy
• Moonshine: @nharada1

I'm doing a related works survey and wanted to document basic stats on features supported by other geospatial ML libraries. I included all of your libraries in my list due to their popularity. I'm hoping this will make it easier for us to advertise other libraries and help users decide which library is right for them.

If you're willing, could you check the above preview and make sure I didn't get anything wrong? I'm especially not sure about which libraries support exactly which features. I'll merge this PR in about a week, but if anything ever changes, feel free to submit a PR to update this table, and feel free to share this table in your own documentation!

P.S. I couldn't get all of the PaddleRS or GeoTorchAI tests to pass no matter what I tried, so if you can tell me the test coverage when all tests pass, I can update that.

[AdeelH]

Hey, thank you for doing this! Regarding Raster Vision, the only change I would suggest is to add pillow to the "I/O Backend" column, since Raster Vision also supports pre-chipped datasets.

[robmarkcole]

Worth also considering:

• https://github.com/Doodleverse/segmentation_gym
• https://github.com/open-mmlab/mmsegmentation -> used by https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M

I'm sure you are also aware of the longer list I maintain at https://github.com/satellite-image-deep-learning/software#deep-learning-projects--frameworks but as a word of caution, keeping these lists fresh is quite time consuming!

[adamjstewart]

Yes, I definitely agree it can be time consuming. I started from:

• https://github.com/satellite-image-deep-learning/software#deep-learning-projects--frameworks
• https://github.com/sacridini/Awesome-Geospatial#deep-learning
• https://github.com/topics/earth-observation
• https://github.com/topics/satellite-imagery
• https://github.com/topics/geospatial

then removed all projects that:

• are abandoned (no commits in the last year): removed most things from your list
• have no stable releases on PyPI/CRAN: removed all the research code
• are not yet popular (less than 100 stars): removed Segmentation Gym
• are not specific to geospatial ML: removed MMSegmentation

I'm hoping that this shorter, more focused list will be easier to maintain. I'm planning to keep TorchGeo up-to-date as frequently as I can, but will update the rest of the list less frequently (a couple times per year). The GitHub/Download stuff is easy to scrape, the rest was much harder to determine.

[gilbertocamara]

@adamjstewart
Many thanks for reaching out to fellow developers of Torch-related geospatial packages.

I will provide feedback regarding sits. I assume the concept of "spatial backend" means packages which provide the basis for the spatial operations, and that "transform backend" means packages that perform data transformation.

As for the datasets, please consider that sits comes with an online book (https://e-sensing.github.io/sitsbook/) which uses two additional data packages (https://github.com/e-sensing/sitsdata and https://github.com/e-sensing/rondonia20LMR). Together, these data packages contain 22 datasets with 12 GB. Many of these data sets are relatively large image collections that cannot be accepted by CRAN.

Please also note that the examples of the on-line book are on Kaggle (https://www.kaggle.com/esensing/code).

As for the models, it depends on how you count. The online book provides many examples. If you count each chapter of the book as providing a model, we would have 11 models.

As for CLI, sits runs on GUIs such as Rstudio and R Jupyter Notebooks. sits has an API which tries to be concise and high-level. A brief description of the API is available in Table 1 of the book's introductory chapter (https://e-sensing.github.io/sitsbook/introduction.html). Please also look at Code example 3 on the sits paper on Remote Sensing( https://www.mdpi.com/2072-4292/13/13/2428); it shows how to classify the Cerrado biome in Brazil (200 million ha) with jsix R commands.

Here are some suggested changes regarding sits:

(a) spatial backend: terra, sf, stars, gdalcubes, rstac;
(b) transform backend: tidyverse (which includes dplyr, purrr, tidyr and lubridate).
(c) datasets: 22
(d) models: 11
(e) CLI: yes

Again, many thanks for reaching out and congratulations on your effort and you respect and concern for the work of the community.

Best
Gilberto

[adamjstewart]

Thanks @gilbertocamara! Here are my thoughts:

(a): spatial backend: this is intended to be the thing that is used as a database for spatial joins and intersection checking. In the Python ecosystem, R-tree is an example of this. GDAL handles file I/O and warping, but isn't really a spatial database. Not sure if that distinction makes sense or is just arbitrary. Do any of those R dependencies fit this description? I'm happy to add some or all of these.
(b): transform backend: yes, your understanding is correct, will change.
(c): datasets: here, datasets is used in the PyTorch sense, so it's more like the thing that loads the data, not the data itself. I'm assuming all of these datasets can be used easily in sits without having to write your own data loader? If so, I'm happy to count these.
(d): models: here we mean pre-trained model weights, not model architectures. For example, TorchGeo sits on top of timm, so technically we support ~1K model architectures, but we only have ~40 pre-trained weights. Also, these are pre-trained on multispectral satellite imagery, ImageNet doesn't count.
(e): CLI: if I'm understanding correctly, you have a high-level API (similar to Keras) but you can't use sits without writing any code. A CLI would be a program that can be run on the command line with a simple config file and that's it. LightningCLI is what we use in TorchGeo.

So basically, I think I agree with your proposed changes for a-c, but not d-e.

[gilbertocamara]

Hi @adamjstewart

Points taken. I understand your reasoning for points d-e.

One also needs to consider the target user community of each package. Most users of geospatial data take Google Earth Engine as their reference. GEE's API provides a benchmark for all implementations of ML methods on Earth observation data. Users love GEE's API. That is why we designed sits to achieve the same tasks as GEE with a similar number of API calls.

Many good packages require their users to have programming experience. This leads to examples that use many more commands than GEE to achieve the same tasks. Xarray-based software such as Open Data Cube or Pangeo is a case in point.

In terms of usability to real users (i.e. non-programmers), a good measure would be to count the number of API calls required to perform a task in GEE and those required by other packages. In this respect, consider including a further item in your comparison chart. Such an item could be called "GEE-style", which would measure on a 1..10 scale how close to GEE will be the user experience when using the software.

Best
Gilberto

[adamjstewart]

I think GEE-style would end up being very subjective. TorchGeo is modeled after torchvision, so we would also need a torchvision-style scale for users familiar with that API. I'd rather not get into API design debates, so I stuck with what features they do or do not support. All APIs are valid, as long as they are intuitive and useful for large groups of users. We could possibly have an "API style" field with values like torchvision, GEE, etc., but I don't know whether or not all packages on the list are modeled after another API.

[gilbertocamara]

Hi @adamjstewart
Ok, let's leave it there.

We have different mileages and constituencies, and the field is big enough to have space for everyone. I have one question regarding torchgeo. The package provides many relevant models (i.e., FCN, ResNet). However, I did not find DL models explicitly designed for image time series (e.g., Temporal CNN, Temporal Attention Encoder). Could you please provide me with more information on how torchgeo deals with image time series?

[adamjstewart]

Time-series modeling in TorchGeo is still admittedly immature. We almost have three categories for time-series support:

✅ full support: Raster Vision, SITS
🚧 partial support: TorchGeo, eo-learn, GeoTorchAI
❌ no support: PaddleRS, DeepForest, segment-geospatial, Moonshine

By partial support I mean that these libraries explicitly model the time dimension in their data cubes and offer ways to sample based on time or perform change detection but otherwise have no builtin models for time-series analysis.

Do you think this is a fair evaluation? I can update the table with this split.

[adamjstewart]

P.S. Which of terra, sf, stars, gdalcubes, rstac would you describe as the spatial database you use?

[gilbertocamara]

In sits, image data cubes are stored as GeoTIFF files, and the metadata for them is available in R data frames, which are managed by the tidyverse functions.

[gilbertocamara]

Hi @adamjstewart As for the time series support, I welcome your suggestion of updating the table.

[adamjstewart]

How is the metadata loaded? If you want to find files that have spatial or temporal overlap, how do you do that? Excuse my ignorance of the entire R ecosystem, I've never used it before.

[AdeelH]

By partial support I mean that these libraries explicitly model the time dimension in their data cubes and offer ways to sample based on time or perform change detection but otherwise have no builtin models for time-series analysis.

Under that rubric, Raster Vision should qualify as "partial support" as well.

[gilbertocamara]

Considering your question:

If you want to find files that have spatial or temporal overlap, how do you do that?

We don't. We assume users know which collections are available in each major cloud provider. The package is designed to extract image collections from cloud repositories (e.g., AWS, Microsoft Planetary Computer, Digital Earth Africa) accessing the content of those collections with STAC. The chosen image collection can be regularized for ML analysis.

In simple terms, sits aims to be a software package with a GEE-like API that does what GEE does not allow users to do, specifically in land use and land cover classification of image time series.

The sits developers have a clear mission: help to preserve Brazil's endangered ecosystems, especially Amazonia and Cerrado biomes, support sustainable agriculture production, and provide high-quality LUCC estimates for compliance with market controls. Examples of such controls are the EU deforestation-free goods regulation and the carbon market agreements enabled by Article 6 of the Paris Agreement.

We are focused on our mission and on helping other users with similar aims, especially users in developing countries, which do not have the same programming expertise as Python experts in the North. Thus, sits has clear aims, which hopefully help to keep our design clean.

[adamjstewart]

Under that rubric, Raster Vision should qualify as "partial support" as well.

Raster Vision at least has a tutorial specifically for working with time-series data and a SeriesResNet model: https://docs.rastervision.io/en/latest/usage/tutorials/temporal.html. I think that's a step above the rest.

[adamjstewart]

@gilbertocamara in that case maybe SITS should be listed as having no spatial backend. I'm not sure about the other libraries either. I'm confident about TorchGeo and GeoTorchAI but less confident about the rest. I just picked dependencies that looked relevant for spatial intersection/union-related computations. I don't know too many other libraries that behave like TorchGeo and let you combine arbitrary datasets based on spatiotemporal metadata.

[gilbertocamara]

In that case maybe SITS should be listed as having no spatial backend.

Ok, now that I understand your points better, I agree. We do not rely on Xarray or some similar spatial data structure, which also do not exist in R. We do like GEE, by parallelizing the code in virtual machines along the multiple cores. Each core reads a chunk of the data cube and processes it. Thus, in that sense sits is its own backend.

Thanks for your patience.

[adamjstewart]

But GEE has the ability to combine multiple datasets based on spatial metadata. Does SITS offer that or does it require a user to do that themselves? Or does it query cloud repos and request data from a certain region/CRS, so the cloud repos themselves are the spatial backend?

[gilbertocamara]

But GEE has the ability to combine multiple datasets based on spatial metadata. Does SITS offer that or does it require a user to do that themselves? Or does it query cloud repos and request data from a certain region/CRS, so the cloud repos themselves are the spatial backend?

Not exactly. the combination provided by GEE is based on some sort of temporal averaging. The final image which is used for processing in GEE is a multispectral band image. GEE's combination usually results in an information loss.

[gilbertocamara]

Image time series classification is done on a time-first, space-later basis. Each pixel of a data cube is associated to a time series. Methods such LTAE classify each time series. After the classification, then we apply a spatial smoothing filter based on Bayesian statistics. That's not how GEE works.

[adamjstewart]

But how is your data cube constructed? How do you, for example, combine Landsat and CDL in such a way that each pixel of Landsat lines up with each pixel of CDL geospatially?

[gilbertocamara]

The data cube starts from an image collection. Take an MGRS tile used in Sentinel-2 ARD data. Consider, for example, a one-year period. In this period, there are typically anywhere from 50 to 100 S2 images (Sentinel-2 and Sentinel-2A). Then, select a reference period (say, 16 days). Within each 16 day period, S2 images are combined to get the best cloud-free composite. This produces a regular data-cube.

Areas with persistent cloud cover are flagged as missing data. When the time series is to be classified, such missing data is estimated by temporal interpolation.

[gilbertocamara]

We do not combine Landsat and Sentinel ourselves. We use NASA's HLS product.

[bw4sz]

Hi all,

Speaking for DeepForest, this table seems fair. I want to underline that i'd like DeepForest to play nicely with torchgeo models in the future (following https://torchgeo.readthedocs.io/en/stable/api/trainers.html#torchgeo.trainers.ObjectDetectionTask). I am retraining the tree backbone now. Both packages use pytorch lightning as a wrapper. DeepForest aims and user base seems to be much more focused on drone UAV imagery and less on satellite, but i'd like to compare the utility of pre-trained weights on transfer learning among resolutions. Stay tuned for that. Thanks everyone for their great open source work, i'm all ears on how to better integrate efforts.

Best,

Ben

[Bobholamovic]

Thanks, @adamjstewart ! Regarding PaddleRS, I would suggest the following changes:

Features:

• I/O backend: The I/O backends are GDAL and OpenCV.
• Datasets: PaddleRS has 6 dataset classes, each for a specific remote sensing task.
• Models: PaddleRS provides 35 models in total, 22 of which support multispectral input and 15 of which are specially designed for RS tasks.
• CLI: PaddleRS provides a set of command-line tools for image pre-processing, model deployment, etc.
• Time-series: PaddleRS partially supports time-series modeling. Specifically, PaddleRS supports change detection with exactly two temporal images.

GitHub:

• The test coverage was 48%.

[adamjstewart]

Thanks @Bobholamovic! My thoughts:

• I/O backend: 👍
• Datasets: I'm only counting data loaders for specific datasets, not base classes. If I counted all dataset classes in TorchGeo, we would actually have 116 datasets.
• Models: I'm only counting pre-trained model weights, not model architectures. If I counted all model architectures supported by TorchGeo, we would actually have ~1K models. I'll change the name to "Weights" to make this more clear.
• CLI: I tried pip install paddlers but I don't see any command-line tools. Where do these scripts get installed to?
• Time-series: 👍
• Test coverage: 👍

[Bobholamovic]

Thanks @Bobholamovic! My thoughts:

• I/O backend: 👍
• Datasets: I'm only counting data loaders for specific datasets, not base classes. If I counted all dataset classes in TorchGeo, we would actually have 116 datasets.
• Models: I'm only counting pre-trained model weights, not model architectures. If I counted all model architectures supported by TorchGeo, we would actually have ~1K models. I'll change the name to "Weights" to make this more clear.
• CLI: I tried pip install paddlers but I don't see any command-line tools. Where do these scripts get installed to?
• Time-series: 👍
• Test coverage: 👍

• Datasets: The dataset handling in PaddleRS follows a different procedure. First, a script is used to pre-process the original dataset into a standard format (e.g. prepare_dota.py). Then, a data loader is used to load the data in the standard format during training. Hence, PaddleRS does not provide a data loading interface for each specific dataset (in this sense, there are only base classes), but only provides data pre-processing scripts. If we are couting the programming interfaces for specific datasets, there is 0 in PaddleRS. But if we count the specific datasets supported by PaddleRS, I think there are 7, which is the total number of the data pre-processing scripts, as can be found here.

• Models: The new name "Weights" is more clear. As for model weights, PaddleRS provides 14 weight files pre-trained on remote sensing datasets (LEVIR-CD and iSAID), which can be found here.

  I recommend also taking into account the model architectures dedicated to remote sensing, because the architecture has an important impact on the result, too. By "dedicated to remote sensing" I mean that the model should be originally designed to solve problems in the remote sensing field. These models include FarSeg, ChangeStar, etc. They are born for remote sensing tasks and therefore distinguished from general computer vision models like ResNet and ViT.

• CLI: PaddleRS does not set command-line entry points during installation. This is because PaddleRS is a toolkit, which consists of both a Python library and a set of out-of-the-box tools. Command-line tools included in the repository are ready to be used after installing the core library (e.g. data processing tools and deployment tool). I'm not sure if these tools fall in the "CLI" scope here or only installed entry points are counted.

[adamjstewart]

• Datasets: Thanks for the explanation. I count 12 preprocessing scripts for specific datasets, so I'll update the table.
• Weights: Thanks for pointing me to this file, I'm not sure how I missed it. Will update with 14 pre-trained weights.
• Models: I'll consider adding a separate column for RS-specific model architectures in the future. I'll first need to compile a list of RS-specific models.
• CLI: I'll add those under "partial" 🚧 since the tools exist, they just aren't installed with the library.

[Bobholamovic]

• Datasets: Thanks for the explanation. I count 12 preprocessing scripts for specific datasets, so I'll update the table.
• Weights: Thanks for pointing me to this file, I'm not sure how I missed it. Will update with 14 pre-trained weights.
• Models: I'll consider adding a separate column for RS-specific model architectures in the future. I'll first need to compile a list of RS-specific models.
• CLI: I'll add those under "partial" 🚧 since the tools exist, they just aren't installed with the library.

Thanks for the updates! Actually, there are 12 pre-processing scripts, but 5 of them are empty (not implemented yet), so I guess the count should be 7.

[calebrob6]

Nicely done @adamjstewart! I unfortunately do not know how to make the tables look nice under our current theme (if anything it would be a nasty CSS hack).