Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.1.0 release #226

Merged
merged 27 commits into from
Nov 8, 2021
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ab41d56
0.1.0 release
adamjstewart Nov 7, 2021
363df27
Train deps needed for release testing
adamjstewart Nov 7, 2021
d510400
Update development status
adamjstewart Nov 7, 2021
0c4e6b5
setup.py should not be run directly
adamjstewart Nov 7, 2021
aa8b6b2
Test more trainers
adamjstewart Nov 7, 2021
4e708a5
Fix local docs build
adamjstewart Nov 7, 2021
1a9de99
Update installation instructions
adamjstewart Nov 7, 2021
e22cf22
Specify test data dir in config
adamjstewart Nov 7, 2021
f436f04
Fix tutorial docs
adamjstewart Nov 7, 2021
e9b8dee
Trainers should default to num_workers=0, download=False
adamjstewart Nov 7, 2021
0f0687d
Correct location for root_dir
adamjstewart Nov 7, 2021
ad0439d
Try different GDAL name
adamjstewart Nov 7, 2021
046f69a
Try again
adamjstewart Nov 7, 2021
8cc4d5e
Various fixes to release tests
adamjstewart Nov 7, 2021
49705cc
Update pip installs in tutorials
adamjstewart Nov 7, 2021
8ce9b60
Fix some bugs
adamjstewart Nov 7, 2021
5fae8c0
Config file not being picked up
adamjstewart Nov 7, 2021
3c26048
Get back to 100% test coverage
adamjstewart Nov 7, 2021
43924f2
Added correct weight string to UCMerced
calebrob6 Nov 8, 2021
b222b5b
yolo fix
calebrob6 Nov 8, 2021
cb90dfc
yolo fix pt 2
calebrob6 Nov 8, 2021
8d7f02a
yolo fix 2 pt. 1
calebrob6 Nov 8, 2021
ea219cc
Simplify tests a bit
adamjstewart Nov 8, 2021
2b4f60d
Make the trainer notebook look stupid
calebrob6 Nov 8, 2021
8a990c7
UCMerced should download by default in the trainers
calebrob6 Nov 8, 2021
881dc9b
Revert
calebrob6 Nov 8, 2021
162b6bf
Fix logo/author, include LICENSE in upload
adamjstewart Nov 8, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,16 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install apt dependencies
run: |
sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable
sudo apt-get update
sudo apt-get install gdal-bin libgdal-dev
- name: Install pip dependencies
run: pip install .[tests]
run: |
pip install gdal tqdm # TODO: these deps shouldn't be needed
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isaaccorley the indices tutorial imports things that aren't TorchGeo deps, does it need to?

Copy link
Collaborator

@isaaccorley isaaccorley Nov 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revamp them but requiring that notebooks only use torchgeo deps might be too restrictive. E.g we may want to use visualization libraries in a tutorial but not add them as a dependency to torchgeo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code definitely uses these two libraries.

  • tqdm is simple to include as a dependency, it will be indirectly included anyway, and is quite useful in notebooks and command line scripts
  • The gdal bit can be replaced quite easily with rasterio

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, let's do this for the next release

pip install .[datasets,tests,train]
pip install -r docs/requirements.txt
- name: Run notebook checks
run: pytest --nbmake docs/tutorials
integration:
Expand All @@ -32,6 +40,6 @@ jobs:
with:
python-version: 3.9
- name: Install pip dependencies
run: pip install .[tests]
run: pip install .[datasets,tests,train]
- name: Run integration checks
run: pytest -m slow
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Tests:
The recommended way to install TorchGeo is with [pip](https://pip.pypa.io/):

```console
$ pip install git+https://github.com/microsoft/torchgeo.git
$ pip install torchgeo
```

For [conda](https://docs.conda.io/) and [spack](https://spack.io/) installation instructions, see the [documentation](https://torchgeo.readthedocs.io/en/latest/user/installation.html).
Expand Down
8 changes: 4 additions & 4 deletions conf/task_defaults/bigearthnet.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ experiment:
in_channels: 14
adamjstewart marked this conversation as resolved.
Show resolved Hide resolved
num_classes: 19
datamodule:
num_classes: 19
batch_size: 128
num_workers: 6
root_dir: "tests/data/bigearthnet"
bands: "all"
num_classes: 19
num_classes: ${experiment.module.num_classes}
batch_size: 128
num_workers: 0
9 changes: 5 additions & 4 deletions conf/task_defaults/byol.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,12 @@ experiment:
learning_rate: 1e-3
learning_rate_schedule_patience: 6
datamodule:
batch_size: 64
num_workers: 6
root_dir: "tests/data/chesapeake/cvpr"
train_splits:
- "de-train"
- "de-test"
val_splits:
- "de-val"
- "de-test"
test_splits:
- "de-test"
batch_size: 64
num_workers: 0
12 changes: 9 additions & 3 deletions conf/task_defaults/chesapeake_cvpr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,15 @@ experiment:
num_classes: 7
num_filters: 256
datamodule:
train_state: "de"
root_dir: "tests/data/chesapeake/cvpr"
train_splits:
- "de-test"
val_splits:
- "de-test"
test_splits:
- "de-test"
patches_per_tile: 200
patch_size: 256
batch_size: 64
num_workers: 4
num_classes: ${experiment.module.num_classes}
num_workers: 0
class_set: ${experiment.module.num_classes}
3 changes: 2 additions & 1 deletion conf/task_defaults/cowc_counting.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@ experiment:
learning_rate: 1e-3
learning_rate_schedule_patience: 2
datamodule:
root_dir: "tests/data/cowc_counting"
batch_size: 32
num_workers: 4
num_workers: 0
3 changes: 2 additions & 1 deletion conf/task_defaults/cyclone.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@ experiment:
learning_rate: 1e-3
learning_rate_schedule_patience: 2
datamodule:
root_dir: "tests/data/cyclone"
batch_size: 32
num_workers: 4
num_workers: 0
3 changes: 2 additions & 1 deletion conf/task_defaults/landcoverai.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@ experiment:
num_classes: 6
num_filters: 256
datamodule:
root_dir: "tests/data/landcoverai"
batch_size: 32
num_workers: 4
num_workers: 0
4 changes: 3 additions & 1 deletion conf/task_defaults/naipchesapeake.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,7 @@ experiment:
num_classes: 13
num_filters: 64
datamodule:
naip_root_dir: "tests/data/naip"
chesapeake_root_dir: "tests/data/chesapeake/BAYWIDE"
batch_size: 32
num_workers: 4
num_workers: 0
7 changes: 2 additions & 5 deletions conf/task_defaults/resisc45.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,6 @@ experiment:
in_channels: 3
num_classes: 45
datamodule:
root_dir: "tests/data/resisc45"
batch_size: 128
num_workers: 6
weights: ${experiment.module.weights}
unsupervised_mode: false
val_split_pct: 0.2
test_split_pct: 0.2
num_workers: 0
3 changes: 2 additions & 1 deletion conf/task_defaults/sen12ms.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@ experiment:
in_channels: 15
num_classes: 11
datamodule:
root_dir: "tests/data/sen12ms"
batch_size: 32
num_workers: 4
num_workers: 0
3 changes: 2 additions & 1 deletion conf/task_defaults/so2sat.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ experiment:
in_channels: 3
num_classes: 17
datamodule:
root_dir: "tests/data/so2sat"
batch_size: 128
num_workers: 6
num_workers: 0
bands: "rgb"
6 changes: 2 additions & 4 deletions conf/task_defaults/ucmerced.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ experiment:
in_channels: 3
num_classes: 21
datamodule:
root_dir: "tests/data/ucmerced"
batch_size: 128
num_workers: 6
unsupervised_mode: false
val_split_pct: 0.1
test_split_pct: 0.1
num_workers: 0
2 changes: 1 addition & 1 deletion docs/tutorials/benchmarking.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install git+https://github.com/microsoft/torchgeo.git"
"%pip install torchgeo"
]
},
{
Expand Down
2 changes: 1 addition & 1 deletion docs/tutorials/getting_started.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install git+https://github.com/microsoft/torchgeo.git"
"%pip install torchgeo"
]
},
{
Expand Down
4 changes: 1 addition & 3 deletions docs/tutorials/indices.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.11"
}
},
Expand Down Expand Up @@ -86,7 +85,7 @@
"id": "wOwsb8KT_uXR"
},
"source": [
"%pip install git+https://github.com/microsoft/torchgeo.git"
"%pip install torchgeo"
],
"execution_count": null,
"outputs": []
Expand Down Expand Up @@ -117,7 +116,6 @@
"import rasterio\n",
"import rasterio.features\n",
"import shapely\n",
"import tifffile\n",
"import torch\n",
"import torch.nn as nn\n",
"import torchvision.transforms as T\n",
Expand Down
3 changes: 1 addition & 2 deletions docs/tutorials/trainers.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"metadata": {},
"outputs": [],
"source": [
"%pip install git+https://github.com/microsoft/torchgeo.git"
"%pip install torchgeo"
]
},
{
Expand Down Expand Up @@ -226,7 +226,6 @@
],
"source": [
"trainer = pl.Trainer(\n",
" gpus=1,\n",
" callbacks=[checkpoint_callback, early_stopping_callback],\n",
" logger=[csv_logger],\n",
" default_root_dir=experiment_dir,\n",
Expand Down
7 changes: 3 additions & 4 deletions docs/tutorials/transforms.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.11"
},
"accelerator": "GPU"
Expand Down Expand Up @@ -84,7 +83,7 @@
"id": "wOwsb8KT_uXR"
},
"source": [
"%pip install git+https://github.com/microsoft/torchgeo.git"
"%pip install torchgeo"
],
"execution_count": null,
"outputs": []
Expand Down Expand Up @@ -538,7 +537,7 @@
"outputId": "fa0443da-8b4d-47f7-e713-93e1e4976e87"
},
"source": [
"!nvidia-smi"
"%nvidia-smi"
],
"execution_count": 8,
"outputs": [
Expand Down Expand Up @@ -724,4 +723,4 @@
"outputs": []
}
]
}
}
38 changes: 9 additions & 29 deletions docs/user/installation.rst
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
Installation
============

TorchGeo is simple and easy to install. We support installation using the `pip <https://pip.pypa.io/>`_, `conda <https://docs.conda.io/>`_, and `spack <https://spack.io/>`_ package managers, although you can also install from source if you want to.
TorchGeo is simple and easy to install. We support installation using the `pip <https://pip.pypa.io/>`_, `conda <https://docs.conda.io/>`_, and `spack <https://spack.io/>`_ package managers.

pip
---

..
Since TorchGeo is written in pure-Python, the easiest way to install it is using pip:
Since TorchGeo is written in pure-Python, the easiest way to install it is using pip:

.. code-block:: console
.. code-block:: console

$ pip install torchgeo
$ pip install torchgeo


If you want to install a development version, you can use a VCS project URL:
Expand All @@ -32,16 +31,10 @@ or a local git checkout:

By default, only required dependencies are installed. TorchGeo has a number of optional dependencies for specific datasets or development. These can be installed with a comma-separated list:

..
.. code-block:: console

$ pip install torchgeo[datasets]
$ pip install torchgeo[style,tests]

.. code-block:: console

$ pip install .[datasets]
$ pip install .[style,tests]
$ pip install torchgeo[datasets]
$ pip install torchgeo[style,tests]


See the ``setup.cfg`` for a complete list of options. See the `pip documentation <https://pip.pypa.io/>`_ for more details.
Expand All @@ -57,12 +50,11 @@ If you need to install non-Python dependencies like PyTorch, it's better to use
$ conda config --set channel_priority strict


..
Now, you can install the latest stable release using:
Now, you can install the latest stable release using:

.. code-block:: console
.. code-block:: console

$ conda install torchgeo
$ conda install torchgeo


Conda does not directly support installing development versions, but you can use conda to install our dependencies, then use pip to install TorchGeo itself.
Expand Down Expand Up @@ -110,15 +102,3 @@ Optional dependencies can be installed by enabling build variants:
Run ``spack info py-torchgeo`` for a complete list of variants.

See the `spack documentation <https://spack.readthedocs.io/>`_ for more details.

source
------

TorchGeo can also be installed from source using the ``setup.py`` file and setuptools.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direct invocation of setup.py is deprecated: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html


.. code-block:: console

$ git clone https://github.com/microsoft/torchgeo.git
$ cd torchgeo
$ python setup.py build
$ python setup.py install
4 changes: 2 additions & 2 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/microsoft/torchgeo
classifiers =
Development Status :: 1 - Planning
Development Status :: 3 - Alpha
Intended Audience :: Science/Research
Programming Language :: Python :: 3
Programming Language :: Python :: 3.6
Expand All @@ -20,7 +20,7 @@ classifiers =
Operating System :: OS Independent
Topic :: Scientific/Engineering :: Artificial Intelligence
Topic :: Scientific/Engineering :: GIS
keywords = pytorch, deep learning, machine learning
keywords = pytorch, deep learning, machine learning, remote sensing, satellite imagery, geospatial

[options]
setup_requires =
Expand Down
16 changes: 0 additions & 16 deletions tests/test_setup.py

This file was deleted.

20 changes: 17 additions & 3 deletions tests/test_train.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,20 +134,34 @@ def test_config_file(tmp_path: Path) -> None:
subprocess.run(args, check=True)


@pytest.mark.parametrize("task", ["cowc_counting", "cyclone", "sen12ms", "landcoverai"])
@pytest.mark.parametrize(
"task",
[
"bigearthnet",
"byol",
"chesapeake_cvpr",
"cowc_counting",
"cyclone",
"landcoverai",
"naipchesapeake",
"resisc45",
"sen12ms",
"so2sat",
"ucmerced",
],
)
def test_tasks(task: str, tmp_path: Path) -> None:
output_dir = tmp_path / "output"
data_dir = os.path.join("tests", "data", task)
log_dir = tmp_path / "logs"
args = [
sys.executable,
"train.py",
"experiment.name=test",
"program.output_dir=" + str(output_dir),
"program.data_dir=" + data_dir,
"program.log_dir=" + str(log_dir),
"trainer.fast_dev_run=1",
"experiment.task=" + task,
"program.overwrite=True",
"config_file=" + os.path.join("conf", "task_defaults", task + ".yaml"),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@calebrob6 without this line, the default configs aren't being loaded, so there might be something broken with default loading.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default config is loaded by name here -- https://github.com/microsoft/torchgeo/blob/main/train.py#L97.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but if you comment out this line, you'll see that it isn't loading the default config.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It definitely is loading the default config, else it would throw an error about not finding the default config

Copy link
Member

@calebrob6 calebrob6 Nov 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the way configurations are merged, experiment.datamodule.root_dir was being overridden by the defaults in defaults.yaml (i.e. program.data_dir). I'm not sure it makes sense to have the test datasets be the default data for each dataset. The fixes here are:

  • do as you've done
  • remove the default mapping between program.data_dir and experiment.datamodule.root_dir in defaults.yaml
  • remove the default root_dirs that you added to all the datasets and pass those instead as arguments to program.data_dir
  • rewrite the configuration system :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I realize I've been abusing the config system by modifying it to suit the tests. What if we:

  1. Keep doing this until we get the release out since the conf directory won't get installed anyway
  2. After the release is out, create a tests/conf directory for test-specific configuration settings
  3. Think about Re-think how configs are handled in train.py #227 post-rebuttal when we have more time

We could also do 2 real quick, it wouldn't take that long.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make a TODO here or somewhere that we'll remember?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll create an issue so we don't re-trigger the tests

]
subprocess.run(args, check=True)
Loading