Added custom semantic segmentation trainer tutorial

[calebrob6]

This is a tutorial notebook that shows users how to override a custom semantic segmentation class for training on LandCoverAI.

[calebrob6]

@adamjstewart I think there is a bug (or at least some weird behavior going on) here

I can create a new class that extends SemanticSegmentationTask:

class CustomSemanticSegmentationTask(SemanticSegmentationTask):

    # any keywords we add here between *args and **kwargs will be found in self.hparams
    def __init__(self, *args, tmax=50, eta_min=1e-6, **kwargs) -> None:
        super().__init__(*args, **kwargs)  # pass args and kwargs to the parent class

Then instantiate and everything works great

task = CustomSemanticSegmentationTask(model="unet", tmax=100, ...)

However when I go to load from file:

task = CustomSemanticSegmentationTask.load_from_checkpoint("lightning_logs/version_3/checkpoints/epoch=0-step=117.ckpt")

I get an error:

TypeError: SemanticSegmentationTask.__init__() got an unexpected keyword argument 'ignore'

I can add del kwargs["ignore"] before super().... in the constructor of CustomSemanticSegmentationTask but this feels like a bad hack.

(@isaaccorley in case you've seen this)

[robmarkcole]

Note that https://www.reviewnb.com/ is free for 'educational' use - would enable previewing the notebook

[calebrob6]

@robmarkcole thanks for the review! Was that easy enough to do (vs reviewnb)?

On my side, I just have to run jupytext --sync custom_segmentation_trainer.ipynb whenever I change either file and commit both (which seems simple enough). We could have CI that makes sure that all py and ipynb are in sync.

[robmarkcole]

Yep easy enough

[isaaccorley]

@adamjstewart I think there is a bug (or at least some weird behavior going on) here

I can create a new class that extends SemanticSegmentationTask:

class CustomSemanticSegmentationTask(SemanticSegmentationTask):



    # any keywords we add here between *args and **kwargs will be found in self.hparams

    def __init__(self, *args, tmax=50, eta_min=1e-6, **kwargs) -> None:

        super().__init__(*args, **kwargs)  # pass args and kwargs to the parent class

Then instantiate and everything works great

task = CustomSemanticSegmentationTask(model="unet", tmax=100, ...)

However when I go to load from file:

task = CustomSemanticSegmentationTask.load_from_checkpoint("lightning_logs/version_3/checkpoints/epoch=0-step=117.ckpt")

I get an error:

TypeError: SemanticSegmentationTask.__init__() got an unexpected keyword argument 'ignore'

I can add del kwargs["ignore"] before super().... in the constructor of CustomSemanticSegmentationTask but this feels like a bad hack.

(@isaaccorley in case you've seen this)

I don't see an ignore param in your custom class. Did you modify the class code you used after training that checkpoint? If so, one thing you can do is to just load the checkpoint, delete that param in the hparam dict and then save the checkpoint which would fix the error.

[calebrob6]

The ignore param is stored in task.hparams because SemanticSegmentationTask passes it to BaseTask, -- https://github.com/microsoft/torchgeo/blob/main/torchgeo/trainers/segmentation.py#L98.

I want to be able to do something like this:

class CustomSemanticSegmentationTask(SemanticSegmentationTask):
    def __init__(self, *args, my_custom_arg, **kwargs) -> None:
        super().__init__(*args, **kwargs)

i.e. use the constructor from SemanticSegmentationTask and not have to copy paste the args and logic from SemanticSegmentationTask.

This works fine but, when I try to load a version of this class from checkpoint ignore is passed through to SemanticSegmentationTask as it is a kwarg saved in hparams.

[calebrob6]

One workaround that I'm checking is just adding "ignore" to the list of args ignored in save_hyperparameters in BaseTask.

[isaaccorley]

One workaround that I'm checking is just adding "ignore" to the list of args ignored in save_hyperparameters in BaseTask.

I think this makes the most sense since it's not an actual hparam.

[calebrob6]

The way that I've come up with to check whether an .ipynb is in sync with the corresponding .py is diff <(jupytext --output - --to py custom_segmentation_trainer.ipynb) custom_segmentation_trainer.py. This should only output something like:

3a4
> #     formats: ipynb,py

[calebrob6]

I don't know why notebook test is being cancelled (maybe because it is trying to run the LandCoverAI split script?).

[adamjstewart]

I don't know why notebook test is being cancelled (maybe because it is trying to run the LandCoverAI split script?).

Tests being canceled means the job either ran out of time, space, or memory. Here, my guess would be space. We want to use the smallest datasets possible, such as EuroSAT100. Might be worth creating a LandCoverAI100 or something like that.

[adamjstewart]

Haven't yet had time to evaluate jupytext to decide whether or not it's what we should use. @nilsleh what did you end up using for lightning-uq-box?

[calebrob6]

I really don't like the idea of making custom datasets/datamodules just to have pretty CI -- it is a large overhead for something that makes the tutorial less cool.

[adamjstewart]

And I really don't like having tutorials that take 30+ minutes to download a dataset and train a model for hundreds of epochs, or tutorials that can't be tested in CI because they involve more data than our runners can store. There's always a tradeoff. You can also find a smaller dataset instead of making your own.

[calebrob6]

Luckily none of that happens here ;). LandCover.ai is 1.5GB (this will take 30+ minutes to download if your download speed is < 0.83 MB/s), training happens for 1 batch (and I can reduce the batch size to make this faster).

I'm saying that we shouldn't be catching bugs with overly sanitized examples -- if LandCoverAI breaks or Lightning training breaks then our other tests will break. If the example notebooks are catching bugs then we should ask ourselves why. Downloading LandCoverAI and running this now and per release doesn't seem to be a big burden.

[calebrob6]

How about-- is it possible to change LandCoverAI datamodule to use the test data that we already have spent time creating for this notebook (then comments saying if you actually want to play with data, then do this other thing)?

[nilsleh]

Haven't yet had time to evaluate jupytext to decide whether or not it's what we should use. @nilsleh what did you end up using for lightning-uq-box?

So I tried jupytext, but for my tutorials I couldn't get the jupytext scripts to execute and be displayed on the documentation. I went back to notebooks, and the notebooks are now run when the documentation builds. However, I don't need to download any data and model fitting is fast, since it's just toy problems.

[adamjstewart]

How about-- is it possible to change LandCoverAI datamodule to use the test data that we already have spent time creating for this notebook (then comments saying if you actually want to play with data, then do this other thing)?

This is also undesirable because the notebook by default will train and display predictions on random noise. I would much rather have a tiny dataset with real data. I'm happy to make this myself (maybe next week).

[calebrob6]

I don't think it needs to display predictions -- as if we're only training for a batch for "making CI pretty" reasons then it will display noise regardless.

[adamjstewart]

We can train for multiple epochs in the tutorial, but use fast_dev_run in CI. The trainers tutorial does this with nbmake variable mocking. This also means we could use the synthetic dataset during CI and the real dataset during the tutorial. My only hope is that we don't train for longer than it takes to explain what we're doing in a live tutorial session. If we use this notebook in a tutorial and end up sitting there watching it awkwardly for an hour then it gets tedious.

[isaaccorley]

Lgtm

[adamjstewart]

Is the plan to delete these commented out files or uncomment them?

[adamjstewart]

Same here

[adamjstewart]

I think we can remove this

[adamjstewart]

At some point we transitioned from markdown to code comments. I would prefer markdown.

[adamjstewart]

I usually strip outputs from the notebooks before uploading them so the outputs don't change every time someone submits a PR. I know you prefer to include the output, and I'm trying to find ways to make both of us happy. Ideally, I want to store only .py files and generate the .ipynb on-the-fly but haven't figured out a way to do that yet.

[calebrob6]

TorchVision, Kornia, Sklearn, Lightning, etc. all have notebook output. If you don't show the notebook output then you should just have .py files?

https://pytorch.org/vision/stable/auto_examples/transforms/plot_transforms_getting_started.html#sphx-glr-auto-examples-transforms-plot-transforms-getting-started-py

https://kornia.github.io/tutorials/nbs/image_matching_lightglue.html

[adamjstewart]

I know this is like the whole purpose of #2317, but also completely unnecessary to demonstrate a custom trainer. If you remove this, we could backport this to 0.6.1 so it makes it into the stable docs immediately.

[calebrob6]

I'm not in a rush to get this into the stable docs, and think it is a nice use case.

[adamjstewart]

Output metrics are pretty bad. At least with our EuroSAT100 examples, the accuracies aren't horrible (much higher than random guessing). Maybe if we start with a pre-trained model (even one created solely for the purpose of this tutorial) we can get better results? Would also be nice to include example plots at the end to visualize how well the model works. Again, I know this is something you wish we didn't have to do and we could instead train for minutes/hours, but we want to keep the tutorial simple and easy to walkthrough.

[calebrob6]

A few things:

• Integrating pre-trained models is outside the scope of this tutorial. If we get to a point where we post benchmark models for different datasets then we can revisit.
• Performance on this task is close to meaningless -- it is semantic segmentation over 15 samples training for a single look on 70 samples. It may be the 15 random samples in the test set are all of the same class.
• It'd be cute to maximize performance under the constraints of CPU only Github Action CI runners but I don't think it adds any value to the tutorial and eats up CI time.
• It would be nice to have pretty plots, but again, you are very unlikely to get anything coherent from training for a single 70 sample epoch. Also, plotting should be covered by other tutorials.

[calebrob6]

The point is to show people how to build custom classes that extend our base trainer classes and why they would do that (to have different metrics, to change optimizer schedule, ....).

[calebrob6]

@adamjstewart is this good to go?

[adamjstewart]

Need to give this another detailed review, but there are a lot of lines of code that are commented out that we should clean up. Also, this PR depends on multiple new features that won't exist until 0.7.0/1.0.0, so we probably can't use it for AGU (unless we backport some of the changes without advertising them).