segment UAS multispectral imagery

[vonnonn]

I would like to segment imagery (~0.02 m) we collect from a MicaSense RedEdge-P sensor (B,G,R,RE,NIR) but I have some questions on preprocessing. Can I use a function like this to normalize the samples:

def normalize(image):
    return (image - image.min()) / (image.max() - image.min())

def preprocess(sample):
    if "image" in sample:
        sample["image"] = normalize(sample["image"]).float()
        #sample["image"] = (sample["image"] / 255.0).float()  # inputs are normalized to [0, 1]
        assert sample["image"].min() >= 0 and sample["image"].max() <= 1

    if "mask" in sample:
        sample["mask"] = sample["mask"].squeeze().long()
    del sample["bounds"]
    return sample

And then perform the same function with a band loop on the predicted imagery like this?

for b in range(image.shape[0]):
    image[b,:,:] = (image[b,:,:] - image[b,:,:].min()) / (image[b,:,:].max() - image[b,:,:].min())`

I would also like to use different indices than what are provided in the transform section, specifically, brightness and maxdiff:

BRI = (image[0,:,:] + image[1,:,:] + image[2,:,:]) + (image[3,:,:] + image[4,:,:]) / 5
MDIF = np.maximum.reduce([abs(image[0,:,:] - image[1,:,:]),
                          abs(image[0,:,:] - image[2,:,:]),
                          abs(image[1,:,:] - image[2,:,:]),
                          abs(image[0,:,:] - image[3,:,:]),
                          abs(image[0,:,:] - image[4,:,:]),
                          abs(image[1,:,:] - image[3,:,:]),
                          abs(image[1,:,:] - image[4,:,:]),
                          abs(image[2,:,:] - image[3,:,:]),
                          abs(image[2,:,:] - image[4,:,:]),
                          abs(image[3,:,:] - image[4,:,:])])  `

I was adding these to the image stack along with NDVI, NDRE, and MNDWI for a total of 10 bands (channels) and then writing out a new geotiff. Is there a better way to do this? I normalized all the bands then wrote out the tiff to be read in by torchgeo and commented out the normalizing functions, but I couldn't get any predictions from it.

What would be the optimal setting for weights, True or None, or could I train just the RGB with pretrained weights and then random weights for the remaining channels?

Looking forward to regularly implementing torchgeo with our workflow!

Cheers,
Josh

[isaaccorley]

Hi @vonnonn thanks for using TorchGeo! We have a band normalization function at torchgeo.datasets.utils.percentile.normalization which is the same as the min-max normalization when lower=0 and upper=100.

Regarding the indices, we do have some transforms for computing and appending indices in torchgeo.transforms.indices If you don't want to compute and save new files you can do it on the fly during training instead which is what I would recommend so that you can experiment with different indices.

For weights, I would always recommend weights=True and to set the number of channels for the total number of bands+indices you are using. We use timm in the backend which will repeat pretrained ImageNet RGB weights like RGBRGBR for example id you have 7 channels.

Hope this helps.

[vonnonn]

Thanks @isaaccorley for the prompt reply! Would it make sense, and would it be possible, to use the complimenting bands from a pretrained sentinel 2 model and then timm or random for the indices?

[adamjstewart]

Adding to what @isaaccorley has already said:

Can I use a function like this to normalize the samples

You can. But you shouldn't. In machine learning, we usually normalize all images in an entire dataset to the same dynamic range. Otherwise, an image with a small cloud becomes mostly dark and an image without any clouds will become very bright. My suggestion would be to calculate the mean and std dev of all of your images, then normalize all of them using the same values. You can use a transform like Normalize to do this instead of writing your own code.

I would also like to use different indices than what are provided in the transform section, specifically, brightness and maxdiff.

Your brightness calculation looks identical to our RandomGrayscale transform with the following settings:

transform = RandomGrayscale(p=1)

I've never seen your maxdiff transform before, so would have to think about that.

P.S. We don't yet have builtin transforms for MNDWI or MaxDiff or builtin data loaders for MicaSense RedEdge-P, but if you feel like adding these, please open a PR! MNDWI in particular would be quite easy, as it seems to be identical to NDSI (which we already have).

[vonnonn]

Thanks @adamjstewart! I'll have a look at what it may take to code a dataloader for the RE-P and MaxDiff. Meanwhile, is there a forum where I could find some help to improve my results? I've labeled a subset (3337x5296) with 4 classes and 150 masks but my results have been not great and It's not clear to me what parameters need optimizing.
We are working with a contractor who is getting some great results using ArcPro's deep learning, which uses pytorch in the backend, with a Mask R-CNN model and Resnet-50 backbone, but we need to re-tool it outside of ArcPro in order to run it on our HPC.

[adamjstewart]

is there a forum where I could find some help to improve my results?

If you join our Slack workspace (link in the README) there is a #help channel where you can ask for help.

I've labeled a subset (3337x5296) with 4 classes and 150 masks but my results have been lousy and It's not clear to me what parameters need optimizing.

For such a small area, you'll certainly want to start with a pre-trained model and fine-tune it on your application. We don't have any models pre-trained on the MicaSense RedEdge-P sensor, but you could start with DOFA which is able to adapt to any number of input channels.

[vonnonn]

Thanks again, @adamjstewart. I'll work on mustering the courage to post in the help channel.