Training issue - multiple image pairs

[quizz0n]

Hi @remicres,

After a successful run a while ago (only one pair of images: Sentinel-2/Spot-6 - 1276 patches), I wanted to train a new model with 4 additional pairs of images above different cities. So in total 5 image pairs - 2208 total number of patches (1276+345+158+264+165 patches).

I fallowed the same steps when preparing the patches images, however the training with 5 image pairs is not successful (incomplete parts in the final image).

I also did a run with only 2 image pairs, first image pair used in the first run a while ago + a random image pair to check if maybe there's an issue with one of the pairs. It looks to be the same just not as spread, probably because of the smaller quantity of additional patches.

Another test that I did was to apply the initial successful model to all the new Sentinel-2 images used for the new patches of the 5 image pairs. I thought I can check by this if there's something wrong with the dynamic of the new images or anything else. All of the resulting images were good.

TensorBoard:

• successful run with only one pair of images (a while ago):

• unsuccessful run with 5 pairs of images / unsuccessful run with 2 pairs of images:

python sr4rs/code/train.py \
--lr_patches 4/patch_extraction/lr_patches.tif 4/patch_extraction/lr_b_patches.tif 4/patch_extraction/lr_d_patches.tif 4/patch_extraction/lr_s2_patches.tif 4/patch_extraction/lr_s_patches.tif \
--hr_patches 4/patch_extraction/hr_patches.tif 4/patch_extraction/hr_b_patches.tif 4/patch_extraction/hr_d_patches.tif 4/patch_extraction/hr_s2_patches.tif 4/patch_extraction/hr_s_patches.tif \
--preview 4/images/lr_subset.tif \
--vggfile sr4rs/code/vgg19.npy \
--save_ckpt sr4rs/code/ckpts/ \
--logdir sr4rs/code/results/logs/ \
--lr_scale 0.0001 \
--hr_scale 0.0001 \
--epochs 150 \
--vggweight 0.000025 \
--l1weight 180 \
--l2weight 0.0 \
--depth 32 \
--vggfeatures 1234 \
--batchsize 4 \
--adam_lr 0.002 \
--savedmodel sr4rs/code/savedmodel_t4

python sr4rs/code/train.py \
--lr_patches 4/patch_extraction/lr_patches.tif 4/patch_extraction/lr_s2_patches.tif \
--hr_patches 4/patch_extraction/hr_patches.tif 4/patch_extraction/hr_s2_patches.tif \
--preview 4/images/lr_subset_old.tif \
--vggfile sr4rs/code/vgg19.npy \
--save_ckpt sr4rs/code/ckpts/ \
--logdir sr4rs/code/results/logs/ \
--lr_scale 0.0001 \
--hr_scale 0.0001 \
--epochs 150 \
--vggweight 0.000025 \
--l1weight 180 \
--l2weight 0.0 \
--depth 32 \
--vggfeatures 1234 \
--batchsize 4 \
--adam_lr 0.002 \
--savedmodel sr4rs/code/savedmodel_t5

Applying the model to a Sentinel-2 image:

• successful run with only one pair of images (a while ago):

• unsuccessful run with 5 pairs of images / unsuccessful run with 2 pairs of images:

I'm thinking maybe it's the way the files are read in the train command?
Thanks!

[remicres]

Hi @quizz0n

Nice images!

I suspect that this comes from the dynamic of the input HR images used to train the model.

Indeed, when you train a model from several (LR, HR) pairs, the most important thing - and difficult to achieve, is to preserve the LR --> HR radiometric transformation: it must be the same for all images. Else, the network might perform some "random" radiometric transform, because it has learnt multiple "plausible" mappings, but that do not transform the radiometry the same way. The dark spots impacting your images still looks "plausible" locally, because the model might have learnt multiple
radiometric transformations from (LR, HR) couples having very different radiometry transformations.

Maybe you want that your model preserves the input image radiometry. In this case, one simple thing do to prior to training, is to align radiometrically the HR patches over the LR patches. To do that, you can for instance normalize each HR patches image over the LR patches image (e.g. compute the y=ax+b from the averaged HR patches and the LR patches, or from their 1st and 2nd order statistics).
You can then transform easily your HR patches with BandMathX once you have computed your a and b:

# Generate quickly a new patches image
otbcli_BandMathX 
-il 4/patch_extraction/hr_patches.tif 
-out 4/patch_extraction/hr_patches_calibrated.tif 
-exp "{1.2*im1b1+0.2; 0.8*im1b2-0.3; ...; 1.8*im1b4+0.02}"  # here is y=ax+b applied

[quizz0n]

Hi @remicres, thank you for the fast reply!
That makes sense.

In order to normalize each HR patches image over the LR patches image, I'm not sure I understand what would the a and b represent in this case and how to compute them.

[remicres]

a and b are some vector (N components, like your images) to linear transform the HR images.

For instance you can compute them with least square regression for an entire patches-image, considering that one point is the average pixel value for a single patch.

[quizz0n]

I've seen that in GRASS there's the r.regression.line algorithm:

r.regression.line calculates a linear regression from two raster maps, according to the formula
y = a + b*x
where
x
y
represent the input raster maps.
Optionally, it saves regression coefficients.

This should compute the a and b, but I'm not sure which patches image (LR/HR) should be the x / y here.

[remicres]

Ah nice if there is such thing in another OSGeo software ! 👍

You could compute the averaged HR image over the same pixel spacing as LR. Maybe using gdalwarp -r average -tr 10 10 input_hr.tif avg_input_hr.tif as x (and y is your LR patches)

[quizz0n]

Thanks @remicres!
Unfortunately r.regression.line doesn't work properly. Do you have any suggestion for computing the a and b ?

[remicres]

Yes, you could compute them with least square regression for an entire patches-image, considering that one point is the average pixel value for a single patch.
Use gdal or otbApplication to read your LR and HR and patches images, then use numpy to average per patches and compute the polynomial fitting

[quizz0n]

Hi @remicres,
I was able to fix the issue then with your suggestion (many thanks for that!) but didn't manage to give you an update so here it is.

From left to right: pre-trained model, original Spot-6 image, own model (about 2k patches).

[remicres]

Hi @quizz0n ,

Thanks for the feedback.
This is an interesting comparison, and your images look great 👍

Out of curiosity, which part of the world is that? (our pre-trained model has been only trained from patches over France mainland)

[quizz0n]

Thanks!
This is above one city in Romania, Alba-Iulia.
Looking at the results over the Montpellier area I found the overall characteristics to be pretty similar between the two.
Own model seems to be better however this still has to go through a more in-depth validation.