images to sr.py

[raialvaro]

Hi Remi!

I have managed to get a training model with Sentinel2 images in float32 and I am about to test the weights.

I have trained with a 696 patches and a 120 epochs (default)

I have two questions:

1. I have put the complete training image 1199x1441 in float32

and the result of passing sr.py is the following

I don't know if it requires more epochs or patches to train or is it not the optimal image format to predict

2. I have tried to put a tile in the same format as it has trained, 32x32 in float32, and sr.py gets an error

Traceback (most recent call last):
File "sr.py", line 74, in
infer.ExecuteAndWriteOutput()
File "/work/otb/superbuild_install/lib/otb/python/otbApplication.py", line 2801, in ExecuteAndWriteOutput
return _otbApplication.Application_ExecuteAndWriteOutput(self)
RuntimeError: Exception thrown in otbApplication Application_ExecuteAndWriteOutput: /work/otb/otb/Modules/Remote/otbtf/include/otbTensorflowMultisourceModelFilter.hxx:480:
itk::ERROR: TensorflowMultisourceModelFilter(0x1af6e760): Error occured during tensor to image conversion.
Context: Output image buffered region: ImageRegion (0x7ffd9ed9faa0)
Dimension: 2
Index: [0, 0]
Size: [512, 512]

Input #0:
Requested region: ImageRegion (0x7ffd9ed9fad0)
Dimension: 2
Index: [0, 0]
Size: [32, 32]

Tensor shape ("lr_input": {1, 32, 32, 3}
User placeholders:
Error:
itk::ExceptionObject (0xb89f53f0)
Location: "unknown"
File: /work/otb/otb/Modules/Remote/otbtf/include/otbTensorflowCopyUtils.cxx
Line: 191
Description: itk::ERROR: Number of elements in the tensor is 0 but image outputRegion has 786432 values to fill.
Buffer region:
ImageRegion (0x7ffd9ed9fcc0)
Dimension: 2
Index: [0, 0]
Size: [512, 512]

Number of components: 3
Tensor shape:
{1, 0, 0, 3}
Please check the input(s) field of view (FOV), the output field of expression (FOE), and the output spacing scale if you run the model in fully convolutional mode (how many strides in your model?)

What should be the format of the images that sr.py needs to predict correctly?

Thank so much!!

[remicres]

Hi @raialvaro ,

It looks like your model hasn't converged, or, it has been applied to an image with the wrong dynamic.

To answer your questions:

1. I recommend you to start a tensorboard, and monitor your losses, and (important!) a preview image. This way, you can see if the model is still learning. If the losses reach a plateau, and you have a nice preview: it is good. But in your case, 700 patches is very small. If you don't have that much patches, you can also try with a smaller model (e.g. you can set the --depth to 32 in train.py, this will be also faster) and/or more epochs.
2. The sr.py does produce the SR image in a way that limits the blocking artifacts. The --pad controls the size of the buffer around the output image. So if you have a 32x32 image, which generates a SR image of 256x256, it might be cropped to... 0x0 image (because default --pad value is 128, I believe). This application aims large scale image generation: you can test on a subset of your S2 image instead, like 1024x1024 or something like that (which should result in a 768x768 image using default --pad).
   Also bear in mind that you must apply sr.py on an image which has the same dynamic as the LR patches used in train.py.

[franjamonga]

Hello Remi, very interesting what you are saying. In my case I have the same problem. I would like to ask you how to put Tensorboard to be able to visualize the training curve as well as the validation in images. More specifically how should I add it to the train.py code to make tensorboar work correctly?.

Thank you very much for your attention.

Best regards

[remicres]

You have to use train.py with --logdir /path/to/some/folder.

In the mean time, you start tensorboard with something like

tensorboard --logdir /path/to/some/folder

Then open you favorite web browser and connect to your tensorboard (e.g. http://192.168.xxx.xxx:6006/ if you run tensorboard on the server with ip 192.168.xxx.xxx)
You will find plenty of tutorial on tensorboard, but it is very simple.

Tensorboard will help you to find "good" losses weights for L1, L2, GANs , etc. since it enables you to watch how losses evolves during training. My advice is to start with losses that have the same magnitude order. Do not change the GAN loss weight, but carefully adjust L1 (or L2) and VGG losses weights (if you use perceptual loss with pretrained VGG).

[remicres]

I will improve the documentation soon!