The parameter length check might need to be changed to this form, at least it was used in another file accepting DataFrames too:

x_size = X.index.size if isinstance(X, pd.DataFrame) else X.shape[0]
if x_size != y.size:
raise NonMatchingParameterLengthsException(f"X and y must have the length {x_size} != {y.size}.")

Short notice @nmaarnio: I'll have a look into this by Monday or even Friday, depends on diff. things.

Hi @nmaarnio , I tried to get into this, however, not sure if I do understand the complete context.

In my view, it is sufficient to only use X.shape[0] for both dataframes and numpy arrays. Behind the scenes, pandas 🐼 uses numpy.

If you have a numpy raster stack which originates from a list of 10 rasters, you will have the number of rasters on index[1] and the number of rows (pixels) on index[0].

The underlying question here is: Where does the pd.DataFrame comes from ❓ Maybe the following example (illustrating the complete workflow idea) helps:

Example:

# Imports
import rasterio
import numpy as np
import pandas as pd

from imblearn.combine import SMOTETomek

# Load layers
input_layers = []
for file in file_list:
    # Load
    layer_raster = rasterio.open(file)
    layer_array = layer_raster.read(1)
    layer_array = np.where(layer_array == layer_raster.nodata, np.nan, layer_array)

    # Make it ML compatible (x*y rows, one column)
    layer_array = layer_array.reshape(-1, 1)

# Stacking
layer_stack = np.column_stack(input_layers)       

# layer_stack.shape[0]: number of pixels
# layer_stack.shape[1]: number of layers

# Labels
labels_raster = rasterio.open(file)
labels_array = labels.read(1)
labels_array = np.where(labels_array == labels_raster.nodata, np.nan, labels_array)

# ML-compatibility (all rows in one column)
labels_array = labels_array.reshape(-1, 1)

# Append labels (could be done in one run with the loop above, but better for readability)
raster_stack = np.column_stack([layer_stack, labels_array])

# Remove any rows containing NaN values
raster_stack[
        ~np.isnan(raster_stack).any(axis=1)
    ]

You can easily convert this one to a pd.DataFrame with pd.DataFrame(raster_stack). The first index (0) will be the number of rows and the second (1) the number of layers + one label column.

To further propagate this stuff into the sampling, split it again into X and y

# Split
X, y = raster_stack[:, :-1], raster_stack[:, -1]

# Example for sampling
X_smote_tomek, y_smote_tomek = SMOTETomek(sampling_strategy=0.5).fit_resample(X, y)

To my knowledge, all of the sampling packages and helpers expect X and y to be in shape rows, columns, with rows = spatial_x * spatial_y and columns = number of evidence layers (+ attached labels, if so)

The goal before putting any data into the sampler is to have the right data (cleaned) in the right format (shape). I'm not aware of how the data got prepared before, but if you used .reshape(1, -1), the stacking methods might be changed to np.vstack and additionally transposed (otherwise rows and columns would be in the wrong order and the indizies are upside down as well).

Another, general command
I would not recommend trying to save any of the sampled data as rasters. Any sampling will modify the data and increase/decrease the amount of data used for training by creating some kind of virtual samples (or by dropping), but without a spatial relation.

Thus:

• the number of rows will change, and the spatial_x and spatial_y will not be the same (other dimensions)
• you cannot generally overly the sampled outputs with the original data

Instead, propagate the resulting arrays under the hood and directly feed it [any supervised method] with these - no intermediate saving, since the spatial contstraints cannot be kept. If you want to save those, save it as .csv or .feather (binary and way, way faster and smaller) and then load the file again as input.

Did that help to clarify some points?

Hi @msmiyels and thanks for the comments! I'll get back to this later more in depth, but your last point was what we were looking for most. We were not sure if there is some accepted logic to "squeeze in " the extra synthetic data produced by this tool, but now I understood that there is not. I gather that we should not offer this as a separate tool for the user, but rather parameterize the supervised model training tools that could run this under the hood (?)

A small nitpick, if multiline docstring, the """ should be on a blank line

We need to come up with more descriptive names for these, I think. For users it will be confusing what is a "sampling strategy float" and they might not know what "literal" means in this context.

I don't think we can return the stacked rasters as one. We have to return individual rasters that correspond to the input rasters, like in the unify_rasters CLI function. However, as I was digging into this, I learned that the balancing process changes the data size and spatial integrity of the data (idk if this is a concern). I think we need to think about this more and consult someone who knows about this method in this context.