[ENH]: Allow for matching on the full path

[TomAugspurger]

This adds a parameter to Storage.walk, allowing users to specif whether the matches applies just to the filename or to the full path, including the filename.

The same keyword is added to the task and task input, to let this be set in the datasset.yaml.

Recent changes to the data published in the ecmwf container requires a new feature in pctasks' create-chunks: the ability to match on a full path rather than just a filename.

Data are being published under both the

- <date-prefix>/ifs/...
- <date-prefix>/aifs/...

We list both sets of data, but our stactools package can't handle the aifs data properly yet. So we want to filter it out.

Previously, matches could only target the filename being listed. We need that to filter to specific products. We want to extend that match expression to include the prefix, so that we can filter out the aifs data.

How Has This Been Tested?

In addition to the unit tests, some manual tests to mimic what will run on the server:

# tst.py

from pctasks.dataset.chunks.task import *

input_ = CreateChunksInput.parse_obj(
    {
        "src_uri": "blob://ai4edataeuwest/ecmwf/20240517/00z/",
        "dst_uri": "assets-new",
        "options": {
            "chunk_length": 30000,
            "since": "2024-01-01T10:00:00+00:00",
            "extensions": [".grib2"],
            "matches": "/ifs/(0p25|0p4-beta)/(enfo|oper|waef|wave)(?!-opendata)",
            "match_full_path": True,
            "list_folders": False,
            "chunk_file_name": "uris-list",
            "chunk_extension": ".csv",
        },
    }
)

CreateChunksTask().run(input_, TaskContext(StorageFactory(), "test"))


input_ = CreateChunksInput.parse_obj(
    {
        "src_uri": "blob://ai4edataeuwest/ecmwf/20240517/00z/",
        "dst_uri": "assets-old",
        "options": {
            "chunk_length": 30000,
            "since": "2024-01-01T10:00:00+00:00",
            "extensions": [".grib2"],
            "matches": "/ifs/(0p25|0p4-beta)/(enfo|oper|waef|wave)(?!-opendata)",
            "list_folders": False,
            "chunk_file_name": "uris-list",
            "chunk_extension": ".csv",
        },
    }
)
CreateChunksTask().run(input_, TaskContext(StorageFactory(), "test"))

The first run (match_full_path=True) outputs files, and we can confirm that there aren't any aifs files:

cat assets-new/all/ai4edataeuwest/ecmwf/20240517/00z/0/uris-list.csv | grep aifs | wc -l

The second one (with match_full_path unset, equal to False the old behavior) it doesn't match anything.

Checklist:

Please delete options that are not relevant.

• I have performed a self-review
• Changelog has been updated
• Documentation has been updated
• Unit tests pass locally (./scripts/test)
• Code is linted and styled (./scripts/format)