Run separate CI job for latest boto/botocore version when needed

[potiuk]

By using aiobotocore, we are pinning botocore and boto to a specific
version (in aiobotocore extra of amazon provider). This means that
the boto/botocore dependencies will be updated less frequent in main
(every 3-4 months instead of every few days). This has the danger,
that we will detect errors resulting from the upgrade much later,
so we should run our amazon tests and depending tests with
the latest boto/botocore (having aibotocore uninstalled).

All deferrable tests should skip when aiobotocore is missing so only
non-deferrable tests should run in this job.

Depends on #30127 (so only last commit should be checked).

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

This is unrelated - but I found it along the way. Separate PR follows.

[o-nikolas]

Nice! Not only will this help us catch issues early, but we'll also be able to solve them incrementally. Rather than receiving one large batch of issues after each version bump.

[potiuk]

Nice! Not only will this help us catch issues early, but we'll also be able to solve them incrementally. Rather than receiving one large batch of issues after each version bump.

Precisely

[potiuk]

This seemed to work from the first run (The failing tests were environmental issues) This is ...

But yeah it seems to work indeed:

  Found existing installation: aiobotocore 2.5.0
  Uninstalling aiobotocore-2.5.0:
    Successfully uninstalled aiobotocore-2.5.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Requirement already satisfied: boto3 in /usr/local/lib/python3.7/site-packages (1.26.76)
  Collecting boto3
    Downloading boto3-1.26.92-py3-none-any.whl (134 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.7/134.7 kB 37.4 MB/s eta 0:00:00
  Requirement already satisfied: botocore in /usr/local/lib/python3.7/site-packages (1.29.76)
  Collecting botocore
    Downloading botocore-1.29.92-py3-none-any.whl (10.5 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/10.5 MB 148.1 MB/s eta 0:00:00
  Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3) (0.10.0)
  Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/site-packages (from boto3) (0.6.0)
  Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/site-packages (from botocore) (2.8.2)
  Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.7/site-packages (from botocore) (1.26.15)
  Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore) (1.16.0)
  Installing collected packages: botocore, boto3
    Attempting uninstall: botocore
      Found existing installation: botocore 1.29.76
      Uninstalling botocore-1.29.76:
        Successfully uninstalled botocore-1.29.76
    Attempting uninstall: boto3
      Found existing installation: boto3 1.26.76
      Uninstalling boto3-1.26.76:
        Successfully uninstalled boto3-1.26.76
  Successfully installed boto3-1.26.92 botocore-1.29.92
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

and:

  All Warning errors can be found in the /files/warnings-Providers-postgres.txt file.
  =========== 1694 passed, 3 skipped, 16 warnings in 367.28s (0:06:07) ===========

The 3 skipped were the deferrable tests.

[eladkal]

That is very nice improvement!

[ferruzzi]

Clever solution. 👍