Status of testing Providers that were prepared on December 08, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.13.0rc1

• Support IAM authentication for Redshift serverless (#35897): @hussein-awala
• Implement is_authorized_variable in AWS auth manager (#35804): @vincbeck
• Enhance DynamoDBValueSensor to support any attribute value in list (#35831): @flolas
• Fix handling of single quotes in RedshiftToS3Operator (#35986): @vincbeck
• Fix a bug in get_iam_token for Redshift Serverless (#36001): @hussein-awala
• Fix reraise outside of try block in AthenaHook.get_output_location (#36008): @Taragolis
• Fix a bug with accessing hooks in EKS trigger (#35989): @hussein-awala
• Fix a bug in method name used in GlacierToGCSOperator (#35978): @hussein-awala
• Fix EC2Hook get_instance for client_type api (#35960): @hussein-awala
• Avoid creating the hook in the EmrServerlessCancelJobsTrigger init (#35992): @hussein-awala
• Stop getting message from event after migrating EmrContainerTrigger to AwsBaseWaiterTrigger (#35892): @hussein-awala
• Fix for EksCreateClusterOperator deferrable mode (#36079): @syedahsn
• Update boto3 and botocore versions notes (#36073): @Taragolis
• Improve typing hints for only_client_type decorator (#35997): @hussein-awala
• Refactor some methods in EmrContainerHook (#35999): @hussein-awala
• Refactor get_output_location method in AthenaHook (#35996): @hussein-awala
• Move RDS hook to a cached property in RDS trigger (#35990): @hussein-awala
• Replace default empty dict value by None in AzureBlobStorageToS3Operator (#35977): @hussein-awala
• Refacto CloudwatchTaskHandler set_context method (#35975): @hussein-awala
• Use S3 hook instead of AwsGenericHook in AWS S3 FS (#35973): @hussein-awala
• AWS auth manager: implement all is_authorized_* methods (but is_authorized_dag) (#35928): @vincbeck
• Remove setting a non-existing object param and use local var instead in S3Hook (#35950): @hussein-awala

Provider cncf.kubernetes: 7.11.0rc1

• fix: KPO typing env_vars (#36048): @raphaelauv
• Stop converting state to TaskInstanceState when it's None (#35891): @hussein-awala
• Feature pass dictionary configuration in application_file in SparkKubernetesOperator (#35848): @romsharon98

Provider docker: 3.9.0rc1

• Fix enable_logging=True not working in DockerSwarmOperator (#35677): @stavdav143

Provider elasticsearch: 5.3.0rc1

• Cleanup code for elasticsearch<8 (#35707): @eladkal

Provider google: 10.13.0rc1

• GCP Secrets Backend Impersonation (#36072): @nathadfield
• Add OpenLineage support to GCS operators (#35838): @kacpermuda
• Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel (#35136): @ginolegigot
• Fix DataprocSubmitJobOperator to retrieve failed job error message (#36053): @VladaZakharova
• Fix CloudRunExecuteJobOperator in deferrable mode (#36012): @VladaZakharova
• Fix gcs listing - ensure blobs are loaded (#34919): @atrbgithub
• allow multiple elements in impersonation chain (#35694): @melugoyal
• Deprecate CloudComposerEnvironmentSensor in favor of CloudComposerCreateEnvironmentOperator with defer mode (#35775): @VladaZakharova

Provider http: 4.8.0rc1

• Fix HttpOperator pagination with str data (#35782): @Joffreybvn

Provider jdbc: 4.2.0rc1

• Add documentation about "providers.jdbc" in provider.yaml (#35580): @pankajkoti

Provider microsoft.azure: 8.4.0rc1

• Fix reraise outside of try block in AzureSynapsePipelineRunLink.get_fields_from_url (#36009): @Taragolis
• Do not catch too broad exception in WasbHook.delete_container (#36034): @Taragolis

Provider microsoft.mssql: 3.6.0rc1

• Remove workaround for pymssql failing compilation with new Cython (#35924): @potiuk

Provider openai: 1.1.0rc1

• Bump up openai version to >=1.0 & use get_conn (#36014): @pankajkoti

Provider openlineage: 1.3.0rc1

• OpenLineage: Add parent_run_id for COMPLETE and FAIL events (#36067): @kacpermuda
• openlineage: add basic metrics to stats collector. (#35368): @JDarDagran
• Repair different run_id for OpenLineage START and FAIL events. (#36051): @kacpermuda
• openlineage: add ClassVar to Airflow facets. (#36084): @JDarDagran

Provider trino: 5.5.0rc1

• Feature/trino provider timezone (#35963): @romsharon98

Provider weaviate: 1.1.0rc1

• Add object methods in weaviate hook (#35934): @ephraimbuddy
• Add a cache for weaviate client (#35983): @ephraimbuddy
• Add more ways to connect to weaviate (#35864): @ephraimbuddy

All users involved in the PRs:
@Taragolis @vincbeck @romsharon98 @syedahsn @kacpermuda @raphaelauv @melugoyal @ginolegigot @atrbgithub @flolas @JDarDagran @nathadfield @potiuk @ephraimbuddy @Joffreybvn @pankajkoti @hussein-awala
@VladaZakharova @stavdav143 @eladkal

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[melugoyal]

#35694 working as expected

[atrbgithub]

An issue has been raised with #34919

Unfortunately it looks like it has broken GCSObjectsWithPrefixExistenceSensor.

Testing myself I can confirm that the following code is able to create the error:

from airflow.providers.google.cloud.hooks.gcs import GCSHook
res = GCSHook().list(
        bucket_name='a-testbucket',
        prefix='a/prefix/in/the/bucket/'
)
print(res)

Please do not include this change.

Edit: I've raised a PR #36130 which appears to fix this issue for me when testing locally.

[eladkal]

Docker provider will be excluded from rc1

[hussein-awala]

I tested my change on the Kubernetes executor, and most of the changes in the Amazon provider, all work as expected.

[pankajastro]

As @atrbgithub has mentioned above #34919 is breaking an existing DAG. so we might need to exclude Google provider

[eladkal]

so we might need to exclude Google provider

Yes. Google provider will also have RC2

[potiuk]

Tested mssql - all looks good.

One more thing I noticed - the removed daskexecutor has not been prepared. We have this in the release process, so it's easy to miss:

NOTE! When you want to release a provider marked for removal (needed in order to prepare last release of the provider), package for the provider will not be prepared when you prepare documentation for all providers - you have to specifically use the provider name in a separate command. For example to prepare documentation for removed.provider provider marked for removal you need to run separately this command:

breeze release-management prepare-provider-packages --package-format both removed.provider

But maybye we will change it in the future - we have more removed providers coming. So possibly we shoudl automatically add removed providers to the list of providers prepared (both for documentation and package building - so that we do not have to remember to prepare them. That was probably bad idea to not add them in the first place.

@eladkal -> I guess we should include rc2 (even if there was no rc1) for daskexecutor.

[potiuk]

One small update - as @pankajastro noticed in #32319 (comment) - there is a case where we are not handling properly the fact that handler can return either list of rows or row - in both ODBC and DAtabricks oviders, so IMHO they should also be moved to RC2 (cc: @Joffreybvn )

[potiuk]

Fix to the process to automatically generate removed providers here: #36160

[Joffreybvn]

Fix for #32319 (comment) is here -> #36161

[nathadfield]

#36072 has been tested and working as expected.

[eladkal]

Providers: Google, ODBC, Databricks, Docker are excluded from RC1.

[eladkal]

Thank you everyone.
Providers are released (excluding Google, ODBC, Databricks, Docker which we will have RC2 soon)
I invite everyone to help improve providers for the next release, a list of open issues can be found here.