Status of testing Providers that were prepared on November 24, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider celery: 3.8.5rc1

• Allow for retry when tasks are stuck in queued (#43520): @dimberman

Provider cncf.kubernetes: 10.0.1rc1

• Bugfix KubernetesJobOperator.on_kill() (#44131): @moiseenkov
• Allow for retry when tasks are stuck in queued (#43520): @dimberman
• Rename execution_date to logical_date across codebase (#43902): @sunank200

Provider google: 11.0.0rc1

• Remove deprecated functionality from Google provider (#43953): @moiseenkov
• feat: add OpenLineage support for BigQueryToBigQueryOperator (#44214): @kacpermuda
• Introduce gcp advance (V3) API translate native datasets operators (#44271): @olegkachur-e
• Introduce new gcp TranslateText and TranslateTextBatch operators (#43860): @olegkachur-e
• Add gcloud command to DataprocCreateClusterOperator to be able to create dataproc on GKE cluster (#44185): @MaksYermak
• Fix incorrect query in BigQueryAsyncHook.create_job_for_partition_get (#44225): @sean-rose
  Linked issues:
  • Linked Issue #37655: @Silviu-Surcica
• Fix Dataplex Data Quality partial update (#44262): @amirmor1
• Bump google-cloud-translate to 3.16 (#44297): @kaxil
• fix google datacatalog operator tests (#44281): @gopidesupavan
• Rename execution_date to logical_date across codebase (#43902): @sunank200

Provider standard: 0.0.2rc1

• Fix TriggerDagRunOperator extra_link when trigger_dag_id is templated (#42810): @fredthomsen
• Move TriggerDagRunOperator to standard provider (#44053): @hardeybisey
  Linked issues:
  • Linked Issue #43641: @gopidesupavan
• Move filesystem sensor to standard provider (#43890): @kunaljubce
  Linked issues:
  • Linked Issue #43641: @gopidesupavan
• Rename execution_date to logical_date across codebase (#43902): @sunank200

All users involved in the PRs:
@amirmor1 @dimberman @hardeybisey @fredthomsen @MaksYermak @sunank200 @kunaljubce @kacpermuda @sean-rose @kaxil @olegkachur-e @moiseenkov @gopidesupavan

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[moiseenkov]

Hi,
#44131 and #43953 look good

[hardeybisey]

Hi, #44053 works as expected.

[fredthomsen]

#42810 is good.

[kacpermuda]

#44214 is good

[amirmor1]

#44262 works as expected

[sean-rose]

I verified that #44225 works.

[kunaljubce]

@eladkal So I verified that #43890 works with apache-airflow>=2.10.0 -

However, there's an error when I do the same with apache-airflow==2.9.3 -

Unless I am doing something wrong, in which case please let me know, it seems that this is happening because of #41021 which seems to be linked to the airflow 2.10.0 milestone. If that's the case, maybe we should update the documentation for it on PyPi?

[potiuk]

However, there's an error when I do the same with apache-airflow==2.9.3

Good catch. Apparently we miss compatibility class for that in "common.compat" and the sensor has no corresponding tests that would detect it in "providers/tests/standard/sensors" - the test for filesystem sensor is still in "tests/sensors" that's why our test suite did not catch it.

If that's the case, maybe we should update the documentation for it on PyPi?

Nope. We want to keep 2.8+ compatibility. We should add compatibilty StartTriggerArgs class in "common.compat" similarly to other "after 2.8" features, and import the class from there in the sensor.

I guess @eladkal - since it's not a regression and standard provider is still 0.* it's not a blocker for the release and can be fixed as a follow-up.

[eladkal]

@potiuk is correct, we don't consider this as a regression as standard provider is not yet "stable".
Specifically for 2.9.x the sensor is reachable by importing from core path. The PR in question just moved it from core to providers so while it doesn't work as intended there is simple workaround for 2.9

[eladkal]

@kunaljubce if you can raise a PR with a fix that would be much appreciated

[eladkal]

Thank you everyone. Providers are released.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.