Status of testing of Apache Airflow 2.7.1rc2

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 2.7.1rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Fix rendering the mapped parameters when using expand_kwargs method (#32272): @Talvarenga @hussein-awala
  Linked issues:
  • Apparently the Jinja template does not work when using dynamic task mapping with SQLExecuteQueryOperator (#32260)
• Bring back hive support for Python 3.11 (#32607): @potiuk
• [Models] [Postgres] Check if the dynamically-added index is in the table schema before adding (#32731): @ohaibbq
• Validate SqoopHook connection string and disable extra options from public hook methods (#33039): @pankajkoti
• Upgrade Elasticsearch to 8 (#33135): @potiuk @Owen-CH-Leung
  Linked issues:
  • Migrate elasticsearch client to 8.* (or latest 7.*) (#31256)
• Fix handling of default value and serialization of Param class (#33141): @jens-scheffler-bosch
  Linked issues:
  • Allow optional defaults in required fields with manual triggered dags (#31301)
• Refactor: Simplify code in configuration.py (#33160): @eumiro
• respect soft_fail argument when ExternalTaskSensor runs in deferrable mode (#33196): @Lee-W
• Fix pydantic warning about orm_mode rename (#33220): @ephraimbuddy
• Refactor: Simplify code in providers/amazon (#33222): @eumiro
• Refactor: Simplify code in providers/cncf (#33230): @eumiro
• Refactor: Simplify code in smaller providers (#33234): @eumiro
• Refactor: Simplify dict manipulation in metrics (#33264): @eumiro
• Refactor: Simplify code in serialization (#33266): @eumiro
• Refactor: Simplify code in settings (#33267): @eumiro
• Refactor: Simplify code in utils (#33268): @eumiro
• Refactor: Simplify code in www (#33270): @eumiro
• Refactor str.starts/endwith with tuples (#33292): @eumiro
• Refactor: Simplify code in tests (#33293): @eumiro
• Refactor: Simpify code in dev (#33294): @eumiro
• D205 Support - Root files (#33297): @ferruzzi
• D205 Support - WWW (#33298): @ferruzzi
• D205 Support - Stragglers (#33301): @ferruzzi
• Remove the replace to
  tag in Provider's view (#33326): @eladkal @pankajkoti
  Linked issues:
  • providers view shows description with HTML element (#33325)
• D401 Support - Api to Auth (Inclusive) (#33333): @ferruzzi
• Fix some missing type hints (#33334): @ferruzzi
• D401 Support - Callbacks to Decorators (Inclusive) (#33335): @ferruzzi
• D401 Support - Example_dags to Listeners (Inclusive) (#33336): @ferruzzi
• D401 Support - Macros to Operators (Inclusive) (#33337): @ferruzzi
• D401 Support - Secrets to Triggers (Inclusive) (#33338): @ferruzzi
• D401 Support - Utils (#33339): @ferruzzi
• D401 Support - Root and Misc Files (#33352): @ferruzzi
• Fix DagFileProcessor interfering with dags outside its processor_subdir (#33357): @pierrejeambrun
• openlineage: do not try to redact Proxy objects from deprecated config (#33393): @mobuchowski
• Respect "soft_fail" argument when "poke" is called (#33401): @Lee-W
• Respect "soft_fail" for core async sensors (#33403): @Lee-W
• Allow timetable to slightly miss catchup cutoff (#33404): @uranusjr @BohdanSemonov @potiuk
  Linked issues:
  • Fix lost runs in CronTriggerTimetable (#32921)
  • CronTriggerTimetable lost one task occasionally (#27399)
• More complete fix for dev release scripts to filter commit for PR (#33418): @ephraimbuddy @potiuk
  Linked issues:
  • Fix getting correct commit from multiple referenced PR (#33411)
  • Fix missing related PRs when cherry-picking (#33261)
• Create a new method used to resume the task in order to implement specific logic for operators (#33424): @Lee-W @hussein-awala
  Linked issues:
  • respect soft_fail argument when ExternalTaskSensor runs in deferrable mode (#33196)
  • respect "soft_fail" argument when running BatchSensor in deferrable mode (#33405)
• Import utc from datetime and normalize its import (#33450): @eumiro
• make conf.set case insensitive (#33452): @vandonr-amz
• Simplify 'X for X in Y' to 'Y' where applicable (#33453): @eumiro
• Static check fixes (#33462): @ferruzzi
• Simplify code around enumerate (#33476): @eumiro
• Consolidate import and usage of itertools (#33479): @eumiro
• Consolidate import and usage of pandas (#33480): @eumiro
• Fix a bug in js formatDuration method (#33486): @hussein-awala
• Fix Cluster Activity historical metrics for ongoing dagrun if end_date is provided (#33488): @pierrejeambrun
• Remove openlineage exclusion (#33491): @potiuk
• Use dialect.name in custom SA types (#33503): @agomez-etsy @Taragolis
  Linked issues:
  • SQL_ALCHEMY_CONN_CMD causes triggerers to fail liveness probes on peak (#33485)
• Remove Pydantic limitation for version < 2 (#33507): @potiuk
• Replace OrderedDict with plain dict (#33508): @eumiro
• Update version_added field for configs in config file (#33509): @ephraimbuddy
• Set strict to True when parsing dates in webserver views (#33512): @klexadoc @hussein-awala
  Linked issues:
  • Rendered template malfunction when execution_date parameter is malformed (#33478)
• Stop adding values to rendered templates UI when there is no dagrun (#33516): @klexadoc @hussein-awala
  Linked issues:
  • Rendered template malfunction when execution_date parameter is malformed (#33478)
• Replace repr() with proper formatting (#33520): @eumiro
• Sort data before groupby in TIS duration calculation (#33535): @Taragolis
• Simplify conditions on len() in utils (#33567): @eumiro
• Simplify conditions on len() in jobs (#33568): @eumiro
• Simplify conditions on len() in other providers (#33569): @eumiro
• Add MySQL 8.1 to supported versions. (#33576): @potiuk
• Fix dependencies for celery and opentelemetry for Python 3.8 (#33579): @potiuk @nathadfield
  Linked issues:
  • Airflow 2.7 is incompatible with SodaCore versions 3.0.24 and beyond (#33577)
• Improve detection of when breeze CI image needs rebuilding (#33603): @s0neq @potiuk
  Linked issues:
  • Resume yandex provider (#33574)
• Add a fallback in case no first name and last name are set (#33617): @vincbeck @cesar-vermeulen
  Linked issues:
  • AIP-56 - Create auth manager interface and very basic implementation of FAB auth manager (#32217)
  • Airflow 2.7 Webserver unreacheable with new authentication manager (#33586)
• Bind engine before attempting to drop archive tables (#33622): @SamWheating
• Fix typos (double words and it's/its) (#33623): @eumiro
• Treat dag-defined access_control as authoritative if defined (#33632): @SamWheating
• Use a trimmed version of README.md for PyPI (#33637): @potiuk @pankajkoti
  Linked issues:
  • Trim down the PyPI readme (#33505)
• Set context inside templates (#33645): @ivan-afonichkin
• Upgrade botocore/aiobotocore minimum requirements (#33649): @potiuk
• Fix 2.7.0 db migration job errors (#33652): @LipuFei
  Linked issues:
  • 2.7.0 db migration job fails with CeleryKubernetesExecutor and Sentry (#33651)
• Add back get_url_for_login in security manager (#33660): @vincbeck @cesar-vermeulen
  Linked issues:
  • Airflow 2.7 Webserver unreacheable with new authentication manager (#33586)
• Add limit 1 if required first value from query result (#33672): @Taragolis
• SECURITY_MANAGER_CLASS should be a referrence to class, not a string (#33690): @Michalosu
• Fix cleaning zombie RESTARTING tasks (#33706): @Bisk1
  Linked issues:
  • Zombie tasks in RESTARTING state are not cleaned (#33661)
• Fix MappedTaskGroup tasks not respecting upstream dependency (#33732): @ephraimbuddy
• Add documentation explaining template_ext (and how to override it) (#33735): @SamWheating
• Fix version heads map pre_commit rule (#33749): @jedcunningham
• Use f-string instead of format in Airflow core (#33753): @hussein-awala
• Limit Redis dependencies to match celery limits (#33773): @alexbegg @potiuk
  Linked issues:
  • Celery Executor is not working with redis-py 5.0.0 (#33744)
• Limit hive provider check for Python 3.11 temporarily (#33774): @potiuk
• Fix UI DAG counts including deleted DAGs (#33778): @joeknize-bc @boushphong
  Linked issues:
  • UI DAG counts including deleted DAGs (#33698)
• Remove "eager upgrade" from PROD image completely (#33784): @potiuk
• Raise 404 from Variable PATCH API if variable is not found (#33885): @abhishekbhakat
• Suspend qubole provider (#33889): @potiuk
• Add minimum requirement for sqlalchemy to 1.4.24 (#33892): @potiuk @aipatr
  Linked issues:
  • Airflow db migrate AttributeError: 'Session' object has no attribute 'scalars' (#33887)
• Ensure that tasks wait for running indirect setup (#33903): @dstandish
• Reorganize devel_only extra in airflow's setup.py (#33907): @potiuk
  Linked issues:
  • Suspend qubole provider (#33889)
• Bumping FAB to 4.3.6 in order to fix issues with filters (#33931): @SamWheating
• Upgrade to Pydantic 2 (#33956): @potiuk
• Differentiate 0 and unset as a default param values (#33965): @SamWheating
• Remove print introduced in PR #32261 (#34008): @pankajkoti
• Limit celery by excluding 5.3.2 and 5.3.3 (#34031): @potiuk
• Bring back Pydantic 1 compatibility (#34081): @potiuk
• Remove Pydantic 1 compatibility code (#33998): @uranusjr
• Attempt to fix the flaky TestTriggererJob test (#34075): @potiuk

Thanks to all who contributed to the release (probably not a complete list!):
@agomez-etsy @Lee-W @jedcunningham @potiuk @cesar-vermeulen @pankajkoti @abhishekbhakat @dstandish @vandonr-amz @SamWheating @joeknize-bc @ephraimbuddy @ohaibbq @alexbegg @Michalosu @BohdanSemonov @pierrejeambrun @ferruzzi @eumiro @mobuchowski @Taragolis @Talvarenga @aipatr @LipuFei @Bisk1 @uranusjr @hussein-awala @vincbeck @s0neq @klexadoc @boushphong @eladkal @jens-scheffler-bosch @Owen-CH-Leung @nathadfield @ivan-afonichkin

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pankajkoti]

hi @ephraimbuddy , wondering why #33039 is part of Airflow RC? I think it went in the Sqoop provider, no?

[ephraimbuddy]

hi @ephraimbuddy , wondering why #33039 is part of Airflow RC? I think it went in the Sqoop provider, no?

It's because of conflict resolution, trying to cherry-pick cleanly. We had many refactoring PRs that addressed issues across core and providers. If you look at the description on the sync PR: #33826 I mentioned about provider changes getting cherry-picked to core.
You do not need to test this in this case

[cesar-vermeulen]

I encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic upgrade. Do I open a seperate issue for this or can this be discussed here?
Great expectations does not yet support pydantic > 2.0:
great-expectations/great_expectations#8604

Because of this, we are unable to upgrade our airflow version

[Taragolis]

Check on fresh Airflow 2.7.1.rc1

• Add limit 1 if required first value from query result #33672 - Less intensive queries to DB. Shorter execution time, less traffic between backend and Airflow
• Use dialect.name in custom SA types #33503 - Seems working with Postgres/MySQL as expected
• Sort data before groupby in TIS duration calculation #33535 - Work as expected in the UI

[ephraimbuddy]

I encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic upgrade. Do I open a seperate issue for this or can this be discussed here? Great expectations does not yet support pydantic > 2.0: great-expectations/great_expectations#8604

Because of this, we are unable to upgrade our airflow version

Ops. Looks like we don't currently have great expectation in our codebase but would like to hear what others think cc @potiuk

[hussein-awala]

I encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic

upgrade. Do I open a seperate issue for this or can this be discussed here? Great expectations does not yet support pydantic > 2.0: great-expectations/great_expectations#8604 <great-expectations/great_expectations#8604> Airflow doesn't have an official integration/support for great expectations, so if great expectations doesn't support pydantic v2 which we use in 2.7.1, IMHO this should not be considered a breaking change.

…

On Mon, Sep 4, 2023 at 5:28 PM Andrey Anshin ***@***.***> wrote: Check on fresh Airflow 2.7.1.rc1 - #33672 <#33672> - Less intensive queries to DB. Shorter execution time, less traffic between backend and Airflow - #33503 <#33503> - Seems working with Postgres/MySQL as expected - #33535 <#33535> - Work as expected in the UI — Reply to this email directly, view it on GitHub <#34065 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCS775S3YTSFWWQA7LUH6LXYXXQLANCNFSM6AAAAAA4J6KWGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ephraimbuddy]

I encountered an issue when upgrading to 2.7.1rc1, linked to the pydantic
upgrade. Do I open a seperate issue for this or can this be discussed here? Great expectations does not yet support pydantic > 2.0: great-expectations/great_expectations#8604 <great-expectations/great_expectations#8604> Airflow doesn't have an official integration/support for great expectations, so if great expectations doesn't support pydantic v2 which we use in 2.7.1, IMHO this should not be considered a breaking change.
…
On Mon, Sep 4, 2023 at 5:28 PM Andrey Anshin @.> wrote: Check on fresh Airflow 2.7.1.rc1 - #33672 <#33672> - Less intensive queries to DB. Shorter execution time, less traffic between backend and Airflow - #33503 <#33503> - Seems working with Postgres/MySQL as expected - #33535 <#33535> - Work as expected in the UI — Reply to this email directly, view it on GitHub <#34065 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFCS775S3YTSFWWQA7LUH6LXYXXQLANCNFSM6AAAAAA4J6KWGM . You are receiving this because you were mentioned.Message ID: @.>

Not a breaking change but great expectation is widely used, I think we should consider it

[hussein-awala]

Not a breaking change but great expectation is widely used, I think we should consider it

My only concern is that pydantic v2 was added to fix a bug or just an improvement? if not maybe we should move it to 2.8.0

[ephraimbuddy]

Not a breaking change but great expectation is widely used, I think we should consider it

My only concern is that pydantic v2 was added to fix a bug or just an improvement? if not maybe we should move it to 2.8.0

Yeah, I think it's for improvement

[potiuk]

Not a breaking change but great expectation is widely used, I think we should consider it

My only concern is that pydantic v2 was added to fix a bug or just an improvement? if not maybe we should move it to 2.8.0

Yeah, I think it's for improvement

Specifically for AIP-44 so yes ww move it down to v1 and we will release for 2.8.0 (when I hope AIP-44 will be ready to ship). By then Pydantic 2 should be much more widespread.

[pankajkoti]

hi,
Tested the below PRs are included and they all work as expected.

1. Remove the replace to <br> tag in Provider's view #33326 - The BR tag no longer appears in the providers view. Works as expected
2. Use a trimmed version of README.md for PyPI #33637 - The generated/PYPI_README.md file is present in the RC and is pointed to in the setup.cfg for long_description for the package.
3. Remove print introduced in PR #32261 #34008 : The endpoint no longer prints - Works as expected

As noted in comment #34065 (comment),
Ignored cherry-picked PR for conflict resolution #33039

Thank you for including all of these 🙏🏽

[Bisk1]

Hi,
I verified that this change works as expected in UI in 2.7.1rc1

• Fix cleaning zombie RESTARTING tasks #33706 - a task that was in RESTARTING state always goes into RUNNING state after scheduler start, the scheduler logs indicate that it was adopted

[potiuk]

Confirmed all my changes are in.! Looks good.

[pankajkoti]

+1. Checked my changes from RC1 (#34065 (comment)) are included here too and all work fine.

[LipuFei]

Confirmed that #33651 and #33652 are fixed in RC2.

[klexadoc]

Confirmed that #33478 is fixed

[ohaibbq]

Confirmed #32731 is fixed in RC1

[khmelevskiy]

i checked the version of airflow:2.7.1rc2-python3.10.

run dag (test)
-> let it work
-> rename dag (add _v2, eg = test_v2)
result - bug, there are 2 dags test and test_v2.

#33778 (#33698)

The key moment, dag worked before renaming

[SamWheating]

I have pulled the 2.7.1rc2 commit and verified the following fixes in Breeze:

#33622
#33931
#33965

#33735 is a docs-only fix and should be fine.

I'm trying to verify #33632 but running into issues triggering the permissions sync (this issue might predate this release?) I will keep investigating and confirm.

Update: I have found some issues preventing #33632 from working as expected. I will work on a fix and provide updates in #34114, but in the meantime we could also remove this commit from the release and save it for 2.7.2? I will leave it up to you.

[vandonr-amz]

good for me

[Lee-W]

Tested #33403, #33401, #33196

[Lee-W]

Also tested #33424

[jscheffl]

Checked #33141 and can confirm it is working.

Found a minor "glitch" of string formatting as we forked code of Flask-AppBuilder (manager.py) into the Airflow repo and the Flask Appbuilder has changed the string formatting upstream, the constants we source in manager.py:499 for example are now printed with %s instead of the content.
I assume not critical, will open a follow-up bug ticket (if not already open)

[ephraimbuddy]

Airflow 2.7.1 is now released. Thank you all for testing this release

[hussein-awala]

Found a minor "glitch" of string formatting as we forked code of Flask-AppBuilder (manager.py) into the Airflow repo and the Flask Appbuilder has changed the string formatting upstream, the constants we source in manager.py:499 for example are now printed with %s instead of the content. I assume not critical, will open a follow-up bug ticket (if not already open)

@jens-scheffler-bosch could you check if #34139 fix the bug you are talking about?

[jscheffl]

Found a minor "glitch" of string formatting as we forked code of Flask-AppBuilder (manager.py) into the Airflow repo and the Flask Appbuilder has changed the string formatting upstream, the constants we source in manager.py:499 for example are now printed with %s instead of the content. I assume not critical, will open a follow-up bug ticket (if not already open)

@jens-scheffler-bosch could you check if #34139 fix the bug you are talking about?

Perfect. Yes. that was what I was referring to. Did not find the issue by search... so fixed already, GREAT!