Upgrade FAB to 4.1.1

[potiuk]

The Flask Application Builder have been updated recently to
support a number of newer dependencies. This PR is the
attempt to migrate FAB to newer version.

This includes:

• update setup.py and setup.cfg upper and lower bounds to
  account for proper version of dependencies that
  FAB < 4.0.0 was blocking from upgrade
• added typed Flask application retrieval with a custom
  application fields available for MyPy typing checks.
• fix typing to account for typing hints added in multiple
  upgraded libraries optional values and content of request
  returned as Mapping
• switch to PyJWT 2.* by using non-deprecated "required" claim as
  list rather than separate fields
• add possibiliyt to install providers without constraints
  so that we could avoid errors on conflicting constraints when
  upgrade-to-newer-dependencies is used
• add pre-commit to check that 2.4+ only get_airflow_app is not
  used in providers
• avoid Bad Request in case the request sent to Flask 2.0 is not
  JSon content type
• switch imports of internal classes to direct packages
  where classes are available rather than from "airflow.models" to
  satisfy MyPY
• synchronize changes of FAB Security Manager 4.1.1 with our copy
  of the Security Manager.
• add error handling for a few "None" cases detected by MyPY
• corrected test cases that were broken by immutability of
  Flask 2 objects and better escaping done by Flask 2
• updated test cases to account for redirection to "path" rather
  than full URL by Flask2

Fixes: #22397

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

[potiuk]

Ok. Seems that all tests are passing and we are ready to start testing FAB 4.1.1. I run airflow with the upgraded FAB and it seems all the basic functionality works. It was a bit simpler than I thought.

I iwll ask interested parties to do more testing.

Upgraded packages

• Flask-AppBuilder: 3.4.5 -> 4.1.1
• Flask-JWT-Extended: 3.25.1 -> 4.4.1
• Flask-Login: 0.4.1 -> 0.6.1
• Flask-WTF: 0.14.3 -> 0.15.1
• Flask: 1.1.2 -> 2.1.2
• Jinja2: 3.0.3 -> 3.1.2
• MarkupSafe: 2.0.1 -> 2.1.1
• PyJWT: 1.7.1 -> 2.4.0
• Werkzeug: 1.0.1 -> 2.1.2
• itsdangerous: 1.1.0 -> 2.1.2

Removed packages

• Flask-OpenID
• jwcrypto
• python3-openid

[potiuk]

Would love some reviews - I made some changes to make them "work" but possibly some typing changes especially might be done better - looking forward to the reviews.

[potiuk]

All is green. I fixed all the tests and it seems that airflow with upgraded FAB and resulting dependencies (see above) works in basic scenarios..

As discused in #22397 - since you are interested in getting the dependencies upgrade I'd love if you could help with testing:
@lacohen @johannesjung @sk0928 @OfirSabanBitSight @blag @josh-fell @Bowrna @thesuperzapper @gmsantos

I prepared some draft package, constraint file and image to make it easy to test.

Downloads

• Airflow package

Python 3.7 image and constraints:

• Constraints 3.7
• Image can be pulled via docker pull potiuk/airflow:fab-4-1-1-python3.7

Python 3.10 image and constraints:

• Constraints 3.10
• Image can be pulled via docker pull potiuk/airflow:fab-4-1-1-python3.10

Installation

You can install Airflow in your venv via:

1. downloading the .whl

curl -L https://github.com/potiuk/airflow/releases/download/fab-4-1-1-v0-2/apache_airflow-2.4.0.dev0-py3-none-any.whl \
  -o apache_airflow-2.4.0.dev0-py3-none-any.whl

2. downloading the constraints.txt file:

curl -L https://github.com/potiuk/airflow/releases/download/fab-4-1-1-v0-2/constraints-fab-4-1-1-python3.7.txt\
   -o constraints-fab-4-1-1-python3.7.txt

3. installing airflow via

pip install "apache_airflow-2.4.0.dev0-py3-none-any.whl[ADD_YOUR_EXTRAS_HERE]" \
   --constraint constraints-fab-4-1-1-python3.7.txt

They are all "dev" version of upcoming 2.4.0 - we are quite far from the "alpha/beta/rc*" release so this is really "for your eyes and hands only " :).

What to test

I have tested the basics, but I would really appreciate if you could test it with "real configuratios" where FAB features are used. I looked at the breaking changes introduced by FAB and I think it would be great to test:

• celery log retrieval with secret (PyJWT): we have new mechanism of signing/verifying the log requests
• FAB authentication integration: OAuth/OpenID and related
• API and API authentication integration (we have some changes in API implementation) - mostly related to typing
• LDAP integration (there are some changes/fixes in FAB Security Manager releated to LDAP integration)
• Some more complex Jinja usages (Jinja is upgraded together with markupsafe library)

I'd really appreciate comments here if you manage to test it in your environment/staging with real configuration! It will be really helpful to get some validation from our users.

[potiuk]

I am also re-running this PR with "full tests needed" and if needs be I will be able to get images, constraints for all other python versions.

[potiuk]

I updated rich-click to >= 1.5 as it seems that the "yesterday-released" 1.5 version introduced some typing issues

[potiuk]

Merging it now.

@lacohen @johannesjung @sk0928 @OfirSabanBitSight @blag @josh-fell @Bowrna @thesuperzapper - if you have some time to test it and get some feedback here, this might make us speed up some decisisons on when to release it. For now I am marking it as 2.4.0, but, who knows..