Add parsing context to DAG Parsing

[potiuk]

Adds proper context in the form of context managers setting
evironment variables to indicate whethere the
dag file is parsed in context of DAG processor or Task Execution
and allows to retrieve DAG_ID and TASK_ID easily.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

This is the first attempt to implement the "robust execution context" as a follow up after #25121 . I am not 100% if I got everything, but I think it might be quite close. @pingzh I know you have a ton of experience with varioous runners and the way they are implemented is a bit "convoluted" so I'd appreciate thorough review (note that it is based on #25121 which was documentation only so only last commit matter.

[potiuk]

A bit cleaner interface (also appropriate to be implemented as "future".

[pingzh]

i like this idea.

one thing is if https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-45+Remove+double+dag+parsing+in+airflow+run is released, we don't need to inject the context in the executor level, as airflow tasks run --local process does not parse dag file anymore. This means we only need the context manager in the task_runner

[potiuk]

Addressed all comments and got rid of the "non-robust" solution.

[ashb]

General point: lets mark anything we do here as experimental as Dag Fetcher (AIP-5) may give us a nicer way of doing this

[potiuk]

General point: lets mark anything we do here as experimental as Dag Fetcher (AIP-5) may give us a nicer way of doing this

We did already:

There is an experimental approach that you can take to optimize this behaviour. Note that it is not always
possible to use (for example when generation of subsequent DAGs depends on the previous DAGs) or when
there are some side-effects of your DAGs generation. Also the code snippet below is pretty complex and while
we tested it and it works in most circumstances, there might be cases where detection of the currently
parsed DAG will fail and it will revert to creating all the DAGs or fail. Use this solution with care and
test it thoroughly.

[potiuk]

I think it should get green this time

[ashb]

We did already:

Ah, but it's not marked with the "experimental" sphinx tag https://github.com/apache/airflow/blame/main/docs/apache-airflow/listeners.rst#L31

[potiuk]

We did already:

Ah, but it's not marked with the "experimental" sphinx tag https://github.com/apache/airflow/blame/main/docs/apache-airflow/listeners.rst#L31

Added.

[potiuk]

Jsut remaining question is about where to inject the parsing_context - @pingzh - any comments here?

[potiuk]

Now I think it should all be fine.

[potiuk]

Right. Finally Green.

[ashb]

LGTM other than the compat issue