1446 dag run failed 979s

[jgreben]

Fixes #1446

It is not possible to simply update the state of a dag to "QUEUED" (or anything else) and have it run again (see apache/airflow#35729 (comment)). This PR makes it so that a new DAG is launched with the previous DAG's conf. Also, the DAG that finds the previously failed DAGS will only look for ones that are 30 days old, so that it will not keep trying to re-rerun the subsequently failed ones. I made that number 30 days because I vaguely remember that was the period we thought we would run this new DAG. If you want, I can make that number into a Variable or change it for this PR.

Using the airflow.models.DagRun clear_dags method now.

[shelleydoljack]

It's kind of confusing to separate out commits that were previously merged but I managed. I like the idea of making the 30 days a variable. Especially since we are almost at that point from when we first ran the digital bookplate dags. We'd want to retry the failed ones monthly, so 30 days from the original run seems okay. The MARC records that we can add a 979 to may not appear for a whole year so I guess the same one would get rerun every 30 days, the only difference being the execution_date.

Does it not work to clear the state (rather than setting it to queued) and just let the scheduler queue them?

[jgreben]

The draft PR was not actually ready for a review, but it is now. I didn't see a way using the DagRunState model to remove or delete a state. I will add the Variable for the start_date.

[jgreben]

https://sul-libsys-airflow-dev.stanford.edu/api/v1/dags/{dag_id}/dagRuns/{dag_run_id}/clear does work if we want to create a https function instead of using the DagRun class. I also found a clear method in the Dag model https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html#airflow.models.dag.DAG.clear that I can try.

[shelleydoljack]

https://sul-libsys-airflow-dev.stanford.edu/api/v1/dags/{dag_id}/dagRuns/{dag_run_id}/clear does work if we want to create a https function instead of using the DagRun class. I also found a clear method in the Dag model https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html#airflow.models.dag.DAG.clear that I can try.

I thought you could use the DagRun.set_state() to None but maybe not?

[jgreben]

https://sul-libsys-airflow-dev.stanford.edu/api/v1/dags/{dag_id}/dagRuns/{dag_run_id}/clear does work if we want to create a https function instead of using the DagRun class. I also found a clear method in the Dag model https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dag/index.html#airflow.models.dag.DAG.clear that I can try.

I thought you could use the DagRun.set_state() to None but maybe not?

This works:
dag.clear_dags( dags=[dag], only_failed=True, dry_run=False, )

[shelleydoljack]

Looks good. I'm not sure about the execution_date for dag runs more than 30 days ago.