Creates backstage selection dag #1463
Conversation
| logger.info( | ||
| f"Removed MARC fields and serialized records for '{kind}' files: {marc_file_list[kind]}" | ||
| ) | ||
| for kind, file_list in marc_file_list.items(): |
There was a problem hiding this comment.
I like this approach of using the key in the marc_file_list over hard-coding the kind values in a list; more extensible this way.
… dict and adds test.
shelleydoljack
force-pushed
the
t1439-backstage-select
branch
from
9cc2d98 to
8c1a4f5
Compare
|
I rebased main and deployed this branch to sul-libsys-airflow-dev. I uploaded a file of 10 instance ID's and selected "new". Here is that dag run: https://sul-libsys-airflow-dev.stanford.edu/dags/select_backstage_records/grid?run_id=manual__2024-12-03T18%3A17%3A22.205158%2B00%3A00&execution_date=2024-12-03+18%3A17%3A22.205158%2B00%3A00&dag_run_id=manual__2024-12-03T18%3A17%3A22.205158%2B00%3A00&tab=graph The scheduled dag run with data interval 11/23/24-11/30/24 also ran: https://sul-libsys-airflow-dev.stanford.edu/dags/select_backstage_records/grid?run_id=manual__2024-12-03T18%3A17%3A22.205158%2B00%3A00&execution_date=2024-12-03+18%3A17%3A22.205158%2B00%3A00&dag_run_id=scheduled__2024-11-23T06%3A30%3A00%2B00%3A00 It returned 0 new since there aren't newly cataloged records on folio-test for the time period. |
|
I ran gobi selection https://sul-libsys-airflow-dev.stanford.edu/dags/select_gobi_records/grid?run_id=scheduled__2024-11-20T06%3A30%3A00%2B00%3A00&execution_date=2024-11-20+06%3A30%3A00%2B00%3A00&tab=graph&dag_run_id=scheduled__2024-11-20T06%3A30%3A00%2B00%3A00 and it found some instances that were updated. Since gobi only transmit new, it didn't create a file in the generate_isbn_list task. I'll create a ticket for fixing gobi (depends on this PR). |
|
I ran pod selection https://sul-libsys-airflow-dev.stanford.edu/dags/select_pod_records/grid?run_id=scheduled__2024-12-02T06:00:00%2B00:00&execution_date=2024-12-02+06:00:00%2B00:00&tab=graph&dag_run_id=scheduled__2024-12-02T06:00:00%2B00:00 and it found updated records and created the file: |
Closes #1439
I tried running locally but then I remembered I can't connect to postgres.