2085 add proportions nhsn #2111
Conversation
nhsn/delphi_nhsn/pull.py
Outdated
| updated_timestamp = datetime.utcfromtimestamp(int(response["rowsUpdatedAt"])) | ||
| now = datetime.utcnow() | ||
| recently_updated = (now - updated_timestamp) < timedelta(days=1) |
There was a problem hiding this comment.
issue: I think this "recently-updated" logic is sufficient but not robust. For example, if we fail to pull data for multiple days, the next day we run we would not pull data we had never seen before if it was not posted in the last day.
The more robust solution would be to save last pull's updated_timestamp to a local file. We would then load that and compare updated_timestamp to that -- if exactly equal, skip update; if unequal, pull data.
There was a problem hiding this comment.
definitely makes sense and something I didn't think about! The only thing I did different was use the api instead of scanning the file since I imagine the file list is going to go and doesn't make much sense to scan the file list every day
I was originally planning for a seperate one. I was considering adding the new signal in this PR, but when I tried to add just the new columns and locally ran the tests and ran across issues and looks like it might be more involved, so i think it'd be better to create a seperate one. I also kinda shoved in other issues (retry and daily checking) and didn't want to add more things on top |
Description
address #2085
Changelog
added new function that checks the metadata for last update
-> the function also checks for 503 error
-> added signals for total reporting hospitals
Associated Issue(s)