Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nssp patching code #2000

Open
wants to merge 46 commits into
base: main
Choose a base branch
from
Open

nssp patching code #2000

wants to merge 46 commits into from

Conversation

minhkhul
Copy link
Contributor

@minhkhul minhkhul commented Jul 23, 2024

Description

Add patching feature to nssp.

One detail that set this implementation apart from most of our other indicators is that nssp data comes in our db as weekly form instead of daily.

Basically, all weekly data coming in to our db has the issue_date column always marked as the first date of the epiweek that the reporting date falls into, rather than the reporting date itself. For example, I query nssp data from our api that was just put into db on July 5th 2024 and got something like this:

geo_value                  signal source geo_type time_type time_value  \
0      54079  pct_ed_visits_combined   nssp   county      week 2022-09-25
**issue**  lag  missing_value  missing_stderr  missing_sample_size  value  \
0  **2024-06-30**   93              0               1                    1   3.27

Note the issue is 2024-06-30 despite this data coming in on July 5th. It's because 2024-06-30 is the start date of epiweek 202427, so any data coming in that week will be marked under that issue date.

Furthermore, the reason why we use this directory format is because of this batch_issue_format implementation in acquisition, which demands folder format to be issue_yyyymmdd including for patches of weekly data.

Fixes

nssp/tests/test_patch.py Outdated Show resolved Hide resolved
@minhkhul minhkhul requested a review from aysim319 July 25, 2024 19:35
nssp/delphi_nssp/patch.py Outdated Show resolved Hide resolved
nssp/delphi_nssp/pull.py Outdated Show resolved Hide resolved
nssp/delphi_nssp/run.py Outdated Show resolved Hide resolved
nssp/tests/test_patch.py Outdated Show resolved Hide resolved
nssp/tests/test_patch.py Outdated Show resolved Hide resolved
nssp/tests/test_patch.py Outdated Show resolved Hide resolved
@minhkhul minhkhul requested review from aysim319 and nmdefries August 9, 2024 15:11
@minhkhul
Copy link
Contributor Author

minhkhul commented Sep 4, 2024

Note: About custom_run flag, non-patch custom run doesn't really exist in nssp like it does in google-symptoms due to the differences in how the two sources handle revision data and how we grab those data. We'll keep the flag in nssp still to maintain consistency and to disambiguate params.json.

nssp/delphi_nssp/pull.py Outdated Show resolved Hide resolved
nssp/delphi_nssp/pull.py Outdated Show resolved Hide resolved
@minhkhul minhkhul requested a review from nmdefries September 10, 2024 18:11
Copy link
Contributor

@nmdefries nmdefries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One issue with generating duplicate data, and a couple clarifying questions.

nssp/README.md Show resolved Hide resolved
nssp/delphi_nssp/patch.py Outdated Show resolved Hide resolved
nssp/delphi_nssp/patch.py Show resolved Hide resolved
nssp/delphi_nssp/patch.py Outdated Show resolved Hide resolved
nssp/delphi_nssp/patch.py Outdated Show resolved Hide resolved
@minhkhul minhkhul requested a review from aysim319 January 8, 2025 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

NSSP patching script and NSSP patch april 2024 to july 2024
4 participants