Stage1 data cleaning

[ZoeMZou]

PR: Refinements for Script Structure & Data Processing

(Based on discussions on 28th January 2025)

🔹 1. Structural Changes
📌 1.1 Naming Conventions

• Removed _stage1 from scripts to improve clarity and consistency.

• The old stage1_data_cleaning.R script was handling:

  1. Setting reference levels for categorical variables
  2. Applying quality assurance
  3. Applying inclusion/exclusion criteria

• The new structure separates these tasks into three dedicated functions:

  • fn-ref.R → Sets reference levels
  • fn-qa.R → Applies quality assurance
  • fn-inex.R→ Applies inclusion/exclusion criteria

• The main script data_cleaning.R calls these functions, ensuring: Project-specific edits are made in function scripts and data_cleaning.R remains clear and workflow-focused.

📌 1.2 Output Dataset Naming

• The cleaned dataset output is now: input_{cohort}.rds

• The preprocessed dataset output is now:
  input_{cohort}.rds → input_{cohort}_0.rds (to avoid YAML conflicts)

🔹 2. fn-ref.R — Reference Level Settings

📌 2.1 Data Formatting Fixes

• Some categorical variables were not factors, and some numeric variables were not numeric. I added code back to preprocess_data.R to ensure correct formatting:
  🔗 See fix in preprocess_data.R

📌 2.2 Code Deletions from Old Repo
Index Date: Already correct—no correction needed.
🔗 Old repo reference
Deprivation (IMD): No need for re-categorization; new repo includes IMD (1–5) directly.
🔗 Old repo reference

📌 2.3 Sex Category Adjustments
Old repo: Only male (M) and female (F).
New repo: Four levels → female, male, intersex, unknown.
Fix:
Set non-male/non-female values to missing. Retain three levels: female, male, unknown.

🔹 3. fn-qa.R — Quality Assurance Fixes

📌 3.1 Fixing "Date of Death" Message
The original message incorrectly stated that NA values were being removed.
Fix: Message now correctly states:
"Quality assurance: Date of death is invalid (on or before 1/1/1900 or after current date)"

🔗 Old repo reference

📌 3.2 Fixing Missing Data Handling
Issue: Patients with missing sex were marked as missing for the entire record, which was unintended.
Fix: The new script preserves records and only marks sex as missing.
🔗 Updated QA code

📌 3.3 Update to qa_bin_prostate_cancer
Issue:
The current method introduced missing values due to logic operations with NA values. NA values were assigned to alive patients who had no recorded prostate cancer diagnosis but were affected by the logic.
The check for cause of death due to prostate cancer was unnecessary. Patients who died from prostate cancer before the index date are already excluded later in the study. Removing this check ensures all alive participants receive a valid classification (TRUE/FALSE) rather than NA.
🔗 Updated method

🔹 4. fn-inex.R — Inclusion/Exclusion Criteria Fixes

📌 4.1 Code Corrections
The new repo includes inex_bin_alive, so we can use it directly instead of recalculating.
🔗 Old repo reference
🔗 New implementation

📌 4.2 Variable Renaming
death_date → cens_date_death
has_follow_up_previous_6months → inex_bin_6m_reg
deregistration_date → cens_date_dereg

📌 4.3 Removal of Unnecessary Code
Issue: The old repo had an unnecessary step for active registration at index.
Fix:
Removed redundant filtering for active registration, as inex_bin_6m_reg already ensures this. I rewrote the print message to reflect this.
🔗 Old repo reference
🔗 Updated function

📌 4.4 Fix for Exclusion Criteria in the Vax Cohort
Issue: Errors in the vax cohort exclusion criteria due to incorrect variable types.
Fix:
Redefined data types to ensure they are numeric before calculating vax_mixed.
🔗 Updated fix

🔹 5. Other Updates

📌 5.1 Cause of Death Extraction Simplified
A more efficient method for extracting cause of death has become available in OpenSAFELY.
Fix: Updated our code to align with the latest OpenSAFELY documentation.
🔗 Updated function

[ZoeMZou]

Hi @venexia,

This PR is now ready for your review. It runs successfully locally. Since the revisions in this script are quite detailed, I’ve listed the key changes at the beginning of the PR to make it easier to navigate and review.

Thank you very much.

Best wishes,
Zoe