Iterative categorical matching

[Hussein-Mahfouz]

As described in #13

[sgreenbury]

Adding notes from discussion to check and implement, then can merge (happy to do both these):

• Confirm we can run for the whole population
• Add config table matching: for columns to include optional matching and a value for n_matches (if not passed, no threshold)

[Hussein-Mahfouz]

The new iterative_match_categorical function does not need to be iterative. We can run an exact match on specified column as shown in the last commit. We do the following:

• add all columns to required_columns
• keep optional_cols empty - that means we will only have one iteration see here
• make n_matches = 0

# Define required columns for matching
required_columns_fixed = [
    "number_adults",
    "number_children",
    "number_cars",
    "num_pension_age",
    "rural_urban_2_categories",
]

# Define optional columns in order of importance (most to least important)
optional_columns_fixed = []

# # Match using iterative matching
matcher_fixed = MatcherExact(
    df_pop=spc_matching,
    df_pop_id="hid",
    df_sample=nts_matching,
    df_sample_id="HouseholdID",
    matching_dict = matching_dfs_dict,
    fixed_cols=required_columns_fixed,
    optional_cols=optional_columns_fixed,
    n_matches=0,  
    chunk_size=50000,
    show_progress=True,
)

matches_hh_level_fixed = matcher_fixed.iterative_match_categorical()

[sgreenbury]

• also add docs in the config toml (base.toml) explaining order is the order in which features are dropped

[sgreenbury]

Great to have the test fixture and tests added here. Perhaps we can add a test for the match_psm too using this in a subsequent PR, I'll open an issue for this.

[sgreenbury]

Thanks for adding this @Hussein-Mahfouz, this looks great with the matching algorithm clearer, configurable and tested. As discussed, I'll go ahead and merge this now following running for Greater London as part of #58.