586 sic sut mapping #113
Merged
586 sic sut mapping #113
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhubbardONS
approved these changes
Oct 21, 2024
| } | ||
|
|
||
| for i in files_and_column_names: | ||
| mapping_path = mapping_folder + i |
There was a problem hiding this comment.
Minor comment: worth having something in the docstring to say the mapping_folder needs to have a '/' at the end for a readable file path? (or validation?)
There was a problem hiding this comment.
Updated the doc string to be more explicit about this, nice spot!
AntonZogk
pushed a commit
that referenced
this pull request
Oct 28, 2024
* Creating mapping validation function * Docstrings * Update docstring and leave unmatched as set * Update to raise warning and created wraper to test multiple mapping files in one go * adding test that passes when a warning raised * update docstring to ask that mapping file be a folder and not a file
AntonZogk
added a commit
that referenced
this pull request
Oct 29, 2024
* Build pbj-workbench-python3.9-standard.Dockerfile and test * Check files * Build full path instead of cd * Test whole process with 3.8 * Update push branch for testing * Navigate to project folder * Add ls for debugging * docker run -t cml_3.9 * Run docker in detached mode * Run docker in -i mode * Split process to many steps * Add matrix with cml versions * Typo matrix * Tupo matrix * Typo matrix * Use string type for versions * continue-on-error: true * Tidy up workflow Break steps Add 3.11 runtime Change name tags * Set wd in pytest * Add pre commit hooks * Use checkout v3 * Move files around with docker cp * Fix typo * Use checkouts * Copy files to container * Mount parent volume * Create container after checkout * Trigger on pull requests in main * 586 sic sut mapping (#113) * Creating mapping validation function * Docstrings * Update docstring and leave unmatched as set * Update to raise warning and created wraper to test multiple mapping files in one go * adding test that passes when a warning raised * update docstring to ask that mapping file be a folder and not a file * Correct design and calibration values (#108) *Add reusable function *Add test data *Add unit test *Add TODO to use this function in other parts of the pipeline * 632 module restructure (#116) * move files to correct location * change relevant module imports * Move files to apropiate folders, rename folders * Move all data to equivalent test folder structure * Update estimation test paths and imports * Update imputation test paths and imports * Update outlier detection test paths and imports * Update outpus test paths and imports * Update utilities test paths and imports * Remove tests/imputation/test_pivot_imputation_value.py * Run hooks * Add tests tree into tests readme * Run hooks * Remove duplicated test data * These were copied instead of moved, hence duplicated * these files aren't needed and covered by other tests * update tree --------- Co-authored-by: Wil Roberts <[email protected]> * Use list, passing dict not support in pandas 2.1.4 * Enforce constrain marker str type in tests * Create separate job for pre commit hooks * Use legacy job for hooks * Run hooks * Pre commit mig config, user python instead of python3 * Use python 3.9 for hooks * Use 3.10 for hooks * Use python 3.10.13 --------- Co-authored-by: Jordan-Day-ONS <[email protected]> Co-authored-by: Wil Roberts <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Functions created to validate mapping files against input data. will produce a warning if there is unmatched data in the dataframe. No warning or errors if unused mappings are present in the mapping file
Checklists
This pull request meets the following requirements:
If you feel some of these conditions do not apply for this pull request, please
add a comment to explain why.