Skip to content

Commit

Permalink
Merge branch 'dev' into bwmac/AG-1455/updates-README
Browse files Browse the repository at this point in the history
  • Loading branch information
BWMac committed Nov 6, 2024
2 parents 4e14185 + 6fd726d commit 7573e1a
Show file tree
Hide file tree
Showing 11 changed files with 1,165 additions and 7 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -139,3 +139,5 @@ test_staging_dir/

# dev config file
dev_config.yaml

.vscode/
10 changes: 5 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,25 +198,25 @@ These expectations are defined in the `/great_expectations/gx/plugins/expectatio

#### Nested Columns

If the transform includes nested columns (example: `druggability` column in `gene_info` tranform), please follow these steps:
1. Add the nested column name to the `gx_nested_columns` flag in the configuration file for the specific transform. This will convert the column values to a JSON parsable string.
If the transform includes nested columns (example: `druggability` column in `gene_info` tranform), please follow these four steps:
1. In the config file, add the nested column name to the `gx_nested_columns` flag for the specific transform. This will convert the column values to a JSON parsable string.
```
gx_nested_columns:
- <nested_column_name>
```
1. When creating the validator object in the gx_suite_definitions notebook, the nested column(s) must be included in the `nested_columns` list.
2. When creating the validator object in the gx_suite_definitions notebook, the nested column(s) must be included in the `nested_columns` list.
```
df = pd.read_json(<data_file>)
nested_columns = ['<nested_column_name>']
df = GreatExpectationsRunner.convert_nested_columns_to_json(df, nested_columns)
validator = context.sources.pandas_default.read_dataframe(df)
validator.expectation_suite_name = "<suite_name>"
```
1. When validating the value type of the nested column, make sure to specify it as a string (see Step 1 for reasoning):
3. When validating the value type of the nested column, specify it as a string (see Step 1 for reasoning):
```
validator.expect_column_values_to_be_of_type("<nested_column_name>", "str")
```
1. A JSON file containing the expected schema must be added here: `src/agoradatatools/great_expectations/gx/json_schemas/<transform_name>/<column_name>.json`. Use the [JSON schema tool](https://jsonschema.net/app/schemas/0) to create the schema template for your nested column.
4. A JSON file containing the expected schema must be added here: `src/agoradatatools/great_expectations/gx/json_schemas/<transform_name>/<column_name>.json`. Use the [JSON schema tool](https://jsonschema.net/app/schemas/0) to create the schema template for your nested column.

### DockerHub

Expand Down
269 changes: 269 additions & 0 deletions gx_suite_definitions/biomarkers.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,269 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import synapseclient\n",
"\n",
"import great_expectations as gx\n",
"import pandas as pd\n",
"import json\n",
"\n",
"context = gx.get_context(project_root_dir='../src/agoradatatools/great_expectations')\n",
"\n",
"from agoradatatools.gx import GreatExpectationsRunner\n",
"from expectations.expect_column_values_to_have_list_length import ExpectColumnValuesToHaveListLength\n",
"from expectations.expect_column_values_to_have_list_members import ExpectColumnValuesToHaveListMembers\n",
"from expectations.expect_column_values_to_have_list_members_of_type import ExpectColumnValuesToHaveListMembersOfType"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create Expectation Suite for Biomarkers Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"syn = synapseclient.Synapse()\n",
"syn.login()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"biomarkers_data_file = syn.get(\"syn63540269\").path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Validator Object on Data File"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_json(biomarkers_data_file)\n",
"nested_columns = ['points']\n",
"df = GreatExpectationsRunner.convert_nested_columns_to_json(df, nested_columns)\n",
"validator = context.sources.pandas_default.read_dataframe(df)\n",
"validator.expectation_suite_name = \"biomarkers\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Add Expectations to Validator Object For Each Column"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Get the list of unique values for fields\n",
"with open(\"../src/agoradatatools/great_expectations/gx/json_schemas/immunohisto/biomarkers_unique_field_values.json\", \"r\") as file:\n",
" unique_field_values = json.load(file)\n",
"print(unique_field_values)\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# points\n",
"validator.expect_column_values_to_be_of_type(\"points\", \"str\")\n",
"with open(\"../src/agoradatatools/great_expectations/gx/json_schemas/immunohisto/points.json\", \"r\") as file:\n",
" points_schema = json.load(file)\n",
"validator.expect_column_values_to_match_json_schema(\"points\", json_schema=points_schema)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# model\n",
"validator.expect_column_values_to_be_of_type(\"model\", \"str\")\n",
"validator.expect_column_values_to_not_be_null(\"model\")\n",
"\n",
"# List of accepted values\n",
"field_name = \"model\"\n",
"if field_name in unique_field_values:\n",
" validator.expect_column_distinct_values_to_be_in_set(field_name, unique_field_values[field_name])\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# type\n",
"validator.expect_column_values_to_be_of_type(\"type\", \"str\")\n",
"validator.expect_column_values_to_not_be_null(\"type\")\n",
"\n",
"# List of accepted values\n",
"field_name = \"type\"\n",
"if field_name in unique_field_values:\n",
" validator.expect_column_distinct_values_to_be_in_set(field_name, unique_field_values[field_name])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# units\n",
"validator.expect_column_values_to_be_of_type(\"units\", \"str\")\n",
"validator.expect_column_values_to_not_be_null(\"units\")\n",
"\n",
"# List of accepted values\n",
"field_name = \"units\"\n",
"if field_name in unique_field_values:\n",
" validator.expect_column_distinct_values_to_be_in_set(field_name, unique_field_values[field_name])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# age_death\n",
"validator.expect_column_values_to_be_of_type(\"age_death\", \"int\")\n",
"validator.expect_column_values_to_not_be_null(\"age_death\")\n",
"validator.expect_column_values_to_be_between(\"age_death\", strict_min_value=0, max_value=100)\n",
"\n",
"# List of accepted values\n",
"field_name = \"age_death\"\n",
"if field_name in unique_field_values:\n",
" validator.expect_column_distinct_values_to_be_in_set(field_name, unique_field_values[field_name])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# tissue\n",
"validator.expect_column_values_to_be_of_type(\"tissue\", \"str\")\n",
"validator.expect_column_values_to_not_be_null(\"tissue\")\n",
"\n",
"# List of accepted values\n",
"field_name = \"tissue\"\n",
"if field_name in unique_field_values:\n",
" validator.expect_column_distinct_values_to_be_in_set(field_name, unique_field_values[field_name])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# unique entries ExpectSelectColumnValuesToBeUniqueWithinRecord\n",
"validator.expect_select_column_values_to_be_unique_within_record(column_list=[\"model\", \"type\", \"age_death\", \"tissue\"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"validator.save_expectation_suite(discard_failed_expectations=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Checkpoint and View Results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"checkpoint = context.add_or_update_checkpoint(\n",
" name=\"agora-test-checkpoint\",\n",
" validator=validator,\n",
")\n",
"checkpoint_result = checkpoint.run()\n",
"context.view_validation_result(checkpoint_result)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build Data Docs - Click on Expectation Suite to View All Expectations"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"context.build_data_docs()\n",
"context.open_data_docs()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading

0 comments on commit 7573e1a

Please sign in to comment.