Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TotalEDInpatientVisits, TotalEDVisits, TotalInpatientVisits - incorrect treatment of >9 at several endpoints #297

Closed
karafecho opened this issue Oct 27, 2023 · 5 comments
Closed

TotalEDInpatientVisits, TotalEDVisits, TotalInpatientVisits - incorrect treatment of >9 at several endpoints #297

karafecho opened this issue Oct 27, 2023 · 5 comments
Assignees
@hyi

Comments

@karafecho
Copy link
Contributor

This issue is to report that the variables TotalEDInpatientVisits, TotalEDVisits, and TotalInpatientVisits are not treating >9 correctly at the Discover Cohort, Features, and Association to All Features endpoint.

Discover Cohort - note that one can correctly create a cohort in which TotalEDInpatientVisits > 9, but for logical consistency, we should not allow users to create cohorts defined as, e.g., TotalEDInpatientVisits > 10.

curl -X 'POST' \
  'https://icees-pcd.renci.org/patient/cohort' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{"TotalEDInpatientVisits":{"operator":">","value":"10"}}'

  "return value": {
    "cohort_id": "COHORT:3",
    "size": 527
  }

Features

curl -X 'GET' \
  'https://icees-pcd.renci.org/patient/cohort/COHORT%3A1/features?year=2020' \
  -H 'accept: text/tabular'

+-----------------------------+---------+
| feature                     | count   |
+=============================+=========+
| TotalEDInpatientVisits = 0  | 222447  |
|                             | 66.27%  |
+-----------------------------+---------+
| TotalEDInpatientVisits = 1  | 40211   |
|                             | 11.98%  |
+-----------------------------+---------+
| TotalEDInpatientVisits = 2  | 21748   |
|                             | 6.48%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 3  | 13456   |
|                             | 4.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 4  | 9319    |
|                             | 2.78%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 5  | 6335    |
|                             | 1.89%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 6  | 4593    |
|                             | 1.37%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 7  | 3441    |
|                             | 1.03%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 8  | 2523    |
|                             | 0.75%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 9  | 2067    |
|                             | 0.62%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = >9 | 0       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 10 | 1793    |
|                             | 0.53%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 11 | 1309    |
|                             | 0.39%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 12 | 1144    |
|                             | 0.34%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 13 | 918     |
|                             | 0.27%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 14 | 590     |
|                             | 0.18%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 15 | 762     |
|                             | 0.23%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 16 | 482     |
|                             | 0.14%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 17 | 439     |
|                             | 0.13%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 18 | 341     |
|                             | 0.10%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 19 | 209     |
|                             | 0.06%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 20 | 230     |
|                             | 0.07%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 21 | 183     |
|                             | 0.05%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 22 | 120     |
|                             | 0.04%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 23 | 198     |
|                             | 0.06%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 24 | 153     |
|                             | 0.05%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 25 | 54      |
|                             | 0.02%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 26 | 90      |
|                             | 0.03%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 27 | 76      |
|                             | 0.02%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 28 | 38      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 29 | 28      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 30 | 63      |
|                             | 0.02%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 31 | 47      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 32 | 34      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 33 | 39      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 34 | 25      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 35 | 31      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 37 | 33      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 38 | 10      |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 39 | 6       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 41 | 22      |
|                             | 0.01%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 42 | 16      |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 44 | 12      |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 45 | 12      |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 48 | 8       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 51 | 3       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 55 | 6       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 63 | 1       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 66 | 8       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 70 | 8       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 83 | 4       |
|                             | 0.00%   |
+-----------------------------+---------+
| TotalEDInpatientVisits = 94 | 1       |
|                             | 0.00%   |
+-----------------------------+---------+

Association to All Features

curl -X 'POST' \
  'https://icees-pcd.renci.org/patient/cohort/COHORT%3A1/associations_to_all_features' \
  -H 'accept: text/tabular' \
  -H 'Content-Type: application/json' \
  -d '{
  "feature": {
    "TotalEDInpatientVisits": {
      "operator": "=",
      "value": "0"
    }
  },
  "maximum_p_value": 1,
  "correction": {
    "method": "bonferroni"
  }
}'


+-----------------------------+------------------------------+-------------------------------+---------+
| feature                     | TotalEDInpatientVisits = 0   | TotalEDInpatientVisits <> 0   |         |
+=============================+==============================+===============================+=========+
| TotalEDInpatientVisits = 0  | 30370    100.00%             | 0      0.00%                  | 30370   |
|                             | 100.00%  66.79%              | 0.00%  0.00%                  | 66.79%  |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 1  | 0      0.00%                 | 5619    100.00%               | 5619    |
|                             | 0.00%  0.00%                 | 37.20%  12.36%                | 12.36%  |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 2  | 0      0.00%                 | 2943    100.00%               | 2943    |
|                             | 0.00%  0.00%                 | 19.48%  6.47%                 | 6.47%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 3  | 0      0.00%                 | 1791    100.00%               | 1791    |
|                             | 0.00%  0.00%                 | 11.86%  3.94%                 | 3.94%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 4  | 0      0.00%                 | 1217   100.00%                | 1217    |
|                             | 0.00%  0.00%                 | 8.06%  2.68%                  | 2.68%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 5  | 0      0.00%                 | 815    100.00%                | 815     |
|                             | 0.00%  0.00%                 | 5.40%  1.79%                  | 1.79%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 6  | 0      0.00%                 | 591    100.00%                | 591     |
|                             | 0.00%  0.00%                 | 3.91%  1.30%                  | 1.30%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 7  | 0      0.00%                 | 439    100.00%                | 439     |
|                             | 0.00%  0.00%                 | 2.91%  0.97%                  | 0.97%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 8  | 0      0.00%                 | 314    100.00%                | 314     |
|                             | 0.00%  0.00%                 | 2.08%  0.69%                  | 0.69%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = 9  | 0      0.00%                 | 261    100.00%                | 261     |
|                             | 0.00%  0.00%                 | 1.73%  0.57%                  | 0.57%   |
+-----------------------------+------------------------------+-------------------------------+---------+
| TotalEDInpatientVisits = >9 | 0      null                  | 0      null                   | 0       |
|                             | 0.00%  0.00%                 | 0.00%  0.00%                  | 0.00%   |
+-----------------------------+------------------------------+-------------------------------+---------+
|                             | 30370                        | 15104                         | 45474   |
|                             | 66.79%                       | 33.21%                        | 100.00% |
+-----------------------------+------------------------------+-------------------------------+---------+
@hyi hyi mentioned this issue Oct 31, 2023
Closed
@karafecho
Copy link
Contributor Author

karafecho commented Nov 1, 2023 •
edited
Loading

Per discussion with Hong:

Leave Discover Cohort endpoint as is; in other words, allow users to define cohorts per their choice.

Fix >9 issues with Features and Association to All Features endpoints; issue involves db and YAML file.

@karafecho
Copy link
Contributor Author

Update:

Turns out that this is a very complicated issue. In brief, AgeStudyStart and AgeStudyStart2 are binned in FHIR PIT as part of preprocAge() routine. For example, for AgeStudyStart2, the bin labels are enumerated as '<5', '5-17', '18-44', '45-64', '65-89', which are strings stored in the ICEES+ database ingested from patient data generated from FHIR PIT. Currently, if an operator is not specified as part of user input, then all feature variables are compared with an equal operator for feature associations and multilevel analysis endpoints. In fact, the values or levels to compare with the equal operator for each feature variable are specified in the all_features YAML files, either through enumeration or minimum/maximum specifications. As a result, AgeStudyStart2, which includes a bin of <5, is stored in the database as a string and compared with <5 level string specified in the all_features YAML file with the == equal operator. On the other hand, TotalEDInpatientVisits, TotalEDVisits, and TotalInpatientVisits are enumerated as integers and treated as == in the same way as other feature variables. We tried to include >9 in the all_features YAML file for visit variables and also tried to parse it to apply > operator to aggregate visit variable integers into >9 bins for feature association and multivariate endpoints, but that raises the conflict with the string comparison with <5 for age variables, which should be compared with the equal operator instead. This complexity made us examine the current design and implementation for binning between FHIR -PIT, all_features YAML specification, and ICEES+ applications. If we want to expand the current design and implementation to allow > and < operators, in addition to == operator, then how can we specify this in all_features YAML file to inform the different operator comparisons? Currently, in the all_features YAML files, if we change the enum to min/max (range) and set it to min=0 and max = 100, then 0 through 100 will be returned, regardless of whether the values are null. Is this desired behavior? How can we improve it to make the specification more consistent across different feature variables to address different use cases?

@hyi hyi mentioned this issue Nov 13, 2023
Merged
@karafecho
Copy link
Contributor Author

karafecho commented Dec 2, 2023 •
edited
Loading

Adding on to this ticket ... For the Features endpoint (but not the other endpoints), the output for the visit variables is binned as 0 ... 9 not 0 ... 9, >9 in the pre 2014 tables. A quick review suggests that the issue is specific to the visit variables. Any ideas why?

2014

+----------------------------+---------+
| feature                    | count   |
+============================+=========+
| TotalEDInpatientVisits = 0 | 17679   |
|                            | 64.24%  |
+----------------------------+---------+
| TotalEDInpatientVisits = 1 | 4355    |
|                            | 15.82%  |
+----------------------------+---------+
| TotalEDInpatientVisits = 2 | 2030    |
|                            | 7.38%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 3 | 1084    |
|                            | 3.94%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 4 | 651     |
|                            | 2.37%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 5 | 540     |
|                            | 1.96%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 6 | 250     |
|                            | 0.91%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 7 | 223     |
|                            | 0.81%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 8 | 146     |
|                            | 0.53%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 9 | 96      |
|                            | 0.35%   |
+----------------------------+---------+
| TotalEDInpatientVisits > 9 | 467     |
|                            | 1.70%   |
+----------------------------+---------+

2013

+----------------------------+---------+
| feature                    | count   |
+============================+=========+
| TotalEDInpatientVisits = 0 | 16488   |
|                            | 100.00% |
+----------------------------+---------+
| TotalEDInpatientVisits = 1 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 2 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 3 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 4 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 5 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 6 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 7 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 8 | 0       |
|                            | 0.00%   |
+----------------------------+---------+
| TotalEDInpatientVisits = 9 | 0       |
|                            | 0.00%   |
+----------------------------+---------+

2014

+----------------------------+-----------------+------------------+---------+
| feature                    | Sex2 = Female   | Sex2 <> Female   |         |
+============================+=================+==================+=========+
| TotalEDInpatientVisits = 0 | 1255    57.46%  | 929     42.54%   | 2184    |
|                            | 64.79%  37.89%  | 67.56%  28.05%   | 65.94%  |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 1 | 310     60.19%  | 205     39.81%   | 515     |
|                            | 16.00%  9.36%   | 14.91%  6.19%    | 15.55%  |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 2 | 149    64.50%   | 82     35.50%    | 231     |
|                            | 7.69%  4.50%    | 5.96%  2.48%     | 6.97%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 3 | 69     55.65%   | 55     44.35%    | 124     |
|                            | 3.56%  2.08%    | 4.00%  1.66%     | 3.74%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 4 | 42     56.00%   | 33     44.00%    | 75      |
|                            | 2.17%  1.27%    | 2.40%  1.00%     | 2.26%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 5 | 31     53.45%   | 27     46.55%    | 58      |
|                            | 1.60%  0.94%    | 1.96%  0.82%     | 1.75%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 6 | 21     77.78%   | 6      22.22%    | 27      |
|                            | 1.08%  0.63%    | 0.44%  0.18%     | 0.82%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 7 | 11     45.83%   | 13     54.17%    | 24      |
|                            | 0.57%  0.33%    | 0.95%  0.39%     | 0.72%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 8 | 13     81.25%   | 3      18.75%    | 16      |
|                            | 0.67%  0.39%    | 0.22%  0.09%     | 0.48%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 9 | 6      54.55%   | 5      45.45%    | 11      |
|                            | 0.31%  0.18%    | 0.36%  0.15%     | 0.33%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits > 9 | 30     63.83%   | 17     36.17%    | 47      |
|                            | 1.55%  0.91%    | 1.24%  0.51%     | 1.42%   |
+----------------------------+-----------------+------------------+---------+
|                            | 1937            | 1375             | 3312    |
|                            | 58.48%          | 41.52%           | 100.00% |
+----------------------------+-----------------+------------------+---------

2013

+----------------------------+-----------------+------------------+---------+
| feature                    | Sex2 = Female   | Sex2 <> Female   |         |
+============================+=================+==================+=========+
| TotalEDInpatientVisits = 0 | 1033     58.49% | 733      41.51%  | 1766    |
|                            | 100.00%  58.49% | 100.00%  41.51%  | 100.00% |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 1 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 2 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 3 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 4 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 5 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 6 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 7 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 8 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits = 9 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
| TotalEDInpatientVisits > 9 | 0      null     | 0      null      | 0       |
|                            | 0.00%  0.00%    | 0.00%  0.00%     | 0.00%   |
+----------------------------+-----------------+------------------+---------+
|                            | 1033            | 733              | 1766    |
|                            | 58.49%          | 41.51%           | 100.00% |
+----------------------------+-----------------+------------------+---------+

How come the features endpoint is not returning the >9 bin with a total of zero, but the 1 x N endpoint does? I only see this in the pre-2014 tables (see examples above). This is by no means a high-priority issue, but I feel like this is an inconsistency that we should probably resolve, as I'm a bit worried that it might be indicative of something bigger.

@hyi
Copy link
Collaborator

hyi commented Dec 4, 2023

This issue should be fixed which can be tested on the ICEES PCD dev instance.

@hyi hyi mentioned this issue Dec 6, 2023
Merged
@karafecho
Copy link
Contributor Author

Closing issue, but first noting that Hong and I agreed to leave the Discover Cohort as is, in order to provide users with flexibility ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hyi hyi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hyi @karafecho