Nadia Wood and Diana Gonzalez Santillan
Video Presentation Link => Access here
Silde Deck => Access here
Our Paper => Access here
The code in this repository is an adaptation of the original code sent to us by Xianli Zhang through email ([email protected]), downloadable from this Google Drive folder: https://drive.google.com/file/d/1hfhM93zu_pc-SC2ppC6PTp5cFdfLjnsG/view?usp=sharing
Our work is based on a paper by Xianli Zhang et al. from Xi'an Jiaotong University, in Xi'an, China:
Zhang, Xianli, et al. “INPREM: An Interpretable and Trustworthy Predictive Model for Healthcare.” Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020, https://doi.org/10.1145/3394486.3403087
In order to run the code provided by the authors of the paper, we had to install the following dependencies. We used Visual Studio Code to import the project, and then installed the following required libraries:
sudo pip3 install torch torchvision
sudo pip3 install Cython
sudo pip3 install torchsparseattn
sudo pip3 install pandas
sudo pip3 install torchmetrics
sudo pip3 install torchsummary
sudo pip3 install tensorboardOnce the dependencies are installed you can run the preprocessing, training, and evaluation code by executing the follwing command:
python3 main.py --task=[TASK] --emb_dim=256 --d_k=256 --d_v=256 --d_inner=256where [TASK] must be one of diagnoses, or heart. Note: Original code included diabetes and kidney disease tasks as well, but we have excluded that from our reproduction for simplicity.
Feel free to change any of the other hyperparameters to see the changes in the results of the model. Here is the full list of options available for you to run the model, along with their default values:
--task, choices=('diagnoses', 'heart'), default='heart', task to run
--data_root, type=str, default='../datasets/', dataset root directory
--fold, choices=(1, 2, 3, 4, 5), default=1, number of fold
--use_cuda, action='store_true', default=False, if use GPU
--gpu_devices, type=str, default='0', device IDs for GPU
--epochs, type=int, default=25, number of epochs
--batch_size, type=int, default=32
--drop_rate, type=float, default=0.5
--optimizer, choices=('Adam', 'SGD', 'Adadelta'), required=False, default='Adam'
--lr, type=float, default=5e-4, learning rate
--weight_decay, type=float, default=1e-4
--n_head, type=int, default=2, number of head of self-attention for the visit attention
--n_depth, type=int, default=2, number of layers of self-attention for the visit attention
--emb_dim, type=int, default=128, size of medical variable (or code) embedding.
--d_k, type=int, default=128, size of vector before self attention
--d_v, type=int, default=128, size of vector before self attention
--d_inner, type=int, default=128
--dvp, action='store_true', default=False, Weather use position embedding
--dp, action='store_true', default=False, Weather use position embedding
--ds, action='store_true', default=False, whether delete the sparse_max
--cap_uncertainty, action='store_true', Weather capture uncertainty, default=True
--monto_carlo_for_aleatoric, type=int, default=100, size of Monto Carlo Sample
--monto_carlo_for_epistemic, type=int, default=200, size of Monto Carlo Sample
--analysis_dir, type=str, default='../../output_for_analysis_final/', nalysis output dir
--write_performance, action='store_true', default=False, Weather write performance result
--performance_dir, type=str, default='../../metric_results/', performance dir
--save_model_dir, type=str, default='../../saved_model', set dir to save the model which has the best performance
--resume, type=str, default=None, Choose the model dict to load for test or fine-tune
--data_scale, default=1, type=float
NOTE: If you plan to use GPU computation, install CUDA: https://developer.nvidia.com/cuda-downloads and include the --use_cuda=True flag.
We used the publicly available MIMIC III dataset to acquire the diagnosis codes dataset, and the Heart Failure dataset needed for the project. Specifically, we followed these instructions to access the MIMIC III demo data: https://mimic.mit.edu/docs/gettingstarted/cloud/bigquery/
Then, we used the following queries to get the specific datasets we needed for the project:
SELECT * FROM 'physionet-data.mimiciii_demo.admissions
SELECT * FROM 'physionet-data.mimiciii_demo.diagnoses_icd
SELECT c.SUBJECT_ID, c.CHARTDATE, c.CPT_CD,
CASE WHEN d.ICD9_CODE IS null THEN 0 ELSE 1 END AS HAS_DIAG
FROM 'physionet-data.mimiciii_demo.cptevents' c
LEFT JOIN 'physionet-data.mimiciii_demo.diagnoses_icd' d
ON c.SUBJECT_ID=d.SUBJECT_ID and d.ICD9_CODE='42731'
WHERE c.CHARTDATE IS NOT null
ORDER BY c.SUBJECT_ID, c.CHARTDATE
Mimiciii Demo Admissions Data Stats
mimiciiiDemoData.csv contains demo data from the table mimiciii_demo.admissions. Using panda-profiling we were quickly able to get a sense of the data and statistics about this dataset. A detailed report is avaiable here=> Data Profiling Report
Diagnosis Code Data Stats
A detailed report is avaiable here=> Data Profiling Report
Heart Disease Data Stats
A detailed report is avaiable here=> Data Profiling Report
Alerts for the dataset that was used to train the model. High cerdinality and duplicate rows were found in the dataset.
Data for training model: We split subject IDs into 3 groups (75% train, 10% valid, 15% test)
Model analysis results are in the folder=>Results
| Epocs | Batch Size | Drop Rate | Learning Rate | Weight Decay | Accuracy | F1 Score |
|---|---|---|---|---|---|---|
| 5 | 32 | 0.5 | 0.0005 | 0.0001 | 43% | 0.33 |
| 10 | 32 | 0.5 | 0.0005 | 0.0001 | 86% | 0.89 |
| 15 | 32 | 0.5 | 0.0005 | 0.0001 | 71% | 0.75 |
| 20 | 32 | 0.5 | 0.0005 | 0.0001 | 57% | 0.73 |
| 25 | 32 | 0.5 | 0.0005 | 0.0001 | 57% | 0.67 |
| 30 | 32 | 0.5 | 0.0005 | 0.0001 | 57% | 0.40 |
Inprem(
(embedding): Linear(in_features=2, out_features=128, bias=False)
(position_embedding): Embedding(35, 128)
(encoder): Encoder(
(layer_stack): ModuleList(
(0): EncoderLayer(
(slf_attn): MultiHeadAttention(
(w_qs): Linear(in_features=128, out_features=256, bias=True)
(w_ks): Linear(in_features=128, out_features=256, bias=True)
(w_vs): Linear(in_features=128, out_features=256, bias=True)
(attention): ScaledDotProductAttention(
(dropout): Dropout(p=0.5, inplace=False)
(softmax): Softmax(dim=2)
)
(layer_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
(fc): Linear(in_features=256, out_features=128, bias=True)
(dropout): Dropout(p=0.5, inplace=False)
)
(pos_ffn): PositionwiseFeedForward(
(w_1): Conv1d(128, 128, kernel_size=(1,), stride=(1,))
(w_2): Conv1d(128, 128, kernel_size=(1,), stride=(1,))
(layer_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.5, inplace=False)
)
)
(1): EncoderLayer(
(slf_attn): MultiHeadAttention(
(w_qs): Linear(in_features=128, out_features=256, bias=True)
(w_ks): Linear(in_features=128, out_features=256, bias=True)
(w_vs): Linear(in_features=128, out_features=256, bias=True)
(attention): ScaledDotProductAttention(
(dropout): Dropout(p=0.5, inplace=False)
(softmax): Softmax(dim=2)
)
(layer_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
(fc): Linear(in_features=256, out_features=128, bias=True)
(dropout): Dropout(p=0.5, inplace=False)
)
(pos_ffn): PositionwiseFeedForward(
(w_1): Conv1d(128, 128, kernel_size=(1,), stride=(1,))
(w_2): Conv1d(128, 128, kernel_size=(1,), stride=(1,))
(layer_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.5, inplace=False)
)
)
)
)
(w_alpha_1): Linear(in_features=128, out_features=1, bias=True)
(w_alpha_2): Linear(in_features=128, out_features=1, bias=True)
(w_beta): Linear(in_features=128, out_features=128, bias=True)
(variance): Linear(in_features=128, out_features=1, bias=True)
(predict): Linear(in_features=128, out_features=2, bias=True)
(dropout): Dropout(p=0.5, inplace=False)
(sparsemax): Sparsemax()
)
Note: We did not choose to contact Professor Sum as we were able to find a way to wrangle the data they way we needed for it to work with the model.