pretraining/notebooks/03_create-pretraining-data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Pretraining Data\n",
    "Before starting the pretraining: \n",
    "\n",
    "1. Convert the text-reports to file-format, see [here](https://github.com/kbressem/bert-for-radiology/blob/master/pretraining/sentencizing.ipynb).    \n",
    "2. Create the custom vocabulary file, see [here](https://github.com/kbressem/bert-for-radiology/blob/master/pretraining/bert-vocab-builder.ipynb).   \n",
    "3. Create pretraining data with a [script](https://github.com/google-research/bert/blob/master/create_pretraining_data.py) supplied by Google. \n",
    "\n",
    "The  `create_pretraining_data.py` was copied to this directory, so it can be executed locally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initalizing the enviroment\n",
    "\n",
    "The BERT vocabulary Anaconda enviroment from the [bert-custom-vocabulary.ipyn](https://github.com/kbressem/bert-for-radiology/blob/master/pretraining/bert-vocab-builder.ipynb) will be used. \n",
    "\n",
    "The 3.6 GB grew to over 100 GB in memory, because of which the data needed to be split using the `split` bash command.  To split each 1,000,000 lines, the following code was used:\n",
    "\n",
    "```bash\n",
    "  split -l 1000000 report-dump.txt report-data-chunk-\n",
    "```\n",
    "\n",
    "The output are multiple 1,000,000-line files named: `report-data-chunk-aa`, `report-data-chunk-ab`, ... , `report-data-chunk-zz`  \n",
    "In our case, it resulted in 55 files. \n",
    "\n",
    "In order to create the pretraining data from the first split, one must execute the Google script with the following command. \n",
    "\n",
    "```bash\n",
    "python create_pretraining_data.py \\\n",
    "    --input_file='../data/small-splits/report-data-chunk-aa' \\\n",
    "    --output_file='../tmp/tf_examples.tfrecord' \\\n",
    "    --vocab_file='../vocab.txt' \\\n",
    "    --do_lower_case=False \\\n",
    "    --max_seq_length=128 \\\n",
    "    --max_predictions_per_seq=20 \\\n",
    "    --masked_lm_prob=0.15 \\\n",
    "    --random_seed=12345 \\\n",
    "    --dupe_factor=5\n",
    " ```\n",
    " \n",
    "To advoid copy-pasting the code multiple times for all data splits, we used a bash file which runs a loop over all data files stored in `../data/small-splits/` and stores the output in `../tmp`. \n",
    "The script also contains a logger, telling which file is currently processed and how many more remain."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}