practice1_bert_nlu_solution.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python37764bit0813dialoguesystemcondad6fc42c0cb704d67bf0b79f4f56e2656",
   "display_name": "Python 3.7.7 64-bit ('0813_dialogue_system': conda)"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practice 1. BERT를 이용한 NLU 실습\n",
    "\n",
    "# 목차\n",
    "## Step 1. 데이터 전처리 (Data Preprocessing)\n",
    "## Step 2. BERT 세부조정하기 (Fine-tuning)\n",
    "## Step 3. 학습한 모델 평가하기\n",
    "\n",
    "최근 자연어처리 모듈에서 높은 성능을 보이는 대규모 사전학습 언어모델의 하나인 BERT를 활용하여 대화시스템의 모듈 중 하나인 NLU를 구현해보는 것이 이번 실습의 목표입니다.\n",
    "\n",
    "## Step 0. 사전설정\n",
    "아래 코드 블럭에서는 이 실습을 수행하기 위해 필요한 모듈들을 import합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# For data preprocessing\n",
    "import json\n",
    "import os\n",
    "import zipfile\n",
    "import sys\n",
    "from collections import Counter\n",
    "from nltk.tokenize import word_tokenize\n",
    "import nltk\n",
    "nltk.download('punkt')\n",
    "\n",
    "# For Bert pretraining and postprecessing\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "import random\n",
    "import numpy as np\n",
    "import torch\n",
    "from tqdm import tqdm\n",
    "from transformers import AdamW, get_linear_schedule_with_warmup\n",
    "from convlab2.nlu.jointBERT.dataloader import Dataloader\n",
    "from convlab2.nlu.jointBERT.jointBERT import JointBERT\n",
    "\n",
    "CUDA_IDX = '1'\n",
    "os.environ['CUDA_VISIBLE_DEVICES'] = CUDA_IDX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래 코드 블럭에서는 다른 코드 구현을 간편하게 해주는 helper function들을 정의합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def set_seed(seed):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "\n",
    "\n",
    "def read_zipped_json(filepath, filename):\n",
    "    archive = zipfile.ZipFile(filepath, 'r')\n",
    "    return json.load(archive.open(filename))\n",
    "\n",
    "\n",
    "def phrase_in_utt(phrase, utt):\n",
    "    phrase_low = phrase.lower()\n",
    "    utt_low = utt.lower()\n",
    "    return (' ' + phrase_low in utt_low) or utt_low.startswith(phrase_low)\n",
    "\n",
    "\n",
    "def phrase_idx_utt(phrase, utt):\n",
    "    phrase_low = phrase.lower()\n",
    "    utt_low = utt.lower()\n",
    "    if ' ' + phrase_low in utt_low or utt_low.startswith(phrase_low):\n",
    "        return get_idx(phrase_low, utt_low)\n",
    "    return None\n",
    "\n",
    "\n",
    "def get_idx(phrase, utt):\n",
    "    char_index_begin = utt.index(phrase)\n",
    "    char_index_end = char_index_begin + len(phrase)\n",
    "    word_index_begin = len(utt[:char_index_begin].split())\n",
    "    word_index_end = len(utt[:char_index_end].split()) - 1\n",
    "    return word_index_begin, word_index_end\n",
    "\n",
    "\n",
    "def da2triples(dialog_act):\n",
    "    triples = []\n",
    "    for intent, svs in dialog_act.items():\n",
    "        for slot, value in svs:\n",
    "            triples.append([intent, slot, value])\n",
    "    return triples\n",
    "\n",
    "\n",
    "def das2tags(sen, das):\n",
    "    tokens = word_tokenize(sen)\n",
    "    new_sen = ' '.join(tokens)\n",
    "    new_das = {}\n",
    "    span_info = []\n",
    "    intents = []\n",
    "    for da, svs in das.items():\n",
    "        new_das.setdefault(da, [])\n",
    "        if da == 'inform':\n",
    "            for s, v in svs:\n",
    "                v = ' '.join(word_tokenize(v))\n",
    "                if v != 'dontcare' and phrase_in_utt(v, new_sen):\n",
    "                    word_index_begin, word_index_end = phrase_idx_utt(v, new_sen)\n",
    "                    span_info.append((da, s, v, word_index_begin, word_index_end))\n",
    "                else:\n",
    "                    intents.append(da + '+' + s + '*' + v)\n",
    "                new_das[da].append([s, v])\n",
    "        else:\n",
    "            for s, v in svs:\n",
    "                new_das[da].append([s, v])\n",
    "                intents.append(da + '+' + s + '*' + v)\n",
    "    tags = []\n",
    "    for i, _ in enumerate(tokens):\n",
    "        for span in span_info:\n",
    "            if i == span[3]:\n",
    "                tag = \"B-\" + span[0] + \"+\" + span[1]\n",
    "                tags.append(tag)\n",
    "                break\n",
    "            if span[3] < i <= span[4]:\n",
    "                tag = \"I-\" + span[0] + \"+\" + span[1]\n",
    "                tags.append(tag)\n",
    "                break\n",
    "        else:\n",
    "            tags.append(\"O\")\n",
    "\n",
    "    return tokens, tags, intents, da2triples(new_das)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1. 데이터 전처리 (Data Preprocessing)\n",
    "\n",
    "- BERT는 토큰화된 자연어(Natural Language)들을 입력받는 모델인데, NLU의 예측 타겟(prediction target)들은 모두 구조화된 데이터입니다. \n",
    "- 데이터 전처리 단계에서는 특수토큰(special token)들을 이용하여 구조화된 데이터를 자연어로 바꿔주는 처리를 합니다.\n",
    "- 또한 원래 하나였던 데이터셋을 train, validation, test 셋으로 분리합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "먼저 아래 코드블럭에서는 실습에서 사용할 데이터셋인 Camrest 데이터셋을 불러옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cur_dir = os.path.abspath(os.curdir)\n",
    "data_dir = \"ConvLab-2/data/camrest\"\n",
    "processed_data_dir = os.path.join(cur_dir, 'data/all_data')\n",
    "if not os.path.exists(processed_data_dir):\n",
    "    os.makedirs(processed_data_dir)\n",
    "\n",
    "data_key = ['train', 'val', 'test']\n",
    "data = {}\n",
    "for key in data_key:\n",
    "    data[key] = read_zipped_json(os.path.join(data_dir, key + '.json.zip'), key + '.json')\n",
    "    print('load {}, size {}'.format(key, len(data[key])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래 코드블럭에서는 본격적으로 전처리 작업을 진행합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mode = 'all'\n",
    "processed_data = {}\n",
    "all_da = []\n",
    "all_intent = []\n",
    "all_tag = []\n",
    "context_size = 3\n",
    "for key in data_key:\n",
    "    processed_data[key] = []\n",
    "    for dialog in data[key]:\n",
    "        context = []\n",
    "        for turn in dialog['dial']:\n",
    "            if mode == 'usr' or mode == 'all':\n",
    "                tokens, tags, intents, new_das = das2tags(turn['usr']['transcript'], turn['usr']['dialog_act'])\n",
    "\n",
    "                processed_data[key].append([tokens, tags, intents, new_das, context[-context_size:]])\n",
    "\n",
    "                all_da += [da for da in turn['usr']['dialog_act']]\n",
    "                all_intent += intents\n",
    "                all_tag += tags\n",
    "\n",
    "            context.append(turn['usr']['transcript'])\n",
    "\n",
    "            if mode == 'sys' or mode == 'all':\n",
    "                tokens, tags, intents, new_das = das2tags(turn['sys']['sent'], turn['sys']['dialog_act'])\n",
    "\n",
    "                processed_data[key].append([tokens, tags, intents, new_das, context[-context_size:]])\n",
    "                all_da += [da for da in turn['sys']['dialog_act']]\n",
    "                all_intent += intents\n",
    "                all_tag += tags\n",
    "\n",
    "            context.append(turn['sys']['sent'])\n",
    "    \n",
    "    all_da = [x[0] for x in dict(Counter(all_da)).items() if x[1]]\n",
    "    all_intent = [x[0] for x in dict(Counter(all_intent)).items() if x[1]]\n",
    "    all_tag = [x[0] for x in dict(Counter(all_tag)).items() if x[1]]\n",
    "\n",
    "    print('loaded {}, size {}'.format(key, len(processed_data[key])))\n",
    "    json.dump(processed_data[key], \n",
    "              open(os.path.join(processed_data_dir, '{}_data.json'.format(key)), 'w'),\n",
    "              indent=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래 코드블럭에서는 전처리 작업결과를 각각 json 파일에 저장합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('dialog act num:', len(all_da))\n",
    "print('sentence label num:', len(all_intent))\n",
    "print('tag num:', len(all_tag))\n",
    "json.dump(all_da, open(os.path.join(processed_data_dir, 'all_act.json'), 'w'), indent=2)\n",
    "json.dump(all_intent, open(os.path.join(processed_data_dir, 'intent_vocab.json'), 'w'), indent=2)\n",
    "json.dump(all_tag, open(os.path.join(processed_data_dir, 'tag_vocab.json'), 'w'), indent=2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2. BERT 세부조정하기 (Fine-tuning)\n",
    "Pre-training된 BERT의 파라메터를 불러온 뒤 Camrest 데이터셋을 맞게 파라메터를 Fine-tuning 하는 부분을 실습해봅시다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2.1 환경설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "### Configurations ###\n",
    "main_dir = processed_data_dir\n",
    "config = {\n",
    "  \"data_dir\": main_dir,\n",
    "  \"output_dir\": main_dir + \"/output/all\",\n",
    "  \"zipped_model_path\": main_dir + \"/output/all/bert_camrest_all.zip\",\n",
    "  \"log_dir\": main_dir + \"/log/all\",\n",
    "  \"DEVICE\": \"cuda:\"+CUDA_IDX,\n",
    "  \"seed\": 2019,\n",
    "  \"cut_sen_len\": 40,\n",
    "  \"use_bert_tokenizer\": True,\n",
    "  \"model\": {\n",
    "    \"finetune\": True,\n",
    "    \"context\": False,\n",
    "    \"context_grad\": False,\n",
    "    \"pretrained_weights\": \"bert-base-uncased\",\n",
    "    \"check_step\": 1000,\n",
    "    \"max_step\": 10000,\n",
    "    \"batch_size\": 20,\n",
    "    \"learning_rate\": 3e-5,\n",
    "    \"adam_epsilon\": 1e-8,\n",
    "    \"warmup_steps\": 0,\n",
    "    \"weight_decay\": 0.0,\n",
    "    \"dropout\": 0.1,\n",
    "    \"hidden_units\": 768\n",
    "  }\n",
    "}\n",
    "\n",
    "data_dir = config['data_dir']\n",
    "output_dir = config['output_dir']\n",
    "log_dir = config['log_dir']\n",
    "DEVICE = config['DEVICE']\n",
    "\n",
    "set_seed(config['seed'])\n",
    "\n",
    "print('-' * 20 + 'dataset:camrest' + '-' * 20)\n",
    "from convlab2.nlu.jointBERT.camrest.postprocess import is_slot_da, calculateF1, recover_intent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2.2 Pre-trained BERT 모델 불러오기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [],
   "source": [
    "intent_vocab = json.load(open(os.path.join(data_dir, 'intent_vocab.json')))\n",
    "tag_vocab = json.load(open(os.path.join(data_dir, 'tag_vocab.json')))\n",
    "dataloader = Dataloader(intent_vocab=intent_vocab, tag_vocab=tag_vocab,\n",
    "                        pretrained_weights=config['model']['pretrained_weights'])\n",
    "print('intent num:', len(intent_vocab))\n",
    "print('tag num:', len(tag_vocab))\n",
    "for data_key in ['train', 'val', 'test']:\n",
    "    print(data_key)\n",
    "    dataloader.load_data(json.load(open(os.path.join(data_dir, '{}_data.json'.format(data_key)))), data_key,\n",
    "                            cut_sen_len=config['cut_sen_len'], use_bert_tokenizer=config['use_bert_tokenizer'])\n",
    "    print('{} set size: {}'.format(data_key, len(dataloader.data[data_key])))\n",
    "\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "if not os.path.exists(log_dir):\n",
    "    os.makedirs(log_dir)\n",
    "\n",
    "writer = SummaryWriter(log_dir)\n",
    "\n",
    "model = JointBERT(config['model'], DEVICE, dataloader.tag_dim, dataloader.intent_dim, dataloader.intent_weight)\n",
    "model.to(DEVICE)\n",
    "\n",
    "if config['model']['finetune']:\n",
    "    no_decay = ['bias', 'LayerNorm.weight']\n",
    "    optimizer_grouped_parameters = [\n",
    "        {'params': [p for n, p in model.named_parameters() if\n",
    "                    not any(nd in n for nd in no_decay) and p.requires_grad],\n",
    "            'weight_decay': config['model']['weight_decay']},\n",
    "        {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay) and p.requires_grad],\n",
    "            'weight_decay': 0.0}\n",
    "    ]\n",
    "    optimizer = AdamW(optimizer_grouped_parameters, lr=config['model']['learning_rate'],\n",
    "                        eps=config['model']['adam_epsilon'])\n",
    "    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=config['model']['warmup_steps'],\n",
    "                                                num_training_steps=config['model']['max_step'])\n",
    "else:\n",
    "    for n, p in model.named_parameters():\n",
    "        if 'bert' in n:\n",
    "            p.requires_grad = False\n",
    "    optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()),\n",
    "                                    lr=config['model']['learning_rate'])\n",
    "\n",
    "for name, param in model.named_parameters():\n",
    "    print(name, param.shape, param.device, param.requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2.3 불러온 BERT 모델을 Fine-tuning 하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "max_step = config['model']['max_step']\n",
    "check_step = config['model']['check_step']\n",
    "batch_size = config['model']['batch_size']\n",
    "model.zero_grad()\n",
    "train_slot_loss, train_intent_loss = 0, 0\n",
    "best_val_f1 = 0.\n",
    "\n",
    "writer.add_text('config', json.dumps(config))\n",
    "\n",
    "for step in tqdm(range(1, max_step + 1)):\n",
    "    model.train()\n",
    "    batched_data = dataloader.get_train_batch(batch_size)\n",
    "    batched_data = tuple(t.to(DEVICE) for t in batched_data)\n",
    "    word_seq_tensor, tag_seq_tensor, intent_tensor, word_mask_tensor, tag_mask_tensor, context_seq_tensor, context_mask_tensor = batched_data\n",
    "    if not config['model']['context']:\n",
    "        context_seq_tensor, context_mask_tensor = None, None\n",
    "    _, _, slot_loss, intent_loss = model.forward(word_seq_tensor, word_mask_tensor, tag_seq_tensor, tag_mask_tensor,\n",
    "                                                    intent_tensor, context_seq_tensor, context_mask_tensor)\n",
    "    train_slot_loss += slot_loss.item()\n",
    "    train_intent_loss += intent_loss.item()\n",
    "    loss = slot_loss + intent_loss\n",
    "    loss.backward()\n",
    "    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "    optimizer.step()\n",
    "    if config['model']['finetune']:\n",
    "        scheduler.step()  # Update learning rate schedule\n",
    "    model.zero_grad()\n",
    "    if step % check_step == 0:\n",
    "        train_slot_loss = train_slot_loss / check_step\n",
    "        train_intent_loss = train_intent_loss / check_step\n",
    "        print('[%d|%d] step' % (step, max_step))\n",
    "        print('\\t slot loss:', train_slot_loss)\n",
    "        print('\\t intent loss:', train_intent_loss)\n",
    "\n",
    "        predict_golden = {'intent': [], 'slot': [], 'overall': []}\n",
    "\n",
    "        val_slot_loss, val_intent_loss = 0, 0\n",
    "        model.eval()\n",
    "        for pad_batch, ori_batch, real_batch_size in dataloader.yield_batches(batch_size, data_key='val'):\n",
    "            pad_batch = tuple(t.to(DEVICE) for t in pad_batch)\n",
    "            word_seq_tensor, tag_seq_tensor, intent_tensor, word_mask_tensor, tag_mask_tensor, context_seq_tensor, context_mask_tensor = pad_batch\n",
    "            if not config['model']['context']:\n",
    "                context_seq_tensor, context_mask_tensor = None, None\n",
    "\n",
    "            with torch.no_grad():\n",
    "                slot_logits, intent_logits, slot_loss, intent_loss = model.forward(word_seq_tensor,\n",
    "                                                                                    word_mask_tensor,\n",
    "                                                                                    tag_seq_tensor,\n",
    "                                                                                    tag_mask_tensor,\n",
    "                                                                                    intent_tensor,\n",
    "                                                                                    context_seq_tensor,\n",
    "                                                                                    context_mask_tensor)\n",
    "            val_slot_loss += slot_loss.item() * real_batch_size\n",
    "            val_intent_loss += intent_loss.item() * real_batch_size\n",
    "            for j in range(real_batch_size):\n",
    "                predicts = recover_intent(dataloader, intent_logits[j], slot_logits[j], tag_mask_tensor[j],\n",
    "                                            ori_batch[j][0], ori_batch[j][-4])\n",
    "                labels = ori_batch[j][3]\n",
    "\n",
    "                predict_golden['overall'].append({\n",
    "                    'predict': predicts,\n",
    "                    'golden': labels\n",
    "                })\n",
    "                predict_golden['slot'].append({\n",
    "                    'predict': [x for x in predicts if is_slot_da(x)],\n",
    "                    'golden': [x for x in labels if is_slot_da(x)]\n",
    "                })\n",
    "                predict_golden['intent'].append({\n",
    "                    'predict': [x for x in predicts if not is_slot_da(x)],\n",
    "                    'golden': [x for x in labels if not is_slot_da(x)]\n",
    "                })\n",
    "\n",
    "        for j in range(10):\n",
    "            writer.add_text('val_sample_{}'.format(j),\n",
    "                            json.dumps(predict_golden['overall'][j], indent=2, ensure_ascii=False),\n",
    "                            global_step=step)\n",
    "\n",
    "        total = len(dataloader.data['val'])\n",
    "        val_slot_loss /= total\n",
    "        val_intent_loss /= total\n",
    "        print('%d samples val' % total)\n",
    "        print('\\t slot loss:', val_slot_loss)\n",
    "        print('\\t intent loss:', val_intent_loss)\n",
    "\n",
    "        writer.add_scalar('intent_loss/train', train_intent_loss, global_step=step)\n",
    "        writer.add_scalar('intent_loss/val', val_intent_loss, global_step=step)\n",
    "\n",
    "        writer.add_scalar('slot_loss/train', train_slot_loss, global_step=step)\n",
    "        writer.add_scalar('slot_loss/val', val_slot_loss, global_step=step)\n",
    "\n",
    "        for x in ['intent', 'slot', 'overall']:\n",
    "            precision, recall, F1 = calculateF1(predict_golden[x])\n",
    "            print('-' * 20 + x + '-' * 20)\n",
    "            print('\\t Precision: %.2f' % (100 * precision))\n",
    "            print('\\t Recall: %.2f' % (100 * recall))\n",
    "            print('\\t F1: %.2f' % (100 * F1))\n",
    "\n",
    "            writer.add_scalar('val_{}/precision'.format(x), precision, global_step=step)\n",
    "            writer.add_scalar('val_{}/recall'.format(x), recall, global_step=step)\n",
    "            writer.add_scalar('val_{}/F1'.format(x), F1, global_step=step)\n",
    "\n",
    "        if F1 > best_val_f1:\n",
    "            best_val_f1 = F1\n",
    "            torch.save(model.state_dict(), os.path.join(output_dir, 'pytorch_model.bin'))\n",
    "            print('best val F1 %.4f' % best_val_f1)\n",
    "            print('save on', output_dir)\n",
    "\n",
    "        train_slot_loss, train_intent_loss = 0, 0\n",
    "\n",
    "writer.add_text('val overall F1', '%.2f' % (100 * best_val_f1))\n",
    "writer.close()\n",
    "\n",
    "model_path = os.path.join(output_dir, 'pytorch_model.bin')\n",
    "zip_path = config['zipped_model_path']\n",
    "print('zip model to', zip_path)\n",
    "\n",
    "with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zf:\n",
    "    zf.write(model_path)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3. 학습한 모델 평가하기\n",
    "학습한 모델을 양적 (quantitatively), 질적(qualitatively)으로 각각 평가해봅시다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3.1 양적 평가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "intent_vocab = json.load(open(os.path.join(data_dir, 'intent_vocab.json')))\n",
    "tag_vocab = json.load(open(os.path.join(data_dir, 'tag_vocab.json')))\n",
    "dataloader = Dataloader(intent_vocab=intent_vocab, tag_vocab=tag_vocab,\n",
    "                        pretrained_weights=config['model']['pretrained_weights'])\n",
    "print('intent num:', len(intent_vocab))\n",
    "print('tag num:', len(tag_vocab))\n",
    "for data_key in ['val', 'test']:\n",
    "    dataloader.load_data(json.load(open(os.path.join(data_dir, '{}_data.json'.format(data_key)))), data_key,\n",
    "                            cut_sen_len=0, use_bert_tokenizer=config['use_bert_tokenizer'])\n",
    "    print('{} set size: {}'.format(data_key, len(dataloader.data[data_key])))\n",
    "\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "if not os.path.exists(log_dir):\n",
    "    os.makedirs(log_dir)\n",
    "\n",
    "model = JointBERT(config['model'], DEVICE, dataloader.tag_dim, dataloader.intent_dim)\n",
    "model.load_state_dict(torch.load(os.path.join(output_dir, 'pytorch_model.bin'), DEVICE))\n",
    "model.to(DEVICE)\n",
    "model.eval()\n",
    "\n",
    "batch_size = config['model']['batch_size']\n",
    "GT_sent = []\n",
    "GT_slot = []\n",
    "data_key = 'test'\n",
    "predict_golden = {'intent': [], 'slot': [], 'overall': []}\n",
    "slot_loss, intent_loss = 0, 0\n",
    "for pad_batch, ori_batch, real_batch_size in dataloader.yield_batches(batch_size, data_key=data_key):\n",
    "    pad_batch = tuple(t.to(DEVICE) for t in pad_batch)\n",
    "    word_seq_tensor, tag_seq_tensor, intent_tensor, word_mask_tensor, tag_mask_tensor, context_seq_tensor, context_mask_tensor = pad_batch\n",
    "    if not config['model']['context']:\n",
    "        context_seq_tensor, context_mask_tensor = None, None\n",
    "\n",
    "    with torch.no_grad():\n",
    "        slot_logits, intent_logits, batch_slot_loss, batch_intent_loss = model.forward(word_seq_tensor,\n",
    "                                                                                        word_mask_tensor,\n",
    "                                                                                        tag_seq_tensor,\n",
    "                                                                                        tag_mask_tensor,\n",
    "                                                                                        intent_tensor,\n",
    "                                                                                        context_seq_tensor,\n",
    "                                                                                        context_mask_tensor)\n",
    "    slot_loss += batch_slot_loss.item() * real_batch_size\n",
    "    intent_loss += batch_intent_loss.item() * real_batch_size\n",
    "    for j in range(real_batch_size):\n",
    "        predicts = recover_intent(dataloader, intent_logits[j], slot_logits[j], tag_mask_tensor[j],\n",
    "                                    ori_batch[j][0], ori_batch[j][-4])\n",
    "        labels = ori_batch[j][3]\n",
    "        GT_sent.append(ori_batch[j][0])\n",
    "        GT_slot.append(ori_batch[j][1])\n",
    "        predict_golden['overall'].append({\n",
    "            'predict': predicts,\n",
    "            'golden': labels\n",
    "        })\n",
    "        predict_golden['slot'].append({\n",
    "            'predict': [x for x in predicts if is_slot_da(x)],\n",
    "            'golden': [x for x in labels if is_slot_da(x)]\n",
    "        })\n",
    "        predict_golden['intent'].append({\n",
    "            'predict': [x for x in predicts if not is_slot_da(x)],\n",
    "            'golden': [x for x in labels if not is_slot_da(x)]\n",
    "        })\n",
    "    print('[%d|%d] samples' % (len(predict_golden['overall']), len(dataloader.data[data_key])))\n",
    "\n",
    "total = len(dataloader.data[data_key])\n",
    "slot_loss /= total\n",
    "intent_loss /= total\n",
    "print('%d samples %s' % (total, data_key))\n",
    "print('\\t slot loss:', slot_loss)\n",
    "print('\\t intent loss:', intent_loss)\n",
    "\n",
    "for x in ['intent', 'slot', 'overall']:\n",
    "    precision, recall, F1 = calculateF1(predict_golden[x])\n",
    "    print('-' * 20 + x + '-' * 20)\n",
    "    print('\\t Precision: %.2f' % (100 * precision))\n",
    "    print('\\t Recall: %.2f' % (100 * recall))\n",
    "    print('\\t F1: %.2f' % (100 * F1))\n",
    "\n",
    "output_file = os.path.join(output_dir, 'output.json')\n",
    "json.dump(predict_golden['overall'], open(output_file, 'w', encoding='utf-8'), indent=2, ensure_ascii=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3.2 질적 평가 (Exercise)\n",
    "\n",
    "위 양적 평가에서 BERT가 slot과 intention 예측에서 뛰어난 정확도를 보이고 있는 것을 확인하셨을 것입니다.\n",
    "\n",
    "이번 파트에서는 문장 별 예측 결과를 실제로 출력해보고, 얼마나 모델이 정확하게 예측하고 있는지 실제로 확인해 봅시다.\n",
    "\n",
    "실습과제 구현을 시작하시기 전에, 다음 List와 Dictionary들을 각각 출력해보시기를 권장합니다.\n",
    "\n",
    "- `GT_slot`, `GT_sent` : 각각 모든 Ground-truth 슬롯 값들과 문장들의 정보가 저장되어 있는 List\n",
    "- `predict_golden` : slot/intent 예측값이 구조화되어 저장되어 있는 Dictionary\n",
    "\n",
    "최종적으로 다음과 같은 형식으로 예측 결과를 출력하세요.\n",
    "\n",
    "### Example #1\n",
    "\n",
    "Query(slot_loc): Yes(O) .(O) what(O) type(O) of(O) food(O) do(O) you(O) want(O) ?(O) \n",
    "\n",
    "SLOT   predict : []    / label: []\n",
    "\n",
    "INTENT predict : [['request', 'food', '?']]    / label: [['request', 'food', '?']]\n",
    "\n",
    "### Example #2\n",
    "\n",
    "Query(slot_loc): How(O) about(O) Italian(B-inform+food) ?(O) \n",
    "\n",
    "SLOT   predict : [['inform', 'food', 'Italian']]    / label: [['inform', 'food', 'italian']]\n",
    "\n",
    "INTENT predict : []    / label: []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print()\n",
    "print('Qualitative results:')\n",
    "for i in range(len(GT_sent)):\n",
    "    # print('Slot location: ', GT_slot[i])\n",
    "    # print('Query:         ', GT_sent[i])\n",
    "    \n",
    "    # Solution\n",
    "    print('Sentence #'+str(i+1))\n",
    "    print('Query(slot_loc):', ''.join([GT_sent[i][j]+'('+GT_slot[i][j]+') ' for j in range(len(GT_slot[i]))] ) )\n",
    "    print('SLOT   predict :', predict_golden['slot'][i]['predict'], '   / label:', predict_golden['slot'][i]['golden'])\n",
    "    print('INTENT predict :', predict_golden['intent'][i]['predict'], '   / label:', predict_golden['intent'][i]['golden'])\n",
    "    print()\n"
   ]
  }
 ]
}