CIFAR10_Classification_VGG16.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/YenaChoi00/ML-Final-Project/blob/main/CIFAR10_Classification_VGG16.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8H2mplXJU1d6"
      },
      "source": [
        "# ***Final Project***\n",
        "---\n",
        "\n",
        "In this project, you will develop a convolutional neural network (CNN) to perform classification, and test it out on the CIFAR-10 dataset.\n",
        "\n",
        "\n",
        "The given code is a simple implement for the CIFAR-10 classification. \n",
        "\n",
        "The goal of the final project is to check whether you understand important concepts for training CNN such as:\n",
        "\n",
        "*   Network architecture\n",
        "*   Optimization\n",
        "*   Loss function\n",
        "*   Data preprocessing\n",
        "*   Regularization\n",
        "*   ...\n",
        "\n",
        "Therefore, you can **modify the given code to improve the performance** of the network **except for the dataset (i.e. trainset and testset)**.\n",
        "\n",
        "You should **upload your code and your report** including an explaination of your code and what you have changed or added."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YmVSPVANZ0uG"
      },
      "source": [
        "class : 10개\n",
        "나오는 output : 4개의 batch size에 대해 4*10\n",
        "output feature가 10이 되는..\n",
        "\n",
        "#수정해야 할 부분\n",
        "data preprocessing\n",
        "network\n",
        "loss function - cross entro..는 그대로 두기! 다른거 추가해서 사용해도 됨\n",
        "adam도 바꿔도 됨,\n",
        "scheduls???? 이런것도 바꿔도 됨\n",
        "regulization 추가 가능\n",
        "epochs 수 조정"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fr9zgCgCmdNq"
      },
      "outputs": [],
      "source": [
        "# AS usual, a bit of setup\n",
        "# If you need other libraries, you should import the libraries.\n",
        "\n",
        "# os와 sys: 파일 및 시스템 관련 기능을 제공하는 모듈입니다.\n",
        "# torch: PyTorch 라이브러리를 import합니다.\n",
        "# nn (torch.nn): 신경망 모델 구축을 위한 클래스들을 제공하는 모듈입니다.\n",
        "# functional (torch.nn.functional): 활성화 함수, 손실 함수 등과 같은 기능을 포함하는 함수들을 제공하는 모듈입니다.\n",
        "# 그리고 CIFAR-10 데이터셋을 사용하기 위해 torchvision 라이브러리에서 transforms와 datasets를 import하고 있습니다. 데이터 시각화를 위한 matplotlib.pyplot과 배열 연산을 위한 numpy도 import하고 있습니다.\n",
        "\n",
        "import os, sys\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.nn import functional as F\n",
        "\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "import torchvision.datasets as datasets\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "from torchsummary import summary as summary_  # 모델의 summary 보기 위해 추가"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V9pg5nB1pXEO",
        "outputId": "a8e576f2-b9d3-46c3-a052-2e8042c7791d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cuda\n"
          ]
        }
      ],
      "source": [
        "# Set the device\n",
        "\n",
        "# 이 코드는 사용 가능한 GPU가 있는 경우 디바이스를 CUDA로 설정하고, 그렇지 않은 경우 CPU로 설정합니다.\n",
        "\n",
        "# torch.cuda.is_available(): CUDA가 사용 가능한지 확인하는 함수입니다. GPU가 사용 가능한 경우 True를 반환하고, 그렇지 않은 경우 False를 반환합니다.\n",
        "# device 변수에는 'cuda' 또는 'cpu'가 할당됩니다. 그리고 print(device)를 통해 현재 설정된 디바이스를 출력합니다.\n",
        "\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "print(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q-G1AJVsbqwc"
      },
      "source": [
        "# **Data Loader**\n",
        "\n",
        "You can load the CIFAR-10 dataset using the library `torchvision.datasets`\n",
        "\n",
        "The details of CIFAR-10 dataset can be found in https://www.cs.toronto.edu/~kriz/cifar.html\n",
        "\n",
        "`transforms_cifar10` is used to assemble several transforms for data preprossing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HOEFPBpcmmHU",
        "outputId": "e58e049c-fff5-4fe1-9170-27b61dbd4788"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n"
          ]
        }
      ],
      "source": [
        "# 이 코드는 CIFAR-10 데이터셋을 불러와서 전처리를 수행하고, 훈련 및 테스트 데이터로 나누는 부분입니다.\n",
        "\n",
        "# transforms.Compose: 여러 개의 데이터 전처리 작업을 한 번에 적용하기 위한 함수입니다. 인자로 전달된 전처리 함수들을 순차적으로 적용합니다.\n",
        "# transforms.Resize: 이미지 크기를 조정하는 전처리 함수입니다. CIFAR-10의 이미지는 32x32 크기로 조정됩니다.\n",
        "# transforms.ToTensor: 이미지를 PyTorch Tensor로 변환하는 전처리 함수입니다.\n",
        "# transforms.Normalize: 이미지의 픽셀 값을 정규화하는 전처리 함수입니다. 평균과 표준편차를 사용하여 정규화합니다. 여기서는 각 채널의 평균과 표준편차를 (0.5, 0.5, 0.5)로 설정하여 정규화합니다.\n",
        "# transforms_cifar10은 위의 전처리 함수들을 순차적으로 적용하는 변환 파이프라인입니다.\n",
        "\n",
        "# trainset: CIFAR-10 훈련 데이터셋을 생성합니다. root는 데이터셋의 다운로드 위치를 지정합니다. train=True는 훈련 데이터셋을 의미하며, download=True는 데이터셋이 없는 경우 다운로드합니다. transform에는 위에서 정의한 transforms_cifar10를 전달하여 데이터셋의 전처리를 수행합니다.\n",
        "# trainloader: 훈련 데이터셋을 미니배치로 나누고 데이터를 섞어주는 데이터 로더입니다. batch_size=4는 미니배치의 크기를 4로 설정합니다. shuffle=True는 데이터를 섞어서 무작위로 골고루 가져올 수 있도록 합니다. num_workers=2는 데이터 로딩을 위한 병렬 작업자의 수를 지정합니다.\n",
        "# 위와 마찬가지로, testset과 testloader는 테스트 데이터셋에 대한 데이터셋과 데이터 로더를 생성하는 역할을 합니다.\n",
        "\n",
        "# 마지막으로, classes는 CIFAR-10 데이터셋의 클래스들을 나타내는 튜플입니다. 각 클래스는 \"plane\", \"car\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"으로 표현됩니다.\n",
        "\n",
        "train_cifar10 = transforms.Compose([transforms.Resize((32, 32)),     ## 32.32는 수정불가\n",
        "                                  transforms.RandomHorizontalFlip(),\n",
        "                                  transforms.RandomRotation(10),\n",
        "                                  transforms.RandomAffine(0, shear=10, scale=(0.8, 1.2)),\n",
        "                                  transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),\n",
        "                                  transforms.ToTensor(),\n",
        "                                  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))    # 여기에 preprocessing할 때 추가적으로 넣어도 됨. preprocessing은 tensor로 바뀌기 전에 \n",
        "                                 ])\n",
        "\n",
        "# # color Jitter 조정, randomvertival flip\n",
        "# train_cifar10 = transforms.Compose([transforms.Resize((32, 32)),     ## 32.32는 수정불가\n",
        "#                                   transforms.RandomHorizontalFlip(),\n",
        "#                                   transforms.RandomVerticalFlip(),\n",
        "#                                   transforms.RandomRotation(10),\n",
        "#                                   transforms.RandomAffine(0, shear=10, scale=(0.8, 1.2)),\n",
        "#                                   transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),\n",
        "#                                   transforms.ToTensor(),\n",
        "#                                   transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))    # 여기에 preprocessing할 때 추가적으로 넣어도 됨. preprocessing은 tensor로 바뀌기 전에 \n",
        "#                                  ])\n",
        "\n",
        "\n",
        "\n",
        "test_cifar10 = transforms.Compose([transforms.Resize((32, 32)),     ## 32.32는 수정불가\n",
        "                                  transforms.ToTensor(),\n",
        "                                  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))    # 여기에 preprocessing할 때 추가적으로 넣어도 됨. preprocessing은 tensor로 바뀌기 전에 \n",
        "                                 ])\n",
        "\n",
        "# Train dataset\n",
        "trainset = datasets.CIFAR10(root='./data', train=True, download=True, transform=train_cifar10)\n",
        "trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)    # batch size 수정가능. shuffle : True/False 선택가능, num_workers는 CPU 수 그래서  수정 불필요\n",
        "\n",
        "# Test dataset\n",
        "testset = datasets.CIFAR10(root='./data', train=False, download=True, transform=test_cifar10)\n",
        "testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)\n",
        "\n",
        "# Classes of CIFAR-10 dataset 10개의 dataset. 수정 불필요\n",
        "classes = (\"plane\", \"car\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "id": "d7JdtMAzoCac",
        "outputId": "2644113c-9dae-4370-bfc0-405c1b0a25fc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  dog plane   cat truck\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAACwCAYAAACviAzDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFz0lEQVR4nO29eXBc1Zn3/9zeu9Wb9sWSbBkbbLANjo2NTBYSnBCGH4FAzSS8ZHASMilm7AzgqgFMBlIJIXZlfjUQphynkmEgkwlxhlRwZsiEZQyBGLxhbMAYL4AX2dYu9aLeu+95/8jLPc/zyN22ZKllWc+nSlX39Ll977nnnnv76HyfxVBKKRAEQRAEQSgTtolugCAIgiAIUwuZfAiCIAiCUFZk8iEIgiAIQlmRyYcgCIIgCGVFJh+CIAiCIJQVmXwIgiAIglBWZPIhCIIgCEJZkcmHIAiCIAhlRSYfgiAIgiCUFZl8CIIgCIJQVsZt8rF+/XqYMWMGeDweWLp0KezYsWO8TiUIgiAIwiTCGI/cLr/+9a/htttug5/85CewdOlSePTRR+Hpp5+GAwcOQF1dXcnvmqYJJ0+ehEAgAIZhjHXTBEEQBEEYB5RSEI/HoampCWy206xtqHFgyZIlauXKlVa5UCiopqYmtXbt2tN+t6OjQwGA/Mmf/Mmf/Mmf/E3Cv46OjtP+1jtgjMlms7Br1y5Ys2aN9ZnNZoPly5fD1q1bh+2fyWQgk8lYZfX/FmLuvvtucLvdY908QRAEQRDGgUwmA4888ggEAoHT7jvmk4++vj4oFApQX19PPq+vr4f9+/cP23/t2rXw3e9+d9jnbrdbJh+CIAiCMMk4E5OJCfd2WbNmDUSjUeuvo6NjopskCIIgCMI4MuYrHzU1NWC326G7u5t83t3dDQ0NDcP2lxUOQRAEQZhajPnKh8vlgkWLFsHmzZutz0zThM2bN0N7e/tYn04QBEEQhEnGmK98AACsXr0aVqxYAYsXL4YlS5bAo48+ColEAr72ta+d9bE/+7m/IGXTLBTd1wA7Kdvtxedadrve1zD4fmPv8ns6B+dCwbS28/k8qTNBfzmXM4FW0rbn8umi50hlYtZ2X98xUvfee6/T9hZ0G1xO2q+2Ev1TyNK259F8NxSeXfR7jzzySNE6YXJx9913l6xft25dmVoijDf33Xdf0bqHHnqojC05O+742jdIOegLW9uhcJDUHfjgkLXtdDlJnc/ltbYH4wOkrsJPlYCKqgprO5OhakBL/QX6HO4hUjetoZaU40l9zo4eui84s9ZmY5hWJVJUrYhHD+qv0V0hk6+Bs2VcJh9f+tKXoLe3Fx588EHo6uqCyy67DJ577rlhRqiCIAiCIEw9xmXyAQCwatUqWLVq1XgdXhAEQRCEScqEe7sIgiAIgjC1GLeVj/HCYPMlWwl/Ymw3AQBgGNpWAtt4APzZKFbX8TkZNdCw23W3lbI5yeeZPQZquxpBVHve1kIuX2RPAIfLYGWt//X1nyR1R4/utba7uw8Xa+qf22DzWdupLLcj0dfigBypySdSpByuovqkIGDC4TAp42BFg4ODpA4HJ+TwIEeplB6H+FkHAPB4PEXL3BPP6dTqd0VFBanz+/3WNn++eajp1157rWjbpyLfWbOSlLMZ/R5xu1ykboi+UgjxZJKUvR59v/KpLKmzofeqzUV/CtNx+h7z+vQ4MNjvQTiobUDcbi8U47K2y0n5SMdxUo706eNWBKiVRSar2374g05SN9jfR88z/2JdMH2k7oOT+t19Ik/PUVfdRsr2nH4Osqn3YayRlQ9BEARBEMqKTD4EQRAEQSgrk052KQxTObDMQJdT8RIpAABWaPgyqM2mK7NZuuTGyWT0EpjDQc9RQC6pXFkp5eqrFJVLbDbs+ksPhMs5JoEMDkaKnuPwkfdIubP7A2s7neondd2d1EXL5aBLn5hgpV5uNg0qCTkdVDJyOXQfFBeshnPp4k+Q8vJll1nbqWyc1HWeiFjbuSw9vz9Ar6O2Ri+d+3z0cbApdL+Y+3WejTWzhIxmQ99VXMJD83+Hg55fmXRM/OD/P//djxsbG0kZP6dVVVWk7tgx6h4eiUSs7VgsRupwuGcuY3JX9kQiccptACAee+k0ffbwcfg5eLjpT3ziE0Xr8DXz91SpfV988UWYLHzlL79Mynb2fAVC+r3qsNNnNpPR76aeAfquZooa2PJamjNzVHZx+7REwuVhw6T3r6GhCYrR1KQztafSdCzF4vq4CaYXzWybRcq9g/rdEAjTse606bdlMEClFJeTXnQuX0DbTJo0dBvsTCIfGmIhHdC7M1+g7ybaO6NDVj4EQRAEQSgrMvkQBEEQBKGsyORDEARBEISyMulsPrBtBgCAaWINneujpeZW9DjY5sLpdLE6qqklElpzTKWoa5cLhdd1MRexkmmGFW1rDoUlV8w6IpnQ9hnvHTxQ/JgA8MHhD63twX7qoqVA63+pLNUjd+44RMpeT3ELjcsWzrW2/T4aetih6PeMiNbiQyPwunV7qM755jvaNXhma5jUmXk9Jgo5qmsWsnTIx2P6/tmBnsPt1veyYFK9WLFx6EA2OlynJ/o/G5MKi9TMbsTJ7Im+c/+9+nxuNraQLQ0oqsi6bXRfG2gdWPmou2g2r68rkqT37ocP3gPjDXeDxc8Qt4m58MILSTmJ3Cz37t0LxSgww7FSbu/8me3t7bW2+X3GDLcpo+WTJ7Xbe3NzM6nD18n7o5TNx3XXXVd0X/49bpNSik2bNp3xvmeK28Wui9Xje+Rmsb3jUf3+TSfo8+1g4c2TOX1kr5uOH/zMZoHaOzTU15HyUCxibbNTQGJIn2MgQm3lchl9HQ3TqD1TRYDakSTSemzZs9St3BPQ9imtM2ho85pq+gwHQtoGbzBO7eECyKXY46Mu5rEY3TePwkiYzA7SXsIG8EyRlQ9BEARBEMqKTD4EQRAEQSgrMvkQBEEQBKGsTDqbj+GhzrXeNlyDLRXCnNt86H2pHclwcPhlrpP19upQtzxUNMbvqyRlHoo9X9AaW//ACVK3//3d1vbO3btKtvV4h9YRvcxf3mYU130dBtURc8hdPM9sUHa/2WFt19VQm48A80l32CLW9pLai+FMcbmoPulA8UOiUerL3lCv+7bjGI1fonggAFMLuJk01zX1OVwu/qjQMYI1aoPbfCC9nZsXGCSei8Hq2BkNHIuG2qB4nLifS9gWAY2FYLBQ/T1DunJoqHj48vEiGo2SMg6Lzm2oeOhzXN/QQNOVd3ZqeyceL4TH+cD2EPydMjBA06KPFvy+KWXHUdJOjNXzuEb4Ori9DC+XuubxoLGxmn7AYtokkb2RKtB4Kv5AyNq226itGn++IkPomWbX7EJ2Ux5WNzhAbTc6Tuh08/kcbU8B2evlc8yeCD2LBZaKY7aH/ga4DBT6/NgHpO6CuRdY2/MvmUPqqmuo8dzb7+jv1vhpv6aRDcogiosDAODx0j5w5o/otjlovJt4lv4+jAZZ+RAEQRAEoazI5EMQBEEQhLIy6WQXviSIly+5y1xpV9tS5+DuvNzdTS9RhkJUZsCZLTmxmF5SPnyUusi62BJyPq+X1QsmXVr0uvV1ffIKminxwOH9pKxsuk/27f2Q1iEv4cpAmNTNamHLoohjPdQNLBnXy/M9GbpsHppF+yOlSoeuLwrLYOqy6/7KshDYgwXdX/V1dHkwmaBL7PmsvreJAh0/ptLHCYao7FPho/cLu0rnsrSt+axe6rQZLIQ6GqPDQnJD8bDJdi+VINJp3Va3i7Ytm6XX7EKu5OkCuw50zrxrLIIonx055CrNn2+e1RY/p1y+wfJEnLkf8mcWH4dLGcEgfd6Lwd8ZQ0N0GR9fS18fzUo6WkpJK1yy4vvi9+pI3HBHi5udv6eXvlOc6H1oMpnDUPq7VWE6ftNsrCOve3Awn10sj+ZS9HvHOmhYArdPy5p2Jz1nAYVisDE/3Lypj+t203EWGegl5Wi8D23T/ohFdTkWp1JyIEil7YYafV0zW2l2Z/s+/Z7o7qbn9/qoHOlHknkyGoGxRlY+BEEQBEEoKzL5EARBEAShrMjkQxAEQRCEsjLpbD7GCu5Oy+08zmUaG2Zb251dh0rsOXqiLGxxqMJZZM/y4Hbw8NCokKcubAUk3xYK9Dp8Pjrfxl6WdjsLcY/sDfJ5ev3c3sCFQqE7nfSxSqW0Zq14uHk0/+e6PA9ob6J9VYZelw27AjN3Q7uDXpeBQiOb7BUQH9Lh710+2p6/v/97pPzYDx6EsWbhwoWkXFGhbXYOHaJjvb+fat/4fuVYWH1s88Fda7l9CLZ/4LYa+DiBANXTsR0HD+HOweco5drKj8PtMXC5q6ur6HeH2ROVcOEth81HkrnP2jy0PU6PHpf5FLV5sxv6/nnd1E4hGqO2EoaJ3JZN/p7Qx+keoHZAzDQLXNjGy0ZtNwrY7Z3ZGdpA11X46PfiQ/ScJ7t0yH0nc+03c/o4OL0HAEB/H3X/LsT1MzzYE6HtQekTTGab1tdN7VxyQf1c2HiIgjFAVj4EQRAEQSgrMvkQBEEQBKGsTDrZhS8J4qXF4a62Z35c7hpH4QcqHgGVL91jsEufyZYAP/jwXVLuHdAR9fx+6uaJ6e8/Tso4SysAQFenXpoe7KXLyxU2nSmxJ9VD6hbNo9FHmxt0lsdjXX8idRfPbbG260N0abG6ikbxGy0ugy4RGigLMdiZu6iBsjEy2YVGAgVQyI0vw6Qmt18fN8+z49rpmFAGclFlWZBtZCByuU8/gnbuGs4GcKFE5F0sJeRNOgaCLnrNBeR+3dNPl2wNQ481I0/7PFxx9pksT8djjz1GylhK+M1vfkPqcGZYAICeHj2G+btg1y4dCfh00T6xey1/32C5LRaLkTqv13vK/QBKv19KyS7Dot6yMm77SLL14rbyfXF24PEiEKaSld9PyybSTqNp2p5CQfdBVy+ti9BXHGTQd70BOn599VrS8wfpeyuZprIQdv1NcZkXyUDKziLHGnoseTzU7X+gn7q65vL6HRNm782AT0d1zWfo+dNJOtaiyG35ZAeV4gZj+r67WMTrKJMYkw7dB14+UxgDZU5WPgRBEARBKCsy+RAEQRAEoayMePLx6quvwvXXXw9NTU1gGAZs2rSJ1Cul4MEHH4TGxkbwer2wfPnyYVbqgiAIgiBMXUZs85FIJODSSy+Fr3/963DTTTcNq//hD38Ijz32GPz85z+HtrY2eOCBB+Caa66Bffv2kWywYwV2a+QSZ2l3N27HgTOP8nDqo2wccL1WC2UmO0d8KELKhw6+X/SoLdNmWtuDLMnmvj3UBuToca35BZmr142fu7boOebMmEbKO3e/ZW1nUlQb/PiVH7O2p9UVD8sOAPC75160the1zCyxJ+UPf3yJlK/9zNXWtttF9Wsg7qylQ5Z73Xr85HIsQ2ZBz825hu9gtj02nKWUtd2G7AYUt/kArRfnCvSbpknPgTP5FthgN3MozYCdXmOa2cuAQuHVWVZblx89Tya9z84SWZDHCv7MNjU1Wdt33nknqeP3pKNDZ1fu7aV6+oYNG6xtHk6duzinmd5frG7/fprKoFQWWW47gu08uru7oRjcboPbjnC3YQxuA7cV4d/D+/Jz4rpS5xsJbha2vpelbMiiDNOxOD1nLouyO+eZy66N9ruB3My5m7vLqd8bNgf7reDHwf+n83Qb+Gvs6fd59VhzsbDsoVCYlAcTesx6WCh2nL7B7WEpK7L0Oc0he5nBfto/OTuy6eJuwaxcyOhrSaaYPRxt+qgY8eTj2muvhWuvPfWPllIKHn30UfjHf/xHuOGGGwAA4N///d+hvr4eNm3aBF/+8pfPrrWCIAiCIEx6xtTm4/Dhw9DV1QXLly+3PguFQrB06VLYunXrKb+TyWQgFouRP0EQBEEQzl/GdPLxkVtcfX09+by+vn5Y9L2PWLt2LYRCIeuvpaXllPsJgiAIgnB+MOFxPtasWQOrV6+2yrFYrOQEhIdFT6HQu1kc5vYUhELIV5qlT7ehuA12pvdlMvS4uRLnwdogD/GMwzi/9dYeUrfzjTdIef/+fUXPMXCRTiFvM6gdTTpL55MepGt6nHTf515+2dq+YtGlpG4wFiLlzVu1zUWgkmqOeVNf57bdbxZtNwDAiaiOO7Ko5J6lcfq0H3yOxQFwe1Doam6/w+Kr4PTzwQD1wx9CNgX5LL2XGRabwXDqspPZEOA46Vx7Lyh9XBYiAEwbvV+mgdLL59j5kV7LZXmT25k49HEcHmqLoLI65LPPyWxHmI3FeMBjXpQKEc5jVcyerdMOXHTRRaRuYEAbR/HvOZn9weuvv25tRyKRou3hbcX2GPzZx+8eAGpHwcO7v/3226c8H0BpG5AUC0OOCYfDpByPx4uW+RhdtEg/qdu3by96jpHQ291Hy73UbsGFbBMyaR5TR48Dl4c+aw4nLWdS+l1pMHsM/EjnsrRf3R56v5xOHYckwGKvp23oebLRMVpbW2ttVwRpLBOXiz5frgr93cFBOibiKd32SJTeO6+PXlcOHTZrMgs01PQUC68OLJ2Cg9icjb2915iufDQ0NADAcAOq7u5uq47jdrshGAySP0EQBEEQzl/GdPLR1tYGDQ0NsHnzZuuzWCwG27dvh/b29rE8lSAIgiAIk5QRyy5DQ0Pw/vvaDfTw4cOwZ88eqKqqgtbWVrjrrrvg+9//PsyePdtytW1qaoIbb7xxTBrsYhlDDx08Ym0fOXoESnHVVdoQFpjbYBQZukaj1Oj15Anqvorp7qEufYmYXhIbGIiQulgsgrYTpO7osaPsyFoC8DBP0k998kpr+8mf/4rU8SylARQ2WBWKu8m9vIWGTP/jn14m5ZpWnT3yorkXkrp0QV9LZR1druzqopkSK+tpqO/R0tOvz1kXZjJQXi8/s5VEUCbtA0Pp/nK76PJ7HskVXGpTysXKxduayWh3NxuTDhwoW6/BlzYNunRvKuSqDbQ9bje+z3SpPse8CAt5/T9HT4S6OFaHtat0wMVkhRJpBcYKLoHgUN9cKijlzsrlimXLllnbVVU0E2qpfbmUcfjwYWv7oYceInWJhB6TXNo5epQ+39hllbdn3rx51jbvD+4WjMuDg/ReYrdgfo2877DbMj8Hl5BGy7e+scLaduboCvfctiZSjqD7fvF86vZfP22Wtf3uOwdLnnMAufA6nXT85nO4b2n/zJpJ33HU1ZaeI6NQeHUmZeB3bs6kddNaakm5962ItZ1K03tgIPf53l56n2vrw6SMZd+mlhpSNzik73tPkrrh2p0slQA0WttBg8Z0GIsctyOefLzxxhvw6U9/2ip/ZK+xYsUKePLJJ+Gee+6BRCIB3/zmNyESicDHP/5xeO6558YlxocgCIIgCJOPEU8+rrrqqpIJiwzDgO9973vwve9976waJgiCIAjC+YnkdhEEQRAEoaxMuKvtSFEstG0EadZh5s62dSd1C/vEVTok98ZfUVuJKHJtGmKphQcHI0Xbw9NP57JaR8uWcE10MC2Xh3yOotzQBtB9D73/od6PuemZbN8AcR9l6bjtegXL5uR1dGgsXrJAn8Og96A/2gPFyOaobUs44C2yZ2lqm6grMCCXazNP74HLo+scDir3MU84SOe17QQPZ2636T5x+GhoZKeDhW1H3Vco0P6x23F4aqqf4zVEngyARzM3kDt4zk7vlx1pwoq5AiaZ+/XggB7fDgfTlpHbdJwN36rg6O7dSOCrqtid9XShxfF3h7k0I5sHbjvCbRywvUapVd7qappKAAdI5G3ldhMVFfq55C672M6j1HXw83B3Xlzm5+Bl3D5+zrGy+fC7tKupl4XyNrPU/iCE7A8aqmg/z56tbT4GBun7JdrNXOkbqS0JBqc6cBosRHmG9oFhohQJNmpvhV14DTd9RlRa2wztf/cAa8EsUrrq6qXW9v++uIPU9fQgN/sCfRflcxFSbmwMW9uFNH0XVFehNAyKtvVIN32POitQeH7g4SXO3nZPVj4EQRAEQSgrMvkQBEEQBKGsyORDEARBEISyMulsPrjmibVTzv5975HyL37xC2v7T1u2kToHEti55smxIx/5bJr5dSPNnOvFuMzTgQ87B0rZnmT+2Dt36hDmdm7EwCTqxJDWRF1ueruvWKYDv/kDVEe026jO63FpzS+TYRqwA2nULPy9xx8m5ZTJLRvOjIJJ23f4hNZoI0zXnF6DxgSLq+HzcrsOFL4bWAwQNDd3cI06T+9fMq+1dzdzK8cavtPLQqajc5gGtdfJZGhfftirx4HhpKGafR7d9nSBjt9AmGq7Pq/et5JpuQ1+HEuEhfbO0X0f/q6Oc/Ht7zwAYwF/ZrBNFbfx4HYLOHYFr8MxQPjzzWNyYHiIgIULF1rb9913H6n7/ve/X7St/L3l82nN/HTh3jGlbDd4HU43wes4+Jy87Tys/Wip8OtnOBGj8VPycXpPTBQfo7f3NVKXyuq62RdeQureSuwh5WiXfi7NPO1XB+qSQqaf1Clm02DY9D0qKGZrZNPnsGeo3YTd0NfZ0kJzngUq6X3/nxd2WtuRHvqO9fu1XZLdTu1TeLaPKhT3aMig4+7IkSO6YGPRxNl9z6f1sxiuov3RXzxk1BkjKx+CIAiCIJQVmXwIgiAIglBWJp3swpl3yXxre++770xgS0YGF3bGP3D12OF2UwnkdBLSWDDQRV3PmmZqt+nBBHW3a6nTGW/zLKtjnsUFzqKlRg97GiqDeqnRzsJRc3daAy1N82VzvK/iWXYdqC9ttF93vbmblO1V2jXPV0GXQbPouC43lQryefo/RjqtUwLUV9OlV5XT0o5jWFRiJj2VCNc/WnhmauwuijPTAgwPfY6lDB4+HN8TXjcSsCTxwgsvkDrsesvHAHeJx/Vc2iklc/A6LCHxOuyGW8plmNdz1+MEe75GiwItJXTH6TEdBh3PisizVDoYGNLSRjjD5BtFJWoT5RZQTBJ2uXRdlmXG5j75iqQ24OmndT8bPPB4QY/nri6acLXgYeHes/pabOz8Jsp+nWVuyVd9Yi4pV/rqrO3fvkazo9tQ6IEsCwVvsCzWs+v1OSur6JjoPwFnjax8CIIgCIJQVmTyIQiCIAhCWZHJhyAIgiAIZWXS2XyY3FoCaZWNjY2kysPcGjdt2oT2bSZ1mRTV0cg5mXuoB2nqXJ4tIBmPh1BXheKJiLMs7DbW/Lha63JqV9JgsLSW29evQ5+7AlTfr23QthHJFNXTwcbcTrPF2+5AVX43DRNfYO6rDnMskjEDQAHpo2xIpNC9DDFddSgaI2UcXdwOVNeMD+nvegPUtdXtYG57xKaApz3XDTQLtD0ZpesGBuk96O6l5Qum67413NwuQD/KOYP2ucdJ2+Ow62upcNP/P1ROt6+Qo5qw00VtUsxxsFTiobyxGyq31QgG6XjGbrk8vDm2HeHn4DYN2P6hlGvrsmXLSF0pmw8ceh2AujzyffF1cjsObruBbT64Oy/+Lu8PDrbj4teM7Vw++clPkrpXX3215HFJW/F1svQNymS2LOg9rwxqB5RAz3cPe2aG4tQNtVBAz6XBngOnvu8m0PeWwUKPm6Dbl2P3AGc6MGysn9F73OGmdi1+Hy1HHXqMZF30pWZDIQH4+D1+opO2pzVsbXsCNaQug0JD8CgVpo0+Bynksu/I8PFz9usWsvIhCIIgCEJZkcmHIAiCIAhlZdLJLjz6HpZd+BLlggULSDkaLb5U7/V50H6DpM7Jlq1dHr2Ux923DLQU7XCy7kWrjsMyRebZUh4Kv2djWUrxqp/XWzq7YMHUS5YVfrq829d/0toeytJrrg/TDJl4MbP3ZC+pCyBJIpunS782HtCU9eVocTnRvJmlf+3q0u0LOCtJXdhP+xJn6OULiyRCJJPlcna6FOz342VaKlfYHH5UQ5d3+5J6PEeyVNaoqJ9Oyg50r4ctojv0cRSTulxMolEouqTDXUvqPG6UOZdFAk2xDM7DxvcYwJf8sVzAn30uJeAyd8PFsgd3beVSBj4Pjz6K3VBnz55N6rAMxNtaU0OXvx999FFrm7sXl8oie7rstBicKbtUtOVTlYud43TyTSnwq9Ku2AvYZNmeUYZpHsTZYegD5VnfFZgsb8fu6yZ79gs4szCTyHn/oG3+2+FA57QZ9Hs42rHNQ999PpZJ3ETvTgPoRdtQBnLFIhscP0nf3Zl8h7Vd20Sf7xOHdZ2NS3o8jTZ6jwwOuFnd2Wc6lpUPQRAEQRDKikw+BEEQBEEoKzL5EARBEAShrEw6mw8b06Gxex3XTjMZWna6tK6XyxV3reWaud1BNWK3C5WZzQc4uDaG0ZqeyVxrDea26EKubypL2xONU40P42PuxTWV2h5jzoVNpC6d1pkc8wXaHye6i/dPdaCKfqD0daWTVGuvq6aaY99QvOhxR4JCmSVtQPXaeEprl4c6qOvd/FnUlsXp1jqswTLXGmgc2A2qdXtZCOoCGgc2B90Xh3gfSNC6gSGUBdlB7UGaZtKwyThDscNDx5kb/R/hZ9qyyewfvC7dP9lhmWL1tsFsjezMPbKE5/io4XZb+Jnmddy1FO/L7Saw3UI6zex32L74OIOD9FnDthvc3iIcDlvb3LX2wgsvJOVFixZZ29g2AwDg0KFD1nZXVxeUArvl8my92IX4dFltsY0Md2nG/XO6jN8lQXYe/H1XYCHUibGYonUGevbDYeov2slsUgooq6uyM5shNL4L3IaBuasXyHFZX6LUBqaNXlcO5XMIuZgdB7MPyWdRWxWzgcFF9pPDPOJBIXuRmjAdW71OfZ9dNvqeyNnZO8Wns/DmS/zmjBZZ+RAEQRAEoazI5EMQBEEQhLIikw9BEARBEMrKpLP5MBTVzZJRbUPQ1Uf10e4eGo8ih+0smE9zKqFtRwwb7ZYMS2Hc2X3M2uapj/1eGvIZ40CaeTJNhbpcjtmZuHRbE7HieluKaYNckp3eqG0uptdT+4vjJ4/qc6SpXcAwnNqWpJrFAHEgewdun1Ltp/tmT5Pa+0wxkF7qYuGpc2mtnbpc1Aamb5Dqx7W1ut7hpJqnE/R9t59G6nbijldUSx1CfdI5GCF1eRTSPZOkYyKbp/p1Kqvvtb+ChbFHNig1fqqDJ6P0nMFKfc05ZvPhQ/FuHCyaCI83Mzh0mjEzCrjdFo6dwe0W+L7YloPHacD2Irwuk6H9ju08uF0JtuXgdhw4fkgp2xUAgC9+8YvWNrejwPYgP/vZz0hdJBKBYvC24jIOnw4wvC9xn/D+wMc5mzgfhRJGQjxUv0Jl/h8ythfxsDDkKs+eoRwKz09fEyT1Q47FHeG/MzgEvs1O+9lEz2k8QX8r8ln9+1RRRd+Fys5ss5Ddi2HQ93oyodvqZqGS4rEoKVej+5dKR0hdY71+nkIh+g7pH6Jt9/qxbSO9C2lq0jQqZOVDEARBEISyMqLJx9q1a+Hyyy+HQCAAdXV1cOONN8KBAwfIPul0GlauXAnV1dXg9/vh5ptvhu7u7jFttCAIgiAIk5cRyS6vvPIKrFy5Ei6//HLI5/Nw//33w+c+9znYt28fVPy/FHl33303/P73v4enn34aQqEQrFq1Cm666SZ47bXXxqbBDjpfigxqd9FXXn6Z1A0M9JOyvYSbmMLLbmyJyVVBy0NJveYUZ1lS+9AKmM9Ll9mCAZRZ02SurHZ6jgIKteuvKB6SnHnIgirQZXQcHnpamMouFfniy6CHo1SyMlEm39gQvWYvyuIaYG7JXSfoxJPcARr5fETU1uoMotF+6r5rU9hljS6RJhJ0WdaLlmJDfu5qht1wWdh4tkzsROGZnX4qvbnREu7rb2wjdVmvdn/2uGkob75Ujtm9cycpt0xvtbZrwyx7ZgU9rhe5nCcTNGS6B90gh4/eyzwbW24nD8d89lx66aWkvBNdJ3cBPX78OCljqYPvy11fMf399D2B3VsreOpPBP/H64ILLrC2eVh2/g8Yli9aW1tJHXa95XW8raXkJOwie7qw7Fj64cfBbS0Vhv105JB8o9jzYzCZykDn5OEVbEi+drC6ULCa7gs61EA+Rd93CoV0Zz8r4A/Q+4dDMySTEVLnQmMNh0EHAMjktDSZTtGssZk0fU5x5lrDoJJmuqBlu2k1NGRCNsf7TocXOPBeB6lrrNMv3dZZbaTO2UfHFpbJmmtpKoH+3W/D2TKiycdzzz1Hyk8++STU1dXBrl274JOf/CREo1F4/PHH4amnnoLPfOYzAADwxBNPwNy5c2Hbtm1wxRVXnHWDBUEQBEGY3JyVzUc0+ud/86uq/hx0ateuXZDL5WD58uXWPnPmzIHW1lbYunXrKY+RyWQgFouRP0EQBEEQzl9GPfkwTRPuuusuuPLKK2HevHkA8OflSpfLRSL9AQDU19cXjdS3du1aCIVC1l9LS8tomyQIgiAIwiRg1K62K1euhL1798KWLVvOqgFr1qyB1atXW+VYLDZuExBfhdawk4ni4cPPNfIOqms68mPjrloOPMzeIJ0YG/dM7LZcV0ONR6KgXSW9XjrEfdxOAWmpJ473kapMldZkpzVRt7Q8s5exId23r5uu3r21T9sGDMaoy2Vzg7YPSaaoPYrPx3VnXR8OB0hdIjKgv+ei19zf00PKYa/ur0KWuvQN4RDuPJf5sBDdelw+9J3vkJoHvvtdGAs6OzutbW7/gF1bAWiqBe4SisOHn85uIY/cj/H5AQAaGxutbR4WHdtfcLfXkydPkvIHH3xwynYDAJw4ccLafvPNN0kdt9UoFfoctyc/LIx+cZsPDu6vswmv3tuLUh2wFPYG6y/8ypt32XxS13qBdnHuPzlA6qa1zix6fjNCn4MjJ/WzGGpoIHV9kSOkfPzI+9Z2iqUrwLZZ1VU1pG56c521nU3S98IH+6kdkB2FRU/mqB0bToPQ1ETtWkJ+2vbtu96xtlWB3q+ZM3Xfud3UNi0Rp2M9iWz7WuZeAGPNqCYfq1atgmeffRZeffVVaG5utj5vaGiAbDYLkUiErH50d3dDA7u5H+F2u0sa1gmCIAiCcH4xItlFKQWrVq2CZ555Bl566SVoa6PWsosWLQKn0wmbN2+2Pjtw4AAcO3YM2tvbx6bFgiAIgiBMaka08rFy5Up46qmn4He/+x0EAgHLjiMUCoHX64VQKAS33347rF69GqqqqiAYDMK3vvUtaG9vHzNPl9gQXWaLDukl9nf3vcN3JwyiLI+VVXWkDrtz+fzUtfWieVQGSmaL+4gOxCPWdqCCLgtDTp8jEWMZMYNhUi7lFlxdobMN7nqHuvtx1183ytTa3VU83kqogq4+1bHlw2MDWpIoeOm+sUG9RGhWhIueAwAgE9dLzKX3LM3WLf9lbV9+2adJXUuzlit8LMOsh/nUOV3aldLJJBl/UC8NZ9mjcriXLhvHTL0UG2JRXT21Ojttay1tD3Z17Tx+jNTNaKTLq90D+v41N9D70zZjurV97OC7pK6pisoDCsk7Hicd63ipPhqnGYE9PJOuW8tSdFF/7OhBkhG3JePSCnZD5RFGsezAJQge0RNLElwSwZli+fdwhFUua3R0UJfHvj79PO3fv5/UYRdefo38nMXaDUCjw3K4fIK/y8+Jo7NyOWkkkHMw5YtLYYYNt4+7yOpykGW1NU0miUS0hHW8n71zq3R27ncPUPMBD8tO6/OgCLBpKp3a0Ds24GUZrVF07BTLptw4jT7fOdTPdfU0eu67+/QYefW1HaRuRiMNoeBCawodnVRq2r9Py3gO2xxSF3bTtnd+qO00X49wm00WLnYUjGjysWHDBgAAuOqqq8jnTzzxBHz1q18FAIBHHnkEbDYb3HzzzZDJZOCaa66BH//4x2fdUEEQBEEQzg9GNPk4kwAzHo8H1q9fD+vXrx91owRBEARBOH+R3C6CIAiCIJSVSZfVdt/bNKw0oNWY+oZqKMXJvVpnDbBQ0VgirqyhYW/tbqrVBVCo8UiSumdW1ui6hsoqUpca0nptXSV1lYR8cRsPD7OxmFar3f3eevcQqVNMr61BtiR5s7heexzZzgAAZFlYcg9KpdjLXMYAuZ0motSuJOiimqwBxUO6j5YPOmlWx6yp7SFmT6ND3GRZgPMZbdfgdlHbiERK65qxOLUhiOWYtuvQY2YoScdWX78OVV9TT8cEoBDPeZZZePcOGordRG6x1VVhUmcMaXuH6VV0/Hq8dGy5nMXHmh25MLtdVNe1s7SgGWQ78Y9j5FrLwfYH8ThzP2SZYvHK7EcBED+iVJj0UnYM3BMPtwe7uQJQF0xuU8H3ra7W76pSthm83UND1A4Hn4fbfJQK2MhtYvh3i9Vxe5mRYEPZwgv8OCy9RLag7+XePftInYFcUhdfsZjU7Xl7Dykffl+7ONcU6PMVCFxmbTfUcbs+On4qA/rdEK6lNha4nw8epXZbbmRTxW3KfFF6byPIXZ47h3pQ1up0koZpf/vt96AYLh8dWx6vHuu5As8CT48biaLfNpbR2ldN+2A0yMqHIAiCIAhlRSYfgiAIgiCUFZl8CIIgCIJQViadzYcjR3VeH9JE/8+NXyB1m154kZRNkmqdeu54UC7xujqW6the3Lc+k6c6oseu25NI0rosiq/gAhpfgaOQHUUFi8ER9umy0061ZZOlfo8m4mibnsMT0N+N5KmezjVhf1DHNmHmIJDM03uCMZidgMG/PAbkWF929uprcThp3y2YTeO7OFGEikye2YPk9XEHaPdAhvu5o/GUcdO+S6T1fe/YsYfU+Xx6rDVPayZ1u3dSm49rrtXxTEyDnt+Pwth7mVOaAbTPTWzjYDDtH8UswKnLAQAyWRYPwzn+kYkPHjyoz8fsEpxO2gc43Drfd2BA6+l8bHN7DExlJY3pUyqFPY4BgsOw87YB0HgdLhcdv9jO5OjRo6TO4aD3BNuA8HNef/311vYLL7xA6kp5LvK24mtOp0efliIY1nYTdvaeSsTpcbPomeFxR7Bdkr+CvqvzSWqrkYxpu4VeF93XmdVtKJj0HZZnaQf6B7VNFbZdAQDw+/U7383s87JJfdxchp7j/fc/JOVACI1fZpblQPHmFUt7UFEZpjuj7po3j6YkKKAUDc89/zZta5aOCTOn++cXz2wkdQ888ACcLbLyIQiCIAhCWZHJhyAIgiAIZWXSyS41VTR0dV+kr8ieAHm2lOav0Mt+hknnXQ2N2iXJ42FLknm6vBtPR3SBqQg55IJ5LEZD23rR0llNBb0OThK5olUwF9kACmsd9tO2nRyg17zlXR1yfuElNDNhLqWvs7lpGqmLJalLXxQt71Z46PIlLseHqD5hGKwvnWM/33U4uISl+y6aov2xff8JUg4H9JIp8ywFr18vvcZz1DXQZO6qLo+WPWwuukzc1Ibc+Fg23Lmz9T1xO+lyd6Sfuj8Hw9pN9/jx46TuCJKM5jZQFz6+rA823QYXC6+O3U4LBdrWgkn7wGmM/+sDyyelMroCUPfaUiHT+fe4C28pt1MsSXAXXa9XjwHe5/wcWAbCYeEBaEh37k7M215To93KP/1pmmbg4osvtra5yzDPzIqvhUs7WPbYunUrjJaAF7vaskzdIfpOgZCWFnIFHnpd98EO1p73D35Aym4k+zp8xV2aM8zNnbsCX3yR7kvedzjzsZs9TznQ98/h4m7/dPxcMneBtT1jJpVgB/r1eHlnL3Wttdtp/0QH9Lv6w0M0m3Ima6Jtev4ce8d9atml1vYvnoExR1Y+BEEQBEEoKzL5EARBEAShrMjkQxAEQRCEsjLpbD76kxFSNpCO9vrOV0ldPk/dt6pCWh81mY5YXavtQQajxVPPA1D32io/DZPePaBD7eYd9BwOl9aEIzmqG/qZJmv3onTuTFvuj2qNuCpI7QTmTCse9rY+TDXPPNLwswnqhxvw8BDzxd0qIyn93aogtWVJZmhK8oKDuomNCWwKXVBaT45nqI5pK1CtOZLR98GDNHsAgCpD67WmogYhDpNq73Zk/GMDpmcjHXjGzOmkTimtwXYco3Yc3IN52+u7rO3aOhqm3WXTY713iOrO06ppB2E3vjwbW3nkym4wN1w7u3cGctd86P5vk7oHfvAwjAU//OEPre177rmH1M2YMYOUuRaPwWHIua0Gt6vA9hrcnRcfp1SI8u5u+g7h7qLYLZfbp2zbpl2suR3H0qVLSRm7AnNbDXxdF110UdG2AgwPB1+s7dzm5PXXXy/6PU5kUNswpdP0mnn/ON26fW4nfS57u7X9Axj0OvyBMCnbkVuq10ffTTkUbj2TpWOnbUYbKX/2c5+1tjtPdJI67MY8MEDtd5zIkIy7BSeS9N2YQyk/XH7a1hkB/TsTZ6kD9u3fT8pZ5Cbc0tJE6uZcOEsXTGYrEqN2SQsXzIHxRFY+BEEQBEEoKzL5EARBEAShrEw62eXYCRoVrh8tFcXjfNmVLTejq3V76DJfJqeXsrwuKmWkWGTHygotX1T4qOySiOl9Q166/B30FI9qysWIbE4viSXTdDnsRK9e9ruwjUaw4+WARx/5ZDxC6k5GdXkwQZcA7Wkqwzi8tE+K4WPuZIMRetyEMfZZbXnM1IqgdjWtrKb3x+6h8lEyhZaRDVoXR/c97KfX5WEZKqGgl7h59FobWtIeYv3a2auXaRMsC6nfT6UVmx25Q7Loo1gC4TJCfIg+BxUVuj0VAboUjN1r08z90J5n14wkIy5PjAeHDtEMzkm2bI0lCBw5lsOjdPIl/1LRP7HLalUVvT9YzuES0Ny5c0m5rU0v6/PMtbjtl1xyCakLBOh4xm0vJaXw9nCJBn+XXz++t8uWLSN1jz32GJwpaZz1l7W1kKfnxCELhhR1+08M6WfWw96xNpY1O0DkCzp+sSTsqaDSDr9ON3Ln9zAJuha5Ow8xl2pvtb63EfS+BQDI5YtHPDVY+ID0kP5u53EaLqC2mmVPd+n+mTVzBqlra9MRnnm4CQNouIV8jr9ZxxZZ+RAEQRAEoazI5EMQBEEQhLIikw9BEARBEMrKpLP5GC86O7RO1tgyPvp1DGU4DPpKZ7U918intE5+pvYf5SLWQV2sZ1/9FWubZx2OMjc1QNlpeTZPn0frtfkc1Z37YgOkHApo3TU+QG0Rkqjvsgnanqpw2NoO1FeTuoCPlgcG9TnTTMMPuLUtQI65Fxf8PCy51tfTadrWQFC3h7tfOpirbSZdPJvxeMDvHbdjwHYLM2fOJHXYPoPbWPDj4OvmtizV1fqe8JDl2P6CjyV+HGx3wkOx33TTTadsC8BwV9dS4efxcbk7Lz8udhvmrsjYBoTb2YwEbNXB7SbyLPSAC/WXzc7ccJ26fYaNpT1gIcLzyBbL7qD9k0D2ghUeavNRzTLF5gu637lNTAXKaqtM7saN3JRZyHaT2bkkE/od886eXaQOu8D7WObcyz82n5T37ztibdvYT7xCdh6FHO1XA4rbOo0HsvIhCIIgCEJZkcmHIAiCIAhlRSYfgiAIgiCUlUln83G06xj7ROuaTfU0xsWJE32kbNp1HIWKcPEw3waLHMFjV9gMfc4s01IrcBwJJqHZULjqag/1zW5pqCFlrG8rG4tNgc4ZMakGG4/T8vxKrX0vbm4gdS/s3mlt92epHpnJ0lgIGA/TWbEkm7BTLTnIYqZAtngI7LGiP6ZjaeTYdeQTNPyxC82/41ka3txEcVlw+H0AgJCbjp/sYK+17WUxJvwohoARpMfJ57WW7GT32QDazw6b7ug6ZCsCAOBFdjiFHB2Tvf3UVqKuWsfDAIPaELicKHVAFQuVz2wjnCjsdWMTDeM8HuD08QDD43XgZ6ajo4PUYZsHbvOB44MAUJuHUIj2Abbz4Ofn9hgYHksEn4PbjmC4jYWXpQDA5+S2GtgG5HRxWPB3uT1IHtkqlArDfjpqavX9U2xss7A14HHr9rpYeHUw8mg/1neK2s+4UEh1BzvOUIf+Lalw0/dUVxdNRW+gmDaRQfo82VC+gjSLCQUo1YLBYpDg7wEABJAdYGU1TYVRiVJjNDXQFBrhMH2nuJBtlsPO1hcUbs/o7+VYICsfgiAIgiCUlRFNPjZs2AALFiyAYDAIwWAQ2tvb4Q9/+INVn06nYeXKlVBdXQ1+vx9uvvnmYQmWBEEQBEGY2oxIdmluboZ169bB7NmzQSkFP//5z+GGG26A3bt3wyWXXAJ33303/P73v4enn34aQqEQrFq1Cm666SZ47bXXxqzB0SEagjqf0ktwZoAua81oostTDfVU6sB80KeXaRMsG25dRZiU/W69zFUw2PKhvbis4HHp5V7fadxVC2SZli6nepz6tgW9LLx7krY9gsKmmzbaP30oy2RdfWPJ9sSyKCw4ywjsREuxPDR9zKT9U+GmS4TjwUDHB6jEQgQX6LLodJRlNs3C89uQhKTYmEgzd7tcXvetPc7m9B59v1zMBdOLZDqVp98zCywjrw1di5PWRSMRa7syRJdsVYG1B2UCNVlmy0wGhYl3M0nRx5a/EVzeGg+4KykPoY7dRfPMrbGrq8va5m6nPFMsPk6chcvGdVySwZlqefhy7p45NKTdKvl1YBdZfs38OKXkEv5dTCn5hMtHuMzlo5Hg9+mx7nTRseR2U0lNFZALfAV7V6I+yOdof+TZT1oqq/snm6FjtICkZruNfm9YFuK4flcair5T/D79TjNYpmMXeoc47FT6SibpcT5+5WJru7KWSoEVSF7C7/9TsbRdH6egqDu8Iu9uPgbK62o7osnH9ddfT8oPP/wwbNiwAbZt2wbNzc3w+OOPw1NPPQWf+cxnAADgiSeegLlz58K2bdvgiiuuGLtWC4IgCIIwaRm1zUehUICNGzdCIpGA9vZ22LVrF+RyOVi+fLm1z5w5c6C1tRW2bt1a9DiZTAZisRj5EwRBEATh/GXEk4933nkH/H4/uN1uuOOOO+CZZ56Biy++GLq6usDlckGYWeHX19eTJU/O2rVrIRQKWX8tLS0jvghBEARBECYPI3a1veiii2DPnj0QjUbhN7/5DaxYsQJeeeWVUTdgzZo1sHr1aqsci8VKTkDqglQbzKP0wYOd1AVqwYzZpHxsgLpSYuwoZK+PdUull7rmOZGLVI6lPva4dTnK0qdjn1SeUvq9Ix+Sci6vdWm/t7idRCLBXG2ZK+vRAT3xq1VUow6Hka7I9NAq5tZoi+m2x5O0n50OrWUqZg8ylCtvCG4AgFjX22e8747e98f8/MuWXk3KOMV9AWh/uEw0fliK63SK2iYQd0ST6sc25F53vLOX1LU01ZGyUljr5TYfyCW1QLV2g/2rgm0Kksnxd6HeuHHjuBx3xowZpIz/gbIzDR+7vvI6bI/Bw6uX2pfXYbdY7j47WldXHsKdh6rH9dwtF9t8lLIjOR11VfXWdsGkz0Ge2WYpFJYgWEHff1lkq5Flbv8qT2017Ibu53ffeY/U1dboUPkXzm4jdQMD1FHCi1xxu/upu/6BA/ut7Ytm0nAGtdX0PYopqOI2FpU+arcFCl1nntmxsXuCXW+H2Lu6kEM2Qszmg4dXN/gDP8aMePLhcrlg1qxZAACwaNEi2LlzJ/zoRz+CL33pS5DNZiESiZCHt7u7GxoaGooc7c8+7qX83AVBEARBOL8466mNaZqQyWRg0aJF4HQ6YfPmzVbdgQMH4NixY9De3n62pxEEQRAE4TxhRCsfa9asgWuvvRZaW1shHo/DU089BX/84x/h+eefh1AoBLfffjusXr0aqqqqIBgMwre+9S1ob28XTxdBEARBECxGNPno6emB2267DTo7OyEUCsGCBQvg+eefh89+9rMAAPDII4+AzWaDm2++GTKZDFxzzTXw4x//eEwbbBaopockLPB46UJOBVNz5jY2W9vvHTtA6pwoZXJNgOpttSGqOWLd80gfDeFO2pZnPtamPkdPmur5ceY7rpz6OlN56gHkRLpmKkV1O8ViMaSV1qgdSdo/Xm9xuas/Qq8rl9Jt5ynIvX6tF1ewvpvFyicGi/fX+cLr2zeffqcysm2iG3CO89Of/nTMj/mv//qvpMzjfmDbDR5TAttYnM7GA9uW8NgmGP7M8uPic3L7kEBAxxLCsUxGDIqlMW/ORaQqkqTxVE6e1OHNe3sHSN0gCm9eYHFq8szmDJtVKNbPkQEduyMSpe9YX47avQwldf8cOvgBqbMZel9uQtA6XZscuB20zu5gdkAm/g1g9iDI/uJ0Zj8xZs9D20oOSk8BtD1XXPfF0ic6S0Y0+Xj88cdL1ns8Hli/fj2sX7/+rBolCIIgCML5i+R2EQRBEAShrEy6rLZK0aUzb0AvO4bD1CX2SBfNbDm9WodbbwxVk7pMWi9L2llm1uPd9Dh+f9jaDrqZy6NPtyHZR921Mkja4WtnaUVDGmdKZMg0UJ3DoC59TtZ2vLSXKhQ/5tAQXfa0sWXiLFrSVWxp0YOklWSmtGttdah4iHtBOF/4xje+MdFNOOdIoNADv9n036RuwfwFpNxQr+WKVIq+m9woNcWWV94gdUmWXgK7kwaZfJ5F79Etr9FAmM0sS3Ntvf7t8PiptB0f0LLQzOmzSF0AZ5tmJgOKpU+44v+7GaYSsvIhCIIgCEJZkcmHIAiCIAhlRSYfgiAIgiCUFUPxHM0TTCwWg1AoBPfdd59EPhUEQRCESUImk4F169ZBNBqFYDBYcl9Z+RAEQRAEoazI5EMQBEEQhLIikw9BEARBEMqKTD4EQRAEQSgrMvkQBEEQBKGsnHMRTj9yvslkMqfZUxAEQRCEc4WPfrfPxIn2nHO1PX78OLS0tEx0MwRBEARBGAUdHR3Q3Nxccp9zbvJhmiacPHkSlFLQ2toKHR0dp/UXnorEYjFoaWmR/imC9E9ppH9KI/1TGumf4kzlvlFKQTweh6amJrDZSlt1nHOyi81mg+bmZojFYgAAEAwGp9wNHAnSP6WR/imN9E9ppH9KI/1TnKnaN6FQ6Iz2E4NTQRAEQRDKikw+BEEQBEEoK+fs5MPtdsN3vvMdye9SBOmf0kj/lEb6pzTSP6WR/imO9M2Zcc4ZnAqCIAiCcH5zzq58CIIgCIJwfiKTD0EQBEEQyopMPgRBEARBKCsy+RAEQRAEoazI5EMQBEEQhLJyzk4+1q9fDzNmzACPxwNLly6FHTt2THSTys7atWvh8ssvh0AgAHV1dXDjjTfCgQMHyD7pdBpWrlwJ1dXV4Pf74eabb4bu7u4JavHEsm7dOjAMA+666y7rs6nePydOnICvfOUrUF1dDV6vF+bPnw9vvPGGVa+UggcffBAaGxvB6/XC8uXL4dChQxPY4vJRKBTggQcegLa2NvB6vXDBBRfAQw89RJJiTaX+efXVV+H666+HpqYmMAwDNm3aROrPpC8GBgbg1ltvhWAwCOFwGG6//XYYGhoq41WMH6X6J5fLwb333gvz58+HiooKaGpqgttuuw1OnjxJjnE+98+IUecgGzduVC6XS/3bv/2bevfdd9Xf/M3fqHA4rLq7uye6aWXlmmuuUU888YTau3ev2rNnj/qLv/gL1draqoaGhqx97rjjDtXS0qI2b96s3njjDXXFFVeoZcuWTWCrJ4YdO3aoGTNmqAULFqg777zT+nwq98/AwICaPn26+upXv6q2b9+uPvzwQ/X888+r999/39pn3bp1KhQKqU2bNqm33npLfeELX1BtbW0qlUpNYMvLw8MPP6yqq6vVs88+qw4fPqyefvpp5ff71Y9+9CNrn6nUP//zP/+jvv3tb6vf/va3CgDUM888Q+rPpC8+//nPq0svvVRt27ZN/elPf1KzZs1St9xyS5mvZHwo1T+RSEQtX75c/frXv1b79+9XW7duVUuWLFGLFi0ixzif+2eknJOTjyVLlqiVK1da5UKhoJqamtTatWsnsFUTT09PjwIA9corryil/jzgnU6nevrpp6193nvvPQUAauvWrRPVzLITj8fV7Nmz1Ysvvqg+9alPWZOPqd4/9957r/r4xz9etN40TdXQ0KD+6Z/+yfosEokot9utfvWrX5WjiRPKddddp77+9a+Tz2666SZ16623KqWmdv/wH9cz6Yt9+/YpAFA7d+609vnDH/6gDMNQJ06cKFvby8GpJmecHTt2KABQR48eVUpNrf45E8452SWbzcKuXbtg+fLl1mc2mw2WL18OW7duncCWTTzRaBQAAKqqqgAAYNeuXZDL5UhfzZkzB1pbW6dUX61cuRKuu+460g8A0j//9V//BYsXL4a//Mu/hLq6Oli4cCH87Gc/s+oPHz4MXV1dpH9CoRAsXbp0SvTPsmXLYPPmzXDw4EEAAHjrrbdgy5YtcO211wKA9A/mTPpi69atEA6HYfHixdY+y5cvB5vNBtu3by97myeaaDQKhmFAOBwGAOkfzjmX1bavrw8KhQLU19eTz+vr62H//v0T1KqJxzRNuOuuu+DKK6+EefPmAQBAV1cXuFwua3B/RH19PXR1dU1AK8vPxo0b4c0334SdO3cOq5vq/fPhhx/Chg0bYPXq1XD//ffDzp074e///u/B5XLBihUrrD441bM2Ffrnvvvug1gsBnPmzAG73Q6FQgEefvhhuPXWWwEApnz/YM6kL7q6uqCuro7UOxwOqKqqmnL9lU6n4d5774VbbrnFymwr/UM55yYfwqlZuXIl7N27F7Zs2TLRTTln6OjogDvvvBNefPFF8Hg8E92ccw7TNGHx4sXwgx/8AAAAFi5cCHv37oWf/OQnsGLFiglu3cTzn//5n/DLX/4SnnrqKbjkkktgz549cNddd0FTU5P0jzBqcrkc/NVf/RUopWDDhg0T3ZxzlnNOdqmpqQG73T7MI6G7uxsaGhomqFUTy6pVq+DZZ5+Fl19+GZqbm63PGxoaIJvNQiQSIftPlb7atWsX9PT0wMc+9jFwOBzgcDjglVdegcceewwcDgfU19dP6f5pbGyEiy++mHw2d+5cOHbsGACA1QdT9Vn7h3/4B7jvvvvgy1/+MsyfPx/++q//Gu6++25Yu3YtAEj/YM6kLxoaGqCnp4fU5/N5GBgYmDL99dHE4+jRo/Diiy9aqx4A0j+cc27y4XK5YNGiRbB582brM9M0YfPmzdDe3j6BLSs/SilYtWoVPPPMM/DSSy9BW1sbqV+0aBE4nU7SVwcOHIBjx45Nib66+uqr4Z133oE9e/ZYf4sXL4Zbb73V2p7K/XPllVcOc80+ePAgTJ8+HQAA2traoKGhgfRPLBaD7du3T4n+SSaTYLPRV6DdbgfTNAFA+gdzJn3R3t4OkUgEdu3aZe3z0ksvgWmasHTp0rK3udx8NPE4dOgQ/O///i9UV1eT+qneP8OYaIvXU7Fx40bldrvVk08+qfbt26e++c1vqnA4rLq6uia6aWXlb//2b1UoFFJ//OMfVWdnp/WXTCatfe644w7V2tqqXnrpJfXGG2+o9vZ21d7ePoGtnliwt4tSU7t/duzYoRwOh3r44YfVoUOH1C9/+Uvl8/nUf/zHf1j7rFu3ToXDYfW73/1Ovf322+qGG244b11JOStWrFDTpk2zXG1/+9vfqpqaGnXPPfdY+0yl/onH42r37t1q9+7dCgDUP//zP6vdu3db3hpn0hef//zn1cKFC9X27dvVli1b1OzZs88bV9JS/ZPNZtUXvvAF1dzcrPbs2UPe15lMxjrG+dw/I+WcnHwopdS//Mu/qNbWVuVyudSSJUvUtm3bJrpJZQcATvn3xBNPWPukUin1d3/3d6qyslL5fD71xS9+UXV2dk5coycYPvmY6v3z3//932revHnK7XarOXPmqJ/+9Kek3jRN9cADD6j6+nrldrvV1VdfrQ4cODBBrS0vsVhM3Xnnnaq1tVV5PB41c+ZM9e1vf5v8WEyl/nn55ZdP+b5ZsWKFUurM+qK/v1/dcsstyu/3q2AwqL72ta+peDw+AVcz9pTqn8OHDxd9X7/88svWMc7n/hkphlIonJ8gCIIgCMI4c87ZfAiCIAiCcH4jkw9BEARBEMqKTD4EQRAEQSgrMvkQBEEQBKGsyORDEARBEISyIpMPQRAEQRDKikw+BEEQBEEoKzL5EARBEAShrMjkQxAEQRCEsiKTD0EQBEEQyopMPgRBEARBKCv/FzVp25aAktH+AAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# 이 코드는 데이터셋에서 가져온 예시 이미지들을 시각화하고, 해당 이미지들의 라벨을 출력하는 부분입니다.\n",
        "\n",
        "# imshow: 이미지를 시각화하기 위한 함수입니다. 입력된 이미지를 0.5로 나누고 0.5를 더하여 범위를 [0, 1]로 변환합니다. 그리고 이미지를 numpy 배열로 변환하고, plt.imshow를 사용하여 이미지를 표시합니다.\n",
        "# dataiter: trainloader를 이터레이터로 변환합니다.\n",
        "# imgs, labels: next(dataiter)를 통해 다음 미니배치의 이미지와 라벨을 가져옵니다.\n",
        "# torchvision.utils.make_grid: 이미지들을 그리드 형태로 결합하여 하나의 이미지로 만들어줍니다.\n",
        "# plt.imshow를 사용하여 결합된 이미지를 표시합니다.\n",
        "# ' '.join('%5s' % classes[labels[j]] for j in range(4)): 4개의 이미지에 대한 라벨을 출력합니다. labels에는 이미지들의 정답 라벨이 들어있고, classes에는 CIFAR-10 클래스들이 들어있습니다.\n",
        "#  이를 활용하여 정답 라벨을 문자열로 변환하고, ' '.join()을 사용하여 공백을 포함하여 출력합니다.\n",
        "\n",
        "# Examples of dataset\n",
        "# 예시 이미지를 가져온 것\n",
        "def imshow(img):\n",
        "  img = img /2 + 0.5\n",
        "  npimg = img.numpy()\n",
        "  plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
        "\n",
        "dataiter = iter(trainloader)\n",
        "imgs, labels = next(dataiter)\n",
        "\n",
        "imshow(torchvision.utils.make_grid(imgs))\n",
        "print(' '.join('%5s' % classes[labels[j]] for j in range(4)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y77itjthc5Xh"
      },
      "source": [
        "# **Your own CNN**\n",
        "\n",
        "You can implement your own network using libraries such as `torch.nn`and `torch.nn.functional`.\n",
        "\n",
        "`SimpleNet` and `VGG11` are examples to help your understand the implementation of the network.\n",
        "\n",
        "So, you can modify the given codes or create another awesome neural network for CIFAR-10 classification."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "11oMT_o1mnL2"
      },
      "outputs": [],
      "source": [
        "# VGG Model\n",
        "\n",
        "# 이 코드는 CIFAR-10 이미지 분류를 위한 VGG11 모델을 정의하는 부분입니다.\n",
        "\n",
        "# cfg: VGG11 아키텍처의 구성을 나타내는 딕셔너리입니다. 딕셔너리의 값은 컨볼루션 레이어와 풀링 레이어의 출력 채널 수를 나타냅니다.\n",
        "# VGG11: nn.Module을 상속하는 클래스입니다. 이 클래스는 모델의 구조와 forward 연산을 정의합니다.\n",
        "# __init__ 메서드: 모델의 레이어들을 초기화합니다. make_layers 함수를 호출하여 컨볼루션 레이어와 분류기 레이어를 생성합니다.\n",
        "# self.features: 컨볼루션 레이어들로 구성된 특성 추출기입니다. make_layers 함수를 통해 생성된 레이어들을 포함합니다.\n",
        "# self.classifier: 분류기 레이어입니다. 512개의 입력을 받아 10개의 출력을 생성하는 선형 레이어입니다.\n",
        "# make_layers 메서드: 주어진 구성(cfg)에 따라 컨볼루션 레이어들을 생성하여 순차적으로 연결합니다.\n",
        "# layers 리스트에 컨볼루션, 배치 정규화 및 ReLU 활성화 함수를 포함한 레이어들을 추가합니다.\n",
        "# nn.MaxPool2d를 사용하여 풀링 레이어를 추가합니다.\n",
        "# forward 메서드: 입력 데이터를 받아 순전파 연산을 수행합니다. 특성 추출기를 통과한 후에는 2D 텐서를 1D로 평탄화하여 분류기에 입력으로 제공합니다.\n",
        "# net: VGG11 클래스의 인스턴스를 생성하고, to(device)를 사용하여 모델을 지정한 디바이스 (GPU 또는 CPU)로 이동합니다.\n",
        "\n",
        "cfg = {\n",
        "    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],\n",
        "    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M']\n",
        "    }  \n",
        "\n",
        "class VGG16(nn.Module):\n",
        "    def __init__(self,\n",
        "                 in_channels=3,\n",
        "                 num_classes=1000,\n",
        "                 init_weights=True,\n",
        "                 model='VGG16'):\n",
        "        super(VGG16, self).__init__()\n",
        "        self.in_channels = in_channels\n",
        "        self.features = self.make_layers(cfg[model])\n",
        "        self.classifier = nn.Linear(512, 10)\n",
        "\n",
        "        self.fc_layers = nn.Sequential(\n",
        "            nn.Linear(512 * 4 * 4, 4096),\n",
        "            nn.ReLU(True),\n",
        "            nn.Dropout(),\n",
        "            nn.Linear(4096, 4096),\n",
        "            nn.ReLU(True),\n",
        "            nn.Dropout(p=0.5),\n",
        "            nn.Linear(4096, num_classes)\n",
        "        )\n",
        "\n",
        "        if init_weights == True:\n",
        "          self._initialize_weights()\n",
        "\n",
        "\n",
        "    def forward(self, x):\n",
        "        #out = F.normalize(x, dim=0)\n",
        "        #out = F.normalize(x, p=2.0, dim=1, eps=1e-12, out=None)\n",
        "        out = self.features(x)\n",
        "        out = out.view(out.size(0), -1)\n",
        "        out = self.classifier(out)\n",
        "        return out\n",
        "\n",
        "    # def make_layers(self, cfg):\n",
        "    def make_layers(self, cfg):\n",
        "        layers = []\n",
        "        in_channels = 3\n",
        "        for x in cfg:\n",
        "            if x == 'M':\n",
        "                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]\n",
        "            else:\n",
        "                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),\n",
        "                           nn.BatchNorm2d(x),\n",
        "                           nn.ReLU(inplace=True)]\n",
        "                in_channels = x\n",
        "        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]\n",
        "        return nn.Sequential(*layers)\n",
        "\n",
        "    def _initialize_weights(self):\n",
        "      # modules -> Sequential의 모든 layer\n",
        "      for m in self.modules():\n",
        "        # Conv layer 초기화\n",
        "        if isinstance(m, nn.Conv2d):\n",
        "            # he_initialization(ReLU에 최적화)\n",
        "            nn.init.kaiming_normal_(m.weight,\n",
        "                                    mode='fan_out',\n",
        "                                    nonlinearity='relu')\n",
        "            if m.bias is not None:\n",
        "                nn.init.constant_(m.bias, 0)\n",
        "\n",
        "        elif isinstance(m, nn.BatchNorm2d):\n",
        "            nn.init.constant_(m.weight, 1)\n",
        "            nn.init.constant_(m.bias, 0)\n",
        "\n",
        "        elif isinstance(m, nn.Linear):\n",
        "            nn.init.normal_(m.weight, 0, 0.01)\n",
        "            nn.init.normal_(m.bias, 0)\n",
        "\n",
        "\n",
        "net = VGG16().to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t6UTnTJsBC7B",
        "outputId": "ea37bfeb-02cb-43d8-f94b-047a01bf7661"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "VGG16(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (24): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (25): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (26): ReLU(inplace=True)\n",
            "    (27): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (35): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (36): ReLU(inplace=True)\n",
            "    (37): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (38): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (39): ReLU(inplace=True)\n",
            "    (40): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (44): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            "  (fc_layers): Sequential(\n",
            "    (0): Linear(in_features=8192, out_features=4096, bias=True)\n",
            "    (1): ReLU(inplace=True)\n",
            "    (2): Dropout(p=0.5, inplace=False)\n",
            "    (3): Linear(in_features=4096, out_features=4096, bias=True)\n",
            "    (4): ReLU(inplace=True)\n",
            "    (5): Dropout(p=0.5, inplace=False)\n",
            "    (6): Linear(in_features=4096, out_features=1000, bias=True)\n",
            "  )\n",
            ")\n",
            "----------------------------------------------------------------\n",
            "        Layer (type)               Output Shape         Param #\n",
            "================================================================\n",
            "            Conv2d-1           [-1, 64, 32, 32]           1,792\n",
            "       BatchNorm2d-2           [-1, 64, 32, 32]             128\n",
            "              ReLU-3           [-1, 64, 32, 32]               0\n",
            "            Conv2d-4           [-1, 64, 32, 32]          36,928\n",
            "       BatchNorm2d-5           [-1, 64, 32, 32]             128\n",
            "              ReLU-6           [-1, 64, 32, 32]               0\n",
            "         MaxPool2d-7           [-1, 64, 16, 16]               0\n",
            "            Conv2d-8          [-1, 128, 16, 16]          73,856\n",
            "       BatchNorm2d-9          [-1, 128, 16, 16]             256\n",
            "             ReLU-10          [-1, 128, 16, 16]               0\n",
            "           Conv2d-11          [-1, 128, 16, 16]         147,584\n",
            "      BatchNorm2d-12          [-1, 128, 16, 16]             256\n",
            "             ReLU-13          [-1, 128, 16, 16]               0\n",
            "        MaxPool2d-14            [-1, 128, 8, 8]               0\n",
            "           Conv2d-15            [-1, 256, 8, 8]         295,168\n",
            "      BatchNorm2d-16            [-1, 256, 8, 8]             512\n",
            "             ReLU-17            [-1, 256, 8, 8]               0\n",
            "           Conv2d-18            [-1, 256, 8, 8]         590,080\n",
            "      BatchNorm2d-19            [-1, 256, 8, 8]             512\n",
            "             ReLU-20            [-1, 256, 8, 8]               0\n",
            "           Conv2d-21            [-1, 256, 8, 8]         590,080\n",
            "      BatchNorm2d-22            [-1, 256, 8, 8]             512\n",
            "             ReLU-23            [-1, 256, 8, 8]               0\n",
            "        MaxPool2d-24            [-1, 256, 4, 4]               0\n",
            "           Conv2d-25            [-1, 512, 4, 4]       1,180,160\n",
            "      BatchNorm2d-26            [-1, 512, 4, 4]           1,024\n",
            "             ReLU-27            [-1, 512, 4, 4]               0\n",
            "           Conv2d-28            [-1, 512, 4, 4]       2,359,808\n",
            "      BatchNorm2d-29            [-1, 512, 4, 4]           1,024\n",
            "             ReLU-30            [-1, 512, 4, 4]               0\n",
            "           Conv2d-31            [-1, 512, 4, 4]       2,359,808\n",
            "      BatchNorm2d-32            [-1, 512, 4, 4]           1,024\n",
            "             ReLU-33            [-1, 512, 4, 4]               0\n",
            "        MaxPool2d-34            [-1, 512, 2, 2]               0\n",
            "           Conv2d-35            [-1, 512, 2, 2]       2,359,808\n",
            "      BatchNorm2d-36            [-1, 512, 2, 2]           1,024\n",
            "             ReLU-37            [-1, 512, 2, 2]               0\n",
            "           Conv2d-38            [-1, 512, 2, 2]       2,359,808\n",
            "      BatchNorm2d-39            [-1, 512, 2, 2]           1,024\n",
            "             ReLU-40            [-1, 512, 2, 2]               0\n",
            "           Conv2d-41            [-1, 512, 2, 2]       2,359,808\n",
            "      BatchNorm2d-42            [-1, 512, 2, 2]           1,024\n",
            "             ReLU-43            [-1, 512, 2, 2]               0\n",
            "        MaxPool2d-44            [-1, 512, 1, 1]               0\n",
            "        AvgPool2d-45            [-1, 512, 1, 1]               0\n",
            "           Linear-46                   [-1, 10]           5,130\n",
            "================================================================\n",
            "Total params: 14,728,266\n",
            "Trainable params: 14,728,266\n",
            "Non-trainable params: 0\n",
            "----------------------------------------------------------------\n",
            "Input size (MB): 0.01\n",
            "Forward/backward pass size (MB): 6.57\n",
            "Params size (MB): 56.18\n",
            "Estimated Total Size (MB): 62.77\n",
            "----------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "print(net)\n",
        "summary_(net, input_size=(3, 32, 32), device=device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXLnGlLgdyZ2"
      },
      "source": [
        "# **Loss function and optimizer**\n",
        "\n",
        "Set the **loss function and optimizer** for training CNN.\n",
        "You can modify the loss function or optimizer for better performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oB0f6C2npeoj"
      },
      "outputs": [],
      "source": [
        "# Loss function and optimizer\n",
        "\n",
        "# 이 코드는 손실 함수와 옵티마이저를 설정하는 부분입니다.\n",
        "\n",
        "# loss_fun: 분류 작업을 위한 손실 함수로서, nn.CrossEntropyLoss()를 사용합니다. 이 함수는 다중 클래스 분류에서 사용되며, 입력으로 예측값과 정답 라벨을 받아 손실을 계산합니다.\n",
        "# optimizer: 모델의 파라미터를 업데이트하는데 사용되는 최적화 알고리즘입니다. 여기서는 Adam 옵티마이저를 사용하며, net.parameters()를 통해 모델의 학습 가능한 파라미터들을 전달합니다.\n",
        "#  lr=0.001은 학습률(learning rate)로서, 파라미터 업데이트의 크기를 결정합니다.\n",
        "# scheduler: 학습률 스케줄러입니다. 여기서는 StepLR 스케줄러를 사용합니다. step_size는 학습률을 조절하는 주기(step size)를 나타내며, gamma는 학습률을 감소시키는 비율을 나타냅니다. 주어진 주기마다 학습률을 감소시키는 데 사용됩니다.\n",
        "\n",
        "\n",
        "\n",
        "loss_fun = nn.CrossEntropyLoss()    # -log에 대해 값들이 10개로 나옴. 이 값들에 exponential을 취함. 왜 하\n",
        "optimizer = torch.optim.Adam(net.parameters(), lr=0.0001)\n",
        "# optimizer의 매개변수로 weight decay value를 넣어줄 수 있는데, 이때 이 값은 앞선 식에서 lambda를 의미한다. \n",
        "scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.9)\n",
        "#scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1) # 지? log를 활용하기 때문에 계산을 편하게 하기 위해. CrossEntropyLoss는 필수 활용해라"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DInMRCnReWno"
      },
      "source": [
        "# **Train the network**\n",
        "\n",
        "Train your own network using the above loss function and optimizer."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# tensor board\n",
        "from torch.utils.tensorboard import SummaryWriter\n",
        "writer = SummaryWriter('./logs/vgg16')"
      ],
      "metadata": {
        "id": "i1bivW9w3m-j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2vvD0eNCpqxj",
        "outputId": "9394e79d-db5d-4a6d-c88c-595f8fb5c691"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[Epoch - 1, Iteration -  5000] Loss: 1.839\n",
            "[Epoch - 1, Iteration - 10000] Loss: 0.742\n",
            "[Epoch - 1] Loss: 1.593\n",
            "[Epoch - 2, Iteration -  5000] Loss: 1.227\n",
            "[Epoch - 2, Iteration - 10000] Loss: 0.557\n",
            "[Epoch - 2] Loss: 1.146\n",
            "[Epoch - 3, Iteration -  5000] Loss: 0.983\n",
            "[Epoch - 3, Iteration - 10000] Loss: 0.475\n",
            "[Epoch - 3] Loss: 0.955\n",
            "[Epoch - 4, Iteration -  5000] Loss: 0.861\n",
            "[Epoch - 4, Iteration - 10000] Loss: 0.421\n",
            "[Epoch - 4] Loss: 0.846\n",
            "[Epoch - 5, Iteration -  5000] Loss: 0.775\n",
            "[Epoch - 5, Iteration - 10000] Loss: 0.379\n",
            "[Epoch - 5] Loss: 0.762\n",
            "[Epoch - 6, Iteration -  5000] Loss: 0.703\n",
            "[Epoch - 6, Iteration - 10000] Loss: 0.339\n",
            "[Epoch - 6] Loss: 0.686\n",
            "[Epoch - 7, Iteration -  5000] Loss: 0.644\n",
            "[Epoch - 7, Iteration - 10000] Loss: 0.319\n",
            "[Epoch - 7] Loss: 0.638\n",
            "[Epoch - 8, Iteration -  5000] Loss: 0.599\n",
            "[Epoch - 8, Iteration - 10000] Loss: 0.304\n",
            "[Epoch - 8] Loss: 0.602\n",
            "[Epoch - 9, Iteration -  5000] Loss: 0.571\n",
            "[Epoch - 9, Iteration - 10000] Loss: 0.282\n",
            "[Epoch - 9] Loss: 0.564\n",
            "[Epoch - 10, Iteration -  5000] Loss: 0.526\n",
            "[Epoch - 10, Iteration - 10000] Loss: 0.268\n",
            "[Epoch - 10] Loss: 0.534\n",
            "[Epoch - 11, Iteration -  5000] Loss: 0.498\n",
            "[Epoch - 11, Iteration - 10000] Loss: 0.247\n",
            "[Epoch - 11] Loss: 0.495\n",
            "[Epoch - 12, Iteration -  5000] Loss: 0.465\n",
            "[Epoch - 12, Iteration - 10000] Loss: 0.239\n",
            "[Epoch - 12] Loss: 0.470\n",
            "[Epoch - 13, Iteration -  5000] Loss: 0.443\n",
            "[Epoch - 13, Iteration - 10000] Loss: 0.223\n",
            "[Epoch - 13] Loss: 0.446\n",
            "[Epoch - 14, Iteration -  5000] Loss: 0.424\n",
            "[Epoch - 14, Iteration - 10000] Loss: 0.212\n",
            "[Epoch - 14] Loss: 0.424\n",
            "[Epoch - 15, Iteration -  5000] Loss: 0.408\n",
            "[Epoch - 15, Iteration - 10000] Loss: 0.204\n",
            "[Epoch - 15] Loss: 0.412\n",
            "[Epoch - 16, Iteration -  5000] Loss: 0.380\n",
            "[Epoch - 16, Iteration - 10000] Loss: 0.195\n",
            "[Epoch - 16] Loss: 0.386\n",
            "[Epoch - 17, Iteration -  5000] Loss: 0.366\n",
            "[Epoch - 17, Iteration - 10000] Loss: 0.184\n"
          ]
        }
      ],
      "source": [
        "# 이 코드는 모델을 학습시키는 부분입니다.\n",
        "\n",
        "# epochs: 전체 데이터셋을 몇 번 반복하여 학습할지를 결정하는 변수입니다.\n",
        "# for epoch in range(epochs):: 주어진 epoch 수만큼 반복합니다.\n",
        "# loss_tmp와 epoch_loss는 손실(loss) 값을 기록하기 위한 변수입니다.\n",
        "# for i, data in enumerate(trainloader, start=0):: trainloader를 이용하여 미니배치 단위로 데이터를 가져옵니다. i는 현재 미니배치의 인덱스를 나타냅니다.\n",
        "# inputs, labels = data: 데이터로부터 입력(inputs)과 라벨(labels)을 가져옵니다.\n",
        "# inputs = inputs.to(device), labels = labels.to(device): 데이터를 지정한 디바이스(GPU 또는 CPU)로 이동합니다.\n",
        "# outputs = net(inputs): 입력을 모델에 전달하여 예측값을 계산합니다.\n",
        "# loss = loss_fun(outputs, labels): 모델의 출력과 정답 라벨을 사용하여 손실을 계산합니다.\n",
        "# optimizer.zero_grad(): 옵티마이저의 그래디언트를 초기화합니다.\n",
        "# loss.backward(): 손실에 대한 역전파를 수행하여 그래디언트를 계산합니다.\n",
        "# optimizer.step(): 그래디언트를 사용하여 모델의 파라미터를 업데이트합니다.\n",
        "# loss_tmp += loss.data, epoch_loss += loss.data: 현재 미니배치의 손실 값을 누적합니다.\n",
        "# if i % 5000 == 4999:: 일정한 주기로 손실 값을 출력합니다. 여기서는 매 5000번째 미니배치마다 출력합니다.\n",
        "# scheduler.step(): 학습률 스케줄러를 통해 학습률을 업데이트합니다.\n",
        "# print('[Epoch - %d] Loss: %.3f' %(epoch + 1, epoch_loss / (i+1))): 에포크마다 손실 값을 출력합니다.\n",
        "# print('Finished Training'): 학습이 완료되었음을 출력합니다.\n",
        "\n",
        "# Train the model\n",
        "epochs =30  # number of epochs  \n",
        "\n",
        "for epoch in range(epochs):\n",
        "\n",
        "    loss_tmp = 0.0\n",
        "    epoch_loss = 0.0 \n",
        "    \n",
        "    for i, data in enumerate(trainloader, start=0):\n",
        "        # Load the data\n",
        "        inputs, labels = data\n",
        "        inputs = inputs.to(device)    # cpu에 있는 코드를 gpu로 보내줌\n",
        "        labels = labels.to(device)\n",
        "\n",
        "        # Estimate the output using the network\n",
        "        outputs = net(inputs)\n",
        "\n",
        "        # Calculate the loss between the output of the network and label\n",
        "        loss = loss_fun(outputs, labels)\n",
        "\n",
        "        # Optimize the network \n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()   # backpropagation 해서 gradient 구한다\n",
        "        optimizer.step()\n",
        "\n",
        "        loss_tmp += loss.data\n",
        "        epoch_loss += loss.data\n",
        "\n",
        "        if i % 5000 == 4999:    # Print loss every 5000 mini-batches\n",
        "            print('[Epoch - %d, Iteration - %5d] Loss: %.3f' %\n",
        "                  (epoch + 1, i + 1, loss_tmp / (i+1)))\n",
        "            loss_tmp = 0.0\n",
        "\n",
        "    # Update the learning rate according to the learnig rate scheduler\n",
        "    scheduler.step()\n",
        "\n",
        "    # tensorboard: train_loss 기록\n",
        "    writer.add_scalar('Loss/train', epoch_loss / (i+1), epoch)\n",
        "\n",
        "    # Print the epoch loss\n",
        "    print('[Epoch - %d] Loss: %.3f' %(epoch + 1, epoch_loss / (i+1)))\n",
        "\n",
        "print('Finished Training')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "erdS6tqsBBYk"
      },
      "outputs": [],
      "source": [
        "writer.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CgDA6_LlfNhE"
      },
      "source": [
        "# **Test the network**\n",
        "\n",
        "Test the trained network using the testset.\n",
        "\n",
        "Accuracy of the network on the 10,000 test images is the final accuracy of your network. \n",
        "\n",
        "The closer the accuray is to 100%, the better the network classifies the input image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fNfptwqLqpdA"
      },
      "outputs": [],
      "source": [
        "# dataiter_test = iter(testloader): 테스트 데이터 로더에서 데이터를 가져옵니다.\n",
        "# img_test, labels_test = next(dataiter_test): 테스트 데이터에서 이미지와 라벨을 가져옵니다.\n",
        "# imshow(torchvision.utils.make_grid(img_test)): 테스트 이미지를 시각화하여 출력합니다.\n",
        "# print('GroundTruth: ', ' '.join('%5s' % classes[labels_test[j]] for j in range(4))): 실제 정답 라벨을 출력합니다.\n",
        "# img_test = img_test.to(device), labels_test = labels_test.to(device): 데이터를 지정한 디바이스(GPU 또는 CPU)로 이동합니다.\n",
        "# outputs_test = net(img_test): 테스트 이미지를 모델에 전달하여 예측값을 계산합니다.\n",
        "# _, predicted = torch.max(outputs_test.data, 1): 예측값 중 가장 큰 값을 가지는 클래스를 선택합니다.\n",
        "# print('Predicted: ', ' '.join('%5s' % classes[predicted[j]] for j in range(4))): 예측된 클래스를 출력합니다.\n",
        "\n",
        "# Test the trained model with sample\n",
        "\n",
        "# test코드는 고치지 말고 그대로 사용해도 ㄱㅊ\n",
        "\n",
        "dataiter_test = iter(testloader)\n",
        "img_test, labels_test = next(dataiter_test)\n",
        "\n",
        "imshow(torchvision.utils.make_grid(img_test))\n",
        "print('GroundTruth: ', ' '.join('%5s' % classes[labels_test[j]] for j in range(4)))\n",
        "\n",
        "img_test = img_test.to(device)\n",
        "labels_test = labels_test.to(device)\n",
        "\n",
        "# Prediction\n",
        "outputs_test = net(img_test)\n",
        "_, predicted = torch.max(outputs_test.data, 1)\n",
        "print('Predicted: ', ' '.join('%5s' % classes[predicted[j]] for j in range(4)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DNrXRT1NrJft"
      },
      "outputs": [],
      "source": [
        "# epochs: 전체 데이터셋을 몇 번 반복하여 학습할지를 결정하는 변수입니다.\n",
        "# for epoch in range(epochs):: 주어진 epoch 수만큼 반복합니다.\n",
        "# loss_tmp와 epoch_loss는 손실(loss) 값을 기록하기 위한 변수입니다.\n",
        "# for i, data in enumerate(trainloader, start=0):: trainloader를 이용하여 미니배치 단위로 데이터를 가져옵니다. i는 현재 미니배치의 인덱스를 나타냅니다.\n",
        "# inputs, labels = data: 데이터로부터 입력(inputs)과 라벨(labels)을 가져옵니다.\n",
        "# inputs = inputs.to(device), labels = labels.to(device): 데이터를 지정한 디바이스(GPU 또는 CPU)로 이동합니다.\n",
        "# outputs = net(inputs): 입력을 모델에 전달하여 예측값을 계산합니다.\n",
        "# loss = loss_fun(outputs, labels): 모델의 출력과 정답 라벨을 사용하여 손실을 계산합니다.\n",
        "# optimizer.zero_grad(): 옵티마이저의 그래디언트를 초기화합니다.\n",
        "# loss.backward(): 손실에 대한 역전파를 수행하여 그래디언트를 계산합니다.\n",
        "# optimizer.step(): 그래디언트를 사용하여 모델의 파라미터를 업데이트합니다.\n",
        "# loss_tmp += loss.data, epoch_loss += loss.data: 현재 미니배치의 손실 값을 누적합니다.\n",
        "# if i % 5000 == 4999:: 일정한 주기로 손실 값을 출력합니다. 여기서는 매 5000번째 미니배치마다 출력합니다.\n",
        "# scheduler.step(): 학습률 스케줄러를 통해 학습률을 업데이트합니다.\n",
        "# print('[Epoch - %d] Loss: %.3f' %(epoch + 1, epoch_loss / (i+1))): 에포크마다 손실 값을 출력합니다.\n",
        "# print('Finished Training'): 학습이 완료되었음을 출력합니다.\n",
        "\n",
        "\n",
        "# Test the trained model with overall test dataset\n",
        "\n",
        "correct = 0\n",
        "total = 0\n",
        "for data in testloader:\n",
        "    # Load the data\n",
        "    inputs_test, labels_test = data\n",
        "    inputs_test = inputs_test.to(device)\n",
        "    labels_test = labels_test.to(device)\n",
        "\n",
        "    # Estimate the output using the trained network\n",
        "    outputs_test = net(inputs_test)\n",
        "    _, predicted = torch.max(outputs_test.data, 1)\n",
        "    \n",
        "    # Calculate the accuracy\n",
        "    total += labels_test.size(0)\n",
        "    correct += (predicted == labels_test).sum()\n",
        "\n",
        "# Final accuracy\n",
        "print('Accuracy of the network on the 10,000 test images: %d %%' % (100 * correct / total))\n",
        "\n",
        "\n",
        "## [SimpleNet / Training 5 epochs] Accuracy of the network on the 10,000 test images: 9 %\n",
        "## [VGG11 / Training 5 epochs] Accuracy of the network on the 10,000 test images: 11 %\n",
        "\n",
        "## [VGG11 / Training 5 epochs] Accuracy of the network on the 10,000 test images:  68%\n",
        "## [VGG11 / Training 10 epochs] Accuracy of the network on the 10,000 test images: 79 %\n",
        "\n",
        "## schedular 변경, 68%\n",
        "# epoch 50 2시간 이상 소요.... 89%\n",
        "# epoch 10 30분 소요 ...83%\n",
        "# 데마개 no initialize weight epoch 10 ...77%\n",
        "# 데마개 with initialize wiehgt epoch 10 ...80%\n",
        "# 데마개 with initialize wiehgt epoch 30 ... 1시간 30분 소요 예상하나.. 오늘은 여기까지 \n",
        "\n",
        "# 다시 이전 모델로 돌아옴\n",
        "# epoch 5... 20분 소요 ... 78%\n",
        "# epoch 5, def forward(self, x):    out = F.normalize(x, dim=0) ... 77%\n",
        "# epoch 10, def forward(self, x):    out = F.normalize(x, dim=0) ... 83%\n",
        "# epoch 30 def forward(self, x):    out = F.normalize(x, dim=0) ... 88%\n",
        "\n",
        "# batch normalization이나 dropout, 기타 l1norm, l2norm 의 regularization\n",
        "# epoch 10   l2. weight_deacy=0.9 ... 10% (왜지\n",
        "\n",
        "# epoch 10 weight decay 없애고 data augmentation 수정.. 80%\n",
        "# epoch 30 ... data augmentation 수정 ... 81% ( 과도한 augmentation도 accuracy를 떨어뜨릴 수 있구나)\n",
        "# epoch 10 data augmentation에서 hue가 바뀌었을 때 accuracy가 떨어지는 것 같음 ... 76%\n",
        "\n",
        "\n",
        "# 6/4일 epoch 50 ... 89%\n",
        "\n",
        "# 특이점 : 앞서 Color Jitter (0.2, 0.2, 0.2, 0.1)로 변경, \n",
        "# RandomHorizontalFlip()해서 최종 결과 76% 나왔음에도 불구하고 (10epoch 기준, 변환 이전엔 77%) Predicted에서 cat, ship, ship, plane으로 라벨을 모두 맞춤. \n",
        "# 그러나 현 epoch50을 돌린 모델은 89%가 나옴에도 불구하고 cat, ship, car, plane으로 라벨을 제대로 맞추지 못하는 모습을 보임.\n",
        "\n",
        "# 주요 변환 : \n",
        "# DataLoader → Augmentation\n",
        "# DataLoader - batch size 4 변경 / https://inhovation97.tistory.com/32 : 참고 글. batch size와 lr의 상관관계. (lr과 batch size가 작도록 설정함)\n",
        "# Your own CNN - 인터넷에서 기존 모델 사용 / Your own CNN - 인터넷에서 기존 모델 사용\n",
        "\n",
        "# Loss function and optimizer - Adam 그대로 사용\n",
        "#   learning rate = 0.0001\n",
        "#   step size : 5\n",
        "#   gamma = 0.9"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%reload_ext tensorboard\n",
        "%tensorboard --logdir=/content/logs"
      ],
      "metadata": {
        "id": "LqBmJd4-3tVS"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}