9.0.Logging-Make-Inference-Pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Module 9.0] Inferencde Pipeline 생성 로그 확인"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이 노트북에서는 아래의 내용을 진행을 하고 추론에 대한 로그를 남기어서, Inference Pipeline Model 이 어떻게 작동을 하는지 알아 봅니다.\n",
    "    - Feature Transfomer(전처리 학습 모델) 생성\n",
    "    - Train 데이타를 Feature Transfomer를 통해서 전처리 데이타 생성\n",
    "    - Validation 데이타를 Feature Transfomer를 통해서 전처리 데이타 생성\n",
    "    - XGBoost를 학습\n",
    "    - Inference Pipeline Model 생성 (전처리, XGboost, 휴처리 모델)\n",
    "    - Realtime Endpoint 생성\n",
    "    - 한개의 테스트 데이터 추론\n",
    "- 소요 시간은 약 10분 걸립니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Transformer (전처리 학습 모델) - log-preprocessing.py 파일\n",
    "- Numerical 데이타는 <a href=https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html>StandardScaler</a>를 사용하여 Normalization을 함. \n",
    "    * z = (x - u) / s \n",
    "        * (z: 표준화된 값. 이 값을 학습시에 사용, x: 각 테이타의 값, u: 데이타 항목의 평균, s: 데이타 항목의 표준편차)\n",
    "- 아래 Account Length, ..CustServ Calls까지 모두 위의 방법으로 전처리 함.\n",
    "    - 아래 imputer는 결측값이 있을 경우에 해당 컬럼의 median 값을 사용 함.\n",
    "\n",
    "```python\n",
    "    numeric_features = list([\n",
    "    'Account Length',\n",
    "    'VMail Message',\n",
    "    'Day Mins',\n",
    "    'Day Calls',\n",
    "    'Eve Mins',\n",
    "    'Eve Calls',\n",
    "    'Night Mins',\n",
    "    'Night Calls',\n",
    "    'Intl Mins',\n",
    "    'Intl Calls',\n",
    "    'CustServ Calls'])\n",
    "\n",
    "    numeric_transformer = Pipeline(steps=[\n",
    "        ('imputer', SimpleImputer(strategy='median')),\n",
    "        ('scaler', StandardScaler())])\n",
    "```\n",
    "- Categorical 데이타는 One Hot Encoding 방식으로 전처리 함. (예, 남자:0, 여자:1 일 경우에 남자:(1,0), 여자:(0,1) 방식으로 처리)\n",
    "    - State, Area Code, Int'l Plan, VMail Plan을 적용 함\n",
    "```python\n",
    "    categorical_features = ['State','Area Code',\"Int'l Plan\",'VMail Plan']\n",
    "    categorical_transformer = Pipeline(steps=[\n",
    "        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n",
    "        ('onehot', OneHotEncoder(handle_unknown='ignore'))])\n",
    "```\n",
    "- 최종적으로 Numerical and Categorical Transformer를 합쳐서 Transformer 생성하고, 학습하여 Transformer의 모델을 S3에 업로드 함.\n",
    "```python\n",
    "    preprocessor = ColumnTransformer(\n",
    "        transformers=[\n",
    "            ('num', numeric_transformer, numeric_features),\n",
    "            ('cat', categorical_transformer, categorical_features)],\n",
    "        remainder=\"drop\")\n",
    "\n",
    "    preprocessor.fit(concat_data)\n",
    "\n",
    "    joblib.dump(preprocessor, os.path.join(args.model_dir, \"model.joblib\"))\n",
    "```\n",
    "- Phone 데이타 항목은 위의 전처리 항목에서 제외 함. 유저별로 고유한 번호이기에 피쳐로서 의미가 없을 것으로 보임"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "from time import strftime, gmtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Transformer (전처리 학습 모델) 생성\n",
    "아래는 다음과 같은 작업을 합니다.\n",
    "- SKLearn 이라는 Estimator를 생성 합니다. \n",
    "    - s3_input_train의 학습 데이타를 SKLearn 입력으로 제공 합니다.\n",
    "    - \"전처리 학습 모델 (Featurizer)\" 을 생성할 수 있는 소스 코드 preprocessing.py 를 지정 합니다. \n",
    "    - 사용할 리소스로 instance_type = 'local' 를 지정 합니다. (이미 노트북 인스턴스에 설치된 Docker-compose를 이용 합니다.)\n",
    "        - **Local 이 아니라 SageMaker Cloud Instance도 사용 가능 합니다. (예: ml.m4.xlarge)**\n",
    "        - **아래 XGBoost 학습 알고리즘을 사용시에는 SageMaker Cloud Instance 사용함**\n",
    "- SKLearn의 \"전처리 학습 모델\"이 완료가 되면 결과인 모델 아티펙트 파일이 (model.tar.gz)  s3://{bucket_name}/{job_name}/output.tar.gz 에 저장 됩니다. \n",
    "    - (예: s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-07-15-08-39-41-035/model.tar.gz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 아래는 약 1분 정도가 소요 됩니다. 아래 셀의 [*] 의 표시가 [숫자] (에: [3])로 바뀔 때까지 기다려 주세요"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "This is not the latest supported version. If you would like to use version 0.23-1, please add framework_version=0.23-1 to your constructor.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating tmpelg3l0ow_algo-1-u3hcr_1 ... \n",
      "\u001b[1BAttaching to tmpelg3l0ow_algo-1-u3hcr_12mdone\u001b[0m\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,881 sagemaker-containers INFO     Imported framework sagemaker_sklearn_container.training\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,884 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,892 sagemaker_sklearn_container.training INFO     Invoking user training script.\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,991 sagemaker-containers INFO     Module log-preprocessing does not provide a setup.py. \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Generating setup.py\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,991 sagemaker-containers INFO     Generating setup.cfg\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,992 sagemaker-containers INFO     Generating MANIFEST.in\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:56,992 sagemaker-containers INFO     Installing module with the following command:\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m /miniconda3/bin/python -m pip install . \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Processing /opt/ml/code\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Building wheels for collected packages: log-preprocessing\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m   Building wheel for log-preprocessing (setup.py) ... \u001b[?25ldone\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \u001b[?25h  Created wheel for log-preprocessing: filename=log_preprocessing-1.0.0-py2.py3-none-any.whl size=10100 sha256=3849313afe3e748ebc734b20062e9f92a519e5164b80edd1221e86c62b1d508e\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m   Stored in directory: /tmp/pip-ephem-wheel-cache-jsnq1gyr/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Successfully built log-preprocessing\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Installing collected packages: log-preprocessing\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Successfully installed log-preprocessing-1.0.0\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:57,918 sagemaker-containers INFO     No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:57,927 sagemaker-containers INFO     Invoking user script\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Training Env:\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m {\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"additional_framework_parameters\": {},\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"channel_input_dirs\": {\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         \"train\": \"/opt/ml/input/data/train\"\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"current_host\": \"algo-1-u3hcr\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"framework_module\": \"sagemaker_sklearn_container.training:main\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"hosts\": [\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         \"algo-1-u3hcr\"\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     ],\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"hyperparameters\": {},\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"input_data_config\": {\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         \"train\": {\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m             \"TrainingInputMode\": \"File\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m             \"ContentType\": \"csv\"\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         }\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"input_dir\": \"/opt/ml/input\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"is_master\": true,\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"job_name\": \"sagemaker-scikit-learn-2020-08-09-12-13-54-868\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"log_level\": 20,\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"master_hostname\": \"algo-1-u3hcr\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"model_dir\": \"/opt/ml/model\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"module_dir\": \"s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-08-09-12-13-54-868/source/sourcedir.tar.gz\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"module_name\": \"log-preprocessing\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"network_interface_name\": \"eth0\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"num_cpus\": 16,\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"num_gpus\": 0,\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"output_dir\": \"/opt/ml/output\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"resource_config\": {\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         \"current_host\": \"algo-1-u3hcr\",\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         \"hosts\": [\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m             \"algo-1-u3hcr\"\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m         ]\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     },\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m     \"user_entry_point\": \"log-preprocessing.py\"\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m }\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Environment variables:\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_HOSTS=[\"algo-1-u3hcr\"]\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_NETWORK_INTERFACE_NAME=eth0\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_HPS={}\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_USER_ENTRY_POINT=log-preprocessing.py\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_FRAMEWORK_PARAMS={}\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_RESOURCE_CONFIG={\"current_host\":\"algo-1-u3hcr\",\"hosts\":[\"algo-1-u3hcr\"]}\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_INPUT_DATA_CONFIG={\"train\":{\"ContentType\":\"csv\",\"TrainingInputMode\":\"File\"}}\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_OUTPUT_DATA_DIR=/opt/ml/output/data\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_CHANNELS=[\"train\"]\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_CURRENT_HOST=algo-1-u3hcr\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_MODULE_NAME=log-preprocessing\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_LOG_LEVEL=20\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_FRAMEWORK_MODULE=sagemaker_sklearn_container.training:main\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_INPUT_DIR=/opt/ml/input\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_INPUT_CONFIG_DIR=/opt/ml/input/config\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_OUTPUT_DIR=/opt/ml/output\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_NUM_CPUS=16\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_NUM_GPUS=0\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_MODEL_DIR=/opt/ml/model\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_MODULE_DIR=s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-08-09-12-13-54-868/source/sourcedir.tar.gz\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"train\":\"/opt/ml/input/data/train\"},\"current_host\":\"algo-1-u3hcr\",\"framework_module\":\"sagemaker_sklearn_container.training:main\",\"hosts\":[\"algo-1-u3hcr\"],\"hyperparameters\":{},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"train\":{\"ContentType\":\"csv\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"sagemaker-scikit-learn-2020-08-09-12-13-54-868\",\"log_level\":20,\"master_hostname\":\"algo-1-u3hcr\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-08-09-12-13-54-868/source/sourcedir.tar.gz\",\"module_name\":\"log-preprocessing\",\"network_interface_name\":\"eth0\",\"num_cpus\":16,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1-u3hcr\",\"hosts\":[\"algo-1-u3hcr\"]},\"user_entry_point\":\"log-preprocessing.py\"}\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_USER_ARGS=[]\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m SM_CHANNEL_TRAIN=/opt/ml/input/data/train\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m PYTHONPATH=/miniconda3/bin:/miniconda3/lib/python37.zip:/miniconda3/lib/python3.7:/miniconda3/lib/python3.7/lib-dynload:/miniconda3/lib/python3.7/site-packages\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m Invoking script with the following command:\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m /miniconda3/bin/python -m log-preprocessing\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m \n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m /miniconda3/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m   import imp\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m saved model!\n",
      "\u001b[36malgo-1-u3hcr_1  |\u001b[0m 2020-08-09 12:13:58,785 sagemaker-containers INFO     Reporting training SUCCESS\n",
      "\u001b[36mtmpelg3l0ow_algo-1-u3hcr_1 exited with code 0\n",
      "\u001b[0mAborting on container exit...\n",
      "===== Job Complete =====\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.sklearn.estimator import SKLearn\n",
    "sagemaker_session = sagemaker.Session()\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "script_path = 'log-preprocessing.py'\n",
    "# instance_type = 'ml.m4.2xlarge'\n",
    "instance_type = 'local'\n",
    "\n",
    "sklearn_preprocessor = SKLearn(\n",
    "    entry_point=script_path,\n",
    "    role=role,\n",
    "    train_instance_type = instance_type\n",
    ")\n",
    "sklearn_preprocessor.fit({'train': s3_input_train})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Transfomer를 사용하여 전처리된 학습 및 검증 데이타 생성 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Transformer_Train](img/Fig2.1.transformer_train.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessed Training data (Feature) 만들기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 아래는 약 1분 정도가 소요 됩니다. 아래 셀의 [*] 의 표시가 [숫자] (에: [4])로 바뀔 때까지 기다려 주세요"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attaching to tmpmxj0vrhw_algo-1-tf4z8_1\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m Processing /opt/ml/code\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m Building wheels for collected packages: log-preprocessing\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m   Building wheel for log-preprocessing (setup.py) ... \u001b[?25ldone\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m \u001b[?25h  Created wheel for log-preprocessing: filename=log_preprocessing-1.0.0-py2.py3-none-any.whl size=10099 sha256=afc0a7b2d6822f5205928ded481ce9e1872bddfcfb03c24520b1523cde3b55ad\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m   Stored in directory: /tmp/pip-ephem-wheel-cache-xn9m4_pn/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m Successfully built log-preprocessing\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m Installing collected packages: log-preprocessing\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m Successfully installed log-preprocessing-1.0.0\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m /miniconda3/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m   import imp\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [72] [INFO] Starting gunicorn 19.9.0\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [72] [INFO] Listening at: unix:/tmp/gunicorn.sock (72)\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [72] [INFO] Using worker: gevent\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [75] [INFO] Booting worker with pid: 75\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [91] [INFO] Booting worker with pid: 91\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [92] [INFO] Booting worker with pid: 92\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [108] [INFO] Booting worker with pid: 108\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [109] [INFO] Booting worker with pid: 109\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:02 +0000] [155] [INFO] Booting worker with pid: 155\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [156] [INFO] Booting worker with pid: 156\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [157] [INFO] Booting worker with pid: 157\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [188] [INFO] Booting worker with pid: 188\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [204] [INFO] Booting worker with pid: 204\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [205] [INFO] Booting worker with pid: 205\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [222] [INFO] Booting worker with pid: 222\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [253] [INFO] Booting worker with pid: 253\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [254] [INFO] Booting worker with pid: 254\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [255] [INFO] Booting worker with pid: 255\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m [2020-08-09 12:14:03 +0000] [271] [INFO] Booting worker with pid: 271\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 2020-08-09 12:14:04,541 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m /miniconda3/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m   import imp\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 2020-08-09 12:14:04,869 INFO - root - model_fn, Mode: feature_transform\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 172.18.0.1 - - [09/Aug/2020:12:14:04 +0000] \"GET /ping HTTP/1.1\" 200 0 \"-\" \"-\"\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 172.18.0.1 - - [09/Aug/2020:12:14:04 +0000] \"GET /execution-parameters HTTP/1.1\" 404 232 \"-\" \"-\"\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 2020-08-09 12:14:04,992 INFO - root - Input_fn, Mode: feature_transform\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 2020-08-09 12:14:05,001 INFO - root - predict_fn, Mode: feature_transform\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 2020-08-09 12:14:05,011 INFO - root - Output_fn: prdiction - '[[ 0.          0.1194137  -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m  [ 0.         -1.85265164 -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m  [ 1.          1.18129503 -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m  ...\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m  [ 0.          0.70092014 -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m  [ 0.          1.45940681 -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m  [ 0.         -0.05756653  1.79907451 ...  0.          0.\n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m    1.        ]]' \n",
      "\u001b[36malgo-1-tf4z8_1  |\u001b[0m 172.18.0.1 - - [09/Aug/2020:12:14:05 +0000] \"POST /invocations HTTP/1.1\" 200 1054525 \"-\" \"-\"\n",
      "Gracefully stopping... (press Ctrl+C again to force)\n",
      "Waiting for transform job: sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-13-59-398\n",
      ".s3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/transformtrain-train-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-13-59-398\n"
     ]
    }
   ],
   "source": [
    "# 아웃풋 경로 지정\n",
    "transform_train_output_path = 's3://{}/{}/{}/'.format(bucket, prefix, 'transformtrain-train-output')\n",
    "instance_type = 'local'\n",
    "# instance_type = 'ml.m4.2xlarge'\n",
    "\n",
    "# scikit_learn_inferencee_model 이름으로 전처리 학습 모델 생성\n",
    "# TRANSFORM_MODE의 환경 변수는 전처리 모드라는 것을 알려 줌.\n",
    "    # 추론시에는 환경 변수를 TRANSFORM_MODE\": \"inverse-label-transform\" 설정 함.\n",
    "    # 위의 두개의 과정을 분리할 수 있으나, 한개의 소스를 (preprocessor.py)를 사용하기 위해서, 환경 변수를 통해서 구분함.\n",
    "scikit_learn_inferencee_model = sklearn_preprocessor.create_model(\n",
    "    env={'TRANSFORM_MODE': 'feature-transform'})\n",
    "# scikit_learn_inferencee_model 에서 Train Transformer 생성\n",
    "transformer_train = scikit_learn_inferencee_model.transformer(\n",
    "    instance_count=1, \n",
    "    instance_type= instance_type,\n",
    "    assemble_with = 'Line',\n",
    "    output_path = transform_train_output_path,\n",
    "    accept = 'text/csv')\n",
    "\n",
    "\n",
    "# Preprocess training input\n",
    "transformer_train.transform(s3_input_train.config['DataSource']['S3DataSource']['S3Uri'], \n",
    "                            content_type='text/csv')\n",
    "print('Waiting for transform job: ' + transformer_train.latest_transform_job.job_name)\n",
    "transformer_train.wait()\n",
    "preprocessed_train_path = transformer_train.output_path + transformer_train.latest_transform_job.job_name\n",
    "print(preprocessed_train_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training 전처리된 학습 파일 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/transformtrain-train-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-13-59-398\n"
     ]
    }
   ],
   "source": [
    "print(preprocessed_train_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-09 12:14:06    1054526 sagemaker/customer-churn/transformtrain-train-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-13-59-398/train.csv.out\n"
     ]
    }
   ],
   "source": [
    "! aws s3 ls {preprocessed_train_path} --recursive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0.0</th>\n",
       "      <th>0.11941369588439606</th>\n",
       "      <th>-0.5962380254245051</th>\n",
       "      <th>1.744368057672484</th>\n",
       "      <th>0.9789570533336895</th>\n",
       "      <th>-0.028992907038264654</th>\n",
       "      <th>-0.8931854019845896</th>\n",
       "      <th>-0.8017032037830547</th>\n",
       "      <th>-1.9825286353116254</th>\n",
       "      <th>-1.5305589315744583</th>\n",
       "      <th>...</th>\n",
       "      <th>0.0.48</th>\n",
       "      <th>0.0.49</th>\n",
       "      <th>0.0.50</th>\n",
       "      <th>0.0.51</th>\n",
       "      <th>0.0.52</th>\n",
       "      <th>0.0.53</th>\n",
       "      <th>1.0.1</th>\n",
       "      <th>0.0.54</th>\n",
       "      <th>1.0.2</th>\n",
       "      <th>0.0.55</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.852652</td>\n",
       "      <td>-0.596238</td>\n",
       "      <td>0.140284</td>\n",
       "      <td>-0.310405</td>\n",
       "      <td>0.970689</td>\n",
       "      <td>-0.689888</td>\n",
       "      <td>0.146389</td>\n",
       "      <td>1.232901</td>\n",
       "      <td>0.124852</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.181295</td>\n",
       "      <td>-0.596238</td>\n",
       "      <td>1.835130</td>\n",
       "      <td>0.185503</td>\n",
       "      <td>0.030988</td>\n",
       "      <td>-0.639063</td>\n",
       "      <td>1.568529</td>\n",
       "      <td>-0.063643</td>\n",
       "      <td>-0.846802</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.776769</td>\n",
       "      <td>-0.596238</td>\n",
       "      <td>0.216227</td>\n",
       "      <td>0.334276</td>\n",
       "      <td>0.136954</td>\n",
       "      <td>1.393914</td>\n",
       "      <td>1.394712</td>\n",
       "      <td>-0.634123</td>\n",
       "      <td>0.844596</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.234547</td>\n",
       "      <td>1.508734</td>\n",
       "      <td>-0.459859</td>\n",
       "      <td>0.483049</td>\n",
       "      <td>-0.230929</td>\n",
       "      <td>0.224952</td>\n",
       "      <td>1.056954</td>\n",
       "      <td>0.921730</td>\n",
       "      <td>-0.810815</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.751486</td>\n",
       "      <td>1.218393</td>\n",
       "      <td>0.231046</td>\n",
       "      <td>-0.756723</td>\n",
       "      <td>0.516833</td>\n",
       "      <td>0.275776</td>\n",
       "      <td>1.043127</td>\n",
       "      <td>-2.138114</td>\n",
       "      <td>0.232814</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 70 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0.0  0.11941369588439606  -0.5962380254245051  1.744368057672484  \\\n",
       "0  0.0            -1.852652            -0.596238           0.140284   \n",
       "1  1.0             1.181295            -0.596238           1.835130   \n",
       "2  0.0             0.776769            -0.596238           0.216227   \n",
       "3  0.0            -0.234547             1.508734          -0.459859   \n",
       "4  0.0             0.751486             1.218393           0.231046   \n",
       "\n",
       "   0.9789570533336895  -0.028992907038264654  -0.8931854019845896  \\\n",
       "0           -0.310405               0.970689            -0.689888   \n",
       "1            0.185503               0.030988            -0.639063   \n",
       "2            0.334276               0.136954             1.393914   \n",
       "3            0.483049              -0.230929             0.224952   \n",
       "4           -0.756723               0.516833             0.275776   \n",
       "\n",
       "   -0.8017032037830547  -1.9825286353116254  -1.5305589315744583  ...  0.0.48  \\\n",
       "0             0.146389             1.232901             0.124852  ...     0.0   \n",
       "1             1.568529            -0.063643            -0.846802  ...     0.0   \n",
       "2             1.394712            -0.634123             0.844596  ...     0.0   \n",
       "3             1.056954             0.921730            -0.810815  ...     0.0   \n",
       "4             1.043127            -2.138114             0.232814  ...     0.0   \n",
       "\n",
       "   0.0.49  0.0.50  0.0.51  0.0.52  0.0.53  1.0.1  0.0.54  1.0.2  0.0.55  \n",
       "0     0.0     1.0     0.0     0.0     0.0    1.0     0.0    1.0     0.0  \n",
       "1     0.0     0.0     0.0     0.0     0.0    1.0     0.0    1.0     0.0  \n",
       "2     0.0     0.0     0.0     0.0     0.0    1.0     0.0    1.0     0.0  \n",
       "3     0.0     0.0     0.0     0.0     0.0    1.0     0.0    0.0     1.0  \n",
       "4     0.0     0.0     0.0     0.0     0.0    1.0     0.0    0.0     1.0  \n",
       "\n",
       "[5 rows x 70 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_train_path_file = os.path.join (preprocessed_train_path, 'train.csv.out')\n",
    "df_pre_train = pd.read_csv(preprocessed_train_path_file)\n",
    "df_pre_train.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessed Validation data (Feature) 만들기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 아래는 약 1분 정도가 소요 됩니다. 아래 셀의 [*] 의 표시가 [숫자] (에: [8])로 바뀔 때까지 기다려 주세요"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attaching to tmp_vvrbz37_algo-1-f7xrm_1\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m Processing /opt/ml/code\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m Building wheels for collected packages: log-preprocessing\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m   Building wheel for log-preprocessing (setup.py) ... \u001b[?25ldone\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m \u001b[?25h  Created wheel for log-preprocessing: filename=log_preprocessing-1.0.0-py2.py3-none-any.whl size=10099 sha256=9cac433c353a8d53b88615fde77217dec60bf3ae969cdd11d661aae5049d7d82\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m   Stored in directory: /tmp/pip-ephem-wheel-cache-a0agijrc/wheels/35/24/16/37574d11bf9bde50616c67372a334f94fa8356bc7164af8ca3\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m Successfully built log-preprocessing\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m Installing collected packages: log-preprocessing\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m Successfully installed log-preprocessing-1.0.0\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m /miniconda3/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m   import imp\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [73] [INFO] Starting gunicorn 19.9.0\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [73] [INFO] Listening at: unix:/tmp/gunicorn.sock (73)\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [73] [INFO] Using worker: gevent\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [76] [INFO] Booting worker with pid: 76\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [77] [INFO] Booting worker with pid: 77\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [108] [INFO] Booting worker with pid: 108\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [109] [INFO] Booting worker with pid: 109\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [110] [INFO] Booting worker with pid: 110\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [111] [INFO] Booting worker with pid: 111\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [142] [INFO] Booting worker with pid: 142\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [158] [INFO] Booting worker with pid: 158\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [159] [INFO] Booting worker with pid: 159\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [175] [INFO] Booting worker with pid: 175\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [206] [INFO] Booting worker with pid: 206\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [222] [INFO] Booting worker with pid: 222\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [243] [INFO] Booting worker with pid: 243\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [271] [INFO] Booting worker with pid: 271\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [272] [INFO] Booting worker with pid: 272\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m [2020-08-09 12:14:10 +0000] [273] [INFO] Booting worker with pid: 273\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 2020-08-09 12:14:11,934 INFO - sagemaker-containers - No GPUs detected (normal if no gpus installed)\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m /miniconda3/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m   import imp\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 2020-08-09 12:14:12,274 INFO - root - model_fn, Mode: feature_transform\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 172.18.0.1 - - [09/Aug/2020:12:14:12 +0000] \"GET /ping HTTP/1.1\" 200 0 \"-\" \"-\"\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 172.18.0.1 - - [09/Aug/2020:12:14:12 +0000] \"GET /execution-parameters HTTP/1.1\" 404 232 \"-\" \"-\"\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 2020-08-09 12:14:12,401 INFO - root - Input_fn, Mode: feature_transform\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 2020-08-09 12:14:12,406 INFO - root - predict_fn, Mode: feature_transform\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 2020-08-09 12:14:12,413 INFO - root - Output_fn: prdiction - '[[ 0.         -1.37227675  1.43614836 ...  0.          0.\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m    1.        ]\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m  [ 0.         -1.80208586 -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m  [ 0.          0.1194137   1.72648928 ...  0.          0.\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m    1.        ]\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m  ...\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m  [ 0.          0.27111103  1.58131882 ...  0.          0.\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m    1.        ]\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m  [ 0.         -0.7907703  -0.59623803 ...  0.          1.\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m    0.        ]\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m  [ 0.         -0.51265853  2.74268247 ...  0.          0.\n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m    1.        ]]' \n",
      "\u001b[36malgo-1-f7xrm_1  |\u001b[0m 172.18.0.1 - - [09/Aug/2020:12:14:12 +0000] \"POST /invocations HTTP/1.1\" 200 300974 \"-\" \"-\"\n",
      "Gracefully stopping... (press Ctrl+C again to force)\n",
      "Waiting for transform job: sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-14-06-784\n",
      ".s3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/transformtrain-validation-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-14-06-784\n"
     ]
    }
   ],
   "source": [
    "# 아웃풋 경로 지정\n",
    "transform_validation_output_path = 's3://{}/{}/{}/'.format(bucket, prefix, 'transformtrain-validation-output')\n",
    "# scikit_learn_inferencee_model 에서 Validation Transformer 생성\n",
    "transformer_validation = scikit_learn_inferencee_model.transformer(\n",
    "    instance_count=1, \n",
    "    instance_type= instance_type,\n",
    "    assemble_with = 'Line',\n",
    "    output_path = transform_validation_output_path,\n",
    "    accept = 'text/csv')\n",
    "# Preprocess validation input\n",
    "transformer_validation.transform(s3_input_validation.config['DataSource']['S3DataSource']['S3Uri'], content_type='text/csv')\n",
    "print('Waiting for transform job: ' + transformer_validation.latest_transform_job.job_name)\n",
    "transformer_validation.wait()\n",
    "preprocessed_validation_path = transformer_validation.output_path+transformer_validation.latest_transform_job.job_name\n",
    "print(preprocessed_validation_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/transformtrain-validation-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-14-06-784'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessed_validation_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-09 12:14:13     300975 sagemaker/customer-churn/transformtrain-validation-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-14-06-784/validation.csv.out\n"
     ]
    }
   ],
   "source": [
    "! aws s3 ls {preprocessed_validation_path} --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Train with XGBoost on SageMaker Cloud Instance\n",
    "아 과정은 위의 전처리된 데이타를 가지고 실제 SageMaker Built-in Algorithm XGBoost를 이용하여 학습을 수행 합니다.<br>\n",
    "실제의 학습 과정은 SageMaker Cloud Instance에서 실제 학습니 됩니다.\n",
    "\n",
    "\n",
    "Built-in XGboost 알고리즘 컨테이너를 가져옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "container = get_image_uri(boto3.Session().region_name, 'xgboost', '1.0-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "S3에 있는 Train, Validation 전처리된 (Features) 데이타의 경로 및 파일 포맷등을 지정하는 오브젝트를 생성 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S3 Train input: \n",
      "\n",
      "{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/transformtrain-train-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-13-59-398', 'S3DataDistributionType': 'FullyReplicated'}}, 'ContentType': 'text/csv'}\n",
      "\n",
      "S3 Validation input: \n",
      "\n",
      "{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix', 'S3Uri': 's3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/transformtrain-validation-output/sagemaker-scikit-learn-2020-08-09-12-13-2020-08-09-12-14-06-784', 'S3DataDistributionType': 'FullyReplicated'}}, 'ContentType': 'text/csv'}\n"
     ]
    }
   ],
   "source": [
    "s3_input_train_processed = sagemaker.session.s3_input(\n",
    "    preprocessed_train_path, \n",
    "    distribution='FullyReplicated',\n",
    "    content_type='text/csv', \n",
    "    s3_data_type='S3Prefix')\n",
    "print(\"S3 Train input: \\n\")\n",
    "print(s3_input_train_processed.config)\n",
    "s3_input_validation_processed = sagemaker.session.s3_input(\n",
    "    preprocessed_validation_path, \n",
    "    distribution='FullyReplicated',\n",
    "    content_type='text/csv', \n",
    "    s3_data_type='S3Prefix')\n",
    "print(\"\\nS3 Validation input: \\n\")\n",
    "print(s3_input_validation_processed.config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 아래는 약 5분 정도가 소요 됩니다. 아래 셀의 [*] 의 표시가 [숫자] (에: [13])로 바뀔 때까지 기다려 주세요"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-09 12:14:14 Starting - Starting the training job...\n",
      "2020-08-09 12:14:16 Starting - Launching requested ML instances.........\n",
      "2020-08-09 12:15:46 Starting - Preparing the instances for training...\n",
      "2020-08-09 12:16:34 Downloading - Downloading input data...\n",
      "2020-08-09 12:16:58 Training - Downloading the training image..\u001b[34mINFO:sagemaker-containers:Imported framework sagemaker_xgboost_container.training\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:Failed to parse hyperparameter objective value binary:logistic to Json.\u001b[0m\n",
      "\u001b[34mReturning the value itself\u001b[0m\n",
      "\u001b[34mINFO:sagemaker-containers:No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34mINFO:sagemaker_xgboost_container.training:Running XGBoost Sagemaker in algorithm mode\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[12:17:20] 2333x69 matrix with 160977 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[12:17:20] 666x69 matrix with 45954 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34mINFO:root:Single node training.\u001b[0m\n",
      "\u001b[34mINFO:root:Train matrix has 2333 rows\u001b[0m\n",
      "\u001b[34mINFO:root:Validation matrix has 666 rows\u001b[0m\n",
      "\u001b[34m[12:17:20] WARNING: /workspace/src/learner.cc:328: \u001b[0m\n",
      "\u001b[34mParameters: { num_round, silent } might not be used.\n",
      "\n",
      "  This may not be accurate due to some parameters are only used in language bindings but\n",
      "  passed down to XGBoost core.  Or some parameters are not used but slip through this\n",
      "  verification. Please open an issue if you find above cases.\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[34m[0]#011train-error:0.07715#011validation-error:0.09910\u001b[0m\n",
      "\u001b[34m[1]#011train-error:0.05058#011validation-error:0.08108\u001b[0m\n",
      "\u001b[34m[2]#011train-error:0.04886#011validation-error:0.07508\u001b[0m\n",
      "\u001b[34m[3]#011train-error:0.04672#011validation-error:0.07207\u001b[0m\n",
      "\u001b[34m[4]#011train-error:0.04801#011validation-error:0.07357\u001b[0m\n",
      "\u001b[34m[5]#011train-error:0.04672#011validation-error:0.07057\u001b[0m\n",
      "\u001b[34m[6]#011train-error:0.04544#011validation-error:0.07357\u001b[0m\n",
      "\u001b[34m[7]#011train-error:0.04372#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[8]#011train-error:0.04501#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[9]#011train-error:0.04244#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[10]#011train-error:0.04029#011validation-error:0.06456\u001b[0m\n",
      "\u001b[34m[11]#011train-error:0.03901#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[12]#011train-error:0.03858#011validation-error:0.06456\u001b[0m\n",
      "\u001b[34m[13]#011train-error:0.03772#011validation-error:0.06456\u001b[0m\n",
      "\u001b[34m[14]#011train-error:0.03772#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[15]#011train-error:0.03943#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[16]#011train-error:0.03858#011validation-error:0.06306\u001b[0m\n",
      "\u001b[34m[17]#011train-error:0.03772#011validation-error:0.06456\u001b[0m\n",
      "\u001b[34m[18]#011train-error:0.03943#011validation-error:0.06306\u001b[0m\n",
      "\u001b[34m[19]#011train-error:0.03986#011validation-error:0.06006\u001b[0m\n",
      "\u001b[34m[20]#011train-error:0.03943#011validation-error:0.06306\u001b[0m\n",
      "\u001b[34m[21]#011train-error:0.03858#011validation-error:0.06306\u001b[0m\n",
      "\u001b[34m[22]#011train-error:0.03815#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[23]#011train-error:0.03686#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[24]#011train-error:0.03601#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[25]#011train-error:0.03429#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[26]#011train-error:0.03386#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[27]#011train-error:0.03386#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[28]#011train-error:0.03386#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[29]#011train-error:0.03386#011validation-error:0.07207\u001b[0m\n",
      "\u001b[34m[30]#011train-error:0.03386#011validation-error:0.07057\u001b[0m\n",
      "\u001b[34m[31]#011train-error:0.03429#011validation-error:0.07207\u001b[0m\n",
      "\u001b[34m[32]#011train-error:0.03472#011validation-error:0.07057\u001b[0m\n",
      "\u001b[34m[33]#011train-error:0.03472#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[34]#011train-error:0.03300#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[35]#011train-error:0.03386#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[36]#011train-error:0.03386#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[37]#011train-error:0.03300#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[38]#011train-error:0.03343#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[39]#011train-error:0.03300#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[40]#011train-error:0.03172#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[41]#011train-error:0.03172#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[42]#011train-error:0.03086#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[43]#011train-error:0.03129#011validation-error:0.06907\u001b[0m\n",
      "\u001b[34m[44]#011train-error:0.03086#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[45]#011train-error:0.03129#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[46]#011train-error:0.03086#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[47]#011train-error:0.03086#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[48]#011train-error:0.03043#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[49]#011train-error:0.03000#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[50]#011train-error:0.03000#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[51]#011train-error:0.03000#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[52]#011train-error:0.03000#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[53]#011train-error:0.03000#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[54]#011train-error:0.03000#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[55]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[56]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[57]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[58]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[59]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[60]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[61]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[62]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[63]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[64]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[65]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[66]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[67]#011train-error:0.03043#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[68]#011train-error:0.02915#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[69]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[70]#011train-error:0.02915#011validation-error:0.06757\u001b[0m\n",
      "\u001b[34m[71]#011train-error:0.02872#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[72]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[73]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[74]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[75]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[76]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[77]#011train-error:0.02915#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[78]#011train-error:0.02958#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[79]#011train-error:0.03000#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[80]#011train-error:0.02958#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[81]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[82]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[83]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[84]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[85]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[86]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[87]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[88]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[89]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[90]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[91]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[92]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[93]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[94]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[95]#011train-error:0.02786#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[96]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[97]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[98]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\u001b[34m[99]#011train-error:0.02829#011validation-error:0.06607\u001b[0m\n",
      "\n",
      "2020-08-09 12:17:30 Uploading - Uploading generated training model\n",
      "2020-08-09 12:17:30 Completed - Training job completed\n",
      "Training seconds: 56\n",
      "Billable seconds: 56\n"
     ]
    }
   ],
   "source": [
    "sess = sagemaker.Session()\n",
    "instance_type = 'ml.m4.2xlarge'\n",
    "\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(container, # Built-in XGBoost Container\n",
    "                                    role, \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type= instance_type,\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                    sagemaker_session=sess\n",
    "                                   )\n",
    "xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        silent=0,\n",
    "                        objective='binary:logistic',\n",
    "                        num_round=100,\n",
    "                       )\n",
    "\n",
    "\n",
    "xgb.fit({'train': s3_input_train_processed, 'validation': s3_input_validation_processed}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference Pipeline <a class=\"anchor\" id=\"pipeline_setup\"></a>\n",
    "\n",
    "아래 그림과 같이 위에서 생성한 전처리, 알고리즘 학습, 후처리의 세가지 모델을 가지고 1개의 단일 모델을 만들어 Inference Pipleline을 생성 합니다. <br>\n",
    "**입력 데이타 가공이 없이 실제 데이타가 입력이 되면, 1개의 단일 모델을 통해서 최종적으로 예측 결과인 True, False의 결과 값이 제공 됩니다.**\n",
    "\n",
    "![Inference-pipeline](img/Fig2.2.inference_pipeline.png)\n",
    "\n",
    "\n",
    "**Machine Learning Model Pipeline (Inference Pipeline)는 create_model() 를 호출하여 만들 수 있습니다.** <br>\n",
    "예를 들어 여기서는 the fitted Scikit-learn inference model, the fitted Xgboost model and the psotprocessing model 의 세가지 모델을 가지고 만듦니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래는 세개 모델을 생성함. 전처리, 후처리 모델 생성시에는 환경 변수를 제공 함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "# 전처리 모델\n",
    "scikit_learn_pre_process_model = sklearn_preprocessor.create_model(\n",
    "    env={'TRANSFORM_MODE': 'feature-transform'})    \n",
    "# 학습 모델\n",
    "xgb_model = xgb.create_model()\n",
    "# 후처리 모델\n",
    "scikit_learn_post_process_model = sklearn_preprocessor.create_model(\n",
    "    env={'TRANSFORM_MODE': 'inverse-label-transform'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Transformer Model:\n",
      " s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-08-09-12-13-54-868/model.tar.gz\n",
      "\n",
      "XGBoost Model:\n",
      " s3://sagemaker-us-east-2-057716757052/sagemaker/customer-churn/output/sagemaker-xgboost-2020-08-09-12-14-13-917/output/model.tar.gz\n",
      "\n",
      "Post-Processing Model :\n",
      " s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-08-09-12-13-54-868/model.tar.gz\n"
     ]
    }
   ],
   "source": [
    "print(\"Feature Transformer Model:\\n {}\".format(scikit_learn_pre_process_model.model_data))\n",
    "print(\"\\nXGBoost Model:\\n {}\".format(xgb_model.model_data))\n",
    "print(\"\\nPost-Processing Model :\\n {}\".format(scikit_learn_post_process_model.model_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "전처리 모델의 기타 설정 변수 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "entry_point:  log-preprocessing.py\n",
      "framework_version:  0.20.0\n",
      "env:  {'TRANSFORM_MODE': 'feature-transform'}\n",
      "model_data:  s3://sagemaker-us-east-2-057716757052/sagemaker-scikit-learn-2020-08-09-12-13-54-868/model.tar.gz\n",
      "name:  sagemaker-scikit-learn-2020-08-09-12-13-54-868\n"
     ]
    }
   ],
   "source": [
    "print(\"entry_point: \", scikit_learn_pre_process_model.entry_point)\n",
    "print(\"framework_version: \", scikit_learn_pre_process_model.framework_version)\n",
    "print(\"env: \", scikit_learn_pre_process_model.env)\n",
    "print(\"model_data: \", scikit_learn_pre_process_model.model_data)\n",
    "print(\"name: \", scikit_learn_pre_process_model.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "from sagemaker.pipeline import PipelineModel\n",
    "import boto3\n",
    "\n",
    "from time import gmtime, strftime\n",
    "\n",
    "timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "model_name = 'churn-model-inference-pipeline-' + timestamp_prefix\n",
    "\n",
    "\n",
    "sklearn_preprocessor.create_model\n",
    "\n",
    "pipeline_model = PipelineModel(\n",
    "    name = model_name,\n",
    "    role = role,\n",
    "    models = [\n",
    "        scikit_learn_pre_process_model,\n",
    "        xgb_model,\n",
    "        scikit_learn_post_process_model        \n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------!CPU times: user 304 ms, sys: 24.9 ms, total: 329 ms\n",
      "Wall time: 9min 32s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "instance_type='ml.t2.medium'\n",
    "# instance_type='local'\n",
    "endpoint_name= 'churn-model-pipeline-endpoint-' + timestamp_prefix\n",
    "\n",
    "deployed_model = pipeline_model.deploy(\n",
    "    initial_instance_count=1, \n",
    "    instance_type= instance_type, \n",
    "    endpoint_name = endpoint_name,        \n",
    "    wait = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래의 local endpoint는 에러가 발생함. \n",
    "추정되는 이유는 전처리 모델, 후처리 모델은 'local' mode 로 생성이 되었고 <br>\n",
    "xgboost 모델은 Sagemaker Host mode 로 생성이 되었을 겻으로 추정 함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# instance_type='local'\n",
    "\n",
    "# deployed_local_model = pipeline_model.deploy(\n",
    "#     initial_instance_count=1, \n",
    "#     instance_type= instance_type, \n",
    "#     wait = False\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import json_serializer, csv_serializer, json_deserializer, RealTimePredictor\n",
    "from sagemaker.content_types import CONTENT_TYPE_CSV, CONTENT_TYPE_JSON\n",
    "import sagemaker\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "predictor = RealTimePredictor(\n",
    "    endpoint = endpoint_name,\n",
    "    sagemaker_session = sagemaker_session,\n",
    "    serializer = csv_serializer,\n",
    "    content_type = CONTENT_TYPE_CSV,\n",
    "    accept = CONTENT_TYPE_JSON\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def make_inference_format(sample):\n",
    "    instance = str()\n",
    "    for i, token in enumerate(sample):\n",
    "        # print(token)\n",
    "        if i > 0:\n",
    "            instance = instance  + ',' + str(token) \n",
    "        else:\n",
    "            instance = instance  +  str(token) \n",
    "    return instance\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "instance: \n",
      " KS,186,510,400-6454,no,no,0,137.8,97,23.43,187.7,118,15.95,146.4,85,6.59,8.7,6,2.35,1\n",
      "Churn result?: \n",
      " b'False\\n'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_df = pd.read_csv(\"churn_data/batch_transform_test.csv\", header=None)\n",
    "\n",
    "for i in range(10):\n",
    "    sample = test_df.iloc[i,:]\n",
    "    instance = make_inference_format(sample)\n",
    "    print(\"instance: \\n\", instance)\n",
    "\n",
    "    payload = instance\n",
    "    churn_result = predictor.predict(payload)\n",
    "    print(\"Churn result?: \\n\", churn_result)\n",
    "    print(\"\")\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}