zadanie3_s24216.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"7W8u8FZFIL2m"},"source":["# Data Science z Python 3.10. Optymalizator, funkcja straty i early stopping\n","## 🇬🇧 Data Science with Python 3.10. Optimizer, loss function and early stopping\n","#### 👨‍🏫 PhD Wojciech Oronowicz-Jaśkowiak\n","#### 🤖 https://github.com/aipogodzinach"]},{"cell_type":"markdown","metadata":{"id":"Mn_XRemkR6fM"},"source":["Poniżej przedstawiony jest kod znany z poprzednich zajęć. Tym razem wprowadzimy do niego \"ulepszenia\" i zastanowimy się nad właściwym optymalizatorem i funkcją straty. Wprowadzimy również do naszego modelu podstawowy Early Stopping, który będzie zapobiegać przeuczeniu się naszego modelu."]},{"cell_type":"markdown","metadata":{"id":"gchK-2kPR6fM"},"source":["### 1. Import bibliotek\n","#### 🇬🇧 Import libraries\n","W tym miejscu importujemy biblioteki, które będą nam potrzebne do wytrenowania sieci neuronowej. Jeżeli w tym miejscu widoczny będzie błąd, to konieczne jest zainstalowanie biblioteki w danym środowisku. W takim wypadku wchodzimy w terminal, aktywujemy środowisko pracy, instalujemy biblioteki (pip install X)."]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"0hk22wZDYbVQ","outputId":"1982e6e3-26c4-4eec-fb18-d0cf7ed0bd03","executionInfo":{"status":"ok","timestamp":1649519379023,"user_tz":-120,"elapsed":6248,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}}},"outputs":[{"output_type":"stream","name":"stdout","text":["Requirement already satisfied: mnist in /usr/local/lib/python3.7/dist-packages (0.2.2)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from mnist) (1.21.5)\n"]}],"source":["!pip install mnist\n","\n","import tensorflow as tf\n","import numpy as np\n","import mnist\n","\n","from keras.models import Sequential\n","from keras.layers import Dense\n","from keras.layers import Dropout\n","from keras.callbacks import EarlyStopping\n","from keras.callbacks import ModelCheckpoint\n","from keras.models import load_model\n","from matplotlib import pyplot\n","from tensorflow.keras.utils import to_categorical\n","\n","from tensorflow import keras\n","from tensorflow.keras import backend as K\n","\n","# Rozpoczynamy od importu potrzebnych nam bibliotek, nowością jest EarlyStopping"]},{"cell_type":"markdown","metadata":{"id":"APpHwz_fR6fN"},"source":["### 2. Przygotowanie danych do analizy\n","#### 🇬🇧 Data preparation"]},{"cell_type":"code","execution_count":2,"metadata":{"id":"2CPZuB2vsqBS","executionInfo":{"status":"ok","timestamp":1649519380478,"user_tz":-120,"elapsed":1463,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}}},"outputs":[],"source":["train_images = mnist.train_images()\n","train_labels = mnist.train_labels()\n","test_images = mnist.test_images()\n","test_labels = mnist.test_labels()\n","\n","train_images = (train_images / 255) - 0.5\n","test_images = (test_images / 255) - 0.5\n","\n","train_images = train_images.reshape((-1, 784))\n","test_images = test_images.reshape((-1, 784))\n","\n","model = Sequential([\n","  Dense(64, activation='relu', input_shape=(784,)),\n","  Dense(64, activation='relu'),\n","  Dense(10, activation='softmax'),\n","])"]},{"cell_type":"code","source":["train_labels_ = to_categorical(train_labels)\n","test_labels_ = to_categorical(test_labels)"],"metadata":{"id":"z0huzZt10mNL","executionInfo":{"status":"ok","timestamp":1649519380479,"user_tz":-120,"elapsed":3,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}}},"execution_count":3,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"iLsO0jydsqtG"},"source":["Zatrzymajmy się na chwilę w tym miejscu. Na ostatnich zajęciach modyfikowaliśmy architekturę sieci neuronowej. Teraz przyszedł czas na wybór odpowiedniego optymalizatora, metryk oraz funkcji straty."]},{"cell_type":"code","execution_count":4,"metadata":{"id":"T-nhhMPgV6VD","executionInfo":{"status":"ok","timestamp":1649519380839,"user_tz":-120,"elapsed":2,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}}},"outputs":[],"source":["model.compile(\n","  optimizer='adam',\n","  loss='categorical_crossentropy',\n","  metrics=['accuracy'],\n",")"]},{"cell_type":"markdown","metadata":{"id":"2TyGYkqBR6fO"},"source":["### 3. Trenowanie modelu\n","#### 🇬🇧 Model training"]},{"cell_type":"code","execution_count":5,"metadata":{"id":"OhIvq1x3R6fO","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1649519714738,"user_tz":-120,"elapsed":333900,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}},"outputId":"9e570c7f-4a97-4d17-fe10-c3cb14fe7352"},"outputs":[{"output_type":"stream","name":"stdout","text":["\n","Epoch 1: accuracy improved from -inf to 0.89255, saving model to model1.h5\n","\n","Epoch 2: accuracy improved from 0.89255 to 0.94462, saving model to model1.h5\n","\n","Epoch 3: accuracy improved from 0.94462 to 0.95755, saving model to model1.h5\n","\n","Epoch 4: accuracy improved from 0.95755 to 0.96347, saving model to model1.h5\n","\n","Epoch 5: accuracy improved from 0.96347 to 0.96842, saving model to model1.h5\n","\n","Epoch 6: accuracy improved from 0.96842 to 0.97167, saving model to model1.h5\n","\n","Epoch 7: accuracy improved from 0.97167 to 0.97328, saving model to model1.h5\n","\n","Epoch 8: accuracy improved from 0.97328 to 0.97528, saving model to model1.h5\n","\n","Epoch 9: accuracy improved from 0.97528 to 0.97772, saving model to model1.h5\n","\n","Epoch 10: accuracy improved from 0.97772 to 0.97885, saving model to model1.h5\n","\n","Epoch 11: accuracy improved from 0.97885 to 0.98010, saving model to model1.h5\n","\n","Epoch 12: accuracy improved from 0.98010 to 0.98032, saving model to model1.h5\n","\n","Epoch 13: accuracy improved from 0.98032 to 0.98237, saving model to model1.h5\n","\n","Epoch 14: accuracy improved from 0.98237 to 0.98310, saving model to model1.h5\n","\n","Epoch 15: accuracy improved from 0.98310 to 0.98410, saving model to model1.h5\n","\n","Epoch 16: accuracy improved from 0.98410 to 0.98503, saving model to model1.h5\n","\n","Epoch 17: accuracy did not improve from 0.98503\n","\n","Epoch 18: accuracy improved from 0.98503 to 0.98577, saving model to model1.h5\n","\n","Epoch 19: accuracy improved from 0.98577 to 0.98625, saving model to model1.h5\n","\n","Epoch 20: accuracy improved from 0.98625 to 0.98675, saving model to model1.h5\n","\n","Epoch 21: accuracy improved from 0.98675 to 0.98822, saving model to model1.h5\n","\n","Epoch 22: accuracy did not improve from 0.98822\n","\n","Epoch 23: accuracy improved from 0.98822 to 0.98910, saving model to model1.h5\n","\n","Epoch 24: accuracy improved from 0.98910 to 0.98918, saving model to model1.h5\n","\n","Epoch 25: accuracy did not improve from 0.98918\n","\n","Epoch 26: accuracy improved from 0.98918 to 0.98940, saving model to model1.h5\n","\n","Epoch 27: accuracy improved from 0.98940 to 0.99025, saving model to model1.h5\n","\n","Epoch 28: accuracy did not improve from 0.99025\n","\n","Epoch 29: accuracy did not improve from 0.99025\n","\n","Epoch 30: accuracy improved from 0.99025 to 0.99133, saving model to model1.h5\n","\n","Epoch 31: accuracy did not improve from 0.99133\n","\n","Epoch 32: accuracy did not improve from 0.99133\n","\n","Epoch 33: accuracy improved from 0.99133 to 0.99207, saving model to model1.h5\n","\n","Epoch 34: accuracy did not improve from 0.99207\n","\n","Epoch 35: accuracy did not improve from 0.99207\n","\n","Epoch 36: accuracy did not improve from 0.99207\n","\n","Epoch 37: accuracy improved from 0.99207 to 0.99237, saving model to model1.h5\n","\n","Epoch 38: accuracy did not improve from 0.99237\n","\n","Epoch 39: accuracy improved from 0.99237 to 0.99287, saving model to model1.h5\n","\n","Epoch 40: accuracy did not improve from 0.99287\n","\n","Epoch 41: accuracy did not improve from 0.99287\n","\n","Epoch 42: accuracy improved from 0.99287 to 0.99323, saving model to model1.h5\n","\n","Epoch 43: accuracy did not improve from 0.99323\n","\n","Epoch 44: accuracy did not improve from 0.99323\n","\n","Epoch 45: accuracy did not improve from 0.99323\n","\n","Epoch 46: accuracy did not improve from 0.99323\n","\n","Epoch 47: accuracy did not improve from 0.99323\n","\n","Epoch 48: accuracy improved from 0.99323 to 0.99392, saving model to model1.h5\n","\n","Epoch 49: accuracy did not improve from 0.99392\n","\n","Epoch 50: accuracy improved from 0.99392 to 0.99403, saving model to model1.h5\n","\n","Epoch 51: accuracy did not improve from 0.99403\n","\n","Epoch 52: accuracy did not improve from 0.99403\n","\n","Epoch 53: accuracy improved from 0.99403 to 0.99432, saving model to model1.h5\n","\n","Epoch 54: accuracy did not improve from 0.99432\n","\n","Epoch 55: accuracy did not improve from 0.99432\n","\n","Epoch 56: accuracy did not improve from 0.99432\n","\n","Epoch 57: accuracy improved from 0.99432 to 0.99447, saving model to model1.h5\n","\n","Epoch 58: accuracy did not improve from 0.99447\n","\n","Epoch 59: accuracy improved from 0.99447 to 0.99450, saving model to model1.h5\n","\n","Epoch 60: accuracy did not improve from 0.99450\n","\n","Epoch 61: accuracy did not improve from 0.99450\n","\n","Epoch 62: accuracy did not improve from 0.99450\n","\n","Epoch 63: accuracy did not improve from 0.99450\n","\n","Epoch 64: accuracy did not improve from 0.99450\n","\n","Epoch 65: accuracy did not improve from 0.99450\n","\n","Epoch 66: accuracy improved from 0.99450 to 0.99473, saving model to model1.h5\n","\n","Epoch 67: accuracy improved from 0.99473 to 0.99493, saving model to model1.h5\n","\n","Epoch 68: accuracy did not improve from 0.99493\n","\n","Epoch 69: accuracy improved from 0.99493 to 0.99533, saving model to model1.h5\n","\n","Epoch 70: accuracy did not improve from 0.99533\n","\n","Epoch 71: accuracy did not improve from 0.99533\n","\n","Epoch 72: accuracy did not improve from 0.99533\n","\n","Epoch 73: accuracy improved from 0.99533 to 0.99573, saving model to model1.h5\n","\n","Epoch 74: accuracy did not improve from 0.99573\n","\n","Epoch 75: accuracy did not improve from 0.99573\n","\n","Epoch 76: accuracy did not improve from 0.99573\n","\n","Epoch 77: accuracy did not improve from 0.99573\n","\n","Epoch 78: accuracy did not improve from 0.99573\n","\n","Epoch 79: accuracy did not improve from 0.99573\n","\n","Epoch 80: accuracy did not improve from 0.99573\n","\n","Epoch 81: accuracy did not improve from 0.99573\n","\n","Epoch 82: accuracy did not improve from 0.99573\n","\n","Epoch 83: accuracy did not improve from 0.99573\n","Epoch 83: early stopping\n","1875/1875 [==============================] - 4s 2ms/step - loss: 0.0125 - accuracy: 0.9957\n","313/313 [==============================] - 1s 3ms/step - loss: 0.1888 - accuracy: 0.9735\n","[0.012455979362130165, 0.9957333207130432, 0.18879833817481995, 0.9735000133514404]\n"]}],"source":["from os import path\n","#1\n","f1 = EarlyStopping(monitor='accuracy', mode='auto', verbose=1, patience=10)\n","\n","#2\n","f2 = ModelCheckpoint(filepath='model1.h5', monitor='accuracy', mode='auto', verbose=1, save_best_only=True)\n","\n","#3\n","history = model.fit(train_images, train_labels_, batch_size = None, validation_data = (test_images, test_labels_), callbacks = [f1,f2], epochs=100, verbose=0)\n","\n","#4\n","trained_model = load_model('model1.h5')\n","\n","#5\n","train_accuracy = trained_model.evaluate(train_images, train_labels_)\n","\n","#6\n","test_accuracy = trained_model.evaluate(test_images, test_labels_)\n","\n","#7\n","print(train_accuracy + test_accuracy)"]},{"cell_type":"code","source":["print(f'Model przerwal uczenie w epoce: { f1.stopped_epoch}')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"lAyXiYW156kk","executionInfo":{"status":"ok","timestamp":1649519714738,"user_tz":-120,"elapsed":12,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}},"outputId":"e17bb78a-50c7-439e-82c5-2e9d4795b1e5"},"execution_count":6,"outputs":[{"output_type":"stream","name":"stdout","text":["Model przerwal uczenie w epoce: 82\n"]}]},{"cell_type":"markdown","metadata":{"id":"li4Y1654R6fP"},"source":["### 📚 Zadanie 1.\n","##### Przeprowadź proces uczenia się z wykorzystaniem early stopping, wykonaj poniższe kroki."]},{"cell_type":"markdown","metadata":{"id":"8InTZwPRCfJy"},"source":["1. Stwórz nową zmienną, którą nazwiesz f1, odwołaj się do early stopping, wprowadź odpowiednie wartości (takie jakie chcesz, tylko dla zmiennej patience wprowadź \"10\"), do parametrów takich jak monitor, mode, verbose oraz patience.\n","2. Stwórz nową zmienną, którą nazwiesz f2, odwołaj się do ModelCheckpoint, wprowadź nazwę dla swojego modelu oraz odpowiednie wartości dla parametru monitor, mode, verbose oraz save_best_only (znowu - takie jakie chcesz, poeksperymentuj z nimi 🔬).\n","3. Stwórz nową zmienną, którą nazwiesz history. Zmienna ta powinna zawierać stworzone w poprzednim kroku dane treningowe (x), etykiety danych trningowych (y), dane testowe (a), etykiey danych testowych (b) oraz odpowiednie wartości parametrów epochs oraz verbose.\n","4. Stwórz nową zmienną, którą nazwiesz trained_model, załaduj model używając nazwy modelu z punktu drugiego.\n","5. Stwórz nową zmienną, którą nazwiesz train_accuracy, odwołaj się do trained_model.evaluate, użyj danych treningowych (x) oraz etykiet danych treningowych (y).\n","6. Stwórz nową zmienną, którą nazwiesz test_accuracy, odwołaj się do trained_model.evaluate, użyj danych testowych (a) oraz etykiet danych testowych (b).\n","7. Przeprowadź uczenie sieci dla liczby epok wynoszącej 100. Po której epoce proces trenowania sieci się zatrzyma?\n","8. Wyświetl informacje dotyczące dokładności klasyfikacji na materiale treningowym oraz testowym."]},{"cell_type":"markdown","metadata":{"id":"9-vMmyrGR6fP"},"source":["### 📚 Zadanie 2.\n","##### Uruchom ponownie proces treningu sieci, tym razem wprowadzając zmianę związaną z funkcją straty oraz optymalizatora."]},{"cell_type":"markdown","metadata":{"id":"hpxxzs6sR6fQ"},"source":["1. Wypróbuj inną funkcję straty (rozkład Poisona). Czy w przypadku naszego zbioru danych taka funkcja ma sens?\n","2. Wypróbuj inny optymalizator (z wykorzystaniem algorytmu Adadelta). Czy obserwujesz jakieś zmiany w procesie uczenia się w porównaniu do algorytmu wykorzystanego w tutorialu (adam)? Gdzie może leżeć przyczyna tych różnic? (zob. https://keras.io/api/optimizers/adadelta)."]},{"cell_type":"code","source":["#Zadanie 2\n","from os import path\n","p = tf.keras.losses.Poisson()\n","a = tf.keras.optimizers.Adadelta()\n","\n","model2 = Sequential([\n","  Dense(64, activation='relu', input_shape=(784,)),\n","  Dense(64, activation='relu'),\n","  Dense(10, activation='softmax'),\n","])\n","\n","model2.compile(\n","  optimizer=a,\n","  loss=p,\n","  metrics=['accuracy'],\n",")\n","\n","#1\n","f1 = EarlyStopping(monitor='accuracy', mode='auto', verbose=1, patience=10)\n","\n","#2\n","f2 = ModelCheckpoint(filepath='model2.h5', monitor='accuracy', mode='auto', verbose=1, save_best_only=True)\n","\n","#3\n","history = model2.fit(train_images, train_labels_, batch_size = None, validation_data = (test_images, test_labels_), callbacks = [f1,f2], epochs=100, verbose=0)\n","\n","#4\n","trained_model = load_model('model2.h5')\n","\n","#5\n","train_accuracy = trained_model.evaluate(train_images, train_labels_)\n","\n","#6\n","test_accuracy = trained_model.evaluate(test_images, test_labels_)\n","\n","#7\n","print(train_accuracy + test_accuracy)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"_KqSG2Jj6cdF","executionInfo":{"status":"ok","timestamp":1649520203882,"user_tz":-120,"elapsed":389197,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}},"outputId":"735ed5b7-0483-4687-f933-5896d0fc0293"},"execution_count":8,"outputs":[{"output_type":"stream","name":"stdout","text":["\n","Epoch 1: accuracy improved from -inf to 0.08215, saving model to model2.h5\n","\n","Epoch 2: accuracy improved from 0.08215 to 0.12783, saving model to model2.h5\n","\n","Epoch 3: accuracy improved from 0.12783 to 0.21850, saving model to model2.h5\n","\n","Epoch 4: accuracy improved from 0.21850 to 0.28918, saving model to model2.h5\n","\n","Epoch 5: accuracy improved from 0.28918 to 0.33958, saving model to model2.h5\n","\n","Epoch 6: accuracy improved from 0.33958 to 0.38138, saving model to model2.h5\n","\n","Epoch 7: accuracy improved from 0.38138 to 0.42262, saving model to model2.h5\n","\n","Epoch 8: accuracy improved from 0.42262 to 0.46018, saving model to model2.h5\n","\n","Epoch 9: accuracy improved from 0.46018 to 0.49350, saving model to model2.h5\n","\n","Epoch 10: accuracy improved from 0.49350 to 0.52485, saving model to model2.h5\n","\n","Epoch 11: accuracy improved from 0.52485 to 0.55310, saving model to model2.h5\n","\n","Epoch 12: accuracy improved from 0.55310 to 0.58070, saving model to model2.h5\n","\n","Epoch 13: accuracy improved from 0.58070 to 0.60523, saving model to model2.h5\n","\n","Epoch 14: accuracy improved from 0.60523 to 0.62868, saving model to model2.h5\n","\n","Epoch 15: accuracy improved from 0.62868 to 0.65092, saving model to model2.h5\n","\n","Epoch 16: accuracy improved from 0.65092 to 0.66912, saving model to model2.h5\n","\n","Epoch 17: accuracy improved from 0.66912 to 0.68783, saving model to model2.h5\n","\n","Epoch 18: accuracy improved from 0.68783 to 0.70325, saving model to model2.h5\n","\n","Epoch 19: accuracy improved from 0.70325 to 0.71768, saving model to model2.h5\n","\n","Epoch 20: accuracy improved from 0.71768 to 0.72975, saving model to model2.h5\n","\n","Epoch 21: accuracy improved from 0.72975 to 0.73967, saving model to model2.h5\n","\n","Epoch 22: accuracy improved from 0.73967 to 0.74983, saving model to model2.h5\n","\n","Epoch 23: accuracy improved from 0.74983 to 0.75840, saving model to model2.h5\n","\n","Epoch 24: accuracy improved from 0.75840 to 0.76515, saving model to model2.h5\n","\n","Epoch 25: accuracy improved from 0.76515 to 0.77135, saving model to model2.h5\n","\n","Epoch 26: accuracy improved from 0.77135 to 0.77758, saving model to model2.h5\n","\n","Epoch 27: accuracy improved from 0.77758 to 0.78360, saving model to model2.h5\n","\n","Epoch 28: accuracy improved from 0.78360 to 0.78892, saving model to model2.h5\n","\n","Epoch 29: accuracy improved from 0.78892 to 0.79390, saving model to model2.h5\n","\n","Epoch 30: accuracy improved from 0.79390 to 0.79837, saving model to model2.h5\n","\n","Epoch 31: accuracy improved from 0.79837 to 0.80308, saving model to model2.h5\n","\n","Epoch 32: accuracy improved from 0.80308 to 0.80685, saving model to model2.h5\n","\n","Epoch 33: accuracy improved from 0.80685 to 0.81035, saving model to model2.h5\n","\n","Epoch 34: accuracy improved from 0.81035 to 0.81357, saving model to model2.h5\n","\n","Epoch 35: accuracy improved from 0.81357 to 0.81697, saving model to model2.h5\n","\n","Epoch 36: accuracy improved from 0.81697 to 0.81992, saving model to model2.h5\n","\n","Epoch 37: accuracy improved from 0.81992 to 0.82283, saving model to model2.h5\n","\n","Epoch 38: accuracy improved from 0.82283 to 0.82503, saving model to model2.h5\n","\n","Epoch 39: accuracy improved from 0.82503 to 0.82760, saving model to model2.h5\n","\n","Epoch 40: accuracy improved from 0.82760 to 0.83062, saving model to model2.h5\n","\n","Epoch 41: accuracy improved from 0.83062 to 0.83235, saving model to model2.h5\n","\n","Epoch 42: accuracy improved from 0.83235 to 0.83532, saving model to model2.h5\n","\n","Epoch 43: accuracy improved from 0.83532 to 0.83692, saving model to model2.h5\n","\n","Epoch 44: accuracy improved from 0.83692 to 0.83872, saving model to model2.h5\n","\n","Epoch 45: accuracy improved from 0.83872 to 0.84137, saving model to model2.h5\n","\n","Epoch 46: accuracy improved from 0.84137 to 0.84315, saving model to model2.h5\n","\n","Epoch 47: accuracy improved from 0.84315 to 0.84487, saving model to model2.h5\n","\n","Epoch 48: accuracy improved from 0.84487 to 0.84663, saving model to model2.h5\n","\n","Epoch 49: accuracy improved from 0.84663 to 0.84817, saving model to model2.h5\n","\n","Epoch 50: accuracy improved from 0.84817 to 0.84935, saving model to model2.h5\n","\n","Epoch 51: accuracy improved from 0.84935 to 0.85082, saving model to model2.h5\n","\n","Epoch 52: accuracy improved from 0.85082 to 0.85185, saving model to model2.h5\n","\n","Epoch 53: accuracy improved from 0.85185 to 0.85283, saving model to model2.h5\n","\n","Epoch 54: accuracy improved from 0.85283 to 0.85418, saving model to model2.h5\n","\n","Epoch 55: accuracy improved from 0.85418 to 0.85518, saving model to model2.h5\n","\n","Epoch 56: accuracy improved from 0.85518 to 0.85638, saving model to model2.h5\n","\n","Epoch 57: accuracy improved from 0.85638 to 0.85773, saving model to model2.h5\n","\n","Epoch 58: accuracy improved from 0.85773 to 0.85857, saving model to model2.h5\n","\n","Epoch 59: accuracy improved from 0.85857 to 0.85955, saving model to model2.h5\n","\n","Epoch 60: accuracy improved from 0.85955 to 0.86043, saving model to model2.h5\n","\n","Epoch 61: accuracy improved from 0.86043 to 0.86135, saving model to model2.h5\n","\n","Epoch 62: accuracy improved from 0.86135 to 0.86203, saving model to model2.h5\n","\n","Epoch 63: accuracy improved from 0.86203 to 0.86280, saving model to model2.h5\n","\n","Epoch 64: accuracy improved from 0.86280 to 0.86387, saving model to model2.h5\n","\n","Epoch 65: accuracy improved from 0.86387 to 0.86455, saving model to model2.h5\n","\n","Epoch 66: accuracy improved from 0.86455 to 0.86520, saving model to model2.h5\n","\n","Epoch 67: accuracy improved from 0.86520 to 0.86615, saving model to model2.h5\n","\n","Epoch 68: accuracy improved from 0.86615 to 0.86678, saving model to model2.h5\n","\n","Epoch 69: accuracy improved from 0.86678 to 0.86745, saving model to model2.h5\n","\n","Epoch 70: accuracy improved from 0.86745 to 0.86813, saving model to model2.h5\n","\n","Epoch 71: accuracy improved from 0.86813 to 0.86875, saving model to model2.h5\n","\n","Epoch 72: accuracy improved from 0.86875 to 0.86952, saving model to model2.h5\n","\n","Epoch 73: accuracy improved from 0.86952 to 0.87022, saving model to model2.h5\n","\n","Epoch 74: accuracy improved from 0.87022 to 0.87092, saving model to model2.h5\n","\n","Epoch 75: accuracy improved from 0.87092 to 0.87145, saving model to model2.h5\n","\n","Epoch 76: accuracy improved from 0.87145 to 0.87217, saving model to model2.h5\n","\n","Epoch 77: accuracy improved from 0.87217 to 0.87247, saving model to model2.h5\n","\n","Epoch 78: accuracy improved from 0.87247 to 0.87337, saving model to model2.h5\n","\n","Epoch 79: accuracy improved from 0.87337 to 0.87398, saving model to model2.h5\n","\n","Epoch 80: accuracy improved from 0.87398 to 0.87420, saving model to model2.h5\n","\n","Epoch 81: accuracy improved from 0.87420 to 0.87440, saving model to model2.h5\n","\n","Epoch 82: accuracy improved from 0.87440 to 0.87505, saving model to model2.h5\n","\n","Epoch 83: accuracy improved from 0.87505 to 0.87553, saving model to model2.h5\n","\n","Epoch 84: accuracy improved from 0.87553 to 0.87598, saving model to model2.h5\n","\n","Epoch 85: accuracy improved from 0.87598 to 0.87662, saving model to model2.h5\n","\n","Epoch 86: accuracy improved from 0.87662 to 0.87698, saving model to model2.h5\n","\n","Epoch 87: accuracy improved from 0.87698 to 0.87793, saving model to model2.h5\n","\n","Epoch 88: accuracy improved from 0.87793 to 0.87838, saving model to model2.h5\n","\n","Epoch 89: accuracy improved from 0.87838 to 0.87853, saving model to model2.h5\n","\n","Epoch 90: accuracy improved from 0.87853 to 0.87917, saving model to model2.h5\n","\n","Epoch 91: accuracy improved from 0.87917 to 0.87950, saving model to model2.h5\n","\n","Epoch 92: accuracy improved from 0.87950 to 0.88008, saving model to model2.h5\n","\n","Epoch 93: accuracy improved from 0.88008 to 0.88032, saving model to model2.h5\n","\n","Epoch 94: accuracy improved from 0.88032 to 0.88095, saving model to model2.h5\n","\n","Epoch 95: accuracy improved from 0.88095 to 0.88120, saving model to model2.h5\n","\n","Epoch 96: accuracy improved from 0.88120 to 0.88143, saving model to model2.h5\n","\n","Epoch 97: accuracy improved from 0.88143 to 0.88168, saving model to model2.h5\n","\n","Epoch 98: accuracy improved from 0.88168 to 0.88200, saving model to model2.h5\n","\n","Epoch 99: accuracy improved from 0.88200 to 0.88225, saving model to model2.h5\n","\n","Epoch 100: accuracy improved from 0.88225 to 0.88298, saving model to model2.h5\n","1875/1875 [==============================] - 3s 2ms/step - loss: 0.1429 - accuracy: 0.8833\n","313/313 [==============================] - 1s 2ms/step - loss: 0.1409 - accuracy: 0.8913\n","[0.14287178218364716, 0.883316695690155, 0.14088217914104462, 0.8913000226020813]\n"]}]},{"cell_type":"code","source":["'''\n","W modelu 2, proces uczenia przebiegal znacznie wolniej. Pulap z ktorego zaczynal byl bardzo niski, bo w porownaniu \n","z modelem 1, w tym przypadku zaczal z bardzo niskiego accuracy i w 100 epoce osiagnal accuracy mniej wiecej takie\n","z jakiego zaczynal model 1. Z informacji znalezionych w sieci zauwazylem ze optymizer Adadelta podczas trenowania\n","dziala znaczej wolnej co potiwerdzilo sie w moim eksperymencie. Obecnie Adam jest znaczenie czesciej uzywany co \n","potwierdza fakt znacznie lepszych wynikow w modelu 1.\n","'''\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":70},"id":"ZEhjgQcN-0a6","executionInfo":{"status":"ok","timestamp":1649520558437,"user_tz":-120,"elapsed":312,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}},"outputId":"9f80e0b6-f9e4-4208-f936-131f835bbdd1"},"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'\\nW modelu 2, proces uczenia przebiegal znacznie wolniej. Pulap z ktorego zaczynal byl bardzo niski, bo w porownaniu \\nz modelem 1, w tym przypadku zaczal z bardzo niskiego accuracy i w 100 epoce osiagnal accuracy mniej wiecej takie\\nz jakiego zaczynal model 1. Z informacji znalezionych w sieci zauwazylem ze optymizer Adadelta podczas trenowania\\ndziala znaczej wolnej co potiwerdzilo sie w moim eksperymencie. Obecnie Adam jest znaczenie czesciej uzywany co \\npotwierdza fakt znacznie lepszych wynikow w modelu 1.\\n'"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"}},"metadata":{},"execution_count":10}]},{"cell_type":"markdown","metadata":{"id":"x61bwRH5R6fR"},"source":["### 📚 Zadanie 3.\n","##### Niekiedy przydaje nam się możliwość wprowadzenia modyfikacji do funkcji straty lub zbudowania własnej funkcji straty od postaw. W Keras/Tensorflow jest to dosyć prosta rzecz, którą możemy zrealizować za pomocą podstawowych operacji."]},{"cell_type":"markdown","metadata":{"id":"Xjpqs_zRR6fR"},"source":["Poniższy przykład ilustruje w jaki sposób możemy skorzystać z funkcji straty Root Mean Square Error (RMSE) dla dwóch zmiennych (y_true i y_pred)."]},{"cell_type":"code","execution_count":11,"metadata":{"id":"XSF0qsNyR6fR","executionInfo":{"status":"ok","timestamp":1649520571430,"user_tz":-120,"elapsed":308,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}}},"outputs":[],"source":["y_true = 43.2\n","y_pred = 4.2"]},{"cell_type":"code","execution_count":12,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"i5kU6V0IR6fR","outputId":"8f441c30-e50c-4faf-abca-2686bca00787","executionInfo":{"status":"ok","timestamp":1649520572795,"user_tz":-120,"elapsed":321,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}}},"outputs":[{"output_type":"execute_result","data":{"text/plain":["39.0"]},"metadata":{},"execution_count":12}],"source":["x = tf.keras.metrics.RootMeanSquaredError()\n","x(y_true, y_pred).numpy()"]},{"cell_type":"markdown","metadata":{"id":"Gc5btaV4R6fS"},"source":["Czasami jednak możemy chcieć stworzyć naszą własną funkcję straty (podajemy przy tym dwa argumenty, tj. y_true oraz y_pred)."]},{"cell_type":"code","execution_count":13,"metadata":{"id":"dZa95_oER6fS","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1649520574366,"user_tz":-120,"elapsed":371,"user":{"displayName":"Dominik Zalewski","userId":"03487575896574739301"}},"outputId":"3aea9c39-39b0-42ff-cabf-cf9788e935ab"},"outputs":[{"output_type":"stream","name":"stdout","text":["39.0\n"]}],"source":["import numpy as np\n","\n","def custom_loss_function(y_true, y_pred):\n","    return np.sqrt((y_true - y_pred)**2)\n","\n","rmse = custom_loss_function(y_true, y_pred)\n","print(rmse)"]},{"cell_type":"markdown","metadata":{"id":"6YSGpIQuR6fS"},"source":["Odtwórz wskazaną wcześniej funkcję straty Root Mean Squared Error tworząc od podstaw nową funkcję."]},{"cell_type":"markdown","metadata":{"id":"cj4DgCl2UmaC"},"source":["![2fae41d2-cda7-466c-b672-f909e745e126.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAeMAAACXCAQAAACMuvvQAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAAAmJLR0QA/4ePzL8AAAAHdElNRQfmAhEPOiMSPtmTAAARFklEQVR42u2dTXKjSBqG3+7o9YTwEcQNTB1B9AkGH8HcoOkjFHWCwUdo1QkaH8HUYrYTeD8b45jNzERMBLMgkfjNTERiSPw+iqgKSwIlEl9+v/nlTyC2EuCEu5HX3vC49vDIx/HT2gMgN5PjuPYQCCHzKJCuPQSyDX5eewCEkLlQjAmxHooxIdZDMSbEeijGhFgPxZgQ66EYE2I9FGNCrIdiTIj1UIwJsR6KMSHWQzEmxHooxoRYD8WYEOuhGBNiPRRjQqyHYkyI9VCMCbEeijEh1kMxJgCQIEeOCADgIEKGHAXOcNceGCH7xlxLvRgxgAQlQnjIEQIAXJQo4Kx9mYTsGXNiXAAAIpTIkcO7PJ+hhL/2ZRI1v6w9AHIz/8NfjJwnxDMA4AuAI35F1nqVvbAtgL6xvfyCfxk601cAwD2Ap5aGPwJ4WfsyCdkzZtvNOyhRNgxqwEOJcvGrcD6t2e6biztQG5OKE4D3lkH9AODHwp/q4nXtC1+NN7y2ps0hHMRIkSPHGRHDjXvErDZOUOLceiZHKWLWS+GiEEku+bv2StCxf7pEyBDBh4sAZ5QoNb4tYhlmxbgrtJVJXc3/59tOqfGZ6jPHu46Wx5KUXtD5dkKUKBGvPWRiFpNiXHnGTb0XoxTn95AvMv5UJLpkRChR7lqQs05m4EqBomMNpb1fiViPSTEOUHaEKrto5xTBAqMPUSrPG6FEDAeZwvi0GXdEw1YTa3sCDaiP94dJMU5QIuk9EwCIO8+bwdEYfdjwBc8odivI6cV5aZP1fOHK0Vni9yCrYVKMowExSVAgX2juV3u8XW2doNipOen3gouydzLMtTPs3abcRTnqEdbkPTFfxi7YArpOQ9zL7RPrsVeMY+qUFoGmPi4uYUeyG+wV42LEG/y86NTLnVEi5/e2N2wVY78XgSWZstQmRIl8p9GBT42tYnxmtLVHrDCXA5RIqYn3iK1ivHyRp30EUrM64MS3X+wUY0cRbfWRobg0FGofmW3KNzQ5Utm3ErSmPXfHFW2fEjvFWK53ImRwURWfdDPW20q3mB7pmI0SdjL6Eau49oWdYhxJAlzeZaFA1CsOrcJAy69/1sP8SLNBwzlE1tHqg6WxbOJDPpYjgPeR157wIETiC4BD59V7LL/+WRfzI31Dv2FShK/4hkfx1x0OuMMJv6998cQkdmrjdDQm6ze0dNHT2f7gsoAQyQre8hIjPffOU6/u6j7IrrBTjLPRiqXzJVgUDNQOxwMLFv2VFjEuMdKkY5y7I0LMjPvOsFOM89HkSXDRVudGy4KaIX/T1Wo8YJ4lRhoP+NjkE6AW42RkRjf5mDqVFMoVs1XyJVM+p084+ar0JgeTI43mGMwMcZF5nOBOMvTeegGhLg8AgKfWc18AQPTTns5L52xq9D7J/EjJp+O/+IfiHQ6KhoaZvpGLAwcuPHgIECERTWS6j2n1RbnyiGzAUB3yN9fH5EhpVH9S/o2/K98TtATuVqO0iYtQeIS3xU4zpck6ZKrnm4zRmhxpwvDV50QvxNX2j81VAAXihp3aj+Ks8Ke9gXE6PYHxRPfmNfWz2ZGeDU2zxDJ0I9V5S5BNFjMGwmifokVixe1ahaPalUrdtI4rCiHzVdctmx1pqh1YG4C7RuyfX1t/PRu88c844hnAcYJWfIV8e7cDgG4N1EmMvOZPPCIHcIc6pLQGZkd6FN8N+WTo543bCRfT2WbVWtk2zTb2Q1SFEs3l8W7HAw8u/bPLVbWx2ZFuMYRHPoAp5R/toJTp1b4JpjRB75uiTaroetj4O+9MPqG43ZMFpqQpmBzp2lMSWY1pVVzLecgAkE5IO2WKUJuPUizOcxAiF2mufhhNPh18BOZGGjJO/VmZJsbeopW57gRdEil1k4szchTIkMARlkR34gk3kWc1NdKz0SyCtGwtR4azwiALOsfoDa2bh5QbaA4ipChQokDW2C4yGNQIkeSK+g+blmRPralufxOma5EjbQ3vTBTAcvD9Kp2+BrePNDdrITnw4SO8VP+E8MUjQISzeD6Whij8Rk5RJxN2/bQYAXzp5TgiTV5tF+kjRCYMFn8kf+nAg9+YnkL48C4PHwFCRKIex66OR9OXRrSrsNYzSNMJ4ZzhZX9uY7rfym92+0jdpUzqbHTGjsWcI9eYqSg7UM+6DjKhjdU/h4d84IsKUApNI5sACkWU1kMxsZBhbaaLcbs4c729+cZaxLnIUHR+33TwLrpmn6dVdJvC7EgXa78vm6/PGt5VIeYmtcd0FgKk1g7eqKgGyinDvejicaLVQybTuGWhot8S4/WqhobLIdLenRWM3BnpZTpXuXnLYHakxTJevlwEA6iMMg8l6tio3OIPkFxETC7wjlRb5gpvL9TQP94CEdwluW29cbyJWMCwC5R3xuSMLmuMxZQcrbLe2OxIdbaIvYlYqm9rIR83giOkqGcs2QAdZKhFTKUZ5DWniULTnjVMfGdFM/MWbm0bkLUEea2Sg3RgxVWKrJeHHZtoltzvUWf05kZaLGUVZVIxjZQzeYoI9WQgs/nP8FGLmM5S8vFzxQoRLDSisw62t4ZGxq1i3G4PM335ohncgTvMRSGmfQcRipWXPshHb2qk8XI2oHyertNDgeR4D7WWHRcdX7xWaGiFUPGJkdRX1/GMq5/GJm5v4tNOC65VCRUM/OpVHrZEMdzIdUOYGamnvCtvpjaah3GgKiDwxLG+wliuOvDqecYqzS7f+3bMM45aP4Bn2UKxOb242pn6teLz8W63HtfDQbFcuiyWil8dIhk3BCJxe7lSfzQRIqTnGceQ+7bybPOYZ5x3jrIpwDW3pd6yxZm6JJ+6BDFbMkCXSXxVXwiE7GdPL7P7uFb3Lheg4xlfzcDsptl72DP2YJcv3L+qOWLcLs5cy0P+zJuZLHzl455xJEJfqiVXtZDno0me65YWOp7x1ZSvzPmk0WJUzZhnnFnZIPbK3Aa37cLbdRI3ZCGGPGMHPmIUKHFWasOmjhtLOcUXkXIh88Ob9Ku9M8RaxmB9ZP1eBy4CZDf4hNPqs5cOIc3vU90uzuRmpdbSb3B7Ev/XPsud6HLwjketGfvU6H1QdTPo9npwcX8RoOrTdPa7SfCGP1rP3OMev+FJefvVV/TSe2VqG9If+D7xiFs+5eN4wGuj3ezf8PypPdVdkfVilw58US+qkxtLG8eGg/nnpn+r5xlfRxINNFlVTS51NXVzOUSiaQNsGRO7RrSLMynEu2EscnlW1mTVx1+P9QeMyqg1Reh5xl28jjjLRjXmGaeWpZf6mNn8pV2cuZXVQmQW4zljPS+2Hf3tp5zclvDoe8ZDOJfmrbLbeSxnnFofIVW3m9ejXZy57aILokUsEQudDGPUObYrpmnreL2csQz1aqvxnHHbBnAtyxrrtZvXYe7eEnrcEhzkQ/640G1wOx5yqrWZvE3nqRPQeW0dG+JHS2i77UCHiKTx5Gr/nDvpiMY+oz3h/E3adnWL/Af/NHKe4rIRNtkFdXZ2yFcNJK9d6WrrtHGM09O7Op5xqlgjJdfnetXUVdMA2zC5Mep1bwnbbBKCbsKp1rRDt0f92pvkbB7QEalKG98jBfDUmfVdkeyQ34wnfJO8WunhZ8nRULyj4iu+anxbPh6V+wF2eceLBV54iAccADxYH/j7lLTFWJbFrQXiXXK2U09crpnjAG+dW+Svkk+rUcWwq1E9KV5/V6RSPJzEJpdy7sWYp3GwQIwjHAD8zlquPdDtadCk51YPkPb82Drl5AzM8qnk02pUXYpU22xWZrvq5ly0RH0xzBnVvta3RKxA5hm7A2Lc96L6npUrIsnJwFl1PONMGoeOUSKXRFbrUcuTKJGlHqEpMfZm5wvIhqiDWENi0V9l7PTWivqDurocmek9De1eHz2sjwOoOnSGkitqvmdrt7DeKi4zYlylm3JuPLIXko6gtsk6Ypd0jFlnpNthMZKLjDV0QLW5RoihVVWVJla12ZVnlV3t2rSPxNMUTzNinCknQ7J5fkEVMb7DScSR7xDgHW947aRgHvCCAwAfKRw84XAxh30ccRQx3Bd8wxueG8e+4h6Pjb89HAF8wW8AgCNCvOEdL4MJnxOAEzK84A+84RnPeMU7DjjhAQd8G/WaPdzhgNMlLNc13A+4wz2+4B4A8L4xr/AJ93A+KAGW4h7AA6upbafdYO366ItIvTChQN7aiLl/bFO7xS0hcUY+bawj9lUAA9H7qOp+FEG1tab+Y1sL9HztMc3XxskGr5+QHZCJ2L4aM20Dtp8MI8QyrpaEOuQ0T4yZZCJkIbKLIKvDbnPE2GWSiZBlqDp3p5pm9e1i7IzmDwghM6n6otTZbhW3i3HlfzPJRIhxAuGpOppm9a1ifNaonpuOS91OSLNHmV60es6OiuaTTLmVBa2EGMVvVMXpmdW3iPFSSaaQBSSEAGgYpXpm9XQx9rVz0tNwsd4+UIRslkwjqztVjF0UiySZqvPSMyakg45ZPU2MHbGNp2lxq4TY7m10CFkEHbN6mhhni3TairSLVbaJi6y3RiBBhhwFMmuvimwGtVk9RYyXSDL5jc1V7cQRO6Eklx4yCXIhvA5SsP0+mUmkFBB9MVZt8j4VF1Frf2Rbb/ZExOwDUQ6Ttq7EY6dQMhe1Wa0rxoEhUXPgIbwsF20+bL3V60q2SCyAbSfiHMOT34L8Mv8UZBEK/BBL+uetQvIu+1BOyxYfAFQNhA+4w1HS2PfV0iUWAX6IbHfVvPmlI7J3jX8JuRGVWa2jjd3Wxi7LPKzQVwOcLyPPB2vMAzZVIPNR9fVUi7EzYACbf9iaMQ7EyJ2Roph4kdpz8umQR6vVYhx9gBDbnzEORmyK3OIYPNkQcrPa5B5On5lkMEznatTREaJBbVYPG3YUYzNU1W1dIqvLWsimkJnVFGMTOCPfsFUm9c/zT0EWpEoW3bIBHNGj2oKvu4Vg1U293uLP2Xqgi2K8bb6L/zd+G1nM8Db21dYLtRj/LjYmIORG8tEaLBrVJigGPeMczU2DNr85DrXx1qk0gs7uy2Q6Lg7o62IHR1wtoRDP7G1C5jEeraY2nk84Uql1fdbZvi6mNt4+OV4BMMy1DJVn/Efv+ScARwA+XvBIXUzmU6+/6UJtPJ8QxUhNeIwcBVJrV2+RjTFmVk8X43Bgn+hb8Rg9J2QKw9HqqWLsGyz193ZRTb0b6BvbQBUznRutfsUrvhsQPg8JXtb+SgixjWGzeg3fOEOJHAkyauMtQW1sA9uJVp/wE1yE1MbbgmJsB2bM6vkU809BzEMxtoMqs3lg+oMMQTG2g0yY1Y83He0hRY6cKaK9ws6YtvAdvwF4uKHFm4snPCBHjj9x1zKLdXPINKUJMYLX6wmtG6mudx4uOrFu/Z6ZfS2eMFK9JaiNbSHDK44AHifq40D0kfZwAFoR5i84ap6DAkuIIeJObbWeNg6FLjWrP6mNCbmJrlk9rfzDbIM4ivGmYKTaHjK8A7gtexwCeGe71r1C39gmnvAbgMcbtlt5xLWz1BVGqgn5cNpmtb5R7Tb2KLquk9KPVPeNcRrVm4La2CYyvOMA4GHiHoaPgNg90BX9LgBGqglZiWqjkqqpjL42TlGKbVHPhnYIPFMbbwmGuOyi8m+PE2urfwB4BRABhrYrvwfwZfut5gjZJpVHG2NawilBgXziRuXD59njnoqEfDBXs5ot9YiARrVt3GZWk11DMbaNOUUgZKdQjO2DuyySDhRj+6ga+hxxWHsgZCtQjO0jFWY1IQKKsY38Mf8UZE9QjG3k+/xTkD1BMbYRmtWkBcXYTmhWkwYUYzuhWU0aUIzthGY1aUAxthWa1eQCxdhWvgPUyITYTszlEaTi/1UpzJl8brcVAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDIyLTAyLTE3VDE1OjU4OjM1KzAwOjAwEjAMIgAAACV0RVh0ZGF0ZTptb2RpZnkAMjAyMi0wMi0xN1QxNTo1ODozNSswMDowMGNttJ4AAAAASUVORK5CYII=)"]}],"metadata":{"accelerator":"GPU","colab":{"collapsed_sections":[],"name":"Zadanie 3.ipynb","provenance":[]},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.12"}},"nbformat":4,"nbformat_minor":0}