RL_PS_22_23.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RL_PS_22_23.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "GLwKqTgLUp6V",
        "M5YTQ0VwUg_u",
        "ACyANvFs4r0j",
        "crI79wCF0DbV",
        "PNUAV99MK00U"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/abhipraay/CVI_Projects_PS/blob/main/RL_PS_22_23.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# `Hit the Gym: MiniProject by CVI`\n",
        "\n",
        "The exercises in this notebook are meant for CVI aspirants who wish to work on the RL Games CPP. \n",
        "\n",
        "**About the project:** The goal of the RL Games project is to explore and analyse current Reinforcement Learning Methods. These techniques will then be used to help F1 teams optimize their race strategies.\n",
        "\n",
        "In this notebook, you'll use a popular Reinforcement Learning Technique called Q-learning to train an agent to play simple games using the python library OpenAI Gym. \n",
        "\n",
        "You may refer to the following while coding:\n",
        "\n",
        "Python reference: https://bit.ly/3ajUalZ and https://bit.ly/2UgiZKa\n",
        "\n",
        "OpenAI Gym Documentation: https://gym.openai.com/docs/\n",
        "\n",
        "\n",
        "### `RL Basics:`\n",
        "Reinforcement learning is an approach used in Machine Learning where the agent is allowed to interact with the system to learn its behaviour and come up with an optimal startegy to achieve an objective. The agent models the problem as a probabilistic state machine (a graph where the transition from one node to another has a probability distribution). Nodes in the model graph are called states and a transition from one state to another is called an action. Each state transition (which is a (state, action) pair) has a corresponding reward or penalty. The goal of the RL agent is to maximise the reward. \n",
        "\n",
        "Training an RL agent from scratch requires us to model the state space and the action space. In addition, we must also come up with suitable rewards for each state transition. The RL agent estimates this reward structure and executes actions so as to maximise them. The final performance of the RL agent is heavily dependent on how the system is modelled. Luckily for us, we **do not need to get into the mathematics** of Reinforcement Learning right now, thanks to the Python library Gym.\n",
        "\n",
        "\n",
        "Gym offers many in-built RL environments which you can use to play around with. These environments are Python classes with their state spaces, action spaces and rewards pre-defined. You will use two such environments (Taxi-v3 and Maze) to train an agent accomplish a goal. You can find the documentation for these environments here:\n",
        "\n",
        "Taxi-v3 Documentation: https://gym.openai.com/envs/Taxi-v3/  \n",
        "\n",
        "Maze: Custom environment similar to GYM environments   \n",
        "\n",
        "To create a gym environment of 'Taxi-v3' you do this:"
      ],
      "metadata": {
        "id": "IT87J6j_SwIY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import gym\n",
        "import numpy as np  \n",
        "# Create an environment of Taxi-v3:\n",
        "env = gym.make('Taxi-v3').env \n",
        "env.render()\n",
        "print(env.s)\n",
        "print(env.observation_space)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aCZOJ4_3TNhX",
        "outputId": "d95e30f6-3fde-4329-cc95-027da129f518"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+\n",
            "|\u001b[34;1mR\u001b[0m: | : :G|\n",
            "| : | : :\u001b[43m \u001b[0m|\n",
            "| : : : : |\n",
            "| | : | : |\n",
            "|Y| : |\u001b[35mB\u001b[0m: |\n",
            "+---------+\n",
            "\n",
            "183\n",
            "Discrete(500)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "env.s is the current state of the environment.\n",
        "\n",
        "env.observation_space() returns the type and size of the observation or state space. Discrete implies that the states are discrete and not continuous. For games like Pacman, Gym uses another type of state space- Box- due to the large number of states. \n",
        "\n",
        "**Solving the Problem:** Now the goal of this game is to train the taxi to efficiently pick the passenger from the blue spot and drop them at the purple spot. The taxi can do the following actions: Move Up, Move Down, Move to the Left, Move to the Right, Pickup, Dropoff. The reward structure is such:\n",
        "1. -10 points for illegal dropoff/pickup actions\n",
        "2. +20 points if the passenger is dropped off at the correct location\n",
        "3. -1 point for every other action\n",
        "\n",
        "A paleolithic approach to this problem would be to pick an action at random and execute it. Eventually the passenger would get picked up and then dropped off at the correct location."
      ],
      "metadata": {
        "id": "HOPwjhIjTW20"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "state = env.reset()\n",
        "epochs = 0\n",
        "penalty, reward = 0, 0  # Penalty records the number of times the agent hits a wall\n",
        "frames = []\n",
        "done = # Find out the role of 'done' and complete the statement for its initial condition \n",
        "while #insert condition:\n",
        "    '''\n",
        "      Enter your code here\n",
        "      The code must pick an action from the action space at random, execute it and update 'penalty' accordingly\n",
        "    '''\n",
        "    frames.append({'frame': env.render(mode='ansi' ), 'state': state, 'action': action, 'reward': reward})\n",
        "    epochs += 1\n",
        "print(\"Timesteps taken: {}\".format(epochs))\n",
        "print(\"Penalties incurred: {}\".format(penalty))"
      ],
      "metadata": {
        "id": "pFfOyUuBTYnO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the following cell to visualise the performance of the agent. It won't come as a surprise to see that the approach is quite bad. It is so because, the agent has no memory of the past and hence learns nothing. "
      ],
      "metadata": {
        "id": "sNXW5ZGrTtOG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import clear_output\n",
        "from time import sleep\n",
        "def print_frames(frames):\n",
        "    for i, frame in enumerate(frames):\n",
        "        clear_output(wait = True)\n",
        "        print(frame['frame'])\n",
        "        print(f\"Timestep: {i+1}\")\n",
        "        print(f\"State: {frame['state']}\")\n",
        "        print(f\"Action: {frame['action']}\")\n",
        "        print(f\"Reward: {frame['reward']}\")\n",
        "        sleep(0.1)\n",
        "print_frames(frames)"
      ],
      "metadata": {
        "id": "FUKDckQ1Tvw9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# `Enter Q-learning`\n",
        "\n",
        "A popular technique used in Reinforcement Learning is to have the agent maintain an estimate of the rewards that it would gain from executing a particular state transition (called the Q-table) which is updated after each step based on the reward obtained. The agent picks the action that yields the maximum reward at that particular state. That is, if a particular state transition ((state, action) pair) resulted in a good reward, the agent is better off repeating that action whenever that state is attained. \n",
        "\n"
      ],
      "metadata": {
        "id": "D9lni7i9T9s2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##**TASK 1:**\n",
        "\n",
        "Find out the update rule for Q-learning and implement Q-learning for the Taxi-v3 problem. Also, find out what 'exploration versus exploitation' is and use a suitable way to optimise on exploration and exploitation.\n",
        "\n",
        "You would need to set a few hyperparameters- learning rate ($\\alpha$), reward decay rate ($\\gamma$), number of episodes and exploration probability($\\epsilon$). Obtain the performance characteristics of the agent (that is, number of epochs per episode and average number of penalties per episode) for ($\\alpha$, $\\gamma$, episodes, $\\epsilon$) = (0.6, 0.9, 1000, 0.4)"
      ],
      "metadata": {
        "id": "GLwKqTgLUp6V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''\n",
        "  Q-LEARNING ON Taxi-v3:\n",
        "  Enter Your Code Here\n",
        "'''"
      ],
      "metadata": {
        "id": "qhnGtvbrUepG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##TASK 2:\n",
        "Now that you have trained the agent on Taxi-v3, try Q-learning on the Maze environment. After training, obtain the following performance characteristics- number of epochs per episode and average number of wins. What can you do to improve the performance of the agent?"
      ],
      "metadata": {
        "id": "M5YTQ0VwUg_u"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-bREpf4vqiOD"
      },
      "source": [
        "###Maze\n",
        "###This game is described using a grid like:\n",
        ">   _ _ _ H _ _ _ H _ _<br>   _ _ _ _ _ H _ H _ _<br>   _ _ H _ H _ _ _ _ H<br>   H _ _ _ _ _ H _ _ _<br>   _ H _ H H _ _ _ H _<br>   _ _ _ H _ _ _ H _ _<br>   _ _ _ _ _ H H _ _ _<br>   _ _ H _ _ H _ _ _ H <br>   H _ _ _ _ _ H _ _ _ <br>   _ _ _ H _ _ _ H _ G\n",
        "<br><br> _ : Safe path\n",
        "<br> H : Hole, avoid falling\n",
        "<br> G: Goal, target to reach\n",
        "###Your goal is to reach G and receive reward 1.\n",
        "###The episode ends when you reach G or fall in H.\n",
        "\n",
        "###You receive a reward of 1 if you reach G, 0 otherwise.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Do Not Edit This Code"
      ],
      "metadata": {
        "id": "ACyANvFs4r0j"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fNbZB_Tnhdss"
      },
      "source": [
        "import random\n",
        "import sys\n",
        "import copy\n",
        "class maze:\n",
        "  class action:\n",
        "    def __init__(self):\n",
        "      self.total_actions = 4\n",
        "      self.__out=sys.stdout\n",
        "    \n",
        "    def random_action(self):\n",
        "      act = random.randint(1,4)\n",
        "      return act\n",
        "\n",
        "    def show_actions(self):\n",
        "      actions= \"1->Up, 2->Right, 3->Down, 4->Left\"\n",
        "      self.__out.write(actions)\n",
        "    \n",
        "  class observation:\n",
        "    def __init__(self):\n",
        "      self.total_observations = 100\n",
        "      self.dtype = type(self.total_observations)\n",
        "\n",
        "    def random(self):\n",
        "      obs = random.randint(1,99)\n",
        "      return obs\n",
        "  def __init__(self):\n",
        "    self.observation_space = self.observation()\n",
        "    self.__map=['___H___H__',\n",
        "       '_____H_H__',\n",
        "       '__H_H____H',\n",
        "       'H_____H___',\n",
        "       '_H_HH___H_',\n",
        "       '___H___H__',\n",
        "       '_____HH___',\n",
        "       '__H__H___H',\n",
        "       'H_____H___',\n",
        "       '___H___H_G']\n",
        "    self.action_space = self.action()\n",
        "    self.__x = None\n",
        "    self.__y = None\n",
        "    self.__state = None\n",
        "    self.__out = sys.stdout\n",
        "    self.__action = None\n",
        "    self.__action_dict = {1:'Up',2:'Right',3:'Down',4:'Left'}\n",
        "    self.__done = False\n",
        "\n",
        "  def reset(self):\n",
        "    self.__y = random.randint(0,9)\n",
        "    while True:\n",
        "      self.__x = random.randint(0,9)\n",
        "      if self.__map[self.__y][self.__x]=='_':\n",
        "        break\n",
        "    self.current_state()\n",
        "    self.__action = None\n",
        "    self.__done = False\n",
        "    return self.__state\n",
        "  \n",
        "  def current_state(self):\n",
        "    if self.__y is not None:\n",
        "     self.__state = self.__y*10+self.__x+1\n",
        "    return self.__state\n",
        "\n",
        "  def take_step(self,action):\n",
        "    if self.__done == False :\n",
        "      if action == 1:\n",
        "        if self.__y-1>=0:\n",
        "          self.__y-=1\n",
        "        self.__action = action\n",
        "      elif action == 3:\n",
        "        if self.__y+1<=9:\n",
        "          self.__y+=1\n",
        "        self.__action = action\n",
        "      elif action == 2:\n",
        "        if self.__x+1<=9:\n",
        "          self.__x+=1\n",
        "        self.__action = action\n",
        "      elif action == 4:\n",
        "        if self.__x-1>=0:\n",
        "          self.__x-=1\n",
        "        self.__action = action\n",
        "      else :\n",
        "        self.__out.write(\"Enter a valid action.\")\n",
        "        return\n",
        "      self.current_state()\n",
        "      reward = 0.0\n",
        "      if self.__map[self.__y][self.__x]=='G':\n",
        "        reward=1.0\n",
        "        self.__done= True\n",
        "      if self.__map[self.__y][self.__x]=='H':\n",
        "        self.__done = True\n",
        "      return self.__state,reward,self.__done\n",
        "    else :\n",
        "      self.__out.write(\"\\n\\033[38;5;11mWARN: You are calling 'step()' even though this environment has already returned done = True. You should always call 'reset()' once you receive 'done = True\\033[0;0m\")\n",
        "\n",
        "  def show(self):\n",
        "    map = copy.deepcopy(self.__map)\n",
        "    val = map[self.__y][self.__x]\n",
        "    map[self.__y] = map[self.__y][:self.__x] + 'P' +map[self.__y][self.__x+1:]\n",
        "    map = self.__add_colour_h(map)\n",
        "    map[-1]=map[-1].replace('G',\"\\033[38;5;12mG\\033[0;0m\")\n",
        "    map[self.__y] = map[self.__y].replace('P',f'\\033[48;5;9m{val}\\033[0;0m')\n",
        "    if self.__action is not None:\n",
        "      self.__out.write('\\n'+self.__action_dict[self.__action])\n",
        "    for i in map:\n",
        "      self.__out.write('\\n'+i)\n",
        "    if val =='H':\n",
        "      self.__out.write(\"\\nTRY AGAIN.......You fell in hole!!!\")\n",
        "    if val =='G':\n",
        "      self.__out.write(\"\\nGG!!\")\n",
        "    self.__out.write(\"\\n\")\n",
        "\n",
        "  def __add_colour_h(self,map):\n",
        "    for i in range(len(map)):\n",
        "      map[i]=map[i].replace('H','\\033[48;5;16mH\\033[0;0m')\n",
        "    return map\n",
        "\n",
        "  def set_state(self,state):\n",
        "    if state>100 or state<1:\n",
        "      self.__out.write(\"Enter a valid state.\")\n",
        "      return\n",
        "    self.__state = state\n",
        "    self.__y = (state-1)//10\n",
        "    self.__x = (state-1)%10\n",
        "    if self.__map[self.__y][self.__x]=='_':\n",
        "      self.__done = False\n",
        "    else: \n",
        "      self.__done = True\n",
        "    self.__action = None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "crI79wCF0DbV"
      },
      "source": [
        "###Environment methods and attributes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YyHIbaKN0aVm"
      },
      "source": [
        "env = maze() #Creating object of maze class"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_HRbOXfB1XL0",
        "outputId": "4b60bab2-146c-4ea5-e951-560cec9cb39e"
      },
      "source": [
        "print(env.observation_space.total_observations) #Total observations in observation space\n",
        "print(env.observation_space.random()) # Random observation from observation space"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100\n",
            "40\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "daUf-uSP1lZI",
        "outputId": "8d89757d-a93b-4c04-a253-a797aa715c6e"
      },
      "source": [
        "print(env.action_space.total_actions) #Total actions in action space\n",
        "print(env.action_space.random_action()) #Returns random action from action space\n",
        "env.action_space.show_actions() #Prints details about actions in action space"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4\n",
            "1\n",
            "1->Up, 2->Right, 3->Down, 4->Left"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oqLUzZQJ5XDC",
        "outputId": "2afc670c-af43-4caa-c01a-59e8380390ef"
      },
      "source": [
        "print(env.current_state()) #No state is initailized"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pnrXwQIW7pNM",
        "outputId": "a28f69e2-31fd-420c-fd19-6a5be7300edb"
      },
      "source": [
        "env.reset() #initilizes game to a random state\n",
        "env.show() #prints observation\n",
        "print(env.current_state())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m__\n",
            "_____\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m__\n",
            "__\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m____\u001b[48;5;16mH\u001b[0;0m\n",
            "\u001b[48;5;16mH\u001b[0;0m_____\u001b[48;5;16mH\u001b[0;0m___\n",
            "_\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m_\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m__\n",
            "_____\u001b[48;5;16mH\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m___\n",
            "__\u001b[48;5;16mH\u001b[0;0m__\u001b[48;5;16mH\u001b[0;0m__\u001b[48;5;9m_\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m\n",
            "\u001b[48;5;16mH\u001b[0;0m_____\u001b[48;5;16mH\u001b[0;0m___\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m_\u001b[38;5;12mG\u001b[0;0m\n",
            "79\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lkvDavCy8o1Z",
        "outputId": "b849b89d-56fb-40fb-f260-97148739f7d7"
      },
      "source": [
        "env.set_state(90) #state of environment is changed to state specified\n",
        "env.show() \n",
        "print(env.current_state())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m__\n",
            "_____\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m__\n",
            "__\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m____\u001b[48;5;16mH\u001b[0;0m\n",
            "\u001b[48;5;16mH\u001b[0;0m_____\u001b[48;5;16mH\u001b[0;0m___\n",
            "_\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m_\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m__\n",
            "_____\u001b[48;5;16mH\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m___\n",
            "__\u001b[48;5;16mH\u001b[0;0m__\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m\n",
            "\u001b[48;5;16mH\u001b[0;0m_____\u001b[48;5;16mH\u001b[0;0m__\u001b[48;5;9m_\u001b[0;0m\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m_\u001b[38;5;12mG\u001b[0;0m\n",
            "90\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PNUAV99MK00U"
      },
      "source": [
        "###env.take_step( ) returns THREE values only, state, reward and done (episode completed or not) "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FXNd77KOEdC4",
        "outputId": "be523867-12b1-4284-d7c3-d508fbe88822"
      },
      "source": [
        "from IPython.display import clear_output\n",
        "from time import sleep\n",
        "env.reset()\n",
        "done = False\n",
        "while True:\n",
        "  env.show()\n",
        "  clear_output(wait=True)\n",
        "  sleep(1)\n",
        "  if done: \n",
        "    break\n",
        "  action = env.action_space.random_action()\n",
        "  state,reward,done = env.take_step(action)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Left\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m__\n",
            "_____\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;9mH\u001b[0;0m__\n",
            "__\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m____\u001b[48;5;16mH\u001b[0;0m\n",
            "\u001b[48;5;16mH\u001b[0;0m_____\u001b[48;5;16mH\u001b[0;0m___\n",
            "_\u001b[48;5;16mH\u001b[0;0m_\u001b[48;5;16mH\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m_\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m__\n",
            "_____\u001b[48;5;16mH\u001b[0;0m\u001b[48;5;16mH\u001b[0;0m___\n",
            "__\u001b[48;5;16mH\u001b[0;0m__\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m\n",
            "\u001b[48;5;16mH\u001b[0;0m_____\u001b[48;5;16mH\u001b[0;0m___\n",
            "___\u001b[48;5;16mH\u001b[0;0m___\u001b[48;5;16mH\u001b[0;0m_\u001b[38;5;12mG\u001b[0;0m\n",
            "TRY AGAIN.......You fell in hole!!!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Now you are familiar with Maze environment. You have to implement q-learing on this custom environment. Remember you are not allowed to do any chnages in Maze class."
      ],
      "metadata": {
        "id": "5uJdncEe5Auj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''\n",
        "  Q-LEARNING ON Maze:\n",
        "  Enter Your Code Here\n",
        "'''"
      ],
      "metadata": {
        "id": "YH88-JjAVjI9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## `Further Motivation`\n",
        "\n",
        "In case you are curious about the mathematics of Reinforcement Learning, you check the following resources out:\n",
        "\n",
        "RL Lectures by Dr. David Silver: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html\n",
        "\n",
        "Medium Blog Post on RL techniques: https://medium.com/@jonathan_hui/rl-introduction-to-deep-reinforcement-learning-35c25e04c199\n",
        "\n",
        "Deep Neural Networks (useful for Deep RL): http://cs231n.github.io/"
      ],
      "metadata": {
        "id": "PwLndzEJWzLM"
      }
    }
  ]
}