CVI-RL PS.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RLGames_ps.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6NEcxwh0Ej7T"
      },
      "source": [
        "# `Hit the Gym: MiniProject for RL Games CPP`\n",
        "\n",
        "The exercises in this notebook are meant for CVI aspirants who wish to work on the RL Games CPP. \n",
        "\n",
        "**About the project:** The goal of the RL Games project is to explore and analyse current Reinforcement Learning Methods by deploying them on Atari games, Minesweeper, slither.io etc. Further, the techniques developed can be used to attack more complex problems such as Reconnaissance Blind Chess and Autonomous Driving.\n",
        "\n",
        "In this notebook, you'll use a popular Reinforcement Learning Technique called Q-learning to train an agent to play simple games using the python library OpenAI Gym. \n",
        "\n",
        "You may refer to the following while coding:\n",
        "\n",
        "Python reference: https://bit.ly/3ajUalZ and https://bit.ly/2UgiZKa\n",
        "\n",
        "OpenAI Gym Documentation: https://gym.openai.com/docs/\n",
        "\n",
        "\n",
        "### `RL Basics:`\n",
        "Reinforcement learning is an approach used in Machine Learning where the agent is allowed to interact with the system to learn its behaviour and come up with an optimal startegy to achieve an objective. The agent models the problem as a probabilistic state machine (a graph where the transition from one node to another has a probability distribution). Nodes in the model graph are called states and a transition from one state to another is called an action. Each state transition (which is a (state, action) pair) has a corresponding reward or penalty. The goal of the RL agent is to maximise the reward. \n",
        "\n",
        "Training an RL agent from scratch requires us to model the state space and the action space. In addition, we must also come up with suitable rewards for each state transition. The RL agent estimates this reward structure and executes actions so as to maximise them. The final performance of the RL agent is heavily dependent on how the system is modelled. Luckily for us, we **do not need to get into the mathematics** of Reinforcement Learning right now, thanks to the Python library Gym.\n",
        "\n",
        "\n",
        "Gym offers many in-built RL environments which you can use to play around with. These environments are Python classes with their state spaces, action spaces and rewards pre-defined. You will use two such environments (Taxi-v3 and FrozenLake8x8-v0) to train an agent accomplish a goal. You can find the documentation for these environments here:\n",
        "\n",
        "Taxi-v3 Documentation: https://gym.openai.com/envs/Taxi-v3/  \n",
        "\n",
        "FrozenLake Documentation: https://gym.openai.com/envs/FrozenLake8x8-v0/  \n",
        "\n",
        "To create a gym environment of 'Taxi-v3' you do this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aMtvRD03J_JO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        },
        "outputId": "8b4fcb9f-6e44-4231-ba73-8a90d0af657d"
      },
      "source": [
        "import gym\n",
        "import numpy as np  \n",
        "# Create an environment of Taxi-v3:\n",
        "env = gym.make('Taxi-v3').env \n",
        "env.render()\n",
        "print(env.s)\n",
        "print(env.observation_space)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+---------+\n",
            "|\u001b[34;1mR\u001b[0m: | : :\u001b[35mG\u001b[0m|\n",
            "| : |\u001b[43m \u001b[0m: : |\n",
            "| : : : : |\n",
            "| | : | : |\n",
            "|Y| : |B: |\n",
            "+---------+\n",
            "\n",
            "141\n",
            "Discrete(6)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WauFKBJeKos0"
      },
      "source": [
        "env.s is the current state of the environment.\n",
        "\n",
        "env.observation_space() returns the type and size of the observation or state space. Discrete implies that the states are discrete and not continuous. For games like Pacman, Gym uses another type of state space- Box- due to the large number of states. \n",
        "\n",
        "**Solving the Problem:** Now the goal of this game is to train the taxi to efficiently pick the passenger from the blue spot and drop them at the purple spot. The taxi can do the following actions: Move Up, Move Down, Move to the Left, Move to the Right, Pickup, Dropoff. The reward structure is such:\n",
        "1. -10 points for illegal dropoff/pickup actions\n",
        "2. +20 points if the passenger is dropped off at the correct location\n",
        "3. -1 point for every other action\n",
        "\n",
        "A paleolithic approach to this problem would be to pick an action at random and execute it. Eventually the passenger would get picked up and then dropped off at the correct location."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gPwn4saqKiQN"
      },
      "source": [
        "state = env.reset()\n",
        "epochs = 0\n",
        "penalty, reward = 0, 0  # Penalty records the number of times the agent hits a wall\n",
        "frames = []\n",
        "done = # Find out the role of 'done' and complete the statement for its initial condition \n",
        "while #insert condition:\n",
        "    '''\n",
        "      Enter your code here\n",
        "      The code must pick an action from the action space at random, execute it and update 'penalty' accordingly\n",
        "    '''\n",
        "    frames.append({'frame': env.render(mode='ansi'), 'state': state, 'action': action, 'reward': reward})\n",
        "    epochs += 1\n",
        "print(\"Timesteps taken: {}\".format(epochs))\n",
        "print(\"Penalties incurred: {}\".format(penalty))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Fwe9LT-8qn8"
      },
      "source": [
        "Run the following cell to visualise the performance of the agent. It won't come as a surprise to see that the approach is quite bad. It is so because, the agent has no memory of the past and hence learns nothing. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KgvVPJ9KQWJk"
      },
      "source": [
        "from IPython.display import clear_output\n",
        "from time import sleep\n",
        "def print_frames(frames):\n",
        "    for i, frame in enumerate(frames):\n",
        "        clear_output(wait = True)\n",
        "        print(frame['frame'])\n",
        "        print(f\"Timestep: {i+1}\")\n",
        "        print(f\"State: {frame['state']}\")\n",
        "        print(f\"Action: {frame['action']}\")\n",
        "        print(f\"Reward: {frame['reward']}\")\n",
        "        sleep(0.1)\n",
        "print_frames(frames)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aKcVQ8OV8-Lv"
      },
      "source": [
        "## `Enter Q-learning`\n",
        "\n",
        "A popular technique used in Reinforcement Learning is to have the agent maintain an estimate of the rewards that it would gain from executing a particular state transition (called the Q-table) which is updated after each step based on the reward obtained. The agent picks the action that yields the maximum reward at that particular state. That is, if a particular state transition ((state, action) pair) resulted in a good reward, the agent is better off repeating that action whenever that state is attained. \n",
        "\n",
        "**TASK 1:**\n",
        "\n",
        "Find out the update rule for Q-learning and implement Q-learning for the Taxi-v3 problem. Also, find out what 'exploration versus exploitation' is and use a suitable way to optimise on exploration and exploitation.\n",
        "\n",
        "You would need to set a few hyperparameters- learning rate ($\\alpha$), reward decay rate ($\\gamma$), number of episodes and exploration probability($\\epsilon$). Obtain the performance characteristics of the agent (that is, number of epochs per episode and average number of penalties per episode) for ($\\alpha$, $\\gamma$, episodes, $\\epsilon$) = (0.6, 0.9, 1000, 0.4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ryw4gXzjQ2iO"
      },
      "source": [
        "'''\n",
        "  Q-LEARNING ON Taxi-v3:\n",
        "  Enter Your Code Here\n",
        "'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sl1WvHdREsmv"
      },
      "source": [
        "**TASK 2:**\n",
        "\n",
        "Now that you have trained the agent on Taxi-v3, try Q-learning on the FrozenLake8x8-v0 environment. After training, obtain the following performance characteristics- number of epochs per episode and average number of wins. What can you do to improve the performance of the agent?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGTlFXDrFyME"
      },
      "source": [
        "'''\n",
        "  Q-LEARNING ON FrozenLake8x8-v0:\n",
        "  Enter Your Code Here\n",
        "'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LITRSgg9I33Z"
      },
      "source": [
        "## `Further Motivation`\n",
        "\n",
        "In case you are curious about the mathematics of Reinforcement Learning, you check the following resources out:\n",
        "\n",
        "RL Lectures by Dr. David Silver: http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html\n",
        "\n",
        "Medium Blog Post on RL techniques: https://medium.com/@jonathan_hui/rl-introduction-to-deep-reinforcement-learning-35c25e04c199\n",
        "\n",
        "Deep Neural Networks (useful for Deep RL): http://cs231n.github.io/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bjpYZfLKIOtJ"
      },
      "source": [
        "If you are facing any issue or difficulty, you may PM (on WhatsApp):\n",
        "\n",
        "Shreyas : 8318387585"
      ]
    }
  ]
}