CVI_PS_DL.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "CVI-PS-DL.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/iitmcvg/CVI-Project-apps/blob/master/CVI_PS_DL.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjbIgGuAOz5I"
      },
      "source": [
        "# Image Classification Using CNN\n",
        "#### By The Computer Vision and Intelligence Club (CVI) - IIT Madras\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-sh-27b_jXK8"
      },
      "source": [
        "This aim of this Problem Statement is to introduce Deep Learning, which is one of the most used techniques for Computer Vision Application. \n",
        "Before beginning go thorugh the following articles:\n",
        "1. https://www.nature.com/articles/nature14539 : It is fairly easy to follow and provides an excellent overview of the field. (You will need to login with smail to download the pdf.) \n",
        "\n",
        "2. http://deeplearning.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/ : A short introduction to Multi-layer perceptrons.\n",
        "\n",
        "3. https://cs231n.github.io/convolutional-networks/ : Introduction to Convolutional Neural Networks. CNNs are generally used for computer vision problems.\n",
        "\n",
        "\n",
        "This introduction should be sufficient to get you started with this problem statement. If after going through the PS, you guys are interested in further exploring the field, I would suggest the following resources:\n",
        "1. http://cs231n.stanford.edu/ : Online course by Stanford.\n",
        "2. http://introtodeeplearning.com/: Video Lectures from MIT\n",
        "3. https://www.deeplearningbook.org/: Most popular book on Deep Learning\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a5cK9DWljhoo"
      },
      "source": [
        "We'll build a CNN using Keras to use it to classify thousands of pictures in 6 different categories.\n",
        "\n",
        "Sounds like a huge task, but do not worry! Remember to use Google freely to search for any doubts, errors or documentation. This assignment is not only to check your DL skills, but also how you can step up to learn new things on the go using publicly available sources.\n",
        "\n",
        "Ideally at the end, we expect that you understand the code in EVERY cell\n",
        "\n",
        "Enter your code between areas surrounded by the following comment design\n",
        "######### YOUR CODE HERE #########\n",
        "\n",
        "\n",
        "################################\n",
        "\n",
        "[Link For The Dataset](https://drive.google.com/uc?id=1Qc66kVqetwJIK7cKXnXxbPJy6gnpRSRI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gyr9xCP9jyne"
      },
      "source": [
        "The aim of this get you all familiarized with Deep Learning in TensorFlow, a very popular Deep Learning Library (or in general GPU computation library).\n",
        "\n",
        "Some of the dataloading and preprocessing work has been done, and you guys are expected to fill-in code where you are asked to. \n",
        "\n",
        "Before diving into the code ensure that you copy the notebook to your drive (See the option in File Tab) and that the Runtime Type is set to GPU (Runtime tab -> Change runtime type). To see the importance of GPU in deep learning see [this](https://course.fast.ai/gpu_tutorial.html) short article.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "clXXy5KnAjXB"
      },
      "source": [
        "\n",
        "\n",
        "First we import the necessary libraries. We recommend that you check out what these libraries are if they seem unfamiliar"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "koTFeclBOz5M"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import seaborn as sns\n",
        "sns.set(style=\"whitegrid\")\n",
        "import os\n",
        "import glob as gb\n",
        "import cv2\n",
        "import tensorflow as tf\n",
        "import keras"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t5GqcwrySwkP"
      },
      "source": [
        "# Downloading the Data and Extracting it"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EdEwBoAkPjh6"
      },
      "source": [
        "!gdown https://drive.google.com/uc?id=1Qc66kVqetwJIK7cKXnXxbPJy6gnpRSRI\n",
        "!unzip The_Data.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KS7eV5L3Oz5O"
      },
      "source": [
        "Now define the paths to the train test and pred folders"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Tm9b0ZlDOz5P"
      },
      "source": [
        "######### YOUR CODE HERE #########\n",
        "trainpath = './RandomFolder/RandomSubFolder/TheFolderIWant'\n",
        "testpath = ''\n",
        "predpath = ''\n",
        "\n",
        "##################################"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_YCU8RoOz5Q"
      },
      "source": [
        "# Data Loading\n",
        "\n",
        "\n",
        "Now let's first check the Train folder to have a look to its content"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "Z8aHEePNOz5R"
      },
      "source": [
        "for folder in  os.listdir(trainpath + 'seg_train') : \n",
        "    files = gb.glob(pathname= str( trainpath +'seg_train//' + folder + '/*.jpg'))\n",
        "    print(f'For training data , found {len(files)} in folder {folder}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72M3TdcqOz5S"
      },
      "source": [
        "Ok, how about the test folder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "lSgCHYJcOz5S"
      },
      "source": [
        "for folder in  os.listdir(testpath +'seg_test') : \n",
        "    files = gb.glob(pathname= str( testpath +'seg_test//' + folder + '/*.jpg'))\n",
        "    print(f'For testing data , found {len(files)} in folder {folder}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qokkhkCsOz5T"
      },
      "source": [
        "_____\n",
        "Now for prediction folder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "BcjoyFzJOz5U"
      },
      "source": [
        "files = gb.glob(pathname= str(predpath +'seg_pred/*.jpg'))\n",
        "print(f'For Prediction data , found {len(files)}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMMhtR59Oz5V"
      },
      "source": [
        "_____\n",
        "\n",
        "# Checking Images\n",
        "\n",
        "Now we need to check the images sizes , to know how they look like\n",
        "\n",
        "Since we have 6 categories , we first need to create a dictionary with their names & indices. This is known as integer encoding. Also create a function to get the code back"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "_9H1Ds8pOz5W"
      },
      "source": [
        "code_to_num = {'buildings':0 ,'forest':1, 'glacier':2, 'mountain':3, 'sea':4, 'street':5}\n",
        "num_to_code = {0:'buildings' ,1:'forest', 2:'glacier', 3:'mountain', 4:'sea', 5:'street'}\n",
        "\n",
        "def get_code(n) : \n",
        "    if n in num_to_code:\n",
        "        return num_to_code[n]    \n",
        "\n",
        "def get_num(c):\n",
        "    if c in code_to_num:\n",
        "        return code_to_num[c] "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SLFrYewDOz5X"
      },
      "source": [
        "Now how about the images sizes in train folder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "gfj3TlLQOz5X"
      },
      "source": [
        "size = []\n",
        "for folder in  os.listdir(trainpath +'seg_train') : \n",
        "    files = gb.glob(pathname= str( trainpath +'seg_train//' + folder + '/*.jpg'))\n",
        "    for file in files: \n",
        "        image = plt.imread(file)\n",
        "        size.append(image.shape)\n",
        "pd.Series(size).value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "swsdbbnkOz5Y"
      },
      "source": [
        "______\n",
        "\n",
        "Ok, almost all of them are (150,150,3), how about test images ? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "WY5VwvEPOz5Z"
      },
      "source": [
        "size = []\n",
        "for folder in  os.listdir(testpath +'seg_test') : \n",
        "    files = gb.glob(pathname= str( testpath +'seg_test//' + folder + '/*.jpg'))\n",
        "    for file in files: \n",
        "        image = plt.imread(file)\n",
        "        size.append(image.shape)\n",
        "pd.Series(size).value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PmGxfbuPOz5a"
      },
      "source": [
        "Almost same ratios  \n",
        "Now to prediction images "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "mUPmrwtZOz5a"
      },
      "source": [
        "size = []\n",
        "files = gb.glob(pathname= str(predpath +'seg_pred/*.jpg'))\n",
        "for file in files: \n",
        "    image = plt.imread(file)\n",
        "    size.append(image.shape)\n",
        "pd.Series(size).value_counts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nAWdiCW7Oz5b"
      },
      "source": [
        "Ok , since almost all of pictures are (150,150,3) , we can use all pictures in our model, after resizing it to a particular size"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JfMstFqyOz5c"
      },
      "source": [
        "# Reading Images\n",
        "\n",
        "Now it's time to read all images & convert it into arrays\n",
        "\n",
        "First we'll create a variable s , which refer to size , so we can change it easily \n",
        "\n",
        "Let's use now size = 100 , so it will be suitable amount to contain accuracy without losing so much time in training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "c0B2kG3jOz5d"
      },
      "source": [
        "s = 100"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bvTebpaJOz5e"
      },
      "source": [
        "Now to read all pictues in six categories in training folder, and use OpenCV to resize it. And not to forget assigning the y value from the predefined function "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "OUQ37lzjOz5e"
      },
      "source": [
        "X_train = []\n",
        "y_train = []\n",
        "for folder in  os.listdir(trainpath +'seg_train') : \n",
        "    files = gb.glob(pathname= str( trainpath +'seg_train//' + folder + '/*.jpg'))\n",
        "    for file in files: \n",
        "        image = cv2.imread(file)\n",
        "        image_array = cv2.resize(image , (s,s))\n",
        "        X_train.append(list(image_array))\n",
        "        y_train.append(get_num(folder))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tP-KMcOqOz5f"
      },
      "source": [
        "Great , now how many items in X_train "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "g2EXBm6xOz5g"
      },
      "source": [
        "print(f'we have {len(X_train)} items in X_train')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiM_OtVZOz5h"
      },
      "source": [
        "Also we have have a look to random pictures in X_train , and to adjust their title using the y value"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "jmP7O6-ZOz5h"
      },
      "source": [
        "plt.figure(figsize=(20,20))\n",
        "for n , i in enumerate(list(np.random.randint(0,len(X_train),36))) : \n",
        "    plt.subplot(6,6,n+1)\n",
        "    plt.imshow(X_train[i])   \n",
        "    plt.axis('off')\n",
        "    plt.title(get_code(y_train[i]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VLuUP5V_Oz5i"
      },
      "source": [
        "Great , now to repeat same steps exactly in test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "7ZWqIDeQOz5j"
      },
      "source": [
        "X_test = []\n",
        "y_test = []\n",
        "for folder in  os.listdir(testpath +'seg_test') : \n",
        "    files = gb.glob(pathname= str(testpath + 'seg_test//' + folder + '/*.jpg'))\n",
        "    for file in files: \n",
        "        image = cv2.imread(file)\n",
        "        image_array = cv2.resize(image , (s,s))\n",
        "        X_test.append(list(image_array))\n",
        "        y_test.append(get_num(folder))\n",
        "        "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "DDJ8B8UUOz5k"
      },
      "source": [
        "print(f'we have {len(X_test)} items in X_test')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "zZ5g8l6dOz5l"
      },
      "source": [
        "plt.figure(figsize=(20,20))\n",
        "for n , i in enumerate(list(np.random.randint(0,len(X_test),36))) : \n",
        "    plt.subplot(6,6,n+1)\n",
        "    plt.imshow(X_test[i])    \n",
        "    plt.axis('off')\n",
        "    plt.title(get_code(y_test[i]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XKaTakNMOz5l"
      },
      "source": [
        "Also with Prediction data , without having title ofcourse"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "fPqTXjPJOz5m"
      },
      "source": [
        "X_pred = []\n",
        "files = gb.glob(pathname= str(predpath + 'seg_pred/*.jpg'))\n",
        "for file in files: \n",
        "    image = cv2.imread(file)\n",
        "    image_array = cv2.resize(image , (s,s))\n",
        "    X_pred.append(list(image_array))       "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "MwpbsMcZOz5n"
      },
      "source": [
        "print(f'we have {len(X_pred)} items in X_pred')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "K17f6tciOz5n"
      },
      "source": [
        "plt.figure(figsize=(20,20))\n",
        "for n , i in enumerate(list(np.random.randint(0,len(X_pred),36))) : \n",
        "    plt.subplot(6,6,n+1)\n",
        "    plt.imshow(X_pred[i])    \n",
        "    plt.axis('off')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ddzrCyb6Oz5o"
      },
      "source": [
        "________\n",
        "\n",
        "# Building The Model \n",
        "\n",
        "Now we need to build the model to train our data\n",
        "\n",
        "First we convert the data into arrays using numpy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "ok8jjDkkOz5o"
      },
      "source": [
        "X_train = np.array(X_train)\n",
        "X_test = np.array(X_test)\n",
        "X_pred_array = np.array(X_pred)\n",
        "y_train = np.array(y_train)\n",
        "y_test = np.array(y_test)\n",
        "\n",
        "print(f'X_train shape  is {X_train.shape}')\n",
        "print(f'X_test shape  is {X_test.shape}')\n",
        "print(f'X_pred shape  is {X_pred_array.shape}')\n",
        "print(f'y_train shape  is {y_train.shape}')\n",
        "print(f'y_test shape  is {y_test.shape}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JG1kv36fOz5p"
      },
      "source": [
        "Now to build the CNN model by Keras , using Conv2D layers , MaxPooling & Dropouts and Dense layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "agD8NVLsOz5q"
      },
      "source": [
        "############### YOUR CODE HERE  ####################\n",
        "KerasModel = keras.models.Sequential(\n",
        "    \n",
        "\n",
        "\n",
        "\n",
        "\n",
        ")\n",
        "####################################################"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "akeNayXNOz5r"
      },
      "source": [
        "Now to compile the model , using adam optimizer , & sparse categorical crossentropy loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "SyIt_BFfOz5r"
      },
      "source": [
        "############### YOUR CODE HERE  ####################\n",
        "KerasModel.compile(''' HERE ''')\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "####################################################"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ru7oeFpjOz5s"
      },
      "source": [
        "So how the model looks like ? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "8-jFr_rLOz5s"
      },
      "source": [
        "print('Model Details are : ')\n",
        "print(KerasModel.summary())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mmWJUrYLOz5t"
      },
      "source": [
        "Now to train the model , lets use 50 epochs now"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "aaPiCt74Oz5t"
      },
      "source": [
        "############### YOUR CODE HERE  ####################\n",
        "ThisModel = KerasModel.fit()\n",
        "\n",
        "####################################################"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CIRWfNXMOz5u"
      },
      "source": [
        "How is the final loss & accuracy?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "0Ktc2gK2Oz5u"
      },
      "source": [
        "ModelLoss, ModelAccuracy = KerasModel.evaluate(X_test, y_test)\n",
        "\n",
        "print('Test Loss is {}'.format(ModelLoss))\n",
        "print('Test Accuracy is {}'.format(ModelAccuracy ))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Au_UPNv8TzzG"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XbIAOg4iOz5w"
      },
      "source": [
        "\n",
        "_______\n",
        "\n",
        "Now to predict X_test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "TBbFk7vrOz5w"
      },
      "source": [
        "y_pred = KerasModel.predict(X_test)\n",
        "\n",
        "print('Prediction Shape is {}'.format(y_pred.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3FHZ9lrOz5x"
      },
      "source": [
        "\n",
        "Now it's time to predict X_Predict"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "uCJihtl5Oz5x"
      },
      "source": [
        "y_result = KerasModel.predict(X_pred_array)\n",
        "\n",
        "print('Prediction Shape is {}'.format(y_result.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qB2o63d6Oz5y"
      },
      "source": [
        "And show random redicted pictures & its predicting category\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "H6y9PL4xOz5y"
      },
      "source": [
        "plt.figure(figsize=(20,20))\n",
        "for n , i in enumerate(list(np.random.randint(0,len(X_pred),36))) : \n",
        "    plt.subplot(6,6,n+1)\n",
        "    plt.imshow(X_pred[i])    \n",
        "    plt.axis('off')\n",
        "    plt.title(get_code(np.argmax(y_result[i])))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FU3fSMkabPRb"
      },
      "source": [
        "#### BONUS POINTS FOR SOME TRANSFER LEARNING APPROACH"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjxWBxHEkU1j"
      },
      "source": [
        "[Basics Of Transfer Learning](https://machinelearningmastery.com/transfer-learning-for-deep-learning/)"
      ]
    }
  ]
}