02_linear_regression_pytorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "02-linear-regression.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kotalaraghava/Natural-language-processing/blob/master/02_linear_regression_pytorch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cEhwJq2SGdVx"
      },
      "source": [
        "# Jovian Commit Essentials\n",
        "# Please retain and execute this cell without modifying the contents for `jovian.commit` to work\n",
        "!pip install jovian --upgrade -q\n",
        "import jovian\n",
        "jovian.set_project('02-linear-regression')\n",
        "jovian.set_colab_id('1GPJuLzL_8At8i8wR0nmhjhuIeoxrmpuP')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cu_AojCPGdV3"
      },
      "source": [
        "## Gradient Descent and Linear Regression with PyTorch\n",
        "\n",
        "### Part 2 of \"Deep Learning with Pytorch: Zero to GANs\"\n",
        "\n",
        "This tutorial series is a hands-on beginner-friendly introduction to deep learning using [PyTorch](https://pytorch.org), an open-source neural networks library. These tutorials take a practical and coding-focused approach. The best way to learn the material is to execute the code and experiment with it yourself. Check out the full series here:\n",
        "\n",
        "1. [PyTorch Basics: Tensors & Gradients](https://jovian.ai/aakashns/01-pytorch-basics)\n",
        "2. [Gradient Descent & Linear Regression](https://jovian.ai/aakashns/02-linear-regression)\n",
        "3. [Working with Images & Logistic Regression](https://jovian.ai/aakashns/03-logistic-regression) \n",
        "4. [Training Deep Neural Networks on a GPU](https://jovian.ai/aakashns/04-feedforward-nn)\n",
        "5. [Image Classification using Convolutional Neural Networks](https://jovian.ai/aakashns/05-cifar10-cnn)\n",
        "6. [Data Augmentation, Regularization and ResNets](https://jovian.ai/aakashns/05b-cifar10-resnet)\n",
        "7. [Generating Images using Generative Adversarial Networks](https://jovian.ai/aakashns/06b-anime-dcgan/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmkq81fLGdV3"
      },
      "source": [
        "This tutorial covers the following topics:\n",
        "\n",
        "- Introduction to linear regression and gradient descent\n",
        "- Implementing a linear regression model using PyTorch tensors\n",
        "- Training a linear regression model using the gradient descent algorithm\n",
        "- Implementing gradient descent and linear regression using PyTorch built-in"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wo--LIXYGdV4"
      },
      "source": [
        "### How to run the code\n",
        "\n",
        "This tutorial is an executable [Jupyter notebook](https://jupyter.org) hosted on [Jovian](https://www.jovian.ai) (don't worry if these terms seem unfamiliar; we'll learn more about them soon). You can _run_ this tutorial and experiment with the code examples in a couple of ways: *using free online resources* (recommended) or *on your computer*.\n",
        "\n",
        "#### Option 1: Running using free online resources (1-click, recommended)\n",
        "\n",
        "The easiest way to start executing the code is to click the **Run** button at the top of this page and select **Run on Colab**. [Google Colab](https://colab.research.google.com) is a free online platform for running Jupyter notebooks using Google's cloud infrastructure. You can also select \"Run on Binder\" or \"Run on Kaggle\" if you face issues running the notebook on Google Colab.\n",
        "\n",
        "\n",
        "#### Option 2: Running on your computer locally\n",
        "\n",
        "To run the code on your computer locally, you'll need to set up [Python](https://www.python.org), download the notebook and install the required libraries. We recommend using the [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/) distribution of Python. Click the **Run** button at the top of this page, select the **Run Locally** option, and follow the instructions.\n",
        "\n",
        ">  **Jupyter Notebooks**: This tutorial is a [Jupyter notebook](https://jupyter.org) - a document made of _cells_. Each cell can contain code written in Python or explanations in plain English. You can execute code cells and view the results, e.g., numbers, messages, graphs, tables, files, etc. instantly within the notebook. Jupyter is a powerful platform for experimentation and analysis. Don't be afraid to mess around with the code & break things - you'll learn a lot by encountering and fixing errors. You can use the \"Kernel > Restart & Clear Output\" or \"Edit > Clear Outputs\" menu option to clear all outputs and start again from the top."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p1pLaQyHGdV4"
      },
      "source": [
        "Before we begin, we need to install the required libraries. The installation of PyTorch may differ based on your operating system / cloud environment. You can find detailed installation instructions here: https://pytorch.org ."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cV50WqucGdV4"
      },
      "source": [
        "# Uncomment and run the appropriate command for your operating system, if required\n",
        "\n",
        "# Linux / Binder\n",
        "# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "\n",
        "# Windows\n",
        "# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "\n",
        "# MacOS\n",
        "# !pip install numpy torch torchvision torchaudio"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "waWUiO6rGdV5"
      },
      "source": [
        "## Introduction to Linear Regression\n",
        "\n",
        "In this tutorial, we'll discuss one of the foundational algorithms in machine learning: *Linear regression*. We'll create a model that predicts crop yields for apples and oranges (*target variables*) by looking at the average temperature, rainfall, and humidity (*input variables or features*) in a region. Here's the training data:\n",
        "\n",
        "![linear-regression-training-data](https://i.imgur.com/6Ujttb4.png)\n",
        "\n",
        "In a linear regression model, each target variable is estimated to be a weighted sum of the input variables, offset by some constant, known as a bias :\n",
        "\n",
        "```\n",
        "yield_apple  = w11 * temp + w12 * rainfall + w13 * humidity + b1\n",
        "yield_orange = w21 * temp + w22 * rainfall + w23 * humidity + b2\n",
        "```\n",
        "\n",
        "Visually, it means that the yield of apples is a linear or planar function of temperature, rainfall and humidity:\n",
        "\n",
        "![linear-regression-graph](https://i.imgur.com/4DJ9f8X.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0eZxJHV4GdV5"
      },
      "source": [
        "The *learning* part of linear regression is to figure out a set of weights `w11, w12,... w23, b1 & b2` using the training data, to make accurate predictions for new data. The _learned_ weights will be used to predict the yields for apples and oranges in a new region using the average temperature, rainfall, and humidity for that region. \n",
        "\n",
        "We'll _train_ our model by adjusting the weights slightly many times to make better predictions, using an optimization technique called *gradient descent*. Let's begin by importing Numpy and PyTorch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1d1HuoNHGdV6"
      },
      "source": [
        "import numpy as np\n",
        "import torch"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIsGcjRtGdV6"
      },
      "source": [
        "## Training data\n",
        "\n",
        "We can represent the training data using two matrices: `inputs` and `targets`, each with one row per observation, and one column per variable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iteUoaOgGdV7"
      },
      "source": [
        "# Input (temp, rainfall, humidity)\n",
        "inputs = np.array([[73, 67, 43], \n",
        "                   [91, 88, 64], \n",
        "                   [87, 134, 58], \n",
        "                   [102, 43, 37], \n",
        "                   [69, 96, 70]], dtype='float32')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pcnTjeaXGdV7"
      },
      "source": [
        "# Targets (apples, oranges)\n",
        "targets = np.array([[56, 70], \n",
        "                    [81, 101], \n",
        "                    [119, 133], \n",
        "                    [22, 37], \n",
        "                    [103, 119]], dtype='float32')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GhhgLHPNGdV7"
      },
      "source": [
        "We've separated the input and target variables because we'll operate on them separately. Also, we've created numpy arrays, because this is typically how you would work with training data: read some CSV files as numpy arrays, do some processing, and then convert them to PyTorch tensors.\n",
        "\n",
        "Let's convert the arrays to PyTorch tensors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLvLD0ILGdV7",
        "outputId": "f11c9ede-9257-4e1a-8952-d7f1d62b6aab"
      },
      "source": [
        "# Convert inputs and targets to tensors\n",
        "inputs = torch.from_numpy(inputs)\n",
        "targets = torch.from_numpy(targets)\n",
        "print(inputs)\n",
        "print(targets)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ 73.,  67.,  43.],\n",
            "        [ 91.,  88.,  64.],\n",
            "        [ 87., 134.,  58.],\n",
            "        [102.,  43.,  37.],\n",
            "        [ 69.,  96.,  70.]])\n",
            "tensor([[ 56.,  70.],\n",
            "        [ 81., 101.],\n",
            "        [119., 133.],\n",
            "        [ 22.,  37.],\n",
            "        [103., 119.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcUhLJo9GdV8"
      },
      "source": [
        "## Linear regression model from scratch\n",
        "\n",
        "The weights and biases (`w11, w12,... w23, b1 & b2`) can also be represented as matrices, initialized as random values. The first row of `w` and the first element of `b` are used to predict the first target variable, i.e., yield of apples, and similarly, the second for oranges."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G3GLVe_vGdV8",
        "outputId": "20943ab1-9501-41ef-f139-11778886910b"
      },
      "source": [
        "# Weights and biases\n",
        "w = torch.randn(2, 3, requires_grad=True)\n",
        "b = torch.randn(2, requires_grad=True)\n",
        "print(w)\n",
        "print(b)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-0.2910, -0.3450,  0.0305],\n",
            "        [-0.6528,  0.7386, -0.5153]], requires_grad=True)\n",
            "tensor([-0.9161, -0.7779], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "81Rsc8yAGdV9"
      },
      "source": [
        "`torch.randn` creates a tensor with the given shape, with elements picked randomly from a [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution) with mean 0 and standard deviation 1.\n",
        "\n",
        "Our *model* is simply a function that performs a matrix multiplication of the `inputs` and the weights `w` (transposed) and adds the bias `b` (replicated for each observation).\n",
        "\n",
        "![matrix-mult](https://i.imgur.com/WGXLFvA.png)\n",
        "\n",
        "We can define the model as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9x1KoNYTGdV9"
      },
      "source": [
        "def model(x):\n",
        "    return x @ w.t() + b"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6FqPyN3uGdV9"
      },
      "source": [
        "`@` represents matrix multiplication in PyTorch, and the `.t` method returns the transpose of a tensor.\n",
        "\n",
        "The matrix obtained by passing the input data into the model is a set of predictions for the target variables."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1FQlP5ugGdV9",
        "outputId": "a4cc8320-9244-4552-ac61-a832f2979b03"
      },
      "source": [
        "# Generate predictions\n",
        "preds = model(inputs)\n",
        "print(preds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-43.9569, -21.1025],\n",
            "        [-55.7975, -28.1628],\n",
            "        [-70.6863,  11.5154],\n",
            "        [-44.2982, -54.6685],\n",
            "        [-51.9732, -10.9839]], grad_fn=<AddBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wq62xzlbGdV-"
      },
      "source": [
        "Let's compare the predictions of our model with the actual targets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_XYEktlYGdV-",
        "outputId": "dc72ecad-510b-49e6-c964-f08434529c5e"
      },
      "source": [
        "# Compare with targets\n",
        "print(targets)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ 56.,  70.],\n",
            "        [ 81., 101.],\n",
            "        [119., 133.],\n",
            "        [ 22.,  37.],\n",
            "        [103., 119.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JcY_AXzRGdV-"
      },
      "source": [
        "You can see a big difference between our model's predictions and the actual targets because we've initialized our model with random weights and biases. Obviously, we can't expect a randomly initialized model to *just work*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "28x4jxmVGdV-"
      },
      "source": [
        "## Loss function\n",
        "\n",
        "Before we improve our model, we need a way to evaluate how well our model is performing. We can compare the model's predictions with the actual targets using the following method:\n",
        "\n",
        "* Calculate the difference between the two matrices (`preds` and `targets`).\n",
        "* Square all elements of the difference matrix to remove negative values.\n",
        "* Calculate the average of the elements in the resulting matrix.\n",
        "\n",
        "The result is a single number, known as the **mean squared error** (MSE)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IueXsEV7GdV_"
      },
      "source": [
        "# MSE loss\n",
        "def mse(t1, t2):\n",
        "    diff = t1 - t2\n",
        "    return torch.sum(diff * diff) / diff.numel()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iAhn8A6gGdV_"
      },
      "source": [
        "`torch.sum` returns the sum of all the elements in a tensor. The `.numel` method of a tensor returns the number of elements in a tensor. Let's compute the mean squared error for the current predictions of our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dHTrJ9XxGdV_",
        "outputId": "3981443d-1b6b-4053-ee3e-bb6e4dcdf07b"
      },
      "source": [
        "# Compute loss\n",
        "loss = mse(preds, targets)\n",
        "print(loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(15813.8125, grad_fn=<DivBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V1q2RhYSGdV_"
      },
      "source": [
        "Here’s how we can interpret the result: *On average, each element in the prediction differs from the actual target by the square root of the loss*. And that’s pretty bad, considering the numbers we are trying to predict are themselves in the range 50–200. The result is called the *loss* because it indicates how bad the model is at predicting the target variables. It represents information loss in the model: the lower the loss, the better the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tPUJ9Zf4GdV_"
      },
      "source": [
        "## Compute gradients\n",
        "\n",
        "With PyTorch, we can automatically compute the gradient or derivative of the loss w.r.t. to the weights and biases because they have `requires_grad` set to `True`. We'll see how this is useful in just a moment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m1DpkrARGdWA"
      },
      "source": [
        "# Compute gradients\n",
        "loss.backward()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CM7cv1JVGdWA"
      },
      "source": [
        "The gradients are stored in the `.grad` property of the respective tensors. Note that the derivative of the loss w.r.t. the weights matrix is itself a matrix with the same dimensions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nVACXV5uGdWA",
        "outputId": "43328fe7-13f3-4c08-c99f-fc6abb141789"
      },
      "source": [
        "# Gradients for weights\n",
        "print(w)\n",
        "print(w.grad)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-0.2910, -0.3450,  0.0305],\n",
            "        [-0.6528,  0.7386, -0.5153]], requires_grad=True)\n",
            "tensor([[-10740.7393, -12376.3008,  -7471.2300],\n",
            "        [ -9458.5078, -10033.8672,  -6344.1094]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2zhdXIt4GdWA"
      },
      "source": [
        "## Adjust weights and biases to reduce the loss\n",
        "\n",
        "The loss is a [quadratic function](https://en.wikipedia.org/wiki/Quadratic_function) of our weights and biases, and our objective is to find the set of weights where the loss is the lowest. If we plot a graph of the loss w.r.t any individual weight or bias element, it will look like the figure shown below. An important insight from calculus is that the gradient indicates the rate of change of the loss, i.e., the loss function's [slope](https://en.wikipedia.org/wiki/Slope) w.r.t. the weights and biases.\n",
        "\n",
        "If a gradient element is **positive**:\n",
        "\n",
        "* **increasing** the weight element's value slightly will **increase** the loss\n",
        "* **decreasing** the weight element's value slightly will **decrease** the loss\n",
        "\n",
        "![postive-gradient](https://i.imgur.com/WLzJ4xP.png)\n",
        "\n",
        "If a gradient element is **negative**:\n",
        "\n",
        "* **increasing** the weight element's value slightly will **decrease** the loss\n",
        "* **decreasing** the weight element's value slightly will **increase** the loss\n",
        "\n",
        "![negative=gradient](https://i.imgur.com/dvG2fxU.png)\n",
        "\n",
        "The increase or decrease in the loss by changing a weight element is proportional to the gradient of the loss w.r.t. that element. This observation forms the basis of _the gradient descent_ optimization algorithm that we'll use to improve our model (by _descending_ along the _gradient_).\n",
        "\n",
        "We can subtract from each weight element a small quantity proportional to the derivative of the loss w.r.t. that element to reduce the loss slightly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ffoh4TupGdWA",
        "outputId": "5668e406-571d-4f1f-e991-c83cbc234cb8"
      },
      "source": [
        "w\n",
        "w.grad"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-10740.7393, -12376.3008,  -7471.2300],\n",
              "        [ -9458.5078, -10033.8672,  -6344.1094]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l7D7fuxDGdWB"
      },
      "source": [
        "with torch.no_grad():\n",
        "    w -= w.grad * 1e-5\n",
        "    b -= b.grad * 1e-5"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PUDlN0AJGdWB"
      },
      "source": [
        "We multiply the gradients with a very small number (`10^-5` in this case) to ensure that we don't modify the weights by a very large amount. We want to take a small step in the downhill direction of the gradient, not a giant leap. This number is called the *learning rate* of the algorithm. \n",
        "\n",
        "We use `torch.no_grad` to indicate to PyTorch that we shouldn't track, calculate, or modify gradients while updating the weights and biases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hPvXBOKoGdWB",
        "outputId": "a114dd28-609d-400a-c837-0daf9a1eb3d5"
      },
      "source": [
        "# Let's verify that the loss is actually lower\n",
        "loss = mse(preds, targets)\n",
        "print(loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(15813.8125, grad_fn=<DivBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vQw2TZDtGdWB"
      },
      "source": [
        "Before we proceed, we reset the gradients to zero by invoking the `.zero_()` method. We need to do this because PyTorch accumulates gradients. Otherwise, the next time we invoke `.backward` on the loss, the new gradient values are added to the existing gradients, which may lead to unexpected results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OAKDkU5pGdWC",
        "outputId": "ff0446c8-df4a-496e-aa95-b006a8f84202"
      },
      "source": [
        "w.grad.zero_()\n",
        "b.grad.zero_()\n",
        "print(w.grad)\n",
        "print(b.grad)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0., 0., 0.],\n",
            "        [0., 0., 0.]])\n",
            "tensor([0., 0.])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YnskcylUGdWC"
      },
      "source": [
        "## Train the model using gradient descent\n",
        "\n",
        "As seen above, we reduce the loss and improve our model using the gradient descent optimization algorithm. Thus, we can _train_ the model using the following steps:\n",
        "\n",
        "1. Generate predictions\n",
        "\n",
        "2. Calculate the loss\n",
        "\n",
        "3. Compute gradients w.r.t the weights and biases\n",
        "\n",
        "4. Adjust the weights by subtracting a small quantity proportional to the gradient\n",
        "\n",
        "5. Reset the gradients to zero\n",
        "\n",
        "Let's implement the above step by step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yjJrNKYrGdWC",
        "outputId": "7fca1a0c-016a-4d43-e5f8-c4e3d15deee5"
      },
      "source": [
        "# Generate predictions\n",
        "preds = model(inputs)\n",
        "print(preds)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-24.6101,  -4.7460],\n",
            "        [-30.3494,  -6.6644],\n",
            "        [-40.4230,  36.8703],\n",
            "        [-25.2552, -38.3578],\n",
            "        [-27.4497,   9.6170]], grad_fn=<AddBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2FkBoac1GdWC",
        "outputId": "3dc9785c-4e17-4fe2-d7bd-f89880d81810"
      },
      "source": [
        "# Calculate the loss\n",
        "loss = mse(preds, targets)\n",
        "print(loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(10762.5488, grad_fn=<DivBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rvg5VlU8GdWD",
        "outputId": "7068695f-0d8b-4f78-d14f-04099b4da246"
      },
      "source": [
        "# Compute gradients\n",
        "loss.backward()\n",
        "print(w.grad)\n",
        "print(b.grad)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[ -8741.6377, -10223.4902,  -6143.8101],\n",
            "        [ -7770.2256,  -8220.9961,  -5225.0342]])\n",
            "tensor([-105.8175,  -92.6562])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wN9FwjLGGdWD"
      },
      "source": [
        "Let's update the weights and biases using the gradients computed above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4b0MV45MGdWD"
      },
      "source": [
        "# Adjust weights & reset gradients\n",
        "with torch.no_grad():\n",
        "    w -= w.grad * 1e-5\n",
        "    b -= b.grad * 1e-5\n",
        "    w.grad.zero_()\n",
        "    b.grad.zero_()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yhV-EW2cGdWD"
      },
      "source": [
        "Let's take a look at the new weights and biases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pvCR9tSNGdWD",
        "outputId": "3bc5ea5c-4142-4084-acca-951b97072f05"
      },
      "source": [
        "print(w)\n",
        "print(b)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[-0.0961, -0.1190,  0.1667],\n",
            "        [-0.4805,  0.9211, -0.3996]], requires_grad=True)\n",
            "tensor([-0.9137, -0.7759], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLaFJNMOGdWE"
      },
      "source": [
        "With the new weights and biases, the model should have a lower loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fFn5v2AxGdWE",
        "outputId": "633b200d-943f-4202-a822-0ec4a25c401b"
      },
      "source": [
        "# Calculate loss\n",
        "preds = model(inputs)\n",
        "loss = mse(preds, targets)\n",
        "print(loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(7357.4829, grad_fn=<DivBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wzAmRxtCGdWE"
      },
      "source": [
        "We have already achieved a significant reduction in the loss merely by adjusting the weights and biases slightly using gradient descent."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JxQtNlEeGdWE"
      },
      "source": [
        "## Train for multiple epochs\n",
        "\n",
        "To reduce the loss further, we can repeat the process of adjusting the weights and biases using the gradients multiple times. Each iteration is called an _epoch_. Let's train the model for 100 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2cyhXwqtGdWE"
      },
      "source": [
        "# Train for 100 epochs\n",
        "for i in range(100):\n",
        "    preds = model(inputs)\n",
        "    loss = mse(preds, targets)\n",
        "    loss.backward()\n",
        "    with torch.no_grad():\n",
        "        w -= w.grad * 1e-5\n",
        "        b -= b.grad * 1e-5\n",
        "        w.grad.zero_()\n",
        "        b.grad.zero_()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BgR5o6tlGdWE"
      },
      "source": [
        "Once again, let's verify that the loss is now lower:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9CEYgB-NGdWF",
        "outputId": "9912e9e1-3ac2-4b54-9539-ed4d1b1b2ab6"
      },
      "source": [
        "# Calculate loss\n",
        "preds = model(inputs)\n",
        "loss = mse(preds, targets)\n",
        "print(loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(130.3513, grad_fn=<DivBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmpRFSz5GdWF"
      },
      "source": [
        "The loss is now much lower than its initial value. Let's look at the model's predictions and compare them with the targets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qr5fMGj-GdWF",
        "outputId": "d78ac15d-789e-4227-a4ed-dd634d691cde"
      },
      "source": [
        "# Predictions\n",
        "preds"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 60.8975,  70.5663],\n",
              "        [ 83.9699,  92.9066],\n",
              "        [108.6802, 150.1993],\n",
              "        [ 43.5842,  38.4608],\n",
              "        [ 91.6760, 104.6360]], grad_fn=<AddBackward0>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nl7_x7zkGdWF",
        "outputId": "74c9f2fd-444d-4e5a-da9c-e63a86dac6d4"
      },
      "source": [
        "# Targets\n",
        "targets"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 56.,  70.],\n",
              "        [ 81., 101.],\n",
              "        [119., 133.],\n",
              "        [ 22.,  37.],\n",
              "        [103., 119.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n0fLz6u-GdWF"
      },
      "source": [
        "The predictions are now quite close to the target variables. We can get even better results by training for a few more epochs. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_-k5QBhCGdWG"
      },
      "source": [
        "## Save and upload your notebook\n",
        "\n",
        "Whether you're running this Jupyter notebook online or on your computer, it's essential to save your work from time to time. You can continue working on a saved notebook later or share it with friends and colleagues to let them execute your code. [Jovian](https://jovian.ai/platform-features) offers an easy way of saving and sharing your Jupyter notebooks online.\n",
        "\n",
        "First, you need to install the Jovian python library if it isn't already installed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WMvOOKbtGdWG"
      },
      "source": [
        "!pip install jovian --upgrade -q"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "42KmD61lGdWG"
      },
      "source": [
        "import jovian"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uSmEQ60jGdWG",
        "outputId": "8b31b8e7-818a-44bd-973c-dbf0c18f699f"
      },
      "source": [
        "jovian.commit(project='02-linear-regression')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "window.require && require([\"base/js/namespace\"],function(Jupyter){Jupyter.notebook.save_checkpoint()})"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "[jovian] Attempting to save notebook..\u001b[0m\n",
            "[jovian] Updating notebook \"aakashns/02-linear-regression\" on https://jovian.ai/\u001b[0m\n",
            "[jovian] Uploading notebook..\u001b[0m\n",
            "[jovian] Capturing environment..\u001b[0m\n",
            "[jovian] Committed successfully! https://jovian.ai/aakashns/02-linear-regression\u001b[0m\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'https://jovian.ai/aakashns/02-linear-regression'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BsXwFRqUGdWH"
      },
      "source": [
        "`jovian.commit` uploads the notebook to your Jovian account, captures the Python environment, and creates a shareable link for your notebook, as shown above. You can use this link to share your work and let anyone (including you) run your notebooks and reproduce your work.\n",
        "\n",
        "> The first time you run `jovian.commit`, you may be asked to provide an _API Key_ to securely upload the notebook to your Jovian account. You can get the API key from your [Jovian profile page](https://jovian.ai) after logging in / signing up.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGbrm_saGdWH"
      },
      "source": [
        "## Linear regression using PyTorch built-ins\n",
        "\n",
        "We've implemented linear regression & gradient descent model using some basic tensor operations. However, since this is a common pattern in deep learning, PyTorch provides several built-in functions and classes to make it easy to create and train models with just a few lines of code.\n",
        "\n",
        "Let's begin by importing the `torch.nn` package from PyTorch, which contains utility classes for building neural networks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8G4gywkGdWH"
      },
      "source": [
        "import torch.nn as nn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BIPtKX9TGdWH"
      },
      "source": [
        "As before, we represent the inputs and targets and matrices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQyB-_qXGdWI"
      },
      "source": [
        "# Input (temp, rainfall, humidity)\n",
        "inputs = np.array([[73, 67, 43], \n",
        "                   [91, 88, 64], \n",
        "                   [87, 134, 58], \n",
        "                   [102, 43, 37], \n",
        "                   [69, 96, 70], \n",
        "                   [74, 66, 43], \n",
        "                   [91, 87, 65], \n",
        "                   [88, 134, 59], \n",
        "                   [101, 44, 37], \n",
        "                   [68, 96, 71], \n",
        "                   [73, 66, 44], \n",
        "                   [92, 87, 64], \n",
        "                   [87, 135, 57], \n",
        "                   [103, 43, 36], \n",
        "                   [68, 97, 70]], \n",
        "                  dtype='float32')\n",
        "\n",
        "# Targets (apples, oranges)\n",
        "targets = np.array([[56, 70], \n",
        "                    [81, 101], \n",
        "                    [119, 133], \n",
        "                    [22, 37], \n",
        "                    [103, 119],\n",
        "                    [57, 69], \n",
        "                    [80, 102], \n",
        "                    [118, 132], \n",
        "                    [21, 38], \n",
        "                    [104, 118], \n",
        "                    [57, 69], \n",
        "                    [82, 100], \n",
        "                    [118, 134], \n",
        "                    [20, 38], \n",
        "                    [102, 120]], \n",
        "                   dtype='float32')\n",
        "\n",
        "inputs = torch.from_numpy(inputs)\n",
        "targets = torch.from_numpy(targets)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "54a8Tvs6GdWI",
        "outputId": "0bf9c7bd-ef03-43ed-94cd-6b7e7623176b"
      },
      "source": [
        "inputs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 73.,  67.,  43.],\n",
              "        [ 91.,  88.,  64.],\n",
              "        [ 87., 134.,  58.],\n",
              "        [102.,  43.,  37.],\n",
              "        [ 69.,  96.,  70.],\n",
              "        [ 74.,  66.,  43.],\n",
              "        [ 91.,  87.,  65.],\n",
              "        [ 88., 134.,  59.],\n",
              "        [101.,  44.,  37.],\n",
              "        [ 68.,  96.,  71.],\n",
              "        [ 73.,  66.,  44.],\n",
              "        [ 92.,  87.,  64.],\n",
              "        [ 87., 135.,  57.],\n",
              "        [103.,  43.,  36.],\n",
              "        [ 68.,  97.,  70.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oy3H3RZ7GdWI"
      },
      "source": [
        "We are using 15 training examples to illustrate how to work with large datasets in small batches. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8TFj_u0zGdWI"
      },
      "source": [
        "## Dataset and DataLoader\n",
        "\n",
        "We'll create a `TensorDataset`, which allows access to rows from `inputs` and `targets` as tuples, and provides standard APIs for working with many different types of datasets in PyTorch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fMANMiSUGdWI"
      },
      "source": [
        "from torch.utils.data import TensorDataset"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1fQOEptEGdWI",
        "outputId": "4c6fff7a-16be-415f-929e-d30279c636d2"
      },
      "source": [
        "# Define dataset\n",
        "train_ds = TensorDataset(inputs, targets)\n",
        "train_ds[0:3]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[ 73.,  67.,  43.],\n",
              "         [ 91.,  88.,  64.],\n",
              "         [ 87., 134.,  58.]]),\n",
              " tensor([[ 56.,  70.],\n",
              "         [ 81., 101.],\n",
              "         [119., 133.]]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S_CsHX37GdWJ"
      },
      "source": [
        "The `TensorDataset` allows us to access a small section of the training data using the array indexing notation (`[0:3]` in the above code). It returns a tuple with two elements. The first element contains the input variables for the selected rows, and the second contains the targets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6UnVDy5GdWJ"
      },
      "source": [
        "We'll also create a `DataLoader`, which can split the data into batches of a predefined size while training. It also provides other utilities like shuffling and random sampling of the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9fImeCGbGdWJ"
      },
      "source": [
        "from torch.utils.data import DataLoader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o9E4nlWrGdWJ"
      },
      "source": [
        "# Define data loader\n",
        "batch_size = 5\n",
        "train_dl = DataLoader(train_ds, batch_size, shuffle=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VnCHfqN_GdWJ"
      },
      "source": [
        "We can use the data loader in a `for` loop. Let's look at an example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_XdElIUOGdWJ",
        "outputId": "e4f0d24d-7663-4a9d-c055-91e576e77b86"
      },
      "source": [
        "for xb, yb in train_dl:\n",
        "    print(xb)\n",
        "    print(yb)\n",
        "    break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[102.,  43.,  37.],\n",
            "        [ 92.,  87.,  64.],\n",
            "        [ 87., 134.,  58.],\n",
            "        [ 69.,  96.,  70.],\n",
            "        [101.,  44.,  37.]])\n",
            "tensor([[ 22.,  37.],\n",
            "        [ 82., 100.],\n",
            "        [119., 133.],\n",
            "        [103., 119.],\n",
            "        [ 21.,  38.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6mfnUazHGdWK"
      },
      "source": [
        "In each iteration, the data loader returns one batch of data with the given batch size. If `shuffle` is set to `True`, it shuffles the training data before creating batches. Shuffling helps randomize the input to the optimization algorithm, leading to a faster reduction in the loss."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rtMAIveDGdWK"
      },
      "source": [
        "## nn.Linear\n",
        "\n",
        "Instead of initializing the weights & biases manually, we can define the model using the `nn.Linear` class from PyTorch, which does it automatically."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ivii2KT-GdWK",
        "outputId": "cd74a5ed-5de0-401d-e397-9c21ddbada5f"
      },
      "source": [
        "# Define model\n",
        "model = nn.Linear(3, 2)\n",
        "print(model.weight)\n",
        "print(model.bias)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Parameter containing:\n",
            "tensor([[ 0.1304, -0.1898,  0.2187],\n",
            "        [ 0.2360,  0.4139, -0.4540]], requires_grad=True)\n",
            "Parameter containing:\n",
            "tensor([0.3457, 0.3883], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kY_c1j7GdWK"
      },
      "source": [
        "PyTorch models also have a helpful `.parameters` method, which returns a list containing all the weights and bias matrices present in the model. For our linear regression model, we have one weight matrix and one bias matrix."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAv4ZwWZGdWK",
        "outputId": "0e701d30-8407-484f-ccc7-8fc51dc4b326"
      },
      "source": [
        "# Parameters\n",
        "list(model.parameters())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[ 0.1304, -0.1898,  0.2187],\n",
              "         [ 0.2360,  0.4139, -0.4540]], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([0.3457, 0.3883], requires_grad=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZsVXczQiGdWL"
      },
      "source": [
        "We can use the model to generate predictions in the same way as before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UaaLRRYgGdWL",
        "outputId": "40d702b9-8963-4bfa-a447-4cc29129b24b"
      },
      "source": [
        "# Generate predictions\n",
        "preds = model(inputs)\n",
        "preds"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 6.5493, 25.8226],\n",
              "        [ 9.5025, 29.2272],\n",
              "        [-1.0633, 50.0460],\n",
              "        [13.5738, 25.4576],\n",
              "        [ 6.4278, 24.6221],\n",
              "        [ 6.8695, 25.6447],\n",
              "        [ 9.9110, 28.3593],\n",
              "        [-0.7142, 49.8280],\n",
              "        [13.2536, 25.6355],\n",
              "        [ 6.5161, 23.9321],\n",
              "        [ 6.9578, 24.9546],\n",
              "        [ 9.8227, 29.0494],\n",
              "        [-1.4718, 50.9139],\n",
              "        [13.4855, 26.1476],\n",
              "        [ 6.1076, 24.8000]], grad_fn=<AddmmBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMvnv-N8GdWL"
      },
      "source": [
        "## Loss Function\n",
        "\n",
        "Instead of defining a loss function manually, we can use the built-in loss function `mse_loss`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NmQ3vpy1GdWL"
      },
      "source": [
        "# Import nn.functional\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LfgBCKzBGdWM"
      },
      "source": [
        "The `nn.functional` package contains many useful loss functions and several other utilities. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1RMH0ZGRGdWM"
      },
      "source": [
        "# Define loss function\n",
        "loss_fn = F.mse_loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sf4gumgSGdWM"
      },
      "source": [
        "Let's compute the loss for the current predictions of our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z1I6PQojGdWM",
        "outputId": "28d1659d-3c85-4fc4-df71-25e20e7d9344"
      },
      "source": [
        "loss = loss_fn(model(inputs), targets)\n",
        "print(loss)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(5427.9517, grad_fn=<MseLossBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gF44zaBNGdWM"
      },
      "source": [
        "## Optimizer\n",
        "\n",
        "Instead of manually manipulating the model's weights & biases using gradients, we can use the optimizer `optim.SGD`. SGD is short for \"stochastic gradient descent\". The term _stochastic_ indicates that samples are selected in random batches instead of as a single group."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4NH1u-6mGdWM"
      },
      "source": [
        "# Define optimizer\n",
        "opt = torch.optim.SGD(model.parameters(), lr=1e-5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o5--3gRIGdWN"
      },
      "source": [
        "Note that `model.parameters()` is passed as an argument to `optim.SGD` so that the optimizer knows which matrices should be modified during the update step. Also, we can specify a learning rate that controls the amount by which the parameters are modified."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87jztoKrGdWN"
      },
      "source": [
        "## Train the model\n",
        "\n",
        "We are now ready to train the model. We'll follow the same process to implement gradient descent:\n",
        "\n",
        "1. Generate predictions\n",
        "\n",
        "2. Calculate the loss\n",
        "\n",
        "3. Compute gradients w.r.t the weights and biases\n",
        "\n",
        "4. Adjust the weights by subtracting a small quantity proportional to the gradient\n",
        "\n",
        "5. Reset the gradients to zero\n",
        "\n",
        "The only change is that we'll work batches of data instead of processing the entire training data in every iteration. Let's define a utility function `fit` that trains the model for a given number of epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bya4yq-dGdWN"
      },
      "source": [
        "# Utility function to train the model\n",
        "def fit(num_epochs, model, loss_fn, opt, train_dl):\n",
        "    \n",
        "    # Repeat for given number of epochs\n",
        "    for epoch in range(num_epochs):\n",
        "        \n",
        "        # Train with batches of data\n",
        "        for xb,yb in train_dl:\n",
        "            \n",
        "            # 1. Generate predictions\n",
        "            pred = model(xb)\n",
        "            \n",
        "            # 2. Calculate loss\n",
        "            loss = loss_fn(pred, yb)\n",
        "            \n",
        "            # 3. Compute gradients\n",
        "            loss.backward()\n",
        "            \n",
        "            # 4. Update parameters using gradients\n",
        "            opt.step()\n",
        "            \n",
        "            # 5. Reset the gradients to zero\n",
        "            opt.zero_grad()\n",
        "        \n",
        "        # Print the progress\n",
        "        if (epoch+1) % 10 == 0:\n",
        "            print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lq5PBPvcGdWN"
      },
      "source": [
        "Some things to note above:\n",
        "\n",
        "* We use the data loader defined earlier to get batches of data for every iteration.\n",
        "\n",
        "* Instead of updating parameters (weights and biases) manually, we use `opt.step` to perform the update and `opt.zero_grad` to reset the gradients to zero.\n",
        "\n",
        "* We've also added a log statement that prints the loss from the last batch of data for every 10th epoch to track training progress. `loss.item` returns the actual value stored in the loss tensor.\n",
        "\n",
        "Let's train the model for 100 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UdWKtPZqGdWN",
        "outputId": "1e354ead-1cd9-482b-ec1e-e8bfbf79dd2f"
      },
      "source": [
        "fit(100, model, loss_fn, opt, train_dl)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch [10/100], Loss: 818.6476\n",
            "Epoch [20/100], Loss: 335.3347\n",
            "Epoch [30/100], Loss: 190.3544\n",
            "Epoch [40/100], Loss: 131.6701\n",
            "Epoch [50/100], Loss: 77.0783\n",
            "Epoch [60/100], Loss: 151.5671\n",
            "Epoch [70/100], Loss: 151.0817\n",
            "Epoch [80/100], Loss: 67.6262\n",
            "Epoch [90/100], Loss: 53.6205\n",
            "Epoch [100/100], Loss: 33.4517\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W2_CwjMFGdWO"
      },
      "source": [
        "Let's generate predictions using our model and verify that they're close to our targets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4SzmHtVdGdWO",
        "outputId": "e16f2a39-306f-4a11-8d84-2809c7e20628"
      },
      "source": [
        "# Generate predictions\n",
        "preds = model(inputs)\n",
        "preds"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 58.4229,  72.0145],\n",
              "        [ 82.1525,  95.1376],\n",
              "        [115.8955, 142.6296],\n",
              "        [ 28.6805,  46.0115],\n",
              "        [ 97.5243, 104.3522],\n",
              "        [ 57.3792,  70.9543],\n",
              "        [ 81.9342,  94.1737],\n",
              "        [116.2036, 142.6871],\n",
              "        [ 29.7242,  47.0717],\n",
              "        [ 98.3498, 104.4486],\n",
              "        [ 58.2047,  71.0507],\n",
              "        [ 81.1088,  94.0774],\n",
              "        [116.1137, 143.5935],\n",
              "        [ 27.8550,  45.9152],\n",
              "        [ 98.5680, 105.4124]], grad_fn=<AddmmBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mkd3w78pGdWO",
        "outputId": "5e5d9ab0-602a-4db8-a895-fb34dd48e2a9"
      },
      "source": [
        "# Compare with targets\n",
        "targets"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 56.,  70.],\n",
              "        [ 81., 101.],\n",
              "        [119., 133.],\n",
              "        [ 22.,  37.],\n",
              "        [103., 119.],\n",
              "        [ 57.,  69.],\n",
              "        [ 80., 102.],\n",
              "        [118., 132.],\n",
              "        [ 21.,  38.],\n",
              "        [104., 118.],\n",
              "        [ 57.,  69.],\n",
              "        [ 82., 100.],\n",
              "        [118., 134.],\n",
              "        [ 20.,  38.],\n",
              "        [102., 120.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aq4zwJPiGdWO"
      },
      "source": [
        "Indeed, the predictions are quite close to our targets. We have a trained a reasonably good model to predict crop yields for apples and oranges by looking at the average temperature, rainfall, and humidity in a region. We can use it to make predictions of crop yields for new regions by passing a batch containing a single row of input."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bIKbHe6OGdWO",
        "outputId": "858143b7-85c0-426b-b274-94003453b459"
      },
      "source": [
        "model(torch.tensor([[75, 63, 44.]]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[55.3323, 67.8895]], grad_fn=<AddmmBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZGzr58WkGdWP"
      },
      "source": [
        "The predicted yield of apples is 54.3 tons per hectare, and that of oranges is 68.3 tons per hectare."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AllLKTvyGdWP"
      },
      "source": [
        "## Machine Learning vs. Classical Programming\n",
        "\n",
        "The approach we've taken in this tutorial is very different from programming as you might know it. Usually, we write programs that take some inputs, perform some operations, and return a result. \n",
        "\n",
        "However, in this notebook, we've defined a \"model\" that assumes a specific relationship between the inputs and the outputs, expressed using some unknown parameters (weights & biases). We then show the model some know inputs and outputs and _train_ the model to come up with good values for the unknown parameters. Once trained, the model can be used to compute the outputs for new inputs.\n",
        "\n",
        "This paradigm of programming is known as _machine learning_, where we use data to figure out the relationship between inputs and outputs. _Deep learning_ is a branch of machine learning that uses matrix operations, non-linear activation functions and gradient descent to build and train models. Andrej Karpathy, the director of AI at Tesla Motors, has written a great blog post on this topics, titled [Software 2.0](https://medium.com/@karpathy/software-2-0-a64152b37c35).\n",
        "\n",
        "This picture from book [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python) by Francois Chollet captures the difference between classical programming and machine learning:\n",
        "\n",
        "![](https://i.imgur.com/oJEQe7k.png)\n",
        "\n",
        "Keep this picture in mind as you work through the next few tutorials. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O73nMzrRGdWP"
      },
      "source": [
        "## Commit and update the notebook\n",
        "\n",
        "As a final step, we can record a new version of the notebook using the `jovian` library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Bc67KUqGdWP"
      },
      "source": [
        "import jovian"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DI0PoZlXGdWP",
        "outputId": "f04d8d67-45d7-4c26-b1fe-8f6c444aeed0"
      },
      "source": [
        "jovian.commit(project='02-linear-regression')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "window.require && require([\"base/js/namespace\"],function(Jupyter){Jupyter.notebook.save_checkpoint()})"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "[jovian] Attempting to save notebook..\u001b[0m\n",
            "[jovian] Updating notebook \"aakashns/02-linear-regression\" on https://jovian.ai/\u001b[0m\n",
            "[jovian] Uploading notebook..\u001b[0m\n",
            "[jovian] Capturing environment..\u001b[0m\n",
            "[jovian] Committed successfully! https://jovian.ai/aakashns/02-linear-regression\u001b[0m\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'https://jovian.ai/aakashns/02-linear-regression'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YX8KHqF1GdWP"
      },
      "source": [
        "Note that running `jovian.commit` a second time records a new version of your existing notebook. With Jovian.ml, you can avoid creating copies of your Jupyter notebooks and keep versions organized. Jovian also provides a visual diff ([example](https://jovian.ai/aakashns/keras-mnist-jovian/diff?base=8&remote=2)) so you can inspect what has changed between different versions:\n",
        "\n",
        "![jovian-commenting](https://i.imgur.com/HF1cOVt.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Vbwa9KCGdWQ"
      },
      "source": [
        "## Exercises and Further Reading\n",
        "\n",
        "We've covered the following topics in this tutorial:\n",
        "\n",
        "- Introduction to linear regression and gradient descent\n",
        "- Implementing a linear regression model using PyTorch tensors\n",
        "- Training a linear regression model using the gradient descent algorithm\n",
        "- Implementing gradient descent and linear regression using PyTorch built-in\n",
        "\n",
        "\n",
        "Here are some resources for learning more about linear regression and gradient descent:\n",
        "\n",
        "* An visual & animated explanation of gradient descent: https://www.youtube.com/watch?v=IHZwWFHWa-w\n",
        "\n",
        "* For a more detailed explanation of derivates and gradient descent, see [these notes from a Udacity course](https://storage.googleapis.com/supplemental_media/udacityu/315142919/Gradient%20Descent.pdf). \n",
        "\n",
        "* For an animated visualization of how linear regression works, [see this post](https://hackernoon.com/visualizing-linear-regression-with-pytorch-9261f49edb09).\n",
        "\n",
        "* For a more mathematical treatment of matrix calculus, linear regression and gradient descent, you should check out [Andrew Ng's excellent course notes](https://github.com/Cleo-Stanford-CS/CS229_Notes/blob/master/lectures/cs229-notes1.pdf) from CS229 at Stanford University.\n",
        "\n",
        "* To practice and test your skills, you can participate in the [Boston Housing Price Prediction](https://www.kaggle.com/c/boston-housing) competition on Kaggle, a website that hosts data science competitions.\n",
        "\n",
        "With this, we complete our discussion of linear regression in PyTorch, and we’re ready to move on to the next topic: [Working with Images & Logistic Regression](https://jovian.ai/aakashns/03-logistic-regression)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ruF0WNZ-GdWQ"
      },
      "source": [
        "## Questions for Review\n",
        "\n",
        "Try answering the following questions to test your understanding of the topics covered in this notebook:\n",
        "\n",
        "1. What is a linear regression model? Give an example of a problem formulated as a linear regression model.\n",
        "2. What are input and target variables in a dataset? Give an example.\n",
        "3. What are weights and biases in a linear regression model?\n",
        "4. How do you represent tabular data using PyTorch tensors?\n",
        "5. Why do we create separate matrices for inputs and targets while training a linear regression model?\n",
        "6. How do you determine the shape of the weights matrix & bias vector given some training data?\n",
        "7. How do you create randomly initialized weights & biases with a given shape?\n",
        "8. How is a linear regression model implemented using matrix operations? Explain with an example.\n",
        "9. How do you generate predictions using a linear regression model?\n",
        "10. Why are the predictions of a randomly initialized model different from the actual targets?\n",
        "11. What is a loss function? What does the term “loss” signify?\n",
        "12. What is mean squared error?\n",
        "13. Write a function to calculate mean squared using model predictions and actual targets.\n",
        "14. What happens when you invoke the `.backward` function on the result of the mean squared error loss function?\n",
        "15. Why is the derivative of the loss w.r.t. the weights matrix itself a matrix? What do its elements represent?\n",
        "16. How is the derivate of the loss w.r.t. a weight element useful for reducing the loss? Explain with an example.\n",
        "17. Suppose the derivative  of the loss w.r.t. a weight element is positive. Should you increase or decrease the element’s value slightly to get a lower loss?\n",
        "18. Suppose the derivative  of the loss w.r.t. a weight element is negative. Should you increase or decrease the element’s value slightly to get a lower loss?\n",
        "19. How do you update the weights and biases of a model using their respective gradients to reduce the loss slightly?\n",
        "20. What is the gradient descent optimization algorithm? Why is it called “gradient descent”?\n",
        "21. Why do you subtract a “small quantity” proportional to the gradient from the weights & biases, not the actual gradient itself?\n",
        "22. What is learning rate? Why is it important?\n",
        "23. What is `torch.no_grad`?\n",
        "24. Why do you reset gradients to zero after updating weights and biases?\n",
        "25. What are the steps involved in training a linear regression model using gradient descent?\n",
        "26. What is an epoch?\n",
        "27. What is the benefit of training a model for multiple epochs?\n",
        "28. How do you make predictions using a trained model?\n",
        "29. What should you do if your model’s loss doesn’t decrease while training? Hint: learning rate.\n",
        "30. What is `torch.nn`?\n",
        "31. What is the purpose of the `TensorDataset` class in PyTorch? Give an example.\n",
        "32. What is a data loader in PyTorch? Give an example.\n",
        "33. How do you use a data loader to retrieve batches of data?\n",
        "34. What are the benefits of shuffling the training data before creating batches?\n",
        "35. What is the benefit of training in small batches instead of training with the entire dataset?\n",
        "36. What is the purpose of the `nn.Linear` class in PyTorch? Give an example.\n",
        "37. How do you see the weights and biases of a `nn.Linear` model?\n",
        "38. What is the purpose of the `torch.nn.functional` module?\n",
        "39. How do you compute mean squared error loss using a PyTorch built-in function?\n",
        "40. What is an optimizer in PyTorch?\n",
        "41. What is `torch.optim.SGD`? What does SGD stand for?\n",
        "42. What are the inputs to a PyTorch optimizer? \n",
        "43. Give an example of creating an optimizer for training a linear regression model.\n",
        "44. Write a function to train a `nn.Linear` model in batches using gradient descent.\n",
        "45. How do you use a linear regression model to make predictions on previously unseen data? \n"
      ]
    }
  ]
}