IPYNB Code/02_Autograd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mengimport library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pertama-tama kita akan membuat variabel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Membuat variabel dengan mengaktifkan ```requires_grad```\n",
    "x = torch.randn(3, requires_grad=True)\n",
    "print(x)\n",
    "\n",
    "y = x + 2\n",
    "print(y)\n",
    "\n",
    "z = y * y * 3\n",
    "z = z.mean()\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Menghitung gradien"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# dz/dx\n",
    "z.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradien diatas dapat dihitung karena nilai dari z berupa skalar dan bukan vektor. Apabila misalnya kita hanya melakukan operasi z = y * y * 3, maka nilai dari z tidak berupa vektor dan tidak dapat dihitung gradiennya. Lalu bagaimana jika ingin melakukan backpropagation terhadap nilai yang berupa vektor?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "y = x * 2\n",
    "z = y * y * 3\n",
    "\n",
    "# kita membutuhkan v sebagai array yang akan membantu melakukan backward terhadap z\n",
    "v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)\n",
    "z.backward(v)\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Menghentikan Tracking Gradien"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "x = torch.randn(3, requires_grad=True)\n",
    "y = torch.randn(3, requires_grad=True)\n",
    "z = torch.randn(3, requires_grad=True)\n",
    "print(x)\n",
    "print(y)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Cara pertama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "x.requires_grad_(False)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cara kedua"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "y = y.detach()\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cara ketiga"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    z = x + y\n",
    "    print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradien yang salah akibat dari backpropagation\n",
    "Ketika kita melakukan backpropagation dengan epoch yang berjumlah lebih dari satu, maka kita akan mendapatkan nilai gradien yang salah seperti pada kode di bawah ini. Nilai dari gradien terakumulasi pada setiap perulangan epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "weights = torch.ones(4, requires_grad=True)\n",
    "for epoch in range(3):\n",
    "    model_output = (weights * 3).sum()\n",
    "    model_output.backward()\n",
    "    print(weights.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Untuk mengatasi kesalahan di atas, maka kode di bawah ini akan menghilangkan nilai gradien yang salah dengan cara mengosongkan nilai gradien"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "weights = torch.ones(4, requires_grad=True)\n",
    "for epoch in range(3):\n",
    "    model_output = (weights * 3).sum()\n",
    "    model_output.backward()\n",
    "    print(weights.grad)\n",
    "    weights.grad.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hal yang sama juga perlu dilakukan ketika menggunakan optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "weights = torch.ones(4, requires_grad=True)\n",
    "optimizer = torch.optim.SGD(weights, lr=0.01)\n",
    "optimizer.step()\n",
    "# gunakan optimizer untuk mengoptimalkan nilai weights\n",
    "optimizer.zero_grad()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}