Notebook3_AC_Parameter_estimation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Notebook3_AC-Parameter-estimation.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOGyJhIAzZLf3+YMGSyxnBO",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/levimcclenny/notebooks/blob/main/Notebook3_AC_Parameter_estimation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n-9NquZ6AySw"
      },
      "source": [
        "# AC Parameter Estimation\n",
        "In this example, we seek to estimate the parameters of the Allen-Cahn PDE system used in the last notebook. We can perform this estimation in *tensorDiffEq* using the DiscoveryModel()| \n",
        "\n",
        "First, install the package"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GCTmTejA-Irs",
        "outputId": "622db455-6d5e-474e-8fdf-c89da47470c4"
      },
      "source": [
        "!pip install tensordiffeq"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: tensordiffeq in /usr/local/lib/python3.7/dist-packages (0.1.8)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (3.2.2)\n",
            "Requirement already satisfied: pyDOE2 in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (1.3.0)\n",
            "Requirement already satisfied: pyfiglet in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (0.8.post1)\n",
            "Requirement already satisfied: tensorflow-probability in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (0.12.1)\n",
            "Requirement already satisfied: tensorflow in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (2.4.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (1.19.5)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (4.41.1)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from tensordiffeq) (1.4.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tensordiffeq) (1.3.1)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tensordiffeq) (2.8.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tensordiffeq) (2.4.7)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tensordiffeq) (0.10.0)\n",
            "Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability->tensordiffeq) (0.3.3)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability->tensordiffeq) (4.4.2)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability->tensordiffeq) (1.15.0)\n",
            "Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability->tensordiffeq) (0.1.6)\n",
            "Requirement already satisfied: cloudpickle>=1.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability->tensordiffeq) (1.3.0)\n",
            "Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (1.1.0)\n",
            "Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (1.12.1)\n",
            "Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (3.7.4.3)\n",
            "Requirement already satisfied: grpcio~=1.32.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (1.32.0)\n",
            "Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (3.3.0)\n",
            "Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (1.1.2)\n",
            "Requirement already satisfied: tensorboard~=2.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (2.4.1)\n",
            "Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (1.12)\n",
            "Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (1.6.3)\n",
            "Requirement already satisfied: tensorflow-estimator<2.5.0,>=2.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (2.4.0)\n",
            "Requirement already satisfied: h5py~=2.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (2.10.0)\n",
            "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (3.12.4)\n",
            "Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (0.36.2)\n",
            "Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (0.12.0)\n",
            "Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow->tensordiffeq) (0.2.0)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (2.23.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (0.4.4)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (3.3.4)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (1.28.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (1.8.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.4->tensorflow->tensordiffeq) (56.0.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->tensordiffeq) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->tensordiffeq) (2020.12.5)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->tensordiffeq) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->tensordiffeq) (3.0.4)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow->tensordiffeq) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.4->tensorflow->tensordiffeq) (3.10.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->tensordiffeq) (0.2.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->tensordiffeq) (4.2.1)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->tensordiffeq) (4.7.2)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow->tensordiffeq) (3.1.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard~=2.4->tensorflow->tensordiffeq) (3.4.1)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->tensordiffeq) (0.4.8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDgjpp5sBSPo"
      },
      "source": [
        "Imports as usual, make sure to import the DiscoveryModel class!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rwko8-z69VDv"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import scipy.io\n",
        "import tensordiffeq as tdq\n",
        "from tensordiffeq.models import DiscoveryModel"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H80OdZ2fBX_4"
      },
      "source": [
        "Create a list of tf.Variables for the parameters to be estimated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IM_10XFw9Z-Z"
      },
      "source": [
        "params = [tf.Variable(0.0, dtype=tf.float32), tf.Variable(0.0, dtype=tf.float32)]"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BAEk4OVrBeP-"
      },
      "source": [
        "Generate the strong-form PDE, replacing the parameters we are estimating with the tf.Variables defined in the params variable. In this case, var ends up being the params list described in the cell above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rysAttTf9c0t"
      },
      "source": [
        "# Define f_model, note the `vars` argument. Inputs must follow this order!\n",
        "def f_model(u_model, var, x, t):\n",
        "    u = u_model(tf.concat([x, t], 1))\n",
        "    u_x = tf.gradients(u, x)\n",
        "    u_xx = tf.gradients(u_x, x)\n",
        "    u_t = tf.gradients(u, t)\n",
        "    c1 = var[0]  # tunable param 1\n",
        "    c2 = var[1]  # tunable param 2\n",
        "    f_u = u_t - c1 * u_xx + c2 * u * u * u - c2 * u\n",
        "    return f_u"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_aZFUfVLB2jZ"
      },
      "source": [
        "Import the Allen-Cahn data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jbNmDGU29czs",
        "outputId": "172de69b-d5b8-4a31-e8f6-5dca07c1634d"
      },
      "source": [
        "# Get AC data from repo, requires the ?raw=true so wget doesnt download the webpage instead of the data!\n",
        "!wget --no-check-certificate https://github.com/tensordiffeq/TensorDiffEq/blob/main/examples/AC.mat?raw=true \\\n",
        "    -O AC.mat"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-05-06 05:00:57--  https://github.com/tensordiffeq/TensorDiffEq/blob/main/examples/AC.mat?raw=true\n",
            "Resolving github.com (github.com)... 192.30.255.112\n",
            "Connecting to github.com (github.com)|192.30.255.112|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://github.com/tensordiffeq/TensorDiffEq/raw/main/examples/AC.mat [following]\n",
            "--2021-05-06 05:00:57--  https://github.com/tensordiffeq/TensorDiffEq/raw/main/examples/AC.mat\n",
            "Reusing existing connection to github.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/tensordiffeq/TensorDiffEq/main/examples/AC.mat [following]\n",
            "--2021-05-06 05:00:57--  https://raw.githubusercontent.com/tensordiffeq/TensorDiffEq/main/examples/AC.mat\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 559196 (546K) [application/octet-stream]\n",
            "Saving to: ‘AC.mat’\n",
            "\n",
            "AC.mat              100%[===================>] 546.09K  --.-KB/s    in 0.03s   \n",
            "\n",
            "2021-05-06 05:00:57 (17.7 MB/s) - ‘AC.mat’ saved [559196/559196]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZELzKWdB7-j"
      },
      "source": [
        "Import and reformat data for use in tdq"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IngFvn219cyp"
      },
      "source": [
        "data = scipy.io.loadmat('AC.mat')\n",
        "\n",
        "t = data['tt'].flatten()[:, None]\n",
        "x = data['x'].flatten()[:, None]\n",
        "Exact = data['uu']\n",
        "Exact_u = np.real(Exact)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WwhuKGQaCBUR"
      },
      "source": [
        "Create input data for the target data in AC.mat and generate a list of input data. In this case, the targets are stored in X_star, and the inputs are stored in X"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZvBfDwP9cFv"
      },
      "source": [
        "# generate all combinations of x and t\n",
        "X, T = np.meshgrid(x, t)\n",
        "\n",
        "X_star = np.hstack((X.flatten()[:, None], T.flatten()[:, None]))\n",
        "u_star = Exact_u.T.flatten()[:, None]\n",
        "\n",
        "x = X_star[:, 0:1]\n",
        "t = X_star[:, 1:2]\n",
        "\n",
        "# append to a list for input to model.fit\n",
        "X = [x, t]"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wXQ1dgqQCTYP"
      },
      "source": [
        "Generate col_weights (if using SA solving) and layer sizes for the approimation neural network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0aqjIDhV9cEu"
      },
      "source": [
        "# define col_weights for SA discovery model, can be removed\n",
        "col_weights = tf.Variable(tf.random.uniform([np.shape(x)[0], 1]))\n",
        "\n",
        "# define MLP depth and layer width\n",
        "layer_sizes = [2, 128, 128, 128, 128, 1]"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fbZd9QJICwSN"
      },
      "source": [
        "Import the model and compile"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "njO_GVXA-BDz"
      },
      "source": [
        "model = DiscoveryModel()\n",
        "model.compile(layer_sizes, f_model, X, u_star, params) "
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s7-gfgNdC0NN"
      },
      "source": [
        "This is optional - define a different optimizer from the Keras suite of optimizers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xpx5HNnW-AsY"
      },
      "source": [
        "# an example as to how one could modify an optimizer, in this case the col_weights optimizer\n",
        "model.tf_optimizer_weights = tf.keras.optimizers.Adam(lr=0.005,\n",
        "                                                      beta_1=.95)\n",
        "\n"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8akMobysDEoO"
      },
      "source": [
        "Train the model with 10k Adam iterations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Gh-ql2Q-AIZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9daeae3a-da6d-4544-bbf1-c0ab87691d7c"
      },
      "source": [
        "# train loop\n",
        "model.fit(tf_iter=10000)\n"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[<tf.Variable 'Variable:0' shape=() dtype=float32>, <tf.Variable 'Variable:0' shape=() dtype=float32>]\n",
            "[<tf.Variable 'Variable:0' shape=() dtype=float32>, <tf.Variable 'Variable:0' shape=() dtype=float32>]\n",
            "It: 0, Time: 2.16\n",
            "loss_value: 0.5344747304916382\n",
            "vars estimate(s): [-0.0049884045, 0.004997983]\n",
            "It: 100, Time: 12.81\n",
            "loss_value: 0.23482635617256165\n",
            "vars estimate(s): [0.0076214774, -0.055614177]\n",
            "It: 200, Time: 13.11\n",
            "loss_value: 0.1246507316827774\n",
            "vars estimate(s): [0.0064566415, -0.0753046]\n",
            "It: 300, Time: 13.19\n",
            "loss_value: 0.09062732011079788\n",
            "vars estimate(s): [0.0025977604, -0.08056378]\n",
            "It: 400, Time: 12.91\n",
            "loss_value: 0.0749722570180893\n",
            "vars estimate(s): [0.0016510101, -0.07313789]\n",
            "It: 500, Time: 12.81\n",
            "loss_value: 0.06650445610284805\n",
            "vars estimate(s): [-0.00091971294, -0.05759476]\n",
            "It: 600, Time: 12.81\n",
            "loss_value: 0.06338178366422653\n",
            "vars estimate(s): [-0.0009755291, -0.033789974]\n",
            "It: 700, Time: 12.89\n",
            "loss_value: 0.060310978442430496\n",
            "vars estimate(s): [-0.0013915153, 0.0014047724]\n",
            "It: 800, Time: 13.00\n",
            "loss_value: 0.05681362375617027\n",
            "vars estimate(s): [-0.0034183604, 0.04829152]\n",
            "It: 900, Time: 12.99\n",
            "loss_value: 0.05299413204193115\n",
            "vars estimate(s): [-0.004688485, 0.09918145]\n",
            "It: 1000, Time: 12.91\n",
            "loss_value: 0.04938874766230583\n",
            "vars estimate(s): [-0.0059074904, 0.15289827]\n",
            "It: 1100, Time: 12.87\n",
            "loss_value: 0.04603976011276245\n",
            "vars estimate(s): [-0.007922644, 0.20845892]\n",
            "It: 1200, Time: 12.85\n",
            "loss_value: 0.04316887632012367\n",
            "vars estimate(s): [-0.009575116, 0.26404747]\n",
            "It: 1300, Time: 12.92\n",
            "loss_value: 0.040702082216739655\n",
            "vars estimate(s): [-0.010564709, 0.31860712]\n",
            "It: 1400, Time: 12.98\n",
            "loss_value: 0.03820570558309555\n",
            "vars estimate(s): [-0.011884162, 0.373123]\n",
            "It: 1500, Time: 12.96\n",
            "loss_value: 0.035820718854665756\n",
            "vars estimate(s): [-0.012862196, 0.4276584]\n",
            "It: 1600, Time: 12.94\n",
            "loss_value: 0.03382422402501106\n",
            "vars estimate(s): [-0.013331259, 0.4834011]\n",
            "It: 1700, Time: 12.93\n",
            "loss_value: 0.032066695392131805\n",
            "vars estimate(s): [-0.013025589, 0.54084855]\n",
            "It: 1800, Time: 12.92\n",
            "loss_value: 0.030491149052977562\n",
            "vars estimate(s): [-0.012763535, 0.6007395]\n",
            "It: 1900, Time: 12.90\n",
            "loss_value: 0.0290256068110466\n",
            "vars estimate(s): [-0.012378682, 0.6634415]\n",
            "It: 2000, Time: 12.88\n",
            "loss_value: 0.027584055438637733\n",
            "vars estimate(s): [-0.011966269, 0.72928816]\n",
            "It: 2100, Time: 12.88\n",
            "loss_value: 0.02616490051150322\n",
            "vars estimate(s): [-0.011059937, 0.7985625]\n",
            "It: 2200, Time: 12.88\n",
            "loss_value: 0.024762965738773346\n",
            "vars estimate(s): [-0.010312063, 0.87168306]\n",
            "It: 2300, Time: 12.88\n",
            "loss_value: 0.023371875286102295\n",
            "vars estimate(s): [-0.009700762, 0.9483864]\n",
            "It: 2400, Time: 12.90\n",
            "loss_value: 0.022009657695889473\n",
            "vars estimate(s): [-0.00916214, 1.0283281]\n",
            "It: 2500, Time: 12.93\n",
            "loss_value: 0.0206641536206007\n",
            "vars estimate(s): [-0.008634364, 1.1114107]\n",
            "It: 2600, Time: 12.97\n",
            "loss_value: 0.019392671063542366\n",
            "vars estimate(s): [-0.008211341, 1.1971142]\n",
            "It: 2700, Time: 12.93\n",
            "loss_value: 0.01816282421350479\n",
            "vars estimate(s): [-0.007668869, 1.2847536]\n",
            "It: 2800, Time: 12.88\n",
            "loss_value: 0.016967883333563805\n",
            "vars estimate(s): [-0.0071371947, 1.3742208]\n",
            "It: 2900, Time: 12.85\n",
            "loss_value: 0.01585286110639572\n",
            "vars estimate(s): [-0.006668203, 1.4652625]\n",
            "It: 3000, Time: 12.88\n",
            "loss_value: 0.014752739109098911\n",
            "vars estimate(s): [-0.0062375367, 1.5574292]\n",
            "It: 3100, Time: 12.92\n",
            "loss_value: 0.013736739754676819\n",
            "vars estimate(s): [-0.0058551747, 1.6502938]\n",
            "It: 3200, Time: 12.97\n",
            "loss_value: 0.012769829481840134\n",
            "vars estimate(s): [-0.0054665357, 1.743423]\n",
            "It: 3300, Time: 12.98\n",
            "loss_value: 0.01178671047091484\n",
            "vars estimate(s): [-0.005126364, 1.8366003]\n",
            "It: 3400, Time: 12.96\n",
            "loss_value: 0.010990351438522339\n",
            "vars estimate(s): [-0.004803842, 1.9296197]\n",
            "It: 3500, Time: 12.95\n",
            "loss_value: 0.010141780599951744\n",
            "vars estimate(s): [-0.004427753, 2.022324]\n",
            "It: 3600, Time: 12.93\n",
            "loss_value: 0.009342297911643982\n",
            "vars estimate(s): [-0.004082521, 2.1145942]\n",
            "It: 3700, Time: 12.90\n",
            "loss_value: 0.008584600873291492\n",
            "vars estimate(s): [-0.003800391, 2.2062724]\n",
            "It: 3800, Time: 12.88\n",
            "loss_value: 0.007873738184571266\n",
            "vars estimate(s): [-0.003464905, 2.2972593]\n",
            "It: 3900, Time: 12.88\n",
            "loss_value: 0.007264995947480202\n",
            "vars estimate(s): [-0.0032489426, 2.387433]\n",
            "It: 4000, Time: 12.88\n",
            "loss_value: 0.006664888933300972\n",
            "vars estimate(s): [-0.0029200425, 2.476686]\n",
            "It: 4100, Time: 12.88\n",
            "loss_value: 0.006159359589219093\n",
            "vars estimate(s): [-0.0027377303, 2.5649056]\n",
            "It: 4200, Time: 12.89\n",
            "loss_value: 0.005509064067155123\n",
            "vars estimate(s): [-0.0025139588, 2.652036]\n",
            "It: 4300, Time: 12.91\n",
            "loss_value: 0.005133724771440029\n",
            "vars estimate(s): [-0.0022581508, 2.738012]\n",
            "It: 4400, Time: 12.93\n",
            "loss_value: 0.004741620272397995\n",
            "vars estimate(s): [-0.0021206615, 2.8228161]\n",
            "It: 4500, Time: 12.94\n",
            "loss_value: 0.004178678151220083\n",
            "vars estimate(s): [-0.0019131763, 2.9062984]\n",
            "It: 4600, Time: 12.93\n",
            "loss_value: 0.003925342112779617\n",
            "vars estimate(s): [-0.0017533816, 2.9884174]\n",
            "It: 4700, Time: 12.91\n",
            "loss_value: 0.0035584894940257072\n",
            "vars estimate(s): [-0.0016500854, 3.069014]\n",
            "It: 4800, Time: 12.92\n",
            "loss_value: 0.003214173950254917\n",
            "vars estimate(s): [-0.0014996348, 3.148158]\n",
            "It: 4900, Time: 12.89\n",
            "loss_value: 0.0027867075987160206\n",
            "vars estimate(s): [-0.0013996011, 3.2257025]\n",
            "It: 5000, Time: 12.90\n",
            "loss_value: 0.002608206123113632\n",
            "vars estimate(s): [-0.0012902982, 3.301675]\n",
            "It: 5100, Time: 12.91\n",
            "loss_value: 0.0022631422616541386\n",
            "vars estimate(s): [-0.0011477638, 3.3760426]\n",
            "It: 5200, Time: 12.93\n",
            "loss_value: 0.0021515032276511192\n",
            "vars estimate(s): [-0.0010775923, 3.448806]\n",
            "It: 5300, Time: 12.95\n",
            "loss_value: 0.001980316126719117\n",
            "vars estimate(s): [-0.0009895846, 3.519875]\n",
            "It: 5400, Time: 12.94\n",
            "loss_value: 0.0018771248869597912\n",
            "vars estimate(s): [-0.0009231522, 3.5893683]\n",
            "It: 5500, Time: 12.94\n",
            "loss_value: 0.0015271061565726995\n",
            "vars estimate(s): [-0.00082660286, 3.6571064]\n",
            "It: 5600, Time: 12.92\n",
            "loss_value: 0.0014098708052188158\n",
            "vars estimate(s): [-0.0008019494, 3.7231789]\n",
            "It: 5700, Time: 12.91\n",
            "loss_value: 0.0012902992311865091\n",
            "vars estimate(s): [-0.00072336633, 3.7875884]\n",
            "It: 5800, Time: 12.90\n",
            "loss_value: 0.0010898521868512034\n",
            "vars estimate(s): [-0.0006576429, 3.8502662]\n",
            "It: 5900, Time: 12.90\n",
            "loss_value: 0.0009451819933019578\n",
            "vars estimate(s): [-0.0005846137, 3.9111953]\n",
            "It: 6000, Time: 12.92\n",
            "loss_value: 0.001003153738565743\n",
            "vars estimate(s): [-0.0005646332, 3.970395]\n",
            "It: 6100, Time: 12.95\n",
            "loss_value: 0.0007862008060328662\n",
            "vars estimate(s): [-0.0005299207, 4.0278096]\n",
            "It: 6200, Time: 12.94\n",
            "loss_value: 0.0008246300276368856\n",
            "vars estimate(s): [-0.00047807922, 4.083355]\n",
            "It: 6300, Time: 12.94\n",
            "loss_value: 0.0007694094674661756\n",
            "vars estimate(s): [-0.00044345454, 4.137122]\n",
            "It: 6400, Time: 12.92\n",
            "loss_value: 0.0007125837146304548\n",
            "vars estimate(s): [-0.00040041146, 4.189056]\n",
            "It: 6500, Time: 12.92\n",
            "loss_value: 0.0004796195134986192\n",
            "vars estimate(s): [-0.00036877327, 4.239181]\n",
            "It: 6600, Time: 12.90\n",
            "loss_value: 0.0005615553236566484\n",
            "vars estimate(s): [-0.00034022314, 4.2874684]\n",
            "It: 6700, Time: 12.88\n",
            "loss_value: 0.0004679951525758952\n",
            "vars estimate(s): [-0.00030213484, 4.3339415]\n",
            "It: 6800, Time: 12.88\n",
            "loss_value: 0.0005126340547576547\n",
            "vars estimate(s): [-0.00029871086, 4.3784323]\n",
            "It: 6900, Time: 12.90\n",
            "loss_value: 0.0002738098264671862\n",
            "vars estimate(s): [-0.00023979768, 4.421053]\n",
            "It: 7000, Time: 12.93\n",
            "loss_value: 0.00024341960670426488\n",
            "vars estimate(s): [-0.00021052966, 4.461802]\n",
            "It: 7100, Time: 12.93\n",
            "loss_value: 0.0002844961127266288\n",
            "vars estimate(s): [-0.00018156627, 4.5005836]\n",
            "It: 7200, Time: 12.93\n",
            "loss_value: 0.0002443362318444997\n",
            "vars estimate(s): [-0.00019323097, 4.5374403]\n",
            "It: 7300, Time: 12.91\n",
            "loss_value: 0.0001886010868474841\n",
            "vars estimate(s): [-0.0001738636, 4.5723004]\n",
            "It: 7400, Time: 12.90\n",
            "loss_value: 0.00024154348648153245\n",
            "vars estimate(s): [-0.00016538939, 4.605233]\n",
            "It: 7500, Time: 12.90\n",
            "loss_value: 0.00039564940379932523\n",
            "vars estimate(s): [-0.00012854324, 4.636322]\n",
            "It: 7600, Time: 12.90\n",
            "loss_value: 0.0001188318565255031\n",
            "vars estimate(s): [-0.000119328135, 4.665328]\n",
            "It: 7700, Time: 12.90\n",
            "loss_value: 0.00014506373554468155\n",
            "vars estimate(s): [-0.000107196, 4.6927576]\n",
            "It: 7800, Time: 12.90\n",
            "loss_value: 0.0001257790718227625\n",
            "vars estimate(s): [-0.000106573585, 4.718137]\n",
            "It: 7900, Time: 12.92\n",
            "loss_value: 0.00013156107161194086\n",
            "vars estimate(s): [-9.033054e-05, 4.741745]\n",
            "It: 8000, Time: 12.92\n",
            "loss_value: 7.594846829306334e-05\n",
            "vars estimate(s): [-6.442948e-05, 4.7639236]\n",
            "It: 8100, Time: 12.92\n",
            "loss_value: 9.568499808665365e-05\n",
            "vars estimate(s): [-3.9073857e-05, 4.7841806]\n",
            "It: 8200, Time: 12.92\n",
            "loss_value: 8.8236149167642e-05\n",
            "vars estimate(s): [-3.468388e-05, 4.802824]\n",
            "It: 8300, Time: 12.90\n",
            "loss_value: 0.0002666582877282053\n",
            "vars estimate(s): [-2.9456329e-05, 4.8198843]\n",
            "It: 8400, Time: 12.88\n",
            "loss_value: 0.0002581477165222168\n",
            "vars estimate(s): [-3.4780307e-05, 4.8354645]\n",
            "It: 8500, Time: 12.88\n",
            "loss_value: 0.00026229859213344753\n",
            "vars estimate(s): [-2.8465836e-05, 4.8496614]\n",
            "It: 8600, Time: 12.88\n",
            "loss_value: 5.050301842857152e-05\n",
            "vars estimate(s): [-4.3039568e-06, 4.862691]\n",
            "It: 8700, Time: 12.91\n",
            "loss_value: 0.00022355523833539337\n",
            "vars estimate(s): [-1.4082484e-06, 4.874439]\n",
            "It: 8800, Time: 12.92\n",
            "loss_value: 5.70355259696953e-05\n",
            "vars estimate(s): [1.626091e-05, 4.8851094]\n",
            "It: 8900, Time: 12.93\n",
            "loss_value: 0.00018804465071298182\n",
            "vars estimate(s): [2.1002419e-05, 4.8946276]\n",
            "It: 9000, Time: 12.92\n",
            "loss_value: 6.564476643688977e-05\n",
            "vars estimate(s): [4.1860832e-05, 4.9030714]\n",
            "It: 9100, Time: 12.92\n",
            "loss_value: 2.7917347324546427e-05\n",
            "vars estimate(s): [4.2284235e-05, 4.9106417]\n",
            "It: 9200, Time: 12.92\n",
            "loss_value: 0.0001683023147052154\n",
            "vars estimate(s): [4.0679035e-05, 4.9170985]\n",
            "It: 9300, Time: 12.91\n",
            "loss_value: 0.0001970065204659477\n",
            "vars estimate(s): [7.5736716e-05, 4.9229345]\n",
            "It: 9400, Time: 12.91\n",
            "loss_value: 0.00019721266289707273\n",
            "vars estimate(s): [5.1340496e-05, 4.928307]\n",
            "It: 9500, Time: 12.92\n",
            "loss_value: 6.732447945978492e-05\n",
            "vars estimate(s): [8.38083e-05, 4.9331565]\n",
            "It: 9600, Time: 12.94\n",
            "loss_value: 0.0001885906094685197\n",
            "vars estimate(s): [6.784549e-05, 4.9376254]\n",
            "It: 9700, Time: 12.95\n",
            "loss_value: 0.00018842755525838584\n",
            "vars estimate(s): [0.00010123821, 4.9415946]\n",
            "It: 9800, Time: 12.96\n",
            "loss_value: 5.989978672005236e-05\n",
            "vars estimate(s): [7.9570214e-05, 4.945083]\n",
            "It: 9900, Time: 12.95\n",
            "loss_value: 0.0001880271447589621\n",
            "vars estimate(s): [8.867447e-05, 4.9483433]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPLwdB-E9cDy"
      },
      "source": [
        ""
      ],
      "execution_count": 11,
      "outputs": []
    }
  ]
}