Introduction_to_Python_and_PySpark_Spr2022.ipynb

{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "colab": {
   "name": "Introduction_to_Python_and_PySpark_Spr2022.ipynb",
   "provenance": [],
   "collapsed_sections": []
  },
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4A4BhKYXXTPY"
   },
   "source": [
    "# Introduction to Python and PySpark\n",
    "\n",
    "This is an additional lecture designed for ECE795 Advanced Big Data Analytics course. In this lecture, how to code with Python and PySpark will be discussed.\n",
    "TA: Anchen Sun (asun@miami.edu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d23XWE9CZF4k"
   },
   "source": [
    "## Python Introduction \n",
    "\n",
    "- Python is an open-source general-purpose programming language with powerful library and package support. \n",
    "\n",
    "- Python supports object-oriented, procedural and functional programming. \n",
    "\n",
    "- Python has great interactive environment and interfaces with Spark."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "8WAgFeKcR0-M"
   },
   "source": [
    "## Runing Python: The Python Interpreter\n",
    "\n",
    "### Print the version of Python"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "gDSfuXpLXNDn",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "95816075-947a-4ff1-be9d-69df83376ac1"
   },
   "source": [
    "!python --version"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Python 3.7.12\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UH6D2JWaZpgK"
   },
   "source": [
    "### Use Python as calculator"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "zJ_qGrHVZisR",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "9abbfa76-a8b2-459b-e655-50af1b2ca490"
   },
   "source": [
    "3 * (7 + 2)"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "27"
      ]
     },
     "metadata": {},
     "execution_count": 2
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SNwNMNGpXR8e"
   },
   "source": [
    "### A simple example\n",
    "\n",
    "1. Assignment operator: `=`\n",
    " \n",
    "2. Value comparison: `==`\n",
    "\n",
    "3. Basic arithmetic operators: add(`+`), minus(`-`), multiply(`*`), divide(`/`), and modulo(`%`).\n",
    "\n",
    "    - `+` is also used for string concatenation when the operands are of `str` type.\n",
    "\n",
    "    - `%` is also used for string formatting (as with printf in C)\n",
    "\n",
    "4. Logical operators are in words (`and`, `or`, `not`) but not symbols.\n",
    "\n",
    "5. The basic printing command is `print`."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "QYIfVMwWaBfn",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "7f25ecae-8873-46fb-a6eb-0642ef5f606b"
   },
   "source": [
    "x = 100 - 50                            # A comment\n",
    "y = 'University'                           # Another one\n",
    "z = 23.33\n",
    "if z == 23.33 or y == 'University':\n",
    "  x = x + 1\n",
    "  y = y + ' of Miami'                     # String concat\n",
    "print(x)\n",
    "print(y)"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "51\n",
      "University of Miami\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uXv9Fl03bGvd"
   },
   "source": [
    "### Basic Data Types\n",
    "\n",
    "Number"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "_Yyq4dbTajy5",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "7d6626b3-cce8-4694-d68b-28eabc2a97f9"
   },
   "source": [
    "# Integers (default for numbers)\n",
    "z = int(7 / 3)\n",
    "print(z)\n",
    "\n",
    "# Floating number which is the same as `double` in C\n",
    "z = float(7 / 3)\n",
    "print(z)"
   ],
   "execution_count": 15,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "2\n",
      "2.3333333333333335\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JYpusi50cnmx"
   },
   "source": [
    "Strings"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "CqSOcT2mco54",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "29d16276-ffa5-41ff-acdc-83e61046606a"
   },
   "source": [
    "# Can use \" \" or ' ' to specify.\n",
    "x = \"abc\"\n",
    "y = 'abc'\n",
    "print(x == y)\n",
    "\n",
    "# Unmatched quotes can occur within the string.\n",
    "z = \"matt's\"\n",
    "print(z)\n",
    "\n",
    "# Use triple double-quotes for multi-line strings or strings than contain both 'and \" inside of them:\n",
    "q = \"\"\"a'b\"c\"\"\"\n",
    "print(q)"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True\n",
      "matt's\n",
      "a'b\"c\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ciWRMPpRde0D"
   },
   "source": [
    "### Whitespace\n",
    "\n",
    "Whitespace is meaningful in Python!!!! Especially the indentation and placement of newlines\n",
    "\n",
    "Use a newline to end a line of code: Use \\ when must go to next line prematurely."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "8SGiYEvCdx7l",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "403a9e0f-128b-4779-9f19-cf736715f5e7"
   },
   "source": [
    "x = 1\n",
    "y = 2\n",
    "z = x + \\\n",
    "    y\n",
    "print(z)"
   ],
   "execution_count": 13,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "3\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1E_utZmLeFX8"
   },
   "source": [
    "### Comments\n",
    "\n",
    "Start comments with `#`: the rest of lines are ignored.\n",
    "\n",
    "Can include a \"\"\"documentation string\"\"\" as the first line of any new function or class that you define.\n",
    "\n",
    "The development environment, debugger, and other tools like help() use it: it’s good style to include one."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "U1b9DBGfeDHk"
   },
   "source": [
    "def my_function(x, y):\n",
    "  \"\"\"\n",
    "  This is the docstring. This function does ...\n",
    "\n",
    "  Attributes\n",
    "    x (int): number of ...\n",
    "    y (string): ...\n",
    "  \"\"\"\n",
    "  # The code would go here..."
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bfmi3Al1e_pH"
   },
   "source": [
    "### Multiple Simultaneous Assignments"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "j9L7AzmffAQd",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "5f5b7704-1ca1-4bbd-e67c-2d0fba02b03c"
   },
   "source": [
    "x, y = 1, 100\n",
    "print(x + y)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "101\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "5T9gX0MSR0-S"
   },
   "source": [
    "### Reserved words\n",
    "\n",
    "and, assert, break, class, continue, def, del, elif, else, except, exec, finally,\n",
    "for, from, global, if, import, in, is, lambda, not, or, pass, print, raise, return,\n",
    "try, while"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RwYc-xKQfWV1"
   },
   "source": [
    "### Basic Operators\n",
    "\n",
    "Binary operators on numbers"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Af_LlOIWfY6c",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "5e407d7c-f051-4330-e74f-a73f6eed39dd"
   },
   "source": [
    "x = 95\n",
    "y = 7\n",
    "print(x + y, x - y, x / y, x * y, 2 ** y, x % 7, x // 7)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "102 88 13.571428571428571 665 128 4 13\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "8ajZPAjQge0o",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "outputId": "d88407af-c2b0-4614-94a6-2e0b39d454d3"
   },
   "source": [
    "# Some overloaded to work on strings\n",
    "print('University' + ' of ' + 'Miami')   # concatenation\n",
    "print('I' + 3 * 'E')                     # replication\n",
    "print(eval('1 + 4 / 3'))"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "University of Miami\n",
      "IEEE\n",
      "2.333333333333333\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o6T7ecQDhj_M"
   },
   "source": [
    "### Advanced Data Types\n",
    "\n",
    "1. Tuple\n",
    "    \n",
    "    - A simple immutable ordered sequence of items\n",
    "    \n",
    "    - Items can be of mixed types, including collections\n",
    "\n",
    "2. String\n",
    "\n",
    "    - Immutable\n",
    "\n",
    "    - Conceptually very much like a tuple\n",
    "\n",
    "3. List\n",
    "\n",
    "    - Mutable ordered sequence of items of mixed types\n",
    "\n",
    "All three sequence types (tuples, strings, and lists) share\n",
    "much of the same syntax and functionality.\n",
    "\n",
    "Key difference:\n",
    "\n",
    "1. Tuples and strings are immutable\n",
    "\n",
    "2. Lists are mutable\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "kLwGDYmdR0-T"
   },
   "source": [
    "Lists are defined using square brackets (and commas)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "p61IuJWEkKVx",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "0f3d0ef3-dcea-4dbd-cf1d-6e370a2042f3"
   },
   "source": [
    "L = ['ECE', 795, 'Advanced Big Data Analytics']\n",
    "print(L[0])"
   ],
   "execution_count": 10,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "ECE\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "KozSpkbqlFp9",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 165
    },
    "outputId": "bede2dab-b539-4117-c96f-75f29056f0c1"
   },
   "source": [
    "print(L[3])"
   ],
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "error",
     "ename": "IndexError",
     "evalue": "ignored",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mIndexError\u001B[0m                                Traceback (most recent call last)",
      "\u001B[0;32m<ipython-input-11-4d6e94a9c06a>\u001B[0m in \u001B[0;36m<module>\u001B[0;34m()\u001B[0m\n\u001B[0;32m----> 1\u001B[0;31m \u001B[0mprint\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mL\u001B[0m\u001B[0;34m[\u001B[0m\u001B[0;36m3\u001B[0m\u001B[0;34m]\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m",
      "\u001B[0;31mIndexError\u001B[0m: list index out of range"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "nNkNL_lPlJbo",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "a635bd7e-ffca-43c1-9386-28b63be0013c"
   },
   "source": [
    "print(L[-1])"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Advanced Big Data Analytics\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JdHbrnqM2Ija"
   },
   "source": [
    "Mutability: Lists are Mutable and Operations Only on Lists"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "yf_u_tQ9lQ9U",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "420bec93-01f6-4db9-9160-8e9b13e4ab75"
   },
   "source": [
    "L.append('College of Engineering')\n",
    "print(L)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "['ECE', 795, 'Advanced Big Data', 'College of Engineering']\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Q8EwlJATlaRb",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "9ac17bce-5521-422d-d753-591f488a7530"
   },
   "source": [
    "L.insert(3, 'Dr. Shyu')\n",
    "print(L)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "['ECE', 795, 'Advanced Big Data', 'Dr. Shyu', 'College of Engineering']\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "-v7oMZ5_liFs",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "9b1039ee-d155-4237-944e-696aa2b39b42"
   },
   "source": [
    "L.pop()\n",
    "print(L)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "['ECE', 795, 'Advanced Big Data', 'Dr. Shyu']\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "SUZrcM9OloAm",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "a5661343-ff37-45e2-c88e-c2a328788f51"
   },
   "source": [
    "L.pop(2)\n",
    "print(L)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "['ECE', 795, 'Dr. Shyu']\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ozm4Irq0l-2f",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "246d009c-cf53-41d1-a4fd-f89da8f41e2b"
   },
   "source": [
    "L[2] = 'College of Engineering'\n",
    "print(L)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "['ECE', 795, 'College of Engineering']\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uh5PyuC016EP"
   },
   "source": [
    "Mutability: Tuples are Immutable"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Yb4lLgW4mG_r",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "3a4af020-921b-4b9f-9807-fa79325ae52b"
   },
   "source": [
    "# Tuples are defined using parentheses (and commas).\n",
    "T = ('ECE', 795, 'Advanced Big Data Analytics')\n",
    "print(T)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "('ECE', 795, 'Advanced Big Data Analytics')\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "y0qU04gfvP8i",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 130
    },
    "outputId": "caf5121d-e8a4-4164-aeab-461c40204650"
   },
   "source": [
    "T(1) = 800"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "error",
     "ename": "SyntaxError",
     "evalue": "ignored",
     "traceback": [
      "\u001B[0;36m  File \u001B[0;32m\"<ipython-input-7-51263593ab7c>\"\u001B[0;36m, line \u001B[0;32m1\u001B[0m\n\u001B[0;31m    T(1) = 800\u001B[0m\n\u001B[0m              ^\u001B[0m\n\u001B[0;31mSyntaxError\u001B[0m\u001B[0;31m:\u001B[0m can't assign to function call\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "sY4TiSibvW-S",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "efae1b68-823b-43db-c545-231457992e85"
   },
   "source": [
    "T = ('ECE', 795, 'Advanced Big Data Analytics', ['a', 'b', 'c'])\n",
    "print(T)\n",
    "L = T[3]\n",
    "print(L)\n",
    "L[0] = 1\n",
    "L[1] = 2\n",
    "print(T)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "('ECE', 795, 'Advanced Big Data Analytics', ['a', 'b', 'c'])\n",
      "['a', 'b', 'c']\n",
      "('ECE', 795, 'Advanced Big Data Analytics', [1, 2, 'c'])\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "sc2XzfVmR0-W"
   },
   "source": [
    "Slicing: Return a Copy of a Subset"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "-Ol4oPCcx0fD",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "outputId": "ce2b75ce-8e93-41c8-d5f8-452e6b2a22d0"
   },
   "source": [
    "print(T[1:3])\n",
    "print(T[:3])\n",
    "print(T[2:])"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "(795, 'Advanced Big Data')\n",
      "('ECE', 795, 'Advanced Big Data')\n",
      "('Advanced Big Data', [1, 2, 'c'])\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "PFGbbT6BR0-W"
   },
   "source": [
    "Copy the Entire Sequence"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "IgOTmlXvyIqm",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "outputId": "d2adf1b1-ed3d-4366-cc8c-dc38007b4ab6"
   },
   "source": [
    "A = [1, 2, 3, 4]\n",
    "B = A\n",
    "B[1] = 100\n",
    "print(A, B)\n",
    "\n",
    "A = [1, 2, 3, 4]\n",
    "B = A[:]\n",
    "B[1] = 100\n",
    "print(A, B)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[1, 100, 3, 4] [1, 100, 3, 4]\n",
      "[1, 2, 3, 4] [1, 100, 3, 4]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "_2dZYmHywEs9",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "0a8b2584-202b-4dd3-9959-6bf08f508b65"
   },
   "source": [
    "St = 'ECE 795 Advanced Big Data Analytics'\n",
    "print(St[2])"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "E\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IkE-niV0zE5F"
   },
   "source": [
    "### Membership test\n",
    "\n",
    "Boolean test whether a value is inside a container:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "I_MGLsF9zJf5",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "outputId": "abeaf30a-58c2-4301-fbf3-7ba66c037c02"
   },
   "source": [
    "T = [1, 2, 3]\n",
    "print(3 in T)\n",
    "print(4 in T)\n",
    "print(4 not in T)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n",
      "True\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k8jFcrRUzTAe"
   },
   "source": [
    "For strings, also tests for substrings"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "B3TqImZJzUj5",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "outputId": "2925c663-4d8e-4dc5-d3f6-b74e843084f1"
   },
   "source": [
    "st = 'University of Miami'\n",
    "print('Miami' in st)\n",
    "print('M' in st)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G3kg-g1dzz8x"
   },
   "source": [
    "### Replication\n",
    "\n",
    "The * operator produces a new tuple, list, or string that repeats the original content"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "wRBeCc0qz70Y",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "outputId": "022432c9-12ac-4d4a-9015-f484813dc0ce"
   },
   "source": [
    "print((1, 2) * 3)\n",
    "print([1, 2] * 3)\n",
    "print('Miami' * 3)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "(1, 2, 1, 2, 1, 2)\n",
      "[1, 2, 1, 2, 1, 2]\n",
      "MiamiMiamiMiami\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V3nqRAff2Z09"
   },
   "source": [
    "Lists made out of other lists: List Comprehension"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "V-QWz2Kp2c-W",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "1ad42d7a-3ebe-4b17-86ff-8a374a033698"
   },
   "source": [
    "li = range(5)\n",
    "print(li)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "range(0, 5)\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Q0Xm7ICB2tz3",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "d4075f19-a956-4383-8333-20d1f66b1fd2"
   },
   "source": [
    "squares = [x ** 2 for x in li]\n",
    "print(squares)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[0, 1, 4, 9, 16]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "GTiTWbuo24ea",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "19c73b38-9418-4fbd-8db4-cce945f5b84c"
   },
   "source": [
    "even_squares = [x for x in squares if x%2==0]\n",
    "print(even_squares)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[0, 4, 16]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "ZXw8fRA13Gmv",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "7aff8e17-c8af-4db7-ed2a-99ae29682f82"
   },
   "source": [
    "newlist = [(x,1) for x in even_squares]\n",
    "print(newlist)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[(0, 1), (4, 1), (16, 1)]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "hk991wjV3T0c",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "cdca3494-354c-440e-c196-e65dfb2cb2d0"
   },
   "source": [
    "newlist = [range(x) for x in range(5)]\n",
    "print(newlist)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[range(0, 0), range(0, 1), range(0, 2), range(0, 3), range(0, 4)]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "bXXj3W-P3b0N",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "4900a2f0-a5d6-4e6c-f45d-9f4fe79f66ba"
   },
   "source": [
    "newlist = [y for x in range(5) for y in range(x)] #\"nested\" for loop\n",
    "print(newlist)"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[0, 0, 1, 0, 1, 2, 0, 1, 2, 3]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pej3sXSQ5giP"
   },
   "source": [
    "### Control Flow\n",
    "if statement"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "sCvxbp6q5qIe",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "caddb3ce-4537-4dc6-a98b-3f8eaac6f16a"
   },
   "source": [
    "x = 2\n",
    "if x == 1:\n",
    "  print('x = 1')\n",
    "elif x == 2:\n",
    "  print('x = 2')\n",
    "else:\n",
    "  print('x equals something else')"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "x = 2\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "OsiuaXre58vu",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "006cf768-5b42-48bf-a206-0908bad94fd0"
   },
   "source": [
    "x = 'abc'\n",
    "if 'a' in x:\n",
    "  print('a is in string x')\n",
    "else:\n",
    "  print('a is not in string x')"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "a is in string x\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a5dpp5ay5d6g"
   },
   "source": [
    "while statement: Fibonacci Series:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Z7oYQjQ16XFK",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "outputId": "e4b7c7a6-062d-40c5-a0cd-1d8f3d1364c3"
   },
   "source": [
    "a, b = 0, 1\n",
    "while a < 100:\n",
    "  print(a)\n",
    "  a, b = b, a + b"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "1\n",
      "2\n",
      "3\n",
      "5\n",
      "8\n",
      "13\n",
      "21\n",
      "34\n",
      "55\n",
      "89\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YmByV5uB7Tmu"
   },
   "source": [
    "for loop: Primality Test"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "BByp-wmD7ZMk",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "outputId": "2266449c-a5d0-4b5c-cbd0-5f1d929b0272"
   },
   "source": [
    "for n in range(2, 10):\n",
    "  for x in range(2, n):\n",
    "    if n % x == 0:\n",
    "      print(n, 'equals', x, '*', n//x)\n",
    "      break\n",
    "  else:    #executed only if loop does not break early\n",
    "    print(n, 'is a prime number')"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "2 is a prime number\n",
      "3 is a prime number\n",
      "4 equals 2 * 2\n",
      "5 is a prime number\n",
      "6 equals 2 * 3\n",
      "7 is a prime number\n",
      "8 equals 2 * 4\n",
      "9 equals 3 * 3\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bYaNJVWY80v3"
   },
   "source": [
    "### Function\n",
    "\n",
    "Function Definitions\n",
    "\n",
    "1. def creates a function and assigns it a name\n",
    "\n",
    "2. return sends a result back to the caller\n",
    "\n",
    "3. Arguments are passed by assignment\n",
    "\n",
    "4. Arguments and return types are not declared:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Jv1FMkBs9BP4",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "74b2d1cc-3d72-4e5b-bf6e-2b2266a0fd57"
   },
   "source": [
    "def mean(Num):\n",
    "  sum = 0\n",
    "  for each in Num:\n",
    "    sum += each\n",
    "  return sum / len(Num)\n",
    "\n",
    "print(mean([1, 2, 3, 9, 10])) "
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "5.0\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i4YGWaV_-A4f"
   },
   "source": [
    "### Python Modules\n",
    "\n",
    "Modules comprise functions and variables defined in separate files\n",
    "\n",
    "Functions or variables from a module are imported using from or import"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "iUs4YKup-ILd",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "a8fb163f-3d00-479a-ab26-74264839b10f"
   },
   "source": [
    "import numpy\n",
    "print(numpy.sqrt(144))"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "12.0\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "1ZWec3D3-Yeg",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "6bf872e3-11dd-4d0b-e9aa-0e7723609590"
   },
   "source": [
    "from numpy import sqrt\n",
    "print(sqrt(144))"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "12.0\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "KIGiCHjT-fmT",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "outputId": "991e574b-eca1-4376-d75b-d42dafa899ff"
   },
   "source": [
    "import numpy as np\n",
    "print(np.sqrt(144))"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "12.0\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "S9VosWzr-lUH",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "outputId": "03cbb569-1c85-48fb-925c-b218bd427580"
   },
   "source": [
    "from numpy import *\n",
    "print(sqrt(144))\n",
    "print(exp(2))"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "12.0\n",
      "7.38905609893065\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "om69PjwV_Ic9",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "outputId": "287c055f-6ee7-4e20-f245-ad916f20c366"
   },
   "source": [
    "import numpy as np\n",
    "a = np.array([1, 2, 3])\n",
    "b = np.array([10, 20, 30])\n",
    "print(a + b)\n",
    "print(np.dot(a, b))\n",
    "print(np.outer(a, b))"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[11 22 33]\n",
      "140\n",
      "[[10 20 30]\n",
      " [20 40 60]\n",
      " [30 60 90]]\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DWnzGf5L_sLU"
   },
   "source": [
    "## PySpark Introdction\n",
    "\n",
    "What is Spark?\n",
    "\n",
    "A general engine for large-scale data processing\n",
    "\n",
    "Fast, expressive cluster computing system compatible\n",
    "with Apache Hadoop\n",
    "\n",
    "Improves efficiency through:\n",
    "\n",
    "1. In-memory computing primitives\n",
    "\n",
    "2. General computation graphs\n",
    "\n",
    "Improves usability through:\n",
    "\n",
    "1. Rich APIs in Java, Scala, Python\n",
    "\n",
    "2. Interactive shell\n",
    "\n",
    "Programmer’s point of view:\n",
    "\n",
    "“Normal” python program, with “special” parallel data\n",
    "structures\n",
    "\n",
    "Easy to write short, clear map/reduce operations\n",
    "\n",
    "Full expressibility of java/scala/python “included”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cop8JREuB4ud"
   },
   "source": [
    "### Qucik Tour of Operations\n",
    "Starting point for spark functionalities\n",
    "\n",
    "In pyspark interpreter, automatically loaded\n",
    "variable:\n",
    "\n",
    "`sc`\n",
    "\n",
    "In standalone programs, you create your own\n",
    "\n",
    "Creating an RDD"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "#Setup PySpark\n",
    "!pip install pyspark\n",
    "!pip install findspark\n",
    "\n",
    "import findspark\n",
    "findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.master(\"local[*]\").getOrCreate()\n",
    "\n",
    "from pyspark import SparkConf, SparkContext\n",
    "sc = SparkContext.getOrCreate()"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zB7E0qwl-7k5",
    "outputId": "826aca79-30d7-41b0-c9a0-f53ee7c9a815"
   },
   "execution_count": 4,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Collecting pyspark\n",
      "  Downloading pyspark-3.2.1.tar.gz (281.4 MB)\n",
      "\u001B[K     |████████████████████████████████| 281.4 MB 35 kB/s \n",
      "\u001B[?25hCollecting py4j==0.10.9.3\n",
      "  Downloading py4j-0.10.9.3-py2.py3-none-any.whl (198 kB)\n",
      "\u001B[K     |████████████████████████████████| 198 kB 51.1 MB/s \n",
      "\u001B[?25hBuilding wheels for collected packages: pyspark\n",
      "  Building wheel for pyspark (setup.py) ... \u001B[?25l\u001B[?25hdone\n",
      "  Created wheel for pyspark: filename=pyspark-3.2.1-py2.py3-none-any.whl size=281853642 sha256=f0109fa61dd89a7035bec164bcdf8ede221ba8d5dfc95cc4a6523cb00f198ab6\n",
      "  Stored in directory: /root/.cache/pip/wheels/9f/f5/07/7cd8017084dce4e93e84e92efd1e1d5334db05f2e83bcef74f\n",
      "Successfully built pyspark\n",
      "Installing collected packages: py4j, pyspark\n",
      "Successfully installed py4j-0.10.9.3 pyspark-3.2.1\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "M63SR-u0CwgL",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "a19564f4-8c95-4162-8bee-0ffc814e5475"
   },
   "source": [
    "# Turn a local collection into an RDD\n",
    "rdd = sc.parallelize([1, 2, 3])\n",
    "print(rdd.take(2))"
   ],
   "execution_count": 7,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[1, 2]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "zE6uULqeDpMS",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "0f641ce8-0659-49e9-ca20-fcd335b558f2"
   },
   "source": [
    "rdd = sc.textFile('/content/sample_data/california_housing_test.csv')\n",
    "print(rdd.take(2))"
   ],
   "execution_count": 6,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['\"longitude\",\"latitude\",\"housing_median_age\",\"total_rooms\",\"total_bedrooms\",\"population\",\"households\",\"median_income\",\"median_house_value\"', '-122.050000,37.370000,27.000000,3885.000000,661.000000,1537.000000,606.000000,6.608500,344700.000000']\n"
     ]
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4gHVEmGYEaVC"
   },
   "source": [
    "Elements in the list are lines of the file\n",
    "\n",
    "The list is distributed over multiple machines\n",
    "\n",
    "The list is immutable\n",
    "\n",
    "1. You cannot change it,\n",
    "\n",
    "2. Cannot see individual elements (e.g., 5th element)\n",
    "\n",
    "3. Can only interact with it through specific ops"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "Vvm5MbVFEtzS",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "587e55dc-efe1-42f4-8a4e-e9857b3a2c97"
   },
   "source": [
    "#Basic Transformations\n",
    "nums = sc.parallelize([1, 2, 3])\n",
    "\n",
    "# Pass each element through a function\n",
    "squares = nums.map(lambda x: x * x)\n",
    "print(squares.take(10))\n",
    "\n",
    "# Keep elements passing a predicate\n",
    "even = squares.filter(lambda x: x % 2 == 0)\n",
    "print(even.take(10))"
   ],
   "execution_count": 8,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[1, 4, 9]\n",
      "[4]\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "2v3Mt2u2FGrh",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "ac6cc22b-8d7b-4a37-c752-e3a8ec62f483"
   },
   "source": [
    "#Basic Actions\n",
    "nums = sc.parallelize([1, 2, 3])\n",
    "\n",
    "# Retrieve RDD contents as a local collection\n",
    "print(nums.collect())\n",
    "\n",
    "# Count number of elements\n",
    "print(nums.count())\n",
    "\n",
    "# Write elements to a text file\n",
    "#nums.saveAsTextFile('file.txt')"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[1, 2, 3]\n",
      "3\n"
     ]
    }
   ]
  }
 ]
}