model_training_and_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1cfc1267-442f-4c79-aad2-bdc706fbf4f5",
   "metadata": {},
   "source": [
    "# Gun Detection System for Urban Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efc608c4-6cf4-4953-9285-6cfa024b43e9",
   "metadata": {},
   "source": [
    "This notebook contains the implementation of a **gun detection system** for the detection of gunshots sounds in urban settings. The system is designed to detect gunshots by analysing audio data and uses machine learning algorithms to classify the sounds. The proposed system has several potential applications, including the ability to provide real-time information to law enforcement agencies about the location and nature of gunshots. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5fc624a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import librosa\n",
    "import numpy as np\n",
    "from numpy import nan\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "import math\n",
    "from PIL import Image\n",
    "import os\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "matplotlib.use(\"Agg\")\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "110e5c5f-e49a-44bb-bc3b-537770b74e71",
   "metadata": {},
   "source": [
    "First, some constant values are set to be used across the entire notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6183ca5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "IMG_DIR = \"./mel-specs\"\n",
    "VAL_DIR = \"./mel-specs-val\"\n",
    "TEST_DIR = \"./mel-specs-test\"\n",
    "\n",
    "SAMPLE_RATE = 22050 # Justification for this value is provided later"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece1f1ca",
   "metadata": {},
   "source": [
    "## Training, validation & testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd9ed2c7-9c0f-42ca-9d92-678b130bae55",
   "metadata": {},
   "source": [
    "In this section both proposed approaches will be implemented using different models for comparison."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e906280b-7e8d-41d7-a146-1cfda3861307",
   "metadata": {},
   "source": [
    "***For both of the following approaches, the training data is split equally into ten classes, with only one of them being the target class (gun_shot). Treating this as a binary classification problem by considering only one class as true and all others as false might not result in good classification accuracy, as the correct class will be underrepresented. Therefore, it is better to treat this as a normal classification problem with ten categories. After the classifier predicts the class, the output will be checked for whether it is a gunshot or not. If the predicted class is a gunshot, then it will be considered true; otherwise, it will be considered false.***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a6895bc-89e0-4d7e-8428-b8b3905b3dc5",
   "metadata": {},
   "source": [
    "Tensorflow 2.11 will be used in this project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "28b528a9-fce7-4fe1-a486-67646accf95e",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorflow==2.11.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (2.11.0)\n",
      "Requirement already satisfied: flatbuffers>=2.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (2.0)\n",
      "Requirement already satisfied: keras<2.12,>=2.11.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (2.11.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (0.2.0)\n",
      "Requirement already satisfied: setuptools in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (58.0.4)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (3.3.0)\n",
      "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (0.3.3)\n",
      "Requirement already satisfied: protobuf<3.20,>=3.9.2 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (3.19.1)\n",
      "Requirement already satisfied: tensorflow-estimator<2.12,>=2.11.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (2.11.0)\n",
      "Requirement already satisfied: numpy>=1.20 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.21.6)\n",
      "Requirement already satisfied: h5py>=2.9.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (2.10.0)\n",
      "Requirement already satisfied: six>=1.12.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.16.0)\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.42.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (4.5.0)\n",
      "Requirement already satisfied: libclang>=13.0.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (16.0.0)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (0.32.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.6.3)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.13.3)\n",
      "Requirement already satisfied: absl-py>=1.0.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.4.0)\n",
      "Requirement already satisfied: packaging in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (21.3)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (1.1.0)\n",
      "Requirement already satisfied: tensorboard<2.12,>=2.11 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorflow==2.11.0) (2.11.2)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow==2.11.0) (0.37.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (0.6.1)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (0.4.4)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (2.26.0)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (1.6.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (3.3.4)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (2.0.2)\n",
      "Requirement already satisfied: google-auth<3,>=1.6.3 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from tensorboard<2.12,>=2.11->tensorflow==2.11.0) (1.33.0)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (4.2.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (0.2.8)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (4.7.2)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (1.3.0)\n",
      "Requirement already satisfied: importlib-metadata in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (4.10.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (0.4.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (3.3)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (2.0.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (1.26.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (2021.10.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (3.1.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<2.12,>=2.11->tensorflow==2.11.0) (3.7.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages (from packaging->tensorflow==2.11.0) (3.0.4)\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow==2.11.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "29e315fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "474e1f70-ab2b-4cd2-a993-68b83b48844d",
   "metadata": {},
   "source": [
    "The following functions will be used for the evaluation and comparison of the different models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e4ffdfe8-130c-4554-b311-e0a375d2e986",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "def _sensitivity(y_actual, y_pred):\n",
    "    \"\"\"Calculate the sensitivity score per class for a model\"\"\"\n",
    "    cm = confusion_matrix(y_actual, y_pred)\n",
    "    FN = cm[1, 0]\n",
    "    TP = cm[1, 1]\n",
    "    \n",
    "    if(not TP): # avoid Nan values if both TP and FN are 0s\n",
    "        sensitivity = 0.0\n",
    "    else:\n",
    "        sensitivity = (TP / (TP + FN)).round(2)\n",
    "        \n",
    "    return sensitivity\n",
    "\n",
    "def _specificity(y_actual, y_pred):\n",
    "    \"\"\"Calculate the specificity score per class for a model\"\"\"\n",
    "    cm = confusion_matrix(y_actual, y_pred)\n",
    "    TN = cm[0, 0]\n",
    "    FP = cm[0, 1]\n",
    "    \n",
    "    if(not TN): # avoid Nan values if both TN and FP are 0s\n",
    "        specificity = 0.0\n",
    "    else:\n",
    "        specificity = (TN / (TN + FP)).round(2)\n",
    "        \n",
    "    return specificity\n",
    "\n",
    "def report(y_actual, y_pred):\n",
    "    \"\"\"Print a report with all evaluation metrics for a model\"\"\"\n",
    "    print(classification_report(y_actual, y_pred))\n",
    "    sensitivity = _sensitivity(y_actual, y_pred)\n",
    "    print('Sensitivity: ', sensitivity)\n",
    "    specificity = _specificity(y_actual, y_pred)\n",
    "    print('Specificity: ', specificity)\n",
    "    \n",
    "    cm_matrix = confusion_matrix(y_actual,y_pred)\n",
    "    sns.heatmap(cm_matrix, annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71310969-67c3-41c0-9df7-dd3225c14cf2",
   "metadata": {},
   "source": [
    "### Approach I"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91017f65-8eb6-40f1-b00f-5be5f896e721",
   "metadata": {},
   "source": [
    "This approach involves the classification of mel-spectrogram images representing the audio slices. This moves this task from audio classification into the image classification paradigm. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fedde71-18d6-4851-bb9d-944bb9e8a3b6",
   "metadata": {},
   "source": [
    "For this approach, a 2D deep convolutional neural network (CNN) will be trained on the extracted mel-spectrogram images. To achive this TensorFlow with Keras will be used to construct, train, validate and test the CNN model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f54880b3-75e6-4619-a1d8-b6ca2c7d089b",
   "metadata": {},
   "source": [
    "The training, validation and test sets will be loaded from each respective images folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1e8e9b23-ed3c-4111-85b3-a94c02786719",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 7000 files belonging to 10 classes.\n"
     ]
    }
   ],
   "source": [
    "'''The training images set'''\n",
    "train_ds = tf.keras.utils.image_dataset_from_directory(\n",
    "    IMG_DIR,\n",
    "    labels=\"inferred\",\n",
    "    batch_size=1,\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "03cff9a2-2daa-4026-8c87-65ad49e05026",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 812 files belonging to 10 classes.\n"
     ]
    }
   ],
   "source": [
    "'''The validation images set'''\n",
    "val_ds = tf.keras.utils.image_dataset_from_directory(\n",
    "    VAL_DIR,\n",
    "    labels=\"inferred\",\n",
    "    batch_size=1,\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d08d2108-8ea5-424e-841a-5b0846db49b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 902 files belonging to 10 classes.\n"
     ]
    }
   ],
   "source": [
    "'''The testing images set'''\n",
    "test_ds = tf.keras.utils.image_dataset_from_directory(\n",
    "    TEST_DIR,\n",
    "    labels=\"inferred\",\n",
    "    batch_size=1,\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ea476a68-58af-4594-9675-a8e5d2a5ad34",
   "metadata": {},
   "outputs": [],
   "source": [
    "for example_spectrograms, example_spect_labels in train_ds.take(1):\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdac30fa-9d1f-4d30-a711-c3679eb459ca",
   "metadata": {},
   "source": [
    "The following CNN structure will be used. The CNN takes as an input a 3 channel RGB images. The input layer takes in the image in the original size, then the following layer resizes the images into 32 x 32. Resizing the image reduces the computational complexity of the model. Then, the next layer applies normalization to the image to standardize it. The output then passes through 2 convolutional layers with 16 and 32 filters, respectively. The activation function used is `ReLU`. A max pooling layer then performs a 2x2 pooling operation to reduce the spatial dimensions of the output followed by a dropout layer that randomly drops out 25% of the connections to prevent overfitting. The output is flattened and inputed into a dense layer with 128 nodes. The output goes through another dropout layer with 50% before reaching the last dense layer with ten nodes representing the 10 output classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e2169c7e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/msc1/anaconda3/envs/Env-7146COMP/lib/python3.7/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.\n",
      "Instructions for updating:\n",
      "Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089\n"
     ]
    }
   ],
   "source": [
    "input_shape = example_spectrograms.shape[1:]\n",
    "num_classes = 10\n",
    "\n",
    "norm_layer = tf.keras.layers.Normalization()\n",
    "norm_layer.adapt(data=train_ds.map(map_func=lambda spec, label: spec))\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Input(shape=input_shape),\n",
    "    tf.keras.layers.Resizing(32, 32),\n",
    "    norm_layer,\n",
    "    tf.keras.layers.Conv2D(16, 3, activation='relu'),\n",
    "    tf.keras.layers.Conv2D(32, 3, activation='relu'),\n",
    "    tf.keras.layers.MaxPooling2D(),\n",
    "    tf.keras.layers.Dropout(0.25),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(128, activation='relu'),\n",
    "    tf.keras.layers.Dropout(0.5),\n",
    "    tf.keras.layers.Dense(num_classes),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877a0e12-86d3-4666-a181-b8ca29ce4469",
   "metadata": {},
   "source": [
    "<br/> The model is compiled using adam optimizer and accuracy as evaluation metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "87a13e50-dbdb-4e0b-9c1f-14350929f2e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "  optimizer='adam',\n",
    "  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "  metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6130d8da-3770-4eef-9f77-6618cbcc7d02",
   "metadata": {},
   "source": [
    "The model is trained with 15 epochs. The validation split will be used for fine tunning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5d824e94-1adf-4196-9898-997f78930e42",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-04-07 01:42:33.761602: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape insequential/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer\n",
      "2023-04-07 01:42:34.214479: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:428] Loaded cuDNN version 8100\n",
      "2023-04-07 01:42:35.084285: I tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:630] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.\n",
      "2023-04-07 01:42:35.085275: I tensorflow/compiler/xla/service/service.cc:173] XLA service 0x7fbebe264f00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
      "2023-04-07 01:42:35.085285: I tensorflow/compiler/xla/service/service.cc:181]   StreamExecutor device (0): NVIDIA GeForce RTX 3090, Compute Capability 8.6\n",
      "2023-04-07 01:42:35.087933: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n",
      "2023-04-07 01:42:35.176628: I tensorflow/compiler/jit/xla_compilation_cache.cc:477] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7000/7000 [==============================] - 23s 3ms/step - loss: 1.5077 - accuracy: 0.4991 - val_loss: 1.0758 - val_accuracy: 0.6330\n",
      "Epoch 2/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 1.0686 - accuracy: 0.6484 - val_loss: 1.0033 - val_accuracy: 0.6810\n",
      "Epoch 3/15\n",
      "7000/7000 [==============================] - 20s 3ms/step - loss: 0.8873 - accuracy: 0.7103 - val_loss: 0.8485 - val_accuracy: 0.7229\n",
      "Epoch 4/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.7513 - accuracy: 0.7537 - val_loss: 0.8775 - val_accuracy: 0.7217\n",
      "Epoch 5/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.6499 - accuracy: 0.7900 - val_loss: 1.0231 - val_accuracy: 0.7241\n",
      "Epoch 6/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.5874 - accuracy: 0.8084 - val_loss: 0.9869 - val_accuracy: 0.7414\n",
      "Epoch 7/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.5593 - accuracy: 0.8257 - val_loss: 0.9460 - val_accuracy: 0.7438\n",
      "Epoch 8/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.5223 - accuracy: 0.8314 - val_loss: 1.0155 - val_accuracy: 0.7438\n",
      "Epoch 9/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.5048 - accuracy: 0.8451 - val_loss: 0.9818 - val_accuracy: 0.7562\n",
      "Epoch 10/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.4581 - accuracy: 0.8557 - val_loss: 1.0949 - val_accuracy: 0.7537\n",
      "Epoch 11/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.4641 - accuracy: 0.8579 - val_loss: 0.9873 - val_accuracy: 0.7451\n",
      "Epoch 12/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.4779 - accuracy: 0.8516 - val_loss: 1.1526 - val_accuracy: 0.7488\n",
      "Epoch 13/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.4326 - accuracy: 0.8716 - val_loss: 1.1234 - val_accuracy: 0.7451\n",
      "Epoch 14/15\n",
      "7000/7000 [==============================] - 21s 3ms/step - loss: 0.4362 - accuracy: 0.8689 - val_loss: 1.4347 - val_accuracy: 0.7475\n",
      "Epoch 15/15\n",
      "7000/7000 [==============================] - 20s 3ms/step - loss: 0.4227 - accuracy: 0.8733 - val_loss: 1.0884 - val_accuracy: 0.7525\n"
     ]
    }
   ],
   "source": [
    "model.fit(\n",
    "  train_ds,\n",
    "  validation_data=val_ds,\n",
    "  epochs=15\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b6a710c-f31f-46c2-921e-416beabdf684",
   "metadata": {},
   "source": [
    "<br/> While training the model, the highest validation accuracy achieved was 75% reached on the 9th epoch with a loss of 0.98. Afterwards, the validation accuracy remained almost the same. The average accuracy of the last 5 epochs was **74.78%**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e2d0893-a7ab-44aa-9b0a-3746f5a0bb29",
   "metadata": {},
   "source": [
    "To get the true accuracy of the model, it is tested against the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6cead682-1e3f-4d69-a365-6937d0ad2458",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "902/902 [==============================] - 1s 963us/step - loss: 1.1927 - accuracy: 0.7328\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'loss': 1.1927478313446045, 'accuracy': 0.7328159809112549}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(test_ds, return_dict=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c82ed21-e544-456f-8d51-913180139e47",
   "metadata": {},
   "source": [
    "The model scored an accuracy of 73.28% on the test which is an adequate accuracy. However, this value reflects the accuracy of the model across all classes. For the current task, the model should be evaluated on its ability to differentiate between gunshot sounds and all other noises."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37ac6fd-f6d2-41b4-b091-97acf4327d03",
   "metadata": {},
   "source": [
    "To measure the performance of the model with the class of interest, the predicted array will be transformed into a binary array where 1 indicates a predicted gun_shot class and 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d9de5c00-6851-44c3-9421-93619b5e5683",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "902/902 [==============================] - 1s 872us/step\n"
     ]
    }
   ],
   "source": [
    "predict = model.predict(test_ds) \n",
    "y_pred = np.argmax(predict,axis=1)\n",
    "y_test = [y.numpy()[0] for x, y in test_ds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1175508d-e968-4ddf-a0df-3912f07dccc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "interest_class = label_map['gun_shot']\n",
    "\n",
    "y_gun_shot_pred = [1 if x == interest_class else 0 for x in y_pred]\n",
    "y_gun_shot_test = [1 if x == interest_class else 0 for x in y_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d4c5f613-debd-48c5-b215-6aec60cea600",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.93      0.94       853\n",
      "           1       0.12      0.16      0.14        49\n",
      "\n",
      "    accuracy                           0.89       902\n",
      "   macro avg       0.54      0.55      0.54       902\n",
      "weighted avg       0.91      0.89      0.90       902\n",
      "\n",
      "Sensitivity:  0.16\n",
      "Specificity:  0.93\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAD4CAYAAADSIzzWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAWN0lEQVR4nO3dfZxdVX3v8c83EwhPAZKGhJCkgjKCCS2PEiBFoEES4GICNt6BtkaJd9BGwBf2atJSUWss9kW9tl5iHXlwKg9hBGMCVmIYpbYgeZKAhhAzEg1DQh5QkrTFJDPn1z9mgwec2XOGzMyas/N957VeZ+919l5rnRd5/fhl7bX3VkRgZmb9b1DqAZiZ7a8cgM3MEnEANjNLxAHYzCwRB2Azs0QG93UHe7c/52UW9jvGHX9p6iHYAPTiy2u1r230JOYcMOKt+9zfvnAGbGaWSJ9nwGZm/arUnnoEFXMANrNiaW9LPYKKOQCbWaFElFIPoWIOwGZWLCUHYDOzNJwBm5kl4otwZmaJOAM2M0sjvArCzCwRX4QzM0vEUxBmZon4IpyZWSLOgM3MEvFFODOzRHwRzswsjQjPAZuZpeE5YDOzRDwFYWaWiDNgM7NE2vemHkHFHIDNrFg8BWFmloinIMzMEqmiDNivpTezYimVKi85JJ0gaXVZ2SnpY5KGS1oqaX32OazsnLmSWiStkzSlu6E6AJtZoUT73opLbjsR6yLilIg4BTgd+G9gITAHaI6IWqA520fSeKAOmABMBeZLqsnrwwHYzIolSpWXyk0Gfh4RvwSmAY1ZfSMwPdueBiyIiN0RsQFoAc7Ma9QB2MyKpQdTEJLqJa0sK/VdtFoH3Jttj4qIzQDZ58isfgzwfNk5rVldl3wRzsyKpQeZbUQ0AA15x0g6EHgPMLeb5tRZF3knOACbWbH0/iqIi4EfR8SWbH+LpNERsVnSaGBrVt8KjCs7byywKa9hT0GYWbH0/hzwlfx2+gFgMTAz254JLCqrr5M0RNJxQC2wPK9hZ8BmVixtvfdAdkmHAO8GrimrvhlokjQL2AjMAIiINZKagGeANmB2dPNsTAdgMyuWXrwTLiL+G/i9N9S9RMeqiM6OnwfMq7R9B2AzK5YquhPOAdjMisXPgjAzS8QZsJlZIs6AzcwS6cVVEH3NAdjMiiVybz4bUByAzaxYPAdsZpaIA7CZWSK+CGdmlkh77t2/A4oDsJkVi6cgzMwScQA2M0vEc8BmZmlEyeuAzczS8BSEmVkiXgVhZpaIM2Azs0QcgIvrXxYs5IEHH0YStW87ls/91Q0MGXLgm25v0b8u5auNCwC4ZmYd0y55NwCf/PQXWPPsegYPHsxJ49/OTZ+4jgMG+z9XEax4+hH+c9d/0V5qp72tnSkXzOCrd3yRt9UeC8ARRxzOjh07ufDcK9IOtFr14sN4JB0J3AacRMcr5q8G1gH3AccCvwDeFxG/zo6fC8wC2oHrImJJXvt+K3IPbNm2nbvvX8R9d/wT377rnymVSnz3kX+r6NwPfPQTvLB5y+vqduzcxVfuvId7v/Yl7v3al/jKnfewY+cuAC696AIevPdrLPzGV9i9ew8PPPhwr/8eS+e9l83kwnOvYMoFMwC45uobuPDcK7jw3Cv4zuLv8a8PPpJ4hFWsVKq8dO8fgYcj4kTgZGAtMAdojohaoDnbR9J4oA6YAEwF5kuqyWvcAbiH2trb2b17D21t7bzym90cNWI4G1s3cc0NN/K+q6/l/R/5S5775fMVtfXYslWc/c5TOeLwoRxx+FDOfuepPLZsFQDvOudMJCGJP3jHCWzZur0vf5YNIJdNn8rC+7+TehjVqxSVlxySDgfeBdwOEBF7IuJlYBrQmB3WCEzPtqcBCyJid0RsAFqAM/P66PbftJJOzBoeQ0cKvglYHBFruzu3aEYdNYIPXPleLrzi/Rw05EDOeedpTJp4OrOum8On/u+1vGXcGJ5e8yyfu+VW7vjyzd22t2Xbdo4eedTr2t+y7fWBdm9bGw8uaWbO9R/u9d9jaUQECxbeTkTwjTvv467Gb7723VnnnMH2bS+x4blfJhxhlevBKghJ9UB9WVVDRDRk228FtgF3SjoZWAVcD4yKiM0AEbFZ0sjs+DHAE2VttWZ1XcoNwJI+CVwJLACWZ9VjgXslLYiITqNM+Y+a/w+f40PvvzKvm6qxY+cufvDvT7Dkm3cydOhhfPzGz/Pgku+z+idrueHGz7923J69ewFY+J3vcVfTIgA2vrCJj/zl33DA4AMYc8wo/unvPtXpVJWk1+1/7pZbOf3kkzj9lJP67odZv7psylVseXEbI0YM575v307L+g088fhKAC5/76UsfMDZ776IHlyEy4JtQxdfDwZOA66NiGWS/pFsuqEL6qQuN83uLgOeBUyIiL2v60X6IrAG6DQAl/+ovdufq57bUrrxxMrVjDlmFMOHHQnA5PPOYdmq1QwdeigPNN76O8dffulFXH7pRUDHHPC8v/44Y0aPeu37o0eOYMWTT7+2v2Xbdt556h++tj//jrv59cs7uOnzN/bVT7IEtry4DYDt23/Fdx96hFNP+wOeeHwlNTU1XHLZhVx0/p8kHmGV67074VqB1ohYlu3fT0cA3iJpdJb9jga2lh0/ruz8sXTMGHSpuzngEnBMJ/Wjs+/2K6NHHcXTP32WV37zGyKCZStX847atzFm9NEs+f6/Ax3/vHx2/XMVtTdp4uk8vvzH7Ni5ix07d/H48h8zaeLpANy/+GEeW7aKv//MJxk0yFP1RXHIIQdz6GGHvLZ93gWTeHbtegDedf7ZtKzfwOZNW/KasO5EqfKS10zEi8Dzkk7IqiYDzwCLgZlZ3UxgUba9GKiTNETScUAtv5056FR3GfDHgGZJ64FXryz9PnA88NFuzi2cP5xwIu++4I943wevpaamhhPf/jZmTLuY8yZN5G9v+f98tfFe2trauHjyeZxY+9Zu2zvi8KFc84ErqfvQ9QB8+INXccThQwH421u+zOhRI/nT+hsAuPC8c/jI1X/adz/O+sWIo36PO+/+MgCDawbzrfsf4gfN/wHA9Pde4otvvaF3nwVxLXC3pAOB54AP0pG4NkmaBWwEZgBExBpJTXQE6TZgdkTkTkgrulkzJ2kQHVfyxtAxx9EKrOiu4VcVaQrCes+44y9NPQQbgF58eW1n86g98l+fqqs45hz62QX73N++6HYVRESUeP2VPTOzgcuPozQzS8SPozQzS6Mny9BScwA2s2JxBmxmlogDsJlZIn4gu5lZGn4nnJlZKg7AZmaJeBWEmVkizoDNzBJxADYzSyPaPQVhZpaGM2AzszS8DM3MLBUHYDOzRKpnCtgB2MyKJdqqJwI7AJtZsVRP/HUANrNiqaaLcH7drpkVS6kHpRuSfiHpJ5JWS1qZ1Q2XtFTS+uxzWNnxcyW1SFonaUp37TsAm1mhRCkqLhW6ICJOiYgzsv05QHNE1ALN2T6SxgN1wARgKjBfUk1eww7AZlYsvZgBd2Ea0JhtNwLTy+oXRMTuiNgAtNDxRvkuOQCbWaFEW+VFUr2klWWl/o3NAd+TtKrsu1ERsRkg+xyZ1Y8Bni87tzWr65IvwplZofTkrfQR0QA05BwyKSI2SRoJLJX0bM6x6qyLvP6dAZtZsfTiFEREbMo+twIL6ZhS2CJpNED2uTU7vBUYV3b6WGBTXvsOwGZWKFGqvOSRdKikoa9uAxcBPwUWAzOzw2YCi7LtxUCdpCGSjgNqgeV5fXgKwswKpSdTEN0YBSyUBB2x8p6IeFjSCqBJ0ixgIzADICLWSGoCngHagNkRkfuGUAdgMyuUaO9sKvZNtBPxHHByJ/UvAZO7OGceMK/SPhyAzaxQejED7nMOwGZWKFHqnQy4PzgAm1mhOAM2M0skwhmwmVkSzoDNzBIp9dIqiP7gAGxmheKLcGZmiTgAm5klEtXzQgwHYDMrFmfAZmaJeBmamVki7V4FYWaWhjNgM7NEPAdsZpaIV0GYmSXiDNjMLJH2UvW8ac0B2MwKpZqmIKrnfxVmZhUohSoulZBUI+lJSQ9l+8MlLZW0PvscVnbsXEktktZJmtJd2w7AZlYoEaq4VOh6YG3Z/hygOSJqgeZsH0njgTpgAjAVmC+pJq9hB2AzK5SIykt3JI0FLgVuK6ueBjRm243A9LL6BRGxOyI2AC3AmXnt9/kc8MhjL+rrLqwK7drzSuohWEFVOrVQoS8BnwCGltWNiojNABGxWdLIrH4M8ETZca1ZXZecAZtZobSXBlVcJNVLWllW6l9tR9L/ArZGxKoKu+4s8ufm2V4FYWaF0pNFEBHRADR08fUk4D2SLgEOAg6XdBewRdLoLPsdDWzNjm8FxpWdPxbYlNe/M2AzK5TeWgUREXMjYmxEHEvHxbXvR8SfAYuBmdlhM4FF2fZioE7SEEnHAbXA8rw+nAGbWaH0w8N4bgaaJM0CNgIzOvqNNZKagGeANmB2RLTnNeQAbGaF0hcvRY6IR4FHs+2XgMldHDcPmFdpuw7AZlYo0em1sIHJAdjMCqXNzwM2M0vDGbCZWSJ9MQfcVxyAzaxQnAGbmSXiDNjMLJF2Z8BmZmlU0RuJHIDNrFhKzoDNzNKoojcSOQCbWbH4IpyZWSIleQrCzCyJ3MePDTAOwGZWKF4FYWaWiFdBmJkl4lUQZmaJeArCzCwRL0MzM0ukvYoyYL8V2cwKpdSDkkfSQZKWS3pK0hpJn8nqh0taKml99jms7Jy5klokrZM0pbuxOgCbWaH0VgAGdgN/HBEnA6cAUyWdBcwBmiOiFmjO9pE0no7X108ApgLzJdXkdeAAbGaFEqq85LbT4T+z3QOyEsA0oDGrbwSmZ9vTgAURsTsiNgAtwJl5fTgAm1mh9CQDllQvaWVZqS9vS1KNpNXAVmBpRCwDRkXEZoDsc2R2+Bjg+bLTW7O6LvkinJkVSk9uRY6IBqAh5/t24BRJRwILJZ2U01xnOXXusmRnwGZWKCVVXioVES8Dj9Ixt7tF0miA7HNrdlgrMK7stLHAprx2HYDNrFB6cRXEUVnmi6SDgQuBZ4HFwMzssJnAomx7MVAnaYik44BaYHleH56CMLNC6cUbMUYDjdlKhkFAU0Q8JOlHQJOkWcBGYAZARKyR1AQ8A7QBs7MpjC45AJtZofTWsyAi4mng1E7qXwImd3HOPGBepX04AJtZofhZEGZmifiB7GZmiZSq6IGUDsBmVih+GpqZWSLVk/86AJtZwTgDNjNLpE3VkwM7AJtZoVRP+HUANrOC8RSEmVkiXoZmZpZI9YRfB2AzKxhPQZiZJdJeRTmwA7CZFYozYDOzRMIZsJlZGtWUAfuVRH1o0KBB/Ntji1nwzY53/k27/GIeX/FdXtr5M045Ne/dfra/uP66/8NTq7/P6iebuesbtzJkyJDUQ6p6JaLikpoDcB/68F98gJ+ta3ltf+0zP+P9V/0Fjz+2IuGobKA45pij+ejsq5l41iWccupkampq+N/vm5Z6WFUvelBScwDuI8ccczQXTT2ff2lseq3uZ+t+Tsv6DQlHZQPN4MGDOfjgg6ipqeGQgw9m8+YXUw+p6rURFZfUHID7yOf//kZuuvELlErp/yPbwLRp04t88f/9Mxt+vpzWjU+yY+dOlj7yw9TDqnrRgz95JI2T9ANJayWtkXR9Vj9c0lJJ67PPYWXnzJXUImmdpCndjfVNB2BJH8z5rl7SSkkrd+/d+Wa7qFpTpl7A9m0v8dTqNamHYgPYkUcewXsum8Lxbz+LcW85jUMPPYSrrroi9bCqXm+9lp6ONxt/PCLeAZwFzJY0HpgDNEdELdCc7ZN9VwdMAKYC87M3KndpXzLgz3T1RUQ0RMQZEXHGkAMO34cuqtPEs05n6iWTeWrNo9z+9S9x7nln89Xb/iH1sGyAmTz5XDb8YiPbt/+KtrY2Fn77u5x91hmph1X1eisDjojNEfHjbHsXsBYYA0wDGrPDGoHp2fY0YEFE7I6IDUALcGZeH7nL0CQ93dVXwKjc0e/HPvvpW/jsp28BYNK5E7n2ullc86GPJx6VDTTPb3yBiRNP4+CDD+KVV37DH1/wR6xa9VTqYVW9nixDk1QP1JdVNUREQyfHHUvHK+qXAaMiYjN0BGlJI7PDxgBPlJ3WmtV1qbt1wKOAKcCv3zge4PFuzrU3uPSyd/OFW25ixIjh3PfAbfzk6bX8yfQuZ3Ks4JaveJJvfes7rFi+hLa2NlavXsPXbrs79bCqXntUft0lC7a/E3DLSToMeAD4WETslLp8731nX+QORpEzWEm3A3dGxH908t09EXFVXuMAww473leh7Hfs2vNK6iHYANS254Uuo1ulrnrL5RXHnHt+uTC3P0kHAA8BSyLii1ndOuD8LPsdDTwaESdImgsQEX+XHbcE+HRE/Kir9nPngCNiVmfBN/uu2+BrZtbfenEVhIDbgbWvBt/MYmBmtj0TWFRWXydpiKTjgFpgeV4fvhXZzAqlF29FngT8OfATSauzur8CbgaaJM0CNgIzACJijaQm4Bk6VlDMjoj2vA4cgM2sUHrrFuPsX/9dTVFM7uKcecC8SvtwADazQvHT0MzMEunJKojUHIDNrFAGwlPOKuUAbGaFUk3PA3YANrNC8RywmVkinoIwM0sk7+7egcYB2MwKxa+lNzNLxFMQZmaJeArCzCwRZ8BmZol4GZqZWSK+FdnMLBFPQZiZJeIAbGaWiFdBmJkl4gzYzCyRaloFkftSTjOzatMepYpLdyTdIWmrpJ+W1Q2XtFTS+uxzWNl3cyW1SFonaUp37TsAm1mhRETFpQJfB6a+oW4O0BwRtUBzto+k8UAdMCE7Z76kmrzGHYDNrFBKRMWlOxHxQ+BXb6ieBjRm243A9LL6BRGxOyI2AC3AmXntOwCbWaFED/5Iqpe0sqzUV9DFqIjYDJB9jszqxwDPlx3XmtV1yRfhzKxQSj1YhhYRDUBDL3Xd2SvscwfjDNjMCqUnGfCbtEXSaIDsc2tW3wqMKztuLLApryEHYDMrlN5cBdGFxcDMbHsmsKisvk7SEEnHAbXA8ryGPAVhZoXSkymI7ki6FzgfGCGpFbgJuBlokjQL2AjMAIiINZKagGeANmB2RLTntt/Xt+0NO+z46lkVbf1m155XUg/BBqC2PS90No/aI7VHnV5xzFm/bdU+97cvnAGbWaH0Zgbc1xyAzaxQqulWZAdgMyuU9vxp1wHFAdjMCsWPozQzS8SPozQzS8QZsJlZIl4FYWaWiFdBmJklsg+3GPc7B2AzKxTPAZuZJeI5YDOzRJwBm5kl4nXAZmaJOAM2M0vEqyDMzBLxRTgzs0Q8BWFmlojvhDMzS8QZsJlZItU0B9znL+W035JUHxENqcdhA4v/Xuy/BqUewH6mPvUAbEDy34v9lAOwmVkiDsBmZok4APcvz/NZZ/z3Yj/li3BmZok4AzYzS8QB2MwsEQfgfiJpqqR1klokzUk9HktP0h2Stkr6aeqxWBoOwP1AUg1wK3AxMB64UtL4tKOyAeDrwNTUg7B0HID7x5lAS0Q8FxF7gAXAtMRjssQi4ofAr1KPw9JxAO4fY4Dny/Zbszoz2485APcPdVLn9X9m+zkH4P7RCowr2x8LbEo0FjMbIByA+8cKoFbScZIOBOqAxYnHZGaJOQD3g4hoAz4KLAHWAk0RsSbtqCw1SfcCPwJOkNQqaVbqMVn/8q3IZmaJOAM2M0vEAdjMLBEHYDOzRByAzcwScQA2M0vEAdjMLBEHYDOzRP4HnW3pEplX7KEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "report(y_gun_shot_test, y_gun_shot_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879b92d4-8f19-42c0-b578-08fb153eab0c",
   "metadata": {},
   "source": [
    "The previous report shows all relevant evaluation metrics for the model's performance with the class of interest. The metrics indicate that the model struggles to recognize gun_shot sounds. Although the model performs well in identifying noises that are not gun shots, it has a very low precision and recall for detecting gun shots, at only 12% precision and 16% recall.\n",
    "\n",
    "The confusion matrix analysis shows that the model has a high rate of labeling instances as \"other\" rather than correctly identifying them as gun shots. This suggests that the model is not well-suited for the current task of identifying gun shot sounds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf89dedd",
   "metadata": {},
   "source": [
    "Based on these results, it is absolutely necessary to consider other approaches to address the challenges of this approach. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "78cc9dc8-09e0-4d4b-a65b-89c4e3f5b553",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('model_01.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1a04c3dd-9e2f-4a22-877d-ec69de359bd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = tf.keras.models.load_model('model_01.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7df6157-3bf2-4b61-a385-ea073215fcb6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7b02e7-72f0-493c-bd89-9a0dda9c201e",
   "metadata": {},
   "source": [
    "### Approach II"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4189ced-dced-4984-9140-37ccd158e4a1",
   "metadata": {},
   "source": [
    "This approach follows the multi-feature stacking techniques discussed in [[1]](https://www.researchgate.net/publication/365912955_Multi-feature_stacking_order_impact_on_speech_emotion_recognition_performance). This method involves stacking multiple features extracted from the audio clip into a one-dimensional array. Five features will be computed from each audio: Mel-spectrogram, MFCCs, spectral contrast feature, chromagram and tonnetz. Each of these features will be then converted from a 2D matrix to a 1D array by taking the mean across the vertical axis. The authors in [[1]](https://www.researchgate.net/publication/365912955_Multi-feature_stacking_order_impact_on_speech_emotion_recognition_performance) stress the fact that the orders in which these features are stacked is significantly impactful on the final result. Therefore, following their recommended order, the features are horizontally stacked in the following order: spectral contrast, tonnetz, chromagram, Mel-spectrogram, and MFCC. The resulting vectors are then inputted into the classifier. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5197f834-41eb-4c10-b11b-dfbec79e65c0",
   "metadata": {},
   "source": [
    "The satcked vectors are first loaded in preparation for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "33ccd933",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_stacked_feature_arrays(split='train'):\n",
    "    path = './processed_data/stacked_features'\n",
    "    \n",
    "    print('loading {} set...'.format(split), end=' ')\n",
    "    X = np.loadtxt(\"{}/X_{}.csv\".format(path, split), delimiter=\",\")\n",
    "    y = np.loadtxt('{}/y_{}.csv'.format(path, split), delimiter = ',')\n",
    "    print('done!')\n",
    "    \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6a335192",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading train set... done!\n",
      "loading val set... done!\n",
      "loading test set... done!\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = load_stacked_feature_arrays('train')\n",
    "X_val, y_val = load_stacked_feature_arrays('val')\n",
    "X_test, y_test = load_stacked_feature_arrays('test')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3682f1d2-156e-43aa-82d8-322e5fd6e60d",
   "metadata": {},
   "source": [
    "---\n",
    "Two different classification models are trained for comparison, Support vector machine (SVM) and a 1D convolutional neural network (CNN). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcab88f1-3855-4b15-b253-c02559627e6d",
   "metadata": {},
   "source": [
    "### SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da11035d-05f3-484f-a655-dcb57800acfd",
   "metadata": {},
   "source": [
    "To find the best performing set of parameters, a grid search will be performed with scikit-learn's SVC model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f152eaaf",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 12 candidates, totalling 60 fits\n",
      "[CV 1/5] END ......C=0.1, gamma=0.1, kernel=rbf;, score=0.122 total time=   4.1s\n",
      "[CV 2/5] END ......C=0.1, gamma=0.1, kernel=rbf;, score=0.159 total time=   3.8s\n",
      "[CV 3/5] END ......C=0.1, gamma=0.1, kernel=rbf;, score=0.118 total time=   3.8s\n",
      "[CV 4/5] END ......C=0.1, gamma=0.1, kernel=rbf;, score=0.138 total time=   3.8s\n",
      "[CV 5/5] END ......C=0.1, gamma=0.1, kernel=rbf;, score=0.131 total time=   4.0s\n",
      "[CV 1/5] END ...C=0.1, gamma=0.1, kernel=linear;, score=0.754 total time=  23.7s\n",
      "[CV 2/5] END ...C=0.1, gamma=0.1, kernel=linear;, score=0.794 total time=  22.6s\n",
      "[CV 3/5] END ...C=0.1, gamma=0.1, kernel=linear;, score=0.732 total time=  23.5s\n",
      "[CV 4/5] END ...C=0.1, gamma=0.1, kernel=linear;, score=0.772 total time=  24.1s\n",
      "[CV 5/5] END ...C=0.1, gamma=0.1, kernel=linear;, score=0.785 total time=  25.8s\n",
      "[CV 1/5] END .....C=0.1, gamma=0.01, kernel=rbf;, score=0.153 total time=   4.2s\n",
      "[CV 2/5] END .....C=0.1, gamma=0.01, kernel=rbf;, score=0.204 total time=   4.2s\n",
      "[CV 3/5] END .....C=0.1, gamma=0.01, kernel=rbf;, score=0.129 total time=   4.2s\n",
      "[CV 4/5] END .....C=0.1, gamma=0.01, kernel=rbf;, score=0.148 total time=   4.2s\n",
      "[CV 5/5] END .....C=0.1, gamma=0.01, kernel=rbf;, score=0.151 total time=   4.2s\n",
      "[CV 1/5] END ..C=0.1, gamma=0.01, kernel=linear;, score=0.754 total time=  23.6s\n",
      "[CV 2/5] END ..C=0.1, gamma=0.01, kernel=linear;, score=0.794 total time=  22.5s\n",
      "[CV 3/5] END ..C=0.1, gamma=0.01, kernel=linear;, score=0.732 total time=  23.4s\n",
      "[CV 4/5] END ..C=0.1, gamma=0.01, kernel=linear;, score=0.772 total time=  24.0s\n",
      "[CV 5/5] END ..C=0.1, gamma=0.01, kernel=linear;, score=0.785 total time=  25.8s\n",
      "[CV 1/5] END ....C=0.1, gamma=0.001, kernel=rbf;, score=0.342 total time=   3.7s\n",
      "[CV 2/5] END ....C=0.1, gamma=0.001, kernel=rbf;, score=0.349 total time=   3.7s\n",
      "[CV 3/5] END ....C=0.1, gamma=0.001, kernel=rbf;, score=0.391 total time=   3.7s\n",
      "[CV 4/5] END ....C=0.1, gamma=0.001, kernel=rbf;, score=0.343 total time=   3.7s\n",
      "[CV 5/5] END ....C=0.1, gamma=0.001, kernel=rbf;, score=0.386 total time=   3.7s\n",
      "[CV 1/5] END .C=0.1, gamma=0.001, kernel=linear;, score=0.754 total time=  23.5s\n",
      "[CV 2/5] END .C=0.1, gamma=0.001, kernel=linear;, score=0.794 total time=  22.5s\n",
      "[CV 3/5] END .C=0.1, gamma=0.001, kernel=linear;, score=0.732 total time=  23.3s\n",
      "[CV 4/5] END .C=0.1, gamma=0.001, kernel=linear;, score=0.772 total time=  24.0s\n",
      "[CV 5/5] END .C=0.1, gamma=0.001, kernel=linear;, score=0.785 total time=  25.8s\n",
      "[CV 1/5] END ........C=1, gamma=0.1, kernel=rbf;, score=0.206 total time=   4.0s\n",
      "[CV 2/5] END ........C=1, gamma=0.1, kernel=rbf;, score=0.261 total time=   4.0s\n",
      "[CV 3/5] END ........C=1, gamma=0.1, kernel=rbf;, score=0.205 total time=   4.0s\n",
      "[CV 4/5] END ........C=1, gamma=0.1, kernel=rbf;, score=0.248 total time=   4.0s\n",
      "[CV 5/5] END ........C=1, gamma=0.1, kernel=rbf;, score=0.216 total time=   4.0s\n",
      "[CV 1/5] END .....C=1, gamma=0.1, kernel=linear;, score=0.758 total time= 2.9min\n",
      "[CV 2/5] END .....C=1, gamma=0.1, kernel=linear;, score=0.786 total time= 3.1min\n",
      "[CV 3/5] END .....C=1, gamma=0.1, kernel=linear;, score=0.731 total time= 2.8min\n",
      "[CV 4/5] END .....C=1, gamma=0.1, kernel=linear;, score=0.764 total time= 3.5min\n",
      "[CV 5/5] END .....C=1, gamma=0.1, kernel=linear;, score=0.781 total time= 3.0min\n",
      "[CV 1/5] END .......C=1, gamma=0.01, kernel=rbf;, score=0.414 total time=   4.3s\n",
      "[CV 2/5] END .......C=1, gamma=0.01, kernel=rbf;, score=0.452 total time=   4.3s\n",
      "[CV 3/5] END .......C=1, gamma=0.01, kernel=rbf;, score=0.404 total time=   4.3s\n",
      "[CV 4/5] END .......C=1, gamma=0.01, kernel=rbf;, score=0.439 total time=   4.3s\n",
      "[CV 5/5] END .......C=1, gamma=0.01, kernel=rbf;, score=0.409 total time=   4.3s\n",
      "[CV 1/5] END ....C=1, gamma=0.01, kernel=linear;, score=0.758 total time= 2.9min\n",
      "[CV 2/5] END ....C=1, gamma=0.01, kernel=linear;, score=0.786 total time= 3.1min\n",
      "[CV 3/5] END ....C=1, gamma=0.01, kernel=linear;, score=0.731 total time= 2.8min\n",
      "[CV 4/5] END ....C=1, gamma=0.01, kernel=linear;, score=0.764 total time= 3.5min\n",
      "[CV 5/5] END ....C=1, gamma=0.01, kernel=linear;, score=0.781 total time= 3.0min\n",
      "[CV 1/5] END ......C=1, gamma=0.001, kernel=rbf;, score=0.733 total time=   3.7s\n",
      "[CV 2/5] END ......C=1, gamma=0.001, kernel=rbf;, score=0.762 total time=   3.7s\n",
      "[CV 3/5] END ......C=1, gamma=0.001, kernel=rbf;, score=0.723 total time=   3.5s\n",
      "[CV 4/5] END ......C=1, gamma=0.001, kernel=rbf;, score=0.779 total time=   3.7s\n",
      "[CV 5/5] END ......C=1, gamma=0.001, kernel=rbf;, score=0.760 total time=   3.6s\n",
      "[CV 1/5] END ...C=1, gamma=0.001, kernel=linear;, score=0.758 total time= 2.9min\n",
      "[CV 2/5] END ...C=1, gamma=0.001, kernel=linear;, score=0.786 total time= 3.1min\n",
      "[CV 3/5] END ...C=1, gamma=0.001, kernel=linear;, score=0.731 total time= 2.8min\n",
      "[CV 4/5] END ...C=1, gamma=0.001, kernel=linear;, score=0.764 total time= 3.5min\n",
      "[CV 5/5] END ...C=1, gamma=0.001, kernel=linear;, score=0.781 total time= 3.0min\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SVC(C=0.1, gamma=0.1, kernel='linear')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "svm_estimator = SVC()\n",
    "svm_parameters = {'C': [0.1,1], 'gamma': [0.1,0.01,0.001], 'kernel': ['rbf', 'linear']}\n",
    "svm = GridSearchCV(svm_estimator, svm_parameters, verbose=3)\n",
    "svm.fit(X_train, y_train)\n",
    "svm.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c34e4e7-4a2e-4733-a5be-6ce72d9f7a39",
   "metadata": {},
   "source": [
    "To validate the resulting model performance, the model is tested against the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f588d063",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model validation accuracy: 0.75\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_val_pred = svm.predict(X_val)\n",
    "\n",
    "print('Model validation accuracy: {}'.format(round(accuracy_score(y_val, y_val_pred), 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3513596-57a0-4cd0-9358-2b77922c2356",
   "metadata": {},
   "source": [
    "The model scored a validation accuracy of 75% which is already very promising for the current method. For baseline comparison and further testing the validity of the feature stacking approach, a 1D CNN will be also trained and validated on the same data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c8e4c3c-f072-456d-b814-8ab240def58b",
   "metadata": {},
   "source": [
    "### 1D CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34709060-3eaa-4c60-80ae-1ede7c4ca937",
   "metadata": {},
   "source": [
    "The following CNN structure will be used. The CNN takes a one dimenstional input vector equal to the length of the stacked vectors. The input first goes throgh 2 consecutive 1D convolutional layer with ReLU activation function. The output is then randomly dropped to 50% to avoid overfitting before it undergoes pooling. The pooling output is passed to a two consecutive dense layers, the first contains 128 nodes and the last one consists of 10 nodes representing the 10 classes. The CNN is compiled using the adam optimiser and the accuracy will be used as an evaluation metric for fine-tunning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5cdfe59f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-04-13 13:30:08.639539: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW\n",
      "2023-04-13 13:30:08.639563: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: CMPR01\n",
      "2023-04-13 13:30:08.639569: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: CMPR01\n",
      "2023-04-13 13:30:08.639665: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 470.182.3\n",
      "2023-04-13 13:30:08.639685: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.161.3\n",
      "2023-04-13 13:30:08.639691: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:313] kernel version 470.161.3 does not match DSO version 470.182.3 -- cannot find working devices in this configuration\n",
      "2023-04-13 13:30:08.640069: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "n_timesteps, n_features, n_outputs = X_train.shape[0], X_train.shape[1], y_train.shape[0]\n",
    "input_shape = (n_features, 1)\n",
    "num_classes = 10\n",
    "\n",
    "model_1D = tf.keras.Sequential([\n",
    "    tf.keras.layers.Conv1D(32, 3, activation='relu', input_shape=input_shape),\n",
    "    tf.keras.layers.Conv1D(32, 3, activation='relu'),\n",
    "    tf.keras.layers.Dropout(0.5),\n",
    "    tf.keras.layers.MaxPooling1D(),\n",
    "    tf.keras.layers.Dense(64, activation='relu'),\n",
    "    tf.keras.layers.Dense(num_classes, activation='softmax'),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "daa11b7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_1D.compile(\n",
    "  optimizer='adam',\n",
    "  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "  metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72f40e44-2fb4-4911-a480-cc8df1eb7c48",
   "metadata": {},
   "source": [
    "The model is trained in 20 epochs and the validation sets will be used for model validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "99907357",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 1.6684 - accuracy: 0.4647 - val_loss: 1.2205 - val_accuracy: 0.6305\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 1.1230 - accuracy: 0.6320 - val_loss: 0.9726 - val_accuracy: 0.7278\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.9327 - accuracy: 0.6994 - val_loss: 0.8971 - val_accuracy: 0.7094\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.8276 - accuracy: 0.7411 - val_loss: 0.7775 - val_accuracy: 0.7722\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.7463 - accuracy: 0.7649 - val_loss: 0.7141 - val_accuracy: 0.7697\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.6741 - accuracy: 0.7861 - val_loss: 0.7591 - val_accuracy: 0.7882\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.6236 - accuracy: 0.7974 - val_loss: 0.7190 - val_accuracy: 0.7919\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.5745 - accuracy: 0.8109 - val_loss: 0.6681 - val_accuracy: 0.7956\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.5677 - accuracy: 0.8181 - val_loss: 0.5852 - val_accuracy: 0.8116\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.5387 - accuracy: 0.8281 - val_loss: 0.6336 - val_accuracy: 0.8054\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.5098 - accuracy: 0.8384 - val_loss: 0.5627 - val_accuracy: 0.8313\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.4963 - accuracy: 0.8429 - val_loss: 0.5352 - val_accuracy: 0.8399\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.4633 - accuracy: 0.8481 - val_loss: 0.6784 - val_accuracy: 0.8103\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.4963 - accuracy: 0.8491 - val_loss: 0.4750 - val_accuracy: 0.8387\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.4217 - accuracy: 0.8684 - val_loss: 0.7798 - val_accuracy: 0.8202\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.4163 - accuracy: 0.8676 - val_loss: 0.4373 - val_accuracy: 0.8633\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 9s 1ms/step - loss: 0.4365 - accuracy: 0.8673 - val_loss: 0.5229 - val_accuracy: 0.8350\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.4027 - accuracy: 0.8780 - val_loss: 0.4315 - val_accuracy: 0.8534\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.3923 - accuracy: 0.8776 - val_loss: 0.4623 - val_accuracy: 0.8374\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 10s 1ms/step - loss: 0.3920 - accuracy: 0.8816 - val_loss: 0.7323 - val_accuracy: 0.7968\n"
     ]
    }
   ],
   "source": [
    "model_1D.fit(\n",
    "    X_train,\n",
    "    y_train,\n",
    "    validation_data=(X_val, y_val),\n",
    "    epochs=20,\n",
    "    batch_size=1\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f807b487",
   "metadata": {},
   "source": [
    "While training the model, the highest validation accuracy achieved was 86.33% reached on the 16th epoch with a loss of 0.44. Afterwards, the validation accuracy did not show significant change. The average accuracy of the last 5 epochs was 83.72%."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04ec7757",
   "metadata": {},
   "source": [
    "**This is a higher accuracy than the value achieved by the SVC. This shows a great promise for the multi-feature stacking approach with deep learning.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3298e576-ba86-434d-b30a-ebf8d8ec633e",
   "metadata": {},
   "source": [
    "## Testing for approach II"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f02ede76-5ed9-4af1-a713-a5dc1696c9e4",
   "metadata": {},
   "source": [
    "Since the CNN model outperformed the SVC model in the validation phase, the CNN is chosen as the better model for the current approach. To further test the suitability of the CNN model for the task at hand, the model is tested against the **test set split**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "83e86511",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29/29 [==============================] - 0s 1ms/step\n",
      "Model validation accuracy: 80.04%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = model_1D.predict(X_test)\n",
    "y_pred = np.argmax(y_pred,axis=1)\n",
    "\n",
    "print('Model validation accuracy: {}%'.format(round(accuracy_score(y_test, y_pred) * 100, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f3908d0",
   "metadata": {},
   "source": [
    "The general accuracy score for the model is 80.04%, however, this does not represent the real accuracy of the model for the gunshot detection task. To measure the performance of the model with the class of interest, the predicted array will be transformed into a binary array where 1 indicates a predicted gun_shot class and 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bac30f7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "interest_class = label_map['gun_shot']\n",
    "\n",
    "y_gun_shot_pred = [1 if x == interest_class else 0 for x in y_pred]\n",
    "y_gun_shot_test = [1 if x == interest_class else 0 for x in y_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fca2d584",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00       853\n",
      "           1       0.98      0.86      0.91        49\n",
      "\n",
      "    accuracy                           0.99       902\n",
      "   macro avg       0.98      0.93      0.95       902\n",
      "weighted avg       0.99      0.99      0.99       902\n",
      "\n",
      "Sensitivity:  0.86\n",
      "Specificity:  1.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAD4CAYAAADSIzzWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAX+UlEQVR4nO3de5hV1X3/8feHu+INguAw0EgsxoLPTxsNWu8KEYhWSHwwk+skxd+kkXjJpQlTW/tTQ2qjpr/mifwa2hgn1YoTjTKxjZGgJpoYEJVEAZFRUhwhIBjvisyc7++P2eIRZs6ckcOsmc3n5bOfs8/ae6+98OH5uFxn7b0UEZiZWc/rl7oBZmZ7KwewmVkiDmAzs0QcwGZmiTiAzcwSGbCnb7B9y9OeZmG72Gf0yambYL1Q65vPanfr6E7mDBzxvt2+3+5wD9jMLJE93gM2M+tRhbbULSibA9jM8qWtNXULyuYANrNciSikbkLZHMBmli8FB7CZWRruAZuZJeIf4czMEnEP2MwsjehDsyD8IIaZ5UuhUP7WBUlfkrRS0uOSbpY0RNJwSYslrc0+hxWdXy+pWdIaSVO7qt8BbGb5EoXytxIkVQMXAcdGxJFAf6AGmAssiYjxwJLsO5ImZMcnAtOA+ZL6l7qHA9jM8qXQVv7WtQHAPpIGAPsCG4AZQEN2vAGYme3PABZGxLaIWAc0A5NKVe4ANrN8qVAPOCKeBa4B1gMbgRcj4m5gVERszM7ZCIzMLqkGnimqoiUr65QD2Mzypa217E1SnaTlRVvdW9VkY7szgHHAaGCopE+VuHNHb1Yr+WY2z4Iws3zpxpNwEbEAWNDJ4SnAuoh4DkDSj4ETgE2SqiJio6QqYHN2fgswtuj6MbQPWXTKPWAzy5WItrK3LqwHjpe0ryQBk4HVQBNQm51TCyzK9puAGkmDJY0DxgPLSt3APWAzy5cKPYgREUsl3Qo8ArQCj9LeW94PaJQ0m/aQnpWdv1JSI7AqO39OdJHyitizC1Z4RQzriFfEsI5UYkWMNx5pKjtzhnzgnKQrYrgHbGb54keRzcwSadueugVlcwCbWb74fcBmZol4CMLMLBH3gM3MEnEAm5mlEf4RzswsEY8Bm5kl4iEIM7NE3AM2M0vEPWAzs0TcAzYzS6S176yK7AA2s3xxD9jMLBGPAZuZJeIesJlZIn2oB+w14cwsXyq0LL2k90taUbS9JOkSScMlLZa0NvscVnRNvaRmSWskTe2qqQ5gM8uX1tbytxIiYk1EHB0RRwPHAK8BtwNzgSURMR5Ykn1H0gSgBpgITAPmS+pf6h4OYDPLl4jyt/JNBp6KiP8BZgANWXkDMDPbnwEsjIhtEbEOaAYmlarUAWxm+VIolL1JqpO0vGir66TWGuDmbH9URGwEyD5HZuXVwDNF17RkZZ3yj3Bmli/d+BEuIhbQvtR8pyQNAs4B6ruorqMVlkt2sx3AZpYvlZ+GNh14JCI2Zd83SaqKiI2SqoDNWXkLMLboujHAhlIVewjCzPKlra38rTwf5+3hB4AmoDbbrwUWFZXXSBosaRwwHlhWqmL3gM0sXyo4D1jSvsCHgM8XFV8FNEqaDawHZgFExEpJjcAqoBWYExElU94BbGb5UsEAjojXgPfsVLaV9lkRHZ0/D5hXbv0OYDPLFz+KbGaWRhS6Nb83KQewmeVLH3oXhAPYzPKl/NkNyTmAzSxf3AM2M0vEAZwPP1x4O7f95C4kMf6wQ/nG336ZwYMH7Ti+7JHfcdHcy6muOgSAKaeewBf+6pO7dc8333yT+iuvZdWatRx04AFcc0U91VWjeOLJp7jymu/yyquv0a9/P+o+U8P0Kafu1r0srX9bcC1nfXgKm5/bwtF/3uGsJns3uveSnaT8JFwnNj23hZtuXcQt13+HO278VwqFAj/9+S92Oe8DRx3JbQ3XcVvDdd0K32c3buKzX/zaLuU/vvNuDth/P37aeD2f/thMvj3/egCGDBnMN//+qyy66Xt879pv8E/f+R4vvfzKu/8DWnI//GEjZ529e//Btg5042U8qTmAS2hta2PbtjdpbW3j9Te2cfCI4WVf+5Of3UPN+Rdzbu0cLv/Wd2gr84eBe+5/kBkfngLAmaedzNKHVxARHPonY3jv2PYXK408+D0MH3YQf3zhxe7/oazXuP+BpTz/xxdSNyN/ClH+lliXQxCSjqD9PZfVtL/ZZwPQFBGr93Dbkhp18Ag++/FzmfLRzzBk8CBO+OAHOPG4Y3Y577ePr+ajtRcwcsR7+Oqc8/nT972Xp36/nruW/IL/+NdrGThgAFde813uvPteZkyf0uV9Nz+3lUNGjgBgwID+7Dd0X1548SWGHXTgjnMeW7WG7dtbGVtdVbk/sFle5GUWhKSv0/4iioW8/VKJMcDNkhZGxFWdXFcH1AHMv/YbnP+Zj1euxT3kxZde5t77f8PPfvQD9t9/P77yd9/kJz+7h7+cesaOcya8/zAW39bAvvvuwy9/vYyL6q/gv2/5PkuXr2DVE83UzL4YgG3btjF82EEAXFR/Bc9u2MT21u1s3PQc59bOAeBT583gI2edSXQwfiW9/Za757Y8T/0VVzPv775Cv37+HxiznUUvGFooV1c94NnAxIjYXlwo6dvAStpfSrGL4ndsbt/ydPp+/rvwm+UrqB49akdwTj71BFY8tuodAbzf0KE79k85YRLfuPY6/vjCi0QE50yfwpe+8Lld6v3OP14GtI8BXzrvWm747rfecXzUyBH8YfMWDhl5MK2tbbzy6msceMD+ALzy6qtc8DeXcWFdLUcd+WcV/zOb5UIvGFooV1ddqAIwuoPyquxYblWNOpjfPf4Er7/xBhHB0uUreN97x77jnC1bn9/RY31s1RoKERx04AEcf+zRLL7vAbZm43svvvQyG/6waZd7dOT0k45n0X//HIC777uf4445Ckls376di+uv5Jxpk5l6xskV/JOa5UyFFuXsCV31gC8Blkhay9tLbfwJ8KfAF/dkw1L7XxOP4EOnn8R5n7uQ/v37c8ThhzFrxnRuuf2/APjYR87i7nsf4Jbb/4v+A/ozZNAgrr58LpI4bNx7ufB/f4a6Sy6lEAUGDhjApV++gNGHjOryvh89eyr1V17N9PP+igMP2J+rL58LwF333M/DKx7nhRdf5o4soOdd+mWOOPywPfcvwfaoG//jOk495S8YMWI4v396OZdfcQ0/uGFh6mb1fX2oB6yOxhzfcYLUj/aF5appX3KjBXioq/dcvqWvDkHYnrXPaPfibVetbz7b0bI+3fLqZTVlZ87QKxbu9v12R5ezICKiAPymB9piZrb7esHQQrn8JJyZ5UsfGoLwPCYzy5UoFMreuiLpIEm3SnpC0mpJfyFpuKTFktZmn8OKzq+X1CxpjaSpXdXvADazfKnsk3D/AtwVEUcARwGrgbnAkogYDyzJviNpAlADTASmAfMl9S9VuQPYzPKlQgEs6QDgFOD7ABHxZkS8QPuTwQ3ZaQ3AzGx/BrAwIrZFxDqgmfYJDJ1yAJtZvnRjWXpJdZKWF211RTW9D3gO+IGkRyX9u6ShwKiI2AiQfY7Mzq/m7em60D5jrLpUU/0jnJnlSnfWhCt+arcDA4APABdGxFJJ/0I23NCJjqa0lWyMe8Bmli+VGwNuAVoiYmn2/VbaA3mTpCqA7HNz0fnFj8uOof3lZZ1yAJtZvlTofcAR8QfgGUnvz4omA6uAJqA2K6sFFmX7TUCNpMGSxgHjefslZh3yEISZ5Utl5wFfCNwkaRDwNPA52juujZJmA+uBWQARsVJSI+0h3QrM6eqJYQewmeVLBQM4IlYAx3ZwqMM1pCJiHjCv3PodwGaWK9HmR5HNzNLoQ48iO4DNLFe6Mw0tNQewmeWLA9jMLJG+MwTsADazfInWvpPADmAzy5e+k78OYDPLF/8IZ2aWinvAZmZpuAdsZpaKe8BmZmlEa+oWlM8BbGa50odWpXcAm1nOOIDNzNJwD9jMLJG+FMBeksjMciXaVPbWFUm/l/SYpBWSlmdlwyUtlrQ2+xxWdH69pGZJayRN7ap+B7CZ5UoUyt/KdHpEHB0Rb62MMRdYEhHjgSXZdyRNAGqAicA0YL6k/qUqdgCbWa5EQWVv79IMoCHbbwBmFpUvjIhtEbEOaAYmlarIAWxmudKdHrCkOknLi7a6nasD7pb0cNGxURGxESD7HJmVVwPPFF3bkpV1yj/CmVmuRJTfs42IBcCCEqecGBEbJI0EFkt6osS5Hd245HPR7gGbWa5Ucgw4IjZkn5uB22kfUtgkqQog+9ycnd4CjC26fAywoVT9DmAzy5VCm8reSpE0VNL+b+0DZwKPA01AbXZaLbAo228CaiQNljQOGA8sK3UPD0GYWa7sxo9rOxsF3C4J2rPyPyPiLkkPAY2SZgPrgVkAEbFSUiOwCmgF5kREW6kbOIDNLFcqFcAR8TRwVAflW4HJnVwzD5hX7j0cwGaWK9F3XgfsADazfKngEMQe5wA2s1zpzjS01BzAZpYrbWW846G3cACbWa64B2xmlojHgM3MEvEsCDOzRNwDNjNLpK3Qd96w4AA2s1zxEISZWSIFz4IwM0vD09DMzBLxEESRfUefvKdvYX3QQUOGpm6C5ZSHIMzMEvEsCDOzRPrQCIQD2MzypS8NQfSdvrqZWRkiVPZWDkn9JT0q6c7s+3BJiyWtzT6HFZ1bL6lZ0hpJU7uq2wFsZrlS6MZWpouB1UXf5wJLImI8sCT7jqQJQA0wEZgGzJfUv1TFDmAzy5VAZW9dkTQGOAv496LiGUBDtt8AzCwqXxgR2yJiHdBM+zL2nXIAm1mutIbK3iTVSVpetNXtVN3/Bb7GOzvMoyJiI0D2OTIrrwaeKTqvJSvrlH+EM7NcKadnu+PciAXAgo6OSTob2BwRD0s6rYzqOrpxyUkZDmAzy5VujO125UTgHEkfBoYAB0i6EdgkqSoiNkqqAjZn57cAY4uuHwNsKHUDD0GYWa5Uagw4IuojYkxEHEr7j2v3RMSngCagNjutFliU7TcBNZIGSxoHjAeWlbqHe8BmlisV7AF35iqgUdJsYD0wCyAiVkpqBFYBrcCciGgrVZED2Mxypa0bY8Dlioj7gPuy/a3A5E7OmwfMK7deB7CZ5UofWpHIAWxm+VLYAz3gPcUBbGa54pfxmJkl0gM/wlWMA9jMcqUgD0GYmSVRct5XL+MANrNc8SwIM7NEPAvCzCwRz4IwM0vEQxBmZol4GpqZWSJt7gGbmaXhHrCZWSIOYDOzRMpcbb5XcACbWa70pR6wlyQys1xp68ZWiqQhkpZJ+q2klZIuz8qHS1osaW32OazomnpJzZLWSJraVVsdwGaWKwWVv3VhG3BGRBwFHA1Mk3Q8MBdYEhHjgSXZdyRNoH3tuInANGC+pP6lbuAANrNcKXRjKyXavZJ9HZhtAcwAGrLyBmBmtj8DWBgR2yJiHdAMTCp1DwewmeVKdwJYUp2k5UVbXXFdkvpLWkH70vOLI2IpMCoiNgJknyOz06uBZ4oub8nKOuUf4cwsV7rzLoiIWAAsKHG8DTha0kHA7ZKOLFFdR4MaJZvjHrCZ5UoFx4B3iIgXaF8VeRqwSVIVQPa5OTutBRhbdNkYYEOpeh3AZpYrFZwFcXDW80XSPsAU4AmgCajNTqsFFmX7TUCNpMGSxgHjgWWl7uEhCDPLlULlXkhZBTRkMxn6AY0RcaekB4FGSbOB9cAsgIhYKakRWAW0AnOyIYxOOYDNLFcq9SBGRPwO+PMOyrcCkzu5Zh4wr9x7OIDNLFf8QnYzs0T60qPIDmAzy5VW9Z0+sAPYzHKl78SvA9jMcsZDEGZmiVRwGtoe5wA2s1zpO/HrADaznPEQhJlZIm19qA/sADazXHEP2MwskXAP2Mwsjb7UA/brKHvA4YcfxvKH7t6xbd3yBBddeH7qZlki/fr145777+A/G78HwP+58ms8uPwufvHrJhpuuo4DDtw/cQv7tgJR9paaA7gHPPnkUxz7wTM59oNnMum4abz22uvcseinqZtliXz+C7WsffKpHd/vu/dXnHTcWZx6wjk81byOS778+YSt6/uiG1tqDuAedsYZJ/H00//D+vXPpm6KJVA1ehQfmnoaNzb8aEfZfff8ira29tfGLn/ot4yuPiRV83KhlSh7S80B3MM+dt4MbrnljtTNsETmXXUpl1/2LQqFjkcqP/npc1my+Jc93Kp8iW78k9q7DmBJnytxbMdKo4XCq+/2FrkzcOBAzj77TG697c7UTbEEzpx2Glu2bOW3K1Z2ePxLX/1rWlvb+NEtTT3csnyp1LL0PWF3esCXd3YgIhZExLERcWy/fkN34xb5Mm3a6Tz66GNs3rwldVMsgUnHHcO06ZN55LF7WPCDf+akU47n//3b1QB87BMf4cxpp/PX538lcSv7vkr1gCWNlXSvpNWSVkq6OCsfLmmxpLXZ57Cia+olNUtaI2lqV21VROeNkPS7zg4Bh0fE4K5uMHBQdfp+fi9x443zWXz3fTT8sDF1U5I7cMje/R/mE0+axJyLZvOJ8z7PGVNO5spv1nPO9E+ydesfUzctqS0vPdmNtYo7VnvouWVnTsPvb+v0ftmKx1UR8Yik/YGHgZnAZ4HnI+IqSXOBYRHxdUkTgJuBScBo4Oe052Sn68J1NQ94FDAV2PlvhYBfd3GtFdlnnyFMmXwKF1zw9dRNsV7mqmsuY/CgQdy66AYAHn5oBV/90j+kbVQf1laiU9kdEbER2JjtvyxpNVANzABOy05roH25+q9n5QsjYhuwTlIz7WH8YGf36CqA7wT2i4gVOx+QdF83/ix7vddff4NDqo5M3QzrJX71wDJ+9UD7iuWTjv5Q4tbkS3fm90qqA+qKihZExIIOzjuU9gU6lwKjsnAmIjZKGpmdVg38puiylqysUyUDOCJmlzj2iVLXmpml0J3ZDVnY7hK4xSTtB9wGXBIRL0mdj1p02JwSPA3NzHKlkrMgJA2kPXxviogfZ8WbsvHht8aJN2flLcDYosvHABtK1e8ANrNcqdSjyGrv6n4fWB0R3y461ATUZvu1wKKi8hpJgyWNA8YDy0rdwy/jMbNcqeADFicCnwYek/TW72B/C1wFNEqaDawHZgFExEpJjcAqoBWYU2oGBDiAzSxnKjgL4gE6HtcFmNzJNfOAeeXewwFsZrnSG95yVi4HsJnlSm94xLhcDmAzy5Xe8JKdcjmAzSxXPARhZpZIqffb9DYOYDPLFS9Lb2aWiIcgzMwS8RCEmVki7gGbmSXiaWhmZolU6lHknuAANrNc8RCEmVkiDmAzs0Q8C8LMLBH3gM3MEulLsyC8JJGZ5UpbFMreuiLpekmbJT1eVDZc0mJJa7PPYUXH6iU1S1ojaWpX9TuAzSxXIqLsrQw3ANN2KpsLLImI8cCS7DuSJgA1wMTsmvmS+peq3AFsZrlSqUU5ASLil8DzOxXPABqy/QZgZlH5wojYFhHrgGZgUqn6HcBmlivRjX/epVERsREg+xyZlVcDzxSd15KVdco/wplZrhS6MQ1NUh1QV1S0ICIWvMtbd7SAZ8nGOIDNLFe607PNwra7gbtJUlVEbJRUBWzOyluAsUXnjQE2lKrIQxBmliuVnAXRiSagNtuvBRYVlddIGixpHDAeWFaqIveAzSxXujME0RVJNwOnASMktQD/AFwFNEqaDawHZgFExEpJjcAqoBWYExFtJevf04/tDRxU3XdmRVuPOXDI0NRNsF5oy0tPdjSO2i3jDz6m7MxZ+9zDu32/3eEesJnlSiV7wHuaA9jMcqUvPYrsADazXGkrPezaqziAzSxX/DpKM7NE/DpKM7NE3AM2M0vEsyDMzBLxLAgzs0R24xHjHucANrNc8RiwmVkiHgM2M0vEPWAzs0Q8D9jMLBH3gM3MEvEsCDOzRPwjnJlZIh6CMDNLxE/CmZkl4h6wmVkifWkMeI8vymlvk1QXEQtSt8N6F/+92Hv1S92AvUxd6gZYr+S/F3spB7CZWSIOYDOzRBzAPcvjfNYR/73YS/lHODOzRNwDNjNLxAFsZpaIA7iHSJomaY2kZklzU7fH0pN0vaTNkh5P3RZLwwHcAyT1B64DpgMTgI9LmpC2VdYL3ABMS90IS8cB3DMmAc0R8XREvAksBGYkbpMlFhG/BJ5P3Q5LxwHcM6qBZ4q+t2RlZrYXcwD3DHVQ5vl/Zns5B3DPaAHGFn0fA2xI1BYz6yUcwD3jIWC8pHGSBgE1QFPiNplZYg7gHhARrcAXgZ8Bq4HGiFiZtlWWmqSbgQeB90tqkTQ7dZusZ/lRZDOzRNwDNjNLxAFsZpaIA9jMLBEHsJlZIg5gM7NEHMBmZok4gM3MEvn/DXM8gc1AQBEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "\n",
    "report(y_gun_shot_test, y_gun_shot_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46585ff3",
   "metadata": {},
   "source": [
    "Based on the evaluation metrics presented in the previous report, the CNN model for gunshot detection achieved excellent results. The model was able to discriminate between the gun_shot class and all other classes with nearly perfect scores, resulting in an overall accuracy of 99%. The model also showed strong sensitivity (or recall) with a score of 86%, indicating that the model can accurately detect most instances of gun_shot sounds. In addition, the model demonstrated a perfect specificity, which means that it mostly did not misclassify any non-gun_shot sounds as gun_shot sounds. These results suggest that the CNN model is highly effective at detecting gun_shot sounds, and could be a valuable tool in identifying and responding to potential gun-related incidents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "04ec78b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_1D.save('model_02.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b34d0a6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_1D = tf.keras.models.load_model('model_02.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b94ea99",
   "metadata": {},
   "source": [
    "# Inferencing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a88cbca",
   "metadata": {},
   "source": [
    "Although the model is already tested on the test split, which was unseen by the model before, it is also necessary to evaluate the model performance with data obtained outside of the current dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46afefaf",
   "metadata": {},
   "source": [
    "In this step, the trained model is applied to fresh, unseen data to evaluate its performance in real-world scenarios. This phase involves testing the model on a sample of data that was not previously used in the training, validation or testing phases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f18e556",
   "metadata": {},
   "source": [
    "In the current project, the model is tested on 10 different sound clips downloaded from YouTube in mp3 format. This is a small sample due to the limited time available for the project, but it serves as an experiment to assess how the model would perform with real-world data. The 10 sound clips belong to different classes, but not all classes are represented in the sample. Some of the sound clips contain gunshot sounds representing the target class. The performance of the model on these sound clips can help evaluate its efficacy in detecting gunshot sounds in real-world scenarios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0c938eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "INFERENCING_DIR = './inferencing_set'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8722269",
   "metadata": {},
   "source": [
    "The files are loaded into a numpy array to prepare for inferencing. The sounds are loaded as 2-second-segments. If a sound is smaller, it is padded with silence. If the sound is longer, the first 2 seconds are taken. The original sound files all have a lengths of few ms seconds less or more than 2 seconds; thus, the lengths are not greatly affected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "73a2e07a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 44100)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = librosa.util.find_files(INFERENCING_DIR, ext=['mp3'])\n",
    "files = np.asarray(files)\n",
    "LEN = 44100\n",
    "\n",
    "inf_slices = []\n",
    "inf_labels = []\n",
    "\n",
    "for file in files:\n",
    "    if '.mp3' in file:\n",
    "        filename = file.split('/').pop()\n",
    "        label = filename.split('.')[0].split('-')[1] # class ID\n",
    "        wave_arr, sr = librosa.load(file, sr = SAMPLE_RATE, mono = True)\n",
    "        \n",
    "        if len(wave_arr) < 44100:\n",
    "            diff = LEN - len(wave_arr)\n",
    "            silence = np.zeros(diff) # silence\n",
    "            lead = silence[0 : math.ceil(diff / 2)]\n",
    "            trail = silence[0 : math.floor(diff / 2)]\n",
    "            \n",
    "            wave_arr = np.concatenate((lead, wave_arr, trail))\n",
    "        \n",
    "        inf_slices += [wave_arr.astype(float)[:LEN]]\n",
    "        inf_labels += [int(label)]\n",
    "        \n",
    "inf_slices = np.array(inf_slices, dtype=np.float)\n",
    "y_inf = np.array(inf_labels)\n",
    "\n",
    "inf_slices.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "235a7880",
   "metadata": {},
   "source": [
    "The next step is to extract the stacked features vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "339570cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inferencing set stacked features: extracting........done!\n"
     ]
    }
   ],
   "source": [
    "print('Inferencing set stacked features: ', end='')\n",
    "X_stack_inf = extract_stacked_feature_arrays(inf_slices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca30965",
   "metadata": {},
   "source": [
    "Next, the model is loaded then the sounds arrays passed to the model for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d2587c59",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-04-14 01:29:43.042701: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE: forward compatibility was attempted on non supported HW\n",
      "2023-04-14 01:29:43.042764: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: CMPR01\n",
      "2023-04-14 01:29:43.042781: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: CMPR01\n",
      "2023-04-14 01:29:43.043002: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 470.182.3\n",
      "2023-04-14 01:29:43.043061: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 470.161.3\n",
      "2023-04-14 01:29:43.043076: E tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:313] kernel version 470.161.3 does not match DSO version 470.182.3 -- cannot find working devices in this configuration\n",
      "2023-04-14 01:29:43.044171: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.models.load_model('model_02.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ac1a099b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 96ms/step\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "y_pred = model.predict(X_stack_inf)\n",
    "y_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7ddb28d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "interest_class = label_map['gun_shot']\n",
    "\n",
    "y_gun_shot_pred = [1 if x == interest_class else 0 for x in y_pred]\n",
    "y_gun_shot_test = [1 if x == interest_class else 0 for x in y_inf]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6c69efda",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.75      1.00      0.86         6\n",
      "           1       1.00      0.50      0.67         4\n",
      "\n",
      "    accuracy                           0.80        10\n",
      "   macro avg       0.88      0.75      0.76        10\n",
      "weighted avg       0.85      0.80      0.78        10\n",
      "\n",
      "Sensitivity:  0.5\n",
      "Specificity:  1.0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVQAAAD8CAYAAAAoqlyCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAN40lEQVR4nO3de4yc1X3G8edZs8QGG0iKMb6QOIlTIqiSujVEFUWiN3BDAFVUKaTtHynNtlFTYVQ1oYS2IoUqohJJKqGgESGkCZdYtLTEJQSUgCxaILZbky5eLsaQsGsTl7RRsJXEuzO//uEBTcE7O4N/s2f3+PtBr9iZeS9H4tXD75zzXhwRAgAcvqHSDQCAWhCoAJCEQAWAJAQqACQhUAEgCYEKAEkIVACYhu0TbN9l+0nbY7Z/qdv6R81WwwBgHvqcpPsi4rdtHy3pmG4rmwv7AeD1bB8n6XFJ74geg3LgFerkS7tIbLzOohVnl24C5qCpAxM+3H30kzlHL33nH0ka6fiqERGN9t/vkPTfkr5o+72Stkm6PCL2T7c/xlABHLEiohER6zqWRsfPR0n6BUmfj4i1kvZLurLb/ghUAHVpNXtfuhuXNB4Rj7U/36WDATstJqUA1KU5lbKbiHjR9gu2T42IpyT9mqQd3bYhUAFUJaKVubs/lXRbe4Z/l6QPd1uZQAVQl1ZeoEbEdknrel2fQAVQl9wKtS8EKoC6zDzZNDAEKoC6UKECQI5ImuV/IwhUAHVJnJTqF4EKoC50+QEgCZNSAJCEChUAkjApBQBJmJQCgBwRjKECQA7GUAEgCV1+AEhChQoASZqTxQ5NoAKoC11+AEhClx8AklChAkASAhUAcgSTUgCQhDFUAEhClx8AklChAkASKlQASEKFCgBJpnjANADkoEIFgCSJY6i2n5f0sqSmpKmIWNdtfQIVQF3yK9RfiYiXelmRQAVQl4Kz/EPFjgwAgxCt3pce9ibpftvbbI/MtDIVKoC69DHL3w7JzqBsRESj4/NZEbHb9kmSHrD9ZERsnm5/BCqAukT0sWo0JDW6/L67/e+9tu+WdKakaQOVLj+AurRavS9d2D7W9pJX/pZ0rqTRbttQoQKoS96k1DJJd9uWDmbl7RFxX7cNCFQAdUm6bCoidkl6bz/bEKgA6tJsFjs0gQqgLjxtCgCSEKgAkISHowBAjmj1fh1qNgIVQF3o8gNAEmb5ASAJFSoAJOHxffX70cv7dMUnr9UFl35EF3xoRNtHx0o3CXPAeeeeoydGN+vJHQ/r43/+J6WbU4eI3pdkVKiz5NOfvUlnvW+dPnPd1ZqcnNSPf/LT0k1CYUNDQ/r7z12n9e+/VOPje/ToI/fqa5vu19jYM6WbNr9RodZt3/792vb4qC6+4DxJ0vDwsI5bsrhwq1DamWes1bPPPq/nnvueJicntXHjv+jC9jmCw9CK3pdkM1aott8t6SJJK3Xw6dW7Jd0TEfRZezQ+8aLefMLxuvq6G/TUzl067dR36coNf6xjFi0s3TQUtGLlyXphfPern8cn9ujMM9YWbFElCs7yd61QbX9C0p2SLOnbkra0/77D9pVdthuxvdX21pv/4Y7M9s5LU82mxp7eqd/5rfN11603atGihfrClzeWbhYKaz8W7v+JAYzrHWmi1ep5yTZThXqZpNMjYrLzS9s3SHpC0qcPtVHnU7AnX9p1xJ8hJ590opYtPVHvOf3dkqRzz/ll3fwVAvVINzG+R6esWvHq51Url2vPnu8XbFElCt4pNdMYakvSikN8v7z9G3pw4s+8RSeftFTPfXdckvTotu165+q3Fm4VStuydbvWrHm7Vq8+RcPDw/rgBy/S1zbdX7pZ81/uS/r6MlOFukHSN20/I+mF9ndvlbRG0sfSW1Oxq674qD5xzfWanJrUKSuW62+uuqJ0k1BYs9nU5Ruu1r3/ersWDA3p1i99VTt2PF26WfNfwQrVM43Z2B7SwRdTrdTB8dNxSVsioqeRX7r8OJRFK84u3QTMQVMHJl4/sNyn/X91Sc+Zc+yn7jzs43WacZY/IlqSHs08KAAMDI/vA4AkPL4PAHIM4nKoXhGoAOpChQoASQhUAEjCA6YBIAfvlAKALAQqACRhlh8AklChAkCS5EC1vUDSVkkTEfGBbusSqACqEs30Lv/lksYkHTfTirwCBUBdEl+BYnuVpPMl3dzLoQlUAFWJVvS8dL5dpL2MvGZ3n5X0cfX4/Ge6/ADq0scYaufbRV7L9gck7Y2IbbbP6WV/BCqAuuQNoZ4l6ULb75e0UNJxtr8SEb833QZ0+QFUJaZaPS9d9xPxFxGxKiJWS7pE0re6halEhQqgNgXfdkegAqjKIO7lj4iHJD0003oEKoC6UKECQA6eNgUAWahQASBHTJU7NoEKoCoF3yJNoAKoDIEKADmoUAEgCYEKAEmi6WLHJlABVIUKFQCSRIsKFQBSUKECQJIIKlQASEGFCgBJWszyA0AOJqUAIAmBCgBJotzjUAlUAHWhQgWAJFw2BQBJmszyA0AOKlQASMIYKgAkYZYfAJJQoQJAkmZrqNixCVQAVaHLDwBJWkmz/LYXStos6U06mJV3RcRfd9uGQAVQlcTLpn4q6VcjYp/tYUkP2/56RDw63QYEKoCqZHX5IyIk7Wt/HG4vXfc+8EC9/hf/ctCHwDx0zfJzSjcBleqny297RNJIx1eNiGh0/L5A0jZJayTdGBGPddsfFSqAqvQzy98Oz0aX35uSft72CZLutv1zETE63frlri8AgAGIPpae9xnxQ0kPSVrfbT0CFUBVWuGel25sL21XprK9SNKvS3qy2zZ0+QFUJXGWf7mkL7XHUYckbYyITd02IFABVCXrpacR8R1Ja/vZhkAFUJUQ9/IDQIopnocKADmoUAEgSdYY6htBoAKoChUqACShQgWAJE0qVADIUfANKAQqgLq0qFABIEfBN6AQqADqwqQUACRpmS4/AKRoFjw2gQqgKszyA0ASZvkBIAmz/ACQhC4/ACThsikASNKkQgWAHFSoAJCEQAWAJAVfKUWgAqgLFSoAJOHWUwBIwnWoAJCELj8AJCkZqEMFjw0A6aKPpRvbp9h+0PaY7SdsXz7TsalQAVQlcQx1StKfRcR/2F4iaZvtByJix3QbEKgAqpI1yx8ReyTtaf/9su0xSSslTRuodPkBVKWl6HmxPWJ7a8cycqh92l4taa2kx7odmwoVQFX6mZSKiIakRrd1bC+W9I+SNkTEj7qtS6ACqErmA6ZtD+tgmN4WEf800/oEKoCqZF02ZduSviBpLCJu6GUbAhVAVaacVqOeJen3Jf2X7e3t766KiHun24BABVCVrDiNiIel/t74R6ACqAq3ngJAklbB954SqACqwmukASAJXX4ASNKkyw8AOahQASBJUKECQA4q1MotWf4WXfiZj2rx0uMVrdB/3v4tbfniN0o3C4VxXgwGl01VLpotffPa2/Ti6PM6+tiF+oNN1+q5h0f10jMTpZuGgjgvBqPkZVM8D3UW7Nv7Q704+rwk6cD+n+gHO3drybI3l20UiuO8GIwpRc9LNirUWXb8qhO17PS3aWL7s6WbgjmE8yJPyUmpN1yh2v5wl99efQr2ln073+ghqjN8zJt08U0b9MCnvqwD+35cujmYIzgvcrX6WLIdTpf/mul+iIhGRKyLiHVnLF5zGIeox9BRC3TxTRs0+s//pqfu21q6OZgjOC/yRR//ZOva5bf9nel+krQsvTUVO//6j+gHOyf07Zu/XropmEM4L/LN5cumlkk6T9L/vuZ7S/r3gbSoQqvW/azec/HZ+v7Y9/SH9/6tJOnBv/uqnn3w8cItQ0mcF4PRjLl72dQmSYsjYvtrf7D90EBaVKHxrU/rurf9bulmYI7hvBiMOXsdakRc1uW3D+U3BwAOD7eeAkCSuTyGCgDzypzt8gPAfEOXHwCSzOVZfgCYV+jyA0ASJqUAIAljqACQhC4/ACSJgpNSPGAaQFWaip6Xmdi+xfZe26O9HJtABVCVlqLnpQe3Slrf67Hp8gOoSmaXPyI2217d6/oEKoCqlJyUossPoCr9PLG/83VN7WXkcI5NhQqgKv3cehoRDUmNrGMTqACqQpcfAJJkzvLbvkPSI5JOtT1ue9qH7ktUqAAqkzzLf2k/6xOoAKrCracAkISHowBAkmaUe4AfgQqgKiUfjkKgAqgKY6gAkIQxVABI0qLLDwA5qFABIAmz/ACQhC4/ACShyw8ASahQASAJFSoAJGlGs9ixCVQAVeHWUwBIwq2nAJCEChUAkjDLDwBJmOUHgCTcegoASRhDBYAkjKECQBIqVABIwnWoAJCEChUAkjDLDwBJmJQCgCQlu/xDxY4MAAMQffwzE9vrbT9le6ftK2danwoVQFWyKlTbCyTdKOk3JI1L2mL7nojYMd02BCqAqiSOoZ4paWdE7JIk23dKukhSuUD95Hdv86CPMV/YHomIRul2YG7hvMg1dWCi58yxPSJppOOrRsd/i5WSXuj4bVzS+7rtjzHU2TUy8yo4AnFeFBIRjYhY17F0/o/tUMHctfwlUAHg0MYlndLxeZWk3d02IFAB4NC2SHqX7bfbPlrSJZLu6bYBk1Kzi3EyHArnxRwUEVO2PybpG5IWSLolIp7oto1LXgQLADWhyw8ASQhUAEhCoM6Sfm9hQ/1s32J7r+3R0m1BDgJ1FnTcwvabkk6TdKnt08q2CnPArZLWl24E8hCos+PVW9gi4oCkV25hwxEsIjZL+p/S7UAeAnV2HOoWtpWF2gJgQAjU2dH3LWwA5h8CdXb0fQsbgPmHQJ0dfd/CBmD+IVBnQURMSXrlFrYxSRtnuoUN9bN9h6RHJJ1qe9z2ZaXbhMPDracAkIQKFQCSEKgAkIRABYAkBCoAJCFQASAJgQoASQhUAEjyf6HId76L+K2kAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "\n",
    "report(y_gun_shot_test, y_gun_shot_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "493f15a3",
   "metadata": {},
   "source": [
    "The previous report shows all evaluation metrics for the quality of the inferencing of the model. It is clear that the model achieved an excellent accuracy of 80%, which is a promising outcome. However, this value is lower than the accuracy score obtained on the test set, which was 99%. This variation is not unusual since the test set consists of carefully curated data that the model had never encountered before but they are extracted from the same sources. Whereas the fresh data in the inference phase is pulled from the internet and may contain variables that the model has not been trained on.\n",
    "\n",
    "The report also shows that the model was successful in predicting the target class with 100% precision. This means that of all the instances predicted as the target class, every single one was correct. However, the recall score for the target class was only 50%. This means that the model correctly identified only half of the actual target class instances in the sample. This could indicate that the model is not as sensitive to detecting the target class as it should be, and may require further fine-tuning or adjustment to improve its performance in this area."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc609ec7",
   "metadata": {},
   "source": [
    "# Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09fe45eb",
   "metadata": {},
   "source": [
    "The 1D multi-feature stacking technique proves to be superior to the image mel-spectrogram classification in detecting noise in the given task. The accuracy achieved by the SVC classifier and 1D CNN surpassed that of the 2D CNN in Approach I. Furthermore, when assessing the model's ability to identify the class of interest as opposed to all other classes, the superiority of the 1D multi-feature stacking technique becomes even more apparent. Combining all other classes into a single class showed that the first approach struggled to identify the majority of instances of the target class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffa22c0",
   "metadata": {},
   "source": [
    "The poor performance of the first approach could be due to many reasons including the nature of the gun_shot class. Gunshots are loud and quick. Other types of noises, such as children_playing, happen continously over an extended period of time and they have a relevantly smooth beginning and end. On the other hand, gunshots occur suddenly and their sounds go from extremely low frequency (zero if no background noise is present) to a momentarily very high frequency then back to low frequency again. This results in mel-spectrograms similar to the two examples displayed below. As seen below, the images are mostly black with a small portion of sound variation. Similar images might be confusing for the CNN as they don't provide a big variation in features and they are mostly blank. This might caused the CNN to struggle in learning the right features for the interest class. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15350932",
   "metadata": {},
   "source": [
    "<img src=\"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAUDBAQEAwUEBAQFBQUGBwwIBwcHBw8LCwkMEQ8SEhEPERETFhwXExQaFRERGCEYGh0dHx8fExciJCIeJBweHx7/2wBDAQUFBQcGBw4ICA4eFBEUHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh7/wAARCAFyA7QDASIAAhEBAxEB/8QAHQABAAMAAgMBAAAAAAAAAAAAAAEICQIDBAUGB//EAFUQAQAAAgYECwMFCQ0HBQAAAAABAgMEBgcRMQUYIUEJEhdVV4WUlcTS1BNRcQgyM3KyFSI1NmFzdLPBFBYjNDdWdYGSobHC0SQlQkNikaInRlJTg//EABoBAQEBAAMBAAAAAAAAAAAAAAABAgMEBQb/xAAtEQEAAgIBAgMHBAMBAAAAAAAAARECAwQFEhUhMRMUMjNBQlE0NVJhcXKBIv/aAAwDAQACEQMRAD8AuWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAjjQxwx2+5Kv/AMsG+m1F0MbL/vaqGhq391v3X7f7o0NJPCX2XseLxeJSSZ+0jjjjlAFgBQDXWvShshoGxsfjU6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXXvT5gsZ2Os+oKF/xQDXXvT5gsZ2Os+oNde9PmCxnY6z6goX/FANde9PmCxnY6z6g1170+YLGdjrPqChf8UA1170+YLGdjrPqDXWvT5hsb2Os+oKF/4xhCEYxjhCGcSG3JQDXUvQj87QNjcN/wDslZ/u/h1/5dkAAAEcaGOGO33JV/8Alg302ouhjZf97VQ0NW/ut+6/b/dGhpJ4S+y9jxeLxKSTP2kccccoAsAKAa616UNkNA2Nj8anWfUGuvenzBYzsdZ9QUL/AIoBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/igGuvenzBYzsdZ9Qa696fMFjOx1n1BQv8AigGuvenzBYzsdZ9Qa696fMFjOx1n1BQv+KAa696fMFjOx1n1Brr3p8wWM7HWfUFC/wCKAa696fMFjOx1n1Brr3p8wWM7HWfUFC/4oBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/AIoBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/igGuvenzBYzsdZ9Qa696fMFjOx1n1BQv8AigGuvenzBYzsdZ9Qa696fMFjOx1n1BQv+KAa696fMFjOx1n1Brr3p8wWM7HWfUFC/wCKAa696fMFjOx1n1Brr3p8wWM7HWfUFC/4oBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/AIoBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/igGuvenzBYzsdZ9Qa696fMFjOx1n1BQv8AigGuvenzBYzsdZ9Qa696fMFjOx1n1BQv+KAa696fMFjOx1n1Brr3p8wWM7HWfUFC/wCKAa696fMFjOx1n1Brr3p8wWM7HWfUFC/4oBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/AIoBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/igGuvenzBYzsdZ9Qa696fMFjOx1n1BQv8AigGuvenzBYzsdZ9Qa696fMFjOx1n1BQv+KAa696fMFjOx1n1Brr3p8wWM7HWfUFC/wCKAa696fMFjOx1n1Brr3p8wWM7HWfUFC/4oBrr3p8wWM7HWfUGuvenzBYzsdZ9QUL/AIoBrr3p8wWM7HWfUGutenzDY3sdZ9QUL/xjCEIxjHCEM4kNuSgGupehH52gbG4b/wDZKz/d/Dr/AMuyAAAAAAACmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACApnwmud3/WXhVzFM+E1zu/6y8KsCmYDQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ3ITuAhlFsvBjRDKLZeDMgAgAAAAKZ8Jrnd/1l4VcxTPhNc7v+svCrApmA0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAICmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACAAAAApnwmud3/WXhVzFM+E1zu/6y8KsCmYDQAAAAAAAAAAAAAAAAAAAAAAAAAAAYACcEABgAAAAAAAAAAAAAAAAAJ3ITuAhlFsvBjRDKLZeDMgAgKZ8Jrnd/1l4VcxTPhNc7v+svCrApmA0AAAAAAAAAAAAAAAAAAAAAAAAAACcIodtDJLNjxpoywh+RY8x14Idk8IQmjCWMYw/LDBwjmTFCAEAAAAAAAAAAAAAAAABO5CdwEMotl4MaIZRbLwZkAEAAAABTPhNc7v8ArLwq5imfCa53f9ZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAAAEwyQ5S5AjaYe97Op0Wjp6KSFJGsRpY7Iwkwwx/7OjSVDR0NNCWikppIcXGMKWGEXLOmYx7lp4m33OLlFxcSAAAAAAAAAAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAICmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAABgCYYYwexqM9VoONGlhRT4wynkjH/CL121MI+9vDLtmx5NbpqOaln9nR0fFjHZxZYweLHNKMEyz7psQJwQyAAAAAAAAAAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAIAAAACmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAAABgDso6WeSMJpZowjCOMPyOdPT0tPNxqWkmnjCGEIzRxdUBruyqhEUJ3oZAAAAAAAAAAAAAAAABO5CdwEMotl4MaIZRbLwZkAEBTPhNc7v8ArLwq5imfCa53f9ZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAHKENji5QyCEYRinDDN2UMJIx2zYRfTaZ0Houho6ONRr09PGM0eNCMsIYQcurRlsi4bx1zlEzD5aEMSaWMI7YRe7l0dJVqGNao6allpKOHHljxdmPxeDWdJV2s0nGpKxNPNhhjhBrZpnDyyScZiPN4KN7thRUs00cJJoxzycZ6KkljGE0sYOLsy/CVNW6xOEccExkmhDGMI4JUo4icEIAAAAAAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAIAAAACmfCa53f8AWXhVzFM+E1zu/wCsvCrApmA0AAAAAAAAAAAAAAAAAAAAAAAAAACYZITDIEwhGMcIJnlmljhNDCKIRw2wjgTzTTRxmjjFRCAQAAAAAAAAAAAAAAAAE7kJ3AQyi2XgxohlFsvBmQAQFM+E1zu/6y8KuYpnwmud3/WXhVgUzAaAAAAAAAAAAAAAAAAAAAAAAAABMqHLDZAH01mdDSV6oxpo1Okpo+0jLxoTYQyh/q+4s3d5Pp2lpqOqaMrdHNRSwmmjDjT5x/qfR/J7r1mKrYis0em6Opz1j7oTzSxppONhL7OT8nxfttmrcXU2VnpqWbSOiJo1iEJcPYRkwwxx28WLky6lnp19uvXN/mn1HG4OjDjxty85/CvWnrK6Q0NomuVanrc8tBVqKbj0EavxY8XCMcIxxi+K0TSWV9jNGtaPhNSceOGNamlwhs/J8VkrX6W0RbGv1+h0bouqx0dpT+Coq9R7YQlmhxYzwhhCOza/LLT3Qxq1dll0fWY1yjmo4TzTQq8JMJuNHZ87H3ObTz/bREbYiJdnl9M2R27tGEV9XgWhrlkKtoX2mg65QUVcxlhCPtoz/e74YRwfnWn65Wq1WJ55KSFPJGTixmlo4Q2PEr+h6/Upp56Wrzwkln4u33udW0nR1bR1LU6SpSTUk8JoceMdsMYYe7dg5faTGM45xT5/mcnLflWePbT1X30uOyMMfe9hTU1am0bLLNQTQosNk+Gx67HHfsebHSE0apLV5pMYSww+c62qcal5rwYowdlJNLPNjLLCX4Oylq8JZYRknhNHBiMJn0V4w5TSxhHCMNsHFj0AAAAAAAAAABO5CdwEMotl4MaIZRbLwZkAEAAAABTPhNc7v+svCrmKZ8Jrnd/1l4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAAAAwBMMN5HDcYIwAAAAAAAAAAAAAAAAAAATuQncBDKLZeDGiGUWy8GZABAUz4TXO7/rLwq5imfCa53f9ZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAEwQmAPZ6O07pKoVeNBVKxCjkjNGaMOJLHGMYYb4JptJaQr8vFpaaE/F24cWENv/Z6zF31OnhQRmjhCOMHLhsm+2fRv2mdV3PvLHW70zo+saP0d+76SWgoqWEsZIUMkfveNjhjF+h121mm61SQpdG1+aSghLCE0J6GTGMc/d7n4NRTwlrUta2YQnhNh8IvLn01LNGEfYQhhDD5/97uaJ0YxM5x5/R6vF6xu06p1zNvI01pysVqinoaSmnmm4+MYRkhhj/U9DST8abGKJvvpoxRGEHV27stk+bytmc55TlLyajPVYcaFZlmmh/w4Y/6uusTUUZ5vZyxhLjsxdOw3OPv/APNMEscIvb6Op9Gy8aNaopp4YQwwx2f3vUYJhs2Lq3TqnyWJp31+ehmrU81XhGFHGP3sI5vGSRYme6ZlEAIAAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAIAAAACmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAAAmGSEwyBMNkUzzcaOWDjigAAAAAAAAAAAAAAAAAABO5CdwEMotl4MaIZRbLwZkAEBTPhNc7v+svCrmKZ8Jrnd/1l4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAABzkhxowhjvwedpPR8KnR0c/tePx4Y5ZPAkzd1ZoqajlljSxxhHLbi5cJjtm4HXCf73BwiQyQ47mY8w/rAwQEwRgmGQPdaI0NLXqpNTRrEKPCaMuHFxxwhi9RSS8WeMvuTJSTywjCWaMPgmio5qWaMJcNjmzyxyiIiPNqZinWje7Z6CeXHHDZm6o5uKYmPVlACAAAAAAAAAnchO4CGUWy8GNEMotl4MyACAAAAApnwmud3/WXhVzFM+E1zu/6y8KsCmYDQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ3ITuAhlFsvBjRDKLZeDMgAgKZ8Jrnd/1l4VcxTPhNc7v+svCrApmA0AAAAAAAAAAAAAAAAAAAAAAAACcNiAEy4w2vJrNZmppZZZoYcXY5VSjkmooxmlhHa9pX6nVpJZeJRS4x/K7WrjZZ4XEkRfo9DhscXuaSrUMKnNNCjhjCXHF6eObi26/ZzQhMMkJcQYYphCL2WhKCipva+1khPhhht/K8euSSSU9JCSEISwm+9+Dm9hPs+8h4sIRey0JV/azUnGmjLhDHL8qdF1ehpKvNNSSQmjCbZjF+1S2W0BV4x4mjKOSMc48aaOzH4ufj6O2e+Xo8Dp+XLuYmqfh2kIxo6zSUUNuEcMXhRze8txQ0VWtPXqKgkhJRy0sYQlhueji623K85dLbhOGc4z9EAONxgAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAIAAAACmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACApnwmud3/AFl4VcxTPhNc7v8ArLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAmWCHKWOAQ9loyWHsIxmhCMON/o+mtBQUdHRUPFo4QjjhGP9T4qjpp5IYSzzSw/JHB30mkK3SwhCkrNLNCEcds8Xo6Objr19tOXDOMcZiiu01JLTUlHCePFyweJFynmjNNGaM0Yxjvi4OhllOU3LiHKGTimEdjI93ZmGPtsYQjDY83TVVoJdGTUsKKEJ8YbcHzlDT0lDCPs6SeXHPizYOdJXKxS0cZKSnnml90Y4vR18zDDROvKLlyY7IjGph7qz9HJGpTYwx+/j/hB5ekbS6do6OSMulK1DjY7ITRy3PmaGtU1FJGWjpZ5YZ4QmwcJ6aef50803xjixlyo9nGMQuG7PCKxmnOvVqnrlapKxWKWalpJ5sZppo4xi6ImxDo3bjmZn1ABAAAAAAAABO5CdwEMotl4MaIZRbLwZkAEAAAABTPhNc7v+svCrmKZ8Jrnd/1l4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAAABOAIHKWSabKBNJNDOC1MLX1cQwEQAAAAAAAAAAAAAAAATuQncBDKLZeDGiGUWy8GZABAUz4TXO7/rLwq5imfCa53f9ZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAE7kJ3AQyi2XgxohlFsvBmQAQAAAAFM+E1zu/wCsvCrmKZ8Jrnd/1l4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAAAcoZOLlCGwH2NmqGinoqpCajkmxm24ywjveNeHRUVHpSihRUUlHCNBCP3sMMfvpv2PMsv8yp47pofaeLePh906H9Hh9qZ7vLwx91xmno5YR7vb5WOcXFy96Hgw85ACgAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACApnwmud3/AFl4VcxTPhNc7v8ArLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACAAAAApnwmud3/WXhVzFM+E1zu/6y8KsCmYDQAAAAAAAAAAAAAAAAAAAAAAAAAAOUmcHFyl3A/XLG6MqM9nalWJqCEaSMsY8bjRz40fyvlb1JJZNM0EJcv3ND7U77axP4qVH6kftRfF3r/hqg/RpftTvX5OUzx4fR83XjjwsZiPV8V70J96Hjw+cAFAAAAAAAAAAAAAABO5CdwEMotl4MaIZRbLwZkAEBTPhNc7v+svCrmKZ8Jrnd/1l4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATuQncBDKLZeDGiGUWy8GZABAAAAAUz4TXO7/AKy8KuYpnwmud3/WXhVgUzAaAAAAAAAAAAAAAAAAAAAAAAAAAAByl3fFxcpM4Y+8WH7RYmMP3qVH6kftRfF3r/hmg/Rpftzvv7BVOFLZHR83tIy4yR3f9UXxd79U9lp+ryQnxxqkscY/WneptyjLRGMPqefqyx4GOT8/jBDnHZscXlvlYQAAAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACApnwmud3/WXhVzFM+E1zu/6y8KsCmYDQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ3ITuAhlFsvBjRDKLZeDMgAgAAAAKZ8Jrnd/wBZeFXMUz4TXO7/AKy8KsCmYDQAAAAAAAAAAAAAAAAAAAAAAAAAAOUuUHFylygD9+u8/E7Rv5uP2ovir5vxjq36HL9ud9rd5+J2jfzcftRfFXzfjHVv0OX7c7vR8MPtOpfteL83n+c4xcp/nOMXSn1fFoAQAAAAAAAAAAAAAAE7kJ3AQyi2XgxohlFsvBmQAQFM+E1zu/6y8KuYpnwmud3/AFl4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATuQncBDKLZeDGiGUWy8GZABAAAAAUz4TXO7/rLwq5imfCa53f9ZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAAAHKXKDi5Sg/frvPxO0b+bj9qL4q+b8Y6t+hy/bnfa3efido383H7UXxV834xVb9Dh9ud3/th9p1L9rxfm8/znGLlPni4xdGfV8WgBAAAAAAAAAAAAAAATuQncBDKLZeDGiGUWy8GZABAUz4TXO7/rLwq5imfCa53f9ZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAAThHAECcImEQQAAAWAYJwLECcDAsQJwQWAnAwiCAAAAAAAAE7kJ3AQyi2XgxohlFsvBmQAQAAAAFM+E1zu/wCsvCrmKZ8Jrnd/1l4VYFMwGgAAAAAAAAAAAAAAAAAAAAAAAAAAc5dzg5SYBD9+u8/E7R2XzI/amfFXy7bR1X9Dl+3OsP8AJ9sboHS1gLM01dq1JPPWJPv4wpZpcf4SaG7+p4HyjLtrK6PtpU6CrVOlkkjo+SaONPPH/mUicfqGvbnOvH1h9lysvb8XDjR61an0yHfW5YS1mklhlCaMIOiOaz6y+OyjtmkACAAAAAAAAAAAAAACdyE7gIZRbLwY0Qyi2XgzIAICmfCa53f9ZeFXMUz4TXO7/rLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAA5QwwcXKAJ2x3IwjjlF9hZDQej9IaLnpqzRzTTwpYywjCbDdD/AFezrVmtEyywjLQzwx/6ovQ0dO27ce7F3dfB2bMe6H55hH3G33Puo2e0ZD/lTf2ooks/ozCONFN/ac3g/I/pPcdj4bb7jCPufb/cDRv/ANU39pzo7PaMjDbQzf2jwff/AEvuOx8LCEfcnCL7mls/oyXKim/tOufQGjYS4+ym/tHg+/8ApPcdj4nCPuIQj7n6BUbOaKpKCaaehnjNCbCH3/5HTCz+jIxjD2M39o8H3/014fsp8LhH3GEfdF95NZ7RcJIx9jPjD/qcKvoDRk/G41HNsjD/AIo5J4Pvq5T3DZdPh8ERg/TtMWT0NVtHQpaOhn4+MIfPjvfCWgqtDVa77OghGEsZYR2xdfbws9WHdkxyOJnx/ierihOcYodJ1QAAAAABO5CdwEMotl4MaIZRbLwZkAEAAAABTPhNc7v+svCrmKZ8Jrnd/wBZeFWBTMBoAAAAAAAAAAAAAAAAAAAAAAAAAAHKRxcpBYXy+TF/JzZD6sP1syPlQfj3Uv6Nk/WUifkxfyc2Q+rD9bMj5UH491L+jZP1lI8Tpf6zP/L67V8/X/qotX/43S/Wi8aLya//ABul+tF40XuS+U2/HKAEcYAAAAAAAAAAAAAAnchO4CGUWy8GNEMotl4MyACApnwmud3/AFl4VcxTPhNc7v8ArLwqwKZgNAAAAAAAAAAAAAAAAAAAAAAAAAmGSEwyB+iXe/gSf8/H/CV7it/RyPT3e/gSf8/H/CV7it/RyPrunfJh9LxfkQ8ObeiXKKZt6JcovS+pDjDN2UXzYuuGbsovmxCEU+TrpPo3ZT5Ouk+jSEeZoz6Cb637HRJ86Pxd+jPoJvrfsdEnzo/EhyT6Qmk+im+DqqmU/wAIftdtJ9FN8HVVMp/hD9p9ss/dD6C0P4Kh9aV+WWq/CMPqSv1O0P4Kh9aV+WWq/CMPqSvE6j+n/wCuv1f0h6WG83kN5vfNw8FAAAAAACdyE7gIZRbLwY0Qyi2XgzIAIAAAACmfCa53f9ZeFXMUz4TXO7/rLwqwKZgYNAJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQ5SIwTKEL5fJi/k5sh9WH62ZHyoPx7qX9GyfrKQ+TH/JzZD6kP1sx8qD8e6l/Rsn6ykeH0v9Zn/l9fq+fr/wBVFq//ABul+tF40XkV7+N0v1ovHe7L5Tb8coE4GCONAnAwBAnAwBAnAwBAnAwBAnAwBAnAwBAnAwBCdxgbgIZRbLwY0Qyi2XgzIAICmfCa53f9ZeFXMUz4TXO7/rLwqwKZgYNAJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQJwMAQmGRgbgfol3v4En/Px/wle4rf0cj093v4En/Px/wle4rf0cj67p3nph9LxfkQ8ObeiXKKZt6JcovTqRxhm7KL5sXXDN2UXzYpUkIp8nXSfRuynyddJ9GREo8zRn0E31v2OiT50fi79GfQTfW/Y6JPnR+JEebc+kJpPopvg6qplP8ACH7XbSfRTfB1VTKf4Q/akxWMp90PoLQ/gqH1pX5Zar8Iw+pK/U7Q/gqH1pX5Zar8Iw+pK8TqH6f/AK6/V/SHpYbzeQHzcPBQJwMAQJwMAQJwMAQncYG4CGUWy8GNEMotl4MyACAAAAAppwmmd3/WXhVy3pLT2Ssxaear/vks3obTUKtxvYfdCpUdY9lxsONxYTyxwx4sMcPdAgZBm1rJC6a63DbdtY2Px0HVo/5Dklus6NbGdxVbyLYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byLYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2plxax8kt1nRrYzuOreRHJNddCOy7WxncdW8iWPw/5Mn8nVkYYwx4kNn/6THyoYRhb2pQjCMP8Adsn6ykWK0XZSzmi6vQ1fRugdGVKgoPoqKr1SSjkk24/eyywhCG3ajS1lbO6VrMKxpPQOi67Syy8SWesVSjpJoS444Q40I7NsXQ4vEnTuy2X6vZx6rGOeOdekUyLrsf8Aa6WMN80Xj7WsfJNdfHGM129jZpoxjGMY6Dq0Y/YIXTXW4bbtbGdx1byPQ7vOXkZ5d0zLJzabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWyyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSa6zo1sZ3FVvIWMm4Q2RbLwyfFxumuvhjGS7exsI7v9yVeGH/AIbH2csMJcEsSAgKacJpnd/1l4Vct6S09krMWnmq/wC+SzehtNQq3G9h90KlR1j2XGw43FhPLHDHiwxw90CBkGbWskLprrcNt21jY/HQdWj/AJDklus6NbGdxVbyLYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byLYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9pDHa1k5JbrOjWxncdW8iOSa63b/AOmtjO46t5EmRm/d7h9xJ/z8fsyvcVzCEkuMYbPytDaG7G7ugkjJV7B2XoJc+LR6JoJYY/CErnya3fTfOsPZqPVdD5XscbqsacIx7bepq6hGvXGFM5I5RRLlFo1G7K7z+Y1me6qDyELsrvIf+xbMd00HkdnxzH+BPUI/DOPCOOUXZR5RaMcmN3W+wll+6aDyHJld3DKwtmIdU0HkPHMf4EdQj8M5qaEcMouukhHiYYRaPcmV3cc7CWY7poPIjkwu6/mHZfuig8hHXMY+xJ6hE/Rndo2MP3PNt/4v2OiT50fi0alu0u+llwksPZmWHuhoqgh/lRyZXefzGsz3VQeRPG8f4NeIxXoznpPoo/B01TDCfbDdD/Fo9yZ3e/zFsxHqqg8iIXZ3eS/NsJZmHw0VQQ/yk9civhPEYu6UCtDGH3JhHGGHGlfltqvwjD6kGqU929gp5eLPYuzs0v8A8Y6MocPsuie6q7Kkm49Ld1ZCkm98+havNH/vGR0uT1GN2vsiHHzObHIryZN4bEbWsnJNdb0a2M7jq3kOSW6zo1sZ3HVvI83uecyb2m1rJyS3WdGtjO46t5Dklus6NbGdx1byFjJvabWsnJLdZ0a2M7jq3kOSW6zo1sZ3HVvIWMm9ptayckt1nRrYzuOreQ5JbrOjWxncdW8hYyb2m1rJyS3WdGtjO46t5Dkmus6NbGdxVbyFjJuENkWy8Mnxcbprr4Yxku3sbCO7/clXhh/4bH2csMJcEsSAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//9k=\" \n",
    "/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36d39aa8",
   "metadata": {},
   "source": [
    "Furthermore, computer vision tasks require a huge amount of data in order to achieve satisfactory results. The training set included 700 instances per class which is relatively small for similar tasks. More instances and more variation of sound environments could benifit the model for a better detection accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6516e1e",
   "metadata": {},
   "source": [
    "The multi-feature stacking solves those issues by providing much more data about each sound. This approach provides a more comprehensive understanding of each sound, enhancing the model's ability to classify them accurately. By combining the knowledge gained from mel-spectrogram representations with other audio features, the model gains greater insight into the nature of the sound and its various aspects, resulting in improved classification performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40ee6fd1",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21baf408",
   "metadata": {},
   "source": [
    "This notebook details the implementation of a gunshot detection system using deep learning. A dataset of 8732 labelled sound files divided into 10 different classes was used. The necessary data preprocessing techniques are applied to the data to prepare for training and to address some issues with the data such as the class imbalance. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68445aa8",
   "metadata": {},
   "source": [
    "Two approaches were tested for the task, with the first one involving the conversion of sound files into mel-spectrogram images. A 2D CNN model was then trained and evaluated on these images for mel-spectrogram image classification. Although the model achieved a 73% accuracy in classifying sounds into any of the ten classes, it performed poorly in identifying the target class, i.e., gunshot sounds. Therefore, this approach was found unsuitable for the current task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11ff21a1",
   "metadata": {},
   "source": [
    "The second approach for the gunshot detection system involves extracting five distinct 2D features from each sound instance. These features are then transformed into 1D vectors by averaging across the x-axis, and subsequently stacked in a particular order to form a long 1D vector representing the sound instance. Two models were trained using this approach, SVC and 1D CNN. During the validation phase, both models demonstrated impressive results, with the CNN outperforming the SVC with validation accuracy scores of 86% and 75%, respectively. The CNN model was then evaluated on the test set, and it produced remarkable outcomes.\n",
    "\n",
    "The CNN model achieved an overall classification accuracy of 80% in identifying sounds among the ten classes. However, when examined for the target class, the model demonstrated a 99% accuracy in distinguishing gunshot sounds from all other classes. This result confirms that this approach utilizing deep learning is the most appropriate for the given task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "104352ae",
   "metadata": {},
   "source": [
    "When applied to fresh unseen sound clips pulled from the internet, the 1D CNN achieved an accuracy of 80%. While this value is lower than the accuracy score obtained on the test set, it is still a satisfactory result. The precision score for the target class was 100%, however, the recall score for the target class was only 50%, suggesting that the model may require further improvement to increase its sensitivity to detecting the target class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8751bc57",
   "metadata": {},
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceb74993",
   "metadata": {},
   "source": [
    "\\[1\\]   Y. Tanoko and A. Zahra, “Multi-feature stacking order impact on speech emotion recognition performance,” Bulletin of Electrical Engineering and Informatics, vol. 11, no. 6, pp. 3272–3278, Dec. 2022, doi: 10.11591/EEI.V11I6.4287."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11e2f0d5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}