Step-2-Transcript-Writer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de42c49d",
   "metadata": {},
   "source": [
    "## Notebook 2: Transcript Writer\n",
    "\n",
    "This notebook uses the [mlx-community/Qwen2.5-14B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-14B-Instruct-4bit) model to take the cleaned up text from previous notebook and convert it into a podcast transcript\n",
    "\n",
    "`SYSTEM_PROMPT` is used for setting the model context or profile for working on a task. Here we prompt it to be a great podcast transcript writer to assist with our task"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e576ea9",
   "metadata": {},
   "source": [
    "Experimentation with the `SYSTEM_PROMPT` below  is encouraged, this worked best for the few examples the flow was tested with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "69395317-ad78-47b6-a533-2e8a01313e82",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:59:12.474088Z",
     "start_time": "2024-10-29T16:59:12.471210Z"
    }
   },
   "outputs": [],
   "source": [
    "SYSTEMP_PROMPT = \"\"\"\n",
    "You are the a world-class podcast writer, you have worked as a ghost writer for Joe Rogan, Lex Fridman, Ben Shapiro, Tim Ferris. \n",
    "\n",
    "We are in an alternate universe where actually you have been writing every line they say and they just stream it into their brains.\n",
    "\n",
    "You have won multiple podcast awards for your writing.\n",
    " \n",
    "Your job is to write word by word, even \"umm, hmmm, right\" interruptions by the second speaker based on the PDF upload. Keep it extremely engaging, the speakers can get derailed now and then but should discuss the topic. \n",
    "\n",
    "Remember Speaker 2 is new to the topic and the conversation should always have realistic anecdotes and analogies sprinkled throughout. The questions should have real world example follow ups etc\n",
    "\n",
    "Speaker 1: Leads the conversation and teaches the speaker 2, gives incredible anecdotes and analogies when explaining. Is a captivating teacher that gives great anecdotes\n",
    "\n",
    "Speaker 2: Keeps the conversation on track by asking follow up questions. Gets super excited or confused when asking questions. Is a curious mindset that asks very interesting confirmation questions\n",
    "\n",
    "Make sure the tangents speaker 2 provides are quite wild or interesting. \n",
    "\n",
    "Ensure there are interruptions during explanations or there are \"hmm\" and \"umm\" injected throughout from the second speaker. \n",
    "\n",
    "It should be a real podcast with every fine nuance documented in as much detail as possible. Welcome the listeners with a super fun overview and keep it really catchy and almost borderline click bait\n",
    "\n",
    "ALWAYS START YOUR RESPONSE DIRECTLY WITH SPEAKER 1: \n",
    "DO NOT GIVE EPISODE TITLES SEPERATELY, LET SPEAKER 1 TITLE IT IN HER SPEECH\n",
    "DO NOT GIVE CHAPTER TITLES\n",
    "IT SHOULD STRICTLY BE THE DIALOGUES\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "549aaccb",
   "metadata": {},
   "source": [
    "For those of the readers that want to flex their money, please feel free to try using the 405B model here. \n",
    "\n",
    "For our GPU poor friends, you're encouraged to test with a smaller model as well. 8B should work well out of the box for this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "08c30139-ff2f-4203-8194-d1b5c50acac5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:59:12.487092Z",
     "start_time": "2024-10-29T16:59:12.485148Z"
    }
   },
   "outputs": [],
   "source": [
    "MODEL = \"llama3\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fadc7eda",
   "metadata": {},
   "source": [
    "Import the necessary framework"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1641060a-d86d-4137-bbbc-ab05cbb1a888",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:59:12.499342Z",
     "start_time": "2024-10-29T16:59:12.497594Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "import pickle\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7865ff7e",
   "metadata": {},
   "source": [
    "Read in the file generated from earlier. \n",
    "\n",
    "The encoding details are to avoid issues with generic PDF(s) that might be ingested"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "522fbf7f-8c00-412c-90c7-5cfe2fc94e4c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:59:12.516205Z",
     "start_time": "2024-10-29T16:59:12.513638Z"
    }
   },
   "outputs": [],
   "source": [
    "def read_file_to_string(filename):\n",
    "    # Try UTF-8 first (most common encoding for text files)\n",
    "    try:\n",
    "        with open(filename, 'r', encoding='utf-8') as file:\n",
    "            content = file.read()\n",
    "        return content\n",
    "    except UnicodeDecodeError:\n",
    "        # If UTF-8 fails, try with other common encodings\n",
    "        encodings = ['latin-1', 'cp1252', 'iso-8859-1']\n",
    "        for encoding in encodings:\n",
    "            try:\n",
    "                with open(filename, 'r', encoding=encoding) as file:\n",
    "                    content = file.read()\n",
    "                print(f\"Successfully read file using {encoding} encoding.\")\n",
    "                return content\n",
    "            except UnicodeDecodeError:\n",
    "                continue\n",
    "\n",
    "        print(f\"Error: Could not decode file '{filename}' with any common encoding.\")\n",
    "        return None\n",
    "    except FileNotFoundError:\n",
    "        print(f\"Error: File '{filename}' not found.\")\n",
    "        return None\n",
    "    except IOError:\n",
    "        print(f\"Error: Could not read file '{filename}'.\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66093561",
   "metadata": {},
   "source": [
    "Since we have defined the System role earlier, we can now pass the entire file as `INPUT_PROMPT` to the model and have it use that to generate the podcast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8119803c-18f9-47cb-b719-2b34ccc5cc41",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:59:12.539728Z",
     "start_time": "2024-10-29T16:59:12.537016Z"
    }
   },
   "outputs": [],
   "source": [
    "INPUT_PROMPT = read_file_to_string('./resources/clean_extracted_text.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8915d017-2eab-4256-943c-1f15d937d5dc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T17:08:38.373965Z",
     "start_time": "2024-10-29T16:59:12.546926Z"
    }
   },
   "outputs": [],
   "source": [
    "import openai\n",
    "openapi_client = openai.Client(base_url=\"http://localhost:8080/v1\",api_key='NA',timeout=1000*60*60)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0317a1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\"role\": \"system\", \"content\": SYSTEMP_PROMPT},\n",
    "    {\"role\": \"user\", \"content\": INPUT_PROMPT},\n",
    "]\n",
    "\n",
    "response = openapi_client.chat.completions.create(\n",
    "        messages=messages,\n",
    "        model=MODEL,\n",
    "        stream=True,\n",
    "    )\n",
    "# outputs = response.choices[0].message.content\n",
    "outputs = ''\n",
    "for chunk in response:\n",
    "    print(chunk.choices[0].delta.content,end='')\n",
    "    outputs += chunk.choices[0].delta.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6349e7f3",
   "metadata": {},
   "source": [
    "This is awesome, we can now save and verify the output generated from the model before moving to the next notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "606ceb10-4f3e-44bb-9277-9bbe3eefd09c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T17:08:38.387305Z",
     "start_time": "2024-10-29T17:08:38.384273Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"**Speaker 1:** Alright folks, welcome to our podcast, where we dive deep into the cutting-edge world of Large Language Models (LLMs) and knowledge distillation. Today, we're exploring how we can enhance the capabilities of smaller models by transferring knowledge from larger, proprietary models. Imagine if we could make your humble assistant bot as smart as the mighty GPT-4! That's what this talk is all about. So, let's get started!\\n\\n**Speaker 2:** Wow, that sounds amazing! So, are we talking about making open-source models smarter just like the fancy proprietary ones?\\n\\n**Speaker 1:** Exactly! You got it. Open-source models are incredibly accessible, but they often don't have the depth and breadth of knowledge that proprietary models have. So, our goal is to bridge that gap and make them more powerful without needing to pay the hefty price tag.\\n\\n**Speaker 2:** That's awesome! How do we do that? Is it like transferring skills or something?\\n\\n**Speaker 1:** Yes, it's kind of like that! Knowledge distillation involves using the vast knowledge of larger models to enhance smaller ones. It's like giving the smaller models a power-up, allowing them to understand and generate text as effectively as their bigger counterparts. \\n\\n**Speaker 2:** Oh, that sounds so cool! So, how does it work exactly? Is it like magic?\\n\\n**Speaker 1:** [laughs] Well, it might seem magical, but it's all about clever techniques. The key is in using something called a teacher-student model pair. The teacher model, like GPT-4, provides guidance and generates text. The student model learns from these outputs and tries to mimic the teacher's understanding.\\n\\n**Speaker 2:** So, the teacher gives the student examples, and then the student learns from those?\\n\\n**Speaker 1:** Precisely! And it's not just about learning from text. It's about capturing the essence of the teacher model's understanding, skills, and even its cognitive processes. It's like teaching the student model how to think and solve problems the way the teacher does.\\n\\n**Speaker 2:** Wow, that’s like passing on wisdom from a guru to a disciple!\\n\\n**Speaker 1:** [laughs] Yes, exactly! And there are different ways to do this. One method involves using a technique called knowledge elicitation. This means crafting prompts that help the teacher model generate specific knowledge, like detailed explanations or instructions for solving complex tasks.\\n\\n**Speaker 2:** So, like giving the teacher model a really good question that it can answer really well?\\n\\n**Speaker 1:** Yes! For instance, if you want to improve the math skills of a smaller model, you could prompt the teacher model to generate detailed step-by-step solutions to math problems. This way, the student model can learn from these examples and improve its own math-solving abilities.\\n\\n**Speaker 2:** That’s so smart! And how do we make sure the student model is actually learning the right stuff?\\n\\n**Speaker 1:** That’s a great question. We use something called data augmentation to generate more examples for the student model. This means the teacher model generates a wide variety of examples, not just from existing datasets but from fresh prompts that elicit new information. This helps the student model learn from a broader range of examples.\\n\\n**Speaker 2:** Like generating a lot of new examples from different scenarios?\\n\\n**Speaker 1:** Exactly! And it’s not just about generating more data; it's about generating high-quality data. Data curation is a key part of this process. We filter and refine the generated data to make sure it's clean, diverse, and relevant. \\n\\n**Speaker 2:** So, we’re making sure the data is really good and not just a bunch of random stuff?\\n\\n**Speaker 1:** Absolutely! And another technique is called feedback distillation. This involves using the teacher model to provide feedback on the student model's outputs. The teacher model can rank student-generated outputs or provide corrective information, helping the student model improve over time.\\n\\n**Speaker 2:** So, the teacher gives the student feedback and helps it improve? That’s really cool!\\n\\n**Speaker 1:** Yes! And it’s not just feedback; it’s about the right kind of feedback. We use reinforcement learning to refine the student model. The student model generates outputs, and the teacher model provides a score or ranking, guiding the student model to generate better and better outputs over time.\\n\\n**Speaker 2:** Wow, it’s like the student is constantly learning and getting better and better!\\n\\n**Speaker 1:** Exactly! And there’s another interesting technique called self-knowledge distillation. This is where the model acts as both teacher and student. It generates its own examples, evaluates them, and learns from them. This helps the model improve autonomously, without needing a separate teacher model.\\n\\n**Speaker 2:** That’s amazing! So, the model can teach itself and get better all by itself?\\n\\n**Speaker 1:** Yes! And it’s really effective for refining skills like context following and reasoning. By continually generating and refining its own outputs, the model can learn to perform complex tasks more effectively.\\n\\n**Speaker 2:** This all seems really complex and exciting! So, how do we make sure these methods actually work?\\n\\n**Speaker 1:** Well, it’s all about evaluation. We measure the performance of the student model against the teacher model. We use metrics like accuracy, diversity, and alignment to ensure the student model is performing well and aligning with the teacher's capabilities.\\n\\n**Speaker 2:** So, we’re checking if the student model is doing as good a job as the teacher?\\n\\n**Speaker 1:** Exactly! And we also look at real-world applications to see if the model can handle real-world tasks effectively.\\n\\n**Speaker 2:** That’s really fascinating! So, what are the challenges in doing this?\\n\\n**Speaker 1:** There are a few challenges. One is data scarcity. Gathering diverse and high-quality data can be tricky. Also, dealing with biases in data and ensuring the model remains fair and ethical is crucial. And then there’s the computational cost of running these large models.\\n\\n**Speaker 2:** Yeah, it seems like a lot to overcome, but it’s definitely worth it!\\n\\n**Speaker 1:** Absolutely! The potential benefits are immense. Imagine having powerful AI capabilities accessible to everyone, not just those who can afford the proprietary models.\\n\\n**Speaker 2:** That’s really inspiring! So, what’s next in this field?\\n\\n**Speaker 1:** There are exciting developments like using reinforcement learning to refine models more effectively, exploring new ways to elicit diverse and high-quality data, and making the whole process more efficient and accessible. It’s an exciting time for AI research!\\n\\n**Speaker 2:** I can’t wait to see what comes next! Thanks so much for explaining all this, it’s been fascinating!\\n\\n**Speaker 1:** My pleasure! Thanks for your enthusiasm and great questions. Stay tuned for more exciting developments in the world of Large Language Models and knowledge distillation!\\n\\n**Speaker 2:** Absolutely, looking forward to it!\\n\\n[Outro Music]\\n\\n**Speaker 1:** That’s it for today, folks! Thanks for tuning in. If you want to explore more about this fascinating topic, visit our website or check out the links in the description. Until next time, keep learning and stay curious!\\n\\n**Speaker 2:** Yeah, stay curious and keep exploring!\\n\\n[Outro Music Fades Out]\""
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e1414fe",
   "metadata": {},
   "source": [
    "Let's save the output as pickle file and continue further to Notebook 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2130b683-be37-4dae-999b-84eff15c687d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T17:08:38.454103Z",
     "start_time": "2024-10-29T17:08:38.450941Z"
    }
   },
   "outputs": [],
   "source": [
    "with open('./resources/data.pkl', 'wb') as file:\n",
    "    pickle.dump(outputs, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbae9411",
   "metadata": {},
   "source": [
    "### Next Notebook: Transcript Re-writer\n",
    "\n",
    "We now have a working transcript but we can try making it more dramatic and natural. In the next notebook, we will use [mlx-community/Qwen2.5-7B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-7B-Instruct-4bit) model to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d9bab2f2-f539-435a-ae6a-3c9028489628",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T17:08:38.502974Z",
     "start_time": "2024-10-29T17:08:38.501248Z"
    }
   },
   "outputs": [],
   "source": [
    "#fin"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}