examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf969b3b-a1f0-410c-b5a4-f43e67a92c7d",
   "metadata": {},
   "source": [
    "# mac-llmops Examples\n",
    "\n",
    "First we initialize our model manager. It auto-detects memory size and uses the default HF cache location. Both can be overridden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6fccd576-4519-4cf6-b736-eacd12802d6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mac_llmops\n",
    "\n",
    "\n",
    "models = mac_llmops.Models()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb23349b-a636-403e-86f6-edb685a9b13a",
   "metadata": {},
   "source": [
    "We can display all models in our local cache that might fit in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c7937d89-8996-48be-9775-a50e14cf769a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mlx-community/Llama-3.2-3B-Instruct',\n",
       " 'mlx-community/Llama-3.3-70B-Instruct-3bit',\n",
       " 'mlx-community/Meta-Llama-3.1-8B-Instruct-bf16',\n",
       " 'mlx-community/Llama-3.1-8B-Instruct',\n",
       " 'mlx-community/Meta-Llama-3-8B-Instruct-4bit',\n",
       " 'mlx-community/Meta-Llama-3.1-8B-Instruct-4bit',\n",
       " 'mlx-community/Llama-3.2-3B-Instruct-4bit',\n",
       " 'mlx-community/Meta-Llama-3.1-8B-Instruct-8bit',\n",
       " 'mlx-community/Llama-3.2-3B-Instruct-8bit',\n",
       " 'microsoft/Phi-3-medium-128k-instruct',\n",
       " 'mlx-community/Qwen2.5-Coder-14B-Instruct-bf16',\n",
       " 'mlx-community/phi-4-bf16',\n",
       " 'ibm-granite/granite-3.1-8b-instruct']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4b25ada-1864-445c-a387-4d6912d57a29",
   "metadata": {},
   "source": [
    "Or search the local cache for specific models that might fit in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6e180376-bb71-40a7-b3e3-b8006d8235ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models.search_local(\"mlx llama 3.2 bf16\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ff1ad0d-7992-4299-8c17-31a5385c439e",
   "metadata": {},
   "source": [
    "Since there were no local results, we can do the same search but online."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aa563267-0a62-4768-97d0-10d066ac21a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mlx-community/Llama-3.2-1B-Instruct-bf16',\n",
       " 'mlx-community/Llama-3.2-3B-Instruct-bf16',\n",
       " 'mlx-community/Llama-3.2-3B-bf16',\n",
       " 'mlx-community/Hermes-3-Llama-3.2-3B-bf16']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models.search_online(\"mlx llama 3.2 bf16\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b68250d7-4777-48d8-921a-e8ab0e068b51",
   "metadata": {},
   "source": [
    "If we find a model we like, it can be added to the local cache."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "17349af0-a902-4fb8-84f6-91e0c2c46541",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "96af1c072c584388a1c2ab140fb85eaf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 8 files:   0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a72875abfd8543508b32b398f8a0e412",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/969 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34558e9e66564b62915ade160eab9826",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/16.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e74f2a5e555493ea938e6fa05cbe612",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/296 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6703d27b5641434a8693ecbd581ab047",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/54.5k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4772d690d149410cb3f4fd77688ddda2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors.index.json:   0%|          | 0.00/21.9k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5acdbfa2cbea4d14a6e3bb6dfc16aefc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/9.09M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07273f63dabc4ba9a21a7bd2b3580aff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00001-of-00002.safetensors:   0%|          | 0.00/5.37G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8f59e2223aa4118b38e2d69220c56e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model-00002-of-00002.safetensors:   0%|          | 0.00/1.06G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "models.add(\"mlx-community/Llama-3.2-3B-Instruct-bf16\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74a0512f-29aa-4999-87a5-e72b6f8fe4ac",
   "metadata": {},
   "source": [
    "Now the model is in our local cache."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f2518c29-db28-4d26-92f1-33b1fea8d953",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['mlx-community/Llama-3.2-3B-Instruct-bf16']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models.search_local(\"mlx llama 3.2 bf16\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb62d1bd-1c7f-4d27-8cdf-6f7e82850570",
   "metadata": {},
   "source": [
    "If we need to free up some disk space, we can also delete models from the local cache."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b14d6c7d-a055-4ac3-bb45-c0012693ca27",
   "metadata": {},
   "outputs": [],
   "source": [
    "models.delete(\"mlx-community/Llama-3.2-3B-Instruct-bf16\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cff3fef4-049b-4bba-9a1e-89b859932ebb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models.search_local(\"mlx llama 3.2 bf16\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}