tutorials/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with XGI\n",
    "<div>\n",
    "<img src=\"../logo/logo.svg\" width=\"200\"/>\n",
    "</div>\n",
    "\n",
    "XGI is a Python library to make working with and analyzing complex systems with higher-order interactions easy. We have collected a lot of useful functions, algorithms, and tools for working with hypergraphs and simplicial complexes to make life easier.\n",
    "\n",
    "Main contributors: Nicholas Landry, Leo Torres, Maxime Lucas, Iacopo Iacopini, Giovanni Petri, Alice Patania, Alice Schwarze, and Martina Contisciani.\n",
    "\n",
    "We will\n",
    "1. Create and load hypergraphs\n",
    "2. Visualize hypergraphs and simplicial complexes\n",
    "3. Show how to use the stats interface\n",
    "4. Give an example of how to compare an empirical dataset to a null model.\n",
    "\n",
    "We start off by loading the XGI library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgi\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "from matplotlib.pyplot import cm\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a hypergraph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to start with a hypergraph, and this can be done in several ways:\n",
    "1. Build a hypergraph node-by-node and edge-by-edge (less common, but can be helpful in writing your own generative models)\n",
    "2. Load an existing dataset\n",
    "3. Sample from a random generative model\n",
    "\n",
    "Let's start with the first method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building a hypergraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_build = xgi.Hypergraph()\n",
    "H_build.add_edge([1, 2], id=\"a\")\n",
    "H_build.add_node(0)\n",
    "H_build.add_edges_from([[3, 4], [0, 2, 3]])\n",
    "H_build.add_nodes_from([9, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGI automatically assigns unique edge IDs (if a user doesn't specify the ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_build.edges"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why NodeViews and EdgeViews? These allow users to access many different properties and data structures from nodes and edges in a much simpler way. We will cover this more in depth later. For now, we can get the edges of which each node is a part and the nodes in each edge as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(H_build.nodes.memberships())\n",
    "print(H_build.edges.members())\n",
    "print(H_build.nodes.memberships(2))\n",
    "print(H_build.edges.members(\"a\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving on to method 2, one can load datasets in several different ways. First, we provide a companion data repository, [xgi-data](https://github.com/ComplexGroupInteractions/xgi-data), where users can easily load several datasets in standard format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_enron = xgi.load_xgi_data(\"email-enron\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset, for example, has a corresponding [datasheet](https://github.com/ComplexGroupInteractions/xgi-data/blob/main/data/email-enron/README_email-enron.md) explaining its characteristics. The nodes (individuals) in this dataset contain associated email addresses and the edges (emails) contain associated timestamps. These attributes can be accessed by simply typing `H.nodes[id]` or `H.edges[id]` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The hypergraph has {H_enron.num_nodes} nodes and {H_enron.num_edges} edges\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The first 10 node IDs are:\")\n",
    "print(list(H_enron.nodes)[:10])\n",
    "print(\"\\nThe first 10 edge IDs are:\")\n",
    "print(list(H_enron.edges)[:10])\n",
    "print(\"\\nThe attributes of node '4' are\")\n",
    "print(H_enron.nodes[\"4\"])\n",
    "print(\"\\nThe attributes of edge '6' are\")\n",
    "print(H_enron.edges[\"6\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can clean up this dataset to remove isolates, multi-edges, singletons, and to replace all IDs with integer IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_enron_cleaned = H_enron.cleanup(\n",
    "    multiedges=False, singletons=False, isolates=False, relabel=True, in_place=False\n",
    ")\n",
    "\n",
    "print(\n",
    "    f\"The hypergraph has {H_enron_cleaned.num_nodes} nodes and {H_enron_cleaned.num_edges} edges\"\n",
    ")\n",
    "\n",
    "print(\"The first 10 node IDs are:\")\n",
    "print(list(H_enron_cleaned.nodes)[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, one can load from a file. For example, in the `data` folder, there is a disease-gene hypergraph in bipartite edgelist format, where nodes are diseases and edges are genes. We can read in this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_disgene = xgi.read_bipartite_edgelist(\"../data/disGene.txt\")\n",
    "print(f\"The hypergraph has {H_disgene.num_nodes} nodes and {H_disgene.num_edges} edges\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may also be interested in the *dual* hypergraph formed by considering nodes to be genes and edges to be diseases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "D = H_disgene.dual()\n",
    "print(f\"The hypergraph has {D.num_nodes} nodes and {D.num_edges} edges\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random null models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can create  synthetic hypergraphs using random generative models. For example, we can extract the degree and edge size sequence from this dataset and wire them together at random (according to the Chung-Lu model) to create a random null model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k = D.nodes.degree.asdict()\n",
    "s = D.edges.size.asdict()\n",
    "\n",
    "H_random = xgi.chung_lu_hypergraph(k, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check whether this new hypergraph is connected and if not, the sizes of the connected components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "connected = xgi.is_connected(H_random)\n",
    "if not connected:\n",
    "    size, num = np.unique(\n",
    "        [len(cc) for cc in xgi.connected_components(H_random)], return_counts=True\n",
    "    )\n",
    "    print(f\"The connected components:\")\n",
    "    display(pd.DataFrame([size, num], columns=[\"Component size\", \"Number\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do much more such as computing different properties, reading/writing, converting to/from different data structures, hypergraph null models and much more!\n",
    "\n",
    "See the Read The Docs for more information: https://xgi.readthedocs.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization\n",
    "\n",
    "The first step for drawing a hypergraph is to choose a layout for the nodes. At the moment the available layouts are:\n",
    "* ```random_layout```: positions nodes uniformly at random in the unit square.\n",
    "* ```pairwise_spring_layout```: positions the nodes using the Fruchterman-Reingold force-directed algorithm on the projected graph.\n",
    "* ```barycenter_spring_layout``` and ```weighted_barycenter_spring_layout```: slight modification of ```pairwise_spring_layout```\n",
    "    \n",
    "Each layout returns a dictionary that maps nodes ID into (x, y) coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 0\n",
    "H_viz = xgi.random_hypergraph(20, [0.08, 0.01, 0.001], seed=seed)\n",
    "pos1 = xgi.random_layout(H_viz)\n",
    "pos2 = xgi.pairwise_spring_layout(H_viz)\n",
    "pos3 = xgi.barycenter_spring_layout(H_viz)\n",
    "pos4 = xgi.weighted_barycenter_spring_layout(H_viz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgi.draw(H_viz, pos1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colors of the hyperedges** are designed to match the hyperedge size, but the colormap can be tweaked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sequential colormap\n",
    "cmap = cm.Paired\n",
    "\n",
    "xgi.draw(H_viz, pos4, edge_fc_cmap=cmap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other parameters can be changed as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cmap = cm.Reds\n",
    "dyad_lc = \"gray\"\n",
    "dyad_lw = 4\n",
    "node_fc = \"black\"\n",
    "node_ec = \"white\"\n",
    "node_lw = 2\n",
    "node_size = 20\n",
    "\n",
    "xgi.draw(\n",
    "    H_viz,\n",
    "    pos4,\n",
    "    edge_fc_cmap=cmap,\n",
    "    dyad_lc=dyad_lc,\n",
    "    dyad_lw=dyad_lw,\n",
    "    node_fc=node_fc,\n",
    "    node_ec=node_ec,\n",
    "    node_lw=node_lw,\n",
    "    node_size=node_size,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simplicial complexes can be visualized using the same functions for node layout and drawing.\n",
    "\n",
    "#### Technical note\n",
    "By definition, a simplicial complex object contains all sub-simplices. This would make the visualization heavy since all sub-simplices contained in a maximal simplex would overlap. The automatic solution for this, implemented by default in all the layouts, is to convert the simplicial complex into a hypergraph composed by solely by its maximal simplices.\n",
    "\n",
    "#### Visual note\n",
    "To visually distinguish simplicial complexes from hypergraphs, the ```draw``` function will also show all links contained in each maximal simplices (while omitting simplices of intermediate orders). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SC = xgi.SimplicialComplex()\n",
    "SC.add_simplices_from([[3, 4, 5], [3, 6], [6, 7, 8, 9], [1, 4, 10, 11, 12], [1, 4]])\n",
    "\n",
    "pos = xgi.pairwise_spring_layout(SC)\n",
    "xgi.draw(SC, pos)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even color/draw the nodes and edges based on statistics!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 100\n",
    "is_connected = False\n",
    "while not is_connected:\n",
    "    H = xgi.random_hypergraph(n, [0.03, 0.0002, 0.00001])\n",
    "    is_connected = xgi.is_connected(H)\n",
    "pos = xgi.barycenter_spring_layout(H)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "ax = plt.subplot(111)\n",
    "xgi.draw(\n",
    "    H,\n",
    "    pos,\n",
    "    node_size=H.nodes.degree,\n",
    "    node_lw=H.nodes.average_neighbor_degree,\n",
    "    node_fc=H.nodes.degree,\n",
    "    ax=ax,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The stats package\n",
    "\n",
    "The stats package is one of the features that sets `xgi` apart from other libraries.  It\n",
    "provides a common interface to all statistics that can be computed from a network, its\n",
    "nodes, or edges.\n",
    "\n",
    "Consider the degree of the nodes of a hypergraph `H`.  After computing the values of the\n",
    "degrees, one may wish to store them in a dict, a list, an array, a dataframe, etc.\n",
    "Through the stats package, `xgi` provides a simple interface that seamlessly allows for\n",
    "this type conversion.  This is done via the `NodeStat` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgi\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "H_stats = xgi.Hypergraph([[1, 2, 3], [2, 3, 4, 5], [3, 4, 5]])\n",
    "H_stats.degree()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_stats.nodes.degree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `NodeStat` object allows us to choose the datatype, specify keywords, and get basic statistics from these properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"As a dictionary:\")\n",
    "print(H_stats.nodes.degree.asdict())\n",
    "print(\"\\nAs a list:\")\n",
    "print(H_stats.nodes.degree.aslist())\n",
    "print(\"\\nAs a numpy array:\")\n",
    "print(H_stats.nodes.degree.asnumpy())\n",
    "print(\"\\nAs a pandas dataframe:\")\n",
    "print(H_stats.nodes.degree.aspandas())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_stats.nodes.degree.aspandas().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(H_stats.nodes.degree(order=2).asdict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st = H_stats.nodes.degree\n",
    "np.round([st.max(), st.min(), st.mean(), st.median(), st.var(), st.std()], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, for edges,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st = H_stats.edges.size\n",
    "st"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.round([st.max(), st.min(), st.mean(), st.median(), st.var(), st.std()], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interface for attributes is very similar. If we add nodal attributes, for example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_stats.add_nodes_from(\n",
    "    [\n",
    "        (1, {\"color\": \"red\", \"name\": \"horse\"}),\n",
    "        (2, {\"color\": \"blue\", \"name\": \"pony\"}),\n",
    "        (3, {\"color\": \"yellow\", \"name\": \"zebra\"}),\n",
    "        (4, {\"color\": \"red\", \"name\": \"orangutan\", \"age\": 20}),\n",
    "        (5, {\"color\": \"blue\", \"name\": \"fish\", \"age\": 2}),\n",
    "    ]\n",
    ")\n",
    "print(H_stats.nodes.attrs(\"color\").asdict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(H_stats.nodes.attrs[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can also filter nodes and edges by their attributes as well as any associated statistic. For example,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(H_stats.nodes.filterby(\"degree\", 2))\n",
    "print(H_stats.nodes.filterby_attr(\"color\", \"blue\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_stats.edges.filterby(\"size\", 3).members()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H_stats.nodes.multi([\"degree\", \"clustering\"]).aspandas().groupby(\"degree\").agg(\"mean\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    H_stats.nodes.multi([\"degree\", \"clustering\", \"average_neighbor_degree\"])\n",
    "    .aspandas()\n",
    "    .groupby(\"degree\")\n",
    "    .agg(\"mean\")\n",
    "    .plot(marker=\"o\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For even more functionality, see the Read The Docs: https://xgi.readthedocs.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare the assortativity of an empirical dataset (Let's choose the `email-enron` dataset) to a random null model. We start by loading the Enron dataset and constructing a Chung-Lu hypergraph using the dataset's degree sequence and edge size sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "H = H_enron\n",
    "k = H.nodes.degree.asdict()\n",
    "s = H.edges.size.asdict()\n",
    "\n",
    "H_null = xgi.chung_lu_hypergraph(k, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use the definitions of assortativity in \"Configuration models of random hypergraphs\" by Phil Chodrow to compare the assortativity of the random null model to the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [\"uniform\", \"top-2\", \"top-bottom\"]\n",
    "\n",
    "assort_orig = []\n",
    "assort_null = []\n",
    "for l in labels:\n",
    "    assort_orig.append(xgi.degree_assortativity(H, kind=l))\n",
    "    assort_null.append(xgi.degree_assortativity(H_null, kind=l))\n",
    "\n",
    "x = np.arange(len(labels))\n",
    "width = 0.35\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "rects1 = ax.bar(x - width / 2, assort_orig, width, label=\"Original\")\n",
    "rects2 = ax.bar(x + width / 2, assort_null, width, label=\"Null model\")\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels, fontsize=16)\n",
    "ax.set_ylabel(\"Assortativity\", fontsize=16)\n",
    "ax.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this plot we can see that the original dataset is more assortative than the random null model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dynamics\n",
    "\n",
    "We can also simulate the order parameter of the Kuramoto model for hypergraphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 100\n",
    "H = xgi.random_hypergraph(n, [0.05, 0.001], seed=None)\n",
    "w = 2 * np.random.normal(1, 0.05, n)\n",
    "theta = np.linspace(0, 2 * np.pi, n)\n",
    "timesteps = 1000\n",
    "dt = 0.002\n",
    "R = xgi.compute_kuramoto_order_parameter(\n",
    "    H, k2=2, k3=3, w=w, theta=theta, timesteps=timesteps, dt=dt\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.plot(np.arange(0, timesteps * dt, dt), R)\n",
    "plt.xlabel(\"time\")\n",
    "plt.ylabel(\"Kuramoto order parameter\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# For more details, visit https://xgi.readthedocs.io"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "78db47bee367e3fae825d59af43d695909ce208402094df58551906768c6e91c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}