Merge branch 'batch' into batch

netpyne/batchtools/__init__.py

@@ -1,5 +1,6 @@
 from netpyne.batchtools.runners import NetpyneRunner
 from batchtk.runtk import dispatchers
+
 from netpyne.batchtools import submits
 from batchtk import runtk
 
@@ -11,7 +12,6 @@
 runtk = runtk
 
 
-
 comm = Comm()
 
 

netpyne/batchtools/docs/batchtools.ipynb

@@ -0,0 +1,314 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Jupyter Tutorial: The NetPyNE batchtools subpackage\n",
+    "How to use the `specs` and `comm` to communicate with the `batchtools` `dispatcher`\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "89ec6ca2392a9a0d"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "For each individual `sim`, communication with the `batchtools` `dispatcher` occurs through the `specs` and `comm` objects"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "be50f40d8e61a944"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "from netpyne.batchtools import specs, comm"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "6f321aedb7faf945",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "the `specs` object is an instantiation of a custom class extending the `batchtk` `Runner` ..."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "5f2f08f0b5e582c3"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "help(type(specs))"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "29fa261236494bc3",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "From this `specs` object, we can similarly call `specs.NetParams` and `specs.SimConfig` to create the NetPyNE objects..."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "64ead24451bbad4a"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "help(specs.NetParams)\n",
+    "help(specs.SimConfig)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "43d263d080800019",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The `batchtools` job submission tool uses `environmental variables` to pass values to our `config` object created by `specs.SimConfig`, these `environmental variables` are captured during the `specs` `object creation` which occurs during the batchtools `import` (from the `batchtools` `__init__.py`:\n",
+    "```\n",
+    "from netpyne.batchtools.runners import NetpyneRunner\n",
+    "specs = NetpyneRunner()\n",
+    "```"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "710cc6084bd7af02"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Let's `export` some `environmental variables` to pass values to our `config` object. When this is handled by the `batchtools` `subpackage`, this occurs automatically..."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "52704684f5e80f3c"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "%env STRRUNTK0   =foo.bar=baz\n",
+    "%env FLOATRUNTK1 =float_val=7.7\n",
+    "from netpyne.batchtools import NetpyneRunner\n",
+    "specs = NetpyneRunner()"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "50de117ff7f43aa6",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "One way of retrieving these values is by calling `specs.get_mappings()`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "fac14e517044b980"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "print(specs.get_mappings())"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "257fad390f4abce",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Now, let's create our `config` object using the `specs.SimConfig()` constructor\n",
+    "This `config` object will hold a `dictionary` such that the initial values `foo['bar']` = `not_baz` and a `float_val` = `3.3`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "92d41061bb828744"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "cfg = specs.SimConfig()\n",
+    "cfg.foo = {'bar': 'not_baz', 'qux': 'quux'}\n",
+    "cfg.float_val = 3.3\n",
+    "print(\"cfg.foo['bar'] = {}\".format(cfg.foo['bar']))\n",
+    "print(\"cfg.float_val = {}\".format(cfg.float_val))"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "ca121d6ab30c3e7b",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Finally, calling the `cfg.update_cfg()` method will overwrite the original values with our environment values, (`baz` and `7.7`)...\n",
+    "\n",
+    "in NetPyNE, this was originally handled with the:\n",
+    "```\n",
+    "try:\n",
+    "    from __main__ import cfg\n",
+    "except:\n",
+    "    from cfg import cfg\n",
+    "```\n",
+    "API idiom in the `netParams.py` file...\n",
+    " \n",
+    "as well as the \n",
+    "```\n",
+    "cfg, netParams = sim.readCmdLineArgs(simConfigDefault='src/cfg.py', netParamsDefault='src/netParams.py')\n",
+    "```\n",
+    "API idiom in the `init.py` file...\n",
+    "\n",
+    "using the `batchtools` subpackage, we can treat the `cfg` as an object and pass it between scripts via `import` statements...\n",
+    "in `netParams.py`...\n",
+    "```\n",
+    "from cfg import cfg\n",
+    "cfg.update()\n",
+    "```\n",
+    "in `init.py`...\n",
+    "```\n",
+    "from netParams import cfg, netParams\n",
+    "sim.createSimulateAnalyze(simConfig=cfg, netParams=netParams)\n",
+    "```"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "6ea43f729d0685d4"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "print(\"prior to  cfg.update()\")\n",
+    "print(\"cfg.foo['bar'] = {}\".format(cfg.foo['bar']))\n",
+    "print(\"cfg.float_val = {}\".format(cfg.float_val))\n",
+    "print()\n",
+    "cfg.update() # call update_cfg to update values in the cfg object with values assigned by batch\n",
+    "print(\"after the cfg.update()\")\n",
+    "print(\"cfg.foo['bar'] = {}\".format(cfg.foo['bar']))\n",
+    "print(\"cfg.float_val = {}\".format(cfg.float_val))"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "a9426b6e6594961",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Finally, the `comm object` is used to report to the monitoring `dispatcher object`\n",
+    "the means of communication is dependent on which `dispatcher object` is instantiated, and communicated through environmental variables\n",
+    "in this case, since there is no `dispatcher object` the `comm` methods will simply perform `pass operations`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "65bbb0ef2c76295a"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "comm.initialize() # initializes comm object, establishing channel to communicate with the host dispatcher object"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "e9141d91d6e02aa3",
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "print(comm.is_host()) # returns a boolean IF the calling process is the 0th ranked  parallelcontext, similar to sim.pc.rank == 0"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "5ed6a524bd8a3e0b",
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "comm.send('message') # sends 'message' to the `dispatcher object`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "1966edbf32649352",
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "comm.close() #finalizes communication, closes any resources used to communicate with the `dispatcher object`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "34f021af4127363c"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "648746fff96b8a72"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

netpyne/batchtools/docs/batchtools.rst

@@ -21,6 +21,7 @@ The NetPyNE batchtools subpackage provides a method of automating job submission
 
 
 
+
 1. Setting up batchtools
 -----
 Beyond the necessary dependency installations for NetPyNE and NEURON, several additional `pip` installations are required.
@@ -53,6 +54,7 @@ Examples of NetPyNE batchtools usage can be found in the ``examples`` directory
 Examples of the underlying batchtk package can be in the ``examples`` directory `here <https://github.com/jchen6727/batchtk/tree/release/examples>`_.
 
 3. Retrieving batch configuration values through the ``specs`` object
+
 -----
 Each simulation is able to retrieve relevant configurations through the ``specs`` object, and communicate with
 the dispatcher through the ``comm`` object.
@@ -86,7 +88,9 @@ This replaces the previous idiom for updating the SimConfig object with mappings
 
 
 
+
 4. Communicating results to the ``dispatcher`` with the ``comm`` object
+
 -----
 
 Prior batched simulations relied on ``.pkl`` files to communicate data. The ``netpyne.batch`` subpackage uses a specific ``comm`` object to send custom data back
@@ -97,11 +101,13 @@ In terms of the simulation, the following functions are available to the user:
 * **comm.initialize()**: establishes a connection with the batch ``dispatcher`` for sending data
 
 * **comm.send(<data>)**: sends ``<data>`` to the batch ``dispatcher``
+
     * for ``search`` jobs, it is important to match the data sent with the metric specified in the search function
 
 * **comm.close()**: closes and cleans up the connection with the batch ``dispatcher``
 
 5. Specifying a batch job
+
 -----
 Batch job handling is implemented with methods from ``netpyne.batchtools.search``
 
@@ -275,4 +281,5 @@ Note that the ``metric`` specifies a specific ``string`` (``loss``) to report an
 
 The ``out_json`` output contains a dictionary which includes the ``loss`` metric (calculated as the MSE between observed and expected values)
 
-In a multi-objective optimization, the relevant ``PYR_loss``, ``BC_loss``, and ``OLM_loss`` components are additionally included (see ``mo_optuna_search.py``)
+In a multi-objective optimization, the relevant ``PYR_loss``, ``BC_loss``, and ``OLM_loss`` components are additionally included (see ``mo_optuna_search.py``)
+

netpyne/batchtools/examples/CA3/grid_search.py

@@ -23,11 +23,13 @@
     'realtime': '00:30:00',
     'command': 'mpiexec -n $NSLOTS -hosts $(hostname) nrniv -python -mpi init.py'}
 
+
 run_config = sge_config
 
 search(job_type = 'sge', # or shell
        comm_type = 'socket',
        label = 'grid',
+
        params = params,
        output_path = '../grid_batch',
        checkpoint_path = '../ray',

netpyne/batchtools/examples/CA3/optuna_search.py

@@ -22,6 +22,7 @@
     'realtime': '00:30:00',
     'command': 'mpiexec -n $NSLOTS -hosts $(hostname) nrniv -python -mpi init.py'}
 
+
 run_config = sge_config
 
 search(job_type = 'sge', # or sh