Batch (#818)

* batch tools subpackage

* comm update

* new search methods

* fixed import issues

* updates comm, search

* search.py

* wrapper tools search

* update to batchtools

* codebase moved to netpyne batchtools

https://github.com/jchen6727/netpyne/tree/batch

* update to comm.py, runners.py

* update to search (kwargs)

* config update (should delete two keys @ end?)

* replace pubtk with batchtk for compatibility with pip

* update the submit (zsh->sh)

* update batchtools, submits

* fixed socket functionality for submits.py

* batchtools documentation

* minor updates- search

netpyne/batchtools/__init__.py

@@ -0,0 +1,14 @@
+from netpyne.batchtools.runners import NetpyneRunner
+from batchtk.runtk import dispatchers
+from batchtk import runtk
+
+specs = NetpyneRunner()
+
+from netpyne.batchtools.comm import Comm
+
+comm = Comm()
+
+
+#from ray import tune as space.comm
+#list and lb ub
+

netpyne/batchtools/comm.py

@@ -0,0 +1,48 @@
+from netpyne.batchtools import specs
+from batchtk.runtk.runners import get_class
+from batchtk import runtk
+from neuron import h
+import warnings
+HOST = 0 # for the purposes of send and receive with mpi.
+
+class Comm(object):
+    def __init__(self, runner = specs):
+        self.runner = runner
+        h.nrnmpi_init()
+        self.pc = h.ParallelContext()
+        self.rank = self.pc.id()
+        self.connected = False
+
+    def initialize(self):
+        if self.is_host():
+            try:
+                self.runner.connect()
+                self.connected = True
+            except Exception as e:
+                print("Failed to connect to the Dispatch Server, failover to Local mode. See: {}".format(e))
+                self.runner._set_inheritance('file') #TODO or could change the inheritance of the runner ...
+                self.runner.env[runtk.MSGOUT] = "{}/{}.out".format(self.runner.cfg.saveFolder, self.runner.cfg.simLabel)
+
+    def set_runner(self, runner_type):
+        self.runner = get_class(runner_type)()
+    def is_host(self):
+        return self.rank == HOST
+    def send(self, data):
+        if self.is_host():
+            if self.connected:
+                self.runner.send(data)
+            else:
+                self.runner.write(data)
+
+    def recv(self): #TODO to be tested, broadcast to all workers?
+        if self.is_host() and self.connected:
+            data = self.runner.recv()
+        else:
+            data = None
+        #data = self.is_host() and self.runner.recv()
+        #probably don't put a blocking statement in a boolean evaluation...
+        self.pc.barrier()
+        return self.pc.py_broadcast(data, HOST)
+
+    def close(self):
+        self.runner.close()

netpyne/batchtools/docs/batchtools.ipynb

@@ -0,0 +1,314 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Jupyter Tutorial: The NetPyNE batchtools subpackage\n",
+    "How to use the `specs` and `comm` to communicate with the `batchtools` `dispatcher`\n"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "89ec6ca2392a9a0d"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "For each individual `sim`, communication with the `batchtools` `dispatcher` occurs through the `specs` and `comm` objects"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "be50f40d8e61a944"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "from netpyne.batchtools import specs, comm"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "6f321aedb7faf945",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "the `specs` object is an instantiation of a custom class extending the `batchtk` `Runner` ..."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "5f2f08f0b5e582c3"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "help(type(specs))"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "29fa261236494bc3",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "From this `specs` object, we can similarly call `specs.NetParams` and `specs.SimConfig` to create the NetPyNE objects..."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "64ead24451bbad4a"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "help(specs.NetParams)\n",
+    "help(specs.SimConfig)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "43d263d080800019",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "The `batchtools` job submission tool uses `environmental variables` to pass values to our `config` object created by `specs.SimConfig`, these `environmental variables` are captured during the `specs` `object creation` which occurs during the batchtools `import` (from the `batchtools` `__init__.py`:\n",
+    "```\n",
+    "from netpyne.batchtools.runners import NetpyneRunner\n",
+    "specs = NetpyneRunner()\n",
+    "```"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "710cc6084bd7af02"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Let's `export` some `environmental variables` to pass values to our `config` object. When this is handled by the `batchtools` `subpackage`, this occurs automatically..."
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "52704684f5e80f3c"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "%env STRRUNTK0   =foo.bar=baz\n",
+    "%env FLOATRUNTK1 =float_val=7.7\n",
+    "from netpyne.batchtools import NetpyneRunner\n",
+    "specs = NetpyneRunner()"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "50de117ff7f43aa6",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "One way of retrieving these values is by calling `specs.get_mappings()`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "fac14e517044b980"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "print(specs.get_mappings())"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "257fad390f4abce",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Now, let's create our `config` object using the `specs.SimConfig()` constructor\n",
+    "This `config` object will hold a `dictionary` such that the initial values `foo['bar']` = `not_baz` and a `float_val` = `3.3`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "92d41061bb828744"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "cfg = specs.SimConfig()\n",
+    "cfg.foo = {'bar': 'not_baz', 'qux': 'quux'}\n",
+    "cfg.float_val = 3.3\n",
+    "print(\"cfg.foo['bar'] = {}\".format(cfg.foo['bar']))\n",
+    "print(\"cfg.float_val = {}\".format(cfg.float_val))"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "ca121d6ab30c3e7b",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Finally, calling the `cfg.update_cfg()` method will overwrite the original values with our environment values, (`baz` and `7.7`)...\n",
+    "\n",
+    "in NetPyNE, this was originally handled with the:\n",
+    "```\n",
+    "try:\n",
+    "    from __main__ import cfg\n",
+    "except:\n",
+    "    from cfg import cfg\n",
+    "```\n",
+    "API idiom in the `netParams.py` file...\n",
+    " \n",
+    "as well as the \n",
+    "```\n",
+    "cfg, netParams = sim.readCmdLineArgs(simConfigDefault='src/cfg.py', netParamsDefault='src/netParams.py')\n",
+    "```\n",
+    "API idiom in the `init.py` file...\n",
+    "\n",
+    "using the `batchtools` subpackage, we can treat the `cfg` as an object and pass it between scripts via `import` statements...\n",
+    "in `netParams.py`...\n",
+    "```\n",
+    "from cfg import cfg\n",
+    "cfg.update()\n",
+    "```\n",
+    "in `init.py`...\n",
+    "```\n",
+    "from netParams import cfg, netParams\n",
+    "sim.createSimulateAnalyze(simConfig=cfg, netParams=netParams)\n",
+    "```"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "6ea43f729d0685d4"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "print(\"prior to  cfg.update()\")\n",
+    "print(\"cfg.foo['bar'] = {}\".format(cfg.foo['bar']))\n",
+    "print(\"cfg.float_val = {}\".format(cfg.float_val))\n",
+    "print()\n",
+    "cfg.update() # call update_cfg to update values in the cfg object with values assigned by batch\n",
+    "print(\"after the cfg.update()\")\n",
+    "print(\"cfg.foo['bar'] = {}\".format(cfg.foo['bar']))\n",
+    "print(\"cfg.float_val = {}\".format(cfg.float_val))"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "a9426b6e6594961",
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Finally, the `comm object` is used to report to the monitoring `dispatcher object`\n",
+    "the means of communication is dependent on which `dispatcher object` is instantiated, and communicated through environmental variables\n",
+    "in this case, since there is no `dispatcher object` the `comm` methods will simply perform `pass operations`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "65bbb0ef2c76295a"
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "comm.initialize() # initializes comm object, establishing channel to communicate with the host dispatcher object"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "e9141d91d6e02aa3",
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "print(comm.is_host()) # returns a boolean IF the calling process is the 0th ranked  parallelcontext, similar to sim.pc.rank == 0"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "5ed6a524bd8a3e0b",
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "comm.send('message') # sends 'message' to the `dispatcher object`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "1966edbf32649352",
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "comm.close() #finalizes communication, closes any resources used to communicate with the `dispatcher object`"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "34f021af4127363c"
+  },
+  {
+   "cell_type": "markdown",
+   "source": [],
+   "metadata": {
+    "collapsed": false
+   },
+   "id": "648746fff96b8a72"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}