Add pbp job gen program which includes orchestration script

[spacetimeengineer]

This builds on linux and windows. The pbp-job-agent is now part of the pbp program suite. It accepts a configuration YAML file that contains the job agent settings.

Usage:

    $ pbp-job-agent --config configuration_1.yaml configuration_2.yaml configuration_3.yaml

will spawn three pbp-job-agents as three separate processes. This agent will provision all metadata, nc files and plots.

Important Notes:

The configuration schema itself can be found through examples by entering

    $ pbp-job-agent --help

  >
  usage: pbp-job-agent [-h] --config path [path ...]
  
  Deploys a job agent to process audio files. Each yaml file configuration argument spawns a new process.
  
  optional arguments:
    -h, --help            show this help message and exit
    --config path [path ...]
                          path to the YAML file(s) that contain the configurations for the pbp-job-agent
  
      Examples:
      pbp-job-agent --config /path/to/deployment/global_config_1.yaml /path/to/deployment/global_config_2.yaml
      
      Runs two job-job-agent process concurrently with the configurations provided in the yaml files.
      
      An Example YAML Content (For SoundTrap) ~ global_config_1.yaml:
      
      pbp_job_agent:
        output_prefix: Georges_Bank_2021
        recorder: "SOUNDTRAP"
        audio_base_dir: "/home/user/SOUNDTRAP_DEPLOYMENT/"
        json_base_dir: "/home/user/SOUNDTRAP_DEPLOYMENT/JSON"
        xml_dir: "/home/user/SOUNDTRAP_DEPLOYMENT"
        start: "20220521"
        end: "20220521"
        prefix: "6550"
        nc_output_dir: "/home/user/SOUNDTRAP_DEPLOYMENT/NC"
        global_attrs: "/path/to/deployment/global_config_1.yaml"
        variable_attrs: "/path/to/deployment/variable_config_1.yaml"
        sensitivity_flat_value: "176.6"
        latlon: "-31.711 115.583"
        title: "Georges_Bank_2021"
        cmlim: "36 107"
        ylim: "10 24000"
        meta_output_dir: "/home/user/SOUNDTRAP_DEPLOYMENT/META"
        voltage_multiplier: ""
        sensitivity_uri: ""
        log_dir: "/home/user/SOUNDTRAP_DEPLOYMENT/AGENT"

The --global-attrs file is distinct from the --config file. While they are not the same, the --global-attrs file can serve a similar purpose if it includes the necessary global attributes. The design decision allows users the flexibility to not place all job agent configurations in the global_attrs file, though doing so is entirely acceptable.

If you opt to use the global_attrs file, you must reference its path within the configuration, even if it points to the file itself. This is required because the arguments in the global_attrs file are also used by other pbp program calls.

Additionally, stability on Windows has been improved through better utilization of path libraries.

The tests arent in yet but I am working on them.

[danellecline]

This looks like a valuable contribution. Thanks! Tag me when the tests are done and I will review the accept the PR.

[spacetimeengineer]

Thanks Danelle, Ill try to get them to you asap.

[spacetimeengineer]

Hi Danelle,

I just realized I’m running into an issue with writing these tests. My initial plan was to assert the existence of paths and directories after they’re created by the agent. However, I don’t have any Soundtrap WAV files to include in the repo for testing, as the ones I use locally are too large. I’m unsure how to handle this without compromising the integrity of the tests.

My next thought was to reimplement the tests you already created using the files you had, but I didn’t see an obvious test for pbp-hmb-gen, only for pbp-meta-gen.

Another option I considered was testing that the job agent’s terminal command profile aligns correctly with the pbp command suite proper. (Feels cheap I admit)

Beyond that, the only idea I have left is to test whether the YAML files themselves are being parsed correctly.

Please advise and thankyou.

[danellecline]

Hi @spacetimeengineer
How about you give us a quick demo of this early next week so we can all get up to speed?

[spacetimeengineer]

Ofcourse!

[spacetimeengineer]

Echopype has a great technique for handling large test files that I just discovered. They set up a Docker image containing all the necessary files, and in the documentation, developers are instructed to pull the image and map its contents to a specific location in the local repository. This approach allows test files to be very large while still having a local path for easy access and allowing the pulling of the pbp repo still very quick