[CLI] Accept multiple specification files via stack specification directories

[neilmehta24]

We are finding it helpful to split up our venvstacks.toml file into multiple toml files. We split up our layer specifications up by platform, put them in their own toml files, and concatentate them into a single spec file before we call the CLI. This split helps separate unrelated specifications when developing or code-reviewing.

I think there are a couple of other interesting ways to further split up the layer spec. Splitting by layer type (applications.toml frameworks.toml runtimes.toml), or by project (mlx.toml, llama.toml) could also make sense.

[ncoghlan]

If we allow loading a directory, this would also feed into #36 since we'd presumably want to do something like:

• if venvstacks.toml is present, load that
• if venvstacks.d is present, recursively load all *.toml files in that folder and treat them as a single combined set of stack specifications

At the command line (without depending on #36), wherever the specification file is passed in, a specification folder can be supplied instead.

I thought about offering a way to specify a common prefix for all layers in a file, but that feels like it would be sufficiently confusing as to not be worthwhile (just add the prefix explicitly when naming the layers instead, as then it's clear how to reference them, whether from the same file or from a different file).

One question to be resolved: are path references relative to the individual specifications or relative to the parent directory? Relative to the individual specifications would allow app layer specifications to be grouped with their launch modules which seems like it would be a good thing.

[neilmehta24]

One question to be resolved: are path references relative to the individual specifications or relative to the parent directory?

I would say relative to the parent directory. I would like the freedom to place the code wherever; in venvstacks.d or elsewhere.

[ncoghlan]

I guess one big advantage of the "relative to the spec directory" approach is that the relative paths from venvstacks.toml and venvstacks.d/layers.toml would be the same, so migrating from a single file spec to a multi-file spec dir would be straightforward (only the build invocations would be different).