CTK extensions and package organisation

[cdsre]

Today I was searching for a Hashicorp Consul extension for CTK and didn't find one either in the chaostoolkit or chaostoolkit-incubator organisations. I searched the web but it doesn't seem like any exists.

I built out my own using the extension template and have built is as chaostoolkit-consul. This is only in the early stages and introduces the package structure and actions and probes for the consul KV.

As part of this early work I sent it out to the slack community for early feedback before doing further development. One point that came up is should we have all products of a vendor housed under a single extension for the vendor. In this case example it would be chaostoolkit-Hashicorp.

An example of what that package structure might look like all housed under a single extension could be

chaostoolkit-hashicorp/
└── chaoshashicorp
    ├── consul
    │   ├── acls
    │   ├── intentions
    │   ├── kv
    │   ├── nodes
    │   └── services
    ├── nomad
    ├── vagrent
    └── vault
        ├── aws
        ├── kv1
        ├── kv2
        ├── ssh
        └── transit

There are other examples of this such as chaostoolkit-aws which hosts a large collection of sub packages related AWS services all under a single service. However Where i believe that differs from this use case is while the aws sub packages are a collection of different services they are all controlled mostly through the same botocore library and all have the same dependency so makes sense that these are housed as sub packages.

In my view the Hashicorp Products don't fit that paradigm as they are all separate products, they all have their own SDK or community based packages to interact with the products. There is very little commonality between them and they in most cases don't even have the same dependencies.

My personal view would be to create separate extensions for each of the Hashicorp products, however this discussion should look at defining a community standard for this. Or at least a set of lose guide lines of when you might want to group them under a single extension vs multiple extensions. The following are my reasons for wanting to maintain separate repos for each:

• A package should be a collection of related modules. In this case the only commonality of the products is they are from the same vendor
• By housing all the products as sub packages of a vendor extension all of the product dependencies would be pulled in even when only 1 product might be used in the experiment
• From a security point of view pulling in packages and dependencies of things you do not use increase the attack surface and makes you more vulnerable to supply chain attacks or day 0 vulnerabilities.
• The installing of all products dependencies will have an impact on the SBOM and in certain organisations it could cause the extension to be excluded due to how its packaged.
• The release cadence of the extension could be significant since each product might be making 1 or more releases per month.
• The maintainability of the project might be overwhelmed by a large number of issues or PR's since the scope of the extension would be too wide and covering multiple products. This can lead to issues becoming stale or well written PR's becoming out dated because they were not reviewed and merged fast enough
• The github workflows could become significantly complex as each product might have different testing requirements or workflow processes.
• Down stream projects that depend on this might get dependabot/renovate version updates more frequently than they need to if all products are under a single extension. For example someone using only the consul sub package would be getting version updates even if the change was just to vault, or vagrant or nomad etc.
• There is a chance that a change to say the consul sub package could get blocked because tests unrelated to consul start failing.
• The repo might need a large list of maintainers to cover SME on all products. This can be seen as unhealthy and means that some PR's might be approved, or fixes developed by maintainers who are not so familiar with all of the products. having them as separate extensions means each product could have a smaller set of maintainers who are familiar with that product.
• housing them all together under a vendor extension means we then end up gate keeping any future Hashicorp products which defeats the point of using a plugable architecture.

I am open to hearing discussion points that would support why housing them all together under chaostoolkit-hashicorp might be better or any counter points to my above comments. I fully believe to get the best long term strategy for the tool requires the community to come together and share their views. After all you all are the people who use the tool and have to live with these design choices.

[cdsre]

One thought on how to manage the dependencies could be to use extra/optional dependencies in the pyproject.toml. So by default the pip package wouldnt pull any of the product specific dependencies. And instead users would install with commands like

pip install -y chaostoolkit-hashicorp[consul,vault]

or

pip install -y chaostoolkit-hashicorp[all]

those might be ways to mitigate just installing all the dependencies for all the products and puts it on the user to install the extra dependencies. An example of this can be seen with fastapi's pyproject.toml

[Lawouach]

Hi Chris,

Thank you for bringing this discussion up.

While I'm generally leaning towards a single hashicorp extension, I think you are making good points that I don't want to ignore.

So I'm happy to allow splitting per-Hashicorp product in this case as they fit what yuou are describing. Also, we already have a chaostoolkit-terraform.

Just for the records, here were my concerns of splitting:

• Fragmentation which lead to increased maintainance
• Difficulty of discovery for users
• Not always easier to distribute in the documentation reference (too fine grained)
• Can make contributions from HashiCorp itself (well IBM now) actually a tad more challenging for coporate policy reasons

But I don't think my points hold that strongly.

It doesn't mean we should split in every cases. As you said, sometimes it's easier to group (AWS, GCP, Kubernetes...) because it makes more sense conceptually.

[cdsre]

Thanks for taking to read it all. I have never really developed or maintained an open source active project or extention before so I am sure your first point will be valid around maintainance or fragmentation. If it is discovered there is some commonality then that could be abstracted out to a higher ctk-hashicorp project.

For the 2nd point having them split in my view makes it easier to document. For example consul should be under the network section in the reference, but terraform or vagrant should be under the infrastructure section.

In terms of discovery for users I think you are totally right. Might it be better for me to create a chaostoolkit-hashicorp org in github and then store the projects under that so there is some form of organisation.

You have way more experience of all this so I appreciate your view from past experience.

[Lawouach]

No worries. I don't think there is a right answer here. In fact, if there is one, your approach is closer to it than mine for this case.
So feel free to carry on as you have started, it's the right path.

I'm happy for your extension to live wherever you see fit since you are developing it and therefore you are its owner and copyright holder :)

The documentation will be happy to link to your repo wherever it lives.