Refactor secrets.py

[cdsre]

Currently secrets.py is a python module in the chaostoolkit-lib package. It holds all the code related to loading secrets. Currently this supports only env and vault(hashicorp) secret types. However in todays tech stack there are many other places secrets may be stored. Examples are AWS SSM (secure strings), AWS Secrets Manager, Azure Key Vault, Kubernetes Secret. The code currently maintains two different concerns, ENV and VAULT. This means that there are multiple reasons to have to change the module which goes against the Single responsibility principle. It also means there is more to test and as new secret types are added we would need to update the code to define these

Current loop

    for key, value in secrets_info.items():
        if isinstance(value, dict):
            if extra_vars.get(key, None) is not None:
                secrets[key] = extra_vars.get(key)

            elif value.get("type") == "env":
                secrets[key] = load_secret_from_env(value)

            elif value.get("type") == "vault":
                secrets[key] = load_secrets_from_vault(value, configuration)

            else:
                secrets[key] = load_secrets(
                    value, configuration, extra_vars.get(key, None)
                )

        else:
            secrets[key] = value

This if statement could grown significantly and doesn't lend its self to scaling or being extendable. Refactoring the secret module could see a new secret package created and have two modules, env and vault. This allows us to separate out the functionality and tests for each secret type. The main chaostoolkit-lib would still have a secret module that could define a Protocol for what makes a secret. This will need some thought but based on the existing setup might have two methods initalise_secret_provider and load_secret. The secret module could then have a map that maps secret type to a package name loaded at run time. This would also lend its self to allow in the future others to pass a secret map that would map their own secret types to their own packages outside of the core, allowing users to extend the functionality where the tookit community might not have the time or experience to implement a secret provider for their given platform.

[Lawouach]

Hey,

I think that makes sense to allow extending the supported backends.

I do like your approach of creating a protocol/interface here. We could imagine having a "magic" function (like discover) in extensions that would allow to register a backend for secrets on import.

[cdsre]

by keeping it mostly generic the protocol method intialise_secret_backend could be called on all cases, in the case of the env module this would just be empty since there is nothing to do. but in the case of vault it can then handle all its required initialisation of setting up a client connection etc using the configuration it receives.

However lets keep this discussion going around requirements before we jump to solutioning.

There are a couple of security concerns to consider here, since we would be allowing external packages to provide secret backends, we should look to pass only the configuration relevant to that secret backend. An example of that is if I happen to write a new aws secrets manager backend, When my backend is invoked I should not also receive the configuration for any other secrets backends like I shouldn't get the vault address, or any config that might allow access to another backend.

I am also not sure about the discovery part, I would like to understand more about the idea, but my concern here is that through a supply chain attack some dependency pulls in another dependency which would be discovered as a secrets backend. This backend could be malicious and gather secrets even those not defined in the experiment by receiving the config and then communicating with the secret provider and harvesting secrets etc.

I suspect we will need to start with an MVP approach to this. we are not going to get it all right in one go. but I think this is a solid feature improvement for the tool

[Lawouach]

The supply chain risk exists for sure. That said, it exists by the very nature of how we load extensions as well. I do care for security but I think we shouldn't mix topics. Supply chain, as a general threat, should deserve its own topic for a more holistic approach to it by the framework.

We have a choice, either we let users implement their own secret backend, even if another exist. This could happen on AWS for instance if you have specific requirements on how you need to access secrets. Or we want the CTK implementation to be the sole authority. In that case, I'd rather we keep all the backend in chaostoolkit-lib. In that case, we can always delegate to installing the right dependencies via extras pip install chaostoolkit-lib[secret-backend-aws] would pull the right dependencies.

An example of that is if I happen to write a new aws secrets manager backend, When my backend is invoked I should not also receive the configuration for any other secrets backends like I shouldn't get the vault address, or any config that might allow access to another backend.

That's a fair expectation.