[Feature] Request to support multiple Helm Charts in CNTI Test Catalog

[collivier]

Please contact [email protected] for any detail

Is your feature request related to a problem? Please describe.

In Sylva WG-02 (Validation Center), we want to execute CNTI Test catalog against every CNF that is going to be validated but all the CNFs that are already validated in Sylva have more than 2 Helm Charts (in some cases, up to 50 Helm Charts) so this is a problem because currently CNTI Test catalog only supports to test one Helm Chart at the same time.

Describe the solution you'd like

In order to save time and execute CNTI Test catalog in a more efficient way, we would like it to support the setup of multiple Helm Charts stored in a same filesystem folder to launch the tests battery directly over them at the same time and don’t have to execute it one by one manually.

Describe alternatives you've considered

we have been applying manually “cnf_setup” for each Helm Chart to be tested within a CNF and then launched “cnf testsuite cert” over the result. The problem is that for doing it we have to be changing manually “cnftestsuite.yaml” for each Helm Chart which drives to lose many time and we think that this could be supported in a more efficient manner.

[martin-mat]

Considered ways how it is possible to be reached:

(1) Add possibility to define multiple helm charts and handle this as single CNF in cnf-testsuite
+: looks like clean solution from conceptual point of view (closest to a field deployment handling)
-: cnf-testsuite.yml would become quite complex to support all current features
-: quite high implementation efforts, might be better first to refactor cnf_config
Implementation impacts:
- adapt cnf-config.yml file format to allow multiple helm charts (possibly all subdirs in specified dir)
- adapt CNFManager (methods cnf_resources, workload_resource_test, workload_resources, workload_resource_keys, cnf_workload_resources, cnf_to_new_cluster) - resources need to be collected across all helm deployments
- adapt specific test cases that use a helm chart directly ("helm_deploy", "helm_chart_published", "helm_chart_valid", "in_address", "hardcoded_ip_addresses_in_k8s_runtime_configuration", "immutable_configmap", "reasonable_startup_time", "prometheus_traffic", "open_metrics", "tracing", "oran_e2_connection", "database_persistence")

(2) generate parent helm chart referencing child helm chart as dependencies
- does not look a good fitting feature to cnf-testsuite. That would rather be a a candidate to be done fully externally without any cnf-testsuite impact.
- it would require to install all helm charts into single namespace
+ if implemented externally -> no impact to cnf-testsuite
Implementation impacts if implemented internally in cnf-testsuite:
- adapt cnf-config.yml to accomodate multiple helm charts (and probably all subdirs of a directory)
- add code generating a parent helm chart

(3) Use 3rd party tools for deployment of CNF with multiple helm charts
+ this is basically option (1) with not extending cnf-testsuite.yml but rather adding additional installation option (helmfile, reckoner etc.)
other than that same aspects as (1)

(4) Modularize cnf-testsuite in a way that it's possible to use libraries/modules/plug-ins for cnf deployments
"multiple-helm-charts" would be then one such libraries/modules/plug-ins, not necessarily part of cnf-testsuite
+ easily extendable solution, a real implementation would then be close to option (1)
- complex code redesign needed
- adaptation of the same testcases as in (1) needed
Implementation impacts:
- similar as (1), plus:
- figure out how to modularize (libraries/modules/plug-ins)
- redesign/rewrite cnf-manager for modularization
- redesign cnf-testsuite.yml to allow multiple deployment methods

(5) Use existing concept with calling cnf_setup multiple times per each helm chart
Note: this functionality is not fully working currently (.yml result generation does not respect all CNFs) and needs to be fixed
- Each helm chart would be handled as a separate CNF from cnf-testsuite point of view with all its impacts, including longer test runtime, possibly unnecessarily multiple test runs
+ lower-ish implementation efforts
Implementation impacts:
- .yaml result generation needs to be implemented for such case properly
- maybe some more batch mode can be supported using single cnf-testsuite.yml

[kosstennbl]

If we want to add this feature ASAP - probably (2) would be the way to go. Add new SAM task for generating helm chart and add more functionality to cnf-testsuite.yml to support multiple charts (or leave it to user and let them handle running tests with generated multi-charts by themselves).

Option (5) doesn't look as plausible solution to me, multiple CNF installations weren't ever fully supported, don't have a real purpose for Testsuite and, to me, feel like an old garbage that we should get rid of (#2095). Building new features on top of that would likely bring hidden issues that we'll be dealing with in future, and, in the end, we'll have to redesign it anyway.

Other options are revolving around the idea of installation multiple helm charts and threating them as one CNF. To me that sounds as the most reasonable solution, but quite complicated to implement. Significant changes would be required to be finished before we would be able to start designing this feature with this option of implementation (in example #2036 with all of its sub-issues).

To me - way to go is option (1), then possibly look for splitting CNF installation from testing (i.e. - in form of other application, repo), then consider using modular architecture as mentioned in option (4).

[martin-mat]

@collivier do you think (2) would work as a quick solution to enable sylva with cnf-testsuite testing?

[GuillermoCarreto]

Hello @collivier @martin-mat ,

We have been reviewing all the mentioned options in this post. In Sylva WG-02 we consider that the best option to go is Option 1. Regarding Option 2, we consider that it would require big efforts to build parent Helm Charts and it could lead to non desired failures. The objective is to support this functionality in the most efficient, cleanest and easiest way for the user of the CNF Testsuite when multiple Helm Charts must be analyzed.

Considering it, as mentioned by @kosstennbl, we vote for the development of Option 1.

Thank you.

[martin-mat]

Implementation ongoing as a part of #2120