Limitations of kuttl: Discussing Needs for a More Advanced Testing Tool

[shubham-cmyk]

Hello everyone,

We are currently in the process of developing a tool named chainsaw that aims to address some of the limitations of kuttl. While kuttl has served us well, there are certain challenges that have emerged.

I'd love to kickstart a discussion here around these limitations, and would appreciate it if others could share their experiences, thoughts, and possibly additional challenges they've faced with kuttl. Here are some of the limitations I've identified:

Comprehensive Support for Array/List Assertions

Kubernetes has various resources where arrays play a vital role. For some resources, lists are unordered (like environment variables), whereas for others, the order is crucial (like initContainers). It's crucial to have a method to specify how the assertion should be performed.

Complex Assertion for Partial Object Matching

Kuttl struggles with matching two partial objects exactly.
The tool lacks operators like >, <, etc., restricting complex assertions.
we need to go beyond simple matching and search for something into conditional and comparative assertions.

Verification of CLI Outputs or possibility to check warning when applying resources

No straightforward, declarative method to verify the output of a command, script, or CLI tool.

---

If you've experienced these limitations or any others, or if you have suggestions on how a tool like chainsaw could be designed to overcome them, please share your thoughts.

Thank you for your time and contributions!

[jmckulk]

General improvements around setting timeouts for test suites and test steps would be great. I'd be interested in hearing other peoples thoughts around having the ability to scale timeouts. For example, having the ability to say tests take a 10% longer in a certain environment, so give the test suite 10% longer to complete.

[shubham-cmyk]

We already have defined timeout for
apply, assert, delete etc, Individually with the default also.

by the environment do you mean the underlying infra keeps changing but how big the impact would be

[jmckulk]

Awesome, I'll have to try those when testing out chainsaw.

We've found that tests can take different times depending on the environment. For example, running tests locally is faster than running in a cloud environment. Another case could be running the same test suite against different software configurations. One configuration may take predictably longer to run, so we want to adjust the timeout accordingly.

[eddycharly]

Hi @jmckulk did we meet at KubeCon ? ;)

[jmckulk]

Yep we did :)

[eddycharly]

Awesome, I'm still having fun in the US (in SF right now), I'll really be back at work on Thursday and will dig that up.