scripts: twister: Add CTest harness

[pdgendt]

Introduce a twister harness for CMake's CTest.

[hakehuang]

can we add this to TWG discussion? this is a good tool, but how to use it in our CI need some clarification @pdgendt would you like to propose some idea on TWG meeting?

[pdgendt]

can we add this to TWG discussion? this is a good tool, but how to use it in our CI need some clarification @pdgendt would you like to propose some idea on TWG meeting?

Sure, I can join, when will be the next TWG meeting?

The gist of it:
Similar to pytest use ctest for adding test cases. This can be done directly from the application's CMakeLists.txt file.

The one "hack" I added here was to be able to run the test(s) even if the target isn't runnable. I'd like to be able to run "meta" tests, for example, performing tests on the resulting binary, without actually running/flashing.
I guess the same could be useful with pytest? Maybe some other option could be introduced instead of special casing the CTest harness.

[hakehuang]

Sure, I can join, when will be the next TWG meeting?

The gist of it: Similar to pytest use ctest for adding test cases. This can be done directly from the application's CMakeLists.txt file.

The one "hack" I added here was to be able to run the test(s) even if the target isn't runnable. I'd like to be able to run "meta" tests, for example, performing tests on the resulting binary, without actually running/flashing. I guess the same could be useful with pytest? Maybe some other option could be introduced instead of special casing the CTest harness.

@pdgendt Thanks a lot. out next meeting is on next year i think it is 2025/1/10 @PerMac , correct me if I get it wrong. the CTest gives us a lot of potential, but how to use it efficiently? the way @pdgendt proposed are:

1. add case in CMakefile.txt, but we need update the meta(tests.yaml/sample.yaml) as well. but if we do this, twister already have check in align with ztest assert.
2. running case without flash to board, which can be useful in checking the metadata build. and anything more?
3. for now, the adding feature is to have non-run checking possible. we need think about how to extend more.

[pdgendt]

1. add case in CMakefile.txt, but we need update the meta(tests.yaml/sample.yaml) as well. but if we do this, twister already have check in align with ztest assert.

I don't really understand this sentence, I've added a test that already "works", but will currently build for some platform. The required testcase/sample yaml option is the harness: ctest.

2. running case without flash to board, which can be useful in checking the metadata build. and anything more?

Same as for pytest, I guess. You can run the binary (if it terminates cleanly?), do black-box testing, ...

3. for now, the adding feature is to have non-run checking possible. we need think about how to extend more.

Do you mean creating a DUT in the CTest context?

[hakehuang]

1. add case in CMakefile.txt, but we need update the meta(tests.yaml/sample.yaml) as well. but if we do this, twister already have check in align with ztest assert.

I don't really understand this sentence, I've added a test that already "works", but will currently build for some platform. The required testcase/sample yaml option is the harness: ctest.

@pdgendt, if I understand correctly, make test can call those CMakefile.txt tests, and update the testcase.yaml is not necessary, but it is needed if we want to use twister. But twsiter does not have insight on what testcases are.

2. running case without flash to board, which can be useful in checking the metadata build. and anything more?

Same as for pytest, I guess. You can run the binary (if it terminates cleanly?), do black-box testing, ...

if is only applicable on simulator environment. am I right?

3. for now, the adding feature is to have non-run checking possible. we need think about how to extend more.

Do you mean creating a DUT in the CTest context?

this could make us to have clean view of what target and test scope we can apply to, I would support this idea. But let's discuss this with more audience.

[pdgendt]

@pdgendt, if I understand correctly, make test can call those CMakefile.txt tests, and update the testcase.yaml is not necessary, but it is needed if we want to use twister. But twsiter does not have insight on what testcases are.

Yes, there is a test build target, or you can call ctest (which this PR does). Calling ctest allows us to pass arguments to the ctest executable, and create a JUnit xml output file.

Same as for pytest, I guess. You can run the binary (if it terminates cleanly?), do black-box testing, ...

if is only applicable on simulator environment. am I right?

I don't think so? For ctest you need to call add_test which takes in a command. The command should run on the host.

Do you mean creating a DUT in the CTest context?

this could make us to have clean view of what target and test scope we can apply to, I would support this idea. But let's discuss this with more audience.

👍

[pdgendt]

I've added 2 test cases using the native_sim executable binary:

add_test(NAME exe_help COMMAND ${CMAKE_BINARY_DIR}/zephyr/zephyr.exe --help)
add_test(NAME exe_run COMMAND ${CMAKE_BINARY_DIR}/zephyr/zephyr.exe -stop_at=3 -no-rt)

[pdgendt]

@pdgendt Thanks a lot. out next meeting is on next year i think it is 2025/1/10 @PerMac , correct me if I get it wrong. the CTest gives us a lot of potential, but how to use it efficiently? the way @pdgendt proposed are:

1. add case in CMakefile.txt, but we need update the meta(tests.yaml/sample.yaml) as well. but if we do this, twister already have check in align with ztest assert.
2. running case without flash to board, which can be useful in checking the metadata build. and anything more?
3. for now, the adding feature is to have non-run checking possible. we need think about how to extend more.

Is the next meeting on 2025/1/9? I've added this PR to the Testing WG Project.

[hakehuang]

can we add a ctest_args example as well