twister: coverage: Data collection and reporting per-test instance #66345

golowanow · 2023-12-09T21:41:41Z

Twister code coverage data collection and reporting improvements:

The new --coverage-split mode extend Twister code coverage operations to collect and report statistics on each test instance execution individually in addition to the default reporting mode of data aggregation for all the test instances in current scope of the Twister run.
The split mode allows to identify precisely what amount of code coverage each test case provides and to analyze its contribution to the overall test plan's coverage. Each build directory will have its own coverage report and data files, so the overall disk space and the total execution time increase.
The split mode reads the data files while the tests are running in parallel, so the report aggregation stage doesn't need to repeat the same at its turn.
Another new --disable-coverage-aggregation option allows to execute only the --coverage-split mode if the summary statistics are not needed.
New Twister option --coverage-sections allows to select what coverage data to include into the twister_coverage.json file for its further analysis together with the test meta-data and test results.
The choices are to select either the coverage summary, or the detailed branch coverage, ztest coverage, or all the coverage data collected. Also the coverage run 'status' and coverage 'tool' values are added with any non-default choice. The 'environment' top object has additional 'gcov_tool' property to keep the command line argument selection.
Depending on --coverage-split and --disable-coverage-aggregation options, the coverage data is attached either to its test suite object or/and at the report's top level as the aggregated summary of the test plan execution scope.
Currently this mode is fully supported for the default gcovr reporting tool only, as it is based on the GCOVR json reports. For lcov only coverage run status is reported.

All the new modes are disabled by default for compatibility.

Group Twister code coverage command line arguments for better usability. Add more details in descriptions.

Resolves #67058

golowanow · 2023-12-11T08:58:54Z

for example, with ./scripts/twister -vv -j 1 -p unit_testing --coverage --coverage-tool gcovr -T tests/unit --coverage-split --coverage-report summary:

twister.json

{
    "environment":{
        "os":"posix",
        "zephyr_version":"zephyr-v3.5.0-2849-g38e4b5f370df",
        "toolchain":"zephyr",
        "commit_date":"2023-12-09T22:30:12+01:00",
        "run_date":"2023-12-11T09:00:36+00:00",
        "gcov_tool":"gcov"
    },
    "testsuites":[
        {
            "name":"tests/unit/crc/utilities.crc",
            "arch":"unit",
            "platform":"unit_testing",
            "path":"tests/unit/crc",
            "run_id":"037ce42604eb9fb89aef5d37a15e7e2a",
            "runnable":true,
            "retries":0,
            "status":"passed",
            "execution_time":"0.01",
            "build_time":"2.55",
            "testcases":[
                {
                    "identifier":"utilities.crc.crc32c",
                    "execution_time":"0.00",
                    "status":"passed"
                },
.......
            "coverage":{
                "status":true,
                "tool":"gcovr",
                "summary":{
                    "branch_covered":121,
                    "branch_percent":48.0,
                    "branch_total":252,
                    "files":[
                        {
                            "branch_covered":0,
                            "branch_percent":null,
                            "branch_total":0,
                            "filename":"include/zephyr/sys/crc.h",
                            "function_covered":1,
                            "function_percent":100.0,
                            "function_total":1,
                            "line_covered":2,
                            "line_percent":100.0,
                            "line_total":2
                        },
.........
                    ],
                    "function_covered":38,
                    "function_percent":63.3,
                    "function_total":60,
                    "gcovr/summary_format_version":"0.5",
                    "line_covered":315,
                    "line_percent":55.6,
                    "line_total":567,
                    "root":"../../../../../.."
                }
            }
        },
.........
    ],
    "coverage":{
        "status":true,
        "tool":"gcovr",
        "summary":{
            "branch_covered":1067,
            "branch_percent":76.1,
            "branch_total":1403,
            "files":[
                {
                    "branch_covered":4,
                    "branch_percent":100.0,
                    "branch_total":4,
                    "filename":"include/zephyr/sys/cbprintf.h",
                    "function_covered":5,
                    "function_percent":100.0,
                    "function_total":5,
                    "line_covered":16,
                    "line_percent":100.0,
                    "line_total":16
                },
.........
            ],
            "function_covered":191,
            "function_percent":86.8,
            "function_total":220,
            "gcovr/summary_format_version":"0.5",
            "line_covered":2093,
            "line_percent":83.8,
            "line_total":2499,
            "root":".."
        }
    }
}
```

golowanow · 2023-12-13T12:53:38Z

@gchwier, @gopiotr, @hakehuang, @nashif, @PerMac up to your attention

golowanow · 2023-12-15T17:03:47Z

rebased to resolve conflicts with #66406

hakehuang · 2023-12-21T16:12:02Z

@golowanow , maybe change the PR title to enable json report with coverage for each file?

golowanow · 2023-12-21T16:37:40Z

@golowanow , maybe change the PR title to enable json report with coverage for each file?

That title will be misleading as gcovr coverage report already has details for each file, and now it is included into twister.json.
The example snippet above is to show the gcovr 'summary' mode results embedded.

scripts/pylib/twister/twisterlib/environment.py

hakehuang

LGTM, with some code suggestion on group coverage options. beside the conflict need to be resolved

golowanow · 2023-12-22T14:09:52Z

rebased to resolve conflicts with #66487 and with some minor improvements:

command line arguments re-arranged;
additional properties added to the json (to keep which gcov-tool and gcovr/lcov was used).

golowanow · 2023-12-28T17:46:25Z

rebased resolving merge conflict with #66898

golowanow · 2023-12-29T21:10:11Z

rebased resolving merge conflict with #66905

golowanow · 2024-04-24T05:57:24Z

@gchwier, @gopiotr, @nashif, @PerMac, any objections or comments on this PR ?
some more details are also in the RFC #67058

katgiadla · 2024-04-24T08:06:41Z

@gchwier, @gopiotr, @nashif, @PerMac, any objections or comments on this PR ? some more details are also in the RFC #67058

I want to check it before PR merge. Please give me about 23 days.

golowanow · 2024-05-07T09:13:53Z

I want to check it before PR merge. Please give me about 23 days.

@katgiadla, did you have time to check ?

katgiadla · 2024-05-07T09:17:22Z

I want to check it before PR merge. Please give me about 23 days.

@katgiadla, did you have time to check ?

No, I hadn't. I'll check it at this week.

golowanow · 2024-06-05T07:26:37Z

@katgiadla, do you have any feedback to share ?

github-actions · 2024-08-05T00:30:46Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

github-actions · 2024-10-05T00:32:33Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

kartben · 2024-11-26T13:46:31Z

This needs rebasing but not sure what's the current status of this PR?

cfriedt · 2025-01-10T15:14:07Z

@golowanow - friendly ping to please rebase

Group Twister code coverage command line arguments for better usability. Add more details in descriptions. Signed-off-by: Dmitrii Golovanov <[email protected]>

The new `--coverage-split` mode extends Twister code coverage operations to report coverage statistics on each test instance execution individually in addition to the default reporting mode which aggregates data to one report with all the test instances in the current scope of the Twister run. The split mode allows to identify precisely what amount of code coverage each test case provides and to analyze its contribution to the overall test plan's coverage. Each build directory will have its own coverage report and data files, so the overall disk space and the total execution time increase. The split mode reads the raw data fies while the tests are running in parallel, so the report aggregation stage doesn't need to do the same at its turn. Another new `--disable-coverage-aggregation` option allows to execute only the `--coverage-split` mode if the summary statistics are not needed. Signed-off-by: Dmitrii Golovanov <[email protected]>

golowanow · 2025-01-17T15:00:01Z

rebased;
save coverage to separate twister_coverage.json instead of adding to twister.json - the same way as for memory footprints.

^^ @hakehuang, @nashif, @PerMac

New Twister option `--coverage-sections` allows to select code coverage reports to compose into an additional JSON file (twister_coverage.json). By default, the new option is disabled. Currently this mode is fully supported for the default gcovr reporting tool only, as it is based on the gcovr JSON reports. For lcov, only the coverage run status is reported. The choices are to select either the coverage summary, or the detailed branch coverage, ztest coverage, or all the coverage data collected. Also the coverage run 'status' and coverage 'tool' values are added with any non-default choice. The 'environment' top object has additional 'gcov_tool' property with a full path to what binary tool has been used. Depending on `--coverage-split` and `--disable-coverage-aggregation` options the coverage data is attached either to its test suite object or/and as the report's top level object with the aggregated summary of the current test plan execution scope. In case of the custom report name, or per-platform report, the report name is composed with the rightmost '_coverage.json' suffix. The code coverage report has similar structure as `twister.json` and compelements it having reduced set of test suite properties: - instead of `testcases` it contains `coverage` object with gcov tool JSON report data embedded there; - other properites are limited to represent only the essential test suite context, thus to allow further data processing consistently and independently from the `twister.json`. Signed-off-by: Dmitrii Golovanov <[email protected]>

nashif

per test coverage analysis and processing data using the queue is fine. We should not be capturing this data however just because we can, there are tools that process this data and display it already, so lets use those tools.

#84106 shows how this can be done with available analysis and visualization tools.

golowanow · 2025-01-18T20:55:21Z

per test coverage analysis and processing data using the queue is fine. We should not be capturing this data however just because we can, there are tools that process this data and display it already, so lets use those tools.

#84106 shows how this can be done with available analysis and visualization tools.

@nashifl, It is totally not clear what kind of changes you request with your -1 at this PR now.

Did you read the preamble or the related RFC ?
This change allows to use the same old-known html reports (gcovr, or lcov) as well as to transfer the coverage data for analysis by more powerful tools, not mention machine learning.

golowanow marked this pull request as ready for review December 9, 2023 22:11

zephyrbot added the area: Twister Twister label Dec 9, 2023

zephyrbot requested review from galak, gchwier, gopiotr, hakehuang, nashif and PerMac December 9, 2023 22:12

zephyrbot assigned nashif Dec 9, 2023

golowanow added the area: Code Coverage label Dec 9, 2023

golowanow mentioned this pull request Dec 13, 2023

scripts: twister: coverage: deal with multiple function definitions #66487

Merged

golowanow force-pushed the twister_coverage_20231208 branch from 38e4b5f to 35a1bbe Compare December 15, 2023 17:02

golowanow added the Enhancement Changes/Updates/Additions to existing features label Dec 15, 2023

hakehuang reviewed Dec 21, 2023

View reviewed changes

scripts/pylib/twister/twisterlib/environment.py Outdated Show resolved Hide resolved

hakehuang previously approved these changes Dec 21, 2023

View reviewed changes

golowanow dismissed hakehuang’s stale review via cffe993 December 22, 2023 13:49

golowanow force-pushed the twister_coverage_20231208 branch from 35a1bbe to cffe993 Compare December 22, 2023 13:49

golowanow changed the title ~~twister: coverage: Data collection per-test instance~~ twister: coverage: Data collection and reporting per-test instance Dec 22, 2023

golowanow requested a review from hakehuang December 22, 2023 14:10

golowanow force-pushed the twister_coverage_20231208 branch from cffe993 to ab3047d Compare December 28, 2023 17:45

golowanow mentioned this pull request Dec 28, 2023

RFC: Deeper code coverage profiling with Twister test automation #67058

Open

golowanow force-pushed the twister_coverage_20231208 branch from ab3047d to 3055b2a Compare December 29, 2023 21:06

golowanow force-pushed the twister_coverage_20231208 branch from 7fe17c5 to 440570f Compare April 21, 2024 11:18

hakehuang previously approved these changes Apr 26, 2024

View reviewed changes

github-actions bot added the Stale label Aug 5, 2024

golowanow removed the Stale label Aug 5, 2024

golowanow mentioned this pull request Aug 27, 2024

ci: elasticsearch: Upload script improvements #76889

Merged

github-actions bot added the Stale label Oct 5, 2024

golowanow removed the Stale label Oct 5, 2024

golowanow added the In progress For PRs: is work in progress and should not be merged yet. For issues: Is being worked on label Nov 26, 2024

golowanow added 2 commits January 17, 2025 11:38

twister: coverage: Group command line arguments

4770d50

Group Twister code coverage command line arguments for better usability. Add more details in descriptions. Signed-off-by: Dmitrii Golovanov <[email protected]>

golowanow dismissed hakehuang’s stale review via 5a40034 January 17, 2025 12:25

golowanow force-pushed the twister_coverage_20231208 branch 2 times, most recently from 5a40034 to 8053dae Compare January 17, 2025 14:52

golowanow requested a review from hakehuang January 17, 2025 15:00

golowanow removed the In progress For PRs: is work in progress and should not be merged yet. For issues: Is being worked on label Jan 17, 2025

golowanow force-pushed the twister_coverage_20231208 branch from 8053dae to 43b9852 Compare January 17, 2025 19:45

golowanow mentioned this pull request Jan 18, 2025

Run lcov on each test #84106

Open

nashif requested changes Jan 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

twister: coverage: Data collection and reporting per-test instance #66345

twister: coverage: Data collection and reporting per-test instance #66345

golowanow commented Dec 9, 2023 •

edited

Loading

golowanow commented Dec 11, 2023 •

edited

Loading

golowanow commented Dec 13, 2023

golowanow commented Dec 15, 2023

hakehuang commented Dec 21, 2023

golowanow commented Dec 21, 2023

hakehuang left a comment

golowanow commented Dec 22, 2023

golowanow commented Dec 28, 2023

golowanow commented Dec 29, 2023

golowanow commented Apr 24, 2024

katgiadla commented Apr 24, 2024 •

edited

Loading

golowanow commented May 7, 2024 •

edited

Loading

katgiadla commented May 7, 2024

golowanow commented Jun 5, 2024

github-actions bot commented Aug 5, 2024

github-actions bot commented Oct 5, 2024

kartben commented Nov 26, 2024

cfriedt commented Jan 10, 2025

golowanow commented Jan 17, 2025

nashif left a comment

golowanow commented Jan 18, 2025 •

edited

Loading

twister: coverage: Data collection and reporting per-test instance #66345

Are you sure you want to change the base?

twister: coverage: Data collection and reporting per-test instance #66345

Conversation

golowanow commented Dec 9, 2023 • edited Loading

golowanow commented Dec 11, 2023 • edited Loading

golowanow commented Dec 13, 2023

golowanow commented Dec 15, 2023

hakehuang commented Dec 21, 2023

golowanow commented Dec 21, 2023

hakehuang left a comment

Choose a reason for hiding this comment

golowanow commented Dec 22, 2023

golowanow commented Dec 28, 2023

golowanow commented Dec 29, 2023

golowanow commented Apr 24, 2024

katgiadla commented Apr 24, 2024 • edited Loading

golowanow commented May 7, 2024 • edited Loading

katgiadla commented May 7, 2024

golowanow commented Jun 5, 2024

github-actions bot commented Aug 5, 2024

github-actions bot commented Oct 5, 2024

kartben commented Nov 26, 2024

cfriedt commented Jan 10, 2025

golowanow commented Jan 17, 2025

nashif left a comment

Choose a reason for hiding this comment

golowanow commented Jan 18, 2025 • edited Loading

golowanow commented Dec 9, 2023 •

edited

Loading

golowanow commented Dec 11, 2023 •

edited

Loading

katgiadla commented Apr 24, 2024 •

edited

Loading

golowanow commented May 7, 2024 •

edited

Loading

golowanow commented Jan 18, 2025 •

edited

Loading