Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make test step run tests depending on which modules/files where changed #572

Open
casparvl opened this issue May 14, 2024 · 0 comments
Open
Labels
tests Related to software testing

Comments

@casparvl
Copy link
Collaborator

casparvl commented May 14, 2024

Currently, the test step runs all tests tagged with -t CI -t '1_node|2_nodes'. To save time, we'll limit it to only running the OSU tests for now through this PR. However, long term, we would want the right tests to be run for the right changes.

E.g.

  • If GROMACs was built, we want to run the GROMACS tests (call ReFrame with -n GROMACS)
  • If OpenMPI was built, we want to run e.g. OSU and at least one OpenMPI-based end-user application (e.g. -n OSU -n GROMACS.*foss)

That requires two things:

  1. We need to be able to identify what has changed
  2. We need a mapping between what has changed, and what should be tested

To achieve (1) the easiest is maybe to check the tarball contents. Extracting module names from that should be pretty straightforward. We should also figure out what tests to run for other changes, e.g. a change to the init script.

To achieve (2), we should pick some easy format (yaml?). I'd probably create seperate files for mapping changes to modules and mapping other changes. E.g.

# module_test_mapping.yml
GROMACS:
    GROMACS
OpenMPI:
    OSU
    GROMACS

and

# other_test_mapping.yml
init/bash:
    OSU
scripts/gpu_support:
    OSU.*CUDA

What I don't know is how YAML treats slashes, and other regex characters (the values should be regex patterns, as there are accepted by the -n argument of ReFrame).

@casparvl casparvl added the tests Related to software testing label May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Related to software testing
Projects
None yet
Development

No branches or pull requests

1 participant