Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins RT Pipeline #780

Merged
merged 33 commits into from
Oct 26, 2023
Merged

Conversation

FernandoAndrade-NOAA
Copy link
Collaborator

This PR addresses #779.
Three new files have been added into the /ci directory
There should be no baseline changes as this PR is for enhancement of testing functionality.
Allows for Jenkins to run regression tests on open PRs with the "Jenkins-run-RT" label on Hera and Orion

  1. Jenkinsfile - instructions for the Jenkins Pipeline to run regression tests on supported machines (currently Hera and Orion)
  2. rt-status.py - a utility script to summarize apparent changes in results for test cases after regression tests conclude, outputs to the Jenkins test run console
  3. rt.sh - a refactored copy of the regression test script for the pipeline and developers to utilize with added options and use of the EPIC UPP test suite location for supported machines.
    3a. from a developer's standpoint, starting regression tests may look like the following (ensuring to be synced with latest develop):
cd ci/
./rt.sh -a epic -r `pwd`/rundir -t `pwd`/../

The following options have been added to rt.sh:

  • a - account to submit jobs with
  • w - work directory where job scripts and their output logs will be contained, defaults to the current directory/work-upp
  • h - home directory, where the test suite resides. Consists of the data input, output, job cards, timestamps, etc. and defaults to utilizing the EPIC UPP test suite directory for regression testing
  • r - run directory, where the tests will be run along with their generated files to compare to current baselines in the home directory. defaults to various locations on machines specified in rt.sh, it may be preferable to adjust the location through this rt.sh option as shown above
  • t - test_v, the local path to the UPP test version that will be tested before merging. Should typically be set to the example above for developers to run RTs
  • b - git branch for cloning before testing
  • u - git url for the PR branch that will be cloned
  • c - enable git cloning

Note: use of options b, u, and c will automatically set test_v to the cloned UPP location and will not typically be needed for developer testing as rt.sh will already be contained within the repository, they're legacy options for regression testing.

@FernandoAndrade-NOAA FernandoAndrade-NOAA added the enhancement New feature or request label Sep 8, 2023
update run directory local paths to environment variable
@WenMeng-NOAA WenMeng-NOAA linked an issue Sep 8, 2023 that may be closed by this pull request
cleanup rt log for pushing through jenkins and cleanup paths in rt-status.py for rundir
@FernandoAndrade-NOAA
Copy link
Collaborator Author

please ignore the review request from being taken out of draft

@FernandoAndrade-NOAA
Copy link
Collaborator Author

@WenMeng-NOAA just an FYI, Hera logs have been pushed up with no changes as expected. Orion may need to be put on hold as the disk quota has been exceeded at the moment, so this may need to be pushed back until that's resolved.

@WenMeng-NOAA
Copy link
Collaborator

@FernandoAndrade-NOAA The hera log looks good to me. Thanks for working on this.

@FernandoAndrade-NOAA
Copy link
Collaborator Author

@WenMeng-NOAA I've run under an alternative workspace in /work2 to run the Orion tests, which should be fine in the meantime while work1's disk quota is resolved. Orion log is pushed and this should be ready for review/final testing. Please let me know if there's any desired changes and suggestions e.g. different default path values / additional log output (like the rt-status.py output). There should be no changed results.

One thing to note is that the Jenkins RT Pipeline's configuration is currently hard coded to /work1, so there may still be disk quota hiccups after this is merged until that's fixed.

Thanks!

@FernandoAndrade-NOAA FernandoAndrade-NOAA added No Baseline Change No baseline of the UPP regression tests are made. Ready for Review This PR is ready for code review. labels Oct 23, 2023
ci/rt.sh Outdated Show resolved Hide resolved
ci/rt.sh Outdated Show resolved Hide resolved
Removes the check and paths for retired WCOSS_D machine and updates rundir to include upp-${machine} as a subdirectory for organization.
ci/rt.sh Outdated Show resolved Hide resolved
New WCOSS2 lines will be temporarily removed due to the machine not utilizing slurm scheduler causing failures in the test script. It will be reintroduced at a later time.
@WenMeng-NOAA
Copy link
Collaborator

The UPP RTs were successfully completed on WCOSS2. No baseline updates are needed in this PR.

@WenMeng-NOAA
Copy link
Collaborator

@FernandoAndrade-NOAA My RTs on WCOSS2 was completed. Do you need to rerun tests on Hera and Orion? Otherwise, this PR looks good for me now.

@FernandoAndrade-NOAA
Copy link
Collaborator Author

@FernandoAndrade-NOAA My RTs on WCOSS2 was completed. Do you need to rerun tests on Hera and Orion? Otherwise, this PR looks good for me now.

I'll go ahead and do a final rerun just on Hera to ensure the path update for rundir doesn't break anything, but there should be no baseline changes.

@FernandoAndrade-NOAA
Copy link
Collaborator Author

No issues in the Hera rerun, and no changes. This should be ready for merging. Thanks for your help @WenMeng-NOAA!

@WenMeng-NOAA
Copy link
Collaborator

This PR is ready for merging.

@WenMeng-NOAA WenMeng-NOAA merged commit d1044b1 into NOAA-EMC:develop Oct 26, 2023
4 checks passed
@FernandoAndrade-NOAA FernandoAndrade-NOAA deleted the jenkins-ci branch November 21, 2024 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request No Baseline Change No baseline of the UPP regression tests are made. Ready for Review This PR is ready for code review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Jenkins Regression Test Pipeline
2 participants