Framework: AT is down

[jhux2]

Bug Report

There are a few PR's that are blocked from merging due to reported failures, but nothing shows up on the dashboard:

#13019
#13014

@trilinos/framework

[cgcgcg]

Could someone from @trilinos/framework acknowledge that there is an issue? Thanks!

[jmlapre]

Our Jenkins agents are currently crashing on the GPU machines. We've reported this to DICE and we hope to have this resolved soon.

[achauphan]

The AutoTester should now be running as the rhel8 upgrade hit us with two issues:

1. rhel8 upgrade on our GPU machines had the Java path previously used in their Jenkin's configurations changed unexpectedly.
2. GenConfig and how it uses the SEMS modules in our PR scripts were NOT prepared for the rhel8 upgrade. Specifically, he config rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables attempts to load the SEMS module sems-openmpi/4.0.5-cuda-11.4.2 which no longer exists since CUDA is now built into the module openmpi/4.0.5 in this new version of SEMS modules, so the configuration had issues finding that.

The second issue currently has a temporary fix in-place so that the AutoTester can actually run as the real fix is still being worked on and requires changes to the PR scripts that are in this repo (which would require AutoTester to pass to be implemented). The actual fix should help resolve #13022 as we would implement a rhel8 version of that config.

While the AutoTester should be working today, I will update this further when the desired fixed in place for that second issue.

[jhux2]

Thanks for the update, @achauphan.

[csiefer2]

@achauphan @sebrowne This doesn't look good #13032

[achauphan]

@achauphan @sebrowne This doesn't look good #13032

Responded in this comment

[jhux2]

@achauphan Some PRs have merged, but there are others, e.g., #13017, where there are no obvious failure reports (except for the new non-blocking PR test), but the AT reports that tests failed.

[jhux2]

fixed