Workflow no longer runs on HPC

[Aariq]

The UA HPC is in the process of an OS upgrade from CentOS7 to RHEL9. targets workflows that work on the old CentOS7-running puma do not run on the RHEL9-running puma9 (more details here: wlandau/crew.cluster#50). The issue on puma9 is with crew and may be fixed in the near future if/when it takes advantage of new features of mirai (shikokuchuo/mirai#170).

In the meantime, I tried going back to running on puma, but I can't install all the R package dependencies. E.g. curl won't compile (because of outdated compilers I think) and there are no binaries available for CentOS7 on PPM. Plus, puma is going to eventually be fully converted to RHEL9 (can't remember the date, but it is very soon), so this is not a long term solution anyways.

So, I am a bit stuck here. Options:

• Wait for a fix in crew
• Talk to the HPC folks and try my best to explain how crew/mirai/nanonext work and see if they have any ideas of a fix on their side (sounds like maybe Will Landau could provide some details as he experienced this bug with an upgrade to RHEL9 as well)
• Move things to Jetstream2 (either my allocation, or have Theresa get one)

I kind of like using a SLURM cluster because there are only some targets that require a lot of RAM and most will run with just a few cores, and using Jetstream2 might require less or no parallelization to manage memory more easily.