Can't start from interactive Slurm session

[kcgthb]

Hi!

When trying to start remora from an interactive Slurm session, it immediately exits with the following error:

$ srun --pty bash
[cn] $ remora --help
REMORA Error:  Incorrect syntax: REMORA can't run in parallel
REMORA Howto
remora ./myapp [args]                     (serial applications)
remora ibrun [options] ./myapp [args]     (parallel MPI applications)

Our Slurm setup has MpiDefault=pmi2, and when starting an interactive session with srun --pty bash, the PMI environment is set, so PMI_RANK is defined in the environment, and remora thinks that it's running in a parallel mode (checks are in check_running_parallel() in aux/extra).

Unsetting the PMI_RANK variable allowsremora to start:

$ srun --pty bash
[cn] $ unset PMI_RANK
[cn] $ remora --help
 SYNOPSIS
  remora ./myapp [args]                     (serial applications)
  remora ibrun [options] ./myapp [args]     (parallel MPI applications)

 DESCRIPTION
 REMORA: REsource MOnitoring for Remote Applications
[...]

[antoniogi]

Thanks for reporting this. Can you tell me the values of PMI_RANK and PMI_SIZE when you start your job with "srun --pty bash"?

[kcgthb]

Sure!

$ srun --pty bash
$ echo $PMI_RANK
0
$ echo $PMI_SIZE
1

[antoniogi]

Ok, add this to aux/extra, line 61:

if [ -n "${PMI_SIZE+1}" ] && [ $PMI_SIZE -gt 1 ] ; then
print_error "Incorrect syntax: REMORA can't run in parallel"
usage
exit
fi

After this, leave the rest of the function as it is (starting with "if [ -n "${PMI_ID+1}" ] || [ -n "${PMI_RANK+1}" ] || [ -n "${MPIRUN_RANK+1}" ]; then").

I'll add this to the code, but there are things that I need to figure out before committing new changes.

[kcgthb]

Thanks! I did what you suggested, but it doesn't seem to change anything.

When $PMI_SIZE=1, ie. in serial jobs, the rest of that function will still lead to a non-null value of $my_rank and display the REMORA can't run in parallel error.

[antoniogi]

Can you send me the output of the env command to my email? [email protected] Let see what we can do :)