Parallel version mpi rosetta does not naturally terminate

[lanselibai]

I use the following code to run two cartesian_ddg jobs in parallel. The required files can be generated. But the jobs does not naturally stop until the wall clock time elapses. How to stop the job earlier when the job has completed?

#!/bin/bash
#SBATCH -J Combined_A36_Mutations      # Job name
#SBATCH --ntasks=2                     # Total number of tasks
#SBATCH --ntasks-per-node=2            # Tasks per node
#SBATCH --time=05:00:00                # Maximum time for the job
#SBATCH -o output.%j
#SBATCH -e error.%j

module purge
module load rosetta/2024.09-mpi 
export OMP_NUM_THREADS=1

NNODES=$SLURM_NNODES
NCPUS=$SLURM_NTASKS
PPN=$SLURM_NTASKS_PER_NODE

echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`
echo SLURM job ID is $SLURM_JOBID
echo This jobs runs on the following machine: `echo $SLURM_JOB_NODELIST | uniq`
echo Number of Processing Elements is $NCPUS
echo Number of mpiprocs per node is $PPN

# Start CPU usage logging
mpstat -P ALL 5 > cpu_usage.log &
MPSTAT_PID=$!

# Run two jobs in parallel using MPI
mpirun -np 1 -quiet bash -c "
cd A36C_A266C &&
cartesian_ddg.mpi.linuxgccrelease \
  -s ../0_pdb/Ortho-DimFinal.pdb \
  -ddg:iterations 1 \
  -force_iterations false \
  -ddg::score_cutoff 1.0 \
  -ddg::cartesian \
  -ddg::dump_pdbs true \
  -bbnbrs 1 \
  -fa_max_dis 9.0 \
  -ddg::legacy true \
  -score:weights ref2015_cart \
  -ignore_zero_occupancy false \
  -ddg:mut_file A36C_A266C.mutfile" &
mpirun -np 1 -quiet bash -c "
cd A36D_A266D &&
cartesian_ddg.mpi.linuxgccrelease \
  -s ../0_pdb/Ortho-DimFinal.pdb \
  -ddg:iterations 1 \
  -force_iterations false \
  -ddg::score_cutoff 1.0 \
  -ddg::cartesian \
  -ddg::dump_pdbs true \
  -bbnbrs 1 \
  -fa_max_dis 9.0 \
  -ddg::legacy true \
  -score:weights ref2015_cart \
  -ignore_zero_occupancy false \
  -ddg:mut_file A36D_A266D.mutfile" & 

# Wait for CPU monitoring to finish
wait

# Once mpirun commands are done, kill mpstat
kill $MPSTAT_PID

# End of job info
echo "End time: $(date)"

[roccomoretti]

Are you getting any output files?

Depending on how Rosetta gets launched with MPI, the first process is often reserved for coordinating runs. I don't know the details of cartesian_ddg, but it may be that the single processor launched by -np 1 is setting itself up as the overseer process, and not actually doing any processing. So it sits around waiting for the (nonexistent) "compute" nodes to ask for work.

(If you are getting output, it may be that the one process is managing to do work, but without other processes to coordinate with, the "wait till others are also done" check might be hanging.)

Try launching with -np 2 and see if that helps.

[lanselibai]

All the required files are generated. But the job keeps running without earlier termination until the wall clock time elapses.

-np 2 is tried as well, still the same problem.