Configuration on CENTOS7.8 linux cluster with ifort and openmpi

[RF-1123]

I'm getting an error compiling MOM6 (make NETCDF=3 REPRO=1 libfms.a -j). The error is Package mpich2 was not found in the pkg-config search path.
Perhaps you should add the directory containing mpich2.pc' to the PKG_CONFIG_PATH environment variable No package 'mpich2' found Package mpich2 was not found in the pkg-config search path. Perhaps you should add the directory containing mpich2.pc'
to the PKG_CONFIG_PATH environment variable
No package 'mpich2' found
Package mpich2-f90 was not found in the pkg-config search path.
Perhaps you should add the directory containing `mpich2-f90.pc'
to the PKG_CONFIG_PATH environment variable
No package 'mpich2-f90' found
gfortran -Duse_libMPI -Duse_netCDF -Duse_netCDF -DHAVE_SCHED_GETAFFINITY -Duse_LARGEFILE -I/usr/include -fcray-pointer -fdefault-double-8 -fdefault-real-8 -Waliasing -ffree-line-length-none -fno-range-check -O2 -fbounds-check -c -I../../src/FMS/include -I../../src/FMS/mpp/include ../../src/FMS/mpp/mpp_data.F90
../../src/FMS/mpp/mpp_data.F90:40:0: fatal error: mpif.h: No such file or directory
#include <mpif.h>
The template I use is linux-gnu.mk. What can be done about this problem? Should I use a different template?

[awallcraft]

The gnu filenames are for gfortran, for ifort look at filenames with intel in their name.

The only .mk that explicitly mentions opermpi is triton-gnu.mk:

narwhal08 312> grep openmpi *mk
triton-gnu.mk:# template for the GNU fortran compiler w/ openmpi

This is for gfortran, but it indicates than openmpi uses the "wrapper" script mpif90 to invoke the Fortran compiler and link in the MPI library:

FC = mpif90
CC = mpicc
LD = mpif90 $(MAIN_PROGRAM)

So your best bet is stellar-amd-intel.mk:

narwhal08 315> grep mpif90 intel
stellar-amd-intel.mk:FC = mpif90
stellar-amd-intel.mk:LD = mpif90 $(MAIN_PROGRAM)

Make sure that mpif90 is an oprnMPI script for ifort. Sometimes there are multiple mpif90 implementations available.

The "-amd" part of this script is:

ISA = -march=core-avx2 # The Intel Instruction Set Archetecture (ISA) compile
# option to use. If blank, than use the default SSE
# settings for the host. Current default is to use SSE2.

If you are running on an Intel processor you can leave this blank:

ISA =

Alan

[RF-1123]

Thank you for your reply. But I still have a question, I am running on centos linux, so is it possible to use linux-intel.mk.

[RF-1123]

I have another question. When I execute make NETCDF=3 REPRO=1 libfms.a -j it gives preference to mpirun in anaconda in the environment variable (because I have python installed on my linux server). So I was wondering if I could specify the path to the openmpi I want to use in the template to get around the use of anaconda's mpirun.

[awallcraft]

Almost all of these machines are running some version of Linux, including stellar.

linux-intel.mk is setup for the MPICH2 MPI library:

# Fortran Compiler flags for the MPICH MPI library
FFPPLAGS += $(shell pkg-config --cflags-only-I mpich2-c)

So I think stellar-amd-intel.mk is a better starting point, providing you can find the openMPI mpif90 script. Try the which or whereis commands to find it.

I don't have access to a machine with mpif90, but on Cray's ftn is the analogous wrapper script for ifort:

narwhal09 476> whereis ftn
ftn: /opt/cray/pe/craype/2.7.19/bin/ftn
narwhal09 477> which ftn
/opt/cray/pe/craype/2.7.19/bin/ftn

[awallcraft]

I don't think mpirun is used by Make -j, which just runs multiple steps in parallel on a single node.

However, you are correct in wanting to get mpirun (and mpif90) from openMPI. Most likely you are getting the anaconda version because it is first in your path. Again, see if which and whereis can find what you want. These days mpiexec is typically preferred:

narwhal09 483> whereis mpiexec
mpiexec: /opt/pbs/bin/mpiexec /opt/cray/pe/pals/1.2.12/bin/mpiexec
narwhal09 483> which mpiexec
/opt/cray/pe/pals/1.2.12/bin/mpiexec

If openMPI has a bin directory, make sure it is first in your path before issuing the above commands.

[RF-1123]

Thank you for your answer. I can find mpif90 and mpirun on the server. but when I use stellar-amd-intel.mk as a template and compile it with make NETCDF=3 REPRO=1 libfms.a -j, an error occurs. The error message is:
mpif90 -Duse_libMPI -Duse_netCDF -DSPMD -Duse_netCDF -DHAVE_SCHED_GETAFFINITY -Duse_LARGEFILE -fpp -Wp,-w -I/usr/include -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -i4 -r8 -nowarn -sox -traceback -O2 -debug minimal -fp-model source -march=core-avx2 -c -I../../src/FMS/include -I../../src/FMS/mpp/include ../../src/FMS/mpp/mpp.F90
Using 8-byte addressing
Using pure routines.
Using allocatable derived type array members.
Using cray pointers.
../../src/FMS/mpp/include/mpp_type_mpi.h(28): error #6404: This name does not have a type, and must have an explicit type. [RANK]
if (dtype%ndims /= rank(field)) cycle
---------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(57): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%sizes(rank(field)))
-------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(58): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%subsizes(rank(field)))
----------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(59): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%starts(rank(field)))
--------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(28): error #6404: This name does not have a type, and must have an explicit type. [RANK]
if (dtype%ndims /= rank(field)) cycle
---------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(57): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%sizes(rank(field)))
-------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(58): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%subsizes(rank(field)))
----------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(59): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%starts(rank(field)))
--------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(28): error #6404: This name does not have a type, and must have an explicit type. [RANK]
if (dtype%ndims /= rank(field)) cycle
---------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(57): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%sizes(rank(field)))
-------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(58): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%subsizes(rank(field)))
----------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(59): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%starts(rank(field)))
--------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(28): error #6404: This name does not have a type, and must have an explicit type. [RANK]
if (dtype%ndims /= rank(field)) cycle
---------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(57): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%sizes(rank(field)))
-------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(58): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%subsizes(rank(field)))
----------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(59): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%starts(rank(field)))
--------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(28): error #6404: This name does not have a type, and must have an explicit type. [RANK]
if (dtype%ndims /= rank(field)) cycle
---------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(57): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%sizes(rank(field)))
-------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(58): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%subsizes(rank(field)))
----------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(59): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%starts(rank(field)))
--------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(28): error #6404: This name does not have a type, and must have an explicit type. [RANK]
if (dtype%ndims /= rank(field)) cycle
---------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(57): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%sizes(rank(field)))
-------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(58): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%subsizes(rank(field)))
----------------------------^
../../src/FMS/mpp/include/mpp_type_mpi.h(59): error #6385: The highest data type rank permitted is INTEGER(KIND=8). [RANK]
allocate(dtype%starts(rank(field)))
--------------------------^
compilation aborted for ../../src/FMS/mpp/mpp.F90 (code 1)
How do I solve this problem?

[marshallward]

@RF-1123 Have you made any progress on this issue?

The errors suggest that rank() is not receiving a real array, and the intrinsic rank() is not being used. Perhaps field has not been preprocessed correctly, or something has gone wrong with the MPI header.

But this is just another way of saying that the mkmf template file is not working and needs some modification. I agree with @awallcraft that stellar-amd-intel.mk is a good starting point, but it's hard to know how to proceed without access to your system.

Conda is known to conflict with system installations, and you might have better luck if you simply disable Conda.

If you require Conda to get MPI and netcdf, then you need to be absolutely sure that everything is using Conda, even tools like ld.

---

There is an undocumented method to build some of the executables. If you go inside ocean_only or ice_ocean_SIS and type make -j, then it may work and produce an executable in build.

No promises, but at the least I would be interested to know if it works for you.