Unable to run METviewer container

[jwraby]

I successfully pulled the Docker container (metviewer_latest) from docker://dtcenter/metviewer and converted it to a Singularity container on the HPC. The installation appeared to be successful, but when I run the container, it gives the following error:
true: /opt/tomcat/bin/startup.sh: Permission denied
I assume that this error is occurring because the container cannot run on the HPC. Is it possible to obtain a METviewer container which will run on an HPC? Thanks.

[jprestop]

@jwraby, we're working on finding the best person on our team to help with this task. Someone should follow up soon. Thank you for your patience.

[jwraby]

@jprestop, Thanks for the update. Understand and there is no rush.

[TatianaBurek]

METviewer does not officially support the run of Singularity containers on HPC. We have an issue created sometime ago to implement this feature : dtcenter/METviewer#281
However, some work was done for one of the protects that required a METviewer container running on HPC. For that, we created a custom Dockefile that builds an image that can be converted into the Singularity image for HPC:
https://github.com/dtcenter/METviewer/blob/main_v5.1/docker/Dockerfile_for_Singularity
And this is the issue that describes how to build and run it: dtcenter/METviewer#446
You can build the container or get it from the repository

[jwraby]

Hi Tatiana. Thanks for this information. When I pulled the METviewer container from the Docker repository, was that container the one you describe which was built from the custom Dockerfile? Are you able to bypass the use of Tomcat and produce batch plots without a web server?

[TatianaBurek]

The "Singularity" docker container is located in dtcenter/nwp-container-metviewer-for-singularity repository (NCAR/container-dtc-nwp#94 (comment))
dtcenter/metviewer has a standard docker container.
We could not use the web app/Tomcat features because of the disabled ports but were able to use batch scripts

[jwraby]

I tried to pull the container as follows, but get an error:
singularity pull docker://dtcenter/nwp-container-metviewer-for-singularity
FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://dtcenter/nwp-container-metviewer-for-singularity: Error reading manifest latest in docker.io/dtcenter/nwp-container-metviewer-for-singularity: manifest unknown: manifest unknown
Do you know what the problem is?

[jwraby]

Tatiana - I was looking at my logs and found that I had the same error when attempting to pull the latest METplus docker container. John Halley Gotway recommended that I specify the tag name of the METplus container when I execute the pull command. I guess for the METviewer container that might look like nwp-container-metviewer-for-singularity:5.1.0. Would this work for the METviewer container?

[TatianaBurek]

nwp-container was built for METviewer 4.1:

singularity build --sandbox --fix-perms --force metv4singularity docker://dtcenter/nwp-container-metviewer-for-singularity:4.1.0
or
docker pull dtcenter/nwp-container-metviewer-for-singularity:4.1.0

[jwraby]

The information was very helpful. Successfully pulled and converted the container to Singularity on the HPC. I tried to translate the command you used to run the Docker container from dtcenter/METviewer#446 into the equivalent command for Singularity:
Docker command: docker run --name=mv1 -d -p 8080:8080 -e MYSQL_ROOT_PASSWORD=xxxxxx
Singularity command: singularity run --hostname=mv1 --env MYSQL_ROOT_PASSWORD=xxxxxx nwp-container-metviewer-for-singularity_4.1.0.sif

I am not sure that my translation is completely correct. For Singularity I chose not to run in the background (-d). I also chose not to specify the container port (8080) because I cannot run Tomcat. I assume that "container name" for Docker is the same as "container hostname" in Singularity. For Singularity, I used "--env string" to set the environment variable.

I ran the above Singularity command and received the following error:
Running mysql_install_db ...
mkdir: cannot create directory ‘/var/lib/mysql/mysql’: Read-only file system
Fatal error Can't create database directory '/var/lib/mysql/mysql'

What is your diagnosis for this error? Is there a possible resolution to overcome this error?

[TatianaBurek]

NCAR/container-dtc-nwp#94 contains the instructions on how to run the singularity container:
NCAR/container-dtc-nwp#94 (comment)
it also has a link to the online tutorial :
https://dtcenter.org/nwp-containers-online-tutorial/software-containers/metviewer-database-display

[TatianaBurek]

Looking on metv_load_singularity.ksh I see the commands that are used to create and load a database:

mysql -hlocalhost -uroot -pmvuser -e"drop database <database_name>;"
mysql -hlocalhost -uroot -pmvuser -e"create database <database_name>;"
mysql -hlocalhos -uroot -pmvuser <database_name> < /METviewer/sql/mv_mysql.sql

[jwraby]

Thanks. These commands look familiar. At what point do I execute the commands to create and load a database?
Before or after executing the following?
singularity instance start --writable metv4singularity metv
singularity exec --writable instance://metv bash init_singularity.sh
singularity shell instance://metv
I'm not sure what state the container must be in in order to create and load the database.

[TatianaBurek]

Database creating and loading should happen after
singularity shell instance://metv
I would suggest to check if the database is properly working by logging in/logging out before creating a database:
mysql -hlocalhost -uroot -pmvuser

[jwraby]

Thanks for the suggestion. I ran the following commands to set-up and run the container before executing the mysql command:
pwd
/p/home/jraby/MET/METviewer_NWP_singularity
ll
total 8
-rwx------. 1 jraby 1487I100 719 Feb 27 22:45 init_singularity.sh
drwxr-x---. 21 jraby 1487I100 4096 Feb 27 17:17 metv4singularity
singularity instance start --writable metv4singularity metv
FATAL: instance metv already exists
singularity exec --writable instance://metv bash init_singularity.sh
Running mysql_install_db ...
Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
240305 17:53:54 [Note] /usr/libexec/mysqld (mysqld 5.5.68-MariaDB) starting as process 898 ...
ERROR: 1 Can't create/write to file '/p/work1/tmp/jraby/#sql_382_0.MAI' (Errcode: 2)
240305 17:53:54 [ERROR] Aborting

240305 17:53:54 [Note] /usr/libexec/mysqld: Shutdown complete

Installation of system tables failed! Examine the logs in
/var/lib/mysql for more information.

The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:

shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf

You can also try to start the mysqld daemon with:

shell> /usr/libexec/mysqld --skip-grant-tables --general-log &

and use the command line tool /usr/bin/mysql
to connect to the mysql database and look at the grant tables:

shell> /usr/bin/mysql -u root mysql
mysql> show tables;

Try 'mysqld --help' if you have problems with paths. Using
--general-log gives you a log in /var/lib/mysql that may be helpful.

The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss

Please check all of the above before submitting a bug report
at http://mariadb.org/jira

Finished mysql_install_db
UMASK must be a 3-digit mode with an additional leading 0 to indicate octal.
The first digit will be corrected to 6, the others may be 0, 2, 4, or 6.
UMASK corrected from 002 to 0640 ...
Tomcat started.
(base) [jraby@gaffney03 METviewer_NWP_singularity]$ 240305 17:53:54 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
240305 17:53:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
singularity shell instance://metv
Singularity> mysql -h localhost -uroot -pmvuser
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

[TatianaBurek]

Looks like you have a problem creating tmp files for the database. I assume that you don't have 'write' permissins in '/p/work1/tmp/' directory or this directory is full.
I remember that we had similar issues. Try to use this init_singularity.sh script instead. It assigns the TMPDIR location to your home directory.

[jwraby]

Thanks for looking at this error. I checked the status of '/p/work1/tmp/' and it is a directory containing what appear to be users accounts on the HPC, including mine 'jraby'. I tried to create a file in '/p/work1/tmp/', but I don't have write permission which agrees with your assumption. I went into 'jraby" and I do have write permissions there. The vol of jraby is 40K.
I will try using the init_singularity.sh script you provided to see what happens.

[TatianaBurek]

To run GUI you need to have a port 8080 to be open on the system. Our servers have this and other possible ports closed for the security reasons. So, no, it was impossible in our situation to use GUI.
But wen we used the same Singularity container on AWS where we dictated the system requirements and permissions we were able to use the web GUI

[jwraby]

Is the port 8080 part of the HPC system? I'm not sure I understand how a port works. It seems like another way to communicate with the HPC. For example, I use PuTTY to open a terminal connection to the HPC and that uses port 22 as one of the settings. So, the HPC has a port 22 open to allow terminal sessions. If I am understanding this correctly, having port 8080 open, means that I can interact with the HPC using a web browser. Am I understanding this correctly? Thanks.

[jwraby]

I asked the HPC system admin if port 8080 was open. Their response was "8080 is not open on the firewall" My take is that this means that I will not be able to use the METviewer GUI. My approach will be to learn how to produce plots using the batch method. Can the batch method produce all the same types of plots as the GUI?

[TatianaBurek]

We have the same situation with port 8080 being not available.
And yes, you can create the same plots with batch as with GUI

[jwraby]

Thanks again for your valuable assistance with getting the METview container working on the HPC and for answering all my questions. I've learned a lot from this! I may have more METviewer questions in the future, but for now I think this discussion has achieved all the results I was hoping for!

[georgemccabe]

@jvigh, I wanted to bring your attention to this discussion in case it contains useful information to help with the issues you were experiencing.