METviewer Performance Question

[lindsayrblank]

Hello METplus folks,

I am running METviewer v4.1.0 on an AWS instance. As a note, we are in the process of updating our METviewer instance to v5.0. The problem I am going to outline has occurred in all versions of METviewer.

METviewer takes a very long time to create an individual plot for our use case. It takes anywhere from 20 - 40 minutes to create one line plot. I have attached an example log here. We'd like to know if this behavior is expected or if it is indicative of an issue.

Tagging @daniel-lagrava-spire for visibility.

log_METviewer_example.txt

[j-opatz]

Hi Lindsay,

And thank you for your question. A quick clarification: when you say "...has occurred in all versions of METviewer" do you mean the runtime issue is also present in v5.0?

I'm going to tag @TatianaBurek for her expertise on this one.

Tatiana, is there a noticeable runtime improvement between v4.1.0 and v5.0.1? I see in the beta1 release notes for v5.1.0 that there's support for Python 3.10.4: could this provide an improvement in runtime?

[TatianaBurek]

Lindsay
The aggregation statistics process depends on

• the amount of data : more data - more time to process
• number of bootstrap iterations: the higher this number more sure you can be about the stability your bootstrap but longer the script will run. Parameter "num_iterations", default 1
• number of CPUs: agg stat uses multiprocessing capabilities. More processes - faster the script. Parameter "num_threads", default -1. It means that Python uses all available CPUs
  By changing num_iterations and num_threads or updating your system you could improve the speed.
  Also please note that Python version of agg_stat is much slower than Rscript because Rscript is more efficient as a language. METviewer still works with Rscript. You could try to user it for now and see if there is a difference.

[lindsayrblank]

Thank you, @TatianaBurek and @j-opatz. I am going to write a more comprehensive report detailing METviewer performance that touches on all of the points you two brought up. I will share that report on this discussion thread.

[lindsayrblank]

@TatianaBurek where do the num_iterations and num_threads parameters get set? I cannot find them in the User's Guide (here is the UG I am looking at https://metviewer.readthedocs.io/en/latest/Users_Guide/index.html)

[lindsayrblank]

@TatianaBurek and @j-opatz Here is the report and associated files as promised. Please let me know if you have any questions!

METviewer Performance Report.pdf
METviewer_performance_report_files-20230323T150421Z-001.zip

[TatianaBurek]

@lindsayrblank Thank you very much for the report. @venitahagerty Could you please take a look at the loading part of the report. I will work on the aggregation questions.

[TatianaBurek]

METplus is installed on AWS c5.4xlarge EC2 instance. It has 16 CPUs and 32G RAM. Here are the specs:
https://instances.vantage.sh/aws/ec2/c5.4xlarge
I run similar job with SL1L2 aggregation logic and bootstrapping replication = 1. The number of record that was retrived from DB is 161898.
Python:
agg_stat.py - time 0:01:15.288
line.py - time 0:00:15.483

Rscript:
agg_stat.R - 00:00:05.41
plot_20230323_204639.R - 0:00:00.989

As you could see, Rscript is faster than Python and it is expected. But the amount of time it takes to calculate statistics in general is much smaller in our case most likely because of the more advence type of the EC2 instance.

[lindsayrblank]

Hi Tatiana, thank you very much for sharing the specs of your machine and the results of your tests. This is really helpful information and puts how METviewer is performing for us into perspective. Thanks again!

[venitahagerty]

Hi Lindsay,

I'll have more time to look at this tomorrow. I had a brief glance, and I'm curious whether you're running met_db_load.py in a docker container. If so, the default memory is low at 2G, and should be bumped to 4G.

Thank you,
Venita

[lindsayrblank]

Hi Venita, we are using a docker container. We are looking into how much memory we allocated for the container and will get back to you.

[lindsayrblank]

@venitahagerty We have allocated the container 32 GB of RAM.

[venitahagerty]

Hi Lindsay, I got a message saying: We have allocated the container 32 GB of RAM. I'm not sure if that's what you originally had, or if you have bumped it up to that. If that's a new value, please let me know how that changed your results when running METdataio.

[lindsayrblank]

Hi Venita, I see how my response would be confusing. We bumped up the machine to that. The machine was able to load the data. Thank you for your help.