Unexpected difference in stat files from EnsembleStat

[gsketefian]

Hi,

I'm running the METplus EnsembleStat tool on 3-hour APCP obs and forecast data, and I'm getting different results in the output stat file depending on how I name the APCP variable in the input obs and forecast files (and the METplus config file), with everything else in the input files being identical (as far as I can tell using nccmp and xxdiff). I mean there are differences in the stat files besides the expected difference in the variable name. The output netcdf file, however, does not change (again, except for expected differences in variable names), which is what I expect.

I could be missing something, but I think the stat files in the two cases should be identical. Any idea what could be causing the difference? More details below.

I'm using met-10.1.1 and metplus-4.1.1 as part of EPIC's spack-stack installation on NOAA's jet.

In the first case, the variables in the input obs and forecast files are named "APCP_03", while in the second case, they are named "APCP" (i.e. without a "_03" suffix). The corresponding "name" attributes of these variables are named in the same way, and the "level" attributes are in all cases set to "A3". Thus, in the first case, we have (say for one of the input forecast files):

$ ncdump -h rrfs.t00z.prslev.f003.rrfs_conus_25km_APCP_a03h.mem001.nc
...
	float APCP_03(lat, lon) ;
		APCP_03:name = "APCP_03" ;
		APCP_03:long_name = "Total Precipitation" ;
		APCP_03:level = "A3" ;
...

while in the second case, we have

$ ncdump -h rrfs.t00z.prslev.f003.rrfs_conus_25km_APCP_a03h.mem001.nc
...
	float APCP(lat, lon) ;
		APCP:name = "APCP" ;
		APCP:long_name = "Total Precipitation" ;
		APCP:level = "A3" ;
...

In the METplus config file, the variable settings in the first case are:

ACCUM_HH = 03
FCST_VAR1_NAME = APCP_03
FCST_VAR1_LEVELS = A{ACCUM_HH}
FCST_VAR1_OPTIONS = ens_ssvar_bin_size = 50.0; ens_phist_bin_size = 0.05;
OBS_VAR1_NAME = APCP_03
OBS_VAR1_LEVELS = A{ACCUM_HH}
OBS_VAR1_OPTIONS = {FCST_VAR1_OPTIONS}

and in the second case are:

ACCUM_HH = 03
FCST_VAR1_NAME = APCP
FCST_VAR1_LEVELS = A{ACCUM_HH}
FCST_VAR1_OPTIONS = ens_ssvar_bin_size = 50.0; ens_phist_bin_size = 0.05;
OBS_VAR1_NAME = APCP
OBS_VAR1_LEVELS = A{ACCUM_HH}
OBS_VAR1_OPTIONS = {FCST_VAR1_OPTIONS}

All other parts of the config file are identical. I've attached the resulting stat files. You can see that in the case with the "_03" in the variable names, there is an extra PHIST line type in the stat file, and the remaining line types have differences (other than the renaming of the variable to include/exclude the substring "_03").

If it would help, I have a set of input files (METplus config file, obs data, and forecast data) and a run script that I can make available that demonstrates this difference in the output stat file.

Thanks for any help you can provide on why the stat files for the two cases might be different.
Gerard
ensemble_stat_FV3_GFS_v15p2_CONUS_25km_APCP03h_CCPA_030000L_20190615_030000V.stat.with_03.txt
ensemble_stat_FV3_GFS_v15p2_CONUS_25km_APCP03h_CCPA_030000L_20190615_030000V.stat.without_03.txt

[CPKalb]

Hi Gerard:

It would be helpful for me to take a look at your input files (obs data, forecast data, and config). You can upload these to our ftp following the instructions. Or, if it's easier, you could put the data somewhere on Hera or Cheyenne that I could access and let me know the paths.

Thanks,
Christina

[gsketefian]

@CPKalb Hi Tina,

I've setup up all my files (and testing scripts) on Hera under:

/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/debug_METplus

The EnsembleStat config file and the NetCDF input files for the case WITH the "_03" suffix in the names are in the subdirectory

case_APCP_03/inputs

The corresponding files for the case WITHOUT the "_03" suffix are in the subdirectory

case_APCP/inputs

There is also the common conf file (common.conf) right under the top-level directory (debug_METplus).

There is a script at the top level named run_ensemble_stat.sh that you can call to run ensemble_stat on the two sets of input files. This script sets up several environment variables that are needed in the conf files. To run the case with "_03" suffixes, use (from the top level directory)

> ./run_ensemble_stat.sh "with_03"

This will place output files in the subdirectory

case_APCP_03/outputs

(it will overwrite any existing files). Similarly, for the without "_03" case, use

> ./run_ensemble_stat.sh "without_03"

and the outputs will be in

case_APCP/outputs

The output files that I'm comparing are:

ensemble_stat_FV3_GFS_v15p2_CONUS_25km_APCP03h_CCPA_030000L_20190615_030000V.stat
ensemble_stat_FV3_GFS_v15p2_CONUS_25km_APCP03h_CCPA_030000L_20190615_030000V_ens.nc

If you diff the stat files for the two cases, you'll see that they're different (I use xxdiff to compare the two). They are different even after say changing all occurrences of the string "APCP_03" to "APCP" in the output for the WITH "_03" case. However, the NetCDF files are the same -- after changing the array name in the NetCDF file and then comparing. I know that because I have a script (compare_netcdf_files.sh, at the top level) that compares the NetCDF files, both for input and output, by first changing the array name and array attributes appropriately in the files for one case and comparing the result to the unmodified files from the other case. You can call it on the inputs like this:

./compare_netcdf_files.sh "inputs"

You'll know that the input NetCDF files are identical (after changing "APCP_03" to "APCP" and vice-versa in the array name and metadata) if you see this screen output:

SUMMARY:
  num_files_compared = 5
  num_same_metadata = 10
  num_same_data = 10

Then, after running run_ensemble_stat.sh for the two cases, you can check the output files like this:

./compare_netcdf_files.sh "outputs"

You'll know that the output nc files are identical (after changing "APCP_03" to "APCP" and vice-versa in the array name and metadata) if you see this screen output:

SUMMARY:
  num_files_compared = 1
  num_same_metadata = 2
  num_same_data = 2

So my question is, why do the stat files differ even though the input and output NetCDF files are the same? Thanks!
Gerard

[CPKalb]

Hi Gerard:

I ran the script (run_ensemble_stat.sh) for both "with_03" and "without_03." When I compare the output in the log files, this warning pops up in the "without_03" log:

WARNING: process_grid_vx() -> Disabling observation error logic since the obs error table contains no entry for OBS_VAR(APCP) and MESSAGE_TYPE(CCPA).
WARNING: Specify a custom obs error table using the MET_OBS_ERROR_TABLE environment variable.
WARNING:

This warning is absent in the "with_03" log, which means observation error logic is applied to the APCP_03 version but not to the APCP version. Why is this happening? Well, the obs error table used in MET is located in {MET_BASE}/share/met/table_files/obs_error_table.txt. That table does need an accumulation to be part of the variable name as seen in lines 13-17 (for MET 10.1). To get around this, I grabbed a copy of that table and edited the variable so that it says APCP rather than APCP_[0-9]*. I then added a custom obs error table to the processing by setting MET_OBS_ERROR_TABLE as an environment variable inside the METplus configuration file for the "without_03" run (case_APCP/inputs/EnsembleStat_APCP03h.conf). This addition has to go at the end of the configuration file, since it's adding a configuration section:

[user_env_vars]
MET_OBS_ERROR_TABLE = /scratch2/BMC/det/kalb/debug_METplus/obs_error_table.txt

I then ran METplus again with this environment variable and no warning was produced in the "without_03" version. When I visually examine the new output .stat files, the results look identical for the "with_03" and "without_03" versions. If I change the variable name so that APCP then says APCP_03 and run a diff, it also produces no differences.

So, the differences here in the .stat files are caused by the observation error logic not being applied to the case where the variable name is APCP. The observation error logic only impacts the output statistics, and not the ensemble means, etc. which is why the netCDF files were the same. These differences happen because MET's observation error table needs the accumulation to part of the variable name. However, you can get around this requirement by creating a custom observation error table using APCP as the variable name and setting the environment variable MET_OBS_ERROR_TABLE to the location of your custom table. I've attached the updated configuration file with this environment variable set here for reference.
EnsembleStat_APCP03h.conf.txt

Does making this change produce identical results on your end?

Christina

[gsketefian]

@CPKalb Hi Tina,

Thanks for getting to the bottom of this!

I will try your solution to make sure it works, although I am somewhat reluctant to use this approach because the changes I'm making are in the SRW App (see PR 973), and I would have to add this custom obs error table to the app and would have to make sure it is updated properly every time the app is upgraded to a new version of MET (and there's a high possibility that will be forgotten!). So I'd much rather just use the default table that comes with MET/METplus.

I wonder if it would be possible in a future version of MET to add lines to the obs error table that don't have the _[0-9]*.

Here's some background for how/why I ran into this issue. In the METplus conf files that the SRW App currently uses, the APCP variable names do include the accumulation. Specifically, for the PcpCombine tasks in the app, both the forecast and obs variables have the _01, _03, etc in their names (e.g. [FCST|OBS]_VAR1_NAME = APCP_03). So in PcpCombine's output NetCDF files, the variable names have the _03 (say), but the variables also have the level attribute that is set to A3 (e.g. [FCST|OBS]_VAR1_LEVELS = A3). This seems redundant and can be confusing. Also, in the app's downstream vx tasks that use the PcpCombine output files as inputs (e.g. EnsembleStat but also ensemble mean and probabilistic GridStat and PointStat tasks), we end up with NetCDF variable names like APCP_03_A3_ENS_MEAN instead of the simpler (and clearer) APCP_A3_ENS_MEAN. For these reasons, in PR 973, I'm trying to drop the accumulations in the variable names, and I ran into this issue of the EnsembleStat output not being identical.

I can add a custom obs error table to the app for now, but it would be great to make a change to the default error table to allow obs variable names without the accumulation so I don't have to keep the custom table for long.

Thanks,
Gerard

[CPKalb]

Hi Gerard:

MET actually does not use the level attribute to read data from a gridded NetCDF file. In fact, level specification for gridded netCDF files is completely different from the A03, Z2, etc. that are used for GRIB1/2 files. Levels for netCDF files should be specified based on the number of dimensions of your data and the dimensions to be used. So, the level should actually look like FCST[OBS]_VAR1_LEVELS = "( * , * )" rather than the [FCST|OBS]_VAR1_LEVELS = A3 that is set inside the configuration file. More information about this can be found in section 4.2.11 (there is an example at the top of the section and more information after scrolling down a bit).

In this case, since your data is a simple 2D array, MET read it without the correct level specification. If your variable had more dimensions or multiple time steps in the same file, then specifying the level as A3 would not have worked. But this should give some information on why the level is part of the variable name. Specifically, if we had multiple accumulations in the same file, or if we were doing something like deriving a mean on temperature at different levels (like Z2, P500, etc.), the level would be needed in the variable name to differentiate between them.

If the goal is to have a simpler ensemble mean name, another option would be to do the following to the "with_03" configuration file:
Change
FCST_VAR1_OPTIONS = ens_ssvar_bin_size = 50.0; ens_phist_bin_size = 0.05;
to
FCST_VAR1_OPTIONS = ens_ssvar_bin_size = 50.0; ens_phist_bin_size = 0.05; set_attr_name = "APCP"; set_attr_level = "03";

This will give your output ensemble mean a simpler name (APCP_03_ENS_MEAN), and will still allow the observation error logic to work without a custom table. However, it will also cause some changes in the output .stat file. Specifically, the variable names will be APCP rather than APCP_03, and the variable levels will be 03 rather than A3. I don't know if that change would be an issue on your end.

In terms of updating the observation error logic table, I'm looping @JohnHalleyGotway into this discussion. John, for reference, Gerard is getting different output statistics from Ensemble-Stat when using a variable name of APCP vs ACPC_03. We tracked this down to the observation error table, which requires the accumulation to be part of the variable name. I pointed him to the custom error logic table option, which does fix the issue. However, there are concerns about maintaining a custom table. So, the question has come up as to whether it would be feasible to update the default observation error table so that the accumulation was not required. Do you have any input on this?

Christina