MODE Object CSI vs Object Area CSI in METviewer

[JohnHalleyGotway]

Posting an email question I received about interpreting MODE output to discussions...

I am plotting the output of MODE data in METviewer and have a strange result. The data is a comparison of two forecasts in MODE. The models are very similar in the early lead times and diverge for the later lead times. This trend has been verified with case studies. However, plotting the CSI doesn't seem to produce results that make any sense. I've attached a plot of the OBJCSI. Not only does the CSI increase with lead time (which is opposite of the trend we have seen visually), but the numbers are much too small given how similar the two forecasts should be.

The second image I have attached is OBJACSI. This data is what we would expect both numerically and in the trend across forecast lead time. So, this leads to my question. How are the OBJCSI and OBJACSI different? We are trying to understand why the OBJCSI is so small.

I've also attached XMLs if that is at all helpful. Added .txt suffix to the .xml to satisfy GitHub.
.
OBJACSI.xml.txt
OBJCSI.xml.txt

[JohnHalleyGotway]

The definition of the OBJCSI statistic can be found in the MODE Statistics section of the METviewer User's Guide. Search for OBJCSI and you'll find:

OBJCSI: hits / (hits + misses + false alarms) = [(FSM + OSM)/2] / [(FSM + OSM)/2 + OSU + FSU]

Where:

• FSM is matched simple forecast objects.
• OSM is matched simple observation objects.
• FSU is unmatched simple forecast objects.
• OSU is unmatched simple observation objects.

And see above that table for...

For the area-weighted version of these statistics, replace RATIO_ with AREARAT_ and OBJ with OBJA.

So the answer is that OBJCSI is computed using counts of objects falling into each category while OBJACSI replaces each object count with the area (i.e. number of contiguous grid cells that comprise the object) of the object. Since objects defined by MODE can, and usually do, have wildly different areas, it's not surprising that OBJCSI and OBJACSI would behave very differently.

Generally speaking, the area-weighted version of these statistics will indicate much, much better performance than counts of objects. Traditional contingency table statistics, including CSI, are implicitly area-weighted already. When doing grid-to-grid comparisons in Grid-Stat, each grid point is used to increment one cell of the 2 x 2 contingency table. But the historical issue with contingency table-based categorical statistics is the double-penalty problem, where a small displacement in the event leads to a large decrease in scores. The MODE OBJACSI statistic attempts to address that. If objects are "close-enough" to match, then their areas get included in the "hit" category... even the portions that don't exactly overlap on the grid.

I find the OBJACSI to be pretty intuitive and the count-based OBJCSI statistic to be much less useful. It is still a reasonable number to compute for some datasets and object-definition criteria, but the interpretation is difficult because of differing object areas.

I realize your question is really about the difference in the trends of the data. Why does OBJCSI show an improvement in performance as the forecast hours increase while OBJACSI shows the opposite, a degradation in performance that we'd usually expect? To answer that, we'd need to break down the OBJCSI statistic down into it's components:

• Numerator = "hits"
• Denominator = "hits + misses + false alarms"

And we could check to see how the counts of MODE object hits, misses, and false alarms vary through time for this dataset. Please let me know if you'd like to dive more into those details or whether this higher level explanation is sufficient.

[CPKalb]

Hi John:

Thanks for the information. Besides the trend, I think the major thing that causes confusion and concern is how different the CSI values are. The area weighted version tops out near 1 which is what we would expect whereas the non area weighted peaks at 0.12. Given how similar the forecasts are, this value is still confusing to me. I couldn't get METviewer to put hits, misses, and false alarms on the same plot. However, I've attached the 3 components on separate images (with the same y scale) below. Is the OBJCSI computed directly from these aggregated components?

Thanks,
Tina

[JohnHalleyGotway]

@CPKalb here's a plot from METviewer showing the sum of the APCP 01 object counts: FSM (red solid), OSM (blue solid), FSU (red dashed), OSU (blue dashed)

And here's the corresponding XML and data files (with a .txt suffix to satisfy GitHub):
plot_plot_object_counts.data.txt
plot_plot_object_counts.xml.txt

Using the values in the data file, lets compare the OBJCSI value for forecast hour 1 vs 36.

FYI, I used these commands to strip out the sums for these object type from the data file:

for key in `echo "FSM OSM FSU OSU"`; do
  for lead in `echo "10000 360000"`; do
    echo "$key $lead `grep $key plot_plot_object_counts.data.txt | grep "\t${lead}\t" | awk '{print $9}' | awk '{s+=$1} END {print s}'`"
  done
done

Here's the result:
FSM 10000 154
OSM 10000 154
FSU 10000 3
OSU 10000 6

FSM 360000 91
OSM 360000 102
FSU 360000 35
OSU 360000 41

Plugging that into the OBJCSI equation, we have:

OBJCSI: hits / (hits + misses + false alarms) = [(FSM + OSM)/2] / [(FSM + OSM)/2 + OSU + FSU]

OBJCSI(01) = ((154+154)/2) / ((154+154)/2 + 3 + 6) = 0.9447
OBJCSI(36) = ((91+102)/2) / ((91+102)/2 + 35 + 41) = 0.5594

But those numbers clearly do not match what is being plotted in the OBJCSI plot with a range of 0 to 0.12. So I'd say this definitely warrants further investigation.

[JohnHalleyGotway]

I went poking around the METviewer source code and found the following equations for computing OBJCSI and OBJACSI.

While OBJCSI correctly counts the number of rows and OBJACSI correctly sums the object areas, they do not match the definition listed in the documentation!

calcOBJCSI=function(d){
  return ( nrow(d[ d$fcst_flag == 1 & d$matched_flag == 0, ]) /
  ( nrow(d[ d$simple_flag == 1 & d$matched_flag == 1, ]) + 2 *
  nrow(d[ d$simple_flag == 1 & d$matched_flag == 0, ]) ) ) ;
}

calcOBJACSI=function(d){
  return ( sum( d[ d$simple_flag == 1 & d$matched_flag == 1, ]$area ) /
  (sum( d[ d$simple_flag == 1 & d$matched_flag == 1, ]$area ) + 2 *
  sum( d[ d$simple_flag == 1 & d$matched_flag == 0, ]$area )) ) ;
}

This definitely looks like a bug to me.

@bikegeek and @TatianaBurek can you please take a look at this?

Based on the equation in the User's Guide, I think the code should be as follows:

calcOBJCSI=function(d){
  return (   ( nrow(d[ d$simple_flag == 1 & d$matched_flag == 1, ]) / 2 ) /
           ( ( nrow(d[ d$simple_flag == 1 & d$matched_flag == 1, ]) / 2 ) +
               nrow(d[ d$simple_flag == 1 & d$matched_flag == 0, ]) )
         )
}

calcOBJACSI=function(d){
  return (   ( sum(d[ d$simple_flag == 1 & d$matched_flag == 1, ]$area) / 2 ) /
           ( ( sum(d[ d$simple_flag == 1 & d$matched_flag == 1, ]$area) / 2 ) +
               sum(d[ d$simple_flag == 1 & d$matched_flag == 0, ]$area) )
         )
}

Or compute the same thing by referencing other existing functions:

calcOBJCSI=function(d){
  return ( calcOBJHITS(d) / ( calcOBJHITS(d) + calcOBJMISSES(d) + calcOBJFAS(d) ) ) 
}

calcOBJACSI=function(d){
  return ( calcOBJAHITS(d) / ( calcOBJAHITS(d) + calcOBJAMISSES(d) + calcOBJAFAS(d) ) ) 
}

I'd also recommend doing a very careful review of all the MODE object-based contingency table statistic computations to confirm that the rest match the documented equations.

[CPKalb]

@JohnHalleyGotway, thanks for the information! I was getting the same numbers when I read them in the data tab and computed CSI at a few different times. It seems like there is a bug.

Tina

[bikegeek]

@CPKalb, I created an issue in METcalcpy: dtcenter/METcalcpy#360

please direct any new questions or comments in the 'comments' section of the issue.

[TatianaBurek]

The implementation of these stats in Python matches implementation of the same stats in Rscript

[CPKalb]

Yes, both the R and Python versions seem to suffer from the same bug

[bikegeek]

@christina Kalb ***@***.***>, Tatiana just merged the changes into the R-script version in METviewer. Can you verify whether this fixes the issue? --------------- Minna Win Pronouns: she/her National Center for Atmospheric Research DTC/Research Applications Lab Phone: 303-497-8423 *My work hours may not be your work hours. Please do not feel obliged to reply to this email outside of your normal working hours.* ---------------

…

On Thu, Feb 15, 2024 at 2:19 PM Christina Kalb ***@***.***> wrote: Yes, both the R and Python versions suffer from the same bug — Reply to this email directly, view it on GitHub <#2496 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHSLYNPUMZPYHTT6LJ3YTZ3URAVCNFSM6AAAAABDBXDDFCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIOBVGY2DE> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[CPKalb]

I can verify, but is it in the dev version of METviewer? I want to make sure I test with the right one

[bikegeek]

@christina Kalb ***@***.***> , Tatiana tested on the 'dakota' instance, I think that is the dev version. --------------- Minna Win Pronouns: she/her National Center for Atmospheric Research DTC/Research Applications Lab Phone: 303-497-8423 *My work hours may not be your work hours. Please do not feel obliged to reply to this email outside of your normal working hours.* ---------------

…

On Thu, Feb 15, 2024 at 3:28 PM Christina Kalb ***@***.***> wrote: I can verify, but is it in the dev version of METviewer? I want to make sure I test with the right one — Reply to this email directly, view it on GitHub <#2496 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHTL243FEXMFAP5QD3LYT2DZ7AVCNFSM6AAAAABDBXDDFCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIOBWGIYDQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[CPKalb]

I'm having some trouble getting the dev version to load (I am connected to vpn). I will try again in a little bit...

[CPKalb]

It doesn't appear to be working in the dev version of METviewer

[bikegeek]

I'll re-open the issue in METviewer. Thanks for testing. --------------- Minna Win Pronouns: she/her National Center for Atmospheric Research DTC/Research Applications Lab Phone: 303-497-8423 *My work hours may not be your work hours. Please do not feel obliged to reply to this email outside of your normal working hours.* ---------------

…

On Thu, Feb 15, 2024 at 3:49 PM Christina Kalb ***@***.***> wrote: It doesn't appear to be working in the dev version of METviewer — Reply to this email directly, view it on GitHub <#2496 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHX6SUOXE6JYVJJO72LYT2GHFAVCNFSM6AAAAABDBXDDFCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIOBWGM3DG> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>