What is the proper syntax for taking the complement of a mask of type "data" using gen_vx_mask?

[jvigh]

Dear MET Discussions,

I am trying to take the simple complement of an existing mask (EQUATORIAL.nc" which is a mask for the equatorial regions (defined by lat >-30 and <30 in this case). So the goal is to obtain a mask which is "NOT_EQUATORIAL". I cannot find any guidance in the documentation or tutorial materials that explains how to create compound masks using -type data. The only thing I found was a simpler example for poly masks that doesn't give enough information my case. While I realize that this simple example could probably be done with gen_vx_mask using the poly method, I actually have much more complex masking regions that I'm seeking to take the complement of (which I'm not sure I can post), so this is just an example -- I want to know what the syntax to use gen_vx_mask to take the complement of a data mask.

Here's my attempt:

gen_vx_mask
EQUATORIAL.nc"
EQUATORIAL.nc"
NOT_EQUATORIAL.nc"
-type data
-complement
-input_field 'name="EQUATORIAL"; level="(,)";'
-mask_field 'name="EQUATORIAL"; level="(,)";'
-thresh eq1

Note: I am using EQUATORIAL.nc both as the input_file and mask_file because I'm not sure what other file else to use for these when taking the complement.

If I use -thresh eq1, then I just get back an entire grid of ones. If I use a thresh of 0, I think I just get back the original equatorial grid.

My simple data mask example is attached.

example_mask.zip

Thank you in advance,
Jonathan

[JohnHalleyGotway]

Hi @jvigh. I see that you already have an existing mask and you'd like to run gen_vx_mask to take its complement. The confusion is caused by a feature that was added a while back to make it easier to run gen_vx_mask iteratively to compute complex masks. Whenever you run gen_vx_mask with INPUT_FIELD = the output of a previous run of the tool, it automatically initializes the value at each grid point to the contents of that input mask. That makes it easier to run iteratively since you no longer need to specify the "-input_field" option each time to specify the name of the NetCDF variable containing the input mask data.

But this logic in turn, makes this seemingly simple task you're trying to do confusing.

Let me step back and tell you how you could easily define these EQUITORIAL and NONEQIUTORIAL regions. Since I don't have your original 72x72 NetCDF input file, I'll just manually define that grid on the command line.

For EQUATORIAL, let's just threshold the latitudes:

met-10.0.0/bin/gen_vx_mask \
'latlon 72 72 -88.75 -177.5 2.5 5.0' \
'latlon 72 72 -88.75 -177.5 2.5 5.0' \
EQUATORIAL_new.nc -type lat -thresh '>=-30&&<=30'

And for NONEQUITORIAL, let's just add the -complement option:

met-10.0.0/bin/gen_vx_mask \
'latlon 72 72 -88.75 -177.5 2.5 5.0' \
'latlon 72 72 -88.75 -177.5 2.5 5.0' \
NONEQUATORIAL_new.nc -type lat -thresh '>=-30&&<=30' \
-complement

And the results are exactly what you'd expect:

On the left we have all points with latitudes of +/- 30 degrees. And on the right, we have the opposite of that.

So in general, if you want the complement of mask A, just rerun the command you used to generate mask A and add the -complement option.

Next, let's see how running gen_vx_mask iteratively is useful. For this I'll use a GFS file that contains a land/sea mask. And I'll define a mask for all nonequatorial water points:

1. Run gen_vx_mask to do data masking and pick out all the WATER points. Note that in the land mask field, the water points are stored as a value of 0:

met-10.0.0/bin/gen_vx_mask gfs_2012040900_F012.grib gfs_2012040900_F012.grib WATER.nc \
-type data -mask_field 'name="LAND"; level="L0";' -thresh eq0

2. Run gen_vx_mask a second time, passing the WATER.nc mask file as input and this time defining the non-equatorial logic we used above.

met-10.0.0/bin/gen_vx_mask WATER.nc WATER.nc NONEQUATORIAL_WATER.nc \
-type lat -thresh '>=-30&&<=30' -complement -intersection

Since we passed WATER.nc as input (and that was previously generated by gen_vx_mask), the tool initialized the value for all of the grid points to the input mask. Next, it defined the new mask for this run... the NONEQUATORIAL one which includes the use of the complement option. Note that I could easily have switched the latitude threshold instead, but wanted to show that -complement applies to the definition of the new mask. And lastly, the -intersection option defines how to combine the input mask value with the newly generated mask value. Switching -intersection to -union results in this mask:

Hope these examples help clarify the functionality.

[JohnHalleyGotway]

Here is an email response from @jvigh:

Why did you say "Since I don't have your original 72x72 NetCDF input file". I thought it was attached as a zip file? Here are the contents of the zip file I downloaded from the discussion.

I don't think that you actually answered the question. I still don't have an example that shows how to take the complement of an existing mask just by itself. Let me know if this is incorrect. This is the real problem I'm having -- how do you pass in a pre-existing mask file and take the complement with it without doing anything else. Is this possible?

For instance, could you give an example that simply takes the complement of your existing land-sea mask that you created in the example? e.g., Turn the water mask into a land mask without having to define any lat masking regions or do another operation?

[JohnHalleyGotway]

Jonathan,

EQUATORIAL.nc is an output file that you created by running the gen_vx_mask tool. What I meant was that I don't know exactly what command that was or what your input file(s) to it were. In order to create the NONEQUATORIAL.nc output file, my advice to you is:

• Find the command you ran to create EQUATORIAL.nc.
• Add on the -complement command line option.
• Rename the output file to be NONEQUATORIAL.nc.
  That's the most straight-forward and intuitive way to handle it.

To answer your exact question, no, it is NOT possible using the currently available MET version 10.0.0. The bug that is fixed by MET#1891 is the reason why it doesn't. That bug was fixed after the 10.0.0 release. I also tested using the latest development release for MET version 10.1.0, and it does work there.

So it is possible to reverse the mask in a call to gen_vx_mask. However, since you're using a gen_vx_mask output file as input to another call, we need to add logic to undo its default behavior. It strikes me as being pretty confusing:

MET-develop/met/bin/gen_vx_mask EQUATORIAL.nc EQUATORIAL.nc NONEQUATORIAL.nc \
-input_field 'name="EQUATORIAL"; level="(*,*)"; censor_thresh=[!=0]; censor_val=[0];' \
-type data -mask_field 'name="EQUATORIAL"; level="(*,*)";' -thresh eq1 -complement

So that does produce the desired result (using a development version for MET 10.1.0). Let me explain, line-by-line...

1. The first line sets INPUT_FILE = MASK_FILE = EQUATORIAL.nc and defines the output file name as NONEQUATORIAL.nc.
2. The second line defines the "-input_field" to read from the INPUT_FILE. Since this is output from a previous call to gen_vx_mask, by default it'll read the mask data and initialize all grid point values to the input mask values. I used the censor_thresh/censor_val options to OVERRIDE this default behavior. I just reset all the grid point values to 0.
3. This defines the current mask to be defined, which is done by reading the "-mask_field" from the MASK_FILE. I threshold that input data using eq1 and used the -complement option to reverse it. Obviously, you could instead use -thresh eq0 and omit the complement option.

The MET#1891 bug is problematic in MET version 10.0.0 because the censor_thresh/censor_val options defined in -input_field are ALSO used when reading data for -mask_field. So we end up resetting the mask field to all 0's... and that bug may explain some of the troubles you ran into.

So again, while it is technically possible to read gen_vx_mask output back into the tool and reverse it's values, I think it's a pretty confusing way of running it. I'd recommend the first option I suggested instead.

[JohnHalleyGotway]

@jvigh just wanted to check in on this discussion. Any additional questions or issues to address?

[jvigh]

@JohnHalleyGotway Thank you very much for the clear explanations and update on gen_vx_mask behavior (10.0.0 vs. 10.1.0) and the info about the MET#1891 bug. This is very helpful information, as is the example showing how to take the complement of a previously existing mask. I'm sure this answer will be very helpful to many users, so thanks again for the time you put into this answer.

I have a suggestion for future development that would relate to creating a simpler tool for conducting mask operations. It would work like the following:
mask_operation <mask_filename>
mask_operation <union | intersection> <mask1_filename> <mask2_filename>

I think this would be quite easy to implement (if it assumed that the defaults are always 0/1s).

Best,
Jonathan

[JohnHalleyGotway]

@jvigh I can definitely see how that functionality would be intuitive for how you're generating masks. You're right, the logic and implementation would be very straightforward. I'm uncertain as to whether we'd shoehorn that into the existing gen_vx_mask command line or create a separate utility for it.

If we did want to use the existing tool, perhaps there's some new "-type" option we could define that'd make that functionality very clear? Here's the existing supported types:

"-type string" specify the masking type (required).
		   "poly", "box", "circle", "track", "grid", "data", "solar_alt", "solar_azi", "lat", "lon" or "shape"

Each time gen_vx_mask is run, it defines a new mask and, optionally, intersects or unions it with the input mask. You want to run it where the "input_file" and "mask_file" are both the output from previous gen_vx_mask runs. So how could we set the "-type" option to tell gen_vx_mask to just read the already defined mask from the "mask_file"?

Or maybe we make this a special case of the existing "-type data" mask? Add logic to check if the "mask_file" was previously generated by gen_vx_mask and, if so, automatically use that mask as-is without needing to define the "-mask_field" option?

What would be the most intuitive for the user?

[jvigh]

I think a separate utility would make the most sense. That would free the user from the complexity of gen_vx_mask for the case when they are trying to do something simple (but which can become quite complex if done in a powerful tool like gen_vx_mask). It's good to also have some simple tools for simple tasks :)

[JohnHalleyGotway]

Sounds good. At this point, I'm going to mark this discussion as being answered and lock it to prevent future posts. We've found it difficult in the past to keep track of new posts to discussions that have already been answered. And this is how we encourage to create new discussions for new questions.

You're welcome to write up a GitHub issue feature request for the new tool you describe. And if you do, I'd recommend referencing this discussion (#1219) in the description.