NDBC stations have invalid geometry elements in ISOs

[kwilcox]

Many (if not all) of the ISO records generated from NDBC buoys have nan in their location fields.

data.ioos.us/stations/waf/ndbc/sdf.ndbc.noaa.gov-urn_ioos_station_wmo_*

<gmd:extent>
  <gmd:EX_Extent>
    <gmd:geographicElement>
      <gmd:EX_GeographicBoundingBox>
        <gmd:extentTypeCode>
          <gco:Boolean>1</gco:Boolean>
        </gmd:extentTypeCode>
        <gmd:westBoundLongitude>
          <gco:Decimal>nan</gco:Decimal>
        </gmd:westBoundLongitude>
        <gmd:eastBoundLongitude>
          <gco:Decimal>nan</gco:Decimal>
        </gmd:eastBoundLongitude>
        <gmd:southBoundLatitude>
          <gco:Decimal>nan</gco:Decimal>
        </gmd:southBoundLatitude>
        <gmd:northBoundLatitude>
          <gco:Decimal>nan</gco:Decimal>
        </gmd:northBoundLatitude>
      </gmd:EX_GeographicBoundingBox>
    </gmd:geographicElement>
  </gmd:EX_Extent>
</gmd:extent>

[rsignell-usgs]

@mwengren, do we know who the contact at NDBC is?

[mwengren]

@kwilcox When I was working on this module originally, I made a list of issues to follow up on (mostly wrt NDBC) after the first Catalog version came out, that at a minimum had a record for every provider's stations.

Needless to say, it never happened. @emiliom found and resolved some of them as well recently. This is what I have in my notes for NDBC issues, so I found this as well back then:

• NaN Geographic extent values passed from SensorML

Here's the code in sensorml2iso that parses lat/lon values: https://github.com/ioos/sensorml2iso/blob/master/sensorml2iso/sensorml2iso.py#L289-L298. There's already some special handling for NDBC since they use a different approach than 52N.

But, if I had to guess NDBC's SOS includes some DescribeSensor responses that don't include numbers in the locations we're expecting. Here's an example to show their formatting for most stations:

    <sml:location>
        <gml:Point gml:id="stationLocation51028" srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
          <gml:coordinates>0 -153.913</gml:coordinates>
        </gml:Point>
      </sml:location>

@rsignell-usgs: @kbailey-noaa is our NDBC POC. At the moment, RPS runs a process to generate NDBC's SOS metadata for them. We've reached out to them in the past about running sensorml2iso themselves and managing their own waf, but I don't know if it's on a roadmap for this FY or not.

I looked at some examples of the NaN errors in the Harvest Registry, and tracking down the corresponding DescribeSensor request for one, it looks like the position info is encoded in varying ways. We just haven't accommodated for the different possibilities.

		<!-- Station Location -->
		<sml:positions>
			<sml:PositionList>
				<sml:position name="stationPosition">
					<swe:Position>
					<swe:location>
						<swe:Vector gml:id="stationLocation" definition="urn:ogc:def:property:OGC:location">
							<swe:coordinate name="latitude">
								<swe:Quantity axisID="Y">
									<swe:uom code="deg"/>
									<swe:value>48.04000</swe:value>
								</swe:Quantity>
							</swe:coordinate>
								<swe:coordinate name="longitude">
								<swe:Quantity axisID="X">
									<swe:uom code="deg"/>
									<swe:value>163.49001</swe:value>
								</swe:Quantity>
							</swe:coordinate>
							<swe:coordinate name="altitude">
								<swe:Quantity axisID="Z">
									<swe:uom code="m"/>
									<swe:value>0</swe:value>
								</swe:Quantity>
							</swe:coordinate>
						</swe:Vector>
					</swe:location>
					</swe:Position>
				</sml:position>
				<sml:position name="bprPosition">
					<swe:Position>
					<swe:location>
						<swe:Vector gml:id="bprLocation" definition="urn:ogc:def:property:OGC:location">
							<swe:coordinate name="latitude">
								<swe:Quantity axisID="Y">
									<swe:uom code="deg"/>
									<swe:value>48.12167</swe:value>
								</swe:Quantity>
							</swe:coordinate>
								<swe:coordinate name="longitude">
								<swe:Quantity axisID="X">
									<swe:uom code="deg"/>
									<swe:value>163.32834</swe:value>
								</swe:Quantity>
							</swe:coordinate>
							<swe:coordinate name="altitude">
								<swe:Quantity axisID="Z">
									<swe:uom code="m"/>
									<swe:value>-5788.6</swe:value>
								</swe:Quantity>
							</swe:coordinate>
						</swe:Vector>
					</swe:location>
					</swe:Position>
				</sml:position>
			</sml:PositionList>
		</sml:positions>

So, seems like we either go through and try to find each variation and account for it, or perhaps @kbailey-noaa can put us in contact with someone at NDBC who knows this service well enough to describe the different possibilities. They describe some here: https://sdf.ndbc.noaa.gov/sos/.

If anyone wants to tackle any of this, have at it..... I won't be able to get into it for awhile.

[emiliom]

Regarding this:

At the moment, RPS runs a process to generate NDBC's SOS metadata for them. We've reached out to them in the past about running sensorml2iso themselves and managing their own waf, but I don't know if it's on a roadmap for this FY or not.

@mwengren Is the RPS process you're referring to sensorml2iso, based on the latest release? That was my assumption, but I just want to make sure I didn't misunderstand. If this is the case, when do you expect my fixes to be operational and reflected in the catalog pycsw, for testing/confirmation? Just curious.

Regarding the geometry problem, it's probably worth looking into the coordinate encoding in their GetCapabilities offering records. If the same encoding errors are there, that's that. But if they're free (or freer) from those screwy errors, it should be easy to change sensorml2iso to read coordinates from GetCapabilities instead.

[benjwadams]

sensorml2iso ion catalog is presently running v1.04 on the catalog server. I see a new tagged release has been made on the 21st of August; I have been OoO/busy for the previous two days, so have not had occasion to upgrade the release, but can do so now.

[mwengren]

I need to get a clarification from @ocefpaf on how/when conda-forge will pick up the new release I just made (1.0.5 - dropped the unnecessary 'v' prefix). I thought it would have been picked up by now, but I can't seem to remember how that process works. @benjwadams I presumed you could update after that happened.

[benjwadams]

@ocefpaf, @mwengren I've updated the version of sensorml2iso running on the Catalog server to 1.0.5.

IIRC, we do builds on DockerHub when new PRs are merged, but specific tagged versions aren't pushed up automatically. I checked out 1.0.5 and built it, pushed it to DockerHub with the proper tag, and then pulled it down onto production.

[mwengren]

@emiliom referring to issues in ioos/sensorml2iso#28, I am able to download CSV observations from the CO-OPS stations in the Catalog now (eg. https://data.ioos.us/dataset/urn-ioos-station-noaa-nos-co-ops-9087096-station-port-inland-mi1), so I think the new sensorml2iso version @benjwadams deployed means we're good on that issue now.

[mwengren]

Related sensorml2iso issue: ioos/sensorml2iso#30

[emiliom]

Regarding the NDBC fixes from ioos/sensorml2iso#26 and #63 (fixed on sensorml2iso by PR ioos/sensorml2iso#27), @benjwadams said on 8/23:

I've updated the version of sensorml2iso running on the Catalog server to 1.0.5.

But I'm not seeing the fixes reflected in either the Catalog (eg, https://data.ioos.us/dataset/delaware-bay-26-nm-southeast-of-cape-may-nj1) or via the CSW query in my sample gist at https://gist.github.com/emiliom/ea4ae7e6fe051d8cac2788015e60e0cf#file-iooscatalog_csw_oldtemporalextent-ipynb

[mwengren]

@emiliom I believe the CO-OPS stations and 'None' strings were fixed in Catalog, the ones I tested anyway. You are referring to another problem here specific to NDBC correct?

[emiliom]

@emiliom I believe the CO-OPS stations and 'None' strings were fixed in Catalog, the ones I tested anyway.

As far as I can tell, that's the case.

You are referring to another problem here specific to NDBC correct?

Yeah, I'm referring to the NDBC-specific problems we fixed in ioos/sensorml2iso#27 that should be deployed by now (1.0.5). I brought them up here since they've been mentioned in this issue, too.

[benjwadams]

I updated the harvesters last week to use the NDBC flag. Please check if they're functioning now.

[benjwadams]

I'd like to close this issue out. Could anyone confirm/deny if this is working as intended?

[benjwadams]

Closing due to inactivity and fix applied.