Exception reading data with caching turned on

[gwlucastrig]

This may be a new issue.

I am seeing an exception when attempting to read from a large (10.9 GB) NetCDF file that contains a global elevation and bathymetry data set. The exception occurs when the setCaching(true) is employed. I recognize that this method is deprecated, and I also note that it is not necessary for the particular file that I am accessing. But it is included in my code because it has proven absolutely necessary to get decent performance when reading similar data sets with different chunking schemes... For details, see Discussion number 1399.

I am using cdm-core-5.6.0

The exception is

Exception in thread "main" java.lang.RuntimeException: Read request size of 3732480000 is too large.
	at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:288)
	at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:230)
	at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:204)
	at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122)
	at ucar.nc2.Variable.reallyRead(Variable.java:797)
	at ucar.nc2.Variable._read(Variable.java:736)
	at ucar.nc2.Variable._read(Variable.java:759)
	at ucar.nc2.Variable.read(Variable.java:600)
	at ucar.nc2.Variable.read(Variable.java:546)

The file I am reading is the GEBCO 2019 global bathymetry set which can be downloaded from The GEBCO Historical downloads. To download it, click on the link marked "Download GEBCO 2021 grid".

The variable I am accessing is

float elevation(lat=43200, lon=86400);
  :long_name = "Elevation relative to sea level";
  :units = "m";
  :sdn_parameter_urn = "SDN:P01::ALATZZ01";
  :sdn_parameter_name = "Sea floor height (above mean sea level) {bathymetric height}";
  :sdn_uom_urn = "SDN:P06::ULAA";
  :sdn_uom_name = "Metres";
  :standard_name = "height_above_reference_ellipsoid";
  :_ChunkSizes = 1U, 86400U; // uint

I access it using code as shown below. The code is intended to read data from the file one row at a time. It throws an exception the first time it is called.

       elevation.setCaching(true);
	int readOrigin = new int[]{0, 0};
	int readShape  = new int[](1, 86400};
	 Array array = elevation.read(readOrigin, readShape);

I believe that the "elevation" variable is a single-precision float and that the read request size should only be about 4 x 86400 = 345600 bytes. I do not have an explanation for why the read request ends up being 3732480000. But I note that this value corresponds to the number of cells in grid for the variable, with 43200 x 86400 = 3732480000.

[lesserwhirls]

When you set caching on a variable, the entire data array will be read into memory before any index subsets are performed, which is likely not quite what you are wanting. It's really intended for caching things like coordinate variable data, which tend to be accessed multiple times and do not consume lots of space. It can be useful in other situations, as you have found previously, but it's not always appropriate.

[gwlucastrig]

Thanks. That explains a lot. It also points me toward a better way to use the API than depending on the cache.

Please feel free to close this discussion if you'd like to do so.

Gary