Can't open VIIRS HDF5 files

[rschmunk]

Versions impacted by the bug

5.10 snapshot

What went wrong?

A Panoply user reported an exception trying to open any VIIRS HDF5 files. I found that toolsUI encounters the same trouble. Whether this is something best solved by patching the NJ code, netCDF-C code, or both, I don't know.

Using just netCDF-Java, an EOFException is thrown, apparently while trying to assemble header data and when reading a four-byte integer. The stack trace I get in Panoply is that

java.io.IOException: java.io.EOFException
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:282)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:246)
	at gov.nasa.giss.data.nc.NcDataset.acquireFromFile(NcDataset.java:436)
	...
Caused by: java.io.EOFException
	at ucar.unidata.io.RandomAccessFile.readInt(RandomAccessFile.java:1241)
	at ucar.unidata.io.RandomAccessFile.readLong(RandomAccessFile.java:1327)
	at ucar.unidata.io.RandomAccessFile.readLong(RandomAccessFile.java:1361)
	at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:138)
	at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:58)
	at ucar.nc2.internal.iosp.hdf5.H5headerNew$Vinfo.readArray(H5headerNew.java:1972)
	at ucar.nc2.internal.iosp.hdf5.H5headerNew.makeVariable(H5headerNew.java:1357)
	at ucar.nc2.internal.iosp.hdf5.H5headerNew.makeNetcdfGroup(H5headerNew.java:539)
	at ucar.nc2.internal.iosp.hdf5.H5headerNew.makeNetcdfGroup(H5headerNew.java:529)
	at ucar.nc2.internal.iosp.hdf5.H5headerNew.makeNetcdfGroup(H5headerNew.java:529)
	at ucar.nc2.internal.iosp.hdf5.H5headerNew.read(H5headerNew.java:254)
	at ucar.nc2.internal.iosp.hdf5.H5iospNew.build(H5iospNew.java:126)
	at ucar.nc2.NetcdfFiles.build(NetcdfFiles.java:839)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:774)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:279)
	... 13 more
000033 WARNING: PanOpenDatasetTask openDataset - IOException opening dataset: java.io.EOFException
java.io.IOException: java.io.EOFException
	at gov.nasa.giss.data.nc.NcDataset.acquireFromFile(NcDataset.java:454)
	...

I subsequently gave it a try with netCDF-Java configured to use the netCDF-C binary installed on the system (macOS) via nj22config.xml. This time there was an IOException stating that "-107: netCDF: Can't open HDF attribute". The stacktrace for this exception reads

java.io.IOException: java.io.IOException: -107: NetCDF: Can't open HDF5 attribute
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:282)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:246)
	at gov.nasa.giss.data.nc.NcDataset.acquireFromFile(NcDataset.java:436)
	...
Caused by: java.io.IOException: -107: NetCDF: Can't open HDF5 attribute
	at ucar.nc2.jni.netcdf.Nc4Iosp.makeGroup(Nc4Iosp.java:298)
	at ucar.nc2.jni.netcdf.Nc4Iosp.makeGroup(Nc4Iosp.java:325)
	at ucar.nc2.jni.netcdf.Nc4Iosp.makeGroup(Nc4Iosp.java:325)
	at ucar.nc2.jni.netcdf.Nc4Iosp._open(Nc4Iosp.java:277)
	at ucar.nc2.jni.netcdf.Nc4Iosp.open(Nc4Iosp.java:237)
	at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1610)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:774)
	at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:279)
	... 13 more
000041 WARNING: PanOpenDatasetTask openDataset - IOException opening dataset: java.io.IOException: -107: NetCDF: Can't open HDF5 attribute
java.io.IOException: java.io.IOException: -107: NetCDF: Can't open HDF5 attribute
	at gov.nasa.giss.data.nc.NcDataset.acquireFromFile(NcDataset.java:454)
	…

I tried poking at this with some other utilities and got some differing results.

The HDF View app opens the file. There are attributes on the root level, and on a subgroup named 'Data_Products/VIIRS-M5-SDR'. All the attributes are strings except for one 4-byte integer on the subgroup.

ncdump dumps part of the file but fails midway with the same 'Can't open HDF5 attribute' exception. It appears it is getting to that 'Data_Products/VIIRS-M5-SDR' subgroup and doesn't like what it sees.

Relevant stack trace

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

[rschmunk]

The example file I was given is available via Dropbox

But the user informed me that he encountered the same initial EOFException with any VIIRS file from where he obtained this one.

[rschmunk]

@lesserwhirls, Thanks for looking into this.

I can confirm that Panoply is now able to open the sample VIIRS data file and make a plot if I use just the netCDF-Java library. Woot.

However, if I have things configured so that NJ links in the netCDF-C library binary if I have a nj22config.xml file in place, then the attribute error still occurs. But I guess that should be filed as a netCDF-C bug. No?

[lesserwhirls]

It does not look like this HDF5 file is a netCDF4 file, so all bets are off in terms of the netCDF-C library being able to read it. That said, I'm a little surprised that netCDF-Java is trying to read the file through the C library when it should be identified as an HDF5 file. This looks to be a bug in the order IOSPs are checked when the netCDF-C library is configured through nj22config.xml.

[lesserwhirls]

On, never mind, I see what's happening. Since we don't have a separate netCDF-4 IOSP (reading netCDF-4 goes through the HDF5 IOSP, as netCDF-4 is sort of a metadata convention on top of HDF5), enabling the C library to read netCDF-4 ends up replacing our java HDF5 IOSP (so all HDF5 files end up being read through netCDF-C). I believe there are some checks we can do to try to only read netCDF-4 files through the C library, but it won't be 100% (but I think will be much better). Let me see what I can come up with.

[rschmunk]

I think you mean "only read netCDF-4 files through the Java library", right?