I am a newbie to netcdf library and i am currently using it to read the metadata from a hdf5 file in java. After reading, I thought that netcdf is a decent enough library for my use, so using it.
however in the first step when i try to read the file it throws an error
LOGGER.debug("Inside Try");
//InputStream fileStream = new FileInputStream(h5File);
//parser.parse(fileStream, handler, metadata);
LOGGER.debug("path is :"+ h5File.getPath());
NetcdfFile hf5File = NetcdfFile.open(h5File.getPath());
LOGGER.debug("Got NetCdFile");
I am assuming the problem occurs when i try to open it , as it says :
Inside Try
13:42:04.393 [main] DEBUG e.k.n.c.m.e.HDF5MetadataExtractor - path is :/var/www/webdav/admin/1151/data/XXXX.h5
13:42:04.495 [main] DEBUG ucar.nc2.NetcdfFile - Using IOSP ucar.nc2.iosp.hdf5.H5iosp
13:42:04.544 [main] ERROR ucar.nc2.iosp.hdf5.H5header - shape[0]=0 must be > 0
My Hd5f is a 2 dimensional array of integers, I am not interested in the array as such but with the metadata group associated with the file.
NetCDF-4 creates HDF5 files, it's true. The HDF5 library can read the HDF5-formatted files produced by NetCDF-4, but NetCDF-4 cannot read arbitrary HDF5 files.
Either you have found a bug in Netcdf-Java, or you have an odd HDF5 file. Have you confirmed this file is not corrupted in some way? Things i would try:
- use the C utility 'ncdump -h' to inspect the header
- use the HDF5 C utility 'h5dump -H' to inspect the file via HDF5
If both of those commands give sensible output, then the issue might rest with netcdf-java.
Netcdf-Java is a pure Java implementation that can read most HDF5 files, details here: http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/CDM/index.html#HDF5
If you find a failure, submit a bug report (and the file!) to netcdf-java#unidata.ucar.edu (must sign up here: http://www.unidata.ucar.edu/support/index.html#mailinglists). But first make sure you have tried the latest version (currently 4.6.3).
Netcdf-Java can write netCDF-4 files (which are HDF5 files) through a JNI interface to the netcdf-4 C library. HDF5 java libraries are also JNI interfaces, except directly to the HDF5 C library. HDF5 java libraries cannot write netCDF4 files (unless you really know what you are doing) because HDF5 does not implement shared dimensions, which are essential to the netCDF data model.
For earth science, this leads to the argument (disclaimer: by me) that you should write netCDF-4, not directly HDF5, details here: http://www.unidata.ucar.edu/blogs/developer/en/entry/dimensions_scales.
Well, thats probably more information than you wanted ;^)
Related
This question already has answers here:
How to break a file into pieces using Java?
(2 answers)
Closed 11 months ago.
Hello I need to split a zip file into multiple samaller files,
ex. file.zip is splitted into files with fixed size less than 640MB and format nameOfZip.zip.001
I am currently looking for some kind of a java library that would be able to do this. If there is no such, advice would be very helpful too.
Zip, the format, can do this. You can then unzip the batch of files with plain jane unzip on the command line (or with tools like p7zip and such, or you can just doubleclick the zip on a mac, etcetera).
Unfortunately, the baked in zip support in java can't make split zips. But, Lingala / Zip4j can do it.
Add that library to your list of dependencies and use its API (forget about java.io.ZipOutputStream, or the Zip FileSystem - anything that starts with java.* can't do this.
Alternatively it is trivial to write code in java to just split any file. You'd need to write java code as well to put them back together, or you need to know a few things about your OS to do this (e.g. cat a.bin b.bin >c.bin on posix OSes will put a.bin and b.bin back together). This isn't difficult at all, just your basic file and outputstream support can trivially put it together in less than a page's worth of java code. No libraries exist, and probably never will - that is a very simple task that isn't common enough to make a library for.
So, if that's what you're looking for, go ahead and write it yourself. All you need is the javadoc of java.nio.file.Paths and java.nio.file.Files.
I have large number of files stored in gz format and trying to run map-reduce program (using PIG) by reading those files. Problem I am running into is, native Decompressor in Hadoop (ZlibDecompressor) is not able successfully decompresss some of it due to data check. But I am able to read those files successfully using java GZIPInputStream. Now my question is - Is there a way to disable Zlib? Or are there any alternate GZipCodec in hadoop(2.7.2) which I can use to decompress gzip input files?
Error given below
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1475882463863_0108_m_000022_0 - exited : java.io.IOException: incorrect data check
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native Method)
at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:228)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
Thank you very much for your help.
I found the answer myself. You can set following property to disable all native libraries.
io.native.lib.available=false;
or you can extend org.apache.hadoop.io.compress.GzipCodec.java to remove native implementation only for GzipCompressor.
Is there a way to get the file signature of a file inside a document? I tried checking the Embedded Object Class but it seems that there is no function for getting the file signature. Is there a way to get it just by using lotus script?
If not then I believe I'll need to maybe use a java class agent right? Can you provide a link that is doing this function or maybe can you guide me with some codes for this one. I am familiar with java but when it is being used in lotus notes agents I am not much familiar with the class being used.
Basically what I need to do is check the files in the documents and check if they are a valid file with the valid signature. Just checking the extension name is not enough as it might be renamed but the signature of the file is not valid so I'll need to confirm the file signature in checking. Thanks.
You have to write the file to file system and then you can read the file from there. Use a temp directory and delete every file after usage.
Look here for code to write attachments to file system as a starting point. Property EmbeddedObjects is available for documents too in case you want to analyse all attachments of a document.
You can accomplish the same in Java. Just look for Java classes in Language cross-reference in documentation.
i have a windows app based on mfc that saves its doc using the CArchive (MFC) serialization class
i would like to load these files into my new andriod app but need some java code to understand the serialized data file format. once i pull it apart i can handle the data ok but don't really want to dissect CArchive created files myself. can anyone help, maybe a library out there somewhere?
See previous answers to this question (with respect to Ruby instead of Java) - Parsing CArchive (MFC classes) files in Ruby
Unfortunately, I think that you're going to have dissect it yourself.
How can I split *.wmv file (using java)?
I tryed simple algorythm like read bytes from wmv file and store first half in one file and other half in another file. But the second becomes non-playable.
As I can see i must add to the second file correct header to allow media-players interpret data correct.
Is it true? How can i do splitting if it is not and where can i find wmv header specification if my assumption is correct?
You won't be helping yourself with any format definitions, since WMV files are handled properly only through the Windows Media Format SDK.
Here is some (very little) info on how to call COM from java: http://www.eggheadcafe.com/software/aspnet/29766681/windows-media-encoder-sdk-java.aspx
Then, go to http://sourceforge.net/projects/windowsmedianet/files/WindowsMediaNetSamples/Dec%202008/
and download the samples, look into WMVSPLIT (I guess that's the name of the sample you should read).
Also, you should know that you will be able to split the files ONLY at CLEAN_POINTs (that's WMV lingo for KEYFRAME).
EDIT:
In fact, I would go, in your shoes, for some windows machine and simple .exe or some other kind of extra-process utility that you will execute from java. My strong belief is that it would be simpler.
And if you don't have a windows machine, you'll have to go through the VLC code to find ASF format parser.