I am relatively new to using the netcdf-java library, and I've immediately run into a problem when trying to load a file. The problem is that there doesn't seem to be a way to load a NetcdfFile from a byte array stored in memory, and that is the base form of my data. To elaborate a little, it is actually a .cdf file uploaded through a client, which the client then converts into a byte array for the server code to read. So the server, where my code is running, cannot see the uploaded file at all. I also cannot assume the server itself is writable, so essentially there is no "location" to pass into the typical NetcdfFile loading methods.
The FAQ on ucar.edu does mention the possibility of reading from a non-file source, here. It says I should write my own IOSP, which I am happy to do. However, there is very little guidance on how to do this.
I don't know how to implement isValidFile when the only thing passed into the function is a RandomAccessFile, which the FAQ says can be ignored.
I don't know how my IOSP will obtain the byte array in question for use in readData.
I don't know why the minimal example in the FAQ advises me to make a new NetcdfFile class, when it seems I could just use the default one but pass in my custom IOSP.
This question is a little vague, but I am truly lost without many clues on where to even begin. Any guidance would be appreciated.
EDIT: I'm using 5.4.2 of the netcdf-java library.
I found this answer in the support archives. The solution is to use InMemoryRandomAccessFile. The constructor takes a String location and a byte array containing the file's contents. From my testing, I think the location can be any arbitrary string. Here is the code that worked for me.
byte[] filebytes = retrieveFileBytes(clientFilepath);
InMemoryRandomAccessFile raf = new InMemoryRandomAccessFile(clientFilepath, filebytes);
NetcdfFile file = NetcdfFiles.open(raf, clientFilepath, null, null);
Variable peakRetentionTime = file.findVariable("peak_retention_time");
if (peakRetentionTime == null) {
displayWarning("peak_retention_time null!");
} else {
Array data = peakRetentionTime.read();
displayInfo(Ncdump.printArray(data));
}
Related
I have a method that takes in a byte[] that came from Files.readAllBytes() in a different part of the code for either .txt or .docx files. I want to create a new File object from the bytes to later read contents from, without saving that file to disk. Is this possible? Or is there a better way to get the contents from the File bytes?
That's not how it works. a java.io.File object is a light wrapper: Check out the source code - it's got a String field that contains the path and that is all it has aside from some bookkeeping stuff.
It is not possible to represent arbitrary data with a java.io.File object. j.i.File objects represent literal files on disk and are not capable of representing anything else.
Files.readAllBytes gets you the contents from the bytes, that's.. why the method has that name.
The usual solution is that a method in some library that takes a File is overloaded; there will also be a method that takes a byte[], or, if that isn't around, a method that takes an InputStream (you can make an IS from a byte[] easily: new ByteArrayInputStream(byteArr) will do the job).
If the API you are using doesn't contain any such methods, it's a bad API and you should either find something else, or grit your teeth and accept that you're using a bad API, with all the workarounds that this implies, including having to save bytes to disk just to satisfy the asinine API.
But look first; I bet there is a byte[] and/or InputStream variant (or possibly URL or ByteBuffer or ByteStream or a few other more exotic variants).
I am trying a create a .dat file that is split into 5 columns each with the heading, "timestamp", "deviceId", "testId", "Availability" and "Latency".
This file is going to be pointed to a database where there are tests being ran at the moment.
The idea is that the data from these tests will be collected and stored into this file that I have just created.
I have been told to use and work with RandomAccessFile.
Only problem is I have no idea how to write a file through Java.
I have made somewhat of an attempt but it is by no means much at all;
public long timestamp;
public int deviceId;
public int testId;
public byte availability;
public int latency;
public static void main (String [] args) throws FileNotFoundException
{
String fileName = "hopeToJaysusThisWorks.dat";
RandomAccessFile file = new RandomAccessFile(fileName, "rw");
}
As you can see, its not much at all!
I was told not to worry about getting into the technical side of things yet i.e the collection of data. At the moment all I need is the layout of each column and its 5 headings.
Can anyone help me or give me some advice on where to go from here, it would be much appreciated.
Well the next step for you, would be to make sure the file you just tried to open for reading/writing (that's what the "rw" option means, if you give it just a r or just a w, it's for only reading, or only writing, respectively), is actually open/available. You can do this by simply running a check to see if your 'file' variable is null. Ok great! now you need to actually start writing some information into the file. This can take MANY forms, I recommend checking out the RandomaccessFile api to see exactly what methods are available, and what form you data has to be in to write it.
It'll look something like:
if (file != null){
file.write(myByteArray[]);
}
now, you may have your data structured however, but that's the general gist of it.
Good luck!
Given ZipFile zip, its ZipEntry entry and target File unzippedEntryFile, what is the most readable way to write entry to unzippedEntryFile?
I came up with following solution using Google Guava and Apache.Commons.IO:
InputSupplier<ByteArrayInputStream> entryInputSupplier = ByteStreams.newInputStreamSupplier(IOUtils.toByteArray(zip.getInputStream(entry)));
Files.copy(entryInputSupplier, unzippedEntryFile);
However, something tells me it can be made simplier.
Thanks,
Konrad
I don't know what's not readable about your code, other than that you have a lot of nested calls. These can be broken out and assigned to local variables, which would make the code longer (but, in my opinion, a bit more readable).
You do seem to be processing the data twice -- once to read it into a byte array and again to copy it to the file. I haven't tested this, but it should work and cut the amount of data movement in half:
final InputStream zipStream = zip.getInputStream(entry);
InputSupplier<InputStream> supplier = new InputSupplier<InputStream>() {
InputStream getInput() {
return zipStream;
}
};
Files.copy(supplier, unzippedEntryFile);
You could, in fact, create your own little class that implements InputSuppler<InputStream>. I'm surprised that I couldn't find one in the Guava library. (Apparently, others have been surprised at this as well.)
Here is a quick link that might have all the possible scenarios for Zip file operation. If it is simple file operation, here is the tutorial from Sun. Am sure there are lot of open source API's. Here is the link to IO tutorial. Cheers
I am new to Java. I want to learn to use GZIPstreams. I already have tried this:
ArrayList<SubImage>myObject = new ArrayList<SubImage>(); // SubImage is a Serializable class
ObjectOutputStream compressedOutput = new ObjectOutputStream(
new BufferedOutputStream(new GZIPOutputStream(new FileOutputStream(
new File("....")))));
compressedOutput.writeObject(myObject);
and
ObjectInputStream compressedInput = new ObjectInputStream(
new BufferedInputStream(new GZIPInputStream(new FileInputStream(
new File("....")))));
myObject=(ArrayList<SubImage>)compressedInput.readObject();
When the program writes myObject to a file without throwing any exception, but when it reaches the line
myObject=(ArrayList<SubImage>)compressedInput.readObject();
it throws this exception:
Exception in thread "main" java.io.EOFException: Unexpected end of ZLIB input stream
How can I solve this problem?
You have to flush and close your outputstream. Otherwhise, at least, the BufferedOutputStream will not write everything to the file (it does in big chucks to avoid penalizing performance).
If you call compressedOutput.flush() and compressedOutput.close() it will suffice.
You can try writing a simple string object and checking if the file is well written.
How? If you write a xxx.txt.gz file you can open it with your preferred zip app and look at the xxx.txt. If the app complains, then the content is not full written.
Extended answer to a comment: compressing even more the data
Changing serialization
You could change the standard serialization of SubImage object if it's an object of your own. Check java.io.Serializable javadoc to know how to do it. It's pretty straightforward.
Writing just what you need
Serialization has the drawback that needs to write "it's a SubImage" just before every instance you write. It's not necessary if you know what's going to be there beforehand. So you could try to serialize it more manually.
To write your list, instead of writing an object write directly the values that conform your list. You will need just a DataOutputStream (but ObjectOutputStream is a DOS so you can use it anyway).
dos.writeInt(yourList.size()); // tell how many items
for (SubImage si: yourList) {
// write every field, in order (this should be a method called writeSubImage :)
dos.writeInt(...);
dos.writeInt(...);
...
}
// to read the thing just:
int size = dis.readInt();
for (int i=0; i<size; i++) {
// read every field, in the same order (this should be a method called readSubImage :)
dis.readInt(...);
dis.readInt(...);
...
// create the subimage
// add it to the list you are recreating
}
This method is more manual but if:
you know what's going to be written
you will not need this kind of serialization for many types
it's pretty affordable and definitively more compressed than the Serializable counterpart.
Have in mind that there are alternative frameworks to serialize objects or create string messages (XStream for xml, Google Protocol Buffers for binary messages, and so on). That frameworks could work directly to binary or writing a string that could be then written.
If your app will need more on this, or just curious, maybe you should look at them.
Alternative serialization frameworks
Just looked in SO and found several questions (and answers) addressing this issue:
https://stackoverflow.com/search?q=alternative+serialization+frameworks+java
I've found that XStream is pretty easy and straightforward to use. And JSON is a format pretty readable and succint (and Javascript compatible which could be a plus :).
I should go for:
Object -> JSON -> OutputStreamWriter(UTF-8) -> GZippedOutputStream -> FileOutputStream
It seems all methods expect either files or urls. I see some methods that work with OutputStream, but I haven't managed to open an IContainer using one of those methods; I always get an invalid return value.
Create your own IURLProtocolHandler interface and pass to IContainer.open(...) to open any type of media type you want.
You can look at this answer I posted on another question to write to an OutputStream (which could easily be a ByteArrayOutputStream).
This gist of it would be to use com.xuggle.xuggler.io.XugglerIO to map from an OutputStream to a special kind of file URL so that FFMPEG can access the stream.
IMediaWriter writer = ToolFactory.makeWriter(XugglerIO.map(outputStream));
Keep in mind that you'll now have to manually set your format (because it can't detect it from the filename). For example:
IContainerFormat containerFormat = IContainerFormat.make();
containerFormat.setOutputFormat("ogg", null, "application/ogg");
writer.getContainer().setFormat(containerFormat);