I am having trouble using Deflater to write a GZIP file. I created a default header and used CRC32 to keep track of the checksum.
The file I am zipping is smaller than my buffer, but the output I get for this compressor is ~200 bytes larger than it should be (gzip creates a file of 457 bytes while my code is creating a file of 652 bytes. I printed the compressedSize and it says it was 634 bytes) I did a hexdump on my final file, and it says that both my trailer and my main file is incorrect, but my header is correct. I am not allowed to use GZIPOutputStream for this assignment, but I used it's code to write the header and trailer. The amount of bytes read in is correct.
The "manage" object is an object that does the reading and writing from System.in and System.out in a synchronized matter (this is for multithreading), and I verified that they should read and write a file in order. I looked at the GZIPOutputStream source and the DeflaterOutputStream source, and my code looks similar, so I am unsure why my compressor is giving me such a large compressed byte array. I played with the Deflater levels and the strategies, but they give me the same result.
EDIT: The constructor for my Deflater is
Deflater compressor = new Deflater(Deflater.DEFAULT_LEVEL, true);
CRC32 checksum = new CRC32();
checksum.reset();
int uncompressedLength = 0;
uncompressedLength = manage.read(buff, threadNum, prime);
if (uncompressedLength > 0)
{
checksum.update(buff, 0, uncompressedLength);
compressor.setInput(buff);
compressor.finish();
byte[] output = new byte[BUFFER_SIZE];
compressor.deflate(output);
int compressedDataLength = (int) compressor.getBytesWritten();
manage.write(output, compressedDataLength, threadNum, (int) checksum.getValue(), uncompressedLength);
The Deflater class has three constructors. The one with two arguments uses a boolean that, if true, indicates ZLIB header and checksum fields should not be used, which is what GZIP needs. The other two constructors (one no-args, the other only specifying compression level) default to using those header and checksum fields. In other words, it's like the two-args constructor with false. Maybe try the one with the boolean and set it to true?
Here's the constructor doc.
Related
I am writing an android application using java in which I need to decompress random lzo compressed files in byte arrays. I am doing that with the https://github.com/shevek/lzo-java library. The algorithm, however expects the size of the original file (uncompressed file) in order to decompress the lzo file.
byte[] uncompressed = new byte["Size of the uncompressed byte array which should be known before decompression"];
ByteArrayInputStream is = new ByteArrayInputStream("compressed byte array");
LzopInputStream us = new LzopInputStream(is);
DataInputStream ds = new DataInputStream(us);
System.out.println(ds.readByte());
ds.readFully(uncompressed);
I need to read the uncompressed file size from the header of the lzop compressed file. Where can I find the header structure of such files or read the decompressed file size from the header of an lzop compressed file?
(For the header, I read the first few hundred bytes, but reading them either in BIG_ENDIAN or LITTLE_ENDIAN or as shorts, Ints, and Longs -and with different beginnings for the bytes do not verify the uncompressed file size for a known file)
Thank you in advance for your help!
In a java program I am compressing an InputStream like this:
ChannelBufferOutputStream outputStream = new ChannelBufferOutputStream(ChannelBuffers.dynamicBuffer(BUFFER_SIZE));
GZIPOutputStream compressedOutputStream = new GZIPOutputStream(outputStream);
try {
IOUtils.copy(inputStream, compressedOutputStream);
} finally {
// this should print the byte size after compression
System.out.println(outputStream.writtenBytes());
}
I am testing this code with a json file that is ~31.000 byte uncompressed and ~7.000 byte compressed on disk. Sending a InputStream that is wrapping the uncompressed json file to the code above, outputStream.writtenBytes() returns 10 which would indicate that it compressed down to only 10 byte. That seems wrong, so I wonder where the problem is. ChannelBufferOutputStream javadoc says: Returns the number of written bytes by this stream so far. So it should be working.
Try calling GZIPOutputStream.finish() or flush() methods before counting bytes
If that does not work, you can create a proxy stream, whose mission - to count the number of bytes that have passed through it
I'm looking for something simple (no external lib preferably) to write and load a byte[] from a file. More or less something like [Python's pickle][1].
byte[] bytes = new byte[10];
ByteBuffer bbuf = new ByteBuffer.allocate(bytes.length);
bbuf.wrap(bytes); // edited due to Jon Skeet's answer
CharBuffer cbuf = bbuf.asCharBuffer();
cbuf.put("t");
FileOutputStream test = new FileOutputStream("somebytes");
test.write(bytes);
test.close();
The problem seems to be that I cannot read the Object structure from a file like that. In a hex-editor furthermore the file "somebytes" contains just a couple or 0s. So it doesn't seem the FileOutputStream puts any of the content ("t" or the byte-equivalent) into it.
[1] http://wiki.python.org/moin/UsingPickle
You've allocated a byte buffer with the size of bytes, but that doesn't mean the byte buffer is associated with the byte array. You can wrap a byte array using ByteBuffer.wrap.
Here's a minimal change to your code which does write the t into the file:
byte[] bytes = new byte[10];
ByteBuffer bbuf = ByteBuffer.wrap(bytes);
CharBuffer cbuf = bbuf.asCharBuffer();
cbuf.put("t");
FileOutputStream test = new FileOutputStream("somebytes");
test.write(bytes);
test.close();
Note that for real code you should use a try/finally block to make sure the file is always closed regardless of exceptions.
However, this is a long way from serialization. Java does have its own binary serialization - see ObjectOutputStream and ObjectInputStream. Personally I usually avoid this sort of serialization however, as it can be very brittle in the face of changes to classes. There are various other approaches to serialization, such as using Thrift or Protocol Buffers for binary serialization, or serializing to XML, JSON or some other human-readable format.
You can
seek(position)
b = read(length)
Then b will be an array of bytes of your length.
Sorry - on rereading you're looking for writing, not reading.
A simpler version of the same thing is.
byte[] bytes = new byte[10];
bytes[1] = 't';
FileOutputStream test = new FileOutputStream("somebytes");
test.write(bytes);
test.close();
There are plenty of examples on how to read/write byte[] from files on the web. try google.
This is a newbie question, I know. Can you guys help?
I'm talking about big files, of course, above 100MB. I'm imagining some kind of loop, but I don't know what to use. Chunked stream?
One thins is for certain: I don't want something like this (pseudocode):
File file = new File(existing_file_path);
byte[] theWholeFile = new byte[file.length()]; //this allocates the whole thing into memory
File out = new File(new_file_path);
out.write(theWholeFile);
To be more specific, I have to re-write a applet that downloads a base64 encoded file and decodes it to the "normal" file. Because it's made with byte arrays, it holds twice the file size in memory: one base64 encoded and the other one decoded. My question is not about base64. It's about saving memory.
Can you point me in the right direction?
Thanks!
From the question, it appears that you are reading the base64 encoded contents of a file into an array, decoding it into another array before finally saving it.
This is a bit of an overhead when considering memory. Especially given the fact that Base64 encoding is in use. It can be made a bit more efficient by:
Reading the contents of the file using a FileInputStream, preferably decorated with a BufferedInputStream.
Decoding on the fly. Base64 encoded characters can be read in groups of 4 characters, to be decoded on the fly.
Writing the output to the file, using a FileOutputStream, again preferably decorated with a BufferedOutputStream. This write operation can also be done after every single decode operation.
The buffering of read and write operations is done to prevent frequent IO access. You could use a buffer size that is appropriate to your application's load; usually the buffer size is chosen to be some power of two, because such a number does not have an "impedance mismatch" with the physical disk buffer.
Perhaps a FileInputStream on the file, reading off fixed length chunks, doing your transformation and writing them to a FileOutputStream?
Perhaps a BufferedReader? Javadoc: http://download-llnw.oracle.com/javase/1.4.2/docs/api/java/io/BufferedReader.html
Use this base64 encoder/decoder, which will wrap your file input stream and handle the decoding on the fly:
InputStream input = new Base64.InputStream(new FileInputStream("in.txt"));
OutputStream output = new FileOutputStream("out.txt");
try {
byte[] buffer = new byte[1024];
int readOffset = 0;
while(input.available() > 0) {
int bytesRead = input.read(buffer, readOffset, buffer.length);
readOffset += bytesRead;
output.write(buffer, 0, bytesRead);
}
} finally {
input.close();
output.close();
}
You can use org.apache.commons.io.FileUtils. This util class provides other options too beside what you are looking for. For example:
FileUtils.copyFile(final File srcFile, final File destFile)
FileUtils.copyFile(final File input, final OutputStream output)
FileUtils.copyFileToDirectory(final File srcFile, final File destDir)
And so on.. Also you can follow this tut.
I'm updating some old code to grab some binary data from a URL instead of from a database (the data is about to be moved out of the database and will be accessible by HTTP instead). The database API seemed to provide the data as a raw byte array directly, and the code in question wrote this array to a file using a BufferedOutputStream.
I'm not at all familiar with Java, but a bit of googling led me to this code:
URL u = new URL("my-url-string");
URLConnection uc = u.openConnection();
uc.connect();
InputStream in = uc.getInputStream();
ByteArrayOutputStream out = new ByteArrayOutputStream();
final int BUF_SIZE = 1 << 8;
byte[] buffer = new byte[BUF_SIZE];
int bytesRead = -1;
while((bytesRead = in.read(buffer)) > -1) {
out.write(buffer, 0, bytesRead);
}
in.close();
fileBytes = out.toByteArray();
That seems to work most of the time, but I have a problem when the data being copied is large - I'm getting an OutOfMemoryError for data items that worked fine with the old code.
I'm guessing that's because this version of the code has multiple copies of the data in memory at the same time, whereas the original code didn't.
Is there a simple way to grab binary data from a URL and save it in a file without incurring the cost of multiple copies in memory?
Instead of writing the data to a byte array and then dumping it to a file, you can directly write it to a file by replacing the following:
ByteArrayOutputStream out = new ByteArrayOutputStream();
With:
FileOutputStream out = new FileOutputStream("filename");
If you do so, there is no need for the call out.toByteArray() at the end. Just make sure you close the FileOutputStream object when done, like this:
out.close();
See the documentation of FileOutputStream for more details.
I don't know what you mean with "large" data, but try using the JVM parameter
java -Xmx 256m ...
which sets the maximum heap size to 256 MByte (or any value you like).
If you need the Content-Length and your web-server is somewhat standard conforming, then it should provide you a "Content-Length" header.
URLConnection#getContentLength() should give you that information upfront so that you are able to create your file. (Be aware that if your HTTP server is misconfigured or under control of an evil entity, that header may not match the number of bytes received. In that case, why dont you stream to a temp-file first and copy that file later?)
In addition to that: A ByteArrayInputStream is a horrible memory allocator. It always doubles the buffer size, so if you read a 32MB + 1 byte file, then you end up with a 64MB buffer. It might be better to implement a own, smarter byte-array-stream, like this one:
http://source.pentaho.org/pentaho-reporting/engines/classic/trunk/core/source/org/pentaho/reporting/engine/classic/core/util/MemoryByteArrayOutputStream.java
subclassing ByteArrayOutputStream gives you access to the buffer and the number of bytes in it.
But of course, if all you want to do is to store de data into a file, you are better off using a FileOutputStream.