I need to optimize a application that uses too much heap memory.
I'm having problem in close a ByteArrayOutputStream variable after using the same. I've tried to do using close() but it does not work. this is the code:
ByteArrayOutputStream zipOutTempStream = new ByteArrayOutputStream();
//arquivo.getZipStream() has the XML received by FTP.
//STreamEtils is the function who transfers the XML to zipOutTempStream
StreamUtils.copiarStream(arquivo.getZipStream(), zipOutTempStream);
//Creating a new XML to write over this.
File arquivo1 = new File("C:/XML.xml");
if (arquivo1.exists()) {
System.out.println("ele existe");
} else {
if (arquivo1.createNewFile()) {
System.out.println("arquivo criado");
} else {
System.out.println("arquivo não criado");
}
}
FileOutputStream arquivo2 = new FileOutputStream(arquivo1);
//Copy the unziped XML to the new xml created.
StreamUtils.copiarStream(StreamUtils .uncompressXmlFromZipStream(new ByteArrayInputStream(zipOutTempStream.toByteArray())), arquivo2);
arquivo.setZipStream(null);
arquivo.setXmlStream(null)
return arquivo;
You cannot close a ByteArrayOutputStream, since it's close() method is documented as
Closing a ByteArrayOutputStream has no effect. The methods in this
class can be called after the stream has been closed without
generating an IOException.
This output stream is backed by an array; it is NOT a buffered stream. If you feel it is using too much memory, you should output bytes directly to some endpoint, such as a file or a socket, using an appropriate OutputStream.
I think you are carelessly using too much memory. close() has nothing to do with it. In fact there is no need for closing ByteArrayOutputStream. Here you are copying the ZIP file into wrapped byte[] array:
ByteArrayOutputStream zipOutTempStream = new ByteArrayOutputStream();
StreamUtils.copiarStream(arquivo.getZipStream(), zipOutTempStream);
and few lines later you convert the byte[] array back to InputStream:
StreamUtils.copiarStream(StreamUtils.uncompressXmlFromZipStream(
new ByteArrayInputStream(zipOutTempStream.toByteArray())
), arquivo2);
Seems like this generated byte[] array is pretty huge (confirm with logging). Instead of storing the whole ZIP file in memory (in byte[]) store in a temporary file and read it back.
Related
I have a MultipartFile and I need to compress inputStream as gzip and sent it, but I need to find a way to compress it and know the compressed size of it
param: MultipartFile file
try(var inputStream = file.getInputStream()) {
var outputStream = new GZIPOutputStream(OutputStream.nullOutputStream());
IOUtils.copyLarge(inputStream, outputStream);
var compressedInputStream = someConvertMerthod(outputStream);
sendCompressed(compressedInputStream, compressedSize)
}
Maybe I can do something like this Java: How do I convert InputStream to GZIPInputStream? but I am not gonna be a able to get the compressedSize
I am not finding an easy way to do it :(
CopyLarge() returns the number of bytes copied. I would assume this is true even if the output is discarded, so all you need is to capture the return value of IOUtils.copyLarge(in,out) and you should be good to go, but this does assume the return value is bytes WRITTEN and not bytes READ, which is not really documented. So it might work!
In general though, you are assuming you can turn the output stream back into an input stream, so nullOutputStream() is not going to be an option. Instead you will be creating a temp file, writing your compressed data to it, and then closing it. At that point you can simply ask the file system API how big it is, that should be iron clad.
hey I think I found the solution :)
param: MultipartFile file
try (InputStream inputStream = file.getInputStream()) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream);
inputStream.transferTo(gzipOutputStream);
InputStream compressedInputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
byteArrayOutputStream.size() // is the compressed size
}
Thanks guys!
Calling Simple toBytes() does produce the bytes but exel throws Warning.
Lost Document information
Googling around gave me this link and looking at Javadocs for worksheet and POI HOW-TO say similar things . Basically I can not get Bytes without loosing some information and should use the write method instead.
While write does work fine I really need to send the bytes over . Is there any way I can do that ? That is get the bytes with out getting any warning .
As that mailing list post said
Invoking HSSFWorkbook.getBytes() does not return all of the data necessary to re-
construct a complete Excel file.
You can use the write method with a ByteArrayOutputStream to get at the byte array.
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
workbook.write(bos);
} finally {
bos.close();
}
byte[] bytes = bos.toByteArray();
(The close call is not really needed for a ByteArrayOutputStream, but imho it is good style to include anyway in case its later changed to a different kind of stream.)
How about:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
workbook.write(baos);
byte[] xls = baos.toByteArray();
In order to get a full excel file out, you must call the write(OutputStream) method. If you want bytes from that, just give a ByteArrayOutputStream
I'm calling onto some code that returns me an HTTP response. I can get the contents of the response which returns me a byte array. The bytes represent a zip file that I would like to extract and get the contents of a single file (the zip only contains one file).
Currently I have some messy code (I'll need to clean it up if I keep it) that seems to work:
byte[] bytes = response.out.toByteArray();
ByteArrayInputStream input = new ByteArrayInputStream(bytes);
ZipInputStream zip = new ZipInputStream(input);
ByteArrayOutputStream output = new ByteArrayOutputStream();
zip.getNextEntry();
int data;
while ((data = zip.read()) != -1) output.write(data);
output.close();
zip.close();
input.close();
byte[] kmlBytes = output.toByteArray();
String contents = new String(kmlBytes, "UTF-8");
but was wondering whether there was a cleaner way to do the same thing because the above looks incredibly ugly.
In order to avoid "reinventing the wheel" and reduce clutter and to focus on your app's business logic rather than low-level code like this, you can use Apache Commons Compress.
This is a newbie question, I know. Can you guys help?
I'm talking about big files, of course, above 100MB. I'm imagining some kind of loop, but I don't know what to use. Chunked stream?
One thins is for certain: I don't want something like this (pseudocode):
File file = new File(existing_file_path);
byte[] theWholeFile = new byte[file.length()]; //this allocates the whole thing into memory
File out = new File(new_file_path);
out.write(theWholeFile);
To be more specific, I have to re-write a applet that downloads a base64 encoded file and decodes it to the "normal" file. Because it's made with byte arrays, it holds twice the file size in memory: one base64 encoded and the other one decoded. My question is not about base64. It's about saving memory.
Can you point me in the right direction?
Thanks!
From the question, it appears that you are reading the base64 encoded contents of a file into an array, decoding it into another array before finally saving it.
This is a bit of an overhead when considering memory. Especially given the fact that Base64 encoding is in use. It can be made a bit more efficient by:
Reading the contents of the file using a FileInputStream, preferably decorated with a BufferedInputStream.
Decoding on the fly. Base64 encoded characters can be read in groups of 4 characters, to be decoded on the fly.
Writing the output to the file, using a FileOutputStream, again preferably decorated with a BufferedOutputStream. This write operation can also be done after every single decode operation.
The buffering of read and write operations is done to prevent frequent IO access. You could use a buffer size that is appropriate to your application's load; usually the buffer size is chosen to be some power of two, because such a number does not have an "impedance mismatch" with the physical disk buffer.
Perhaps a FileInputStream on the file, reading off fixed length chunks, doing your transformation and writing them to a FileOutputStream?
Perhaps a BufferedReader? Javadoc: http://download-llnw.oracle.com/javase/1.4.2/docs/api/java/io/BufferedReader.html
Use this base64 encoder/decoder, which will wrap your file input stream and handle the decoding on the fly:
InputStream input = new Base64.InputStream(new FileInputStream("in.txt"));
OutputStream output = new FileOutputStream("out.txt");
try {
byte[] buffer = new byte[1024];
int readOffset = 0;
while(input.available() > 0) {
int bytesRead = input.read(buffer, readOffset, buffer.length);
readOffset += bytesRead;
output.write(buffer, 0, bytesRead);
}
} finally {
input.close();
output.close();
}
You can use org.apache.commons.io.FileUtils. This util class provides other options too beside what you are looking for. For example:
FileUtils.copyFile(final File srcFile, final File destFile)
FileUtils.copyFile(final File input, final OutputStream output)
FileUtils.copyFileToDirectory(final File srcFile, final File destDir)
And so on.. Also you can follow this tut.
This problem seems to happen inconsistently. We are using a java applet to download a file from our site, which we store temporarily on the client's machine.
Here is the code that we are using to save the file:
URL targetUrl = new URL(urlForFile);
InputStream content = (InputStream)targetUrl.getContent();
BufferedInputStream buffered = new BufferedInputStream(content);
File savedFile = File.createTempFile("temp",".dat");
FileOutputStream fos = new FileOutputStream(savedFile);
int letter;
while((letter = buffered.read()) != -1)
fos.write(letter);
fos.close();
Later, I try to access that file by using:
ObjectInputStream keyInStream = new ObjectInputStream(new FileInputStream(savedFile));
Most of the time it works without a problem, but every once in a while we get the error:
java.io.StreamCorruptedException: invalid stream header: 0D0A0D0A
which makes me believe that it isn't saving the file correctly.
I'm guessing that the operations you've done with getContent and BufferedInputStream have treated the file like an ascii file which has converted newlines or carriage returns into carriage return + newline (0x0d0a), which has confused ObjectInputStream (which expects serialized data objects.
If you are using an FTP URL, the transfer may be occurring in ASCII mode.
Try appending ";type=I" to the end of your URL.
Why are you using ObjectInputStream to read it?
As per the javadoc:
An ObjectInputStream deserializes primitive data and objects previously written using an ObjectOutputStream.
Probably the error comes from the fact you didn't write it with ObjectOutputStream.
Try reading it wit FileInputStream only.
Here's a sample for binary ( although not the most efficient way )
Here's another used for text files.
There are 3 big problems in your sample code:
You're not just treating the input as bytes
You're needlessly pulling the entire object into memory at once
You're doing multiple method calls for every single byte read and written -- use the array based read/write!
Here's a redo:
URL targetUrl = new URL(urlForFile);
InputStream is = targetUrl.getInputStream();
File savedFile = File.createTempFile("temp",".dat");
FileOutputStream fos = new FileOutputStream(savedFile);
int count;
byte[] buff = new byte[16 * 1024];
while((count = is.read(buff)) != -1) {
fos.write(buff, 0, count);
}
fos.close();
content.close();
You could also step back from the code and check to see if the file on your client is the same as the file on the server. If you get both files on an XP machine, you should be able to use the FC utility to do a compare (check FC's help if you need to run this as a binary compare as there is a switch for that). If you're on Unix, I don't know the file compare program, but I'm sure there's something.
If the files are identical, then you're looking at a problem with the code that reads the file.
If the files are not identical, focus on the code that writes your file.
Good luck!