I have a Spring Boot application that converts the XML to JSON. The conversion is done by calling another JAVA application internally and passing some information. The implementation of the called application is out of my control and I do not have access to make any changes to it.
The called Java Application requires a OutputStream so I am using the ByteArrayOutputStream and passing it within the method. After receiving the output I am converting the OutputStream to String. During the conversion, I am running into the warning Inefficient conversion from ByteArrayOutputStream.
I wanted to know how can I fix this warning. I researched a bit and found that we need to pass the size of ByteArrayOutputStream but in my case I am not aware how much size it can have because it would depend on the size of the input XML that I am giving. So I am unable to predict it and set it.
Can someone please guide me on what I can do within my Spring Boot application ByteArrayOutputStream so as to fix the warning that I receive in my Intellij IDE:
Inefficient conversion from ByteArrayOutputStream
Following is my code sample:
final InputStream inputStream = new ByteArrayInputStream(xmlEvents.getBytes(StandardCharsets.UTF_8));
final var output = new ByteArrayOutputStream();
new Converter().convert(inputStream, new Handler<>(new Validator(), new StreamCollector(output)));
return new String(output.toByteArray());
I am getting the warning for the line:
new String(output.toByteArray())
The explanation for this warning is that
new String(output.toByteArray());
creates a byte[] from the contents of the ByteArrayOutputStream, then creates a String from the byte[]. That is doing an unnecessary copy of the data.
The fix suggested by Intellij is:
output.toString(StandardCharsets.UTF_8).
which creates the String in a single operation without creating an intermediate byte[].
How does it do this?
Well toString() is passing the ByteArrayOutputStream's internal byte[] buffer to the String constructor. By contrast, output.toByteArray() is copying the buffer to a new byte[] ... so that the caller cannot interfere with the actual buffer's contents.
Related
I'm planing to use SheetJS with rhino. And sheetjs takes a binary object(BLOB if i'm correct) as it's input. So i need to read a file from the system using stranded java I/O methods and store it into a blob before passing it to sheetjs. eg :-
var XLDataWorkBook = XLSX.read(blobInput, {type : "binary"});
So how can i create a BLOB(or appropriate type) from a binary file in java in order to pass it in.
i guess i cant pass streams because i guess XLSX needs a completely created object to process.
I found the answer to this by myself. i was able to get it done this way.
Read the file with InputStream and then write it to a ByteArrayOutputStream. like below.
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
...
buffer.write(bytes, 0, len);
Then create a byte array from it.
byte[] byteArray = buffer.toByteArray();
Finally i did convert it to a Base64 String (which is also applicable in my case) using the "Base64.encodeBase64String()" method in apache.commons.codec.binary package. So i can pass Base64 String as a method parameter.
If you further need there are lot of libraries(3rd-party and default) available for Base64 to Blob conversion as well.
I have a zip file and after decoding it I get a byte array now I want to create a FileInputStream object with that byte[] object. I dont want to create a file instead pass data content do FileInputStream.
Is there any way ?
following is the code:
byte[] decodedHeaderFileZip = decodeHeaderZipFile(headerExportFile);
FileInputStream fileInputStream = new FileInputStream(decodedHeaderZipFileString);
EDIT:
I wanted to build a ZipInputStream object with a FileInputStream.
I have a zip file and after decoding it I get a byte array now I want to create a FileInputStream object with that byte[] object.
But you don't have a file. You have some data in memory. So a FileInputStream is inappropriate - there's no file for it to read from.
If possible, use a ByteArrayInputStream instead:
InputStream input = new ByteArrayInputStream(decodedHeaderFileZip);
Where possible, express your API in terms of InputStream, Reader etc rather than any specific implementation - that allows you to be flexible in which implementation you use. (What I mean is that where possible, make method parameters and return types InputStream rather than FileInputStream - so that callers don't need to provide the specific types.)
If you absolutely have to create a FileInputStream, you'll need to write the data to a file first.
I need to optimize a application that uses too much heap memory.
I'm having problem in close a ByteArrayOutputStream variable after using the same. I've tried to do using close() but it does not work. this is the code:
ByteArrayOutputStream zipOutTempStream = new ByteArrayOutputStream();
//arquivo.getZipStream() has the XML received by FTP.
//STreamEtils is the function who transfers the XML to zipOutTempStream
StreamUtils.copiarStream(arquivo.getZipStream(), zipOutTempStream);
//Creating a new XML to write over this.
File arquivo1 = new File("C:/XML.xml");
if (arquivo1.exists()) {
System.out.println("ele existe");
} else {
if (arquivo1.createNewFile()) {
System.out.println("arquivo criado");
} else {
System.out.println("arquivo não criado");
}
}
FileOutputStream arquivo2 = new FileOutputStream(arquivo1);
//Copy the unziped XML to the new xml created.
StreamUtils.copiarStream(StreamUtils .uncompressXmlFromZipStream(new ByteArrayInputStream(zipOutTempStream.toByteArray())), arquivo2);
arquivo.setZipStream(null);
arquivo.setXmlStream(null)
return arquivo;
You cannot close a ByteArrayOutputStream, since it's close() method is documented as
Closing a ByteArrayOutputStream has no effect. The methods in this
class can be called after the stream has been closed without
generating an IOException.
This output stream is backed by an array; it is NOT a buffered stream. If you feel it is using too much memory, you should output bytes directly to some endpoint, such as a file or a socket, using an appropriate OutputStream.
I think you are carelessly using too much memory. close() has nothing to do with it. In fact there is no need for closing ByteArrayOutputStream. Here you are copying the ZIP file into wrapped byte[] array:
ByteArrayOutputStream zipOutTempStream = new ByteArrayOutputStream();
StreamUtils.copiarStream(arquivo.getZipStream(), zipOutTempStream);
and few lines later you convert the byte[] array back to InputStream:
StreamUtils.copiarStream(StreamUtils.uncompressXmlFromZipStream(
new ByteArrayInputStream(zipOutTempStream.toByteArray())
), arquivo2);
Seems like this generated byte[] array is pretty huge (confirm with logging). Instead of storing the whole ZIP file in memory (in byte[]) store in a temporary file and read it back.
Calling Simple toBytes() does produce the bytes but exel throws Warning.
Lost Document information
Googling around gave me this link and looking at Javadocs for worksheet and POI HOW-TO say similar things . Basically I can not get Bytes without loosing some information and should use the write method instead.
While write does work fine I really need to send the bytes over . Is there any way I can do that ? That is get the bytes with out getting any warning .
As that mailing list post said
Invoking HSSFWorkbook.getBytes() does not return all of the data necessary to re-
construct a complete Excel file.
You can use the write method with a ByteArrayOutputStream to get at the byte array.
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
workbook.write(bos);
} finally {
bos.close();
}
byte[] bytes = bos.toByteArray();
(The close call is not really needed for a ByteArrayOutputStream, but imho it is good style to include anyway in case its later changed to a different kind of stream.)
How about:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
workbook.write(baos);
byte[] xls = baos.toByteArray();
In order to get a full excel file out, you must call the write(OutputStream) method. If you want bytes from that, just give a ByteArrayOutputStream
I'm updating some old code to grab some binary data from a URL instead of from a database (the data is about to be moved out of the database and will be accessible by HTTP instead). The database API seemed to provide the data as a raw byte array directly, and the code in question wrote this array to a file using a BufferedOutputStream.
I'm not at all familiar with Java, but a bit of googling led me to this code:
URL u = new URL("my-url-string");
URLConnection uc = u.openConnection();
uc.connect();
InputStream in = uc.getInputStream();
ByteArrayOutputStream out = new ByteArrayOutputStream();
final int BUF_SIZE = 1 << 8;
byte[] buffer = new byte[BUF_SIZE];
int bytesRead = -1;
while((bytesRead = in.read(buffer)) > -1) {
out.write(buffer, 0, bytesRead);
}
in.close();
fileBytes = out.toByteArray();
That seems to work most of the time, but I have a problem when the data being copied is large - I'm getting an OutOfMemoryError for data items that worked fine with the old code.
I'm guessing that's because this version of the code has multiple copies of the data in memory at the same time, whereas the original code didn't.
Is there a simple way to grab binary data from a URL and save it in a file without incurring the cost of multiple copies in memory?
Instead of writing the data to a byte array and then dumping it to a file, you can directly write it to a file by replacing the following:
ByteArrayOutputStream out = new ByteArrayOutputStream();
With:
FileOutputStream out = new FileOutputStream("filename");
If you do so, there is no need for the call out.toByteArray() at the end. Just make sure you close the FileOutputStream object when done, like this:
out.close();
See the documentation of FileOutputStream for more details.
I don't know what you mean with "large" data, but try using the JVM parameter
java -Xmx 256m ...
which sets the maximum heap size to 256 MByte (or any value you like).
If you need the Content-Length and your web-server is somewhat standard conforming, then it should provide you a "Content-Length" header.
URLConnection#getContentLength() should give you that information upfront so that you are able to create your file. (Be aware that if your HTTP server is misconfigured or under control of an evil entity, that header may not match the number of bytes received. In that case, why dont you stream to a temp-file first and copy that file later?)
In addition to that: A ByteArrayInputStream is a horrible memory allocator. It always doubles the buffer size, so if you read a 32MB + 1 byte file, then you end up with a 64MB buffer. It might be better to implement a own, smarter byte-array-stream, like this one:
http://source.pentaho.org/pentaho-reporting/engines/classic/trunk/core/source/org/pentaho/reporting/engine/classic/core/util/MemoryByteArrayOutputStream.java
subclassing ByteArrayOutputStream gives you access to the buffer and the number of bytes in it.
But of course, if all you want to do is to store de data into a file, you are better off using a FileOutputStream.