Develop Reference

InputStream to GZIPInputStream, and I need to know the size of the GZIPInputStream

I have a MultipartFile and I need to compress inputStream as gzip and sent it, but I need to find a way to compress it and know the compressed size of it
param: MultipartFile file
try(var inputStream = file.getInputStream()) {
var outputStream = new GZIPOutputStream(OutputStream.nullOutputStream());
IOUtils.copyLarge(inputStream, outputStream);
var compressedInputStream = someConvertMerthod(outputStream);
sendCompressed(compressedInputStream, compressedSize)
}
Maybe I can do something like this Java: How do I convert InputStream to GZIPInputStream? but I am not gonna be a able to get the compressedSize
I am not finding an easy way to do it :(

CopyLarge() returns the number of bytes copied. I would assume this is true even if the output is discarded, so all you need is to capture the return value of IOUtils.copyLarge(in,out) and you should be good to go, but this does assume the return value is bytes WRITTEN and not bytes READ, which is not really documented. So it might work!
In general though, you are assuming you can turn the output stream back into an input stream, so nullOutputStream() is not going to be an option. Instead you will be creating a temp file, writing your compressed data to it, and then closing it. At that point you can simply ask the file system API how big it is, that should be iron clad.

hey I think I found the solution :)
param: MultipartFile file
try (InputStream inputStream = file.getInputStream()) {
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream);
inputStream.transferTo(gzipOutputStream);
InputStream compressedInputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
byteArrayOutputStream.size() // is the compressed size
}
Thanks guys!

How can I quickly convert a BufferedImage to a byte[]?

I need to convert a BufferedImage to a byte[], but It is too slow. The byte is eventually base64 encoded and sent to an android client. The Method I have been using is like this:
public static byte[] ImageToBytes(BufferedImage im) throws IOException
{
//make sure its NN
if(im!=null)
{
//create a ByteArrayOutputStream
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//write our image to it
ImageIO.write( im, "png", baos );
//flush the stream
baos.flush();
//get the image in byte form
byte[] imageInByte = baos.toByteArray();
//close the stream
baos.close();
//return our value encoded in base64
return imageInByte;
}
return null;
}
This is far too slow for my program. Changing the png to jpeg makes it fail on the mobile side. The JpegCodec version also fails on the mobile side. By fail I mean the Android method BitmapFactory.decodeByteArray() returns null.

You don't say why it's too slow and there isn't much that can be done to make this code faster because it's the only way to convert a BufferedImage to a PNG byte stream. But here are some pointers:
The class ByteArrayOutputStream is synchronized which costs a lot of performance. You can find a faster implementation in Commons IO (org.apache.commons.io.output.ByteArrayOutputStream)
Depending on the size of your image, the allocation algorithm of ByteArrayOutputStream might be a problem. Start with an initial size of 1 or 10 KiB.
toByteArray() gives you a copy of the existing buffer. If you don't need that (and usually, you don't), writing your own OutputStream might give you an additional speed boost (not to mention avoiding GC runs)

Would not converting to a byte[] be an option? You say "[it] is eventually base64 encoded and sent to an android client". What about wrapping the OutputStream to write to the Android client in something like Base64OutputStream and avoiding the issue entirely?

what is the best way to close a ByteArrayOutputStream?

I need to optimize a application that uses too much heap memory.
I'm having problem in close a ByteArrayOutputStream variable after using the same. I've tried to do using close() but it does not work. this is the code:
ByteArrayOutputStream zipOutTempStream = new ByteArrayOutputStream();
//arquivo.getZipStream() has the XML received by FTP.
//STreamEtils is the function who transfers the XML to zipOutTempStream
StreamUtils.copiarStream(arquivo.getZipStream(), zipOutTempStream);
//Creating a new XML to write over this.
File arquivo1 = new File("C:/XML.xml");
if (arquivo1.exists()) {
System.out.println("ele existe");
} else {
if (arquivo1.createNewFile()) {
System.out.println("arquivo criado");
} else {
System.out.println("arquivo não criado");
}
}
FileOutputStream arquivo2 = new FileOutputStream(arquivo1);
//Copy the unziped XML to the new xml created.
StreamUtils.copiarStream(StreamUtils .uncompressXmlFromZipStream(new ByteArrayInputStream(zipOutTempStream.toByteArray())), arquivo2);
arquivo.setZipStream(null);
arquivo.setXmlStream(null)
return arquivo;

You cannot close a ByteArrayOutputStream, since it's close() method is documented as
Closing a ByteArrayOutputStream has no effect. The methods in this
class can be called after the stream has been closed without
generating an IOException.
This output stream is backed by an array; it is NOT a buffered stream. If you feel it is using too much memory, you should output bytes directly to some endpoint, such as a file or a socket, using an appropriate OutputStream.

I think you are carelessly using too much memory. close() has nothing to do with it. In fact there is no need for closing ByteArrayOutputStream. Here you are copying the ZIP file into wrapped byte[] array:
ByteArrayOutputStream zipOutTempStream = new ByteArrayOutputStream();
StreamUtils.copiarStream(arquivo.getZipStream(), zipOutTempStream);
and few lines later you convert the byte[] array back to InputStream:
StreamUtils.copiarStream(StreamUtils.uncompressXmlFromZipStream(
new ByteArrayInputStream(zipOutTempStream.toByteArray())
), arquivo2);
Seems like this generated byte[] array is pretty huge (confirm with logging). Instead of storing the whole ZIP file in memory (in byte[]) store in a temporary file and read it back.

Byte[] to InputStream or OutputStream

I have a blob column in my database table, for which I have to use byte[] in my Java program as a mapping and to use this data I have to convert it to InputStream or OutputStream. But I don't know what happens internally when I do so. Can anyone briefly explain me what's happening when I do this conversion?

You create and use byte array I/O streams as follows:
byte[] source = ...;
ByteArrayInputStream bis = new ByteArrayInputStream(source);
// read bytes from bis ...
ByteArrayOutputStream bos = new ByteArrayOutputStream();
// write bytes to bos ...
byte[] sink = bos.toByteArray();
Assuming that you are using a JDBC driver that implements the standard JDBC Blob interface (not all do), you can also connect a InputStream or OutputStream to a blob using the getBinaryStream and setBinaryStream methods1, and you can also get and set the bytes directly.
(In general, you should take appropriate steps to handle any exceptions, and close streams. However, closing bis and bos in the example above is unnecessary, since they aren't associated with any external resources; e.g. file descriptors, sockets, database connections.)
1 - The setBinaryStream method is really a getter. Go figure.

I'm assuming you mean that 'use' means read, but what i'll explain for the read case can be basically reversed for the write case.
so you end up with a byte[]. this could represent any kind of data which may need special types of conversions (character, encrypted, etc). let's pretend you want to write this data as is to a file.
firstly you could create a ByteArrayInputStream which is basically a mechanism to supply the bytes to something in sequence.
then you could create a FileOutputStream for the file you want to create. there are many types of InputStreams and OutputStreams for different data sources and destinations.
lastly you would write the InputStream to the OutputStream. in this case, the array of bytes would be sent in sequence to the FileOutputStream for writing. For this i recommend using IOUtils
byte[] bytes = ...;//
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
FileOutputStream out = new FileOutputStream(new File(...));
IOUtils.copy(in, out);
IOUtils.closeQuietly(in);
IOUtils.closeQuietly(out);
and in reverse
FileInputStream in = new FileInputStream(new File(...));
ByteArrayOutputStream out = new ByteArrayOutputStream();
IOUtils.copy(in, out);
IOUtils.closeQuietly(in);
IOUtils.closeQuietly(out);
byte[] bytes = out.toByteArray();
if you use the above code snippets you'll need to handle exceptions and i recommend you do the 'closes' in a finally block.

we can convert byte[] array into input stream by using ByteArrayInputStream
String str = "Welcome to awesome Java World";
byte[] content = str.getBytes();
int size = content.length;
InputStream is = null;
byte[] b = new byte[size];
is = new ByteArrayInputStream(content);
For full example please check here http://www.onlinecodegeek.com/2015/09/how-to-convert-byte-into-inputstream.html

There is no conversion between InputStream/OutputStream and the bytes they are working with. They are made for binary data, and just read (or write) the bytes one by one as is.
A conversion needs to happen when you want to go from byte to char. Then you need to convert using a character set. This happens when you make String or Reader from bytes, which are made for character data.

output = new ByteArrayOutputStream();
...
input = new ByteArrayInputStream( output.toByteArray() )

I do realize that my answer is way late for this question but I think the community would like a newer approach to this issue.

byte[] data = dbEntity.getBlobData();
response.getOutputStream().write();
I think this is better since you already have an existing OutputStream in the response object.
no need to create a new OutputStream.

Copy binary data from URL to file in Java without intermediate copy

I'm updating some old code to grab some binary data from a URL instead of from a database (the data is about to be moved out of the database and will be accessible by HTTP instead). The database API seemed to provide the data as a raw byte array directly, and the code in question wrote this array to a file using a BufferedOutputStream.
I'm not at all familiar with Java, but a bit of googling led me to this code:
URL u = new URL("my-url-string");
URLConnection uc = u.openConnection();
uc.connect();
InputStream in = uc.getInputStream();
ByteArrayOutputStream out = new ByteArrayOutputStream();
final int BUF_SIZE = 1 << 8;
byte[] buffer = new byte[BUF_SIZE];
int bytesRead = -1;
while((bytesRead = in.read(buffer)) > -1) {
out.write(buffer, 0, bytesRead);
}
in.close();
fileBytes = out.toByteArray();
That seems to work most of the time, but I have a problem when the data being copied is large - I'm getting an OutOfMemoryError for data items that worked fine with the old code.
I'm guessing that's because this version of the code has multiple copies of the data in memory at the same time, whereas the original code didn't.
Is there a simple way to grab binary data from a URL and save it in a file without incurring the cost of multiple copies in memory?

Instead of writing the data to a byte array and then dumping it to a file, you can directly write it to a file by replacing the following:
ByteArrayOutputStream out = new ByteArrayOutputStream();
With:
FileOutputStream out = new FileOutputStream("filename");
If you do so, there is no need for the call out.toByteArray() at the end. Just make sure you close the FileOutputStream object when done, like this:
out.close();
See the documentation of FileOutputStream for more details.

I don't know what you mean with "large" data, but try using the JVM parameter
java -Xmx 256m ...
which sets the maximum heap size to 256 MByte (or any value you like).

If you need the Content-Length and your web-server is somewhat standard conforming, then it should provide you a "Content-Length" header.
URLConnection#getContentLength() should give you that information upfront so that you are able to create your file. (Be aware that if your HTTP server is misconfigured or under control of an evil entity, that header may not match the number of bytes received. In that case, why dont you stream to a temp-file first and copy that file later?)
In addition to that: A ByteArrayInputStream is a horrible memory allocator. It always doubles the buffer size, so if you read a 32MB + 1 byte file, then you end up with a 64MB buffer. It might be better to implement a own, smarter byte-array-stream, like this one:
http://source.pentaho.org/pentaho-reporting/engines/classic/trunk/core/source/org/pentaho/reporting/engine/classic/core/util/MemoryByteArrayOutputStream.java

subclassing ByteArrayOutputStream gives you access to the buffer and the number of bytes in it.
But of course, if all you want to do is to store de data into a file, you are better off using a FileOutputStream.