Develop Reference

Unable to create zip file from InputStream

I have a requirement to create a zip file from input stream data, and before writing to zip I need to find the checksum for the input stream.
To do that I am using below codes:
private String writeZipFileToFS(List<ResponsePacks> attachmentList) throws IOException
{
File fileToWrite = new File(getZipPath() + "fileName.zip");
try
{
FileUtils.copyInputStreamToFile(compress(attachmentList), fileToWrite);
}
catch (IOException e)
{
throw e;
}
return fileName;
}
private InputStream compress(List<ResponsePacks> attachmentList)
{
byte buffer[] = new byte[2048];
ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
ZipOutputStream zipFileToSend = new ZipOutputStream(byteStream);
try
{
for (ResponsePacks info : attachmentList)
{
// only for successful requests files would need to be added
zipFileToSend.putNextEntry(new ZipEntry(info.getFileName()));
InputStream in = info.getFileContentStream();
getCheckSum(in, info.getFileName());
int length;
while ((length = in.read(buffer)) >= 0)
{
zipFileToSend.write(buffer, 0, length);
}
zipFileToSend.closeEntry();
}
zipFileToSend.close();
}
catch (IOException e)
{
throw e;
}
return new ByteArrayInputStream(byteStream.toByteArray());
}
private static void getCheckSum(InputStream is, String fileName)
{
byte[] dataCopy = null;
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
try
{
IOUtils.copy(is, outputStream);
dataCopy = outputStream.toByteArray();
printLog("Byte Array Size {}", dataCopy.length);
String checkSum = calculateChecksum(dataCopy);
printLog("Checksum for file {} {}", fileName, checkSum);
outputStream.flush();
outputStream.close();
}
catch (Exception e)
{
printLog("Error on calculationg checksum {}", e.getMessage());
}
}
private static String calculateChecksum(byte[] dataCopy)
{
try (ZipInputStream zipInputStream = new ZipInputStream(new ByteArrayInputStream(dataCopy)))
{
ZipEntry zipEntry;
MessageDigest digest = DigestUtils.getSha256Digest();
DWriter writer = new DWriter(digest);
while ((zipEntry = zipInputStream.getNextEntry()) != null)
{
byte[] entityData = IOUtils.toByteArray(zipInputStream);
if (!zipEntry.isDirectory())
{
writer.write(entityData);
}
}
if (writer.getChecksum() != null)
{
return writer.getChecksum();
}
}
catch (Exception e)
{
throw e;
}
return "";
}
static class DWriter
{
private final MessageDigest myDigest;
DWriter(MessageDigest digest)
{
myDigest = digest;
}
public void write(byte[] data)
{
myDigest.update(data);
}
public String getChecksum()
{
return new String(Hex.encodeHex(myDigest.digest()));
}
}
But problem is if I am adding code to calculate the checksum then zip file creating with empty content and if I am removing the checksum calculation code then zip file creating with proper contents.
And also when I check the log I found InputStream contents different contents but still I am getting the same checkSum (empty string) always as below:
Byte Array Size 20854
Checksum for file 20200910173919142.json e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Byte Array Size 14383
Checksum for file 1599752440405.zip e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
I am unable to find where I am doing wrong, due to which zip file is creating with empty content, and checkSum also creating same always.
Requesting to help me to find where I am doing wrong.

You consume twice the same inputstream: first you read it to get the checksum and the you read it again to write the zip entry.
getCheckSum(in, info.getFileName());
int length;
while ((length = in.read(buffer)) >= 0)
{
zipFileToSend.write(buffer, 0, length);
}
The second time you're trying to read, there's nothing to read anymore, so nothing gets written into the zip entry.
Some input streams can be reset and read multiple times, if that's not the case here you would need to save the data into a ByteArrayOutputStream (as you're already doing inside the getCheckSum() method), and then you could read that data multiple times.

Java get Blob from Database and write

I've got a Table where i store my pdf files as blob.
I get the InputStream and insert it like this.
pstmt.setBinaryStream(1, inputStream);
For this I created a Model with Integer ID and InputStream blob; as variables.
I read the blob like this out of my DB.
blob.setBlob(rs.getBinaryStream("blob_file"));
Now I tried to create the PDF file again with this.
byte[] buffer = new byte[4096];
File file= new File("c:\\MyPath\\myPDF.pdf");
try{
FileOutputStream output= new FileOutputStream(file);
int b = 0;
while ((b = blob.getBlob().read()) != -1) {
output.write(buffer);
}
output.close();
}catch(IOException ex){
System.err.println("Blob Error: " + ex.getMessage());
}
With this method I get a corrupt PDF file which I can't open.
I found an alternative which worked very well like this.
IOUtils.copy(blob.getBlob(), output);
But I don't get why my first Version didn't work and what's the difference between These two.

Try this:
FileOutputStream output = null;
InputStream is = blob.getBlob();
try{
output= new FileOutputStream(file);
int b = 0;
while ((b = is.read(buffer)) != -1) {
output.write(buffer, 0, b);
}
} catch(IOException ex){
System.err.println("Blob Error: " + ex.getMessage());
} finally {
is.close();
if (output != null) {
output.close();
}
}
The problem in your initial code is the fact that you don't use the value of b (which is the the total number of bytes read into the buffer) so you probably write more bytes than you should which is probably the cause of the corruption of your file.

How to convert loaded bytes into image without ImageIO.read()

I can't use ImageIO.read() because of my own restrictions. I can only load bytes after GET request and I need to save this bytes to file as image. But it seems to me, that there also loads some extra data, which browser usually filter (maybe response headers). So I get the array of raw bytes which I even can't open as image.
What should I do with this bytes?
Example:
byte[] buf = ContentLoader.loadBytes(new URL("http://images.visitcanberra.com.au/images/canberra_hero_image.jpg"));
try {
FileOutputStream fileOutputStream = new FileOutputStream(new File("D:\\image.jpg"));
fileOutputStream.write(buf);
fileOutputStream.flush();
} catch (IOException e) {
e.printStackTrace();
}
loadBytes() method:
public static byte[] loadBytes(URL url) {
ByteArrayOutputStream boutArray = new ByteArrayOutputStream();
try {
URLConnection connection = url.openConnection();
BufferedInputStream bin = new BufferedInputStream(connection.getInputStream());
byte[] buffer = new byte[1024 * 16];
while (bin.read(buffer) != -1) {
boutArray.write(buffer);
boutArray.flush();
}
bin.close();
} catch (Exception e) {
return null;
}
return boutArray.toByteArray();
}

Usual problems. The standard way to copy a stream in Java is:
int count;
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
out.close();
in.close();
Note that you need to store the result returned by read() into a variable; that you need to use it in the next write() call; that you shouldn't flush() inside a loop; and that you need to close the input and output streams.
And why you're using a ByteArrayInputStream at all is a mystery. It's just a waste of time and space. Read directly from the URL input stream, and write directly to the FileOutputStream.

The following code works for me:-
URL url = new URL("my url...");
InputStream is = url.openStream();
OutputStream os = new FileOutputStream("img.jpg");
byte[] b = new byte[2048];
int length;
while ((length = is.read(b)) != -1) {
os.write(b, 0, length);
}
is.close();
os.close();

How to Convert Binary Data to Zip file?

Right now I am working on a task to convert binary data in to a zip file
I am calling a url and getting a response from server like
A#B�ArE⏾�7�ϫ���f�걺N�����Yg���o_M^�D�T�U X_���e?� hi\ � �ڂ(� �0 rm��'�ed���� �:6h�k�ڗ� ���fnp���7��)��:��N�U�viR�,) II����M��Np�M��7��
n��
!A!) )AAFAq)Q)�y
y� ��.�����?���
��֞��ͅ��Ɲ_�O�����nc��f��w��ʰ�6��3 2�ƢZZ��N0� O{� mC� ��$��,>����������
���CW/)?�?٥��ߗ�d�=�R�J*E{2L���ח�W���ӑ_PRR�_#�_H��:������Ə�Ջ�J�^v�0wo��+�o���
�-Ä#�R6��P�(���0�WPj�k�
C�E
now I want to save this data to zip file i have searched a lot and find some links but not meet the goal.
here i have done
OutputStreamWriter osw = new OutputStreamWriter(openFileOutput(
"products.zip", Context.MODE_PRIVATE));
osw.write(data);
osw.close();
please guid me if you have any idea about this.

OutputStreamWriter osw
NO!
A Writer is made to write text, not binary.
In the first place, it looks like you read text as well, which you shouldn't.
Use an InputStream to read the original content, and an OutputStream to write into the file:
final OutputStream out = /* open your file using a FileOutputStream here */;
final byte[] buf = new byte[8096]; // size as appropriate
// "in" is the InputStream from the socket
int count;
try {
while ((count = in.read(buf)) != -1)
out.write(buf, 0, count);
out.flush();
} finally {
out.close();
}

Readers are not meant to read octet streams.
Reads text from a character-input stream, buffering characters so as to provide for the efficient reading of characters, arrays, and lines.
You're looking for a BufferedInputStream.
The getContent() method on the HttpEntity returns an InputStream. Wrap this around a BufferedInputStream and write it to a file or a ByteArrayOutputStream.
byte[] buffer = new byte[5 * 1024];
int numRead = -1;
while( (numRead = bufferedInputStream.read(buffer))!= -1)
{
byteArrayOutputStream.write(buffer, 0, numRead);
}
byteArrayOutputStream.flush();
byteArrayOutputStream.close();
byte[] result = byteArrayOutputStream.toByteArray();
To save on memory I'd advise you to write to a BufferedOutputStream instead of trying to get the bytes from the stream into a data structure. The android device is likely to run out of memory for large zip files.

You might be able to avoid the trouble of converting Binary by adding options to your request.
In NodeJS I specify the responseType as arraybuffer
const res = await axios.get('/routToThat/file', {
headers: {
Accept: 'application/zip',
},
responseType: 'arraybuffer',
});
So instead of receiving the server response as a Binary string:
A#B�ArE⏾�7�ϫ���f�걺N�����Yg���o_M^�D�T�U X_���e?� hi\...
I get a Buffer:
Buffer(22781691) [80, 75, 3, …]
NOTE: In my case the response I get is already a ZIP file.
More details on this NodeJS answer.
https://stackoverflow.com/a/62460311/3645464

All I need to do is construct a BufferedInputStream from the entity. Just replace BufferedReader with a BufferedInputStream. I would recommend using ISO-8859-1 .An underlying streaming encoder to read binary data is a waste of processing power.
private class methodName extends
AsyncTask<String, Integer, byte[]> {
#Override
protected byte[] doInBackground(String... params) {
String uri = params[0];
try {
MultipartEntityBuilder entity;
File f;
FileBody fb;
entity = MultipartEntityBuilder.create();
entity.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
f = new File(zipImageFile);
fb = new FileBody(f);
entity.addPart("orderFile", fb);
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(uri);
Log.e("Uploload Missing Image URL", "" + uri);
httppost.setEntity(entity.build());
HttpResponse response = httpclient.execute(httppost);
// byte[] fileBites=null;
BufferedInputStream bufferedInputStream;
ByteArrayOutputStream byteArrayOutputStream;
byte[] buffer = new byte[5 * 1024];
int numRead = -1;
while( (numRead = bufferedInputStream.read(buffer))!= -1)
{
byteArrayOutputStream.write(buffer, 0, numRead);
}
byteArrayOutputStream.flush();
byteArrayOutputStream.close();
byte[] result = byteArrayOutputStream.toByteArray();
// fileBites=stringBuffer.toString().getBytes();
// Log.e("FILE BITES", fileBites+"=>"+fileBites.length);
return ;
// return stringBuffer.toString();
} catch (Exception e) {
return e.toString().getBytes();
}
}
#Override
protected void onPostExecute(byte[] result) {
// TODO Auto-generated method stub
super.onPostExecute(result);
Log.e("Response From Server", "" + result);
writeToFile(result);
}
}
private void writeToFile(byte[] data) {
try {
FileOutputStream fop = null;
File file;
file = new File(AppConstants.DataPath+"/products.zip");
fop = new FileOutputStream(file);
// if file doesnt exists, then create it
if (!file.exists()) {
file.createNewFile();
}
try {
fop.write(data);
} catch (IOException e) {
Log.e("Exception", "File write failed: " + e.toString());
}
unzipImage(AppConstants.DataPath + "/products.zip",
AppConstants.DataPath);
}catch (Exception E)
{
}
}

OutOfMemory when creating Base64 string in java?

I used ostermillerutils library to create base64 string but I get OutOfMemory error if the image is heavy. If the image I try to convert is a simple image, the code is working fine.
public String createBase64String(InputStream in) {
//collect = new ByteArrayOutputStream();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
try {
for(int readNum; (readNum = in.read(buf)) != -1; ) {
bos.write(buf, 0, readNum);
}
}
catch (IOException ex) {
Logger.getInstance().debug("XML createBase64String: IOException");
return null;
}
finally {
if (in != null) {
try {
in.close();
}
catch (IOException ex) {
;
}
}
}
byte[] ba = bos.toByteArray();
String coded = Base64.encodeToString(ba);
return coded;
}
I also tried doing this but the base64 was incorrect when I tried to decode it.
public void createBase64String(InputStream in) throws IOException {
//collect = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int readNum = 0;
try {
while((readNum = in.read(buf)) != -1)
{
smtp.addBase64(Base64.encodeBase64String(buf));
}
}
catch (IOException ex) {
Logger.getInstance().debug("XML createBase64String: IOException");
}
finally {
if (in != null) {
in.close();
}
}
}
Please suggest solutions for JDK 1.4 and also for later versions of Java.

If you like to write the encoded content straight into a file then use the following code
public void encode(File file, OutputStream base64OutputStream) {
InputStream is = new FileInputStream(file);
OutputStream out = new Base64OutputStream(base64OutputStream)
IOUtils.copy(is, out);
is.close();
out.close();
}
IOUtils class from Apache Commons IO.
EDIT
Since you want to do it using BufferedWriter, use it as follows
OutputStream out = Base64OutputStream(smtpSocket.getOutputStream());
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(out));
IOUtils.copy(is, bw);

It sounds like the problem is that you're having to manipulate too much data in memory when you read the entire image. One fix would be to increase the Java heap size until you have enough memory, but that would just be avoiding the problem rather than solving it.
A better option would be to look at a streaming implementation of a Base64 encoder. This would mean you're only working on a subset of the image at any time. I believe that Base64OutputStream from Apache Commons would do the job for you.

I've fixed my problem by using javabase64-1.3.1.jar library.
OutputStream fos2 = FileUtil.getOutputStream(base64FileName, FileUtil.HDD);
InputStream in2 = FileUtil.getInputStream(fileName, FileUtil.HDD);
Base64.encode(in2, fos2);
in2.close();
fos2.close();
I stored the base64 string to a text file first.
public void createBase64String(InputStream in) throws IOException {
baos = new ByteArrayOutputStream();
byte[] buf = new byte[BUFFER_SIZE];
int readNum = 0;
smtp.addBase64("\t\t");
try {
while ((readNum = in.read(buf)) >= 0) {
baos.write(buf, 0, readNum);
smtp.addBase64(baos.toString());
baos.reset();
}
}
catch (IOException ex) {
LogUtil.error("Sending of Base64 String to SMTP: IOException: " + ex);
}
finally {
if (in != null) {
in.close();
baos.close();
}
}
baos = null;
buf = null;
}
then send each line to smtp's socket outputstream.

From Java 8 onwards, there is a simple way to implement base64 encoding in an output stream with one line of code and no external dependencies:
import java.util.Base64;
OutputStream os = ...
OutputStream base64 = Base64.getEncoder().wrap(os);
Base64 also provides other flavors of base64 encoder; see javadocs:
Base64
Base64.Encoder.wrap