Currently I am using below code to copy zip inside another zip.But when zip file size increases eg:2GB, program is throwing out of memory error.I have increased xmx to 1024, still prfoblem is same.Is thr s any alternate method to handle large files ?
public static void zipFile(File srcFile, File zipFile)
throws FileNotFoundException, IOException {
BufferedInputStream origin = null;
FileOutputStream dest = new FileOutputStream(zipFile);
ZipOutputStream out = new ZipOutputStream(
new BufferedOutputStream(dest));
// out.setMethod(ZipOutputStream.DEFLATED);
byte data[] = new byte[BUFFER];
FileInputStream fi = new FileInputStream(srcFile);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(srcFile.getName());
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
out.close();
}
You are only copying bytes when you come right down to it. You don't need to process either file as a Zip file. Just copy the bytes.
Can you try FileChannel.transferTo method? It's more efficient. Since this is done at OS level, I'm assuming it shouldn't depend on Java Heap size.
If that fails too take a look at this question.
Related
I want to perform file compression using zip package of java.util library. The aim is to limit the compressed file to a fixed size. If the compressed file size is greater than this limit, it should be split into multiple files.
try {
fos = new FileOutputStream(p_request.getOutputFilePath() + zipFileName);
ZipOutputStream zos = new ZipOutputStream(fos);
zipEntry1 = new ZipEntry(f.getName());
fis = new FileInputStream(f.getAbsolutePath());
int count;
while ((count = fis.read(fileRAW, 0, BUFFER)) != -1) {
zipEntry1 = new ZipEntry(f.getName());
if (currentSize >= (p_request.getMaxSizePerFileInMB() * 1024 * 1024)) {
zipSplitCount++;
zos.close();
zos = new ZipOutputStream(new FileOutputStream(
p_request.getOutputFilePath() + zipFileName
+ "_" + zipSplitCount + ".zip"));
currentSize = 0;
}
zos.putNextEntry(zipEntry1);
// zos.closeEntry();
currentSize += zipEntry1.getCompressedSize();
zos.write(fileRAW, 0, count);
}
I always get compressed size as -1. Can someone advise a clean approach for this?
EDIT:
So I compressed the file into chunks of fixed size to get multi parts compressed zip of the same file as f.1.zip, f.2.zip. Now when I decompress it, is there some way to restore the original file? Currently, it says the file must be broken.
byte[] buffer = new byte[BUFFER];
ZipInputStream zis = null;
try {
zis = new ZipInputStream(new FileInputStream(f.getAbsolutePath()));
ZipEntry zipEntry = zis.getNextEntry();
while(zipEntry!=null){
String fileName = zipEntry.getName();
File newFile = new File(p_request.getOutputFilePath() + fileName);
System.out.println("file unzip : "+ newFile.getAbsoluteFile());
new File(newFile.getParent()).mkdirs();
FileOutputStream fos = new FileOutputStream(newFile);
int len;
while ((len = zis.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
fos.close();
zipEntry = zis.getNextEntry();
}
zis.closeEntry();
zis.close();
You are getting -1 because the size is not known until the Zip file has been written to disk. Compression takes place when you save the entire zip file, not when you add a new entry.
This means that you have to either:
write the zip to disk after adding each file and then measuring the zip to determine whether to keep adding to it or creating a new file
or guesstimate the size based on an average compression rate and the size of the file before compression on disk.
I want to read data from lets say 4 zip files called zip1, zip2, zip3, zip4. All of these zip files are split from this 1 big zip file called "BigZip". I want to combine the zip files into one and then compare the bytes if the 1 bigzip file matches the size of bytes with the combined zip file of (zip1+zip2+zip3+zip4). I am getting a very small file size when I combine the size of 4 zip files. What am I doing wrong?
Here is my code for the same:
targetFilePath1, targetFilePath2, targetFilePath3, targetFilePath4 belongs to path of 4 zip files.
sourceFilePath is the path to BigZip file
class Test {
public static void main(String args[]) {
ZipOutputStream outStream = new ZipOutputStream(new FileOutputStream(sourceBigZip));
readZip(sourceFilePath, targetFilePath1);
readZip(sourceFilePath, targetFilePath2);
readZip(sourceFilePath, targetFilePath3);
readZip(sourceFilePath, targetFilePath4);
outStream.close();
}
static void readZip(String sourceBigZip, String targetFile) throws Exception {
ZipInputStream inStream = new ZipInputStream(new FileInputStream(targetFile));
byte[] buffer = new byte[1024];
int len = inStream.read(buffer);
while (len != -1) {
outStream.write(buffer, 0, len);
len = inStream.read(buffer);
System.out.print(len);
}
inStream.close();
}
}
Create ZipOutputStream once and pass it to readZip() method, like:
public static void main(String args[]) {
ZipOutputStream outStream = new ZipOutputStream(new FileOutputStream(sourceFilePath));
readZip(outStream , targetFilePath1);
readZip(outStream , targetFilePath2);
readZip(outStream , targetFilePath3);
readZip(outStream , targetFilePath4);
}
Then you have an error dealing with copying the data from one zip to another...
You need to copy each file in the zip file like this:
static void readZip(ZipOutputStream outStream, String targetFile)
throws Exception {
ZipInputStream inStream = new ZipInputStream(new FileInputStream(
targetFile));
byte[] buffer = new byte[1024];
int len = 0;
for (ZipEntry e; (e = inStream.getNextEntry()) != null;) {
outStream.putNextEntry(e);
while ((len = inStream.read(buffer)) > 0) {
outStream.write(buffer, 0, len);
}
}
inStream.close();
}
}
Every time you call new ZipOutputStream, it creates a new empty file, and wipes out everything you have written to it before.
You have to create the stream outside of readZip, and pass it in to each call rather than creating a new stream every time.
I have a list of files from different locations. I create a zip file using the following the code which works without error. But when I try to unzip the file in Windows using Extract All it fails seeing unable to find any bytes, yet if I double click into the zip file itself with Windows Explorer I can see the files and individual ones can be opened and contains the correct data
ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(zipFile));
for (File next : files)
{
ZipEntry zipEntry = new ZipEntry(next.getName());
zos.putNextEntry(zipEntry);
FileInputStream in = new FileInputStream(next);
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0)
{
zos.write(buf, 0, len);
}
zos.closeEntry();
in.close();
}
zos.close();
This may or may not be related but I've found using fixed byte length can lead to a loss of new line characters.
This may help:
final byte[] newLine = System.getProperty(
"line.separator").getBytes("UTF-8");
while ((line = in.readLine()) != null)
final byte[] buffer = line.getBytes("UTF-8");
out.write(buffer, 0, buffer.length);
out.write(newLine, 0, newLine.length);
}
Getting data onto inputStream object from web url
inputStream = AWSFileUtil.getInputStream(
AWSConnectionUtil.getS3Object(null),
"cdn.generalsentiment.com", filePath);
If they are mutliple files then i want to zip them and sent the filetype as "zip" to struts.xml which does the download.
actually am converting the inputstream into byteArrayInputStream
ByteArrayInputStream byteArrayInputStream = new
ByteArrayInputStream(inputStream.toString().getBytes());
while (byteArrayInputStream.read(inputStream.toString().getBytes()) > 0) {
zipOutputStream.write(inputStream.toString().getBytes());
}
and then
zipOutputStream.close();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
fileInputStream = new FileInputStream(file);
while (fileInputStream.read(buffer) > 0) {
byteArrayOutputStream.write(buffer);
}
byteArrayOutputStream.close();
inputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
reportName = "GS_MediaValue_Reports.zip";
fileType = "zip";
}
return fileType;
But the downloaded zip when extracted gives corrupt files.
Please suggest me a solution for this issue.
The short answer is that it's not how ZipOutputStream works. Since it was designed to store multiple files, along with their file names, directory structures and so on, you need to tell the stream about that explicitly.
Furthermore, converting a stream to a string is a bad idea in general, plus it's slow, especially when you're doing it in a loop.
So your solution will be something like:
ZipEntry entry = new ZipEntry( fileName ); // You have to give each entry a different filename
zipOutputStream.putNextEntry( entry );
byte buffer[] = new byte[ 1024 ]; // 1024 is the buffer size here, but it could be anything really
int count;
while( (count = inputStream.read( buffer, 0, 1024 ) ) != -1 ) {
zipOutputStream.write( buffer, 0, count );
}
My application is unable to transfer data properly over a socket connection and write it to a file properly. Files over about 65,535 bytes get corrupted and are no longer recognized by the programs designed to run them.
I have been able to send small .doc and .txt files successfully, but .mp3 .wmv .m4a .avi and just about anything else does not work. Neither do larger docs.
I have looked all over the internet for a solution to this problem. I have repeatedly tweaked the I/O code to fix the problem but it still doesn't work! Here is the I/O code in the super class that handles sending and receiving files. If you need anymore information/other parts of code, let me know.
protected void sendFile() throws IOException {
byte[] bytes = new byte[(int) file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
int bytesRead = buffin.read(bytes,0,bytes.length);
System.out.println(bytesRead);
out = sock.getOutputStream();
out.write(bytes,0,fileBytes);
out.flush();
out.close();
}
protected void receiveFile() throws IOException {
byte[] bytes = new byte[fileBytes];
in = sock.getInputStream();
for(int i=0;i<fileBytes;i++) {
in.read(bytes);
}
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,fileBytes);
buffout.flush();
buffout.close();
}
UPDATED CODE (that works):
protected void sendFile() throws IOException {
if((file.length())<63000) {
byte[] bytes = new byte[(int)file.length()];
buffin = new BufferedInputStream(new FileInputStream(file));
buffin.read(bytes,0,bytes.length);
out = sock.getOutputStream();
out.write(bytes,0,bytes.length);
out.close();
} else {
byte[] bytes = new byte[32000];
buffin = new BufferedInputStream(new FileInputStream(file));
out = sock.getOutputStream();
int bytesRead;
while((bytesRead = buffin.read(bytes))>0) {
out.write(bytes,0,bytesRead);
}
out.close();
}
}
protected void receiveFile() throws IOException {
if(fileBytes<63000) {
byte[] bytes = new byte[32000];
in = sock.getInputStream();
System.out.println(in.available());
in.read(bytes,0,fileBytes);
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
buffout.write(bytes,0,bytes.length);
buffout.close();
} else {
byte[] bytes = new byte[16000];
in = sock.getInputStream();
fos = new FileOutputStream("/Datawire/"+fileName);
buffout = new BufferedOutputStream(fos);
int bytesRead;
while((bytesRead = in.read(bytes))>0) {
buffout.write(bytes,0,bytesRead);
}
buffout.close();
}
}
The issue is that you are sending only chunks of it. That is, you are only sending 64k of the file ever. If the file is ever larger then 64k the other end will never see it.
You want to continously read from the BufferedInputStream until the read() returns either less then the length or -1.
Your code is completely wrong. This is how to copy a stream in Java:
int count;
byte[] buffer = new byte[8192]; // more if you like but no need for it to be the entire file size
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
You should use this both when sending the file and when receiving the file. At present your sending method hopes that the entire file fits into memory; fits into INTEGER_MAX bytes; and is read in one chunk by the read method, without even checking the result. You can't assume any of those things. Your receive method is complete rubbish: it just keeps overwriting the same array, again without checking any read() results.
EDIT: Your revised code is just as bad, or worse. You are calling read() to check for EOS and then throwing that byte away, and then calling read() again and throwing away the read count it returns. You pointlessly have a different path for files < 64000, or 63000, or whatever it is, that has zero benefit except to give you two code paths to test, or possibly four, instead of one. The network only gives you 1460 bytes at a time at best anyway so what is the point? You already have (a) a BufferedInputStream with a default buffersize of 8192, and (b) my code that uses a byte[] buffer of any size you like. My code above works for any amount of data in two lines of executable code. Yours is 20. QED.
I suggest that you use some good library to read and write file contents as well as socket read/write. For example Apache Commons IO. If you insist on writig code yourself, do it smaller chunks rather than the whole file at once.
You have to consider that InputStream.read returns the number of bytes read which may be less than the total number of bytes in the file.
You would probably be better off just letting something like CopyUtils.copy take care of this for you.
You need to loop until bytesRead < 0. You need to make sure that fileBytes is => than the transferred file.
protected void receiveFile() throws IOException {
byte [] bytes = new byte [fileBytes];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream("/Datawire/"+fileName);
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(bytes,0,bytes.length);
int current = bytesRead;
do {
bytesRead =
is.read(bytes, current, (bytes.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(bytes, 0 , current);
bos.flush();
bos.close();
}