I want to perform file compression using zip package of java.util library. The aim is to limit the compressed file to a fixed size. If the compressed file size is greater than this limit, it should be split into multiple files.
try {
fos = new FileOutputStream(p_request.getOutputFilePath() + zipFileName);
ZipOutputStream zos = new ZipOutputStream(fos);
zipEntry1 = new ZipEntry(f.getName());
fis = new FileInputStream(f.getAbsolutePath());
int count;
while ((count = fis.read(fileRAW, 0, BUFFER)) != -1) {
zipEntry1 = new ZipEntry(f.getName());
if (currentSize >= (p_request.getMaxSizePerFileInMB() * 1024 * 1024)) {
zipSplitCount++;
zos.close();
zos = new ZipOutputStream(new FileOutputStream(
p_request.getOutputFilePath() + zipFileName
+ "_" + zipSplitCount + ".zip"));
currentSize = 0;
}
zos.putNextEntry(zipEntry1);
// zos.closeEntry();
currentSize += zipEntry1.getCompressedSize();
zos.write(fileRAW, 0, count);
}
I always get compressed size as -1. Can someone advise a clean approach for this?
EDIT:
So I compressed the file into chunks of fixed size to get multi parts compressed zip of the same file as f.1.zip, f.2.zip. Now when I decompress it, is there some way to restore the original file? Currently, it says the file must be broken.
byte[] buffer = new byte[BUFFER];
ZipInputStream zis = null;
try {
zis = new ZipInputStream(new FileInputStream(f.getAbsolutePath()));
ZipEntry zipEntry = zis.getNextEntry();
while(zipEntry!=null){
String fileName = zipEntry.getName();
File newFile = new File(p_request.getOutputFilePath() + fileName);
System.out.println("file unzip : "+ newFile.getAbsoluteFile());
new File(newFile.getParent()).mkdirs();
FileOutputStream fos = new FileOutputStream(newFile);
int len;
while ((len = zis.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
fos.close();
zipEntry = zis.getNextEntry();
}
zis.closeEntry();
zis.close();
You are getting -1 because the size is not known until the Zip file has been written to disk. Compression takes place when you save the entire zip file, not when you add a new entry.
This means that you have to either:
write the zip to disk after adding each file and then measuring the zip to determine whether to keep adding to it or creating a new file
or guesstimate the size based on an average compression rate and the size of the file before compression on disk.
Related
I have a requirement to split a 100mb zip file(which will be having sub folders and images) into 10 zip files(each of 10mb).Then I need to send each sliced zip files to an API (as multipart reauest), in receiver API i need to combine each of the above 10 zip files back to origin 100mb zip file.
Below is the code for slicing
public static void splitZip(String zipName, String location, String NewZip) throws IOException{
FileInputStream fis = new FileInputStream(location);
ZipInputStream zipInputStream = new ZipInputStream(fis);
ZipEntry entry = null;
int currentChunkIndex = 0;
long entrySize = 0;
ZipFile zipFile = new ZipFile(location);
Enumeration enumeration = zipFile.entries();
String copDest = zipCopyDest + "\\" + NewZip + "_" + currentChunkIndex +".zip";
FileOutputStream fos = new FileOutputStream(new File(copDest));
BufferedOutputStream bos = new BufferedOutputStream(fos);
ZipOutputStream zos = new ZipOutputStream(bos);
long currentSize = 0;
try {
while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {
ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
System.out.println(zipEntry.getName());
System.out.println(zipEntry.getSize());
entrySize = zipEntry.getSize();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
if((currentSize + entrySize) > MAX_FILE_SIZE) {
zos.close();
currentChunkIndex++;
zos = getOutputStream(currentChunkIndex, NewZip);
currentSize = 0;
}else{
currentSize += entrySize;
zos.putNextEntry(new ZipEntry(entry.getName()));
byte[] buffer = new byte[8192];
int length = 0;
while ((length = zipInputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, length);
}
byte[] unzippedFile = outputStream.toByteArray();
zos.write(unzippedFile);
unzippedFile = null;
outputStream.close();
zos.closeEntry();
}
}
} finally {
zos.close();
}
}
When i extract slices zips manually i found some images are corrupted am not able to open it. Also am not getting a proper solution for combining the zip files. Thanks in advance.
Zip specification has a feature to split zip files to any desired length (minimum split length should be 64kb). Zip4j, supports this feature to create split zip files (documentation). You can then pass each split file to the api. The api can then use the merge functionality in zip4j to merge those split files into a single zip file. On a side note: even without merging, it is a perfectly valid zip file. as long as all the split zip files are in the same directory.
The approach will not work if you have any file in the zip greater than 10 mb and also the else condition should be executed in both the cases and in this approach you have to depend on the size of file in zip may be better you go with approach of not creating smaller zips
I have a list of files from different locations. I create a zip file using the following the code which works without error. But when I try to unzip the file in Windows using Extract All it fails seeing unable to find any bytes, yet if I double click into the zip file itself with Windows Explorer I can see the files and individual ones can be opened and contains the correct data
ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(zipFile));
for (File next : files)
{
ZipEntry zipEntry = new ZipEntry(next.getName());
zos.putNextEntry(zipEntry);
FileInputStream in = new FileInputStream(next);
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0)
{
zos.write(buf, 0, len);
}
zos.closeEntry();
in.close();
}
zos.close();
This may or may not be related but I've found using fixed byte length can lead to a loss of new line characters.
This may help:
final byte[] newLine = System.getProperty(
"line.separator").getBytes("UTF-8");
while ((line = in.readLine()) != null)
final byte[] buffer = line.getBytes("UTF-8");
out.write(buffer, 0, buffer.length);
out.write(newLine, 0, newLine.length);
}
I am sending a request XML to the URL and receiving a zip file to the given path.
Sometimes I'm facing troubles when the bandwidth is low this zip file, most likely 120MB size is not getting downloaded properly. And getting an error when extracting the zip file. Extracting happens from the code as well. When I download in high bandwidth this file gets download without issue.
I'm looking for a solution without making the bandwidth high, from program level are there any ways to download this zip file, may be part by part or something like that? Or anyother solution that you all are having is highly appreciated.
Downloading :
url = new URL(_URL);
sc = (HttpURLConnection) url.openConnection();
sc.setDoInput(true);
sc.setDoOutput(true);
sc.setRequestMethod("POST");
sc.connect();
OutputStream mOstr = sc.getOutputStream();
mOstr.write(request.getBytes());
InputStream in = sc.getInputStream();
FileOutputStream out = new FileOutputStream(path);
int count;
byte[] buffer = new byte[86384];
while ((count = in.read(buffer,0,buffer.length)) > 0)
out.write(buffer, 0, count);
out.close();
Extracting :
try {
ZipFile zipFile = new ZipFile(path+zFile);
Enumeration<?> enu = zipFile.entries();
while (enu.hasMoreElements()) {
ZipEntry zipEntry = (ZipEntry) enu.nextElement();
String name = path+"/data_FILES/"+zipEntry.getName();
long size = zipEntry.getSize();
long compressedSize = zipEntry.getCompressedSize();
System.out.printf("name: %-20s | size: %6d | compressed size: %6d\n", name, size, compressedSize);
File file = new File(name);
if (name.endsWith("/")) {
file.mkdirs();
continue;
}
File parent = file.getParentFile();
if (parent != null) {
parent.mkdirs();
}
InputStream is = zipFile.getInputStream(zipEntry);
FileOutputStream fos = new FileOutputStream(file);
byte[] bytes = new byte[86384];
int length;
while ((length = is.read(bytes)) >= 0) {
fos.write(bytes, 0, length);
}
is.close();
fos.close();
}
zipFile.close();
} catch (Exception e) {
log("Error in extracting zip file ");
e.printStackTrace();
}
Currently I am using below code to copy zip inside another zip.But when zip file size increases eg:2GB, program is throwing out of memory error.I have increased xmx to 1024, still prfoblem is same.Is thr s any alternate method to handle large files ?
public static void zipFile(File srcFile, File zipFile)
throws FileNotFoundException, IOException {
BufferedInputStream origin = null;
FileOutputStream dest = new FileOutputStream(zipFile);
ZipOutputStream out = new ZipOutputStream(
new BufferedOutputStream(dest));
// out.setMethod(ZipOutputStream.DEFLATED);
byte data[] = new byte[BUFFER];
FileInputStream fi = new FileInputStream(srcFile);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(srcFile.getName());
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
out.close();
}
You are only copying bytes when you come right down to it. You don't need to process either file as a Zip file. Just copy the bytes.
Can you try FileChannel.transferTo method? It's more efficient. Since this is done at OS level, I'm assuming it shouldn't depend on Java Heap size.
If that fails too take a look at this question.
how do I translate this code into jython?
ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(file + ".zip"));
byte[] buf = new byte[1024];
int len;
//Create a new Zip entry with the file's name.
ZipEntry zipEntry = new ZipEntry(file.toString());
//Create a buffered input stream out of the file
//we're trying to add into the Zip archive.
FileInputStream fin = new FileInputStream(file);
BufferedInputStream in = new BufferedInputStream(fin);
zos.putNextEntry(zipEntry);
//Read bytes from the file and write into the Zip archive.
while ((len = in.read(buf)) >= 0) {
zos.write(buf, 0, len);
}
//Close the input stream.
in.close();
//Close this entry in the Zip stream.
zos.closeEntry();
this is what I have but it Fails badly
buf=None <<<< ?
len=None <<<< ?
zipEntry=ZipEntry(file.toString())
fin=FileInputStream(file)
bin=BufferedInputStream(fin)
self._zos.putNextEntry(zipEntry)
while (len=bin.helpme_im_dying(buf)) >= 0): <<<< ?
self._zos.write(buf,0,len) <<<< ?
len = bin.read(buf) <<<< ?
bin.close()
self._zos.closeEntry()
refer to this page for information https://www.acm.org/crossroads/xrds6-3/ovp63.html
Here's an exact translation of that function (except, like your case, using bin instead of reserved keyword in).
from jarray import zeros
from java.io import BufferedInputStream, FileInputStream, FileOutputStream
from java.util.zip import ZipEntry, ZipOutputStream
def test(file):
zos = ZipOutputStream(FileOutputStream(file + ".zip"))
buf = zeros(1024, 'b')
zipEntry = ZipEntry(file)
fin = FileInputStream(file)
bin = BufferedInputStream(fin)
zos.putNextEntry(zipEntry)
len = bin.read(buf)
while len >= 0:
zos.write(buf, 0, len)
len = bin.read(buf)
bin.close()
zos.closeEntry()
It is not an answer to your question, but related. Here is a CPython version:
from zipfile import ZipFile, ZIP_DEFLATED
def test(file):
ZipFile(file+".zip", "w", ZIP_DEFLATED).write(file)
Don't use ZipFile without ensuring it is closed:
with ZipFile('spam.zip', 'w') as myzip:
myzip.write('eggs.txt')