I need to write an algorithm in java (for an android app) to read a folder containing more folders and each of those containing images and audio files so the structure is this: mainDir/categorySubfolder/myFile1.jpg
My problem is that I need to limit the size of the archive to 16mb and at runtime, create as many archives as needed to contain all my files from my main mainDir folder.
I tried several examples from the net and I read the java documentation but I can't manage to understand and put it all together the way I need it. Has someone done this before or has a link or an example for me?
I resolved the reading of the files with a recursive method but I can't write the logic for the zip creation.
I'm open for suggestions or better a working example.
zip4j is a great library that can create multi-part zip files.
net.lingala.zip4j.core.ZipFile zipFile = new ZipFile("out.zip");
ZipParameters parameters = new ZipParameters();
parameters.setCompressionMethod(Zip4jConstants.COMP_DEFLATE);
parameters.setCompressionLevel(Zip4jConstants.DEFLATE_LEVEL_NORMAL);
zipFile.createZipFileFromFolder("path/to/source/dir", parameters, true, maximum size);
You can find more examples on their web-site.
I am using below code/class to split and zip a large amount/size of files.
I have tested this class on below
number of uncompressed files : 116
total size (uncompressed) : 29.1 GB
ZIP file size limit (each) : 3 GB [MAX_ZIP_SIZE]
total size (compressed) : 7.85 GB
number of ZIP file (splited as MAX_ZIP_SIZE): 3
you have to change the value of MAX_ZIP_SIZE to 16(MB)10241024=16777216-22(zip header size)=16777194.
In my code, MAX_ZIP_SIZE set to 3 GB (ZIP has limitation of 4GB on various things).
final long MAX_ZIP_SIZE = 3221225472L; //3 GB
package com.company;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class QdeZip {
public static String createZIP(String directoryPath, String zipFileName, String filesToZip) {
try {
final int BUFFER = 104857600; // 100MB
final long MAX_ZIP_SIZE = 3221225472L; //3 GB
long currentSize = 0;
int zipSplitCount = 0;
String files[] = filesToZip.split(",");
if (!directoryPath.endsWith("/")) {
directoryPath = directoryPath + "/";
}
byte fileRAW[] = new byte[BUFFER];
ZipOutputStream zipOut = new ZipOutputStream(new FileOutputStream(directoryPath + zipFileName.toUpperCase()));
ZipEntry zipEntry;
FileInputStream entryFile;
for (String aFile : files) {
zipEntry = new ZipEntry(aFile);
if (currentSize >= MAX_ZIP_SIZE) {
zipSplitCount++;
//zipOut.closeEntry();
zipOut.close();
zipOut = new ZipOutputStream(new FileOutputStream(directoryPath + zipFileName.toLowerCase().replace(".zip", "_" + zipSplitCount + ".zip").toUpperCase()));
currentSize = 0;
}
zipOut.putNextEntry(zipEntry);
entryFile = new FileInputStream(directoryPath + aFile);
int count;
while ((count = entryFile.read(fileRAW, 0, BUFFER)) != -1) {
zipOut.write(fileRAW, 0, count);
//System.out.println("number of Bytes read = " + count);
}
entryFile.close();
zipOut.closeEntry();
currentSize += zipEntry.getCompressedSize();
}
zipOut.close();
//System.out.println(directory + " -" + zipFileName + " -Number of Files = " + files.length);
} catch (FileNotFoundException e) {
return "FileNotFoundException = " + e.getMessage();
} catch (IOException e) {
return "IOException = " + e.getMessage();
} catch (Exception e) {
return "Exception = " + e.getMessage();
}
return "1";
}
}
I have returned all Exception Messages as String to work with it. this
my own case related to project.
As far as I can see How to split a huge zip file into multiple volumes? just suggests keeping track of the archive size so far, and when it approaches some arbitrary value (which should be lower than your max), it'll decide to start a new file. So for a 16MB limit, you could set the value to 10MB and start a new zip whenever this is reached, but if you reach 9MB and your next file zips down to 8MB, you'll end up with a zip bigger than your limit.
The code given in that post didn't seem to work for me because 1) it got the size before the ZipEntry was created, so it was always 0 and 2) it didn't write out any zip :-) If I've got that wrong - let me know.
The following works for me. For simplicity, I've taken it out of the Wrapper and just have it all in main(String args[]). There are many, many ways this code could be improved :-)
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.Deflater;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class ChunkedZipTwo {
static final long MAX_LIMIT = 10 * 1000 * 1024; //10MB limit - hopefully this
public static void main(String[] args) throws Exception {
String[] files = {"file1", "file2", "file3"};
int i = 0;
boolean needNewFile = false;
long overallSize = 0;
ZipOutputStream out = getOutputStream(i);
byte[] buffer = new byte[1024];
for (String thisFileName : files) {
if (overallSize > MAX_LIMIT) {
out.close();
i++;
out = getOutputStream(i);
overallSize = 0;
}
FileInputStream in = new FileInputStream(thisFileName);
ZipEntry ze = new ZipEntry(thisFileName);
out.putNextEntry(ze);
int len;
while ((len = in.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
out.closeEntry();
in.close();
overallSize += ze.getCompressedSize();
}
out.close();
}
public static ZipOutputStream getOutputStream(int i) throws IOException {
ZipOutputStream out = new ZipOutputStream(new FileOutputStream("bigfile" + i + ".zip"));
out.setLevel(Deflater.DEFAULT_COMPRESSION);
return out;
}
}
I am currently extracting the contents of a war file and then adding some new files to the directory structure and then creating a new war file.
This is all done programatically from Java - but I am wondering if it wouldn't be more efficient to copy the war file and then just append the files - then I wouldn't have to wait so long as the war expands and then has to be compressed again.
I can't seem to find a way to do this in the documentation though or any online examples.
Anyone can give some tips or pointers?
UPDATE:
TrueZip as mentioned in one of the answers seems to be a very good java library to append to a zip file (despite other answers that say it is not possible to do this).
Anyone have experience or feedback on TrueZip or can recommend other similar libaries?
In Java 7 we got Zip File System that allows adding and changing files in zip (jar, war) without manual repackaging.
We can directly write to files inside zip files as in the following example.
Map<String, String> env = new HashMap<>();
env.put("create", "true");
Path path = Paths.get("test.zip");
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env))
{
Path nf = fs.getPath("new.txt");
try (Writer writer = Files.newBufferedWriter(nf, StandardCharsets.UTF_8, StandardOpenOption.CREATE)) {
writer.write("hello");
}
}
As others mentioned, it's not possible to append content to an existing zip (or war). However, it's possible to create a new zip on the fly without temporarily writing extracted content to disk. It's hard to guess how much faster this will be, but it's the fastest you can get (at least as far as I know) with standard Java. As mentioned by Carlos Tasada, SevenZipJBindings might squeeze out you some extra seconds, but porting this approach to SevenZipJBindings will still be faster than using temporary files with the same library.
Here's some code that writes the contents of an existing zip (war.zip) and appends an extra file (answer.txt) to a new zip (append.zip). All it takes is Java 5 or later, no extra libraries needed.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;
public class Main {
// 4MB buffer
private static final byte[] BUFFER = new byte[4096 * 1024];
/**
* copy input to output stream - available in several StreamUtils or Streams classes
*/
public static void copy(InputStream input, OutputStream output) throws IOException {
int bytesRead;
while ((bytesRead = input.read(BUFFER))!= -1) {
output.write(BUFFER, 0, bytesRead);
}
}
public static void main(String[] args) throws Exception {
// read war.zip and write to append.zip
ZipFile war = new ZipFile("war.zip");
ZipOutputStream append = new ZipOutputStream(new FileOutputStream("append.zip"));
// first, copy contents from existing war
Enumeration<? extends ZipEntry> entries = war.entries();
while (entries.hasMoreElements()) {
ZipEntry e = entries.nextElement();
System.out.println("copy: " + e.getName());
append.putNextEntry(e);
if (!e.isDirectory()) {
copy(war.getInputStream(e), append);
}
append.closeEntry();
}
// now append some extra content
ZipEntry e = new ZipEntry("answer.txt");
System.out.println("append: " + e.getName());
append.putNextEntry(e);
append.write("42\n".getBytes());
append.closeEntry();
// close
war.close();
append.close();
}
}
I had a similar requirement sometime back - but it was for reading and writing zip archives (.war format should be similar). I tried doing it with the existing Java Zip streams but found the writing part cumbersome - especially when directories where involved.
I'll recommend you to try out the TrueZIP (open source - apache style licensed) library that exposes any archive as a virtual file system into which you can read and write like a normal filesystem. It worked like a charm for me and greatly simplified my development.
You could use this bit of code I wrote
public static void addFilesToZip(File source, File[] files)
{
try
{
File tmpZip = File.createTempFile(source.getName(), null);
tmpZip.delete();
if(!source.renameTo(tmpZip))
{
throw new Exception("Could not make temp file (" + source.getName() + ")");
}
byte[] buffer = new byte[1024];
ZipInputStream zin = new ZipInputStream(new FileInputStream(tmpZip));
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(source));
for(int i = 0; i < files.length; i++)
{
InputStream in = new FileInputStream(files[i]);
out.putNextEntry(new ZipEntry(files[i].getName()));
for(int read = in.read(buffer); read > -1; read = in.read(buffer))
{
out.write(buffer, 0, read);
}
out.closeEntry();
in.close();
}
for(ZipEntry ze = zin.getNextEntry(); ze != null; ze = zin.getNextEntry())
{
out.putNextEntry(ze);
for(int read = zin.read(buffer); read > -1; read = zin.read(buffer))
{
out.write(buffer, 0, read);
}
out.closeEntry();
}
out.close();
tmpZip.delete();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I don't know of a Java library that does what you describe. But what you described is practical. You can do it in .NET, using DotNetZip.
Michael Krauklis is correct that you cannot simply "append" data to a war file or zip file, but it is not because there is an "end of file" indication, strictly speaking, in a war file. It is because the war (zip) format includes a directory, which is normally present at the end of the file, that contains metadata for the various entries in the war file. Naively appending to a war file results in no update to the directory, and so you just have a war file with junk appended to it.
What's necessary is an intelligent class that understands the format, and can read+update a war file or zip file, including the directory as appropriate. DotNetZip does this, without uncompressing/recompressing the unchanged entries, just as you described or desired.
As Cheeso says, there's no way of doing it. AFAIK the zip front-ends are doing exactly the same as you internally.
Anyway if you're worried about the speed of extracting/compressing everything, you may want to try the SevenZipJBindings library.
I covered this library in my blog some months ago (sorry for the auto-promotion). Just as an example, extracting a 104MB zip file using the java.util.zip took me 12 seconds, while using this library took 4 seconds.
In both links you can find examples about how to use it.
Hope it helps.
See this bug report.
Using append mode on any kind of
structured data like zip files or tar
files is not something you can really
expect to work. These file formats
have an intrinsic "end of file"
indication built into the data format.
If you really want to skip the intermediate step of un-waring/re-waring, you could read the war file file, get all the zip entries, then write to a new war file "appending" the new entries you wanted to add. Not perfect, but at least a more automated solution.
Yet Another Solution: You may find code below useful in other situations as well. I have used ant this way to compile Java directories, generating jar files, updating zip files,...
public static void antUpdateZip(String zipFilePath, String libsToAddDir) {
Project p = new Project();
p.init();
Target target = new Target();
target.setName("zip");
Zip task = new Zip();
task.init();
task.setDestFile(new File(zipFilePath));
ZipFileSet zipFileSet = new ZipFileSet();
zipFileSet.setPrefix("WEB-INF/lib");
zipFileSet.setDir(new File(libsToAddDir));
task.addFileset(zipFileSet);
task.setUpdate(true);
task.setProject(p);
task.init();
target.addTask(task);
target.setProject(p);
p.addTarget(target);
DefaultLogger consoleLogger = new DefaultLogger();
consoleLogger.setErrorPrintStream(System.err);
consoleLogger.setOutputPrintStream(System.out);
consoleLogger.setMessageOutputLevel(Project.MSG_DEBUG);
p.addBuildListener(consoleLogger);
try {
// p.fireBuildStarted();
// ProjectHelper helper = ProjectHelper.getProjectHelper();
// p.addReference("ant.projectHelper", helper);
// helper.parse(p, buildFile);
p.executeTarget(target.getName());
// p.fireBuildFinished(null);
} catch (BuildException e) {
p.fireBuildFinished(e);
throw new AssertionError(e);
}
}
this a simple code to get a response with using servlet and send a response
myZipPath = bla bla...
byte[] buf = new byte[8192];
String zipName = "myZip.zip";
String zipPath = myzippath+ File.separator+"pdf" + File.separator+ zipName;
File pdfFile = new File("myPdf.pdf");
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipPath));
ZipEntry zipEntry = new ZipEntry(pdfFile.getName());
out.putNextEntry(zipEntry);
InputStream in = new FileInputStream(pdfFile);
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
out.closeEntry();
in.close();
out.close();
FileInputStream fis = new FileInputStream(zipPath);
response.setContentType("application/zip");
response.addHeader("content-disposition", "attachment;filename=" + zipName);
OutputStream os = response.getOutputStream();
int length = is.read(buffer);
while (length != -1)
{
os.write(buffer, 0, length);
length = is.read(buffer);
}
Here are examples how easily files can be appended to existing zip using TrueVFS:
// append a file to archive under different name
TFile.cp(new File("existingFile.txt"), new TFile("archive.zip", "entry.txt"));
// recusively append a dir to the root of archive
TFile src = new TFile("dirPath", "dirName");
src.cp_r(new TFile("archive.zip", src.getName()));
TrueVFS, the successor of TrueZIP, uses Java 7 NIO 2 features under the hood when appropriate but offers much more features like thread-safe async parallel compression.
Beware also that Java 7 ZipFileSystem by default is vulnerable to OutOfMemoryError on huge inputs.
Here is Java 1.7 version of Liam answer which uses try with resources and Apache Commons IO.
The output is written to a new zip file but it can be easily modified to write to the original file.
/**
* Modifies, adds or deletes file(s) from a existing zip file.
*
* #param zipFile the original zip file
* #param newZipFile the destination zip file
* #param filesToAddOrOverwrite the names of the files to add or modify from the original file
* #param filesToAddOrOverwriteInputStreams the input streams containing the content of the files
* to add or modify from the original file
* #param filesToDelete the names of the files to delete from the original file
* #throws IOException if the new file could not be written
*/
public static void modifyZipFile(File zipFile,
File newZipFile,
String[] filesToAddOrOverwrite,
InputStream[] filesToAddOrOverwriteInputStreams,
String[] filesToDelete) throws IOException {
try (ZipOutputStream out = new ZipOutputStream(new FileOutputStream(newZipFile))) {
// add existing ZIP entry to output stream
try (ZipInputStream zin = new ZipInputStream(new FileInputStream(zipFile))) {
ZipEntry entry = null;
while ((entry = zin.getNextEntry()) != null) {
String name = entry.getName();
// check if the file should be deleted
if (filesToDelete != null) {
boolean ignoreFile = false;
for (String fileToDelete : filesToDelete) {
if (name.equalsIgnoreCase(fileToDelete)) {
ignoreFile = true;
break;
}
}
if (ignoreFile) {
continue;
}
}
// check if the file should be kept as it is
boolean keepFileUnchanged = true;
if (filesToAddOrOverwrite != null) {
for (String fileToAddOrOverwrite : filesToAddOrOverwrite) {
if (name.equalsIgnoreCase(fileToAddOrOverwrite)) {
keepFileUnchanged = false;
}
}
}
if (keepFileUnchanged) {
// copy the file as it is
out.putNextEntry(new ZipEntry(name));
IOUtils.copy(zin, out);
}
}
}
// add the modified or added files to the zip file
if (filesToAddOrOverwrite != null) {
for (int i = 0; i < filesToAddOrOverwrite.length; i++) {
String fileToAddOrOverwrite = filesToAddOrOverwrite[i];
try (InputStream in = filesToAddOrOverwriteInputStreams[i]) {
out.putNextEntry(new ZipEntry(fileToAddOrOverwrite));
IOUtils.copy(in, out);
out.closeEntry();
}
}
}
}
}
this works 100% , if you dont want to use extra libs ..
1) first, the class that append files to the zip ..
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class AddZip {
public void AddZip() {
}
public void addToZipFile(ZipOutputStream zos, String nombreFileAnadir, String nombreDentroZip) {
FileInputStream fis = null;
try {
if (!new File(nombreFileAnadir).exists()) {//NO EXISTE
System.out.println(" No existe el archivo : " + nombreFileAnadir);return;
}
File file = new File(nombreFileAnadir);
System.out.println(" Generando el archivo '" + nombreFileAnadir + "' al ZIP ");
fis = new FileInputStream(file);
ZipEntry zipEntry = new ZipEntry(nombreDentroZip);
zos.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {zos.write(bytes, 0, length);}
zos.closeEntry();
fis.close();
} catch (FileNotFoundException ex ) {
Logger.getLogger(AddZip.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(AddZip.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
2) you can call it in your controller ..
//in the top
try {
fos = new FileOutputStream(rutaZip);
zos = new ZipOutputStream(fos);
} catch (FileNotFoundException ex) {
Logger.getLogger(UtilZip.class.getName()).log(Level.SEVERE, null, ex);
}
...
//inside your method
addZip.addToZipFile(zos, pathFolderFileSystemHD() + itemFoto.getNombre(), "foto/" + itemFoto.getNombre());
Based on the answer given by #sfussenegger above, following code is used to append to a jar file and download it:
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
Resource resourceFile = resourceLoader.getResource("WEB-INF/lib/custom.jar");
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (ZipOutputStream zos = new ZipOutputStream(baos, StandardCharsets.ISO_8859_1);) {
try (ZipFile zin = new ZipFile(resourceFile.getFile(), StandardCharsets.ISO_8859_1);) {
zin.stream().forEach((entry) -> {
try {
zos.putNextEntry(entry);
if (!entry.isDirectory()) {
zin.getInputStream(entry).transferTo(zos);
}
zos.closeEntry();
} catch (Exception ex) {
ex.printStackTrace();
}
});
}
/* build file records to be appended */
....
for (FileContents record : records) {
zos.putNextEntry(new ZipEntry(record.getFileName()));
zos.write(record.getBytes());
zos.closeEntry();
}
zos.flush();
}
response.setContentType("application/java-archive");
response.setContentLength(baos.size());
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"custom.jar\"");
try (BufferedOutputStream out = new BufferedOutputStream(response.getOutputStream())) {
baos.writeTo(out);
}
}
I am reading a data from .wav file and then converting it into binary format. and then I write those binaries and create a new .wav file. I want that after getting binary format of .wav file I should do little modifications in its LSB's and then write the file from those modified bits.
How should i implement this? I am not getting any way. Please help me as I want to perform stenography through audio file.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
public class FiletoArrayofBytes
{
public static void main( String[] args )
{
FileInputStream fileInputStream=null;
FileOutputStream fop = null;
File file = new File("C:\\file.wav");
byte[] bFile = new byte[(int) file.length()];
try {
File fileo = new File("c:/newfile.wav");
fop = new FileOutputStream(fileo);
// if file doesnt exists, then create it
if (!fileo.exists()) {
fileo.createNewFile();
}
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
for (int i = 0; i < bFile.length; i++) {
System.out.println(Integer.toBinaryString(0x100 + (bFile[i])).substring(1));
//String a =(Integer.toBinaryString(0x100 + (bFile[i])).substring(1));
int a=bFile[i];
fop.write(a);
System.out.println("\t i am a: " +a);
}
fop.flush();
fop.close();
System.out.println("Done");
}catch(Exception e){
e.printStackTrace();
}
}
}
WAV-files have at least a header. You can't just read/modify/write it byte-by-byte.
I would use some sort of Java-WAV library. For instance this one: Java File IO
WavFile class is really nice and useful. They do have nice read/modify/write examples.
Using that you can implement LSB (...google helped me finding this link...).
I have tried multiple ways to create this zip file in Java/Groovy. The first couple methods I attempted, from various blogs/postings, resulted in corrupt zip files which could not be opened. So, I tried this one (below) which looked fairly promising. The sysouts report valid file paths being passed to the FileInputStream. I am not sure if it is the FQ path being passed to the ZipOutputStream which is causing the problem. Either way, below is the code, which results in small (188kb) zip file (with no entries) being created. Any suggestions?
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.Set;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
class FileZipper {
public static void makeZip(Set fullyQualifiedFileNames, String zipFileName, String outDir) throws IOException, FileNotFoundException
{
// These are the files to include in the ZIP file
Object[] filenames = fullyQualifiedFileNames.toArray();
String fileSeparator = (String) System.getProperties().get("file.separator");
// Create a buffer for reading the files
byte[] buf = new byte[1024];
// Create the ZIP file
String outFilename = outDir + fileSeparator +zipFileName;
FileOutputStream fos = new FileOutputStream(outFilename);
ZipOutputStream zos = new ZipOutputStream(fos);
System.out.println("Zipping to file " +outFilename);
// Compress the files
for (Object fileName: filenames)
{
System.out.println("Adding file: " + fileName);
FileInputStream fis = new FileInputStream((String)fileName);
// Add ZIP entry to output stream.
String[] nodes = ((String)fileName).split("[/[\\\\]]");
String zipEntry = nodes[nodes.length-1];
System.out.println("Adding Zip Entry: " + zipEntry);
zos.putNextEntry(new ZipEntry((String)fileName));
// Transfer bytes from the file to the ZIP file
int len;
int totalBytes = 0;
while ((len = fis.read(buf)) > 0)
{
totalBytes += len;
zos.write(buf, 0, len);
}
System.out.println("Zipped " +totalBytes +" bytes");
// Complete the entry
zos.closeEntry();
fis.close();
}
// Complete the ZIP file
zos.close();
fos.close();
}
}
If you are using Groovy, the easiest way is using AntBuilder:
new AntBuilder().zip(
destfile: "myfile.zip",
basedir: "baseDir")
or as of Groovy 1.8:
ant.zip(destfile: 'file.zip', basedir: 'src_dir')
Have you tried closing the underlying FileOutputStream explicitly to ensure that all data has been flushed to disk?
FileOutputStream fos = new FileOutputStream(outFilename);
ZipOutputStream zos = new ZipOutputStream(fos);
...
zos.Close();
fos.Close();
I ran your code locally and had no problems creating and then opening a zip file.
However, I sometimes used to run into odd issues with the default Java compression utilities, so I started using the Apache Commons compression and have used it since with little difficulty.
Check http://commons.apache.org/compress/index.html for the basic overview, and http://commons.apache.org/compress/examples.html for specific examples.
I want to put some compressed data into a remote repository.
To put data on this repository I can only use a method that take the name of the resource and its content as a String. (like data.txt + "hello world").
The repository is moking a filesystem but is not, so I can not use File directly.
I want to be able to do the following:
client send to server a file 'data.txt'
server compress 'data.txt' into a compressed file 'data.zip'
server send a string representation of data.zip to the repository
repository store data.zip
client download from repository data.zip and his able to open it with its favorite zip tool
The problem arise at step 3 when I try to get a string representation of my compressed file.
Here is a sample class, using the zip*stream and that emulate the repository showcasing my problem.
The created zip file is working, but after its 'serialization' it's get corrupted.
(the sample class use jakarta commons.io )
Many thanks for your help.
package zip;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;
import org.apache.commons.io.FileUtils;
/**
* Date: May 19, 2010 - 6:13:07 PM
*
* #author Guillaume AME.
*/
public class ZipMe {
public static void addOrUpdate(File zipFile, File ... files) throws IOException {
File tempFile = File.createTempFile(zipFile.getName(), null);
// delete it, otherwise you cannot rename your existing zip to it.
tempFile.delete();
boolean renameOk = zipFile.renameTo(tempFile);
if (!renameOk) {
throw new RuntimeException("could not rename the file " + zipFile.getAbsolutePath() + " to " + tempFile.getAbsolutePath());
}
byte[] buf = new byte[1024];
ZipInputStream zin = new ZipInputStream(new FileInputStream(tempFile));
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipFile));
ZipEntry entry = zin.getNextEntry();
while (entry != null) {
String name = entry.getName();
boolean notInFiles = true;
for (File f : files) {
if (f.getName().equals(name)) {
notInFiles = false;
break;
}
}
if (notInFiles) {
// Add ZIP entry to output stream.
out.putNextEntry(new ZipEntry(name));
// Transfer bytes from the ZIP file to the output file
int len;
while ((len = zin.read(buf)) > 0) {
out.write(buf, 0, len);
}
}
entry = zin.getNextEntry();
}
// Close the streams
zin.close();
// Compress the files
if (files != null) {
for (File file : files) {
InputStream in = new FileInputStream(file);
// Add ZIP entry to output stream.
out.putNextEntry(new ZipEntry(file.getName()));
// Transfer bytes from the file to the ZIP file
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
// Complete the entry
out.closeEntry();
in.close();
}
// Complete the ZIP file
}
tempFile.delete();
out.close();
}
public static void main(String[] args) throws IOException {
final String zipArchivePath = "c:/temp/archive.zip";
final String tempFilePath = "c:/temp/data.txt";
final String resultZipFile = "c:/temp/resultingArchive.zip";
File zipArchive = new File(zipArchivePath);
FileUtils.touch(zipArchive);
File tempFile = new File(tempFilePath);
FileUtils.writeStringToFile(tempFile, "hello world");
addOrUpdate(zipArchive, tempFile);
//archive.zip exists and contains a compressed data.txt that can be read using winrar
//now simulate writing of the zip into a in memory cache
String archiveText = FileUtils.readFileToString(zipArchive);
FileUtils.writeStringToFile(new File(resultZipFile), archiveText);
//resultingArchive.zip exists, contains a compressed data.txt, but it can not
//be read using winrar: CRC failed in data.txt. The file is corrupt
}
}
Zip files are binary. String handling in Java is textual and might be mangling what it sees as CRLFs, zero bytes and EOF markers. When it comes to reading and rewriting the zipfile, I suggest you try with readFileToByteArray and writeByteArrayToFile as an experiment. If that works then I'd suspect the String handling is to blame.
server send a string representation of
data.zip to the repository
So you want to get a string (i.e. textual) representation of a zip (i.e. binary) stream.
Base64 is the most popular way to do this.
One popular Java implementation is from Apache commons (codec component)