Getting data onto inputStream object from web url
inputStream = AWSFileUtil.getInputStream(
AWSConnectionUtil.getS3Object(null),
"cdn.generalsentiment.com", filePath);
If they are mutliple files then i want to zip them and sent the filetype as "zip" to struts.xml which does the download.
actually am converting the inputstream into byteArrayInputStream
ByteArrayInputStream byteArrayInputStream = new
ByteArrayInputStream(inputStream.toString().getBytes());
while (byteArrayInputStream.read(inputStream.toString().getBytes()) > 0) {
zipOutputStream.write(inputStream.toString().getBytes());
}
and then
zipOutputStream.close();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
fileInputStream = new FileInputStream(file);
while (fileInputStream.read(buffer) > 0) {
byteArrayOutputStream.write(buffer);
}
byteArrayOutputStream.close();
inputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
reportName = "GS_MediaValue_Reports.zip";
fileType = "zip";
}
return fileType;
But the downloaded zip when extracted gives corrupt files.
Please suggest me a solution for this issue.
The short answer is that it's not how ZipOutputStream works. Since it was designed to store multiple files, along with their file names, directory structures and so on, you need to tell the stream about that explicitly.
Furthermore, converting a stream to a string is a bad idea in general, plus it's slow, especially when you're doing it in a loop.
So your solution will be something like:
ZipEntry entry = new ZipEntry( fileName ); // You have to give each entry a different filename
zipOutputStream.putNextEntry( entry );
byte buffer[] = new byte[ 1024 ]; // 1024 is the buffer size here, but it could be anything really
int count;
while( (count = inputStream.read( buffer, 0, 1024 ) ) != -1 ) {
zipOutputStream.write( buffer, 0, count );
}
Related
I am trying to read extract a given file from zip file. Zip file contains directories & sub-directories as well. I tried Java7 nio file apis but since my zip has subdirectories as well, I need to provide complete path to extract the file, which is not suitable in my scenario. As I have to take filetobeextracted input from user. I have been trying below code for it but somehow read method of ZipInputStream not reading any contents to buffer. On debugging I found out that ZipEntry object value is null inside ZipInputStream due to its read method simply returns -1.But now I am stuck as I am not able to figure out how that value is being set for it.
try(OutputStream out=new FileOutputStream("filetoExtract");) {
zipFile = new ZipFile("zipFile");
Enumeration<? extends ZipEntry> e = zipFile.entries();
while (e.hasMoreElements()) {
ZipEntry entry = e.nextElement();
if (!entry.isDirectory()) {
String entryName = entry.getName();
String fileName = entryName.substring(entryName.lastIndexOf("/") + 1);
System.out.println(i++ + "." + entryName);
if (searchFile.equalsIgnoreCase(fileName)) {
System.out.println("File Found");
BufferedInputStream bufferedInputStream = new BufferedInputStream(zipFile.getInputStream(entry));
ZipInputStream zin = new ZipInputStream(bufferedInputStream);
byte[] buffer = new byte[9000];
int len;
while ((len = zin.read(buffer)) != -1) {
out.write(buffer, 0, len);
}
out.close();
break;
}
}
}
} catch (IOException ioe) {
System.out.println("Error opening zip file" + ioe);
}
Please advice what I am doing wrong here. Thanks
EDIT:
After debugging little more I found out that ZipFile class has inner class of similar name(ZipFileInputStream). So it was creating object of it rather than the outside ZipFileInputStream class. So I tried out below code and it worked out well. But I don't quite understand things here, what has happened. If someone could help me logic behind the scenes would be really great.
// BufferedInputStream bufferedInputStream = new
//BufferedInputStream(zipFile.getInputStream(entry));
//ZipInputStream zin = new ZipInputStream(bufferedInputStream);
InputStream zin= zipFile.getInputStream(entry);
The second line is unnecessary, as zipFile.getInputStream(entry) already returns an InputStream that represents the decompressed data. Therefore there's no need (or in fact it's wrong) to wrap that InputStream in yet another ZipInputStream:
BufferedInputStream bufferedInputStream = new BufferedInputStream(zipFile.getInputStream(entry));
ZipInputStream zin = new ZipInputStream(bufferedInputStream);
I have a requirement to split a 100mb zip file(which will be having sub folders and images) into 10 zip files(each of 10mb).Then I need to send each sliced zip files to an API (as multipart reauest), in receiver API i need to combine each of the above 10 zip files back to origin 100mb zip file.
Below is the code for slicing
public static void splitZip(String zipName, String location, String NewZip) throws IOException{
FileInputStream fis = new FileInputStream(location);
ZipInputStream zipInputStream = new ZipInputStream(fis);
ZipEntry entry = null;
int currentChunkIndex = 0;
long entrySize = 0;
ZipFile zipFile = new ZipFile(location);
Enumeration enumeration = zipFile.entries();
String copDest = zipCopyDest + "\\" + NewZip + "_" + currentChunkIndex +".zip";
FileOutputStream fos = new FileOutputStream(new File(copDest));
BufferedOutputStream bos = new BufferedOutputStream(fos);
ZipOutputStream zos = new ZipOutputStream(bos);
long currentSize = 0;
try {
while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {
ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
System.out.println(zipEntry.getName());
System.out.println(zipEntry.getSize());
entrySize = zipEntry.getSize();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
if((currentSize + entrySize) > MAX_FILE_SIZE) {
zos.close();
currentChunkIndex++;
zos = getOutputStream(currentChunkIndex, NewZip);
currentSize = 0;
}else{
currentSize += entrySize;
zos.putNextEntry(new ZipEntry(entry.getName()));
byte[] buffer = new byte[8192];
int length = 0;
while ((length = zipInputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, length);
}
byte[] unzippedFile = outputStream.toByteArray();
zos.write(unzippedFile);
unzippedFile = null;
outputStream.close();
zos.closeEntry();
}
}
} finally {
zos.close();
}
}
When i extract slices zips manually i found some images are corrupted am not able to open it. Also am not getting a proper solution for combining the zip files. Thanks in advance.
Zip specification has a feature to split zip files to any desired length (minimum split length should be 64kb). Zip4j, supports this feature to create split zip files (documentation). You can then pass each split file to the api. The api can then use the merge functionality in zip4j to merge those split files into a single zip file. On a side note: even without merging, it is a perfectly valid zip file. as long as all the split zip files are in the same directory.
The approach will not work if you have any file in the zip greater than 10 mb and also the else condition should be executed in both the cases and in this approach you have to depend on the size of file in zip may be better you go with approach of not creating smaller zips
I made an upload file servlet, summed up like this:
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
InputStream stream = item.openStream()
File file = new File(path +"/"+ item.getFieldName));
FileOutputStream fout= new FileOutputStream (file);
BufferedOutputStream bout= new BufferedOutputStream (fout);
BufferedInputStream bin= new BufferedInputStream(stream);
byte buf[] = new byte[2048];
while ((bin.read(buf)) != -1){
bout.write(buf);
}
bout.close();
bin.close();
}
I used streams so that the file isn't loaded in memory.
The files are being uploaded smoothly, but I cannot open the resulted file (error differs depending on file type). Also the size of the resulted file is larger then the one of the original.
I tried different types of stream readers and writers but couldn't get any closer, and I couldn't find a similar problem.
I ruled out encoding as I am receiving and writing bytes, so encoding doesn't matter, right?
What could be the problem?
You're writing all the content of buf array. This could be the problem for last read.
Change the while loop like this:
int n;
while ((n = bin.read(buf)) != -1)
{
bout.write(buf, 0, n);
}
Currently I am using below code to copy zip inside another zip.But when zip file size increases eg:2GB, program is throwing out of memory error.I have increased xmx to 1024, still prfoblem is same.Is thr s any alternate method to handle large files ?
public static void zipFile(File srcFile, File zipFile)
throws FileNotFoundException, IOException {
BufferedInputStream origin = null;
FileOutputStream dest = new FileOutputStream(zipFile);
ZipOutputStream out = new ZipOutputStream(
new BufferedOutputStream(dest));
// out.setMethod(ZipOutputStream.DEFLATED);
byte data[] = new byte[BUFFER];
FileInputStream fi = new FileInputStream(srcFile);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(srcFile.getName());
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
out.close();
}
You are only copying bytes when you come right down to it. You don't need to process either file as a Zip file. Just copy the bytes.
Can you try FileChannel.transferTo method? It's more efficient. Since this is done at OS level, I'm assuming it shouldn't depend on Java Heap size.
If that fails too take a look at this question.
I call a service which returns a gzipped file. I have the data as an InputStream (courtesy of javax.activation.DataHandler.getInputStream();) from the response.
What I would like to do is, without writing anything to disk, get an InputStream of the decompressed data in the file that is in the archive. The compressed file in this case is an xml document that I am trying to unmarshal using javax.xml.bind.Unmarshaller which takes an InputStream.
I'm currently trying to write the InputStream to an OutputStream (decompressing the data) and then I'll need to write it back to an InputStream. It's not working yet so I thought I would see if there was a better (I would hope so) approach.
I can write the initial InputStream to disk and get a gz file, and then read that file, get the compressed file out of it and go from there but I'd rather keep it all in memory is possible.
Update 1: Here is my current (not working - get a "Not in GZIP format" exception):
ByteArrayInputStream xmlInput = null;
try {
InputStream in = dh.getInputStream(); //dh is a javax.activation.DataHandler
BufferedInputStream bis = new BufferedInputStream(in);
ByteArrayOutputStream bo = new ByteArrayOutputStream();
int bytes_read = 0;
byte[] dataBuf = new byte[4096];
while ((bytes_read = bis.read(dataBuf)) != -1) {
bo.write(dataBuf, 0, bytes_read);
}
ByteArrayInputStream bin = new ByteArrayInputStream(bo.toByteArray());
GZIPInputStream gzipInput = new GZIPInputStream(bin);
ByteArrayOutputStream out = new ByteArrayOutputStream();
dataBuf = new byte[4096];;
bytes_read = 0;
while ((bytes_read = gzipInput.read(dataBuf)) > 0) {
out.write(dataBuf, 0, bytes_read);
}
xmlInput = new ByteArrayInputStream(out.toByteArray());
If instead of writing to a ByteArrayOutputStream I write to a FileOutputStream the first time around I get a compressed file (which I can manually open to get the xml file within) and the service (eBay) says it should be a gzip file so I'm not sure why I get a "Not in GZIP format" error.
Update 2: I tried something a little different - same error ("Not in GZIP format"). Wow, I just tried to end that parenthesis with a semi-colon. Anyways, here is my second attempt, which still does not work:
ByteArrayInputStream xmlInput = null;
try {
GZIPInputStream gzipInput = new GZIPInputStream(dh.getInputStream());
ByteArrayOutputStream bo = new ByteArrayOutputStream();
int bytes_read = 0;
byte[] dataBuf = new byte[4096];
while ((bytes_read = gzipInput.read(dataBuf)) != -1) {
bo.write(dataBuf, 0, bytes_read);
}
xmlInput = new ByteArrayInputStream(bo.toByteArray());
Decorate the input stream with a GZIPInputStream.
InputStream decompressed = new GZIPInputStream(compressed);
The following code should work. Keep in mind you'll have to handle exceptions properly.
OutputStream out = null;
InputStream in = null;
try {
out = /* some output stream */;
in = new java.util.GZIPInputStream(/*some stream*/);
byte[] buffer = new byte[4096];
int c = 0;
while (( c = in.read(buffer, 0, 4096)) > 0) {
out.write(buffer, 0, c);
}
} finally {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
Take a look at GZIPInputStream. Here's an example; the class handles this very transparently, it's almost no work to use.