I developed a simple media library where you can choose a set of images and download them.
When a client request a download, a servlet receives the blob keys to use for create a zip file and then a Task is launched for the procedure
The task iterate through the received blob keys and zip the images into the archive. When the task has finished a mail with the download link is sent to the user.
Here is my problem:
FileWriteChannel writeChannel = fileService.openWriteChannel(file, lock);
OutputStream blobOutputStream = Channels.newOutputStream(writeChannel);
ZipOutputStream zip = new ZipOutputStream(blobOutputStream);
A single channel can handle only this amount of bytes
BlobstoreService.MAX_BLOB_FETCH_SIZE
Because of that, i must open and close the channel every 1mb of data i have to write (same issue for the read, but for the read i used this code and it works). or the write() method throws a null exception
Opening and closing the channel with a normal outputStream, does not presents issue, like this code
But handling a Zip file i also have to manage
ZipOutputStream zip = new ZipOutputStream(blobOutputStream);
ZipEntry zipEntry = new ZipEntry(image_file_name);
zipOut.putNextEntry(zipEntry);
// while the image has bytes to write
zipOut.write(bytesToWrite);
After i wrote 1MB of data in the ZipEntry i have to close the channel and open it again.
So here the problem: where i open a new channel i can't access to the previouse zipEntry i was writing and then i cannot continue to write the next 1MB of the image i'm processing.
And, after a open a new channel, if i try to write on the zipEntry object (w/o re-initializing it) i get a ClosedChannel exception
Here is the SAMPLE code i wrote, i know is not working, but explains what i am trying to do.
My question then: How (if is possible, off course) can i create a zip file writing 1MB per time?
I'm also available to other approaches, the thing i need is to zip some images into one zip and save it into the blobstore, if you have other ideas to make this, please tell me
You should create you own stream that can manipulate channels. When the blob size limit is reached your stream closes current channel and opens a new one.
Example for local files:
public class ZipChannels {
public static void main(String[] args) throws IOException {
File dirToZip = new File("target\\dependency");
//create zip-files
ChannelOutput out = new ChannelOutput();
ZipOutputStream zip = new ZipOutputStream(out);
int b = 0;
for(File file: dirToZip.listFiles()) {
ZipEntry zipEntry = new ZipEntry(file.getName());
zip.putNextEntry(zipEntry);
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
while((b = bis.read()) != -1) {
zip.write(b);
}
bis.close();
zip.closeEntry();
}
zip.close();
//merge all into one file for check it
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("package_all.zip"));
for (int i = 0; i < out.getChannelCount(); i++) {
BufferedInputStream bis = new BufferedInputStream(new FileInputStream("package_" + i + ".zip"));
while((b = bis.read()) != -1) {
bos.write(b);
}
bis.close();
}
bos.close();
}
public static class ChannelOutput extends OutputStream {
private OutputStream channel;
private int count = 0;
final private int MAX = 1000000;
#Override
public void write(int b) throws IOException {
if(count++ % MAX == 0) {
openNewChannel();
}
channel.write(b);
}
protected void openNewChannel() throws IOException {
if(channel != null) {
channel.close();
}
channel = new BufferedOutputStream(new FileOutputStream("package_" + (count / MAX) + ".zip"));
}
public int getChannelCount() {
return count / MAX + 1;
}
#Override
public void close() throws IOException {
channel.close();
}
#Override
public void flush() throws IOException {
channel.flush();
}
}
}
If you have any questions, please feel free to ask.
Related
I use NanoHTTPD as web server in my Android APP, I hope to compress some files and create a InputStream in server side, and I download the InputStream in client side using Code A.
I have read Code B at How to zip and unzip the files?, but how to create a ZIP InputStream in Android without creating a ZIP file first?
BTW, I don't think Code C is good way, because it make ZIP file first, then convert ZIP file to FileInputStream , I hope to create a ZIP InputStream directly!
Code A
private Response ActionDownloadSingleFile(InputStream fis) {
Response response = null;
response = newChunkedResponse(Response.Status.OK, "application/octet-stream",fis);
response.addHeader("Content-Disposition", "attachment; filename="+"my.zip");
return response;
}
Code B
public static void zip(String[] files, String zipFile) throws IOException {
BufferedInputStream origin = null;
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(zipFile)));
try {
byte data[] = new byte[BUFFER_SIZE];
for (int i = 0; i < files.length; i++) {
FileInputStream fi = new FileInputStream(files[i]);
origin = new BufferedInputStream(fi, BUFFER_SIZE);
try {
ZipEntry entry = new ZipEntry(files[i].substring(files[i].lastIndexOf("/") + 1));
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER_SIZE)) != -1) {
out.write(data, 0, count);
}
}
finally {
origin.close();
}
}
}
finally {
out.close();
}
}
Code C
File file= new File("my.zip");
FileInputStream fis = null;
try
{
fis = new FileInputStream(file);
} catch (FileNotFoundException ex)
{
}
ZipInputStream as per the documentation ZipInputStream
ZipInputStream is an input stream filter for reading files in the ZIP file format. Includes support for both compressed and uncompressed entries.
Earlier I answered to this question in a way that it is not possible using ZipInputStream. I am Sorry.
But after investing some time I found that it is possible as per the below code
It is very much obvious that since you are sending files in zip format
over the network.
//Create proper background thread pool. Not best but just for solution
new Thread(new Runnable() {
#Override
public void run() {
// Moves the current Thread into the background
android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND);
HttpURLConnection httpURLConnection = null;
byte[] buffer = new byte[2048];
try {
//Your http connection
httpURLConnection = (HttpURLConnection) new URL("https://s3-ap-southeast-1.amazonaws.com/uploads-ap.hipchat.com/107225/1251522/SFSCjI8ZRB7FjV9/zvsd.zip").openConnection();
//Change below path to Environment.getExternalStorageDirectory() or something of your
// own by creating storage utils
File outputFilePath = new File ("/mnt/sdcard/Android/data/somedirectory/");
ZipInputStream zipInputStream = new ZipInputStream(new BufferedInputStream(httpURLConnection.getInputStream()));
ZipEntry zipEntry = zipInputStream.getNextEntry();
int readLength;
while(zipEntry != null){
File newFile = new File(outputFilePath, zipEntry.getName());
if (!zipEntry.isDirectory()) {
FileOutputStream fos = new FileOutputStream(newFile);
while ((readLength = zipInputStream.read(buffer)) > 0) {
fos.write(buffer, 0, readLength);
}
fos.close();
} else {
newFile.mkdirs();
}
Log.i("zip file path = ", newFile.getPath());
zipInputStream.closeEntry();
zipEntry = zipInputStream.getNextEntry();
}
// Close Stream and disconnect HTTP connection. Move to finally
zipInputStream.closeEntry();
zipInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}finally {
// Close Stream and disconnect HTTP connection.
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
}
}
}).start();
I want to read data from lets say 4 zip files called zip1, zip2, zip3, zip4. All of these zip files are split from this 1 big zip file called "BigZip". I want to combine the zip files into one and then compare the bytes if the 1 bigzip file matches the size of bytes with the combined zip file of (zip1+zip2+zip3+zip4). I am getting a very small file size when I combine the size of 4 zip files. What am I doing wrong?
Here is my code for the same:
targetFilePath1, targetFilePath2, targetFilePath3, targetFilePath4 belongs to path of 4 zip files.
sourceFilePath is the path to BigZip file
class Test {
public static void main(String args[]) {
ZipOutputStream outStream = new ZipOutputStream(new FileOutputStream(sourceBigZip));
readZip(sourceFilePath, targetFilePath1);
readZip(sourceFilePath, targetFilePath2);
readZip(sourceFilePath, targetFilePath3);
readZip(sourceFilePath, targetFilePath4);
outStream.close();
}
static void readZip(String sourceBigZip, String targetFile) throws Exception {
ZipInputStream inStream = new ZipInputStream(new FileInputStream(targetFile));
byte[] buffer = new byte[1024];
int len = inStream.read(buffer);
while (len != -1) {
outStream.write(buffer, 0, len);
len = inStream.read(buffer);
System.out.print(len);
}
inStream.close();
}
}
Create ZipOutputStream once and pass it to readZip() method, like:
public static void main(String args[]) {
ZipOutputStream outStream = new ZipOutputStream(new FileOutputStream(sourceFilePath));
readZip(outStream , targetFilePath1);
readZip(outStream , targetFilePath2);
readZip(outStream , targetFilePath3);
readZip(outStream , targetFilePath4);
}
Then you have an error dealing with copying the data from one zip to another...
You need to copy each file in the zip file like this:
static void readZip(ZipOutputStream outStream, String targetFile)
throws Exception {
ZipInputStream inStream = new ZipInputStream(new FileInputStream(
targetFile));
byte[] buffer = new byte[1024];
int len = 0;
for (ZipEntry e; (e = inStream.getNextEntry()) != null;) {
outStream.putNextEntry(e);
while ((len = inStream.read(buffer)) > 0) {
outStream.write(buffer, 0, len);
}
}
inStream.close();
}
}
Every time you call new ZipOutputStream, it creates a new empty file, and wipes out everything you have written to it before.
You have to create the stream outside of readZip, and pass it in to each call rather than creating a new stream every time.
Basically i compress video using the customized compressor class in Java. I have assembled my complete code snippets here. My actually problem is, generated video [ A.mp4] from the decompressed byte array is not running. I actually i got this compressor class code over the internet. As i new to Java platform, i am struggling to resolve this problem. Could you please any one help me on this.?
public class CompressionTest
{
public static void main(String[] args)
{
Compressor compressor = new Compressor();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
FileInputStream fis=null;
File file=null;
try
{
URL uri=CompressionTest.class.getResource("/Files/Video.mp4");
file=new File(uri.getPath());
fis = new FileInputStream(file);
}
catch ( FileNotFoundException fnfe )
{
System.out.println( "Unable to open input file");
}
try
{
byte[] videoBytes = getBytesFromFile(file);
System.out.println("CompressionVideoToCompress is: '" +videoBytes + "'");
byte[] bytesCompressed = compressor.compress(videoBytes);
System.out.println("bytesCompressed is: '" +bytesCompressed+ "'");
byte[] bytesDecompressed=compressor.decompress(bytesCompressed);
System.out.println("bytesDecompressed is: '" +bytesDecompressed+ "'");
FileOutputStream out = new FileOutputStream("A.mp4");
out.write(bytesDecompressed,0,bytesDecompressed.length-1);
out.close();
}
catch (IOException e)
{
// TODO Auto-generated catch block
System.out.println("bytesCompressed is: '");
}
}
public static byte[] getBytesFromFile(File file) throws IOException
{
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
}
// Create the byte array to hold the data
byte[] bytes = new byte[1064];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0)
{
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
// Close the input stream and return bytes
is.close();
return bytes;
}
}
class Compressor
{
public Compressor()
{}
public byte[] compress(byte[] bytesToCompress)
{
Deflater deflater = new Deflater();
deflater.setInput(bytesToCompress);
deflater.finish();
byte[] bytesCompressed = new byte[Short.MAX_VALUE];
int numberOfBytesAfterCompression = deflater.deflate(bytesCompressed);
byte[] returnValues = new byte[numberOfBytesAfterCompression];
System.arraycopy
(
bytesCompressed,
0,
returnValues,
0,
numberOfBytesAfterCompression
);
return returnValues;
}
public byte[] decompress(byte[] bytesToDecompress)
{
Inflater inflater = new Inflater();
int numberOfBytesToDecompress = bytesToDecompress.length;
inflater.setInput
(
bytesToDecompress,
0,
numberOfBytesToDecompress
);
int compressionFactorMaxLikely = 3;
int bufferSizeInBytes =
numberOfBytesToDecompress
* compressionFactorMaxLikely;
byte[] bytesDecompressed = new byte[bufferSizeInBytes];
byte[] returnValues = null;
try
{
int numberOfBytesAfterDecompression = inflater.inflate(bytesDecompressed);
returnValues = new byte[numberOfBytesAfterDecompression];
System.arraycopy
(
bytesDecompressed,
0,
returnValues,
0,
numberOfBytesAfterDecompression
);
}
catch (DataFormatException dfe)
{
dfe.printStackTrace();
}
inflater.end();
return returnValues;
}
}
I've tested your code by compressing and decompressing a simple TXT file. The code is broken, since the compressed file, when uncompressed, is different from the original one.
Take for granted that the code is broken at least in the getBytesFromFile function. Its logic is tricky and troublesome, since it only allows files up to length 1064 and the check (throwing IOException when a longer file is read) does not work at all. The file gets read only partially and no exception is thrown.
What you are trying to achieve (file compression/decompression) can be done this way. I've tested it and it works, you just need this library.
import java.io.*;
import java.util.zip.*;
import org.apache.commons.io.IOUtils; // <-- get this from http://commons.apache.org/io/index.html
public class CompressionTest2 {
public static void main(String[] args) throws IOException {
File input = new File("input.txt");
File output = new File("output.bin");
Compression.compress(input, output);
File input2 = new File("input2.txt");
Compression.decompress(output, input2);
// At this point, input.txt and input2.txt should be equal
}
}
class Compression {
public static void compress(File input, File output) throws IOException {
FileInputStream fis = new FileInputStream(input);
FileOutputStream fos = new FileOutputStream(output);
GZIPOutputStream gzipStream = new GZIPOutputStream(fos);
IOUtils.copy(fis, gzipStream);
gzipStream.close();
fis.close();
fos.close();
}
public static void decompress(File input, File output) throws IOException {
FileInputStream fis = new FileInputStream(input);
FileOutputStream fos = new FileOutputStream(output);
GZIPInputStream gzipStream = new GZIPInputStream(fis);
IOUtils.copy(gzipStream, fos);
gzipStream.close();
fis.close();
fos.close();
}
}
This code doesn't come from "credible and/or official sources" but at least it works. :)
Moreover, in order to get more answers, adjust the title stating your real problem: your compressed files don't decompress the right way. There is no 'video' stuff here. Moreover, zipping a .mp4 file is no achievement (compression ratio will likely be around 99.99%).
Two tips:
1) Replace getBytesFromFile with a well known API call, either using Apache commons (IOUtils) or java 7 now provides such a method, too.
2) Test compress and decompress by writing a Junit test:
Create a random huge byte array, write it out, read it back and compare it with the created one.
That's my first question so I hope I write it correctly.
I am trying to send an byte[] array through a Java socket, that array contains an image.
Here is the code to send the file:
public void WriteBytes(FileInputStream dis) throws IOException{
//bufferEscritura.writeInt(dis.available()); --- readInt() doesnt work correctly
Write(String.valueOf((int)dis.available()) + "\r\n");
byte[] buffer = new byte[1024];
int bytes = 0;
while((bytes = dis.read(buffer)) != -1){
Write(buffer, bytes);
}
System.out.println("Photo send!");
}
public void Write(byte[] buffer, int bytes) throws IOException {
bufferEscritura.write(buffer, 0, bytes);
}
public void Write(String contenido) throws IOException {
bufferEscritura.writeBytes(contenido);
}
My image:
URL url = this.getClass().getResource("fuegos_artificiales.png");
FileInputStream dis = new FileInputStream(url.getPath());
sockManager.WriteBytes(dis);
My code to get the image file:
public byte[] ReadBytes() throws IOException{
DataInputStream dis = new DataInputStream(mySocket.getInputStream());
int size = Integer.parseInt(Read());
System.out.println("Recived size: "+ size);
byte[] buffer = new byte[size];
System.out.println("We are going to read!");
dis.readFully(buffer);
System.out.println("Photo received!");
return buffer;
}
public String Leer() throws IOException {
return (bufferLectura.readLine());
}
And to create an image file:
byte[] array = tcpCliente.getSocket().LeerBytes();
FileOutputStream fos = new FileOutputStream("porfavor.png");
try {
fos.write(array);
}
finally {
fos.close();
}
The image file is created but when I try to open it for example with Paint it says that it can't open it because it is damaged...
I also tried to open both images (the original and the new one) with notepad and they have the same data inside!
I don't know what is happening...
I hope you help me.
Thanks!
Don't use available() as a measure of file length. It isn't. There is a specific warning in the Javadoc about that.
Use DataOutputStream.writeInt() to write the length, and DataInputStream.readInt() to read it, and use the same streams to read the image data. Don't use multiple streams on the same socket.
Also in this:
URL url = this.getClass().getResource("fuegos_artificiales.png");
FileInputStream dis = new FileInputStream(url.getPath());
the second line should be:
InputStream in = URL.openConnection.getInputStream();
A class resource is not a file.
I'm developing a Jersey service that uses Dropbox's API.
I need to post a generic file to my service (the service would be able to manage every kind of file as well as you can do with the Dropbox API).
Client Side
So, I've implemented a simple client that:
opens the file,
creates a connection to the URL,
sets the correct HTTP method,
creates a FileInputStream and writes the file on the connection's outputstream using byte buffer.
This is the client test code.
public class Client {
public static void main(String args[]) throws IOException, InterruptedException {
String target = "http://localhost:8080/DCService/REST/professor/upload";
URL putUrl = new URL(target);
HttpURLConnection connection = (HttpURLConnection) putUrl.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("content-Type", "application/pdf");
OutputStream os = connection.getOutputStream();
InputStream is = new FileInputStream("welcome.pdf");
byte buf[] = new byte[1024];
int len;
int lung = 0;
while ((len = is.read(buf)) > 0) {
System.out.print(len);
lung += len;
is.read(buf);
os.write(buf, 0, len);
}
}
}
Server Side
I've a method that:
gets an InputStream as an argument,
creates a file with the same name and type of the original file.
The following code implements a test method to receive a specific PDF file.
#PUT
#Path("/upload")
#Consumes("application/pdf")
public Response uploadMaterial(InputStream is) throws IOException {
String name = "info";
String type = "exerc";
String description = "not defined";
Integer c = 10;
Integer p = 131;
File f = null;
try {
f = new File("welcome.pdf");
OutputStream out = new FileOutputStream(f);
byte buf[] = new byte[1024];
int len;
while ((len = is.read(buf)) > 0)
out.write(buf, 0, len);
out.close();
is.close();
System.out.println("\nFile is created........");
} catch (IOException e) {
throw new WebApplicationException(Response.Status.BAD_REQUEST);
}
//I need to pass a java.io.file object to this method
professorManager.uploadMaterial(name, type, description, c,p, f);
return Response.ok("<result>File " + name + " was uploaded</result>").build();
}
This implementation works only with text files. If I try to send a simple PDF the received file is not readable (after I've saved it on disk).
How can I satisfy my requirements? Could anyone suggest me solution?
You're client code is faulty.
while ((len = is.read(buf)) > 0) {
...
is.read(buf);
...
}
You're reading from the InputStream twice in every iteration. Remove the read statement from the loop's body and you'll be fine.
You've also said that the code presented in your question works with text files. I think that doesn't work either. Reading twice from the file you're trying to upload means you're uploading only half of its contents. Half a text file is still a text file, but half a PDF is only rubbish, so you can't open the latter. You should have double checked if the contents of your uploaded and saved text file is the same as the original.