Develop Reference

How to create a ZIP InputStream in Android without creating a ZIP file first?

I use NanoHTTPD as web server in my Android APP, I hope to compress some files and create a InputStream in server side, and I download the InputStream in client side using Code A.
I have read Code B at How to zip and unzip the files?, but how to create a ZIP InputStream in Android without creating a ZIP file first?
BTW, I don't think Code C is good way, because it make ZIP file first, then convert ZIP file to FileInputStream , I hope to create a ZIP InputStream directly!
Code A
private Response ActionDownloadSingleFile(InputStream fis) {
Response response = null;
response = newChunkedResponse(Response.Status.OK, "application/octet-stream",fis);
response.addHeader("Content-Disposition", "attachment; filename="+"my.zip");
return response;
}
Code B
public static void zip(String[] files, String zipFile) throws IOException {
BufferedInputStream origin = null;
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(new FileOutputStream(zipFile)));
try {
byte data[] = new byte[BUFFER_SIZE];
for (int i = 0; i < files.length; i++) {
FileInputStream fi = new FileInputStream(files[i]);
origin = new BufferedInputStream(fi, BUFFER_SIZE);
try {
ZipEntry entry = new ZipEntry(files[i].substring(files[i].lastIndexOf("/") + 1));
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER_SIZE)) != -1) {
out.write(data, 0, count);
}
}
finally {
origin.close();
}
}
}
finally {
out.close();
}
}
Code C
File file= new File("my.zip");
FileInputStream fis = null;
try
{
fis = new FileInputStream(file);
} catch (FileNotFoundException ex)
{
}

ZipInputStream as per the documentation ZipInputStream
ZipInputStream is an input stream filter for reading files in the ZIP file format. Includes support for both compressed and uncompressed entries.
Earlier I answered to this question in a way that it is not possible using ZipInputStream. I am Sorry.
But after investing some time I found that it is possible as per the below code
It is very much obvious that since you are sending files in zip format
over the network.
//Create proper background thread pool. Not best but just for solution
new Thread(new Runnable() {
#Override
public void run() {
// Moves the current Thread into the background
android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND);
HttpURLConnection httpURLConnection = null;
byte[] buffer = new byte[2048];
try {
//Your http connection
httpURLConnection = (HttpURLConnection) new URL("https://s3-ap-southeast-1.amazonaws.com/uploads-ap.hipchat.com/107225/1251522/SFSCjI8ZRB7FjV9/zvsd.zip").openConnection();
//Change below path to Environment.getExternalStorageDirectory() or something of your
// own by creating storage utils
File outputFilePath = new File ("/mnt/sdcard/Android/data/somedirectory/");
ZipInputStream zipInputStream = new ZipInputStream(new BufferedInputStream(httpURLConnection.getInputStream()));
ZipEntry zipEntry = zipInputStream.getNextEntry();
int readLength;
while(zipEntry != null){
File newFile = new File(outputFilePath, zipEntry.getName());
if (!zipEntry.isDirectory()) {
FileOutputStream fos = new FileOutputStream(newFile);
while ((readLength = zipInputStream.read(buffer)) > 0) {
fos.write(buffer, 0, readLength);
}
fos.close();
} else {
newFile.mkdirs();
}
Log.i("zip file path = ", newFile.getPath());
zipInputStream.closeEntry();
zipEntry = zipInputStream.getNextEntry();
}
// Close Stream and disconnect HTTP connection. Move to finally
zipInputStream.closeEntry();
zipInputStream.close();
} catch (IOException e) {
e.printStackTrace();
}finally {
// Close Stream and disconnect HTTP connection.
if (httpURLConnection != null) {
httpURLConnection.disconnect();
}
}
}
}).start();

.\key.p12: open failed: ENOENT (No such file or directory)

In my android application I'm accessing the Google cloud storage . I have generated the private key xxxxxxxkey.p12 .I have put my key file in assets folder . But while running the project it is not opening the key.p12 file . I have tried putting it outside the assets folder , still no result.
httpTransport = AndroidHttp.newCompatibleTransport();
AssetManager am = getAssets();
InputStream inputStream = am.open("xxxxxxxxxxKey.p12");
File file = createFileFromInputStream(inputStream);
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#developer.gserviceaccount.com")
.setServiceAccountScopes(Collections.singleton(STORAGE_SCOPE))
.setServiceAccountPrivateKeyFromP12File(file).build();
createFileFromInputStream()
private File createFileFromInputStream(InputStream inputStream) {
try {
File f = new File("download/MyKey.p12");
OutputStream outputStream = new FileOutputStream(f);
byte buffer[] = new byte[1024];
int length = 0;
while ((length = inputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, length);
}
outputStream.close();
inputStream.close();
return f;
} catch (IOException e) {
// Logging exception
}
return null;
}
I've done the same in java project.What makes the difference, is it because of android ? or the path to the file location is incorrect?

After some struggle I have got my answer, Thanks a lot for your support. Thumbs up!
File can be retrieved using using AssetManager and also we can get it as a raw resource
Using AssetManager
AssetManager am = getAssets();
InputStream inputStream = am.open("xxxxxxxxxxKey.p12");
File file = createFileFromInputStream(inputStream);
As a raw resource , put the file in raw folder inside res directory
InputStream ins = getResources().openRawResource(R.raw.keyfile);
File file = createFileFromInputStream(ins);
While writing the output file you have to specify where your keyfile actually belongs , in my case I'm using android, I'm creating the file inside the internal storage(emulator/device) inside folder KeyHolder/KeyFile
private File createFileFromInputStream(InputStream inputStream) {
String path = "";
File file = new File(Environment.getExternalStorageDirectory(),
"KeyHolder/KeyFile/");
if (!file.exists()) {
if (!file.mkdirs())
Log.d("KeyHolder", "Folder not created");
else
Log.d("KeyHolder", "Folder created");
} else
Log.d("KeyHolder", "Folder present");
path = file.getAbsolutePath();
try {
File f = new File(path+"/MyKey");
OutputStream outputStream = new FileOutputStream(f);
byte buffer[] = new byte[1024];
int length = 0;
while ((length = inputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, length);
}
outputStream.close();
inputStream.close();
return f;
} catch (IOException e) {
// Logging exception
e.printStackTrace();
}
return null;
}
and that's it !

How to resolve OutOfMemory:Java HeapSpace on my Client Program (with Eclipse)

I am writing a client-server program. The client is on another computer on the same WLAN. The problem I am confronted with is this exception on the client side: java.lang.OutOfMemoryError: Java heap space.
The problem is that if I use the server computer as the client too (Server-Client on the same machine), it does not throw this exception. However, every time I run the client from another computer to download a document, it throws the exception.
Kindly help with an easy way to go about this. I use Eclipse.
Below is my client class:
public void run() {
try {
servsock = new ServerSocket(13330);
while (true) {
System.out.println("Waiting on question server...");
try {
sock = servsock.accept();
System.out.println("Accepted connection : " + sock);
// receive file
byte [] mybytearray = new byte [FILE_SIZE];
InputStream is = sock.getInputStream();
String LocalLocation = "C:\\NewEvalve\\LocalUploadedFiles\\"+clasx+"\\"+subject+"\\"+filename;
File LocalFile = new File(LocalLocation);
File file = new File("C:\\NewEvalve\\LocalUploadedFiles\\"+clasx+"\\"+subject);
if (!file.exists()) {
new File("C:\\NewEvalve\\LocalUploadedFiles\\"+clasx+"\\"+subject).mkdirs();
System.out.println(file+ " now exists.");
}
File outputfile = new File("C:\\NewEvalve\\LocalUploadedFiles\\"+clasx+"\\"+subject+"\\"+filename);
fos = new FileOutputStream(outputfile);
bos = new BufferedOutputStream(fos);
bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
System.out.println("File " + FILE_TO_RECEIVED
+ " downloaded (" + current + " bytes read)");
}
finally {
if (fos != null) fos.close();
if (bos != null) bos.close();
if (sock != null) sock.close();
} }
} catch (IOException e) {
e.printStackTrace();
}
...... and here is my server class that uploads the file:
public void uploadIt(InetAddress retur2, String fileLocation2){
try{
File myFile = new File (fileLocation2);
sock = new Socket(retur2, 13330);
System.out.println("Connection on "+sock.getRemoteSocketAddress());
os = sock.getOutputStream();
byte [] mybytearray = new byte [(int)myFile.length()];
fis = new FileInputStream(myFile);
bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
os.write(mybytearray,0,mybytearray.length);
fis.close();
bis.close();
os.flush();
os.close();
System.out.println("Sent " + fileLocation2 + "(" + mybytearray.length + " bytes)");
} catch (UnknownHostException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
finally {
try {
if (bis != null)bis.close();
if (os != null) os.close();
if (sock!=null) sock.close();
System.out.println("Completed Upload");
} catch (IOException e) {
e.printStackTrace();
}}
}

If you are reading text file then you should use BufferedReader & BufferedWriter instead of reading whole file at a time, like this
File file = new File("your file");
FileWriter fw = new FileWriter(file.getAbsoluteFile());
BufferedWriter bw = new BufferedWriter(fw);
BufferedReader br = new BufferedReader(new FileReader("your file"));
while ((currentLine = br.readLine()) != null) {
bw.write(currentLine);
}
bw.close();
br.close();
If you are reading binary files then it is suggested to use BufferedInputStream & BufferedOutputStream. You cannot read binary data using BufferedReader because it convert binary data into string format & your file will corrupt or damaged.
It may be possible there is no line terminator in your file. In that case, reader continues reading file until there is space in heap memory. Thats why you get OutOfMemory error.
EDIT :
To perform read/write operations on Audio Files there is an extension available in Java Standard Edition - Sound. You can use classes in javax.sound.sampled.* package.
Example for Audio File :
File fileIn = new File("your file");
AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(fileIn);
// do something
File fileOut = new File("new file");
if (AudioSystem.isFileTypeSupported(AudioFileFormat.Type.WAVE, audioInputStream)) {
AudioSystem.write(audioInputStream, AudioFileFormat.Type.WAVE, fileOut);
}
To get detailed idea on sound package look at this Java Sound API
To work with video file format you can do like below :-
There is method in Java getResourceAsStream("your video file") which returns an object of InputStream. This stream you can use to read video file data & write it to another file or whatever you want to do.
InputStream getResourceAsStream(String name) - Java Doc Some sample code will look like this :-
try {
InputStream in = this.getClass().getResourceAsStream("your video file");
BufferedInputStream bis = new BufferedInputStream(in);
BufferedOutputStream bus = new BufferedOutputStream(new FileOutputStream(
new File("your new video file")));
byte[] data = new byte[in.available()];
int read = in.read(data);
bus.write(read);
} catch (IOException ex) {
}

Practically you can just adjust the maximum amount of memory your VM is going to use. Just use -Xmx<size> parameter to adjust this. For example -Xmx512m is going to use maximum of 512 MB of memory.
I've also noticed that you're reading a file as a big chunk into memory. I would do it piece by piece (like pieces of 1 or 2 kb) in order to prevent huge instant memory demands.

HotSpot sizes its heap according to the total physical RAM on a machine. When you change host machines, you implicitly change the heap limits. It would seem that, on your remote machine, you have less RAM, therefore you hit the limit.
However, the real advice would be not to load the entire file into memory; instead read the file in chunks and write them directly to the network socket. Then you won't face this issue no matter what the file size. The same reasoning applies to the server side as well: do not read the whole file from the socket into RAM, but dump it directly into a file.

try to raise perm space, add following parameters to vm start-up
-XX:PermSize=256m -XX:MaxPermSize=256m
also add -XX:MaxPermSize=256m to Tomcat in Eclipse: Server > Open Launch Configuration > Arguments
Reference link is here : Out of memory

GZIPInputStream throws exception when reading GZIP FIle

I am trying to read files from a public anonymous ftp and I am running in to a problem. I can read the plain text files just fine, but when I try to read in gzip files, I get this exception:
Exception in thread "main" java.util.zip.ZipException: invalid distance too far back
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java_io_FilterInputStream$read.call(Unknown Source)
at GenBankFilePoc.main(GenBankFilePoc.groovy:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
I have tried downloading the file and using a FileInputStream wrapped in a GZIPInputStream and got the exact same problem, so I don't think it is a problem with the FTP client (which is apache).
Here is some test code that reproduces the problem. It is just trying to print to stdout:
FTPClient ftp = new FTPClient();
ftp.connect("ftp.ncbi.nih.gov");
ftp.login("anonymous", "");
InputStream is = new GZIPInputStream(ftp.retrieveFileStream("/genbank/gbbct1.seq.gz"));
try {
byte[] buffer = new byte[65536];
int noRead;
while ((noRead = is.read(buffer)) != 1) {
System.out.write(buffer, 0, noRead);
}
} finally {
is.close();
ftp.disconnect();
}
I cannot find any documentation on why this would be happening, and following it through the code in a debugger is not getting me anywhere. I feel like I am missing something obvious.
EDIT: I manually downloaded the file and read it in with a GZIPInputStream and was able to print it out just fine. I have tried this with 2 different Java FTP Clients

Ah, I found out what was wrong. You have to set the file type to FTP.BINARY_FILE_TYPE so that the SocketInputStream returned from retrieveFileStream is not buffered.
The following code works:
FTPClient ftp = new FTPClient();
ftp.connect("ftp.ncbi.nih.gov");
ftp.login("anonymous", "");
ftp.setFileType(FTP.BINARY_FILE_TYPE);
InputStream is = new GZIPInputStream(ftp.retrieveFileStream("/genbank/gbbct1.seq.gz"));
try {
byte[] buffer = new byte[65536];
int noRead;
while ((noRead = is.read(buffer)) != 1) {
System.out.write(buffer, 0, noRead);
}
} finally {
is.close();
ftp.disconnect();
}
}

You need to first download the file completely before, since ftp.retrieveFileStream() doesn't support file seeking.
Your code should be:
FTPClient ftp = new FTPClient();
ftp.connect("ftp.ncbi.nih.gov");
ftp.login("anonymous", "");
File downloaded = new File("");
FileOutputStream fos = new FileOutputStream(downloaded);
ftp.retrieveFile("/genbank/gbbct1.seq.gz", fos);
InputStream is = new GZIPInputStream(new FileInputStream(downloaded));
try {
byte[] buffer = new byte[65536];
int noRead;
while ((noRead = is.read(buffer)) != 1) {
System.out.write(buffer, 0, noRead);
}
} finally {
is.close();
ftp.disconnect();
}

Java Connecting URL and downloading a zip but when extracting the zip it's not properly downloaded

I am sending a request XML to the URL and receiving a zip file to the given path.
Sometimes I'm facing troubles when the bandwidth is low this zip file, most likely 120MB size is not getting downloaded properly. And getting an error when extracting the zip file. Extracting happens from the code as well. When I download in high bandwidth this file gets download without issue.
I'm looking for a solution without making the bandwidth high, from program level are there any ways to download this zip file, may be part by part or something like that? Or anyother solution that you all are having is highly appreciated.
Downloading :
url = new URL(_URL);
sc = (HttpURLConnection) url.openConnection();
sc.setDoInput(true);
sc.setDoOutput(true);
sc.setRequestMethod("POST");
sc.connect();
OutputStream mOstr = sc.getOutputStream();
mOstr.write(request.getBytes());
InputStream in = sc.getInputStream();
FileOutputStream out = new FileOutputStream(path);
int count;
byte[] buffer = new byte[86384];
while ((count = in.read(buffer,0,buffer.length)) > 0)
out.write(buffer, 0, count);
out.close();
Extracting :
try {
ZipFile zipFile = new ZipFile(path+zFile);
Enumeration<?> enu = zipFile.entries();
while (enu.hasMoreElements()) {
ZipEntry zipEntry = (ZipEntry) enu.nextElement();
String name = path+"/data_FILES/"+zipEntry.getName();
long size = zipEntry.getSize();
long compressedSize = zipEntry.getCompressedSize();
System.out.printf("name: %-20s | size: %6d | compressed size: %6d\n", name, size, compressedSize);
File file = new File(name);
if (name.endsWith("/")) {
file.mkdirs();
continue;
}
File parent = file.getParentFile();
if (parent != null) {
parent.mkdirs();
}
InputStream is = zipFile.getInputStream(zipEntry);
FileOutputStream fos = new FileOutputStream(file);
byte[] bytes = new byte[86384];
int length;
while ((length = is.read(bytes)) >= 0) {
fos.write(bytes, 0, length);
}
is.close();
fos.close();
}
zipFile.close();
} catch (Exception e) {
log("Error in extracting zip file ");
e.printStackTrace();
}