Develop Reference

Unzipping a directory/file in Java

I have the following code:
public static void unzip(final File archive) throws FileNotFoundException, IOException
{
ZipInputStream zipInput = null;
try
{
zipInput = new ZipInputStream(new FileInputStream(archive));
ZipEntry zipEntry = null;
while ((zipEntry = zipInput.getNextEntry()) != null)
{
String ename = zipEntry.getName();
final int pos = ename.lastIndexOf(File.separatorChar);
if (pos >= 0)
{
ename = ename.substring(pos + 1);
}
final FileOutputStream outputFile = new FileOutputStream(archive.getParent() + File.separatorChar + ename);
int data = 0;
try
{
while ((data = zipInput.read()) != -1)
{
outputFile.write(data);
}
}catch (final Exception e)
{
LOGGER.error( e);
}finally
{
outputFile.close();
}
}
}catch (final Exception e)
{
LOGGER.error("Error when zipping file ( "+archive.getPath()+" )", e);
}finally
{
if(zipInput !=null)
{
zipInput.close();
}
}
}
What I would like to know is, what does it mean when I get the value -1 from the following line:
(data = zipInput.read()) != -1
I'm guessing it's the reason why the zip file is not being unzipped properly.

It's an expected value to be returned by an InputStream which has no content left to read.
From InputStream's javadoc :
Returns:
the next byte of data, or -1 if the end of the stream is reached.

java OutOfMemoryError about FileOutputStream?

Thanks for everyone ^_^,the problem is solved:there is a single line is too big(over 400M...I download a damaged file while I didn't realize), so throw a OutOfMemoryError
I want to split a file by using java,but it always throw OutOfMemoryError: Java heap space,I searched on the whole Internet,but it looks like no help :(
ps. the file's size is 600M,and it have over 30,000,000 lines,every line is no longer than 100 chars.
(maybe you can generate a "level file" like this:{
id:0000000001,level:1
id:0000000002,level:2
....(over 30 millions)
})
pss. set the Jvm memory size larger is not work,:(
psss. I changed to another PC, problem remains/(ㄒoㄒ)/~~
no matter how large the -Xms or -Xmx I set,the outputFile's size is always same,(and the Runtime.getRuntime().totalMemory() is truely changed)
here's the stack trace:
Heap Size = 2058027008
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
at java.lang.StringBuffer.append(StringBuffer.java:306)
at java.io.BufferedReader.readLine(BufferedReader.java:345)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at com.xiaomi.vip.tools.ptupdate.updator.Spilt.main(Spilt.java:39)
...
here's my code:
package com.updator;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileReader;
public class Spilt {
public static void main(String[] args) throws Exception {
long heapSize = Runtime.getRuntime().totalMemory();
// Print the jvm heap size.
System.out.println("Heap Size = " + heapSize);
String mainPath = "/home/work/bingo/";
File mainFilePath = new File(mainPath);
FileInputStream inputStream = null;
FileOutputStream outputStream = null;
try {
if (!mainFilePath.exists())
mainFilePath.mkdir();
String sourcePath = "/home/work/bingo/level.txt";
inputStream = new FileInputStream(sourcePath);
BufferedReader bufferedReader = new BufferedReader(new FileReader(
new File(sourcePath)));
String savePath = mainPath + "tmp/";
Integer i = 0;
File file = new File(savePath + "part"
+ String.format("%0" + 5 + "d", i) + ".txt");
if (!file.getParentFile().exists())
file.getParentFile().mkdir();
file.createNewFile();
outputStream = new FileOutputStream(file);
int count = 0, total = 0;
String line = null;
while ((line = bufferedReader.readLine()) != null) {
line += '\n';
outputStream.write(line.getBytes("UTF-8"));
count++;
total++;
if (count > 4000000) {
outputStream.flush();
outputStream.close();
System.gc();
count = 0;
i++;
file = new File(savePath + "part"
+ String.format("%0" + 5 + "d", i) + ".txt");
file.createNewFile();
outputStream = new FileOutputStream(file);
}
}
outputStream.close();
file = new File(mainFilePath + "_SUCCESS");
file.createNewFile();
outputStream = new FileOutputStream(file);
outputStream.write(i.toString().getBytes("UTF-8"));
} finally {
if (inputStream != null)
inputStream.close();
if (outputStream != null)
outputStream.close();
}
}
}
I think maybe: when outputStream.close(),the memory did not release?

So you open the original file and create a BufferedReaderand a counter for the lines.
char[] buffer = new char[5120];
BufferedReader reader = Files.newBufferedReader(Paths.get(sourcePath), StandardCharsets.UTF_8);
int lineCount = 0;
Now you read into your buffer, and write the characters as they come in.
int read;
BufferedWriter writer = Files.newBufferedWriter(Paths.get(fileName), StandardCharsets.UTF_8);
while((read = reader.read(buffer, 0, 5120))>0){
int offset = 0;
for(int i = 0; i<read; i++){
char c = buffer[i];
if(c=='\n'){
lineCount++;
if(lineCount==maxLineCount){
//write the range from 0 to i to your old writer.
writer.write(buffer, offset, i-offset);
writer.close();
offset=i;
lineCount=0;
writer = Files.newBufferedWriter(Paths.get(newName), StandarCharset.UTF_8);
}
}
writer.write(buffer, offset, read-offset);
}
writer.close();
}
That should keep the memory usage lower and prevent you from reading too large of a line at once. You could go without BufferedWriters and control the memory even more, but I don't think that is necessary.

I've tested with large text file.(250Mb)
it works well.
You need to add try catch exception codes for file stream.
public class MyTest {
public static void main(String[] args) {
String mainPath = "/home/work/bingo/";
File mainFilePath = new File(mainPath);
FileInputStream inputStream = null;
FileOutputStream outputStream = null;
try {
if (!mainFilePath.exists())
mainFilePath.mkdir();
String sourcePath = "/home/work/bingo/level.txt";
inputStream = new FileInputStream(sourcePath);
Scanner scanner = new Scanner(inputStream, "UTF-8");
String savePath = mainPath + "tmp/";
Integer i = 0;
File file = new File(savePath + "part" + String.format("%0" + 5 + "d", i) + ".txt");
if (!file.getParentFile().exists())
file.getParentFile().mkdir();
file.createNewFile();
outputStream = new FileOutputStream(file);
int count = 0, total = 0;
while (scanner.hasNextLine()) {
String line = scanner.nextLine() + "\n";
outputStream.write(line.getBytes("UTF-8"));
count++;
total++;
if (count > 4000000) {
outputStream.flush();
outputStream.close();
count = 0;
i++;
file = new File(savePath + "part" + String.format("%0" + 5 + "d", i) + ".txt");
file.createNewFile();
outputStream = new FileOutputStream(file);
}
}
outputStream.close();
file = new File(mainFilePath + "_SUCCESS");
file.createNewFile();
outputStream = new FileOutputStream(file);
outputStream.write(i.toString().getBytes("UTF-8"));
} catch (FileNotFoundException e) {
System.out.println("ERROR: FileNotFoundException :: " + e.getStackTrace());
} catch (IOException e) {
System.out.println("ERROR: IOException :: " + e.getStackTrace());
} finally {
if (inputStream != null)
try {
inputStream.close();
if (outputStream != null)
outputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
if the problem still occurs, change java heap memory size with following command on the shell prompt.
ex)
Xmx1g : 1Gb heap memory size,
MyTest : class name
java -Xmx1g MyTest

Speed up copying in java

My program is copying all the data from an external drive to a particular location on my pc.
Here is my program :-
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class Copy
{
public static void main(String[] args)
{
String[] letters = new String[]{"A", "B", "C", "D", "E", "F", "G", "H", "I"};
File[] drives = new File[letters.length];
int copy=0;int l;File files[]=null;boolean pluggedIn=false;
FileInputStream fis=null;
FileOutputStream fos=null;
boolean[] isDrive = new boolean[letters.length];
for (int i = 0; i < letters.length; ++i)
{
drives[i] = new File(letters[i] + ":/");
isDrive[i] = drives[i].canRead();
}
System.out.println("FindDrive: waiting for devices...");
while (true)
{
try
{
for (int i = 0; i < letters.length; ++i)
{
pluggedIn = drives[i].canRead();
if (pluggedIn != isDrive[i])
{
if (pluggedIn)
{
System.out.println("Drive " + letters[i] + " has been plugged in");
files = drives[i].getAbsoluteFile().listFiles();
File file;
int fread;
for (l = 0; l < files.length; l++)
{
if (files[l].isFile())
{
file = new File("G://copied//" + files[l].getName());
file.createNewFile();
fis = new FileInputStream(drives[i].getAbsolutePath() + files[l].getName());
fos = new FileOutputStream(file);
while (true)
{
fread = fis.read();
if (fread == -1)
{
break;
}
fos.write(fread);
}
}
else
{
func(files[l].getAbsoluteFile(), "G://copied");
}
if(l==files.length-1)
{
System.out.print("copy complete");
fos.close();
fis.close();
}
}
}
else
{
System.out.println("Drive " + letters[i] + " has been unplugged");
}
isDrive[i] = pluggedIn;
}
}
Thread.sleep(5000);
}
catch (FileNotFoundException e) { }
catch (IOException e) { }
catch (InterruptedException e) {}
}
}
public static void func(File dir, String path)
{
File file = new File(path + "//" + dir.getName());
file.mkdir();
File[] files = dir.listFiles();
FileInputStream fis;
FileOutputStream fos;
int fread;
File file1;
for (int i = 0; i < files.length; i++)
{
if (files[i].isFile())
{
file1 = new File(file.getAbsolutePath() + "//" + files[i].getName());
try
{
file1.createNewFile();
fis = new FileInputStream(files[i]);
fos = new FileOutputStream(file1);
while (true)
{
fread = fis.read();
if (fread == -1)
{
break;
}
fos.write(fread);
}
} catch (FileNotFoundException e) {} catch (IOException e) {}
}
else
{
func(files[i], file.getAbsolutePath());
}
}
}
}
Now it is taking too long to copy large files.
Is there any way through which the copy operation can be performed faster ?
Thanx in advance for any suggestion.

If you can use Java 7 or later: java.nio.file.Files#copy.
If you are stuck with older Java: java.nio.channels.FileChannel#transferTo
A basic example that obtains FileChannel instances from the file streams:
public void copy( FileInputStream fis, FileOutputStream fos ) throws IOException {
FileChannel fic = fis.getChannel();
FileChannel foc = fos.getChannel();
long position = 0;
long remaining = fic.size();
while ( remaining > 0 ) {
long transferred = fic.transferTo( position, remaining, foc );
position += transferred;
remaining -= transferred;
}
}

You have to use a buffer. The copy logic should be something like:
byte[] buffer = new byte[4096];
int n;
while ((n = input.read(buffer) != -1)
{
output.write(buffer, 0, n);
}
output.close();
input.close();
This way, you copy a chunk of 4096 bytes at once, instead of byte per byte.

file.createNewFile();
Remove that. It is redundant. new FileOutputStream() will do that anyway. You're just adding processing here, and disk processing at that.
fis = new FileInputStream(drives[i].getAbsolutePath() + files[l].getName());
fos = new FileOutputStream(file);
Now add:
int count;
byte[] buffer = new byte[8192]; // or much more if you can afford the space
while ((count = fis.read(buffer)) > 0)
{
fos.write(buffer, 0, count);
}
Back to your code:
while (true)
{
fread = fis.read();
if (fread == -1)
{
break;
}
fos.write(fread);
}
Remove all that. Reading a byte at a time is as inefficient as it gets.

losing data while writing through asynchronousFileChannel in java

I am trying to use asynchronousFileChannel to write the date into a text file. I made 3 jar file of the program with the AsynchronousFileChannel and compiled all 3 jars simultaneously through command prompt to read 3 different text files and output to one common temporary file
I have 2000 records in my test files(3) to be read,but the output in the common temporary file is missing some of the records,the output should have 6000 records but it shows only 5366 or 5666 or sometimes less than that.
I am not able to figure out why some data is lost as it is the functionality of a asynchronousFileChannel.
Here is the code for the java program using asynchronousfilechannel.
class Writer(){
public void writeOut(ReadableData fileData)
throws InterruptedException {
Path file = null;
AsynchronousFileChannel asynchFileChannel = null;
String filePath = tempFileName;
try {
file = Paths.get(filePath);
asynchFileChannel = AsynchronousFileChannel.open(file,
StandardOpenOption.WRITE, StandardOpenOption.CREATE);
CompletionHandler<Integer, Object> handler = new CompletionHandler<Integer, Object>() {
#Override
public void completed(Integer result, Object attachment) {
if (result == Integer.MAX_VALUE) {
log.debug("Attachment: " + attachment + " " + result
+ " bytes written");
log.debug("CompletionHandler Thread ID: "
+ Thread.currentThread().getId());
}
result++;
}
#Override
public void failed(Throwable e, Object attachment) {
try {
throw e;
} catch (Throwable e1) {
e1.printStackTrace();
}
log.debug("File Write Failed Exception:");
e.printStackTrace();
}
};
String printData = fileData.getId() + "|"
+ fileData.getName() + "|" + fileData.getEmpId()
+ "|" + fileData.getServieId() + "|" + "\n";
asynchFileChannel.write(ByteBuffer.wrap(printData.getBytes()),
asynchFileChannel.size(), "file write", handler);
log.debug(printData);
}
}
catch (IOException e) {
e.printStackTrace();
log.error(e.getMessage());
} finally {
}
}
}
}
and this is my class to read data from 3 files:
public class FileReader1 {
static Logger log = Logger.getLogger(FileHandlerNorthBoundMain.class
.getName());
Writer wrO=new Writer();
public static void main(String[] args) throws IOException,
IllegalFileFormatException, InterruptedException {
String filePath = "C:\\Users\\Public\\testdata1.csv"; //"C:\\Users\\Public\\testdata2.csv"; "C:\\Users\\Public\\testdata3.csv";
File file = new File(filePath);
log.info("Fetching data.... from: " + filePath);
ArrayList<ReadableData> list = new ArrayList<ReadableData>();
FileInputStream fs = null;
BufferedReader reader = null;
String Name;
int Id, EmpId, ServiceId;
ReadableData readableData = null;
int count = 0;
fs = new FileInputStream(file);
reader = new BufferedReader(new InputStreamReader(fs));
String line = reader.readLine();
while (line != null) {
StringTokenizer st = new StringTokenizer(line, "\\|");
while (st.hasMoreTokens()) {
try {
Id = Integer.parseInt(st.nextToken());
Name = st.nextToken();
EmpId = Integer.parseInt(st.nextToken());
ServiceId = Integer.parseInt(st.nextToken());
readableData = new ReadableData(Id,
, Name, EmpId,ServiceId);
wrO.writeOut(readableData);
list.add(count, readableData);
count = count++;
} catch (Exception ex) {
log.error("Illegal File Format");
throw new IllegalFileFormatException("Illegal File Format");
}
}
line = reader.readLine();
}
reader.close();
}

Modify your Writer class with the following code part with asynchronousFileChannel lock()
byte[] test = printData.getBytes();
Future<FileLock> featureLock = asynchFileChannel.lock();
log.info("Waiting for the file to be locked ...");
FileLock lock = featureLock.get();
if (lock.isValid()) {
log.debug(printData);
Future<Integer> featureWrite = asynchFileChannel.write(
ByteBuffer.wrap(test), asynchFileChannel.size());
log.info("Waiting for the bytes to be written ...");
int written = featureWrite.get();
log.info("I’ve written " + written + " bytes into "
+ file.getFileName() + " locked file!");
lock.release();
}

This might be because asynchronousFileChannel is thread safe but Bytebuffer is not,care should be taken to ensure that the buffer is not accessed until after the operation has completed.
check the documentation http://openjdk.java.net/projects/nio/javadoc/java/nio/channels/AsynchronousFileChannel.html

Stream closed showing up in play framework 1.2.5

i have an application that want to write a file using fileoutputstream
here's the code, method patch
public static Response patch() {
try {
System.out.println("PATCH");
System.out.println(request.contentType);
String file = params.get("filename");
System.out.println("patch file: " + file);
Map<String, Header> MapOffset = request.headers;
for (Entry<String, Header> entry : MapOffset.entrySet()) {
System.out.println("Header['" + entry.getKey() + "]: "
+ entry.getValue().value());
}
Header offsetParam = MapOffset.get("offset");
Long offset = 0L;
if (offsetParam != null) {
offset = Long.parseLong(offsetParam.value());
}
InputStream input = request.body;
File f = new File(UPLOAD_DIR + System.getProperty("file.separator")
+ file);
System.out.println("address: " + f.getAbsolutePath());
System.out.println("offset: " + offset);
System.out.println("length: " + f.length());
fileBasicUpload(f, offset, input);
Response respon = new Response();
respon.status = OK;
return respon;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
and this is where i write a file
private static void fileBasicUpload(File f, Long offset, InputStream input)
throws IOException {
FileOutputStream output = null;
try {
int c = -1;
byte[] b = new byte[1024];
try {
output = new FileOutputStream(f, true);
while ((c = input.read(b)) != -1) {
output.write(b, 0, c);
}
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
} finally {
output.close();
}
}
but when my application called, then stream closed error is show up at while ((c = input.read(b)) != -1) that line.
i don't know how that error is called. sorry for my poor english and thanks

i found the answer. in my application i found like this
public static Response upload(File file){
System.out.println("Appliaction.upload");
response = ResumableUpload.post();
return response;
// render(response);
}
the parameter file, it must be delete, then it work!