Develop Reference

Extracted ZIP file failed - No files inside, and having .zip25 extension after extract

I was trying to extract the ZIP file from my Linux, I'm able to extract it, but the expected output is failing/wrong. The extract file suddenly has no files inside and the folder extracted has .zip25 extension. I searched on this, and there is saying that it is corrupted. However, I don't think it is corrupted because I am able to open and extract the zip files perfectly in local (Windows directory).
Example:
Zip file: FolderZip.zip
After extract: FolderZip.zip25 (Note: This is already extracted but still has .zip25 extension, also the files inside are missing).
Below is my code, I've worked on this for almost a month, but still can't figure it out. Can someone help me to figure out what did I do wrong?
public void unZipFolder(String zipFile, String outputFolder){
byte[] buffer = new byte[1024];
System.out.println("ZipFileLocation: " + zipFile);
LOG.info(" ZipFileLocation: " + zipFile);
File folder = new File(outputFolder);
if(!folder.exists())folder.mkdirs();
try{
FileInputStream fis = new FileInputStream(zipFile);
ZipInputStream zis = new ZipInputStream(fis);
ZipEntry ze = zis.getNextEntry();
while(ze != null) {
new File(folder.getParent()).mkdirs();
FileOutputStream fos = new FileOutputStream(folder);
File newFile = new File(outputFolder + FilenameUtils.indexOfLastSeparator(ze.getName()));
if (ze.isDirectory()) {
if (!newFile.isDirectory() && !newFile.mkdirs()) {
throw new IOException("Failed to create directory " + newFile);
}else if(ze.isDirectory()){
newFile.mkdirs();
continue;
}else{
int len;
while ((len = zis.read(buffer)) >= 0) {
fos.write(buffer, 0, len);
}
System.out.println("File Unzip: " + newFile);
LOG.info(" File Unzip: " + newFile);
newFile.mkdirs();
fos.close();
zis.closeEntry();
ze = zis.getNextEntry();
}
}
boolean result = Files.deleteIfExists(Paths.get(zipFile));
if (result) {
System.out.println("ZipFile is deleted....");
} else {
System.out.println("Unable to delete the file.....");
}
}
zis.closeEntry();
zis.close();
fis.close();
}catch(IOException ex){
ex.printStackTrace();
}
}

I'd love to be able to tell you exactly what's wrong with your code, but FileOutputStream fos = new FileOutputStream(folder); throws an exception because, well, folder is, a directory, so you can't write to it.
I'm also scratching my head over what your expecting new File(folder.getParent()).mkdirs(); to do.
I basically threw out your code and started again with...
public void unZipFolder(File zipFile, File outputFolder) throws IOException {
byte[] buffer = new byte[1024];
System.out.println("ZipFileLocation: " + zipFile);
System.out.println("outputFolder = " + outputFolder);
if (!outputFolder.exists() && !outputFolder.mkdirs()) {
throw new IOException("Unable to create output folder: " + outputFolder);
} else if (outputFolder.exists() && !outputFolder.isDirectory()) {
throw new IOException("Output is not a directory: " + outputFolder);
}
try (ZipFile zipper = new ZipFile(zipFile)) {
Enumeration<? extends ZipEntry> entries = zipper.entries();
while (entries.hasMoreElements()) {
ZipEntry ze = entries.nextElement();
File destination = new File(outputFolder, ze.getName());
if (ze.isDirectory()) {
if (!destination.exists() && !destination.mkdirs()) {
throw new IOException("Could not create directory: " + destination);
}
} else {
System.out.println("Writing " + destination);
try (InputStream is = zipper.getInputStream(ze); FileOutputStream fos = new FileOutputStream(destination)) {
// You could use is.transferTo(fos) here but I'm a grump old coder
byte[] bytes = new byte[1024 * 4];
int bytesRead = -1;
while ((bytesRead = is.read(bytes)) != -1) {
fos.write(bytes, 0, bytesRead);
}
}
}
}
}
}
Now, what's important to know about this is, it expects the directory contents of the zip files to be relative (ie no root directory information). If your zip file does contain root directory information (ie C:/... or /...), then you're going to need to clean that up yourself.
Now, if you have trouble with this, I would suggest commenting out the "extraction" portion of the code and placing in more System.out.println statements
transferTo
After reading through the code for transferTo, it's basically doing the same thing that the code example above is doing - so, if you wanted to reduce the code complexity (and reduce the risk of possible bugs), you could use it - been some what old school, I'd probably still do it the "old way" in order to provide support for progress monitoring of some kind - but that's beyond the scope of the question.
"Security issues"
This ones a little harder to tie down, as no solution is 100% safe.
I modified the above code to use something like...
Path parent = outputFolder.toPath().toAbsolutePath();
String name = "../" + ze.getName();
Path child = parent.resolveSibling(new File(outputFolder, name).toPath());
And this ended up throwing a NoSuchFileException, so, at least you could "fail fast", assuming that's what you want.
You might also consider removing .., leading / or leading path specifications in an attempt to make the path "relative", but that could become complicated as something like somePath/../file could still be valid within your use case.

File upload/download using nfs-client package not working after n iteration

I am using nfs-client package to realize the NFS protocol to connect to the linux server for file upload/download. Basically the purpose is NFS protocol testing .
This is my sample codebase
import com.emc.ecs.nfsclient.nfs.io.Nfs3File;
import com.emc.ecs.nfsclient.nfs.io.NfsFileInputStream;
import com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream;
import com.emc.ecs.nfsclient.nfs.nfs3.Nfs3;
import com.emc.ecs.nfsclient.rpc.CredentialUnix;
int thread_count = ctx.getThreadNum();
String lrandom_name = service_name + "_" + replica_id + "_" + thread_count;
public static void uploadFileToNfs() {
String localDir = "F:\\look\\"+lrandom_name;
InputStream inputStream = null;
OutputStream outputStream = null;
try {
//Create a local file object
File localFile = new File(localDir);
//Get the file name of the local file, this name is used to create a file with the same name in the specified directory on the remote Nfs server
String localFileName = localFile.getName();
Nfs3 nfs3 = new Nfs3(NFS_IP, NFS_DIR, new CredentialUnix(0, 0, null), 3);
//Create Nfs file object on remote server
Nfs3File NfsFile = new Nfs3File(nfs3, "/" + localFileName);
//Open a file input stream
inputStream = new BufferedInputStream(new FileInputStream(localFile));
//Open a remote Nfs file output stream and copy the file to the destination
outputStream = new BufferedOutputStream(new NfsFileOutputStream(NfsFile));
//Buffer memory
byte[] buffer = new byte[1024];
while ((inputStream.read(buffer)) != -1) {
outputStream.write(buffer);
}
System.out.println("File upload complete!");
} catch (Exception ex) {
ex.printStackTrace();
} finally {
try {
if (outputStream != null) {
outputStream.close();
}
if (inputStream != null) {
inputStream.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
I am running the script from Jmeter , but after n number of iteration (threaded manner) I am getting below error. Similar script for File download from NFS is working fine in threaded manner parallelly .
com.emc.ecs.nfsclient.nfs.NfsException: rpc error, server: 192.168.0.101, RPC error: RPC call is ACCEPTED, but the status is not success, acceptStat=4
at com.emc.ecs.nfsclient.rpc.RpcWrapper.handleRpcException(RpcWrapper.java:311)
at com.emc.ecs.nfsclient.rpc.RpcWrapper.callRpcWrapped(RpcWrapper.java:159)
at com.emc.ecs.nfsclient.nfs.nfs3.Nfs3.wrapped_sendWrite(Nfs3.java:756)
at com.emc.ecs.nfsclient.nfs.nfs3.Nfs3.wrapped_sendWrite(Nfs3.java:90)
at com.emc.ecs.nfsclient.nfs.io.NfsFileBase.write(NfsFileBase.java:866)
at com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream.writeBufferToFile(NfsFileOutputStream.java:296)
at com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream.write(NfsFileOutputStream.java:262)
at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:127)
at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
at java_io_FilterOutputStream$write.call(Unknown Source)
at com.emc.ecs.nfsclient.nfs.io.Script1.run(Script1.groovy:77)
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317)
at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71)
at java.scripting/javax.script.CompiledScript.eval(CompiledScript.java:89)
at org.apache.jmeter.util.JSR223TestElement.processFileOrScript(JSR223TestElement.java:217)
at org.apache.jmeter.protocol.java.sampler.JSR223Sampler.sample(JSR223Sampler.java:72)
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.base/java.lang.Thread.run(Thread.java:834)
Update
Based on feedback instead of one static file for all Threads(Users) , now I have unique file for each Threads(Users) to upload/download
Update
Tried below one more use case randomly picking file from one NFS share folder to other ,same issue coming. Suspecting something wrong with NFS configuration only
package com.emc.ecs.nfsclient.nfs.io;
import org.junit.Test;
import com.emc.ecs.nfsclient.nfs.NfsSetAttributes;
import com.emc.ecs.nfsclient.nfs.io.Nfs3File;
import com.emc.ecs.nfsclient.nfs.io.NfsFileInputStream;
import com.emc.ecs.nfsclient.nfs.io.NfsFileOutputStream;
import com.emc.ecs.nfsclient.nfs.nfs3.Nfs3;
import com.emc.ecs.nfsclient.rpc.CredentialUnix;
import java.util.Random;
import java.io.IOException;
import org.junit.Test;
String nfs_folder = vars.get("nfs_folder");
String files_folder = vars.get("files_folder");
String NFS_IP = vars.get("server_ip");
String NFS_DIR = "/"+nfs_folder+"/"+files_folder;
String NFS_DIR_ROOT = "/"+nfs_folder;
int thread_count = ctx.getThreadNum();
String service_name = vars.get("Service_Name");
String replica_id = vars.get("Replica_ID");
String lrandom_name = service_name + "_" + replica_id + "_" + thread_count;
int transfer_size = Integer.parseInt(vars.get("transfer_size"))
InputStream inputStream = null;
OutputStream outputStream = null;
try {
Nfs3 nfs3 = new Nfs3(NFS_IP, NFS_DIR, new CredentialUnix(0, 0, null), 3);
Nfs3 nfs4 = new Nfs3(NFS_IP, NFS_DIR_ROOT, new CredentialUnix(0, 0, null), 3);
nfsFiles = new Nfs3File(nfs3,"/");
List<String> children = nfsFiles.list();
Random randomGenerator = new Random();
int fileIndex = randomGenerator.nextInt(children.size());
OUT.println(fileIndex)
String NfsFileDir = "/"+children.get(fileIndex);
OUT.println(children.size())
OUT.println(children.get(fileIndex))
//Create Nfs file object on remote server
Nfs3File nfsFile = new Nfs3File(nfs3, NfsFileDir);
Nfs3File NfsFile = new Nfs3File(nfs4, "/" + lrandom_name);
//String localFileName = localDir + lrandom_name;
//Create a local file object
//File localFile = new File(localFileName);
//Open a file input stream
inputStream = new BufferedInputStream(new NfsFileInputStream(nfsFile));
//Open a remote Nfs file output stream and copy the file to the destination
outputStream = new BufferedOutputStream(new NfsFileOutputStream(NfsFile));
//Buffer memory
byte[] buffer = new byte[transfer_size];
while (inputStream.read(buffer) != -1) {
outputStream.write(buffer);
}
System.out.println("File copy complete!");
} catch (IOException ex) {
ex.printStackTrace();
} finally {
try {
if (outputStream != null) {
outputStream.close();
}
if (inputStream != null) {
inputStream.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}

I don't think your use case is valid as uploading the same file into the same destination is not something what real NFS users would be doing, so maybe it worth considering using i.e. Directory Listing Config plugin so each virtual user will have its own file(s) to upload/download
Looking at NFS RPC RFC the status you're getting is:
GARBAGE_ARGS = 4, /* procedure can't decode params */
so double check that your JMeter instance has enough headroom to operate in terms of JVM setup, CPU, RAM, etc. as it looks like that your NFS server cannot properly parse the request so it might be the case JMeter sends some trash. Ensure to follow JMeter Best Practices
Basically the same as point 2 but for NFS server side, it might be the case it is overloaded hence cannot properly parse the incoming valid requests, check NFS logs and your operating system logs for any suspicious entries.

compressing file by independent chunks and then concat them into one valid archive

I wonder if it is possible to compress an arbitrary file (or folder, or any other file structure) by independent chunks and then get a valid archive (e.g. gzip) by concatenating them together. Some requirements:
java 8
chunks <= 16MB
folder structure does not change during the process
chunks are compressed independently, but order is preserved
each compressed chunk is appended to the end of the resulting archive
resulting archive should be valid and decompressable by any standard tool
It looks like to achieve that I would need to create an archive header first and then just append compressed blocks to it https://www.rfc-editor.org/rfc/rfc1952, however I'm not sure if it is supported by any of standard java utils or 3rd party libraries. Does anybody have any ideas on where to start from?
Some background:
I have a client-server app, which allows user to upload files to a cloud storage. Communication via REST api, client side is going to be responsible for dividing files into chunks and upload them one by one. It is possible to do compression in browser, however I wonder if we can move that load to the backend.

Yes. A concatenation of gzip files is a valid gzip file, per the standard (RFC 1952). gzip certainly handles this.
You are correct to be concerned that some code out there might not support it, since it is not very common to have concatenated gzip members. If you want to be super-safe, you can combine the gzip files into a single gzip member, without having to recompress. You do however need to read through all of the compressed data, effectively decompressing it in memory (which is still much faster than compressing). You can find an example of that in gzjoin.c.

You can try something like this for tar + gzip:
Maven dependency:
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.18</version>
</dependency>
Java code to compress into chunks:
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream;
import org.apache.commons.compress.utils.IOUtils;
import java.io.*;
import java.nio.file.Files;
import java.nio.file.Paths;
[..]
private static final int MAX_CHUNK_SIZE = 16000000;
public void compressTarGzChunks(String inputDirPath, String outputDirPath) throws Exception {
PipedInputStream in = new PipedInputStream();
final PipedOutputStream out = new PipedOutputStream(in);
new Thread(() -> {
try {
int chunkIndex = 0;
int n = 0;
byte[] buffer = new byte[8192];
do {
String chunkFileName = String.format("archive-part%d.tar.gz", chunkIndex);
try (OutputStream fOut = Files.newOutputStream(Paths.get(outputDirPath, chunkFileName));
BufferedOutputStream bOut = new BufferedOutputStream(fOut);
GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(bOut)) {
int currentChunkSize = 0;
if (chunkIndex > 0) {
gzOut.write(buffer, 0, n);
currentChunkSize += n;
}
while ((n = in.read(buffer)) != -1 && currentChunkSize + n < MAX_CHUNK_SIZE) {
gzOut.write(buffer, 0, n);
currentChunkSize += n;
}
chunkIndex++;
}
} while (n != -1);
in.close();
} catch (IOException e) {
// logging and exception handling should go here
}
}).start();
try (TarArchiveOutputStream tOut = new TarArchiveOutputStream(out)) {
compressTar(tOut, inputDirPath, "");
}
}
private static void compressTar(TarArchiveOutputStream tOut, String path, String base)
throws IOException {
File file = new File(path);
String entryName = base + file.getName();
TarArchiveEntry tarEntry = new TarArchiveEntry(file, entryName);
tarEntry.setSize(file.length());
tOut.putArchiveEntry(tarEntry);
if (file.isFile()) {
try (FileInputStream in = new FileInputStream(file)) {
IOUtils.copy(in, tOut);
tOut.closeArchiveEntry();
}
} else {
tOut.closeArchiveEntry();
File[] children = file.listFiles();
if (children != null) {
for (File child : children) {
compressTar(tOut, child.getAbsolutePath(), entryName + "/");
}
}
}
}
Java code to concatenate the chunks into a single archive:
public void concatTarGzChunks(List<InputStream> sortedTarGzChunks, String outputFile) throws IOException {
try {
try (FileOutputStream fos = new FileOutputStream(outputFile)) {
for (InputStream in : sortedTarGzChunks) {
int len;
byte[] buf = new byte[1024 * 1024];
while ((len = in.read(buf)) != -1) {
fos.write(buf, 0, len);
}
}
}
} finally {
sortedTarGzChunks.forEach(is -> {
try {
is.close();
} catch (IOException e) {
// logging and exception handling should go here
}
});
}
}

Reading a block of bytes from one file and writing to other until all blocks are read?

I am working a project in which I have to play with some file reading writing tasks. I have to read 8 bytes from a file at one time and perform some operations on that block and then write that block to second file, then repeat the cycle until first file is completely read in chuncks of 8 bytes everytime and the after manipulation the data should be added/appended to the second. However, in doing so, I am facing some problems. Following is what I am trying:
private File readFromFile1(File file1) {
int offset = 0;
long message= 0;
try {
FileInputStream fis = new FileInputStream(file1);
byte[] data = new byte[8];
file2 = new File("file2.txt");
FileOutputStream fos = new FileOutputStream(file2.getAbsolutePath(), true);
DataOutputStream dos = new DataOutputStream(fos);
while(fis.read(data, offset, 8) != -1)
{
message = someOperation(data); // operation according to business logic
dos.writeLong(message);
}
fos.close();
dos.close();
fis.close();
} catch (IOException e) {
System.out.println("Some error occurred while reading from File:" + e);
}
return file2;
}
I am not getting the desired output this way. Any help is appreciated.

Consider the following code:
private File readFromFile1(File file1) {
int offset = 0;
long message = 0;
File file2 = null;
try {
FileInputStream fis = new FileInputStream(file1);
byte[] data = new byte[8]; //Read buffer
byte[] tmpbuf = new byte[8]; //Temporary chunk buffer
file2 = new File("file2.txt");
FileOutputStream fos = new FileOutputStream(file2.getAbsolutePath(), true);
DataOutputStream dos = new DataOutputStream(fos);
int readcnt; //Read count
int chunk; //Chunk size to write to tmpbuf
while ((readcnt = fis.read(data, 0, 8)) != -1) {
//// POINT A ////
//Skip chunking system if an 8 byte octet is read directly.
if(readcnt == 8 && offset == 0){
message = someOperation(tmpbuf); // operation according to business logic
dos.writeLong(message);
continue;
}
//// POINT B ////
chunk = Math.min(tmpbuf.length - offset, readcnt); //Determine how much to add to the temp buf.
System.arraycopy(data, 0, tmpbuf, offset, chunk); //Copy bytes to temp buf
offset = offset + chunk; //Sets the offset to temp buf
if (offset == 8) {
message = someOperation(tmpbuf); // operation according to business logic
dos.writeLong(message);
if (chunk < readcnt) {
System.arraycopy(data, chunk, tmpbuf, 0, readcnt - chunk);
offset = readcnt - chunk;
} else {
offset = 0;
}
}
}
//// POINT C ////
//Process remaining bytes here...
//message = foo(tmpbuf);
//dos.writeLong(message);
fos.close();
dos.close();
fis.close();
} catch (IOException e) {
System.out.println("Some error occurred while reading from File:" + e);
}
return file2;
}
In this excerpt of code, what I did was:
Modify your reading code to include the amount of bytes actually read from the read() method (noted readcnt).
Added a byte chunking system (the processing does not happen until there are at least 8 bytes in the chunking buffer).
Allowed for separate processing of the final bytes (that do not make up a 8 byte octet).
As you can see from the code, the data being read is first stored in a chunking buffer (denoted tmpbuf) until at least 8 bytes are available. This will happen only if 8 bytes are not always available (If 8 bytes are available directly and nothing is chunked, directly process. See "Point A" in code). This is done as a form of optimization to prevent excess array copies.
The chunking system uses offsets which increment every time bytes are written to tmpbuf until it reaches a value of 8 (it will not go over as the Math.min() method used in the assignment of 'chunk' will limit the value). Upon offset == 8, proceed to execute the processing code.
If that particular read produced more bytes than actually processed, continue writing them to tmpbuf, from the beginning again, whilst setting offset appropriately, otherwise set offset to 0.
Repeat cycle.
The code will leave the last few bytes of data that do not fit in an octet in the array tmpbuf with the offset variable indicating how much has actually been written. This data can then be processed separately at point C.
Seems a lot more complicating than it should be, and there probably is a better solution (possibly using existing java library methods), but off the top of my head, this is what I got. Hope this is clear enough for you to understand.

You could use the following, it uses NIO and especially the ByteBuffer class for the long handling. You can of course implement it the standard java way, but since i am a NIO fan, here is a possible solution.
The major problem in your code is that while(fis.read(data, offset, 8) != -1) will read up to 8 bytes, and not always 8 bytes, plus reading in such small portions is not very efficient.
I have put some comments in my code, if something is unclear please leave a comment. My someOperation(...) function just copies the next long value from the buffer.
Update:
added finally block to close the files.
import java.io.File;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
public class TestFile {
static final int IN_BUFFER_SIZE = 1024 * 8;
static final int OUT_BUFFER_SIZE = 1024 *9; // make the out-buffer > in-buffer, i am lazy and don't want to check for overruns
static final int MIN_READ_BYTES = 8;
static final int MIN_WRITE_BYTES = 8;
private File readFromFile1(File inFile) {
final File outFile = new File("file2.txt");
final ByteBuffer inBuffer = ByteBuffer.allocate(IN_BUFFER_SIZE);
final ByteBuffer outBuffer = ByteBuffer.allocate(OUT_BUFFER_SIZE);
FileChannel readChannel = null;
FileChannel writeChannel = null;
try {
// open a file channel for reading and writing
readChannel = FileChannel.open(inFile.toPath(), StandardOpenOption.READ);
writeChannel = FileChannel.open(outFile.toPath(), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
long totalReadByteCount = 0L;
long totalWriteByteCount = 0L;
boolean readMore = true;
while (readMore) {
// read some bytes into the in-buffer
int readOp = 0;
while ((readOp = readChannel.read(inBuffer)) != -1) {
totalReadByteCount += readOp;
} // while
// prepare the in-buffer to be consumed
inBuffer.flip();
// check if there where errors
if (readOp == -1) {
// end of file reached, read no more
readMore = false;
} // if
// now consume the in-buffer until there are at least MIN_READ_BYTES in the buffer
while (inBuffer.remaining() >= MIN_READ_BYTES) {
// add data to the write buffer
outBuffer.putLong(someOperation(inBuffer));
} // while
// compact the in-buffer and prepare for the next read, if we need to read more.
// that way the possible remaining bytes of the in-buffer can be consumed after leaving the loop
if (readMore) inBuffer.compact();
// prepare the out-buffer to be consumed
outBuffer.flip();
// write the out-buffer until the buffer is empty
while (outBuffer.hasRemaining())
totalWriteByteCount += writeChannel.write(outBuffer);
// prepare the out-buffer for writing again
outBuffer.flip();
} // while
// error handling
if (inBuffer.hasRemaining()) {
System.err.println("Truncated data! Not a long value! bytes remaining: " + inBuffer.remaining());
} // if
System.out.println("read total: " + totalReadByteCount + " bytes.");
System.out.println("write total: " + totalWriteByteCount + " bytes.");
} catch (IOException e) {
System.out.println("Some error occurred while reading from File: " + e);
} finally {
if (readChannel != null) {
try {
readChannel.close();
} catch (IOException e) {
System.out.println("Could not close read channel: " + e);
} // catch
} // if
if (writeChannel != null) {
try {
writeChannel.close();
} catch (IOException e) {
System.out.println("Could not close write channel: " + e);
} // catch
} // if
} // finally
return outFile;
}
private long someOperation(ByteBuffer bb) {
// consume the buffer, do whatever you want with the buffer.
return bb.getLong(); // consumes 8 bytes of the buffer.
}
public static void main(String[] args) {
TestFile testFile = new TestFile();
File source = new File("input.txt");
testFile.readFromFile1(source);
}
}

Faster way of copying data in Java?

I have been given a task of copying data from a server. I am using BufferedInputStream and output stream to copy the data and I am doing it byte by byte. Even though it is running but It is taking ages to copy the data as some of them are in 100's MBs, so definitely it is not gonna work. Can anyone suggest me any alternate of Byte by Byte copy so that my code can copy file that are in few Hundred MBs.
Buffer is 2048.
Here is how my code look like:
static void copyFiles(SmbFile[] files, String parent) throws IOException {
SmbFileInputStream input = null;
FileOutputStream output = null;
BufferedInputStream buf_input = null;
try {
for (SmbFile f : files) {
System.out.println("Working on files :" + f.getName());
if (f.isDirectory()) {
File folderToBeCreated = new File(parent+f.getName());
if (!folderToBeCreated.exists()) {
folderToBeCreated.mkdir();
System.out.println("Folder name " + parent
+ f.getName() + "has been created");
} else {
System.out.println("exists");
}
copyFiles(f.listFiles(), parent + f.getName());
} else {
input = (SmbFileInputStream) f.getInputStream();
buf_input = new BufferedInputStream(input, BUFFER);
File t = new File(parent + f.getName());
if (!t.exists()) {
t.createNewFile();
}
output = new FileOutputStream(t);
int c;
int count;
byte data[] = new byte[BUFFER];
while ((count = buf_input.read(data, 0, BUFFER)) != -1) {
output.write(data, 0, count);
}
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (input != null) {
input.close();
}
if (output != null) {
output.close();
}
}
}

Here is a link to an excellent post explaining how to use nio channels to make copies of streams. It introduces a helper method ChannelTools.fastChannelCopy that lets you copy streams like this:
final InputStream input = new FileInputStream(inputFile);
final OutputStream output = new FileOutputStream(outputFile);
final ReadableByteChannel inputChannel = Channels.newChannel(input);
final WriteableByteChannel outputChannel = Channels.newChannel(output);
ChannelTools.fastChannelCopy(inputChannel, outputChannel);
inputChannel.close();
outputChannel.close()

Well since you're using a BufferedInputStream, you aren't reading byte by byte, but rather the size of the buffer. You could just try increasing the buffer size.

Reading/writing byte-by-byte is definitely going to be slow, even though the actual reading/writing is done by chunks of the buffer size. One way to speed it up is to read/write by blocks. Have a look at read(byte[] b, int off, int len) method of BufferedInputStream. However it probably won't give you enough of the improvement.
What would be much better is to use nio package (new IO) to copy data using nio channels. Have a look at nio documentation for more info.

I would suggest to use FileUtils from org.apache.commons.io. It has enough utility methods to perform file operations.
org.apache.commons.io.FileUtils API Here