Develop Reference

Improve performance when reading file from URL and writing it to disk

I made a program which accesses some URLs and downloads the pdfs from there. The files vary between 2MB to 40MB. The program works with no problems but is there a way to improve the perfomance on this? For the larger files it takes a long time to do it.
The code below is the one used for reading / writing the file. This is called in a for loop with different fileNameURLPath.
#Override
public void downloadFile(String fileNameURLPath, String titleCellValue) throws FileException {
try (BufferedInputStream inputStream
= new BufferedInputStream(new URL(fileNameURLPath).openStream())){
FileOutputStream fileOS = new FileOutputStream(FileConstants.MandatoryDownloadProperties.path + titleCellValue + ".pdf");
byte data[] = new byte[32*1024];
int byteContent;
while((byteContent = inputStream.read(data,0 , data.length)) != -1) {
fileOS.write(data, 0 , byteContent);
}
inputStream.close();
fileOS.close();
} catch (MalformedURLException e) {
throw new FileException("Error while processing url. Make sure it is correct");
} catch (IOException e) {
throw new FileException("Error while downloading file. Make sure the download path is correct");
}
}
I read something about Java NIO but I couldn't quite comprehend it or if it can help me in this situation

Error where FileOutputStream only writes to file after the program has been terminated

I've had this error in the past but never fully understood it. After closing an OutputStream, regardless of the location of the java file or the manner in which it is called, completely screws up all sequential runs or attempts to write to another file, even if a different method of writing to a file is used. For this reason I avoid closing streams even though it is a horrible habit not to. In my program, I created was trying a test case that had a close statement which destroyed all of my previous streams, making it for some reason that they only write to files after the program has been terminated.
I kept the file location open and it writes the Text in the text file at the appropriate time, however the "Preview" panel in Windows does not detect it (which used to happen). Note that this all worked perfectly before the stream was accidentally closed. Is there a manner to reset the stream? I've tried flushing it during the process but is still does not run as it did prior.
Here is the method used to create the file:
protected void createFile(String fileName, String content) {
try {
String fileLoc = PATH + fileName + ".txt";
File f = new File(fileLoc);
if(!f.isFile())
f.createNewFile();
FileOutputStream outputStream = new FileOutputStream(fileLoc);
byte[] strToBytes = content.getBytes();
outputStream.write(strToBytes);
} catch (IOException e) {
e.printStackTrace();
return;
}
}
as well as the method used to read the file:
protected String readFile(String fileName) {
try {
StringBuilder sb = new StringBuilder("");
String fileLoc = PATH + fileName + ".txt";
File f = new File(fileLoc);
if(!f.exists())
return "null";
Scanner s = new Scanner(f);
int c = 0;
while(s.hasNext()) {
String str = s.nextLine();
sb.append(str);
if(s.hasNext())
sb.append("\n");
}
return sb.toString();
} catch(Exception e) {
e.printStackTrace();
return "null";
}
}
I'd be happy to answer any clarification questions if needed. Thank you for the assistance.

without try-resource, you need close in final clause to make sure no leak. Or use Stream.flush() if you need more 'in-time' update.
} catch (IOException e) {
e.printStackTrace();
return;
} finally {
outputStream.close();
}

You need to call flush() on the stream to write the bytes to the stream.
You're currently calling write() by itself, like this:
FileOutputStream outputStream = new FileOutputStream(fileLoc);
outputStream.write(content.getBytes());
What you want to do is this:
FileOutputStream outputStream = new FileOutputStream(fileLoc);
outputStream.write(content.getBytes());
outputStream.flush();
From the Javadoc (https://docs.oracle.com/javase/8/docs/api/java/io/OutputStream.html#flush--) for OutputStream (where FileOutputStream is an OutputStream), this is what it says for flush():
Flushes this output stream and forces any buffered output bytes to be written out. The general contract of flush is that calling it is an indication that, if any bytes previously written have been buffered by the implementation of the output stream, such bytes should immediately be written to their intended destination.
Even better would be to close the stream in a finally block, so that no matter what your code always tries to free up any open resources, like this:
FileOutputStream outputStream = null;
try {
outputStream = new FileOutputStream(fileLoc);
outputStream.write(content.getBytes());
outputStream.flush();
} finally {
if (outputStream != null) {
outputStream.close();
}
}
or use automatic resource management, like this:
try (FileOutputStream outputStream = new FileOutputStream(fileLoc)) {
outputStream.write(content.getBytes());
outputStream.flush();
}

java : corrupted zip file created when copy using nio

I have implement following code to copy file(binary file)
code
private void copyFileWithChannels(File aSourceFile, File aTargetFile) {
log("Copying files with channels.");
FileChannel inChannel = null;
FileChannel outChannel = null;
FileInputStream inStream = null;
FileOutputStream outStream = null;
try {
inStream = new FileInputStream(aSourceFile);
inChannel = inStream.getChannel();
outStream = new FileOutputStream(aTargetFile);
outChannel = outStream.getChannel();
long bytesTransferred = 0;
while(bytesTransferred < inChannel.size()){
bytesTransferred += inChannel.transferTo(0, inChannel.size(), outChannel);
}
}
catch(FileNotFoundException e){
log.error("FileNotFoundException in copyFileWithChannels()",e);
}
catch (IOException e) {
log.error("IOException in copyFileWithChannels()",e);
}
catch (Exception e) {
log.error("Exception in copyFileWithChannels()",e);
}
finally {
try{
if (inChannel != null) inChannel.close();
if (outChannel != null) outChannel.close();
if (inStream != null) inStream.close();
if (outStream != null) outStream.close();
}catch(Exception e){
log.error("Exception in copyFileWithChannels() while closing the stream",e);
}
}
}
I have test code with one zip file. when i verify file I found that file which generated is corrupt(size was increased).
Source zip file is about 9GB.

Try this:
while(bytesTransferred < inChannel.size()){
bytesTransferred += inChannel.transferTo(bytesTransferred, inChannel.size() - bytesTransferred, outChannel);
}
Also, I would refer to IOUtils implementation, as a reference
https://github.com/apache/commons-io/blob/master/src/main/java/org/apache/commons/io/FileUtils.java
specifically
private static void doCopyFile(final File srcFile, final File destFile, final boolean preserveFileDate)

The transferTo method's first argument gives the position from which to transfer, not relative to where the stream left off, but relative to the start of the file. Since you put 0 there it will always transfer from the start of the file. So that line needs to be
bytesTransferred += inChannel.transferTo(bytesTransferred , inChannel.size(), outChannel);
mavarazy mentioned in his answer he's not sure if you need a loop when using inChannel.size(), since the expectation is that if you supply the whole size it will copy the entire file. However, the actual transfer might be less than the requested number of bytes if the output channel's buffer has less room available. So you do need the loop as in his second code snippet.

Unless you have a good reason best to use Files.copy(Path, Path, CopyOption...).

Writing to an External File on Android --- File Doesn't Register, But Java Can Read

I'm trying to write to an external txt (or csv) file for Android. I can run an app, close it, and run it again, and readData() will read back to my log what I've stored. However, the dirFile (file directory) appears nowhere within my Android files (even if I connect it to a computer and search).
Something interesting, though: if I clear my log (similar to a list of print statements shown within Eclipse) and disconnect my phone from my computer, then reconnect it, the log reappears with everything I've ever written to my file (even if I later overwrote it)...yet the app isn't even running!
Here is my code. Please help me understand why I cannot find my file!
(Note: I've tried appending a "myFile.txt" extension to the directory, but it just causes an EISDIR exception.)
public void writeData(String dirName){
try
{
File root = new File(getExternalFilesDir(null), dirName);
// Writes to file
//
// The "true" argument allows the file to be appended. Without this argument (just root),
// the file will be overwritten (even though we later call append) rather than appended to.
FileWriter writer = new FileWriter(root, true);
writer.append("Append This Text\n");
writer.flush();
writer.close();
// Checks if we actually wrote to file by reading it back in (appears in Log)
//readData(dirName);
}
catch(Exception e)
{
Log.v("2222", "2222 ERROR: " + e.getMessage());
}
}
If you're interested, here's the function I wrote to read in the data:
public void readData(String dirName){
try
{
File root = new File(getExternalFilesDir(null), dirName);
// Checks to see if we are actually writing to file by reading in the file
BufferedReader reader = new BufferedReader(new FileReader(root));
try {
String s = reader.readLine();
while (s != null) {
Log.v("2222", "2222 READ: " + s);
s = reader.readLine();
}
}
catch(Exception e) {
Log.v("2222", "2222 ERROR: " + e.getMessage());
}
finally {
reader.close();
}
}
catch(Exception e) {
Log.v("2222", "2222 ERROR: " + e.getMessage());
}
}
Thanks!

even if I connect it to a computer and search
if I clear my log (similar to a list of print statements shown within Eclipse) and disconnect my phone from my computer, then reconnect it, the log reappears with everything I've ever written to my file (even if I later overwrote it).
What you are seeing on your computer is what is indexed by MediaStore, and possibly a subset of those, depending upon whether your computer caches information it gets from the device in terms of "directory" contents.
To help ensure that MediaStore indexes your file promptly:
Use a FileOutputStream (optionally wrapped in an OutputStreamWriter), not a FileWriter
Call flush(), getFD().sync(), and close() on the FileOutputStream, instead of calling flush() and close() on the FileWriter (sync() will ensure the bytes are written to disk before continuing)
Use MediaScannerConnection and scanFile() to tell MediaStore to index your file
You can then use whatever sort of "reload" or "refresh" or whatever option is in your desktop OS's file manager, and your file should show up.
This blog post has more on all of this.

public void create(){
folder = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_MOVIES),"video");
boolean success = true;
if (!folder.exists()) {
success=folder.mkdirs();
}
if (success) {
readfile();
} else {
System.out.println("failed");
}
}
The above code will be used to crete the directory in th emobile at desired path
private void readfile() {
// TODO Auto-generated method stub
AssetManager assetManager = getResources().getAssets();
String[] files = null;
try {
files = assetManager.list("clipart");
} catch (Exception e) {
Log.e("read clipart ERROR", e.toString());
e.printStackTrace();
}
for(String filename : files) {
System.out.println("File name => "+filename);
InputStream in = null;
OutputStream out = null;
try {
in = assetManager.open("clipart/" + filename);
out = new FileOutputStream(folder + "/" + filename);
copyFile(in, out);
in.close();
in = null;
out.flush();
out.close();
out = null;
} catch(Exception e) {
Log.e("copy clipart ERROR", e.toString());
e.printStackTrace();
}
}}private void copyFile(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[1024];
int read;
while((read = in.read(buffer)) != -1){
out.write(buffer, 0, read);
}}
this is my code used to write file in internal memory from the assets folder in project. This code can read all type(extension) of file from asset folder to mobile.
Don't forget to add permission in manifest file
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
and call the above function by
readfile();//this call the function to read and write the file
I hope this may help you.
Thank you.

A Problem while Retriving file from FTPS Server

I am working in an application development. On that application i am performing files store, retrieve and delete operations. For identifying the files on server i am using an index(a hash map file) file. every time when i perform upload operation i update "index" file and upload "index" file on server along with other uploading files.
For performing delete operation first i am retrieving the "index" file and based on the index i am deleting the files from server and after updating "index" file i again upload "index" file on server.
I am able to perform file uploading operation successfully but while performing delete operation, i am getting "java.io.EOFException" exception, when i am trying to retrieve "index" file.
i am writing following code to download "index" file from FTPS server
//download index file
if (service.retrFile("INDEX", "") == service.OK) {
try {
ObjectInputStream objIn = new ObjectInputStream(new FileInputStream("INDEX"));
try {
Map<String, FileData> filesUploaded = (HashMap<String, FileData>) objIn.readObject();
} catch (ClassNotFoundException ex) {
ex.printStackTrace();
}
objIn.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
Where "service.ok" returns '0' if it is successfully connected to FTPS server
and "FileData" contains information about file(attributes).
Same code i am using while performing uploading operation. there it is working fine with no exception. but while performing delete operation when i am retrieving "index" file i am getting exception on the statement :
Map filesUploaded = (HashMap) objIn.readObject();
Exception is :
SEVERE: null
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2298)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2767)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:798)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:298)
at com.pixelvault.gui.DeleteServerFilesDialog.startDeleting(DeleteServerFilesDialog.java:447)
I have checked whether FTPS server connections are properly closed after performing corresponding operations.
I am not getting where i am doing wrong.
please give me your valuable suggestions. i thank to all your suggestions which will help me to overcome with this problem.
i am using org.apache.commons.net.ftp and "retrFile" is a method created by me for retrieving files from server.
Here is code for "retrFile"
FTPSClient ftp;
public int retrFile(String filename, String savePath) {
if (!connected) {
return ERR;
}
FileOutputStream fout = null;
InputStream bin = null;
try {
ftp.enterLocalPassiveMode();
fout = new FileOutputStream(savePath + filename);
bin = ftp.retrieveFileStream(filename);
if (bin == null) {
fout.close();
return ERR;
}
byte[] b = new byte[ftp.getBufferSize()];
int bytesRead = 0;
while ((bytesRead = bin.read(b, 0, b.length)) != -1) {
fout.write(b, 0, bytesRead);
}
ftp.completePendingCommand();
fout.close();
} catch (FTPConnectionClosedException ex) {
ex.printStackTrace();
connected = false;
return NOT_CONNECTED;
} catch (IOException ex) {
ex.printStackTrace();
return ERR;
} finally {
try {
fout.close();
} catch (IOException ex) {
ex.printStackTrace();
return ERR;
}
try {
if (bin != null) {
bin.close();
}
} catch (IOException ex) {
ex.printStackTrace();
return ERR;
}
}
return OK;
}

Are you sure the INDEX file is correctly downloaded?
It's present in the filesystem when application is closed?
What FTP lib are you using?. i only know commons.net from Apache and i not recognice the "retrFile" file method. Could it be threaded so that the file is not completely downloaded when the readObject statement is executed?