Develop Reference

I'm having trouble writing copy large files

I'm having problems with my code, I'm encrypting a file with more than 300mb in base 64 but my application gives errors when I open the lra encrypt file
this is my code crashes on the byte, i don't understand why
private void encript(final File file) {
new AsyncTask<Void, Void, Void>() {
#Override
protected Void doInBackground(Void[] p) {
File new_file = null;
try {
new_file = new File(file.getAbsolutePath() + ".enc.txt");
if (!new_file.exists()) {
new_file.createNewFile();
}
BufferedInputStream mInputStream = new BufferedInputStream(new FileInputStream(file));
OutputStream mOutputStream = new DataOutputStream(new FileOutputStream(new_file));
byte[] data = new byte[mInputStream.available()];
int len = 0;
while (true) {
len = mInputStream.read(data);
if (len > 0) {
mOutputStream.write(Base64.encode(data, 0, len, Base64.DEFAULT));
}
break;
}
mOutputStream.flush();
if (mOutputStream != null) {
mOutputStream.close();
}
if (mInputStream != null) {
mInputStream.close();
}
} catch (Exception io) {
Toast.makeText(MainActivity.this, io.toString(), Toast.LENGTH_LONG).show();
}
return null;
}
#Override
protected void onPostExecute(Void res) {
Toast.makeText(MainActivity.this, "Sucesss", Toast.LENGTH_LONG).show();
}
}.execute(new Void[0]);
}

Note that what you are doing here is Base64 encoding the file contents. Don't imagine that someone can't trivially crack this (so-called) "encryption".
There are lots of things wrong with your attempt. I shall go through the more important ones:
#Override
protected Void doInBackground(Void[] p) {
File new_file = null;
try {
Problem: You should be using try with resources to avoid resource leaks.
new_file = new File(file.getAbsolutePath() + ".enc.txt");
if (!new_file.exists()) {
new_file.createNewFile();
}
Problems:
On the one hand, there is no need to use createNewFile to pre-create an output file. Opening the file using FileOutputStream will create it if it doesn't exist already.
On the other hand, this won't prevent (or report) errors in cases where the file's parent directory doesn't exist, is not writeable and so on.
It would be better to use java.nio.file.Path and java.nio.file.Files from Java 7 / Android API 26. Path and Files are better APIs and they will report problems as exceptions so that you can (hypothetically) report them to the user via your exception handler.
There are even some Files.copy methods, though they are not directly applicable to your use-case since you are encoding the data as you copy it.
BufferedInputStream mInputStream =
new BufferedInputStream(new FileInputStream(file));
OutputStream mOutputStream =
new DataOutputStream(new FileOutputStream(new_file));
Problem:
I don't think you need a DataOutputStream. It won't actually be doing anything.
byte[] data = new byte[mInputStream.available()];
Problem:
The available() method should not be used for this. It returns the number of bytes that are "available" to be read right now. The value you get is context dependent. For a socket stream it is typically the number of bytes that are currently in the kernel buffers ready to read. For a "regular" file it may be the length of the input file.
So if you are copying a "really big" file, then you may be attempting to allocate a buffer that will hold the entire file. In the worst case, that will cause your app to OOME!
NOTE - Such an OOME might be the "out of nowhere" problem that you are seeing.
The "best" way is debatable, but I would just use a fixed buffer size ... if I was doing an explicit read / write copy of a stream. The size of the buffer affects throughput, but if you are looking for ultimate performance you shouldn't be doing it this way.
int len = 0;
while (true) {
len = mInputStream.read(data);
if (len > 0) {
mOutputStream.write(
Base64.encode(data, 0, len, Base64.DEFAULT));
}
break;
}
Problem: This loop is simply wrong. You are unconditionally breaking on the first iteration. You should be doing something like this:
int len;
while ((len = mInputStream.read(data)) > 0) {
mOutputStream.write(Base64.encode(data, 0, len, Base64.DEFAULT));
}
In other words, keep reading / writing until read returns a non-positive result.
Note: I'm not sure which Base64 class you are using there. It doesn't appear to be java.util.Base64
mOutputStream.flush();
if (mOutputStream != null) {
mOutputStream.close();
}
if (mInputStream != null) {
mInputStream.close();
}
Problems:
The flush() is not necessary. Closing the stream will flush. And besides, what happens with your attempted flush if mOutputStream is null.
This version leaks resources (file descriptors). If an exception has been thrown, these statements won't be executed, and the stream objects will not be closed.
This is all unnecessary if you use try with resources instead.
} catch (Exception io) {
Toast.makeText(MainActivity.this, io.toString(),
Toast.LENGTH_LONG).show();
}
return null;
}
Problems:
Catching Exception is a bad idea. A better idea is to catch and handle the expected exceptions, and let the unexpected ones propagate so that they can be handled further up the stack.
In this case, it looks like you are assuming that the exception will be some sort of I/O exception. In fact, it could also be an unchecked exception such as an NPE. (An OOME is also possible, though this catch wouldn't catch that because OOMEs are Error exceptions.)
You are throwing away the exception details. Unexpected exceptions should be logged so that you can diagnose them via logcat.

overwrite file content/bytes

I have a (PDF) file that exists on the file system, at a known location. I wish to overwrite the content of that file (with a fresh byte[]).
Best (and most efficient) possible way to do so (using Java APIs)?

public void oneShotAPI(File file, byte[] bytes) throws IOException
{
FileOutputStream fos = null;
try
{
fos = new FileOutputStream(file);
fos.write(bytes);
fos.flush();
} finally
{
if (fos != null)
try
{
fos.close();
} catch (IOException e)
{
// Sad, but true
}
}
}
Call it with:
oneShotAPI(new File("myPDF.png"), byteArray);

There's nothing built into the Java APIs that does this, but if you're looking for a library:
Apache Commons IO has FileUtils.writeByteArrayToFile(File, byte[])
Google Guava has Files.write(byte[], File)
I don't see why any of the short methods posted here wouldn't work, though. There's no actual need for a library IMHO.

javax.imageio.ImageIO Problem creating ImageInputStream

I have a Servlet in Tomcat 5.5 that reads local images sitting on a folder. The image is then sent back to an Applet.
I'm getting this "javax.imageio.IIOException: Can't create an ImageInputStream!" error and not sure whats causing it.
Has anyone had this problem before? Could this be a Thread issue in the ImageIO? I can't reproduce this issue since it occurs about 3 times for every 1000 requests.
EDIT: This is the Servlet code that reads the image. I just use the ImageIO.read(File) in its static form inside the Servlet's doPost method the same way below:
doPost(req,resp){
...
BufferedImage image = ImageIO.read(imageFile);
...
}
Here is the source code for javax.imageio.ImageIO.read(File):
public static BufferedImage read(File input) throws IOException {
if (input == null) {
throw new IllegalArgumentException("input == null!");
}
if (!input.canRead()) {
throw new IIOException("Can't read input file!");
}
ImageInputStream stream = createImageInputStream(input);
if (stream == null) {
throw new IIOException("Can't create an ImageInputStream!");
}
BufferedImage bi = read(stream);
if (bi == null) {
stream.close();
}
return bi;
}

If the sole functional requirement is to read images from local disk and return it unmodified to the HTTP response using a servlet, then you do not need the ImageIO at all. It only adds unnecessary overhead and other problems like you're having now.
Get rid of the ImageIO stuff and just stream the raw image straight from disk to HTTP response, along a set of proper response headers. For example,
String name = request.getParameter("name");
File file = new File("/path/to/images", name);
response.setContentType(getServletContext().getMimeType(file.getName()));
response.setHeader("Content-Length", String.valueOf(file.length()));
response.setHeader("Content-Disposition", "inline; filename=\"" + file.getName() + "\"");
InputStream input = null;
OutputStream output = null;
try {
input = new BufferedInputStream(new FileInputStream(file));
output = new BufferedOutputStream(response.getOutputStream());
byte[] buffer = new byte[8192];
for (int length; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
}
} finally {
if (output != null) try { output.close(); } catch (IOException logOrIgnore) {}
if (input != null) try { input.close(); } catch (IOException logOrIgnore) {}
}
That's all. You only need ImageIO whenever you would like to manipulate the image in server's memory before returning it, e.g. resizing, transforming or something.
Another, more robust, example of such a servlet can be found here and a more advanced one here.

The source I have (Java5 but I doubt it has changed a lot) states that if there are no ImageInputStream service providers registered, the createImageInputStream method returns null and thus you get that exception.
From the JavaDoc on IIORegistry.getDefaultInstance() which is used by ImageIO:
Each ThreadGroup will receive its own instance; this allows different Applets in the same browser (for example) to each have their own registry.
Thus it might actually be a threading problem in that you get a plain new instance of IIORegistry.
Edit: digging deeper into the source I found the following:
Most likely you'd get a FileImageInputStream, since you pass in a file. However, if an exception occurs the service provider returns null. Thus there might be a FileNotFoundException or any other IOException being thrown which causes the stream not to be created.
Unfortunately, there's no logging in the code, thus you'd have to debug somehow. It's probably due to missing file permissions, a corrupted/incomplete file or the file missing.
Here's the Java5 source for FileImageInputStreamSpi#createInputStreamInstance()
public ImageInputStream createInputStreamInstance(Object input,
boolean useCache,
File cacheDir) {
if (input instanceof File) {
try {
return new FileImageInputStream((File)input);
} catch (Exception e) {
return null;
}
} else {
throw new IllegalArgumentException();
}
}

Where is your use of close() methods within the exception handling? Streams have to be closed when there are exceptions, too, as well as normal termination of the block of the code.
The symptom sounds like you run out of heap space or sometime.
It is not the coding errors that others pointed out, since the problem is intermittent.

FFMPEG in Java issue

I have the following code in a java Web Service:
public boolean makeFile(String fileName, String audio)
{
if (makeUserFolder())
{
File file = new File(getUserFolderPath() + fileName + amr);
FileOutputStream fileOutputStream = null;
try
{
file.createNewFile();
fileOutputStream = new FileOutputStream(file);
fileOutputStream.write(Base64.decode(audio));
return true;
}
catch(FileNotFoundException ex)
{
return false;
}
catch(IOException ex)
{
return false;
}
finally{
try {
fileOutputStream.close();
convertFile(fileName);
} catch (IOException ex) {
Logger.getLogger(FileUtils.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
else
return false;
}
public boolean convertFile(String fileName)
{
Process ffmpeg;
String filePath = this.userFolderPath + fileName;
try {
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",filePath + amr,filePath + mp3);
pb.redirectErrorStream();
ffmpeg = pb.start();
} catch (IOException ex) {
return false;
}
return true;
}
It used to work and now it simply won't execute the ffmpeg conversion for some reason. I thought it was a problem with my file but after running the command from terminal no errors are thrown or anything, thought it was maybe permissions issue but all the permissions have been granted in the folder I'm saving the files. I noticed that the input BufferedReader ins being set to null after running the process, any idea what's happening?

First of all, a small nitpick with your code...when you create the FileOutputStream you create it using a string rather than a File, when you have already created the File before, so you might as well recycle that rather than force the FileOutputStream to instantiate the File itself.
Another small nitpick is the fact that when you are writing out the audio file, you should enclose that in a try block and close the output stream in a finally block. If you are allowed to add a new library to your project, you might use Guava which has a method Files.write(byte[],File), which will take care of all the dirty resource management for you.
The only thing that I can see that looks like a definite bug is the fact that you are ignoring the error stream of ffmpeg. If you are blocking waiting for input on the stdout of ffmpeg, then it will not work.
The easiest way to take care of this bug is to use ProcessBuilder instead of Runtime.
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",filePath+amr,filePath+mp3);
pb.redirectErrorStream(); // This will make both stdout and stderr be redirected to process.getInputStream();
ffmpeg = pb.start();
If you start it this way, then your current code will be able to read both input streams fully. It is possible that the stderr was hiding some error that you were not able to see due to not reading it.
If that was not your problem, I would recommend using absolute paths with ffmpeg...in other words:
String lastdot = file.getName().lastIndexOf('.');
File mp3file = new File(file.getParentFile(),file.getName().substring(0,lastdot)+".mp3");
ProcessBuilder pb = new ProcessBuilder("ffmpeg","-i",file.getAbsolutePath(),mp3file.getAbsolutePath());
// ...
If that doesn't work, I would change ffmpeg to be an absolute path as well (in order to rule out path issues).
Edit: Further suggestions.
I would personally refactor the writing code into its own method, so that you can use it elsewhere necessary. In other other words:
public static boolean write(byte[] content, File to) {
FileOutputStream fos = new FileOutputStream(to);
try {
fos.write(content);
} catch (IOException io) {
// logging code here
return false;
} finally {
closeQuietly(fos);
}
return true;
}
public static void closeQuietly(Closeable toClose) {
if ( toClose == null ) { return; }
try {
toClose.close();
} catch (IOException e) {
// logging code here
}
}
The reason that I made the closeQuietly(Closeable) method is due to the fact that if you do not close it in that way, there is a possibility that an exception will be thrown by the close() method, and that exception will obscure the exception that was thrown originally. If you put these in a utility class (although looking at your code, I assume that the class that it is currently in is named FileUtils), then you will be able to use them throughout your application whenever you need to deal with file output.
This will allow you to rewrite the block as:
File file = new File(getUserFolderPath() + fileName + amr);
file.createNewFile()
write(Base64.decode(audio),file);
convertFile(fileName);
I don't know whether or not you should do this, however if you want to be sure that the ffmpeg process has completed, then you should say ffmpeg.waitFor(); to be sure that it has completed. If you do that, then you should examine ffmpeg.exitValue(); to make sure that it completed successfully.
Another thing that you might want to do is once it has completed, write what it output to a log file so you have a record of what happened, just in case something happens.

Java multiple connection downloading

I wanted to get some advice, I have started on a new project to create a java download accelerator that will use multiple connections. I wanted to know how best to go about this.
So far I have figured out that i can use HttpUrlConnection and use the range property, but wanted to know an efficient way of doing this. Once i have download the parts from the multiple connections i will then have to join the parts so that we end up with a fully downloaded file.
Thanks in advance :)

Get the content length of the file to download.
Divide it according to a criteria (size, speed, …).
Run multiple threads to download the file starting at different positions,
and save them in different files: myfile.part1, myfile.part2, …
Once downloaded, join the parts into one single file.
I tried the following code to get the content length:
public Downloader(String path) throws IOException {
int len = 0;
URL url = new URL(path);
URLConnection connectUrl = url.openConnection();
System.out.println(len = connectUrl.getContentLength());
System.out.println(connectUrl.getContentType());
InputStream input = connectUrl.getInputStream();
int i = len;
int c = 0;
System.out.println("=== Content ===");
while (((c = input.read()) != -1) && (--i > 0)) {
System.out.print((char) c);
}
input.close();
}
Here's a sample code to join the files:
public void join(String FilePath) {
long leninfile=0, leng=0;
int count=1, data=0;
try {
File filename = new File(FilePath);
RandomAccessFile outfile = new RandomAccessFile(filename,"rw");
while(true) {
filename = new File(FilePath + count + ".sp");
if (filename.exists()) {
RandomAccessFile infile = new RandomAccessFile(filename,"r");
data=infile.read();
while(data != -1) {
outfile.write(data);
data=infile.read();
}
leng++;
infile.close();
count++;
} else break;
}
outfile.close();
} catch(Exception e) {
e.printStackTrace();
}
}

If you want to avoid joining segments after downloading you could use a FileChannel.
With a FileChannel, you can write to any position of a file (even with multiple threads).
So you could first allocate the whole file, and then
write the segments where they belong as they come in.
See the Javadocs page for more info.

JDownloader is the best downloader I've seen. If you are interested, it's open source and surely you can learn a lot from their code.