I wanted to get some advice, I have started on a new project to create a java download accelerator that will use multiple connections. I wanted to know how best to go about this.
So far I have figured out that i can use HttpUrlConnection and use the range property, but wanted to know an efficient way of doing this. Once i have download the parts from the multiple connections i will then have to join the parts so that we end up with a fully downloaded file.
Thanks in advance :)
Get the content length of the file to download.
Divide it according to a criteria (size, speed, …).
Run multiple threads to download the file starting at different positions,
and save them in different files: myfile.part1, myfile.part2, …
Once downloaded, join the parts into one single file.
I tried the following code to get the content length:
public Downloader(String path) throws IOException {
int len = 0;
URL url = new URL(path);
URLConnection connectUrl = url.openConnection();
System.out.println(len = connectUrl.getContentLength());
System.out.println(connectUrl.getContentType());
InputStream input = connectUrl.getInputStream();
int i = len;
int c = 0;
System.out.println("=== Content ===");
while (((c = input.read()) != -1) && (--i > 0)) {
System.out.print((char) c);
}
input.close();
}
Here's a sample code to join the files:
public void join(String FilePath) {
long leninfile=0, leng=0;
int count=1, data=0;
try {
File filename = new File(FilePath);
RandomAccessFile outfile = new RandomAccessFile(filename,"rw");
while(true) {
filename = new File(FilePath + count + ".sp");
if (filename.exists()) {
RandomAccessFile infile = new RandomAccessFile(filename,"r");
data=infile.read();
while(data != -1) {
outfile.write(data);
data=infile.read();
}
leng++;
infile.close();
count++;
} else break;
}
outfile.close();
} catch(Exception e) {
e.printStackTrace();
}
}
If you want to avoid joining segments after downloading you could use a FileChannel.
With a FileChannel, you can write to any position of a file (even with multiple threads).
So you could first allocate the whole file, and then
write the segments where they belong as they come in.
See the Javadocs page for more info.
JDownloader is the best downloader I've seen. If you are interested, it's open source and surely you can learn a lot from their code.
Related
I am trying to allow a user to download a file (attachment) using Java to serve up the download. I have been partially successful. The file is read, and on the client side there is a prompt for a download. A file is saved successfully, but it has 0 bytes. Here is my server side code:
String stored = "/var/lib/tomcat/webapps/myapp/attachments/" + request.getParameter("stored");
String realname = request.getParameter("realname");
// Open the input and output streams
FileInputStream attachmentFis = new FileInputStream(stored);
FileOutputStream attachmentFos = new FileOutputStream(realname);
try {
// Send the file
byte[] attachmentBuffer = new byte[1024];
int count = 0;
while((count = attachmentFis.read(attachmentBuffer)) != -1) {
attachmentFos.write(attachmentBuffer, 0, count);
}
} catch (IOException e) {
// Exception handling
} finally {
// Close the streams
attachmentFos.flush();
attachmentFos.close();
attachmentFis.close();
}
For context, this is in a servlet. The files have an obfuscated name, which is passed as "stored" here. The actual file name, the name the user will see, is "realname".
What do I need to do to get the actual file to arrive at the client end?
EDIT
Following suggestions in the comments, I changed the write to include the 0, count parameters and put the close stuff in a finally block. However, I am still getting a 0 byte file when I attempt a download.
EDIT 2
Thanks to the logging suggestion from Dave the Dane, I discovered the file was being written locally. A bit of digging and I found I needed to use response.getOutputStream().write instead of a regular FileOutputStream. I have been successful in getting a file to download through this method. Thank you all for your helpful suggestions.
As others have observed, you'd be better off using try-with-resources & let that handle the closing.
Assuming you have some Logging Framework available, maybe the following would cast light on the matter...
try {
LOG.info ("Requesting....");
final String stored = "/var/lib/tomcat/webapps/myapp/attachments/" + request.getParameter("stored");
LOG.info ("stored.......: {}", stored);
final String realname = request.getParameter("realname");
LOG.info ("realname.....: {}", realname);
final File fileStored = new File(stored);
LOG.info ("fileStored...: {}", fileStored .getCanonicalPath());
final File fileRealname = new File(realname);
LOG.info ("fileRealname.: {}", fileRealname.getCanonicalPath());
try(final InputStream attachmentFis = new FileInputStream (fileStored);
final OutputStream attachmentFos = new FileOutputStream(fileRealname))
{
final byte[] attachmentBuffer = new byte[64 * 1024];
int count;
while((count = attachmentFis.read (attachmentBuffer)) != -1) {
; attachmentFos.write(attachmentBuffer, 0, count);
LOG.info ("Written......: {} bytes to {}", count, realname);
}
attachmentFos.flush(); // Probably done automatically in .close()
}
LOG.info ("Done.");
}
catch (final Exception e) {
LOG.error("Problem!.....: {}", request, e);
}
If it won't reach the finally block, you should stop ignoring the IOException which is being thrown:
catch (IOException e) {
// Exception handling
System.err.println(e.getMessage());
}
I'd asssume that the realname is just missing an absolute path.
Background
I'm keeping a relatively large text file in android storage, and appending to it periodically- while limiting the file's size to some arbitrary size (say 2MB)
Hopefully I'm missing a function somewhere, or hopefully there is a better way to do this process.
Currently, when the file a goes over that arbitrary size, I create a temporary file b, copy the relevant portion of the file a (more or less the substring of the file a starting at byte xxx where xxx is the number of bytes too large the file a would be if I wrote the next bit of data to the log) plus the current data, then overwrite the file a with the second file b.
This is obviously terribly inefficient...
Another solution that I'm not terribly fond of is to keep two files, and toggle between the two of them, clearing the next when the current is full, and switching to that file for output.
However, it would be suuuuuper handy if I could just do something like this
File A = new File("output");
A.chip(500);
or maybe
A.subfile(500,A.length()-500);
TLDR;
Is there a function or perhaps library available for Android that can remove a portion of a file?
Did you already take a look at RandomAccessFile? Though you cannot remove portions of a file you can seek any position within the file and even set the length. So if you detect your file grows too large, just grab the relevant portion and jump to the beginning. Set length to 0 and write the new data.
EDIT:
I wrote a small demo. It shows if the file size is limeted to 10 bytes. If you pass in the values 10 to 15 as strings and separate them with commas, after 10,11,12, the file is written from the beginning, so after 15 it reads 13,14,15
public class MainActivity extends Activity {
private static final String TAG = MainActivity.class.getSimpleName();
private static final long MAX = 10;
private static final String FILE_TXT = "file.txt";
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
for (int i = 10; i <= 15; i++) {
if (i > 10) {
writeToFile(",");
}
writeToFile(Integer.toString(i));
}
}
private void writeToFile(String text) {
try {
File f = new File(getFilesDir(), FILE_TXT);
RandomAccessFile file = new RandomAccessFile(f, "rw");
long currentLength = file.length();
if (currentLength + text.length() > MAX) {
file.setLength(0);
}
file.seek(file.length());
file.write(text.getBytes());
file.close();
} catch (IOException e) {
Log.e(TAG, "writeToFile()", e);
}
printFileContents();
}
private void printFileContents() {
StringBuilder sb = new StringBuilder();
try {
FileInputStream fin = openFileInput(FILE_TXT);
int ch;
while ((ch = fin.read()) != -1) {
sb.append((char) ch);
}
fin.close();
} catch (IOException e) {
Log.e(TAG, "printFileContents()", e);
}
Log.d(TAG, "current content: " + sb.toString());
}
}
I have been given a task of copying data from a server. I am using BufferedInputStream and output stream to copy the data and I am doing it byte by byte. Even though it is running but It is taking ages to copy the data as some of them are in 100's MBs, so definitely it is not gonna work. Can anyone suggest me any alternate of Byte by Byte copy so that my code can copy file that are in few Hundred MBs.
Buffer is 2048.
Here is how my code look like:
static void copyFiles(SmbFile[] files, String parent) throws IOException {
SmbFileInputStream input = null;
FileOutputStream output = null;
BufferedInputStream buf_input = null;
try {
for (SmbFile f : files) {
System.out.println("Working on files :" + f.getName());
if (f.isDirectory()) {
File folderToBeCreated = new File(parent+f.getName());
if (!folderToBeCreated.exists()) {
folderToBeCreated.mkdir();
System.out.println("Folder name " + parent
+ f.getName() + "has been created");
} else {
System.out.println("exists");
}
copyFiles(f.listFiles(), parent + f.getName());
} else {
input = (SmbFileInputStream) f.getInputStream();
buf_input = new BufferedInputStream(input, BUFFER);
File t = new File(parent + f.getName());
if (!t.exists()) {
t.createNewFile();
}
output = new FileOutputStream(t);
int c;
int count;
byte data[] = new byte[BUFFER];
while ((count = buf_input.read(data, 0, BUFFER)) != -1) {
output.write(data, 0, count);
}
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (input != null) {
input.close();
}
if (output != null) {
output.close();
}
}
}
Here is a link to an excellent post explaining how to use nio channels to make copies of streams. It introduces a helper method ChannelTools.fastChannelCopy that lets you copy streams like this:
final InputStream input = new FileInputStream(inputFile);
final OutputStream output = new FileOutputStream(outputFile);
final ReadableByteChannel inputChannel = Channels.newChannel(input);
final WriteableByteChannel outputChannel = Channels.newChannel(output);
ChannelTools.fastChannelCopy(inputChannel, outputChannel);
inputChannel.close();
outputChannel.close()
Well since you're using a BufferedInputStream, you aren't reading byte by byte, but rather the size of the buffer. You could just try increasing the buffer size.
Reading/writing byte-by-byte is definitely going to be slow, even though the actual reading/writing is done by chunks of the buffer size. One way to speed it up is to read/write by blocks. Have a look at read(byte[] b, int off, int len) method of BufferedInputStream. However it probably won't give you enough of the improvement.
What would be much better is to use nio package (new IO) to copy data using nio channels. Have a look at nio documentation for more info.
I would suggest to use FileUtils from org.apache.commons.io. It has enough utility methods to perform file operations.
org.apache.commons.io.FileUtils API Here
public void download(String url, String destination) {
BufferedOutputStream localBufferedOutputStream = null;
URLConnection localURLConnection = null;
InputStream localInputStream = null;
try {
URL localURL = new URL(url);
localBufferedOutputStream = new BufferedOutputStream(new FileOutputStream(destination));
localURLConnection = localURL.openConnection();
localInputStream = localURLConnection.getInputStream();
byte[] arrayOfByte = new byte[1024];
int i;
while ((i = localInputStream.read(arrayOfByte)) != -1) {
localBufferedOutputStream.write(arrayOfByte, 0, i);
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
try {
if (localInputStream != null) {
localInputStream.close();
}
if (localBufferedOutputStream != null) {
localBufferedOutputStream.close();
}
} catch (IOException localIOException3) {
System.out.println(localIOException3);
}
}
}
I'm debugging my application and it seems a bit slow. I'm wondering if it's my internet. Is this the proper way to download a file in java? The file is 26mb.
You should always look to libraries such as Apache. They have done all the hard work for you:
http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html
I use
static String readFileToString(File file)
Reads the contents of a file into a String using the default encoding for the VM.
quite a lot.
If you know you have a URL (and so stream) look at:
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html
You can leave out the BufferedOutputStream since you're already using a buffer yourself. But that's not going to make a big difference.
What may (or may not) make a big difference is using the nio channel classes instead of the streams.
As an alternative and just for reference, you can investigate HTMLUnit. This framework will allow you to download files even on sites where there are browser redirects.
http://htmlunit.sourceforge.net/
It is certainly not the best way. Code that throws away all exceptions is rarely the best way to do any thing. You might also consider not usi g strings as parameters. URI and File would be good alternatives.
If you want to copy streams transferTo is a good way.
I have an application that writes information to file. This information is used post-execution to determine pass/failure/correctness of the application. I'd like to be able to read the file as it is being written so that I can do these pass/failure/correctness checks in real time.
I assume it is possible to do this, but what are the gotcha's involved when using Java? If the reading catches up to the writing, will it just wait for more writes up until the file is closed, or will the read throw an exception at this point? If the latter, what do I do then?
My intuition is currently pushing me towards BufferedStreams. Is this the way to go?
Could not get the example to work using FileChannel.read(ByteBuffer) because it isn't a blocking read. Did however get the code below to work:
boolean running = true;
BufferedInputStream reader = new BufferedInputStream(new FileInputStream( "out.txt" ) );
public void run() {
while( running ) {
if( reader.available() > 0 ) {
System.out.print( (char)reader.read() );
}
else {
try {
sleep( 500 );
}
catch( InterruptedException ex ) {
running = false;
}
}
}
}
Of course the same thing would work as a timer instead of a thread, but I leave that up to the programmer. I'm still looking for a better way, but this works for me for now.
Oh, and I'll caveat this with: I'm using 1.4.2. Yes I know I'm in the stone ages still.
If you want to read a file while it is being written and only read the new content then following will help you achieve the same.
To run this program you will launch it from command prompt/terminal window and pass the file name to read. It will read the file unless you kill the program.
java FileReader c:\myfile.txt
As you type a line of text save it from notepad and you will see the text printed in the console.
public class FileReader {
public static void main(String args[]) throws Exception {
if(args.length>0){
File file = new File(args[0]);
System.out.println(file.getAbsolutePath());
if(file.exists() && file.canRead()){
long fileLength = file.length();
readFile(file,0L);
while(true){
if(fileLength<file.length()){
readFile(file,fileLength);
fileLength=file.length();
}
}
}
}else{
System.out.println("no file to read");
}
}
public static void readFile(File file,Long fileLength) throws IOException {
String line = null;
BufferedReader in = new BufferedReader(new java.io.FileReader(file));
in.skip(fileLength);
while((line = in.readLine()) != null)
{
System.out.println(line);
}
in.close();
}
}
You might also take a look at java channel for locking a part of a file.
http://java.sun.com/javase/6/docs/api/java/nio/channels/FileChannel.html
This function of the FileChannel might be a start
lock(long position, long size, boolean shared)
An invocation of this method will block until the region can be locked
I totally agree with Joshua's response, Tailer is fit for the job in this situation. Here is an example :
It writes a line every 150 ms in a file, while reading this very same file every 2500 ms
public class TailerTest
{
public static void main(String[] args)
{
File f = new File("/tmp/test.txt");
MyListener listener = new MyListener();
Tailer.create(f, listener, 2500);
try
{
FileOutputStream fos = new FileOutputStream(f);
int i = 0;
while (i < 200)
{
fos.write(("test" + ++i + "\n").getBytes());
Thread.sleep(150);
}
fos.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
private static class MyListener extends TailerListenerAdapter
{
#Override
public void handle(String line)
{
System.out.println(line);
}
}
}
The answer seems to be "no" ... and "yes". There seems to be no real way to know if a file is open for writing by another application. So, reading from such a file will just progress until content is exhausted. I took Mike's advice and wrote some test code:
Writer.java writes a string to file and then waits for the user to hit enter before writing another line to file. The idea being that it could be started up, then a reader can be started to see how it copes with the "partial" file. The reader I wrote is in Reader.java.
Writer.java
public class Writer extends Object
{
Writer () {
}
public static String[] strings =
{
"Hello World",
"Goodbye World"
};
public static void main(String[] args)
throws java.io.IOException {
java.io.PrintWriter pw =
new java.io.PrintWriter(new java.io.FileOutputStream("out.txt"), true);
for(String s : strings) {
pw.println(s);
System.in.read();
}
pw.close();
}
}
Reader.java
public class Reader extends Object
{
Reader () {
}
public static void main(String[] args)
throws Exception {
java.io.FileInputStream in = new java.io.FileInputStream("out.txt");
java.nio.channels.FileChannel fc = in.getChannel();
java.nio.ByteBuffer bb = java.nio.ByteBuffer.allocate(10);
while(fc.read(bb) >= 0) {
bb.flip();
while(bb.hasRemaining()) {
System.out.println((char)bb.get());
}
bb.clear();
}
System.exit(0);
}
}
No guarantees that this code is best practice.
This leaves the option suggested by Mike of periodically checking if there is new data to be read from the file. This then requires user intervention to close the file reader when it is determined that the reading is completed. Or, the reader needs to be made aware the content of the file and be able to determine and end of write condition. If the content were XML, the end of document could be used to signal this.
There are a Open Source Java Graphic Tail that does this.
https://stackoverflow.com/a/559146/1255493
public void run() {
try {
while (_running) {
Thread.sleep(_updateInterval);
long len = _file.length();
if (len < _filePointer) {
// Log must have been jibbled or deleted.
this.appendMessage("Log file was reset. Restarting logging from start of file.");
_filePointer = len;
}
else if (len > _filePointer) {
// File must have had something added to it!
RandomAccessFile raf = new RandomAccessFile(_file, "r");
raf.seek(_filePointer);
String line = null;
while ((line = raf.readLine()) != null) {
this.appendLine(line);
}
_filePointer = raf.getFilePointer();
raf.close();
}
}
}
catch (Exception e) {
this.appendMessage("Fatal error reading log file, log tailing has stopped.");
}
// dispose();
}
You can't read a file which is opened from another process using FileInputStream, FileReader or RandomAccessFile.
But using FileChannel directly will work:
private static byte[] readSharedFile(File file) throws IOException {
byte buffer[] = new byte[(int) file.length()];
final FileChannel fc = FileChannel.open(file.toPath(), EnumSet.of(StandardOpenOption.READ));
final ByteBuffer dst = ByteBuffer.wrap(buffer);
fc.read(dst);
fc.close();
return buffer;
}
Not Java per-se, but you may run into issues where you have written something to a file, but it hasn't been actually written yet - it might be in a cache somewhere, and reading from the same file may not actually give you the new information.
Short version - use flush() or whatever the relevant system call is to ensure that your data is actually written to the file.
Note I am not talking about the OS level disk cache - if your data gets into here, it should appear in a read() after this point. It may be that the language itself caches writes, waiting until a buffer fills up or file is flushed/closed.
I've never tried it, but you should write a test case to see if reading from a stream after you have hit the end will work, regardless of if there is more data written to the file.
Is there a reason you can't use a piped input/output stream? Is the data being written and read from the same application (if so, you have the data, why do you need to read from the file)?
Otherwise, maybe read till end of file, then monitor for changes and seek to where you left off and continue... though watch out for race conditions.