Develop Reference

Large file download in java incomplete without error

we are downloading a very large file (~70G) but one one occasion the code completed without throwing an exception, but the downloaded file was incomplete, just under 50G.
The code is:
public void download(String url, String filename) throws Exception {
URL dumpUrl = new URL(url);
try (InputStream input = dumpUrl.openStream()) {
Files.copy(input, Paths.get(filename));
}
}
The url is a presigned Google Cloud Storage URL.
Is this just the libraries not detecting a connection reset issue? Or something else?
Are there better libraries I could use. Or do I need to do a HEAD call first and then match downloaded size against content-length.
Don't care that it didn't work, that happens and we have retry logic. My issue is the code thought it did work.
UPDATE: So it seems it failed at exactly 2 hours after starting download. This makes me suspect it may be netops/firewall issue. Not sure at which end, I'll hassle my ops team for starters. Anybody know of time limits at google's end?
Ignore this update - have more instances now, no set time. Anywhere between 20 minutes and 2 hours.

Never resolved core issue. But was able to workaround by comparing the bytes downloaded to the Content-Length header. Work in a loop which resumes incomplete download using the Range header (similar to curl -C -).

Nothing being sent to server via URLConnection

So I've created a program that communicates with both a FTP and SQL server. I run my program on my Windows machine and everything is working fine. But when I run the program on my Ubuntu VM it is not working. My code is as follows:
try {
URL url = new URL(location);
System.out.println("Created url");
URLConnection urlc = url.openConnection();
System.out.println("Created URLConnection");
urlc.setDoOutput(true);
System.out.println("Set Output");
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
System.out.println("Created OutputStreamWriter");
System.out.println("Hello World.");
out.write(Var.pressed);
System.out.println("Wrote String");
out.close();
System.out.println("Closed OutputStreamWriter");
} catch (Exception ex) {
ex.printStackTrace();
}
The program creates a file on my FTP and then writes to it. The file is being created but it is empty. No errors are ever thrown my program prints the following
Created url
Created URLConnection
Set Output
And then does nothing. It doesn't throw an error, it doesn't crash, it doesn't stop. The terminal is just left open. I also tried just running the .jar but nothing happens.
It seems as though the problem is created here:
OutputStreamWriter out = new OutputStreamWriter(urlc.getOutputStream());
Why is nothing being written to the file? And what is wrong with my program if no errors are ever thrown?

It does throw an error,
From everything else, I think you meant to say it doesn't throw an error. My answer is assuming that anyway.
First there are many reasons for a program to hang or wait and not produce an error. Sometimes if you wait long enough a system threshold will be passed and it will timeout even though your program would be happy to wait forever. Other times the system isn't paying attention either. Error conditions are sort of like a courtesy.
I suggest you should try to make the connection with the Ubuntu ftp line command and compare the same with Windows. FTP protocol does some unusual things with ports. The connection opens on one port, but the data transfer connection is switched over to another port. Firewalls can be tripped up with the change of port and prevent the data transfer. This also has created clients that do "passive" versus "active" FTP.

You need to at least get the response code, or the input stream. Otherwise not even a TCP connection is formed. And otherwise you have no idea whether the server accepted your input.
Your title was wildly inaccurate. Obviously you are creating the OutputStreamWriter. Your issue is that nothing is being sent. And the SQL tag was simply irrelevant.

Without an error message it is nearly impossible to figure out why it fails. A few suggestions, though.
First of all, try adding a System.err.println("test") This will prove that the output log will not only catch System.out AKA standard output stream, but also standard error stream.
I asssume, that in after this fix, the ex.printStackTrace() will be visible.
Otherwise: You do not specify how your location String is filled. Are you sure, you have write access to that directory?
If location is /tmp/foo/bar/myFile.txt, make sure the directory /tmp/foo/bar/ exists (mkdir /tmp/foo/bar/) and that the file is accessible (touch /tmp/foo/bar/myFile.txt).
If both report no error AND the first suggestion would bring up no error, the program might really lock up and wait for something from the outside. Difficult to say what at the moment.

What can cause close(2) to fail with EIO for a read-only file?

I'm investigating a problem on Android where an IOException is getting thrown because of a failure to close a file:
java.io.IOException: close failed: EIO (I/O error)
at libcore.io.IoUtils.close(IoUtils.java:41)
at java.io.FileInputStream.close(FileInputStream.java:121)
at com.adamrosenfield.wordswithcrosses.io.JPZIO.convertJPZPuzzle(JPZIO.java:191)
at com.adamrosenfield.wordswithcrosses.net.AbstractJPZDownloader.download(AbstractJPZDownloader.java:56)
at com.adamrosenfield.wordswithcrosses.net.AbstractJPZDownloader.download(AbstractJPZDownloader.java:41)
at com.adamrosenfield.wordswithcrosses.net.AbstractDownloader.download(AbstractDownloader.java:112)
at com.adamrosenfield.wordswithcrosses.net.AbstractDownloader.download(AbstractDownloader.java:108)
at com.adamrosenfield.wordswithcrosses.net.Downloaders.download(Downloaders.java:257)
at com.adamrosenfield.wordswithcrosses.BrowseActivity.internalDownload(BrowseActivity.java:702)
at com.adamrosenfield.wordswithcrosses.BrowseActivity.access$6(BrowseActivity.java:696)
at com.adamrosenfield.wordswithcrosses.BrowseActivity$7.run(BrowseActivity.java:691)
at java.lang.Thread.run(Thread.java:856)
Caused by: libcore.io.ErrnoException: close failed: EIO (I/O error)
at libcore.io.Posix.close(Native Method)
at libcore.io.BlockGuardOs.close(BlockGuardOs.java:75)
at libcore.io.IoUtils.close(IoUtils.java:38)
... 11 more
The relevant code:
public static void convertJPZPuzzle(File jpzFile, File destFile,
PuzzleMetadataSetter metadataSetter) throws IOException {
FileInputStream fis = new FileInputStream(jpzFile);
try {
DataOutputStream dos = new DataOutputStream(new FileOutputStream(destFile));
try {
if (!convertJPZPuzzle(fis, dos, metadataSetter)) {
throw new IOException("Failed to convert JPZ file: " + jpzFile);
}
} finally {
dos.close();
}
} finally {
fis.close();
}
}
The full source is on GitHub.
The exception is being thrown from the line fis.close(). From what I can tell from reading the Android sources, it looks like FileInputStream.close() just calls down into close(2) on the underlying file descriptor in native code.
The manual pages don't seem to specify what can cause an EIO error, they just say things like "An I/O error occurred." or "If an I/O error occurred while reading from or writing to the file system during close()". The Mac OS X man pages say it can occur when "A previously-uncommitted write(2) encountered an input/output error." on those systems.
What exactly can cause close(2) to fail with the error EIO for a file descriptor which was only opened for reading, as in this case? Clearly it's not an uncommitted write(2). In the case of this particular file, it was downloaded using Android's DownloadManager service, which means there might be lingering threads and/or processes trying to access it simultaneously, but I can hardly see how that would affect trying to close it. Also, the file is just about to be deleted after this code runs (here), but unless Android has an undocumented time machine in it, future code ought not to have an affect here.
I'm interested specifically in the answer on Android and/or Linux, but a more general answer for other OSes would also be appreciated.

I'm guessing the EIO comes from bad_file_flush in fs/bad_inode.c. It seems when the kernel has any failure accessing an inode, it transforms the open file description to a pseudo-open-file of with bad_inode_ops as its file ops. I can't find the code that does this for FAT-based filesystems, but perhaps there's some generic code.
As for the reason, it's probably something like attaching a USB cable and mounting the filesystem from an attached computer, removing the SD card, etc.

In general, you should always anticipate IOExceptions when closing streams. The code is very straightforward, but see here for the cleanest example Java can afford:
https://stackoverflow.com/a/156520/1489860
However, in your specific case, I imagine an exception is being thrown because it appears you are changing the value of the InputStream in the unzipOrPassthrough(InputStream) method and then later trying to close it:
if (entry == null) {
is = new ByteArrayInputStream(baos.toByteArray());
When you later call close on the FileInputStream class, it probably freaks out because it is now a ByteArrayInputStream and no longer a FileInputStream.

Downloading files in Java and common errors

I wrote a simple downloader as Java applet. During some tests I discover that my way of downloading files is not even half as perfect as e.g. Firefox's way of doing it.
My code:
InputStream is = null;
FileOutputStream os = null;
os = new FileOutputStream(...);
URL u = new URL(...);
URLConnection uc = u.openConnection();
is = uc.getInputStream();
final byte[] buf = new byte[1024];
for(int count = is.read(buf);count != -1;count = is.read(buf)) {
os.write(buf, 0, count);
}
Sometimes my applet works fine, sometimes unexpected things happen. E.g. from time to time, in the middle of downloading applet throws an IO exception or just lose a connection for a while, without possibility to return to current download and finish it.
I know that really advanced way is too complicated for single unexperienced Java programmer, but maybe you know some techniques to minimalise risk of appearing these problems.

So you want to resume your download.
If you get an IOException on reading from the URL, there was a problem with the connection.
This happens. Now you must note how much you already did download, and open a new connection which starts from there.
To do this, use setRequestProperty() on the second, and send the right header fields for "I want only the range of the resource starting with ...". See section 14.35.2 Range Retrieval Requests in the HTTP 1.1 specification. You should check the header fields on the response to see if you really got back a range, though.

Getting JDOMException on One Machine but the same xml works fine in another

I am hitting an xml and getting the response back in inputstream. This xml is on the remote server.
Whenever I am running my code on the server, it gives me JDOMException,but the same code, I am running on my local machine and hitting the same xml on remote server, it is running fine.
URL url = new URL("http://testwww.net/xml/android.xml");
InputStream stream = url.openStream();
try {
SAXBuilder builder = new SAXBuilder(false);
xmlDocument = builder.build(stream);
} catch (JDOMException e) {
throw new Exception("unable to read android listing document", e);
}
I am not able to figure out what can be the problem here, that the same xml is running fine on one machine and not on other.
Please help me.
Thanks
Hp

The JDOMException is (presumably) happening because the "testwww.net" server is sending a response body that is not XML. Change your program to print out the actual response stream instead of trying to parse it as XML. That should give you some clues as to what the underlying problem is. Even printing out the JDOMException stacktrace could be helpful ... assuming that you can get hold of that output.
If the problem is that you are running that code on some server and you can't get hold of debug info that your code writes to "system.out", the server logs or whatever, then you are in trouble. You will have to use your imagination to figure out some way around that roadblock.