Develop Reference

How can I read a text file from the internet with Java?

I want to read the second line of the text at this URL: "http://vuln2014.picoctf.com:51818/" (this is a capture-the-flag competition but only asking for flags or direction to flags breaks the competition rules). I am attempting to open an input stream from the URL but I get an Invalid HTTP Response exception. Any help is appreciated, and I recognize that my error is likely quite foolish.
Code:
URL url = new URL("http://vuln2014.picoctf.com:51818");
URLConnection con = url.openConnection();
InputStream is = con.getInputStream()
The error occurs at the third line.
java.io.IOException: Invalid Http response at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1342) at name.main(name.java:41)
curl happily gets the text from the page, and it is perfectly accessible from a web browser.

When you do this:
URL url = new URL("http://vuln2014.picoctf.com:51818");
URLConnection con = url.openConnection();
You are entering into a contract that says that this URL uses the http protocol. When you call openConnection it expects to get http responses because you used http:// in the URL as the protocol. The Java Documentation says:
If for the URL's protocol (such as HTTP or JAR), there exists a public, specialized URLConnection subclass belonging to one of the following packages or one of their subpackages: java.lang, java.io, java.util, java.net, the connection returned will be of that subclass. For example, for HTTP an HttpURLConnection will be returned, and for JAR a JarURLConnection will be returned.
The server you are connecting to just returns a couple lines of data. I retrieved them with the command nc vuln2014.picoctf.com 51818. There is no http response code like HTTP/1.1 200 OK:
Welcome to the Daedalus Corp Spies RSA Key Generation Service. The public modulus you should use to send your updates is below. Remember to use exponent 65537.
b4ab920c4772c5247e7d89ec7570af7295f92e3b584fc1a1a5624d19ca07cd72ab4ab9c8ec58a63c09f382aa319fa5a714a46ffafcb6529026bbc058fc49fb1c29ae9f414db4aa609a5cab6ff5c7b4c4cfc7c18844f048e3899934999510b2fe25fcf8c572514dd2e14c6e19c4668d9ad82fe647cf9e700dcf6dc23496be30bb
In this case I would use java.net.Socket to establish a connection and then read the lines. This is a simplistic approach that assumes there are 2 lines of data:
Socket theSocket;
try {
theSocket = new Socket("vuln2014.picoctf.com", 51818);
BufferedReader inFile = new BufferedReader(new InputStreamReader(theSocket.getInputStream()));
String strGreet = inFile.readLine();
String strData = inFile.readLine();
} catch (IOException e) {
e.printStackTrace();
}
As for why curl and browsers may render it properly? They are likely more lenient about the data they read and will just dump what is read from the port even if it doesn't conform to the specified protocol (like http)

Multiple HttpURLConnection calls for get throwing Premature end of file exception with InputStream

I'm trying to make multiple calls to a REST API using HttpURLConnection with GET method and retrieving the response through InputStream.
It worked fine previously for 3 months but now it's throwing below exception:
SAXException Occurred during getArtifactsUrl method:: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) [xercesImpl.jar:6.1.0.Final]
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) [xercesImpl.jar:6.1.0.Final]
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) [:1.7.0_03]
Below is the line of code where I'm making the second call to parse the response:
request = (HttpURLConnection) endpointUrl.openConnection();
inputstream = request.getInputStream();
doc = dBuilder.parse(inputstream);
First call is working fine using request and inputstream objects but second call is failing. I tried all possible answers I found in google but no luck:
after every call:
inputstream.close();
request.disconnect();
Remember that request is an HttpURLConnection object.
I greatly appreciate if you can be able to solve this as I this is a high prioirity production issue now!

First you should check for error cases and not assume it's always working.
Try this:
request = (HttpURLConnection) endpointUrl.openConnection();
request.connect(); // not really necessary (done automatically)
int statusCode = request.getResponseCode();
if (statusCode == 200) { // or maybe other 2xx codes?
// Success - should work if server gives good response
inputstream = request.getInputStream();
// if you get status code 200 and still have the same error, you should
// consider logging the stream to see what document you get from server.
// (see below *)
doc = dBuilder.parse(inputstream);
} else {
// Something happened
// handle error, try again if it makes sense, ...
if (statusCode == 404) ... // resource not found
if (statusCode == 500) ... // internal server error
// maybe there is something interesting in the body
inputstream = request.getErrorStream();
// read and parse errorStream (but probably this is not the body
// you expected)
}
Have a look at the List of HTTP status codes.
And in some nasty cases, there are other problems which are not easy to detect if you just sit behind HttpURLConnection. Then you could enable logging or snoop the TCP/IP traffic with an apropriate tool (depends on your infrastructure, rights, OS, ...). This SO post might help you.
*) In your case I suppose that you're getting a non-error status code from the server but unparseable XML. If logging the traffic is not your thing, you could read the InputStream, write it to a file and then process the stream like before. Of course you can write the stream to a ByteArrayOutputStream, get the byte[] and write that Bytes to a file and then convert them to a ByteArrayInputStream and give this to your XML-parser. Or you could use Commons IO TeeInputStream to handle that for you.
There are cases where connection.getResponseCode() throws an exception. Then it was not possible to parse the HTTP header. This should only happen if there are strange errors in server software, hardware or perhaps a firewall, proxy or load balancer not behaving well.
One more thing: You might consider choosing an HTTP Client library and not directly use HttpURLConnection.

DocumentList API resumable upload not working

Recently it appears that resumable upload is not working. All my requests for upload are returning 500.
Here is the exception stacktrace I receive:
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://docs.google.com/feeds/upload/create-session/default/private/full/folder%3A0B6Qc9CKRbiEMNTQ2NWYzMjEtY2EwNC00NzRhLWFjNGQtNGEzNzEzNzc4MTRj/contents/?convert=false&upload_id=AEnB2UpuikBd2Rd1wk1j8BPAI3KKTJ1pWoAJEPm3KZCBqLIqj6Rm9uOy7NezC8dzROUghRpTI6Clblj8j4zhKO91ductHL2LBA
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
at GoogleDocsManager.uploadWithResumableUpload(GoogleDocsManager.java:1342)
This is the code (no changes were made since yesterday (21-11-2012))
String link = initiateSession(mediaBytes, contentType, title, withConvert);
URL url = new URL(link);
HttpURLConnection copyHttpUrlConn = (HttpURLConnection) url.openConnection();
copyHttpUrlConn.setDoOutput(true);
copyHttpUrlConn.setRequestMethod("PUT");
copyHttpUrlConn.setRequestProperty("Content-Type",contentType);
copyHttpUrlConn.setRequestProperty("Content-Length", mediaBytes.length + "");
String range = "bytes 0-" + (mediaBytes.length - 1) + "/" + (mediaBytes.length);
copyHttpUrlConn.setRequestProperty("Content-Range", range);
OutputStream outputStream = copyHttpUrlConn.getOutputStream();
outputStream.write(mediaBytes);
System.out.println("Code: " + copyHttpUrlConn.getResponseCode());// here I receive the exception
Now I really don't know where to write this kind of posts... From the old DocumentList Api group I was redirected here.
I hope that this issue is fixed soon (I have 2 different programs which use this resumable upload and yesterday both were ok, so I guess that this problem is not on my side)
Best regards,

500 errors in the Drive SDK are a way of life. I get them around 0.5% of the time. Try wrapping your API calls in an exponential backoff and retry.
Because your error is on an upload, you might want to verify that the file you're uploading has valid contents according to its mime type. (eg. if it's text/html, does the file contain valid html).

After 4 days the errors stopped (with no changes made to the code).
PS: For a period there were a strange behaviour (on google appengine not working, and localhost yes).

File not found exception while reading connection.getInputStream()

I am sending a request on a server URL but I am getting File not found exception but when I browse this file through a web browser it seems fine.
URL url = new URL(serverUrl);
connection = getSecureConnection(url);
// Connect to server
connection.connect();
// Send parameters to server
writer = new BufferedWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));
writer.write(parseParameters(CoreConstants.ACTION_PREFIX + actionName, parameters));
writer.flush();
// Read server's response
reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
when I try to getInputStream then it throws error file not found.
It is an .aspx Controller page.

If the request works fine in a browser but not in code, and you've verified that the URL is the same, then the problem probably has something to do with how you are sending your parameters to the server. Specifically, this part:
writer.write(parseParameters(CoreConstants.ACTION_PREFIX + actionName, parameters));
Perhaps there is a bug in the parseParameters() function?
But more generally, I would recommend using something a bit higher-level than a raw URLConnection. HtmlUnit and HttpClient are both fine choices, particularly since it seems like your request is a fairly simple one. I've used both to perform similar client/server interaction in a number of apps. I suggest revising your code to use one of these libraries, and then see if it still produces the error.

Ok finally I have found that the problem was at IIS side it has been resolved in .Net 4.0. for previous version go to your web.config and specify validateRequest==false

java.io.IOException: Server returns HTTP response code 505

I have HTML based queries in my code and one specific kind seems to give rise to IOExceptions upon receiving 505 response from the server. I have looked up the 505 response along with other people who seemed to have similar problems. Apparently 505 stands for HTTP version mismatch, but when I copy the same query URL to any browser (tried firefox, seamonkey and Opera) there seems to be no problem. One of the posts I read suggested that the browsers might automatically handle the version mismatch problem..
I have tried to dig in deeper by using the nice developer tool that comes with Opera, and it looks like there is no mismatch in versions (I believe Java uses HTTP 1.1) and a nice 200 OK response is received. Why do I experience problems when the same query goes through my Java code?
private InputStream openURL(String urlName) throws IOException{
URL url = new URL(urlName);
URLConnection urlConnection = url.openConnection();
return urlConnection.getInputStream();
}
sample link: http://www.uniprot.org/uniprot/?query=mnemonic%3aNUGM_HUMAN&format=tab&columns=id,entry%20name,reviewed,organism,length

There has been some issues in Tomcat with URLs containing space in it. To fix the problem, you need to encode your url with URLEncoder.
Example (notice the space):
String url="http://example.org/test test2/index.html";
String encodedURL=java.net.URLEncoder.encode(url,"UTF-8");
System.out.println(encodedURL); //outputs http%3A%2F%2Fexample.org%2Ftest+test2%2Findex.html

AS a developer at www.uniprot.org I have the advantage of being able to look in the request logs. In the last year according to the logs we have not send a 505 response code. In any case our servers do understand http 1 requests as well as the default http1.1 (though you might not get the results that you expect).
That makes me suspect there was either some kind of data corruption on the way. Or you where affected by a hardware failure (lately we have had some trouble with a switch and a whole datacentre ;). In any case if you ever have questions or problems with uniprot.org please contact help#uniprot.org then we can see if we can help/fix the problem.
Your code snippet seems normal and should work.
Regards,
Jerven Bolleman

Are you behind a proxy? This code works for me and prints out the same text I see through a browser.
final URL url = new URL("http://www.uniprot.org/uniprot/?query=mnemonic%3aNUGM_HUMAN&format=tab&columns=id,entry%20name,reviewed,organism,length");
final URLConnection conn = url.openConnection();
final InputStream is = conn.getInputStream();
System.out.println(IOUtils.toString(is));
conn is an instance of HttpURLConnection

from the API documentation for the URL class:
The URL class does not itself encode or decode any URL components
[...]. It is the responsibility of the caller to encode any fields,
which need to be escaped prior to calling URL, and also to decode any
escaped fields, that are returned from URL.
so if you have any spaces in your url-str encode it before calling new URL(url-str)

#posdef I was having same HTTP error code 505 problem. When I pasted URL that I was using in Java code in Firefox, Chrome it worked. But through code was giving IOException. But at last I came to know that in url string there were brackets '(' and ')', by removing them it worked so it seems I needed URLEncodeing same like browsers.