Some devices (e.g. webrelays) return raw XML in response to HTTPGet requests. That is, the reply contains no valid HTTP header. For many years I have retrieved information from such devices using code like this:
private InputStream doRawGET(String url) throws MalformedURLException, IOException
{
try
{
URL url = new URL(url);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(5000);
con.setReadTimeout(5000);
return con.getInputStream();
}
catch (SocketTimeoutException ex)
{
throw new IOException("Timeout attempting to contact Web Relay at " + url);
}
}
In openJdk 7 the following lines have been added to sun.net.www.protocol.http.HttpURLConnection, which mean any HTTP response with an invalid header generates an IOException:
1325 respCode = getResponseCode();
1326 if (respCode == -1) {
1327 disconnectInternal();
1328 throw new IOException ("Invalid Http response");
1329 }
How do I get 'headless' XML from a server which expects HTTPGet requests in the new world of Java 7?
You could always do it the "socket way" and talk HTTP directly to the host:
private InputStream doRawGET( String url )
{
Socket s = new Socket( new URL( url ).getHost() , 80 );
PrintWriter out = new PrintWriter( s.getOutputStream() , true );
out.println( "GET " + new URL( url ).getPath() + " HTTP/1.0" );
out.println(); // extra CR necessary sometimes.
return s.getInputStream():
}
Not exactly elegant, but it'll work. Strange that JRE7 introduces such "regression" though.
Cheers,
Related
I'm developing my client-server application (server is like a servlet and client is an Android app). I have some difficulties sending information between two entities when the message has special chars (ie: 'è' or others)
On client side I use this code to send the message that can contain special chars:
public static String effettuaPOSTServer (String parameters) {
try {
byte [] parametersBytes = parameters.getBytes("UTF-8");
URL url = new URL("http://" + IP_SERVER + ":" + PORT + PATH_SERVLET);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setConnectTimeout(10000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content_Type", "application/x-www-form-urlencoded; charset=utf-8");
connection.setRequestProperty ("Content-Length", String.valueOf (parametersBytes.length));
connection.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.write(parametersBytes);
wr.flush();
wr.close();
BufferedReader br = new BufferedReader (new InputStreamReader(connection.getInputStream(), "UTF-8"));
StringBuilder sb = new StringBuilder ();
String s;
while ((s = br.readLine ()) != null) {
sb.append (s);
sb.append ("\n");
}
return sb.toString();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (ProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
on server side I use this code to send the response (also the response can contain special chars like 'è' and others) (I'm in servlet doPost context so I work with HttpServletRequest request and HttpServletResponse response):
PrintWriter pw = response.getWriter();
...
String content = ...; //the string is formatted in JSON format
pw.println (content);
but on both sides I'm unable to receive and manage the correct strings where I have special chars.
I'm trying a lot of solutions on web about encoding/decoding etc. but without successful result.
How can I fix my problem? Thank you!
EDIT:
for example, immagine my request from client as follow (I report the request in GET format to show simply the case):
http://MY_URL:PORT/MY_PATH?parameter1=value1¶meter2=value2¶meter3=èqualcosaacaso
but on server I receive:
parameter1=value1
parameter2=value2
parameter3=Äqualcosaacaso
This question already has answers here:
Java - How to find the redirected url of a url?
(6 answers)
Closed 7 years ago.
I tried to access this url in my java program but I got this strange message instead of the page content as I was expecting.
How can I avoid this?
<!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>303 See Other</title>
</head>
<body>
<h1>See Other</h1>
<p>The answer to your request is located here.</p>
</body>
</html>
In a browser though I can navigate there easily. Is there some function or library I can use to evoke that functionality from my java program?
for (String url : list_of_relation_URLs)
{
//System.out.println( url );
//go to relation url
String URL_czech = url;
System.out.println( url );
URL wikidata_page = new URL(URL_czech);
HttpURLConnection wiki_connection = (HttpURLConnection)wikidata_page.openConnection();
InputStream wikiInputStream = null;
try
{
// try to connect and use the input stream
wiki_connection.connect();
wikiInputStream = wiki_connection.getInputStream();
}
catch(IOException error)
{
// failed, try using the error stream
wikiInputStream = wiki_connection.getErrorStream();
}
// parse the input stream using Jsoup
Document docx = Jsoup.parse(wikiInputStream, null, wikidata_page.getProtocol()+"://"+wikidata_page.getHost()+"/");
System.out.println( docx.toString() );
}
I'm trying to do basically the opposite of what is going on here.
When you receive a 303 status code, you simply need to make a second request to the URL supplied with the 303.
The new URL is stored in the Location header.
In your case, you will need to keep following until you get a different status code as you will be redirected two times.
303: Location:"https://www.wikidata.org/wiki/Special:EntityData/P26"
303: Location:"https://www.wikidata.org/wiki/Property:P26"
And yes... if you are using a HttpURLConnection you can ask it to do this for you.
conn.setInstanceFollowRedirects(true);
this is the perfect answer
try {
String url = "http://www.twitter.com";
URL obj = new URL(url);
HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
conn.setReadTimeout(5000);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");
System.out.println("Request URL ... " + url);
boolean redirect = false;
// normally, 3xx is redirect
int status = conn.getResponseCode();
if (status != HttpURLConnection.HTTP_OK) {
if (status == HttpURLConnection.HTTP_MOVED_TEMP
|| status == HttpURLConnection.HTTP_MOVED_PERM
|| status == HttpURLConnection.HTTP_SEE_OTHER)
redirect = true;
}
System.out.println("Response Code ... " + status);
if (redirect) {
// get redirect url from "location" header field
String newUrl = conn.getHeaderField("Location");
// get the cookie if need, for login
String cookies = conn.getHeaderField("Set-Cookie");
// open the new connnection again
conn = (HttpURLConnection) new URL(newUrl).openConnection();
conn.setRequestProperty("Cookie", cookies);
conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
conn.addRequestProperty("User-Agent", "Mozilla");
conn.addRequestProperty("Referer", "google.com");
System.out.println("Redirect to URL : " + newUrl);
}
BufferedReader in = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
String inputLine;
StringBuffer html = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
html.append(inputLine);
}
in.close();
System.out.println("URL Content... \n" + html.toString());
System.out.println("Done");
So I have a problem where if I type this link on the browser and hit enter, an activation happens. I just want to do the same through Java. I don't need any kind of response from the URL. It should just do the same as entering the URL on a browser. Currently my code doesn't throw an error, but I don't think its working because the activation is not happening. My code:
public static void enableMachine(String dns){
try {
String req= "http://"+dns+"/username?username=sputtasw";
URL url = new URL(req);
URLConnection connection = url.openConnection();
connection.connect();
/*BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
String strTemp = "";
while (null != (strTemp = br.readLine())) {
System.out.println(strTemp);
}*/
} catch (Exception ex) {
ex.printStackTrace();
}
}
What's the problem?
If you want to do that with an URLConnection, it isn't sufficient to just open the connection with connect, you also have to send e.g. an HTTP request etc.
That said, i think it would be easier, if you use an HTTP client like the one from Apache HttpComponents (http://hc.apache.org/). Just do a GET request with the HTTP client, this would be the same as visiting the page with a browser (those clients usually also supports redirection etc.).
You may use HttpUrlConnectionClass to do the job:
URL url = new URL("http://my.url.com");
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setRequestProperty("Content-Type", "application/json");
httpCon.setDoOutput(true);
httpCon.setRequestMethod("POST");
String params = "foo=42&bar=buzz";
DataOutputStream wr = new DataOutputStream(httpCon.getOutputStream());
wr.writeBytes(params);
wr.flush();
wr.close();
httpCon.connect();
int responseCode = httpCon.getResponseCode();
You may as well use "GET" request method and just append parameters to the url.
I'm trying to download a file from
http://aula.au.dk/main/document/document.php?action=download&id=%2F%D8velsesvejledning+2012.pdf
but it dosen't appear to be a pdf, when i try downloading it with this code
import java.io.*;
import java.net.*;
public class DownloadFile {
public static void download(String address, String localFileName) throws IOException {
URL url1 = new URL(address);
byte[] ba1 = new byte[1024];
int baLength;
FileOutputStream fos1 = new FileOutputStream(localFileName);
try {
// Contacting the URL
System.out.print("Connecting to " + url1.toString() + " ... ");
URLConnection urlConn = url1.openConnection();
// Checking whether the URL contains a PDF
if (!urlConn.getContentType().equalsIgnoreCase("application/pdf")) {
System.out.println("FAILED.\n[Sorry. This is not a PDF.]");
} else {
try {
// Read the PDF from the URL and save to a local file
InputStream is1 = url1.openStream();
while ((baLength = is1.read(ba1)) != -1) {
fos1.write(ba1, 0, baLength);
}
fos1.flush();
fos1.close();
is1.close();
} catch (ConnectException ce) {
System.out.println("FAILED.\n[" + ce.getMessage() + "]\n");
}
}
} catch (NullPointerException npe) {
System.out.println("FAILED.\n[" + npe.getMessage() + "]\n");
}
}
}
Can you help me out here?
http://aula.au.dk/main/document/document.php?action=download&id=%2F%D8velsesvejledning+2012.pdf is not a pdf. The website gives an error this is why the script doesn't work:
SQL error in file /data/htdocs/dokeos184/www/main/inc/tool_navigation_menu.inc.php at line 70
As Marti said, the root cause of the problem is the fact that the script fails. I tested your program on a working pdf link, it works just fine.
This wouldn't have helped you in this case, but HttpURLConnection is a specialized subclass of URLConnection that makes communications with an http server a lot easier - eg direct access to error codes, etc.
HttpURLConnection urlConn = (HttpURLConnection) url1.openConnection();
// check the responsecode for e.g. errors (4xx or 5xx)
int responseCode = urlConn.getResponseCode();
2 step process with 2 libraries.
// 1. Use Jsoup to get the response.
Response response= Jsoup.connect(location)
.ignoreContentType(true)
// more method calls like user agent, referer, timeout
.execute();
// 2. Use Apache Commons to write the file
FileUtils.writeByteArrayToFile(new File(path), response.bodyAsBytes());
I´m trying to send a post request with cookies. This is the code:
try {
String query = URLEncoder.encode("key", "UTF-8") + "=" + URLEncoder.encode("value", "UTF-8");
String cookies = "session_cookie=value";
URL url = new URL("https://myweb");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setRequestProperty("Cookie", cookies);
conn.setRequestMethod("POST");
conn.setDoInput(true);
conn.setDoOutput(true);
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
out.writeBytes(query);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String decodedString;
while ((decodedString = in.readLine()) != null) {
System.out.println(decodedString);
}
in.close();
// Send the request to the server
//conn.connect();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
The problem is the request is sent without the cookies. If I only make:
conn.connect(); and don´t send data, the cookies are sent OK.
I can´t check exactly what is happening, because the connection is thorugh SSL. I only check the response.
According to the URLConnection javadoc:
The following methods are used to access the header fields and the
contents AFTER the connection is made to the remote object:
* getContent
* getHeaderField
* getInputStream
* getOutputStream
Have you confirmed that in your test case above the request is getting to the server at all? I see you have the call to connect() after getOutputStream() and commented-out besides. What happens if you uncomment it and move up before the call to getOutputStream() ?