This question already has answers here:
Read error response body in Java
(8 answers)
Closed 3 years ago.
When I contact a web service using a Java HttpUrlConnection it just returns a 400 Bad Request (IOException). How do I get the XML information that the server is returning; it does not appear to be in the getErrorStream of the connection nor is it in any of the exception information.
When I run the following PHP code against a web service:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.myclientaddress.com/here/" );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt($ch, CURLOPT_POST,1 );
curl_setopt($ch, CURLOPT_POSTFIELDS,"username=ted&password=scheckler&type=consumer&id=123456789&zip=12345");
$result=curl_exec ($ch);
echo $result;
?>
it returns the following:
<?xml version="1.0" encoding="utf-8"?>
<response>
<status>failure</status>
<errors>
<error name="RES_ZIP">Zip code is not valid.</error>
<error name="ZIP">Invalid zip code for residence address.</error>
</errors>
</response>
so I know the information exists
HttpUrlConnection returns FileNotFoundException if you try to read the getInputStream() from the connection, so you should instead use getErrorStream() whenever the status code is equal or higher than 400.
More than this, please be careful since it's not only 200 to be the success status code, even 201, 204, etc. are often used as success statuses.
Here is an example of how I went to manage it
// ... connection code code code ...
// Get the response code
int statusCode = connection.getResponseCode();
InputStream is = null;
if (statusCode >= 200 && statusCode < 400) {
// Create an InputStream in order to extract the response object
is = connection.getInputStream();
}
else {
is = connection.getErrorStream();
}
// ... callback/response to your handler....
In this way, you'll be able to get the needed response in both success and error cases.
Hope this helps!
If the server is returning XML and you pass the parameters as a url, why not just use a library that supports JAX-RS (Java's REST webservices API), such as Apache CXF?
I know it supports JAX-RS because it has a chapter in the manual.
I had same issue and adding below two lines resolved it.
httpConn.setRequestProperty("Connection", "Close");
System.setProperty("http.keepAlive", "false");
Related
I'm trying to use an existing API that calls for a json object to be "submitted" using POST, but the page also requires a few parameters in request, for this example we will use name, and email. I'm super new to REST so I'm probably making an ignorant mistake somewhere here.
Here is the code I have so far in my servlet:
String path = "http://www.test.com/submit";
URL url = new URL(path);
conn = (HttpURLConnection)url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
Gson gson = new Gson();
//julySchedule is the object I want to submit with this request alongside the parameters.
String input = gson.toJson(julySchedule);
OutputStream os = conn.getOutputStream();
os.write(input.getBytes());
os.flush();
if (conn.getResponseCode() != 200) {
throw new RuntimeException("HTTP POST Request Failed with Error code : "
+ conn.getResponseCode());
}
I've tried to put the parameters in the URL, like:
/submit?name=Name&email=test#gmail.com
But that didn't work, because POST requests wont accept parameters like that.
Then I tried to add it to the Output stream like:
String params = "name=Name&email=test#gmail.com"
OutputStream os = conn.getOutputStream();
os.write(path.toBytes());
os.write(input.getBytes());
os.flush();
But that didn't work either. Am I making a really dumb mistake somewhere?
It is perfectly fine for a POST request to contain query string(Though not best practice).
Since you use application/json as Content-Type, I assume your payload has to be a JSON object. You cannot mix your params in POST body. You have 2 options:
Kep email&name in QueryString, and json in POST payload: What you did should work.
Use POST payload only: Set Content-Type: application/x-www-form-urlencoded as follows: name=NAME&email=EMAIL&jsonPayLoad={Your Json Object}. Server should extract jsonPayLoad from POST payload.
Make sure to escape using encodeURIComponent before writing to POST output stream.
i know you shouldn't send a HTTP GET Request with a body, but ceilometer web api forces me to do so.
I'm developing a ceilometer scala client, so I need a scala/java way to make a get request with a body.
So far I tried with beeClient (http://www.bigbeeconsultants.co.uk) and in plain Java using httpConnection but I get a 404 error.
In curl I can achieve the result in this way:
curl -X GET -H "X-Auth-Token: ..long long token here.."
-H "Content-Type: application/json"
-d '{"q": [{"field": "resource", "op": "eq", "value": "gdfsf"}]}'
http://137.204.57.150:8777/v2/meters/
That's my scala code that uses java HttpURLConnection:
import java.io._
import java.net._
val token = "myToken"
val url = new URL("http://137.204.57.150:8777/v2/meters/")
val body = "{\"q\": [{\"field\": \"resource\", \"op\": \"eq\", \"value\": \"gdfsf\"}]}"
val bodyLenght = body.length.toString
val connection = url.openConnection().asInstanceOf[HttpURLConnection]
connection.setRequestMethod("GET")
connection.setRequestProperty("Content-Type", "application/json")
connection.setRequestProperty("Content-Length", bodyLength)
connection.setRequestProperty("Accept", "*/*")
connection.setRequestProperty("X-Auth-Token", token)
connection.setDoInput(true)
connection.setDoOutput(true)
//SEND REQUEST
val wr = new DataOutputStream(connection.getOutputStream)
wr.write(body.getBytes)
wr.flush
wr.close
if (connection.getResponseCode == 200)
println("ok")
else
println("error")
What's the difference between my Java implementation and the curl command? I can't see any, I tried checking the header of curl calling it with the -v argument and that's what I get:
* Hostname was NOT found in DNS cache
* Trying 137.204.57.150...
* Connected to 137.204.57.150 (137.204.57.150) port 8777 (#0)
> GET /v2/meters/ HTTP/1.1
> User-Agent: curl/7.37.1
> Host: 137.204.57.150:8777
> Accept: */*
> X-Auth-Token: ...Token....
> Content-Type: application/json
> Content-Length: 60
>
* upload completely sent off: 60 out of 60 bytes
* HTTP 1.0, assume close after body
And then I get the response.
Thank you in advance
I resolved the problem using jetty-client implementation, that lets build http requests in anyway you want. Here's the code (it's not immutable but it's not that bad in scala):
val httpClient = new HttpClient()
httpClient.setConnectTimeout(connectTimeout)
httpClient.setFollowRedirects(false)
httpClient.setStopTimeout(readTimeout)
httpClient.start()
val resp = httpClient.newRequest(uri).
method(HttpMethod.GET).
header("X-Auth-Token",s).
send()
Look that i'm using the blocking API but jetty provides also a Java NIO API.
I found a working plain java solution here, using apache's httpclient, httpcore, and commons-logging libs.
You need to create a class and extend HttpEntityEnclosingRequestBase, overriding the method name:
import org.apache.http.client.methods.HttpEntityEnclosingRequestBase;
public class HttpGetWithEntity extends HttpEntityEnclosingRequestBase {
public final static String METHOD_NAME = "GET";
#Override
public String getMethod() {
return METHOD_NAME;
}
}
Then you just use it like this:
HttpClient httpClient = HttpClientBuilder.create().build();
HttpGetWithEntity e = new HttpGetWithEntity();
e.setURI(new URI(yourURL))
e.setEntity(yourEntity);
HttpResponse response = httpclient.execute(e);
Hope it helps.
In general, the specification does not prohibit body on any type of http request (GET, DELETE etc), so you can do it if needed. However by convention this is atypical.
The problem you're having is that there are assumptions about what you can and can't do in the implementation of URLConnection you're using. In general, you'll be using a HttpUrlConnection (as you cast to), which will actually be implemented by your jvm. For example, here is a sun specific implementation.
If you look at this implementation, you will see it assumes that a GET request where you need the output stream is actually a POST.
If you want a GET with a body, you need to use a different connection method, for example a library like apache-http-client. You could start by looking at this question. There may be better scala alternatives for you to start with.
You use HTTP PUT or POST request when sending request body for Celiometer API.
I checked the Ceilometer documentation and found that all requests with request body use HTTP PUT or POST methods. No GET method with request body.
http://docs.openstack.org/developer/ceilometer/webapi/v2.html
After checking the documentation of Ceilometer and cURL I can suggest two things.
Use URL parameters instead of JSON
As per the documentation you can use the URL parameters or JSON. You can modify your request as specified below to achieve the same thing with URL parameters rather than JSON.
URL("http://YOURHOST.COM:8777/v2/meters/?q.field=resource&q.op=eq&q.value=gdfsf")
In case you have a specific reason not to use URL parameters for your JSON approach I guess encoding is what is missing in your request. Parameters are required to be sent in query parameters only rather than body content. For that I guess you need to try with below encoded data as shown below based on your request in the question.
URL("http://YOURHOST.COM:8777/v2/meters/?q=%5B%7B%22field%22%3A%20%22resource%22%2C%20%22op%22%3A%20%22eq%22%2C%20%22value%22%3A%20%22gdfsf%22%7D%5D%7D")
Here q is the root query parameter name, without token I was not able to validate it.
Replace YOURHOST.COM with ip address for your server as it was showing problem to me even after putting them in code block and please let me know.
you can try like this also
#RequestMapping(value = "/listcategories", method = RequestMethod.GET)
private ModelAndView getCategories() {
ModelAndView modelAndView = new ModelAndView("list-of-categories");
List<Category> categories = categoryService.getAllCategories();
modelAndView.addObject("categories", categories);
return modelAndView;
}
I do not know Java and I haven't had success with Google or trial and error testing...
How would I write this with just using CURL on the command line for a RESTful API authentication? (a php or perl solution would also be okay)
This code is from documentation but I don't plan on using Java and need to translate it.
URL url = new URL("http://www.thingsandstuff.com/resfulness/post");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// i don't know if this is relevant to my question or not...
conn.setRequestMethod("POST");
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
// this is what i think might be the problem...
String userNamePassword = "myusername:mypassword";
userNamePassword =
new String(org.apache.commons.codec.binary.Base64.encodeBase64(userNamePassword
.getBytes()));
conn.setRequestProperty("Authorization", userNamePassword);
I always get a 401 unauthorized error.
I've been just playing with the header argument -H. I don't know if what I am doing is not working because I am truly not doing the authentication right or if it is something else. The data is xml and I am testing with something very simple/straightforward. I am assuming that even if my data/xml/post is incorrect, it would be returning an error instead of an unauthorized.
# bXl1c2VybmFtZTpteXBhc3N3b3Jk == myusername:mypassword base64 encoded....
curl -H "bXl1c2VybmFtZTpteXBhc3N3b3Jk" -d "<datums>" -X POST http://www.thingsandstuff/restfulness/post
curl -H "bXl1c2VybmFtZTpteXBhc3N3b3Jk" -d "<datums>" -X POST http://www.thingsandstuff/restfulness/post
# and like this... with encoded things and not encoded things...
curl -d "<datums>" -X POST http://user:password#www.thingsandstuff.com/restfulness/post
curl -u user:password -d "<datums>" -X POST http://www.thingsandstuff.com/restfulness/post
And for all three, I've tried encoding just the password, both together, both separately, neither... And I'm sure it doesn't help that my username does have a \ and the password does end in an ! but I think, when I'm not encoding them, I am escaping properly.
And the curl man page says that -d will cause curl to pass the data to the server using the content-type application/x-www-form-urlencoded, so that means I don't need to pass an additional header saying the content-type, right? Basically... I only partially know what I'm doing and it doesn't work and there are too many things I don't know about enough to isolate my issue or probably even correct it (yes, just enough to be dangerous).
The reason you are getting the 401 is the Authorization header needs a scheme prefix, in your case Basic
So currently you are sending
Authorization: <base64(username:password)>
But the web server expects
Authorization: Basic <base64(username:password)>
as described here
this change should work:
conn.setRequestProperty("Authorization", "Basic " + userNamePassword);
Okay, so this is what I ended up with...
curl -H "Authorization:dXNlclxuYW1lOnBhc3N3b3JkIQ==" -X POST -d 'data=%3C%3Fxml+version....' http://www.stuffandthings.com:8080/restfulness/post
So... username:password is base 64 encoded together. The data is encoded by ISO/IEC 8859-1 and prefixed with "data=" this particular set restful service.
I ended up putting it in PHP just because it was easier to test continuously that way. And this is what ended up working for that...
$user = 'user\\name';
$pass = 'password!';
$userpass = base64_encode($user.':'.$pass);
$url = 'http://www.thingsandstuff.com:8080/restfulness/post';
if (($handle = fopen("foo.csv", "r")) !== FALSE)
{
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
list($thing1,$thing2) = $data;
$xml = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>
<entry xmlns=\"http://purl.org/atom/ns#\">
<xml data goes here>
<ns:thing1>$param1</ns:thing1>
<ns:thing2>$param2</ns:thing2>
</xml data goes here>
</entry>";
$datums = 'data='.utf8_decode($xml);
$result = do_curl($url,$datums);
// not actually using the response at this
// point... just running through everything
var_dump($result);
}
fclose($handle);
}
function do_curl($url, $data) {
global $userpass, $user, $pass;
$ch = curl_init();
$header = array('Authorization:'.$userpass,
'Content-Type:application/x-www-form-urlencoded',
'Content-Length:'.strlen($data));
// this line was the last thing to be added before it worked
// i didn't try it again, removing the authentication in $header
// to see if this was the only thing actually needed...
curl_setopt($ch, CURLOPT_USERPWD, $user.':'.$pass);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$result = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$hout = curl_getinfo($ch, CURLINFO_HEADER_OUT);
//echo "\n"; var_dump($hout); echo "\n";
curl_close($ch);
return array('status'=>$status, 'result'=>$result);
}
And If I var_dump($hout) from the do_curl function... the header actually sent was:
POST /resfulness/post HTTP/1.1
Authorization: Basic dXNlclxuYW1lOnBhc3N3b3JkIQ==
Host: www.thingsandstuff.com:8080
Accept: */*
Content-Length: 409
Content-Type: application/x-www-form-urlencoded
<?xml version="1.0" encoding="UTF-8" ?>
<entry xmlns="http://purl.org/atom/ns#">
<xml data goes here>
<ns:thing1>value1</ns:thing1>
<ns:thing2>value2</ns:thing2>
</xml data goes here>
</entry>
I'm trying to make multiple calls to a REST API using HttpURLConnection with GET method and retrieving the response through InputStream.
It worked fine previously for 3 months but now it's throwing below exception:
SAXException Occurred during getArtifactsUrl method:: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) [xercesImpl.jar:6.1.0.Final]
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) [xercesImpl.jar:6.1.0.Final]
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) [:1.7.0_03]
Below is the line of code where I'm making the second call to parse the response:
request = (HttpURLConnection) endpointUrl.openConnection();
inputstream = request.getInputStream();
doc = dBuilder.parse(inputstream);
First call is working fine using request and inputstream objects but second call is failing. I tried all possible answers I found in google but no luck:
after every call:
inputstream.close();
request.disconnect();
Remember that request is an HttpURLConnection object.
I greatly appreciate if you can be able to solve this as I this is a high prioirity production issue now!
First you should check for error cases and not assume it's always working.
Try this:
request = (HttpURLConnection) endpointUrl.openConnection();
request.connect(); // not really necessary (done automatically)
int statusCode = request.getResponseCode();
if (statusCode == 200) { // or maybe other 2xx codes?
// Success - should work if server gives good response
inputstream = request.getInputStream();
// if you get status code 200 and still have the same error, you should
// consider logging the stream to see what document you get from server.
// (see below *)
doc = dBuilder.parse(inputstream);
} else {
// Something happened
// handle error, try again if it makes sense, ...
if (statusCode == 404) ... // resource not found
if (statusCode == 500) ... // internal server error
// maybe there is something interesting in the body
inputstream = request.getErrorStream();
// read and parse errorStream (but probably this is not the body
// you expected)
}
Have a look at the List of HTTP status codes.
And in some nasty cases, there are other problems which are not easy to detect if you just sit behind HttpURLConnection. Then you could enable logging or snoop the TCP/IP traffic with an apropriate tool (depends on your infrastructure, rights, OS, ...). This SO post might help you.
*) In your case I suppose that you're getting a non-error status code from the server but unparseable XML. If logging the traffic is not your thing, you could read the InputStream, write it to a file and then process the stream like before. Of course you can write the stream to a ByteArrayOutputStream, get the byte[] and write that Bytes to a file and then convert them to a ByteArrayInputStream and give this to your XML-parser. Or you could use Commons IO TeeInputStream to handle that for you.
There are cases where connection.getResponseCode() throws an exception. Then it was not possible to parse the HTTP header. This should only happen if there are strange errors in server software, hardware or perhaps a firewall, proxy or load balancer not behaving well.
One more thing: You might consider choosing an HTTP Client library and not directly use HttpURLConnection.
I know there are a lot of questions related to this issue, but I'm facing an specific requirement for this purpose of posting data from PHP to a Rest Web Service. The details are explained above, but in summary, when I post data to a url (REST WS) and set the CURLOPT_POSTFIELDS the data is not being added to the request.
The scenario: I have a lot of Java Web Services (REST) running as modules, for example, I have a fileUploadModule which is a REST, I have a databaseModule which is another rest and finally a SearchModule, which is another REST.
I can invoke them directly my rest modules using a link like:
http://[MY IP]:8020/system.file.ws.module.ModuleFile/getResults/jsonp?fileName=fileName
http://[MY IP]:8021/system.search.ws.module.ModuleSearch/getResults/jsonp?xmlQuery=myXml
For the case of files and database, the programmer that was managing the code before me used gwt that connected to the module through a proxy; for instance:
http://[MY_PROXY_IP]:8013/system.file.ws.module.ModuleFile/getResults/jsonp?fileName=fileName
and in my proxy I can print the value of the request received, in this case I use a GET and I can print the request as:
GET /system.file.ws.module.ModuleFile/getResults/jsonp?fileName=idc1&folderType=campaign&callback=__gwt_jsonp__.P0.onSuccess&failureCallback=__gwt_jsonp__.P0.onFailure HTTP/1.1
. Now I am responsible for search that should run through PHP. I tested the url directly to the module and it works, but if I try to it by a proxy it does not seems to be working, it reach my proxy but when I print the request it is incomplete:
POST /system.search.ws.module.ModuleSearch/getResults HTTP/1.1
and I am supposed to receive something like the module file, I share my php code, all seems to be ok, but I don't know what can I be doing wrong... when I set the parameters in CURLOPT_POSTFIELDS the string is not being set
$url = "http://192.168.3.41:8013/system.search.ws.module.ModuleSearch/getResults";
try {
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_POST, 1);
$rawXml = $_POST['rawXml'];
$rawXml = str_replace("%", "%25", $rawXml);
$rawXml = str_replace("&", "%26", $rawXml);
$rawXml = str_replace("=", "%3D", $rawXml);
echo $rawXml;
curl_setopt ($ch, CURLOPT_POSTFIELDS,'xmlQuery='.$rawXml);
$info = curl_exec ($ch);
curl_close ($ch);
echo $info;
} catch (Exception $e){
echo $e->getMessage();
}
please I would really appreciate your help or observations. Than you very much in advance.
After long time, I saw this without an answer. I discovered the fail of this some time ago. This issue occurred because, when reaching server side, for some reason a batch file I did not noiced was adding an additional line to my content, and whenever I read the request content with my proxies, I used a "\n" delimiter, I mean, I have been reading my data using as the EOF indicator a line jump, that's why the content was never shown. I had to modify the code inside my proxy to allow reading the request until the end, not when finding a "\n" character. I mean, the content of the post was set in every case, but a batch process was corrupting that data. So thata was the issue, I just solved it by making sure that my reader always read my entire file considering even line jumps and white lines.
Regards.