Develop Reference

Getting java.io.IOException: Server returned HTTP response code: 400 for URL: when using a url which return 400 status code

I am trying to perform a get request using Groovy using the below code:
String url = "url of endpoint"
def responseXml = new XmlSlurper().parse(url)
If the endpoint returns status as 200 then everything works good but there is one case where we have to validate the error response like below and status returned is 400:
<errors>
<error>One of the following parameters is required: xyz, abc.</error>
<error>One of the following parameters is required: xyz, mno.</error>
</errors>
In this case parse method throws :
java.io.IOException: Server returned HTTP response code: 400 for URL: "actual endpoint throwing error"
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1900)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:646)
at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:150)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:831)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:796)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:142)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:644)
at groovy.util.XmlSlurper.parse(XmlSlurper.java:205)
at groovy.util.XmlSlurper.parse(XmlSlurper.java:271)
Can anyone pls suggest how to handle if server give error message by throwing 400 status code?

In the question since we are getting 400 status code for GET request. So in built XmlSlurper().parse(URI) method does not work as it throw io.Exception.
Groovy also support HTTP methods for api request and response and the below worked for me:
def getReponseBody(endpoint) {
URL url = new URL(endpoint)
HttpURLConnection get = (HttpURLConnection)url.openConnection()
get.setRequestMethod("GET")
def getRC = get.getResponseCode()
BufferedReader br = new BufferedReader(new InputStreamReader(get.getErrorStream()))
StringBuffer xmlObject = new StringBuffer()
def eachLine
while((eachLine = br.readLine()) !=null){
xmlObject.append(eachLine)
}
get.disconnect()
return new XmlSlurper().parseText(xmlObject.toString())
}

Getting the response text from the HttpURLConnection class rather than implicitly through XmlSlurper allows you much more flexibility in handling unsuccessful responses. Try something like this:
def connection = new URL('https://your.url/goes.here').openConnection()
def content = { ->
try {
connection.content as String
} catch (e) {
connection.responseMessage
}
}()
if (content) {
def responseXml = new XmlSlurper().parseText(content)
doStuffWithResponseXml(responseXml)
}
Even better would be to use an actual full-featured HTTP client, like the Spring Framework's HttpClient or RestTemplate classes.

You should check the return code and than obtain the error stream from http request instance in case of an error. The problem itself has nothing to do with JsonSlurper, as no instance of "input stream" is returned from http request instance if service returns not successfull return codes (400, 401, 500 etc.) POST example can be seen below:
http= new URL("yourUrl").openConnection() as HttpURLConnection
http.setRequestMethod('POST')
http.setDoOutput(true)
http.setRequestProperty("Content-Type", 'application/json')
http.setRequestProperty("Accept", 'application/json')
http.setRequestProperty("Authorization", "Bearer $yourTokenVariable")
http.outputStream.write(data.getBytes("UTF-8"))
http.connect()
if(http.getResponseCode() != 200 && http.getResponseCode() != 201){
throw new InvalidInputException("There was an error: " + http.getErrorStream().getText("UTF-8"))
} else {
//You can take input stream here
}

How to read an HTTP response from a server (as a web proxy) and send response to client

I am writing a web proxy and so far I can read a GET request from a client, format it, and send it to the server, I believe that I have been able to get the response back from the server, but am unsure of how to send the response to the client.
Scanner readClient = new Scanner(new InputStreamReader(client.getInputStream()));
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(client.getInputStream()));
System.out.println("Client Request: ");
String request;
String host = "";
String path = "";
String[] parts = new String[4];
while((request = bufferedReader.readLine())!= null) {
if (request.indexOf("deflate") != -1) {
break;
}
if(request.indexOf("GET") != -1){
parts = request.split(" ");
path = parts[1];
System.out.println("THIS IS THE PATH: " + path);
}
if(request.indexOf("Host") != -1){
parts = request.split(": ");
host = parts[1];
System.out.println("THIS IS THE HOST: " + host);
}
System.out.println(request);
}
Socket server = new Socket(host, 80);
System.out.println("Successfully connected to host: " + host);
PrintWriter writeServer = new PrintWriter(new DataOutputStream(server.getOutputStream()));
InputStream readServer = server.getInputStream();
writeServer.print("GET " + path + "\r\n" + "Host: " + host + "\r\n" + "Connection: close\r\n\r\n");
writeServer.flush();
OutputStream writeClient = client.getOutputStream();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte buffer[] = new byte[1024];
for(int s; (s=readServer.read(buffer)) != -1; )
{
baos.write(buffer, 0, s);
}
byte result[] = baos.toByteArray();
System.out.println("message sent");
}
catch (Exception e) {
System.out.println("Start Exception: " + e.getMessage());
}
}
** Not sure how I am supposed to record edits made to the question, but I have changed my wording and updated my code as well as included more of it.

You just need to read and copy the input to the output, taking note of the content-length or transfer-encoding headers on the way past, and stop when you exhaust either the content-length or whatever the transfer encoding thinks is the end of the response.

What kind of errors are you trying to catch? Did some homework last term using Scanner(URL.openStream()) and for anything "not normal" that would display as an error in a browser it would throw an Exception. Here's my catch() statement with some comments, it worked for what I needed at the time.
// do we have an error?
catch (Exception ex) {
// rather than specific exceptions related to the type of
// error (network, protocol, webserver content/configuration)
// the java.net.URL.openStream(URL) seems to return
// a different message in .getMessage() that you have to
// parse to figure out what happened.
// would these messages be different in a different java/jvm implementation?
String errorMsg=ex.getMessage();
// nicer errors below
//System.out.println("Error: "+errorMsg+"\n\r");
// what makes up our URL? this lets us get the hostname
// easily as urlParts[2].
String[] urlParts=theURL.split("/");
// on DNS failure (use http://aintthere.example.com as test URL)
// Exception.getMessage() seems to return the desired hostname
if(errorMsg.indexOf(urlParts[2])==0){
System.out.println("DNS error - invalid or unknown hostname");
}
// on a 404 error (use http://www.example.com/aintthere) the
// Exception.getMessage() appears to return the URL requested.
if(errorMsg.indexOf(theURL)==0){
System.out.println("The requested URL does not exist: "+theURL);
}
// no route to host or host off line and/or denying connections
if(errorMsg.indexOf("Connection timed out")==0){
System.out.println("That host is unreachable or is not allowing connections");
}
// turns out lots of different SSL errors - invalid certs, self signed certs, mis-matched hostnames,
// all sorts of things. seems easier to parse for ".security." in the message since
// they seem to come either from java.security.cert.* or sun.security.*
if(errorMsg.indexOf(".security.")!=-1){
System.out.println("Insecure SSL connection attempt - not allowed");
}
// both 500 (Internal Server Error) and 403 (Access to Resource Forbidden)
// produce nice standard looking error messages with the error number in them, so
// we check for that. Why doesn't 404 do that?
if(errorMsg.indexOf("HTTP response code: 500")!=-1){
System.out.println("The webserver is suffering from its own issues - Internal Server Error detected");
}
if(errorMsg.indexOf("HTTP response code: 403")!=-1){
System.out.println("Access to that resource is forbidden by the webserver configuration");
}
} // end catch

Java Jersey ClientResponse returns wrong Status from getClientResponseStatus

I feel like I am missing something here. I have a filter which prints out my server's returned information and I report that I am returning the correct response (403). I wrote a JUnit test to verify this logic and many times I am reporting 200 instead of 403. The weird part is that my server logs still show that I sent a 403. Is there some known bug in Jersey 1.17 that I am not aware of and I need to upgrade to resolve? I am not really in a position to upgrade at this point in time so I am hoping there is some bug in my logic. Here is my test case.
#Test
public void testIdsOwnedBySomeoneElse()
{
final Login user1Cred = Logins.getInstance().getLogin(Logins.LoginType.User1);
final Login user2Cred = Logins.getInstance().getLogin(Logins.LoginType.User2);
final ServiceEndpointAuthenticated authUser1 = LoginHelper.Login(user1Cred);
final ServiceEndpointAuthenticated authUser2 = LoginHelper.Login(user2Cred);
// Create generic entry owned by user 1
BigInteger user1Id = null;
{
final Object payload = endpoint.CreateEntity(authUser1.getUserId());
final ClientResponse response = endpoint.Post(authUser1, payload);
assertTrue(Status.OK == response.getClientResponseStatus());
final byte[] data = Utilities.getBytes(response.getEntityInputStream());
user1Id = endpoint.getEntityId(data);
}
// Using User2, try to use that id from user1!
{
// test 1
final MyEndpointWrapper endpoint = new MyEndpointWrapper(user1Id, validId);
final Object payload = endpoint.CreateEntity(authUser2.getUserId());
final ClientResponse response = endpoint.Post(authUser2, payload);
final Status status = response.getClientResponseStatus();
System.out.println("Returned status = " + status);
if (status != Status.FORBIDDEN)
{
byte[] data = Utilities.getBytes(response.getEntityInputStream());
String toString = null;
try
{
toString = new String(data, "UTF-8");
}
catch (UnsupportedEncodingException e)
{
}
System.out.println("data: " + toString);
}
assertEquals("Status " + status + " is not forbidden!", Status.FORBIDDEN, status);
}
{
// test 2
final MyEndpointWrapper endpoint = new MyEndpointWrapper(validId, user1Id);
final Object payload = endpoint.CreateEntity(authUser2.getUserId());
final ClientResponse response = endpoint.Post(authUser2, payload);
final Status status = response.getClientResponseStatus();
System.out.println("Returned status = " + status);
if (status != Status.FORBIDDEN)
{
int i = 9;
}
assertEquals("Status " + status + " is not forbidden!", Status.FORBIDDEN, status);
}
// Go ahead and delete this data for cleanup
assertTrue(Status.OK == endpoint.Delete(authUser1, user1Id).getClientResponseStatus());
}
My generic code first logs into our server for the creds. These creds are "attached" to the WebResource and it attaches the proper headers automatically when I build my request. I first create an entity, post it, and store the returned id to be used by another user. I create another endpointwrapper which references that violation id and I attempt to post with that id. The server logs:
INFO: RESPONSE: 403 http://myendpoint MediaType:(application/json) Payload: 232 MyErrorMessage
I can even print this message out (as shown above)! The part I dont understand is that getClientResponseStatus returned to me OK. Why?
My Post code looks like:
#Override
public ClientResponse Post(ServiceEndpointAuthenticated endpoint, Object entity)
{
MyUploadData uploadData = (MyUploadData)entity;
return endpoint.getResourceBuilder("/myendpoint")
.accept(MediaTypeExt.APPLICATION_JSON)
.type(MediaTypeExt.APPLICATION_JSON)
.post(ClientResponse.class, gson.toJson(uploadData));
}
[UPDATE]
I ran wire capture and actually do see 200 being sent back! This does appear to be something inside of Jersey Server. Here is what I see:
When working:
Request: 1099 17.021219000 127.0.0.1 127.0.0.1 HTTP 2214 POST /myEndpoint HTTP/1.1 (application/json)
Response: 1153 17.042535000 127.0.0.1 127.0.0.1 HTTP 628 HTTP/1.1 400 Bad Request (application/json)
When not working:
Request: 1161 17.044313000 127.0.0.1 127.0.0.1 HTTP 250 POST /myEndpoint HTTP/1.1 (application/json)
Response: 1217 17.066059000 127.0.0.1 127.0.0.1 HTTP 412 HTTP/1.1 200 OK
When it works I see my normal headers in the response (eg: Access-Control-*, Pragma no cache, etc). When it doesn't work I dont see any of my headers but I do see "Transfer-Encoding: chunked" and my response is my error message but my response code is 200. I added an explicit Trace statement in the server right before I sent my response to ensure I am sending the right Status and I am.
I am okay with allowing chunked transfer but I am not really okay with losing my desired http response.

Incase anyone else encounters something similar. After digging around I finally found the problem. We have a heartbeat on some of our endpoints. Some endpoints can take longer than expected time. To ensure the client doesn't disconnect prematurely we have a component which attaches to the ServletOutputStream. This sends a space to the client to keep the connection alive.
When an error is thrown (caught by our new exception remapper), this keep-alive component was not being shutdown properly. This caused Jersey to switch into chunked mode. Ensuring the keep-alive component was shutdown properly resolved the problem.

Can't get Java's GZIPInputStream to read "gzip" response from server when using "Range" header

I've got a bit of code I've been using for a while to fetch data from a web server and a few months ago, I added compression support which seems to be working well for "regular" HTTP responses where the whole document is contained in the response. It does not seem to be working when I use a Range header, though.
Here is the code doing the real work:
InputStream in = null;
int bufferSize = 4096;
int responseCode = conn.getResponseCode();
boolean error = 5 == responseCode / 100
|| 4 == responseCode / 100;
int bytesRead = 0;
try
{
if(error)
in = conn.getErrorStream();
else
in = conn.getInputStream();
// Buffer the input
in = new BufferedInputStream(in);
// Handle compressed responses
if("gzip".equalsIgnoreCase(conn.getHeaderField("Content-Encoding")))
in = new GZIPInputStream(in);
else if("deflate".equalsIgnoreCase(conn.getHeaderField("Content-Encoding")))
in = new InflaterInputStream(in, new Inflater(true));
int n;
byte[] buffer = new byte[bufferSize];
// Now, just write out all the bytes
while(-1 != (n = in.read(buffer)))
{
bytesRead += n;
out.write(buffer, 0, n);
}
}
catch (IOException ioe)
{
System.err.println("Got IOException after reading " + bytesRead + " bytes");
throw ioe;
}
finally
{
if(null != in) try { in.close(); }
catch (IOException ioe)
{
System.err.println("Could not close InputStream");
ioe.printStackTrace();
}
}
Hitting a URL with the header Accept-Encoding: gzip,deflate,identity works just great: I can see that the data is returned by the server in compressed format, and the above code decompressed it nicely.
If I then add a Range: bytes=0-50 header, I get the following exception:
Got IOException after reading 0 bytes
Exception in thread "main" java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at [my code]([my code]:511)
Line 511 in my code is the line containing the in.read() call. The response includes the following headers:
Content-Type: text/html
Content-Encoding: gzip
Content-Range: bytes 0-50/751
Content-Length: 51
I have verified that, if I don't attempt to decompress the response, I actually get 51 bytes in the response... it's not a server failure (at least that I can tell). My server (Apache httpd) does not support "deflate", so I can't test another compression scheme (at least not right now).
I've also tried to request much more data (like 700 bytes of the total 751 bytes in the target resource) and I get the same kind of error.
Is there something I'm missing?
Update
Sorry, I forgot to include that I'm hitting Apache/2.2.22 on Linux. There aren't any server bugs I'm aware of. I'll have a bit of trouble verifying the compressed bytes that I retrieve from the server, as the "gzip" Content-Encoding is quite bare... e.g. I believe I can't just use "gunzip" on the command-line to decompress those bytes. I'll give it a try, though.

You can use 'gunzip' to decompress it, just keep in mind that the first 50 bytes probably aren't enough for gzip to decompress anything (headers, dictionaries etc). Try this: wget -O- -q <URL> | head -c 50 | zcat with your URL to see whether normal gzip works where your code fails.

Sigh switching to another server (happens to be running Apache/2.2.25) shows that my code does in fact work. The original target server appears to be affected by AWS's current outage in the US-EAST availability zone. I'm going to chalk this up to network errors and close this question. Thanks to those who offered suggestions.

FileNotFoundException while getting the InputStream object from HttpURLConnection

I am trying to send a post request to a url using HttpURLConnection (for using cUrl in java).
The content of the request is xml and at the end point, the application processes the xml and stores a record to the database and then sends back a response in form of xml string. The app is hosted on apache-tomcat locally.
When I execute this code from the terminal, a row gets added to the db as expected. But an exception is thrown as follows while getting the InputStream from the connection
java.io.FileNotFoundException: http://localhost:8080/myapp/service/generate
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.kodeplay.helloworld.HttpCurl.main(HttpCurl.java:30)
Here is the code
public class HttpCurl {
public static void main(String [] args) {
HttpURLConnection con;
try {
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
con.setRequestMethod("POST");
con.setDoOutput(true);
con.setDoInput(true);
File xmlFile = new File("test.xml");
String xml = ReadWriteTextFile.getContents(xmlFile);
con.getOutputStream().write(xml.getBytes("UTF-8"));
InputStream response = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response));
for (String line ; (line = reader.readLine()) != null;) {
System.out.println(line);
}
reader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Its confusing because the exception is traced to the line InputStream response = con.getInputStream(); and there doesn't seem to be any file involved for a FileNotFoundException.
When I try to open a connection to an xml file directly, it doesn't throw this exception.
The service app uses spring framework and Jaxb2Marshaller to create the response xml.
The class ReadWriteTextFile is taken from here
Thanks.
Edit:
Well it saves the data in the DB and sends back a 404 response status code at the same time.
I also tried doing a curl using php and print out the CURLINFO_HTTP_CODE which turns out to be 200.
Any ideas on how do I go about debugging this ? Both service and client are on the local server.
Resolved:
I could solve the problem after referring to an answer on SO itself.
It seems HttpURLConnection always returns 404 response when connecting to a url with a non standard port.
Adding these lines solved it
con.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
con.setRequestProperty("Accept","*/*");

I don't know about your Spring/JAXB combination, but the average REST webservice won't return a response body on POST/PUT, just a response status. You'd like to determine it instead of the body.
Replace
InputStream response = con.getInputStream();
by
int status = con.getResponseCode();
All available status codes and their meaning are available in the HTTP spec, as linked before. The webservice itself should also come along with some documentation which overviews all status codes supported by the webservice and their special meaning, if any.
If the status starts with 4nn or 5nn, you'd like to use getErrorStream() instead to read the response body which may contain the error details.
InputStream error = con.getErrorStream();

FileNotFound is just an unfortunate exception used to indicate that the web server returned a 404.

To anyone with this problem in the future, the reason is because the status code was a 404 (or in my case was a 500). It appears the InpuStream function will throw an error when the status code is not 200.
In my case I control my own server and was returning a 500 status code to indicate an error occurred. Despite me also sending a body with a string message detailing the error, the inputstream threw an error regardless of the body being completely readable.
If you control your server I suppose this can be handled by sending yourself a 200 status code and then handling whatever the string error response was.

For anybody else stumbling over this, the same happened to me while trying to send a SOAP request header to a SOAP service. The issue was a wrong order in the code, I requested the input stream first before sending the XML body. In the code snipped below, the line InputStream in = conn.getInputStream(); came immediately after ByteArrayOutputStream out = new ByteArrayOutputStream(); which is the incorrect order of things.
ByteArrayOutputStream out = new ByteArrayOutputStream();
// send SOAP request as part of HTTP body
byte[] data = request.getHttpBody().getBytes("UTF-8");
conn.getOutputStream().write(data);
if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
Log.d(TAG, "http response code is " + conn.getResponseCode());
return null;
}
InputStream in = conn.getInputStream();
FileNotFound in this case was an unfortunate way to encode HTTP response code 400.

FileNotFound in this case means you got a 404 from your server - could it be that the server does not like "POST" requests?

FileNotFound in this case means you got a 404 from your server
You Have to Set the Request Content-Type Header Parameter
Set “content-type” request header to “application/json” to send the request content in JSON form.
This parameter has to be set to send the request body in JSON format.
Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
Full Script ->
public class SendDeviceDetails extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... params) {
String data = "";
String url = "";
HttpURLConnection con = null;
try {
// From the above URL object,
// we can invoke the openConnection method to get the HttpURLConnection object.
// We can't instantiate HttpURLConnection directly, as it's an abstract class:
con = (HttpURLConnection)new URL(url).openConnection();
//To send a POST request, we'll have to set the request method property to POST:
con.setRequestMethod("POST");
// Set the Request Content-Type Header Parameter
// Set “content-type” request header to “application/json” to send the request content in JSON form.
// This parameter has to be set to send the request body in JSON format.
//Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
//Set Response Format Type
//Set the “Accept” request header to “application/json” to read the response in the desired format:
con.setRequestProperty("Accept", "application/json");
//To send request content, let's enable the URLConnection object's doOutput property to true.
//Otherwise, we'll not be able to write content to the connection output stream:
con.setDoOutput(true);
//JSON String need to be constructed for the specific resource.
//We may construct complex JSON using any third-party JSON libraries such as jackson or org.json
String jsonInputString = params[0];
try(OutputStream os = con.getOutputStream()){
byte[] input = jsonInputString.getBytes("utf-8");
os.write(input, 0, input.length);
}
int code = con.getResponseCode();
System.out.println(code);
//Get the input stream to read the response content.
// Remember to use try-with-resources to close the response stream automatically.
try(BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream(), "utf-8"))){
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
System.out.println(response.toString());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (con != null) {
con.disconnect();
}
}
return data;
}
#Override
protected void onPostExecute(String result) {
super.onPostExecute(result);
Log.e("TAG", result); // this is expecting a response code to be sent from your server upon receiving the POST data
}
and call it
new SendDeviceDetails().execute("");
you can find more details in this tutorial
https://www.baeldung.com/httpurlconnection-post

The solution:
just change localhost for the IP of your PC
if you want to know this: Windows+r > cmd > ipconfig
example: http://192.168.0.107/directory/service/program.php?action=sendSomething
just replace 192.168.0.107 for your own IP (don't try 127.0.0.1 because it's same as localhost)

Please change
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
To
con = (HttpURLConnection) new URL("http://YOUR_IP:8080/myapp/service/generate").openConnection();