I cannot comprehend why doesn't the following code does not put a packet onto wire (confirmed via wireshark). It is a fairly standard method of sending an HTTP POST request, as I believe. I don't intend to read anything just POST.
private void sendRequest() throws IOException {
String params = "param=value";
URL url = new URL(otherUrl.toString());
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoOutput(true);
con.setDoInput(true); //setting this to `false` does not help
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "text/plain");
con.setRequestProperty("Content-Length", "" + Integer.toString(params.getBytes().length));
con.setRequestProperty("Accept", "text/plain");
con.setUseCaches(false);
con.connect();
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
wr.writeBytes(params);
wr.flush();
wr.close();
//Logger.getLogger("log").info("URL: "+url+", response: "+con.getResponseCode());
con.disconnect();
}
What happens is... actually nothing, unless I try to read anything. For example by uncommenting the above log line which reads the response code. Trying to read a response via con.getInputStream(); also works. There is no movement of packets. When I uncomment the getResponseCode, I can see that http POST is sent, and then 200 OK is sent back. The order is proper. I.e. I don't get some wild response before sending POST. Everything else looks exactly the same (I can attach wireshark screenshots if needed.). In the debugger the code executes (i.e. does not block anywhere).
I don't understand under what circumstances this can be happening. I belive it should be possible, to send a POST request with con.setDoInput(false);. Currently it doesn't send anything or fails (when trying to execute con.getResponseCode()) with an exception because I obviously promised I won't read anything.
It might be relevant, that before sendRequest I do request some data from the same site, but I trust I close everything properly. I.e:
public static String getData(String urlAddress) throws MalformedURLException, IOException {
URL url = new URL(urlAddress);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setDoOutput(false);
InputStream in = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder data = new StringBuilder();
String line;
while((line = reader.readLine()) != null) {
data.append(line);
}
reader.close();
in.close();
con.getResponseCode();
con.disconnect();
return data.toString();
}
The server for url in both cases is the same, port also, so I believe it is possible to use the same socket for communication. The above code works and retrieves the data properly.
I am not sure, maybe I don't clean something, and it gets cached, so with out an explicit read the POST gets delayed. There is no other traffic on the socket.
Unless you're using fixed-length or chunked transfer mode, HttpURLConnection will buffer all your output until you call getInputStream() or getResponseCode(), so that it can send a correct Content-length header.
If you call getResponseCode() you should have a look at its value.
Related
NOTICE UPDATE!!
The problem got solved and i added my own answer in the thread
In short, I have attempted to add the parameter "scan_id" value but since it is a POST i can't add the value directly in the url path.
using the code i already have, how would i go about modifying or adding so that the url is correct, that is, so that it accepts my POST?.
somehow i have been unable to find any examples that have helped me in figuring out how i would go about doing this..
I know how to do a POST with a payload, a GET with params. but a post with Params is very confusing to me.
Appreciate any help. (i'd like to continue using HttpUrlConnection unless an other example is provided that also tells me how to send the request and not only configuring the path.
I've tried adding it to the payload.
I've tried UriBuilder but found it confusing and in contrast with the rest of my code, so wanted to ask for help with HttpUrlConnection.
URL url = new URL("http://localhost/scans/{scan_id}/launch");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("tmp_value_dont_mind_this", "432432");
con.setRequestProperty("X-Cookie", "token=" + "43432");
con.setRequestProperty("X-ApiKeys", "accessKey="+"43234;" + " secretKey="+"43234;");
con.setDoInput(true);
con.setDoOutput(true); //NOT NEEDED FOR GETS
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
//First example of writing (works when writing a payload)
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
//second attemp at writing, doens't work (wanted to replace {scan_id} in the url)
DataOutputStream writer = new DataOutputStream(con.getOutputStream());
writer.writeChars("scan_id=42324"); //tried writing directly
//writer.write(payload);
writer.close();
Exception:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 400 for URL: http://localhost/scans/launch
I'd like one of the three response codes because then i know the Url is correct:
200 Returned if the scan was successfully launched.
403 Returned if the scan is disabled.
404 Returned if the scan does not exist.
I've tried several urls
localhost/scans/launch,
localhost/scans//launch,
localhost/scans/?/launch,
localhost/scans/{scan_id}/launch,
So with the help of a friend and everyone here i solved my problem.
The below code is all the code in an entire class explained bit by bit. at the bottom you have the full class with all its syntax etc, that takes parameters and returns a string.
in a HTTP request there are certain sections.
Such sections include in my case, Request headers, parameters in the Url and a Payload.
depending on the API certain variables required by the API need to go into their respective category.
My ORIGINAL URL looked like this: "http://host:port/scans/{scan_id}/export?{history_id}"
I CHANGED to: "https://host:port/scans/" + scan_Id + "/export?history_id=" + ID;
and the API i am calling required an argument in the payload called "format" with a value.
String payload = "{\"format\" : \"csv\"}";
So with my new URL i opened a connection and set the request headers i needed to set.
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
The setDoOutput should be commented out when making a GET request.
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Cookie", "token=" + token);
con.setRequestProperty("X-ApiKeys", "accessKey="+"23243;" +"secretKey="+"45543;");
Here i write to the payload.
//WRITING THE PAYLOAD to the http call
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
After i've written the payload i read whatever response i get back (this depends on the call, when i do a file download (GET Request) i don't have a response to read as i've already read the response through another piece of code).
I hope this helps anyone who might encounter this thread.
public String requestScan(int scan_Id, String token, String ID) throws MalformedInputException, ProtocolException, IOException {
try {
String endpoint = "https://host:port/scans/" + scan_Id + "/export?history_id=" ID;
URL url = new URL(endpoint);
String payload= "{\"format\" : \"csv\"}";
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Cookie", "token=" + token);
con.setRequestProperty("X-ApiKeys", "accessKey="+"324324;" +
"secretKey="+"43242;");
//WRITING THE PAYLOAD to the http call
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
//READING RESPONSE
BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuffer jsonString = new StringBuffer();
String line;
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
con.disconnect();
return jsonString.toString();
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
}
As discussed here the solution would be to change the content type to application/x-www-form-urlencoded, but since you are already using application/json; charset=UTF-8 (which I am assuming is a requirement of your project) you have no choise to redesign the whole thing. I suggest you one of the following:
Add another GET service;
Add another POST service with content type application/x-www-form-urlencoded;
Replace this service with one of the above.
Do not specify the content type at all so the client will accept anything. (Don't know if possible in java)
If there are another solutions I'm not aware of, I don't know how much they would be compliant to HTTP protocol.
(More info)
Hope I helped!
Why you are not using like this. Since you need to do a POST with HttpURLConnection, you need to write the parameters to the connection after you have opened the connection.
String urlParameters = "scan_id=42324";
byte[] postData = urlParameters.getBytes(StandardCharsets.UTF_8);
DataOutputStream dataOutputStream = new DataOutputStream(conn.getOutputStream());
dataOutputStream.write(postData);
Or if you have launch in the end, just change the above code to the following,
String urlParameters = "42324/launch";
byte[] postData = urlParameters.getBytes(StandardCharsets.UTF_8);
DataOutputStream dataOutputStream = new DataOutputStream(conn.getOutputStream());
dataOutputStream.write(postData);
URL url = new URL("http://localhost/scans/{scan_id}/launch");
That line looks odd to me; it seems you are trying to use a URL where you are intending the behavior of a URI Template.
The exact syntax will depend on which template implementation you choose; an implementation using the Spring libraries might look like:
import org.springframework.web.util.UriTemplate;
import java.net.url;
// Warning - UNTESTED code ahead
UriTemplate template = new UriTemplate("http://localhost/scans/{scan_id}/launch");
Map<String,String> uriVariables = Collections.singletonMap("scan_id", "42324");
URI uri = template.expand(uriVariables);
URL url = uri.toURL();
My goal is to create a realtime service alert feed and send it over to a server that I made in Java with an HTTP post request. The first step I did was to create a copy of the example alert feed posted here and it seems I was successfully able to do that as I was able to print it out the message. https://developers.google.com/transit/gtfs-realtime/examples/alerts
The next step that I did is to create an HTTP connection and send the feed over with the POST request. This is what I have in my client code and example here is the feed name.
String url = "https://localhost:8080";
URL obj = new URL(url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
//add reuqest header
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", "Mozilla/5.0");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Content-Type", "application/x-protobuf");
con.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(con.getOutputStream());
example.build().writeDelimitedTo(wr);
wr.flush();
wr.close();
My server code is simply this so far.
ServerSocket server = new ServerSocket(8080);
System.out.println("Listening for connection on port 8080 ....");
while (true) {
try (Socket socket = server.accept()) {
FeedMessage feed = FeedMessage.parseDelimitedFrom(socket.getInputStream());
Date today = new Date();
String httpResponse = "HTTP/1.1 200 OK\r\n\r\n" + today;
socket.getOutputStream().write(httpResponse.getBytes("UTF-8"));
}
}
The question here is that I get the Protocol message contained an invalid tag 0 on the server side. I would like some help on trying to resolve this issue. Maybe I am not parsing it correctly.
Update #2
I have tried to parse the HTTP headers to get to the payload like comments have said. But my code hangs and the output to print the headers on the terminal looks serialized.
DataInputStream isr = new DataInputStream(socket.getInputStream());
Reader reader = new InputStreamReader(isr);
BufferedReader reader2 = new BufferedReader(reader);
String line = reader2.readLine();
System.out.println("get lines");
while (!line.isEmpty()) {
System.out.println(line);
line = reader2.readLine();
}
You are dealing with raw sockets at the server, but the payload is encoded in an http request body. You are going to need to parse the http request through to the payload, and when you have just the body : send that to protobuf. Right now, you're sending the http headers to protobuf, which doesn't make sense. That could mean parsing through to \r\n\r\n, but it would help if you could make use of the content-length header, and even better if you can use a pre-built http library.
I'm writing a small crawler for sites in English only, and doing that by opening a URL connection. I set the encoding to utf-8 both on the request, and the InputStreamReader but I continue to get gobbledigook for some of the requests, while others work fine.
The following code represents all the research I did and advice out there. I have also tried changing URLConnection to HttpURLConnection with no luck. Some of the returned strings continue to look like this:
??}?r?H????P?n?c??]?d?G?o??Xj{?x?"P$a?Qt?#&??e?a#?????lfVx)?='b?"Y(defUeefee=??????.??a8??{O??????zY?2?M???3c??#
What am I missing?
My code:
public static String getDocumentFromUrl(String urlString) throws Exception {
String wholeDocument = null;
URL url = new URL(urlString);
URLConnection conn = url.openConnection();
conn.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
conn.setRequestProperty("Accept-Charset", "utf-8");
conn.setConnectTimeout(60*1000); // wait only 60 seconds for a response
conn.setReadTimeout(60*1000);
InputStreamReader isr = new InputStreamReader(conn.getInputStream(), "utf-8");
BufferedReader in = new BufferedReader(isr);
String inputLine;
while ((inputLine = in.readLine()) != null) {
wholeDocument += inputLine;
}
isr.close();
in.close();
return wholeDocument;
}
The server is sending the document GZIP compressed. You can set the Accept-Encoding HTTP header to make it send the document in plain text.
conn.setRequestProperty("Accept-Encoding", "identity");
Even so, the HTTP client class handles GZIP compression for you, so you shouldn't have to worry about details like this. What seems to be going on here is that the server is buggy: it does not send the Content-Encoding header to tell you the content is compressed. This behavior seems to depend on the User-Agent, so that the site works in regular web browsers but breaks when used from Java. So, setting the user agent also fixes the issue:
conn.setRequestProperty("User-Agent", "Mozilla/5.0"); // for example
I am using HttpURLConnection for making POST requests. I observe always the same behaviour during tests:
first request runs very fast (miliseconds)
all following requests take one second + some miliseconds
So something is causing 1 second delay. What can it be? The delay is happening exactly in HttpURLConnection#getInputStream().
I replaced the implementation with HttpClient - then everything is OK, no second delays (so it is not the server fault). But I really don't want to use any external dependency, so I would like to fix the HttpURLConnection thing... Any ideas?
Below current implementation. I tried some tips from stackoverflow (adding headers to the request), but with no success.
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Length", ""
+ (body == null ? 0 : body.length));
// Send post request
con.setDoOutput(true);
OutputStream wr = con.getOutputStream();
if (body != null) {
wr.write(body);
}
wr.flush();
wr.close();
BufferedReader rd = new BufferedReader(new InputStreamReader(
con.getInputStream()));
String line;
String result = "";
while ((line = rd.readLine()) != null) {
result += line;
}
rd.close();
con.disconnect();
return result;
PS: It is about jse, not android.
You're never closing the input stream from the connection - that may mean that the connection isn't eligible for pooling, and on the next attempt it's waiting for up to a second to see if the previous request's connection will become eligible for pooling.
At the very least, it would be a good idea to close the stream. Use a try-with-resources block if you're using Java 7 - and ditto for the writer.
As an aside, I suggest you explicitly state the encoding you expect when reading - or use a higher-level library which detects that automatically based on headers.
I am trying to send a post request to a url using HttpURLConnection (for using cUrl in java).
The content of the request is xml and at the end point, the application processes the xml and stores a record to the database and then sends back a response in form of xml string. The app is hosted on apache-tomcat locally.
When I execute this code from the terminal, a row gets added to the db as expected. But an exception is thrown as follows while getting the InputStream from the connection
java.io.FileNotFoundException: http://localhost:8080/myapp/service/generate
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.kodeplay.helloworld.HttpCurl.main(HttpCurl.java:30)
Here is the code
public class HttpCurl {
public static void main(String [] args) {
HttpURLConnection con;
try {
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
con.setRequestMethod("POST");
con.setDoOutput(true);
con.setDoInput(true);
File xmlFile = new File("test.xml");
String xml = ReadWriteTextFile.getContents(xmlFile);
con.getOutputStream().write(xml.getBytes("UTF-8"));
InputStream response = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response));
for (String line ; (line = reader.readLine()) != null;) {
System.out.println(line);
}
reader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Its confusing because the exception is traced to the line InputStream response = con.getInputStream(); and there doesn't seem to be any file involved for a FileNotFoundException.
When I try to open a connection to an xml file directly, it doesn't throw this exception.
The service app uses spring framework and Jaxb2Marshaller to create the response xml.
The class ReadWriteTextFile is taken from here
Thanks.
Edit:
Well it saves the data in the DB and sends back a 404 response status code at the same time.
I also tried doing a curl using php and print out the CURLINFO_HTTP_CODE which turns out to be 200.
Any ideas on how do I go about debugging this ? Both service and client are on the local server.
Resolved:
I could solve the problem after referring to an answer on SO itself.
It seems HttpURLConnection always returns 404 response when connecting to a url with a non standard port.
Adding these lines solved it
con.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
con.setRequestProperty("Accept","*/*");
I don't know about your Spring/JAXB combination, but the average REST webservice won't return a response body on POST/PUT, just a response status. You'd like to determine it instead of the body.
Replace
InputStream response = con.getInputStream();
by
int status = con.getResponseCode();
All available status codes and their meaning are available in the HTTP spec, as linked before. The webservice itself should also come along with some documentation which overviews all status codes supported by the webservice and their special meaning, if any.
If the status starts with 4nn or 5nn, you'd like to use getErrorStream() instead to read the response body which may contain the error details.
InputStream error = con.getErrorStream();
FileNotFound is just an unfortunate exception used to indicate that the web server returned a 404.
To anyone with this problem in the future, the reason is because the status code was a 404 (or in my case was a 500). It appears the InpuStream function will throw an error when the status code is not 200.
In my case I control my own server and was returning a 500 status code to indicate an error occurred. Despite me also sending a body with a string message detailing the error, the inputstream threw an error regardless of the body being completely readable.
If you control your server I suppose this can be handled by sending yourself a 200 status code and then handling whatever the string error response was.
For anybody else stumbling over this, the same happened to me while trying to send a SOAP request header to a SOAP service. The issue was a wrong order in the code, I requested the input stream first before sending the XML body. In the code snipped below, the line InputStream in = conn.getInputStream(); came immediately after ByteArrayOutputStream out = new ByteArrayOutputStream(); which is the incorrect order of things.
ByteArrayOutputStream out = new ByteArrayOutputStream();
// send SOAP request as part of HTTP body
byte[] data = request.getHttpBody().getBytes("UTF-8");
conn.getOutputStream().write(data);
if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
Log.d(TAG, "http response code is " + conn.getResponseCode());
return null;
}
InputStream in = conn.getInputStream();
FileNotFound in this case was an unfortunate way to encode HTTP response code 400.
FileNotFound in this case means you got a 404 from your server - could it be that the server does not like "POST" requests?
FileNotFound in this case means you got a 404 from your server
You Have to Set the Request Content-Type Header Parameter
Set “content-type” request header to “application/json” to send the request content in JSON form.
This parameter has to be set to send the request body in JSON format.
Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
Full Script ->
public class SendDeviceDetails extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... params) {
String data = "";
String url = "";
HttpURLConnection con = null;
try {
// From the above URL object,
// we can invoke the openConnection method to get the HttpURLConnection object.
// We can't instantiate HttpURLConnection directly, as it's an abstract class:
con = (HttpURLConnection)new URL(url).openConnection();
//To send a POST request, we'll have to set the request method property to POST:
con.setRequestMethod("POST");
// Set the Request Content-Type Header Parameter
// Set “content-type” request header to “application/json” to send the request content in JSON form.
// This parameter has to be set to send the request body in JSON format.
//Failing to do so, the server returns HTTP status code “400-bad request”.
con.setRequestProperty("Content-Type", "application/json; utf-8");
//Set Response Format Type
//Set the “Accept” request header to “application/json” to read the response in the desired format:
con.setRequestProperty("Accept", "application/json");
//To send request content, let's enable the URLConnection object's doOutput property to true.
//Otherwise, we'll not be able to write content to the connection output stream:
con.setDoOutput(true);
//JSON String need to be constructed for the specific resource.
//We may construct complex JSON using any third-party JSON libraries such as jackson or org.json
String jsonInputString = params[0];
try(OutputStream os = con.getOutputStream()){
byte[] input = jsonInputString.getBytes("utf-8");
os.write(input, 0, input.length);
}
int code = con.getResponseCode();
System.out.println(code);
//Get the input stream to read the response content.
// Remember to use try-with-resources to close the response stream automatically.
try(BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream(), "utf-8"))){
StringBuilder response = new StringBuilder();
String responseLine = null;
while ((responseLine = br.readLine()) != null) {
response.append(responseLine.trim());
}
System.out.println(response.toString());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (con != null) {
con.disconnect();
}
}
return data;
}
#Override
protected void onPostExecute(String result) {
super.onPostExecute(result);
Log.e("TAG", result); // this is expecting a response code to be sent from your server upon receiving the POST data
}
and call it
new SendDeviceDetails().execute("");
you can find more details in this tutorial
https://www.baeldung.com/httpurlconnection-post
The solution:
just change localhost for the IP of your PC
if you want to know this: Windows+r > cmd > ipconfig
example: http://192.168.0.107/directory/service/program.php?action=sendSomething
just replace 192.168.0.107 for your own IP (don't try 127.0.0.1 because it's same as localhost)
Please change
con = (HttpURLConnection) new URL("http://localhost:8080/myapp/service/generate").openConnection();
To
con = (HttpURLConnection) new URL("http://YOUR_IP:8080/myapp/service/generate").openConnection();