I am working on an application which executes a lot of HTTP-Post Request on a specific URL. The data gets included into the request via a JSON File. Performing a single POST request takes about 200ms until I get the response code back. In order to parallelize these Request I started using different Worker Threads, opening a HTTPURLConnection each and posting their requests over this connection.
Unfortunately I do not see much improvement, so I wanted to clarify the underlying situation of the TCP sockets and get some thoughts from you guys.
Each of my workers executes the request like the following example could.
public void submitRequest() {
URL url = new URL("https://example.com");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setRequestProperty("Content-Type", "application/json");
conn.setRequestProperty("Accept", "application/json");
conn.setDoOutput(true);
conn.setRequestMethod("POST");
DataOutputStream output = new DataOutputStream(conn.getOutputStream());
// Construct the POST data.
String content = generatedJSONString();
// Send the request data
output.writeBytes(content);
output.flush();
output.close();
// Check response code here
int response = conn.getResponseCode();
}
Because each worker uses its own HttpsURLConnection, I assume that the necessary TLS handshake is only done once per worker and the requests can then be executed without the need of an additional handshake? I want each worker to use its own TCP connection to the server. I tried to check with Wireshark, but it does not show me the TLS handshake, I don't really know why.
Even with 50 workers in parallel, my code does not execute fast enough to post the relevant data to the url. Therefore I ask myself the question if its because of the TCP connections not getting saturated by the code, because the JSON data (usually 10 key-value pairs) is to little data compared to the TCP/HTTP overhead. If not I might ask the service provider to think about a better way to deliver data in a batch way, to submit more JSON data per individual POST request. I also changed to code to use the Apache HTTPClient, but I got similar performance, therefore I think there is a mistake in the way I think the TLS connections get established and reused.
Related
In order to send a string data to the server once, I do as below:
Make “HttpURLConnection” to my URL address and open it
Set the required headers
for the connection I Set setDoOutput to True
new a DataOutputStream from my connection and finally write my string data to it.
HttpURLConnection myConn = (HttpURLConnection);
myUrl.openConnection();
myConn.setRequestProperty("Accept", "application/json, text/plain, */*");
myConn.setDoOutput(true);
DataOutputStream my_output = new DataOutputStream(myConn.getOutputStream());
my_output.write(myData.getBytes("UTF-8"));
But what to do if I want to send exactly the same data with same URl and headers multiple times?
Can I write to it multiple times?(I mean that is it possible to use the last line of code multiple times?) Or should I repeat the above steps and try it with a new connection?
And if yes should I wait for some second or millisecond before sending the next one?
I also searched for some other alternatives such as “HttpClient” Http API and making synchronous Http request which as far as I got can help me setting the headers only once.
At the end, I appreciate your help and support and any other alternatives would be welcome.
Thanks a million.
I understand that the question has be answered in the comments, but I am leaving this here so that future viewers can see it.
An HTTP request contains 3 main parts:
Request Line: Method, Path, Protocol
Headers: Key-Pairs
Body: Data
Running my_output.write() will just add bytes to the body until my_output.flush() has been executed. Flushing the stream will write the data to the server.
Because HTTP requests are usually closed by the server once all data has been sent/received, whether or not you create a new connection or just add on to the body depends on your intentions. Typically, clients will create a new connection for each request because each response should be handled individually, rather than sending a repetitive body. This will vary though because some servers choose to hold a connection (such as WebSockets).
If you are open to external libraries, you may find this chart insightful:
AsyncHttpClient would be a good fit for your intentions.
Alternatively, you can use cURL by running a terminal command with Runtime.getRuntime().exec(). More information about using cURL with POST Requests can be found here. While cURL is efficient, you have to depend on the fact that your OS supports the command (though usually all devices that can run Java have this command).
Consider the following code.
try {
httpURLConnection = (HttpURLConnection) new URL(strings[0]).openConnection();
httpURLConnection.setConnectTimeout(Config.HTTP_CONNECTION_TIMEOUT);
httpURLConnection.setReadTimeout(Config.HTTP_CONNECTION_TIMEOUT);
httpURLConnection.connect();
responseCode = httpURLConnection.getResponseCode();
httpURLConnection.getHeaderFields();
}
finally {
httpURLConnection.disconnect();
}
The issue is even when I don't use the InputStream to read the response, in my Internet/Wifi connection logs I can see the response-body. What I want is simply to check a field in the header and based upon that field I will continue reading the InputStream.
My questions are these:
Is it correct behavior for the connected stream to automatically download all/partial file even before a BufferedInputStream is created and read from?
If yes, then is it possible to stop the file download until an InputStream is used to read the response?
If not then is there something I am doing wrong or missing?
The response includes both the header and the body, the server does not stop for the client to acknowledge the headers before sending the body.
At the time the client is able to read the response code from the headers, a part of the body has already been sent, the size of which depends on the network latency, buffering, ....
The current implementation of HttpURLConnection.getResponseCode() even use getInputStream() to ensure that the connection is in the correct state.
The client can choose to ignore the body, but it's usually not recommended, because it may prevent a persistent connection to be reused.
I am not sure about Android but since Java 6, a background thread is automatically used to read the remaining data.
If If-Modified-Since is not an option, why not use a HEAD request ? :
The HTTP HEAD method requests the headers that are returned if the
specified resource would be requested with an HTTP GET method. Such a
request can be done before deciding to download a large resource to
save bandwidth, for example.
I need to send data to another system in a Java aplication via HTTP POST method. Using the Apache HttpClient library is not an option.
I create a URL, httpconection without problems. But when sending special character like Spanish Ñ, the system complains it is receiving
Ñ instead of Ñ.
I've read many post, but I don't understand some things:
When doing a POST connection, and writing to the connection object, is it mandatory to do the URLEncode.encode(data,encoding) to the data being sent?
When sending the data, in some examples I have seen they use the
conn.writeBytes(strData), and in other I have seen conn.write(strData.getBytes(encoding)). Which one is it better? Is it related of using the encode?
Update:
The current code:
URL url = new URL(URLstr);
conn1 = (HttpsURLConnection) url.openConnection();
conn1.setRequestMethod("POST");
conn1.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(conn1.getOutputStream());
wr.writeBytes(strToSend);//data sent
wr.flush();
wr.close();
(later I get the response)
strToSend has been previously URLENCODE.encode(,"UTF-8")
I still don't know if I must use urlencode in my code and/or setRequestProperty("Contentype","application/x-www-formurlencode");
Or if I must use .write(strToSend.getByte(??)
Any ideas are welcome. I am testing also the real server (I dont know very much about it)
I'm trying to implement a simple URL availability checker which basically checks if the link is available (No HTTP 403, 404 etc. returned).
I have more than 20,000 links(to different servers/websites) in my database for testing purposes but it doesn't seems to work when I try to create more than 10 Threads.
Here is the code I'm using for opening the connection and reading response code within each WorkerThread.
URL url = new URL(dto.getUrl());
httpUrlConnection = (HttpURLConnection) url.openConnection();
httpUrlConnection.setUseCaches(false);
// httpUrlConnection.setConnectTimeout(6000);
httpUrlConnection.setDoInput(true);
httpUrlConnection.setDoOutput(false);
httpUrlConnection.setRequestMethod("GET");
httpUrlConnection.setRequestProperty("Host", dto.getUrl().replace("http://", ""));
// httpUrlConnection.setRequestProperty("Connection",
// "Keep-Alive");
httpUrlConnection.setRequestProperty("User-Agent", USER_AGENT);
httpUrlConnection.setRequestProperty("Cache-Control", "no-cache");
httpUrlConnection.connect();
int code = httpUrlConnection.getResponseCode();
Few issues I have noticed when having multiple threads opening the connections:
1) Only first 100-200 connections seems to open with no problem, after that, I start getting "Read timeout", "Connection timeout", "Connection reset" etc. Although, if you try to run the code again the links which have thrown above exceptions will return proper response code (if they get processed in first 100).
2) The response code sometimes is not valid (especially if the link was processed after first 100 links). I have noticed that sometimes 404 is returned when in fact it should return 200 (I checked it by putting link in first 100).
I did try using Apache's Http client but it also fails to process links correctly with many threads.
So does anyone know a solution to this problem ? What is the maximum amount of connections you could open using HttpURLConnection using multiple threads ? Is there any other way to open many HTTP connections and check response codes ?
Thank you all in advance !
I am writing a program in JAVA to POST a large number of XML Documents to a specific web address, in addition to a great deal of other data handling that is slightly unrelated to this question. The only trouble is, I'm expect to handle approximately 90,000 records. When POSTing an XML document, each record takes approximately 10 seconds, 9 of which is taken by receiving the response from the sever after POST.
My question is: Is there a way to POST data to a webserver, then ignore the server's response to save time?
Here is a snip of code that's giving me trouble, it takes approximate 9 seconds according to the system timer to go from "writer.close" to "con.getResponseCode()"
URL url = new URL(TargetURL);
con = (HttpsURLConnection) url.openConnection();
//Login with given credentials
String login = (Username)+":"+(Password);
String encoding = new sun.misc.BASE64Encoder().encode(login.getBytes());
con.setRequestProperty ("Authorization", "Basic " + encoding);
// specify that we will send output and accept input
con.setRequestMethod("POST");
con.setDoInput(true);
con.setDoOutput(true);
con.setConnectTimeout(20000) ; // long timeout, but not infinite
con.setReadTimeout(20000);
con.setUseCaches (false);
con.setDefaultUseCaches (false);
// tell the web server what we are sending
con.setRequestProperty ( "Content-Type", "text/xml" );
OutputStreamWriter writer = new OutputStreamWriter( con.getOutputStream() );
writer.write(data);
writer.flush();
writer.close();
//****This is our problem.*****//
int result = con.getResponseCode();
System.err.println( "\nResponse from server after POST:\n" + result );
I see your problem.
Using the strategy to read only the header would not work for you because the problem is not due to voluminous amount of data the server is sending as a response. The problem is that the server takes a long to time to process the data your client had sent and therefore takes a long time to even send a short ack response.
What you are asking for is Asynchronous response. The answer is AJAX and my preference of choice is GWT.
GWT presents three ways to perform async communication with the server.
GWT RPC
RequestBuilder
javascript include
MVP ClientFactory/EventBus
Please read my description at
http://h2g2java.blessedgeek.com/2009/08/gwt-rpc.html
http://h2g2java.blessedgeek.com/2011/06/gwt-requestbuilder-vs-rpc-vs-script.html
But then, you might prefer to use JQuery, with which I have scant and scarce familiarity.
I'd rather use Apache HttpComponents. It lets you not read the response body, and only the headers which you obviously need.
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html#d4e143
That part of the docs has an example of only reading a few bytes of the response.