I have a Java program that uses OAuth for communication with a server to retrieve XML data.
It makes use of the Signpost OAuth library to connect with the source, and uses a standard way of reading the InputStream to access the XML that is returned.
Of late, I've noticed the slow time it's taken to retrieve the information and tests have revealed that some requests can take anywhere from 2000 ms up to 10000 ms (if it matters, the source server is in Europe, I am in Australia).
I added a timestamp after the OAuth communication (request.connect()) and again after the reading of the InputStream and here's the output:
Request #1: Communication: [6351ms] Data process: [403ms] Total: [6754ms]
Request #2: Communication: [1ms] Data process: [3121ms] Total: [3122ms]
Request #3: Communication: [1ms] Data process: [1297ms] Total: [1298ms]
Request #4: Communication: [0ms] Data process: [539ms] Total: [539ms]
Request #4 is actually Request #2 being run a 2nd time. All requests are made in one run of the program (there's no stopping and starting).
My question: is the InputStream returned to the HttpURLConnection object as part of the connect() method, or is it streamed back as I read from it (as the name suggests) and part of the actual connection process?
Secondary question: With the timing above, is the slow time most likely to be a problem with the server or my method of reading the InputStream?
For reference, here is the code in question:
long startTime = System.currentTimeMillis();
URL url = new URL(urlString);
HttpURLConnection request = (HttpURLConnection) url.openConnection();
consumer.sign(request);
request.connect();
long connectionTime = System.currentTimeMillis();
InputStream is = request.getInputStream();
if (is != null) {
final BufferedReader bufferedreader = new BufferedReader(
new InputStreamReader(is, "UTF-8"));
final StringBuffer s2 = new StringBuffer();
String line;
line = bufferedreader.readLine();
if (line != null) {
s2.append(line);
while ((line = bufferedreader.readLine()) != null) {
s2.append('\n');
s2.append(line);
}
}
bufferedreader.close();
rv = s2.toString();
}
long finishTime = System.currentTimeMillis();
long timeTaken = finishTime - startTime;
long totalConnectionTime = connectionTime - startTime;
long processDataTime = finishTime - connectionTime;
String info = "Communication: [" + totalConnectionTime +
"ms] Data process: [" + processDataTime +
"ms] Total: [" + timeTaken + "ms]";
Thanks in advance.
Based on the information provided, here are few observations and suggestions.
To answer your question, the data is streamed back as you read from it. And then you have buffered layers on the top of it. The whole data is not returned and its streamed. I hope I read your question correctly.
The secondary question: The time taken could in both the places viz. the server and in your code as well. Since you are not doing any other processing in your code other than reading the data (except Bufferedreader.close() and s2.toString();) the delay appears to be in server BUT just for being sure, if possible, hit the URL using any browser and see the time taken to fetch the request. (from the code I see that you are just fetching the data from URL and hence should be easy to access the same using browser)
You also have mentioned that you are retrieving XML from the server. I would recommend to use some standard xml parsers (SAX,xstrem etc) which are optimized (hence better performance) for reading xml data from an InputStream.
openConnection() does create the TCP connection, but unless you use a non-default streaming mode no data is sent until you either get the input stream or the reader or the response code. So sending the request is seen as part of getInputStream() in your case.
Related
I am trying to create a proxy server.
I want to read the websites byte by byte so that I can display images and all other stuff. I tried readLine but I can't display images. Do you have any suggestions how I can change my code and send all data with DataOutputStream object to browser ?
try{
Socket s = new Socket(InetAddress.getByName(req.hostname), 80);
String file = parcala(req.url);
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
BufferedReader dis = new BufferedReader(new InputStreamReader(s.getInputStream()));
PrintWriter socketOut = new PrintWriter(s.getOutputStream());
socketOut.print("GET "+ req.url + "\n\n");
//socketOut.print("Host: "+req.hostname);
socketOut.flush();
String line;
while ((line = dis.readLine()) != null){
System.out.println(line);
}
}
catch (Exception e){}
}
Edited Part
This is what I should have to do. I can block banned web sites but can't allow other web sites in my program.
In the filter program, you will open a TCP socket at the specified port and wait for connections. If a
request comes (i.e. the client types a URL to access a web site), the application will process it to
decide whether access is allowed or not and then, using the same socket, it will send the reply back
to the client. After the client opened her connection to WebPolice (and her request has been checked
and is allowed), the real web page needs to be shown to the client. Therefore, since the user already gave her request, now it is WebPolice’s turn to forward the request so that the user can get the web page. Thus, WebPolice acts as a client and requests the web page. This means you need to open a connection to the web server (without closing the connection to the user), forward the request over this connection, get the reply and forward it back to the client. You will use threads to handle multiple connections (at the same time and/or at different times).
I don't know what exactly you're trying to do, but crafting an HTTP request and reading its response incorporates somewhat more than you have done here. Readline won't work on binary data anyway.
You can take a look at the URLConnection class (stolen here):
URL oracle = new URL("http://www.oracle.com/");
URLConnection yc = oracle.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
Then you can read textual or binary data from the in object.
Read line will treat the line read as a String, so unless you want to mess around with conversions over to bytes, I wouldn't recommend that.
I would just read bytes until you can't read anymore, then write them out to a file, this should allow you to grab the images, keeping file headers intact which can be important when dealing with files other than text.
Hope this helps.
Instead of using BufferedReader you can try to use InputStream.
It has several methods for reading bytes.
http://docs.oracle.com/javase/6/docs/api/java/io/InputStream.html
I hope someone can help me. I'm a bit of a noob to Java. But I have a question regarding calling a web service from Java. The question is actually simple but one way works the other does not?
If I call a web service from Java like this, it works:
try {
String parameters = "<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">"+
"<soap:Body>"+
" <HelloWorld xmlns=\"http://np-challenger\" />"+
"</soap:Body>"+
"</soap:Envelope>";
//out.println(parameters);
java.net.URL url = new java.net.URL("http://localhost:50217/WebSite3/Service.asmx");
java.net.HttpURLConnection connjava = (java.net.HttpURLConnection)url.openConnection();
connjava.setRequestMethod("GET");
connjava.setRequestProperty("Content-Length", "" + Integer.toString(parameters.getBytes().length));
connjava.setRequestProperty("Content-Language", "en-US");
connjava.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
connjava.setRequestProperty("SOAPAction", "http://np-challenger/HelloWorld");
connjava.setDoInput(true);
connjava.setDoOutput(true);
connjava.setUseCaches(false);
connjava.setAllowUserInteraction(true);
java.io.DataOutputStream printout = new java.io.DataOutputStream (connjava.getOutputStream());
printout.writeBytes(parameters);
printout.flush();
printout.close();
java.io.BufferedReader in = new java.io.BufferedReader(new java.io.InputStreamReader(connjava.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
/*pagecontent += stuff;*/
}
in.close();
} catch (Exception e) {
System.out.println("Error: "+ e);
}
However, if I try to do it like this, I keep getting a bad request. I'm just about ready to pull my hair out.
try {
String xmlData = "<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">"+
"<soap:Body>"+
" <HelloWorld xmlns=\"http://np-challenger\" />"+
"</soap:Body>"+
"</soap:Envelope>";
//create socket
String hostname = "localhost";
int port = 50217;
InetAddress addr = InetAddress.getByName(hostname);
Socket sock = new Socket(addr,port);
FileWriter fstream = new FileWriter("out.txt");
// Send header
String path = "/WebSite3/Service.asmx";
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(sock.getOutputStream(), "UTF8"));
bw.write("POST " + path + " HTTP/1.1\r\n");
bw.write("Host: localhost\r\n");
bw.write("Content-Type: text/xml; charset=\"utf-8\"\r\n");
bw.write("Content-Length: " + xmlData.length() + "\r\n");
bw.write("SOAPAction: \"http://np-challenger/HelloWorld\"");
bw.write("\r\n");
// Send POST data string
bw.write(xmlData);
bw.flush();
// Process the response from the Web Services
BufferedReader br = new BufferedReader(new InputStreamReader(sock.getInputStream()));
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
bw.close();
br.close();
} catch(Exception e) {
System.err.println(e.getMessage());
e.printStackTrace(System.err);
}
I'm a bit suspicious whether the way you calculate the content length is correct, but more importantly:
Use a testing tool.
You can use a testing tool to compare between good and bad requests. One of such tools is soapUI, it's very convenient in showing you the exact contents of the requests and responses.
Create a new project in soapUI, based on the WSDL of your web service. Make sure to mark the checkboxes "Create sample requests for all operations" and "Create a Web Service Simulation of the imported WSDL". This way, soapUI will be able to act both as a client for your actual .NET web service, and as a server to which your Java client will connect.
Make sure that when soapUI connects acts as a client and connects to your web service, the request is processed correctly. Then run it as a server, send a request from Java, and compare it to the request that was processed successfully.
I chose to emphasize the role of a testing tool instead of addressing the specific problems in your code, because I believe that the ability to analyze the contents of your requests and responses will prove to be valuable time after time.
Use a WS framework.
Working with web services on such a low level requires a lot of unnecessary work from you. There are several frameworks and tools in Java that allow you to work on a higher abstraction level, eliminating the need to handle sockets and HTTP headers yourself. Take a look at the JAX-WS standard. This tutorial shows how to create a client for an existing web service. You'll notice that it's much simpler than your code sample.
Other popular WS frameworks in Java are Apache Axis2 and Apache CXF.
It's actually difference in data that is going to server. Monitor the data that you are actually posting using TCP Monitor. and compare the data i.e. mime header, request xml etc.
You will find the mistake. As far as I can see, first method is using GET method while second method is using POST method. I do not say that this is error just monitor actual data that is going to server and you will automatically get your problem resolved.
I'm currently creating a little webserver (for testing purposes) and I have a problem reading the HTTP request-header (coming from the browser, chromium in my case).
First, I simply tried something like this:
BufferedReader in = new BufferedReader(
new InputStreamReader(client_socket.getInputStream(), "UTF-8")
);
StringBuilder builder = new StringBuilder();
while (in.ready()){
builder.append(in.readLine());
}
return builder.toString();
This worked fine for the first request. However, after the first request was done, the ready()-method only returned false (i closed the client_socket as well as all readers/writers).
After a little searching I stumbled across this older question: Read/convert an InputStream to a String
I tried the first four solutions (two with Apache Commons, the one with the Scanner and the one with the do-while loop). All of them blocked for ever and the browser gave me a "Website not reachable"-error.
I'd like to do this on my own (without using any library's or embedded servers), that's why I even try.
I'm a little lost right now, how would you go about this?
You are reading from the socket until there is no more data to read. That is wrong. You need to keep reading until you encounter a 0-length line, then process the received headers to determine if there is more data to read (look for Content-Length: ... and Transfer-Encoding: chunked headers), eg:
StringBuilder builder = new StringBuilder();
String line;
do
{
line = in.readLine();
if (line == "") break;
builder.append(line);
}
while (true);
// use builder as needed...
// read message body data if headers say there
// is a body to read, per RFC 2616 Section 4.4...
Read RFC 2616 Section 4 for more details. Not only does this facilitate proper request reading, but doing so also allows you to support HTTP keep-alives correctly (for sending multiple requests on a single connection).
The solution suggested above by Remy Lebeau is wrong, as it was shown in a test of mine. This alternative is fail-safe:
StringBuilder builder = new StringBuilder();
String line;
do
{
line = in.readLine();
if (line.equals("")) break;
builder.append(line);
}
while (true);
Refer to: How do I compare strings in Java?
I am currently trying to get HttpComponents to send HttpRequests and retrieve the Response.
On most URLs this works without a problem, but when I try to get the URL of a phpBB Forum namely http://www.forum.animenokami.com the client takes more time and the responseEntity contains passages more than once resulting in a broken html file.
For example the meta tags are contained six times. Since many other URLs work I can't figure out what I am doing wrong.
The Page is working correctly in known Browsers, so it is not a Problem on their side.
Here is the code I use to send and receive.
URI uri1 = new URI("http://www.forum.animenokami.com");
HttpGet get = new HttpGet(uri1);
get.setHeader(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20100101 Firefox/6.0"));
HttpClient httpClient = new DefaultHttpClient();
HttpResponse response = httpClient.execute(get);
HttpEntity ent = response.getEntity();
InputStream is = ent.getContent();
BufferedInputStream bis = new BufferedInputStream(is);
byte[] tmp = new byte[2048];
int l;
String ret = "";
while ((l = bis.read(tmp)) != -1){
ret += new String(tmp);
}
I hope you can help me.
If you need anymore Information I will try to provide it as soon as possible.
This code is completely broken:
String ret = "";
while ((l = bis.read(tmp)) != -1){
ret += new String(tmp);
}
Three things:
This is converting the whole buffer into a string on each iteration, regardless of how much data has been read. (I suspect this is what's actually going wrong in your case.)
It's using the default platform encoding, which is almost never a good idea.
It's using string concatenation in a loop, which leads to poor performance.
Fortunately you can avoid all of this very easily using EntityUtils:
String text = EntityUtils.toString(ent);
That will use the appropriate character encoding specified in the response, if any, or ISO-8859-1 otherwise. (There's another overload which allows you to specify which character encoding to use if it's not specified.)
It's worth understanding what's wrong with your original code though rather than just replacing it with the better code, so that you don't make the same mistakes in other situations.
It works fine but what I don't understand is why I see the same text multiple times only on this URL.
It will be because your client is seeing more incomplete buffers when it reads the socket. Than could be:
because there is a network bandwidth bottleneck on the route from the remote site to your client,
because the remote site is doing some unnecessary flushes, or
some other reason.
The point is that your client must pay close attention to the number of bytes read into the buffer by the read call, otherwise it will end up inserting junk. Network streams in particular are prone not filling the buffer.
I am writing a program in JAVA to POST a large number of XML Documents to a specific web address, in addition to a great deal of other data handling that is slightly unrelated to this question. The only trouble is, I'm expect to handle approximately 90,000 records. When POSTing an XML document, each record takes approximately 10 seconds, 9 of which is taken by receiving the response from the sever after POST.
My question is: Is there a way to POST data to a webserver, then ignore the server's response to save time?
Here is a snip of code that's giving me trouble, it takes approximate 9 seconds according to the system timer to go from "writer.close" to "con.getResponseCode()"
URL url = new URL(TargetURL);
con = (HttpsURLConnection) url.openConnection();
//Login with given credentials
String login = (Username)+":"+(Password);
String encoding = new sun.misc.BASE64Encoder().encode(login.getBytes());
con.setRequestProperty ("Authorization", "Basic " + encoding);
// specify that we will send output and accept input
con.setRequestMethod("POST");
con.setDoInput(true);
con.setDoOutput(true);
con.setConnectTimeout(20000) ; // long timeout, but not infinite
con.setReadTimeout(20000);
con.setUseCaches (false);
con.setDefaultUseCaches (false);
// tell the web server what we are sending
con.setRequestProperty ( "Content-Type", "text/xml" );
OutputStreamWriter writer = new OutputStreamWriter( con.getOutputStream() );
writer.write(data);
writer.flush();
writer.close();
//****This is our problem.*****//
int result = con.getResponseCode();
System.err.println( "\nResponse from server after POST:\n" + result );
I see your problem.
Using the strategy to read only the header would not work for you because the problem is not due to voluminous amount of data the server is sending as a response. The problem is that the server takes a long to time to process the data your client had sent and therefore takes a long time to even send a short ack response.
What you are asking for is Asynchronous response. The answer is AJAX and my preference of choice is GWT.
GWT presents three ways to perform async communication with the server.
GWT RPC
RequestBuilder
javascript include
MVP ClientFactory/EventBus
Please read my description at
http://h2g2java.blessedgeek.com/2009/08/gwt-rpc.html
http://h2g2java.blessedgeek.com/2011/06/gwt-requestbuilder-vs-rpc-vs-script.html
But then, you might prefer to use JQuery, with which I have scant and scarce familiarity.
I'd rather use Apache HttpComponents. It lets you not read the response body, and only the headers which you obviously need.
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html#d4e143
That part of the docs has an example of only reading a few bytes of the response.