Develop Reference

Java -- HTML client returns response before the complete page load

I have to read contents of a certain field from a webpage. I have been told that I need to get the whole page and then extract the text from the html content.
I am using the following program to get the required page html content.
Now the issue is that this webpage takes a few seconds to load the actual text value that I want to read even though the rest of the static page components are loaded earlier. And my program kind of returns the html content after the static components are loaded but before my value is loaded. So, the final HTML that I get has the page loading process pic instead of the actual value.
Could anyone please guide me on the required changes in this program that would help it wait until the page is completely loaded?
HttpPost post = new HttpPost("https://..../login");
//prepare get method
HttpGet httpget = new HttpGet("https://...../value#/123");
// add parameters to the post method
List<NameValuePair> parameters = new ArrayList<NameValuePair>();
parameters.add(new BasicNameValuePair("username", "<name>"));
parameters.add(new BasicNameValuePair("password", "<password>"));
try {
UrlEncodedFormEntity sendEntity = new UrlEncodedFormEntity(parameters, HTTP.DEF_CONTENT_CHARSET);
post.setEntity(sendEntity);
// create the client and execute the post method
HttpClient client = HttpClientBuilder.create().build();
HttpResponse postResponse = client.execute(post);
System.out.println("Statusline: " + postResponse.getStatusLine());
//Output the Response from the POST
System.out.println(getStringFromInputStream(postResponse.getEntity().getContent()));
//releasing POST
EntityUtils.consume(postResponse.getEntity());
//Execute get
HttpContext context = new BasicHttpContext();
HttpResponse getResponse = client.execute(httpget);//, context);
System.out.println("Statusline: " + getResponse.getStatusLine());
if (getResponse.getStatusLine().getStatusCode() != HttpStatus.SC_OK)
throw new IOException(getResponse.getStatusLine().toString());
System.out.print(getStringFromInputStream(getResponse.getEntity().getContent()));

you can also use Jsoup library
visit http://jsoup.org

Get response Uri parameters from apache HttpResponse

I'm sending an http get/head request using Apache HttpClient 4.x. I'm sending a request with a url like "http://example.com/getAccessToken". I'm expecting the response to be a redirect url with parameters in the returned url like "http://redirecturl.com/?code=accessTokenStuff". I want to be able to parse the response redirect url parameters, i.e. I want to get "accessTokenStuff". How can I do that?
HttpClient client = new DefaultHttpClient();
HttpHead request = new HttpHead(authUrl);
HttpResponse response = client.execute(request);
System.out.println(response.getStatusLine().getStatusCode());//returns 200
request.releaseConnection();
In a nutshell: what I want is executing an original url and then getting the result which is another url that has a parameter called "code". Then I want to get the value of that parameter.
EDIT:
I also tried this but it returns the same original URL
DefaultHttpClient client = new DefaultHttpClient();
HttpParams params = client.getParams();
HttpClientParams.setRedirecting(params, false);
HttpGet request = new HttpGet(authUrl);
HttpResponse response = client.execute(request);
String location = response.getLastHeader("Location").getValue();//returns same original url
System.out.println(location);
request.releaseConnection();
Setting HttpClientParams.setRedirecting(params, true); return null

Http response do not take a form of a "redirect url". Redirect and response are both different (but related) concepts. Redirect usually means "get your response from this address instead of original one".
Having said this, you can prevent HttpClient from following a redirect, see this answer: How to prevent apache http client from following a redirect
When your HttpClient is not following the redirect, you can inspect the 'Location:' header of its response, eg:
HeaderIterator iterator = httpResponse.headerIterator("Location");
while(iterator.hasNext()) {
Header header = iterator.nextHeader();
String redirectUrl = header.getValue();
}

Getting Exception while calling soap webservice

When I am using SoapUI to call this web service I am getting the correct response but when I implement this in android, I am getting the below exception,
system.web.services.protocols.soapheaderexception (some information is missing).
This is what I tried,
HttpPost httppost = new HttpPost("http://www.ocrwebservice.com/services/OCRWebService.asmx");
StringEntity se = new StringEntity(SOAPRequestXML,HTTP.UTF_8);
se.setContentType("text/xml");
httppost.setHeader("Content-Type", "text/xml;charset=UTF-8");
httppost.setEntity(se);
HttpClient httpclient = new DefaultHttpClient();
BasicHttpResponse httpResponse =
(BasicHttpResponse) httpclient.execute(httppost);
HttpEntity resEntity = httpResponse.getEntity();
I tried other combinations also like,
httppost.setHeader("Accept-Charset","utf-8")
and
httppost.setHeader("Content-Type","application/soap+xml;charset=UTF-8")
But nothing worked.
The error says,
System.Web.Services.Protocols.SoapHeaderException: WSE012: The input was not a valid SOAP message because the following information is missing: action.

System.Web.Services.Protocols.SoapHeaderException: WSE012: The input
was not a valid SOAP message because the following information is
missing: action.
=> As per the above exception, I can say you forgot to set Action.
Try:
String SOAP_ACTION = "http://stockservice.contoso.com/wse/samples/2005/10/OCRWebServiceAvailablePages";
httppost.setHeader("SOAPAction", SOAP_ACTION);

HttpPost -> Redirect -> Location or body of response needed

Here is Java code that POSTs data to a website and than gets redirected as a response (status 302). It works perfectly on my PC (Eclipse, Java, Ubuntu), it does exactly what I want it to do.
I tried quite everything to post the code functionality but I just am not able to.
Java code:
// Preparing the CLIENT and POST Method
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://na.leagueoflegends.com/ladders/solo-5x5");
try {
// Add your POST METHOD attributes
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
nameValuePairs.add(new BasicNameValuePair("op", "Search"));
nameValuePairs.add(new BasicNameValuePair("player", "Jaiybe"));
nameValuePairs.add(new BasicNameValuePair("ladder_id", "3"));
nameValuePairs.add(new BasicNameValuePair("form_build_id",
"form-526370b788622996caa3669e7b975ccf"));
nameValuePairs.add(new BasicNameValuePair("form_id",
"ladders_filter_form"));
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
// Execute HTTP Post Request
HttpResponse response = httpclient.execute(httppost);
// RESPONE THAT WORKS WITH JAVA
System.out.println("Location:");
String LocationHeader = response.getFirstHeader("location").getValue();
System.out.println(LocationHeader);
System.out.println();
// To get the BODY I would have to parse that again - since its not REDIRECTING automatically
HttpClient httpclient2 = new DefaultHttpClient();
HttpPost httppost2 = new HttpPost(LocationHeader);
response = httpclient2.execute(httppost2);
System.out.println("And EVEN the response body:");
System.out.println(EntityUtils.toString(response.getEntity()));
Code does:
Posts
Gets Redirected - gets header of Location
Parses the Location
And I need android to do the same. Either "Location" or body of repsonse, is ok, I dont need both.
The post: http://www.anddev.org/networking-database-problems-f29/httppost-clientprotocolexception-t56118.html

I have found the problem!
httpclient.getParams().setParameter("http.protocol.version",
HttpVersion.HTTP_1_0);
Just changing this one line - version 1_0 works and 1_1 does not. Don't ask me why :)
Thank you all!

Please try the following code. The location in the header is missing, because the page has already redirected. So we can disable redirection to get the location tag.
httpclient.getParams().setParameter(ClientPNames.HANDLE_REDIRECTS, false);

Try calling this after you create your http client so that it follows your redirect
httpclient.getParams().setParameter("http.protocol.allow-circular-redirects", true);

Obtaining URL from http response when no location header is sent

When communicating with http to http://forecast.weather.gov/zipcity.php I need to obtain the URL that is generated from a request.
I have printed out the headers and their values from the http response message but there is no location header. How can I obtain this URL? (I'm using HttpClient)

It should be similar to:
HttpClient client = new DefaultHttpClient();
HttpParams params = client.getParams();
HttpClientParams.setRedirecting(params, false);
HttpGet method = new HttpGet("http://forecast.weather.gov/zipcity.php?inputstring=90210");
HttpResponse resp = client.execute(method);
String location = resp.getLastHeader("Location").getValue();
EDIT: I had to make a couple minor tweaks, but I tested and the above works.