I'm trying to retrive some data from a web site.
I wrote a java class which seems to work pretty fine with many sites but it doesn't work with this particular site, which use extensive javascript in the input fomr.
As you can see from the code I specified the input fields taking the name from the HTML source, but maybe this website doesn't accept POST request of this kind?
How can I simulate an user-interaction to retrieve the generated HTML?
package com.transport.urlRetriver;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
public class UrlRetriver {
String stationPoller (String url, ArrayList<NameValuePair> params) {
HttpPost postRequest;
HttpResponse response;
HttpEntity entity;
String result = null;
DefaultHttpClient httpClient = new DefaultHttpClient();
try {
postRequest = new HttpPost(url);
postRequest.setEntity((HttpEntity) new UrlEncodedFormEntity(params));
response = httpClient.execute(postRequest);
entity = response.getEntity();
if(entity != null){
InputStream inputStream = entity.getContent();
result = convertStreamToString(inputStream);
}
} catch (Exception e) {
result = "We had a problem";
} finally {
httpClient.getConnectionManager().shutdown();
}
return result;
}
void ATMtravelPoller () {
ArrayList<NameValuePair> params = new ArrayList<NameValuePair>(2);
String url = "http://www.atm-mi.it/it/Pagine/default.aspx";
params.add(new BasicNameValuePair("ctl00$SPWebPartManager1$g_afa5adbb_5b60_4e50_8da2_212a1d36e49c$txt_address_s", "Viale romagna 1"));
params.add(new BasicNameValuePair("ctl00$SPWebPartManager1$g_afa5adbb_5b60_4e50_8da2_212a1d36e49c$txt_address_e", "Viale Toscana 20"));
params.add(new BasicNameValuePair("sf_method", "POST"));
String result = stationPoller(url, params);
saveToFile(result, "/home/rachele/Documents/atm/out4.html");
}
static void saveToFile(String toFile, String pos){
try{
// Create file
FileWriter fstream = new FileWriter(pos);
BufferedWriter out = new BufferedWriter(fstream);
out.write(toFile);
//Close the output stream
out.close();
}catch (Exception e){//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
}
private static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder stringBuilder = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
stringBuilder.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return stringBuilder.toString();
}
}
At my point of view, there could be javascript generated field with dynamic value for preventing automated code to crawl the site. Send concrete site you want to download.
Related
I got a problem while I am using httpclients 4.5.4.
When I close everything that is related to my connection, the PoolingHttpClientConnectionManager objects can not be Collected by java GC.
So as the application runs for about 1 month, I got an OOM exception.
My code is as same as example in https://hc.apache.org/httpcomponents-client-4.5.x/httpclient/examples/org/apache/http/examples/client/ClientConnectionRelease.java.
And InternalHttpClient objects can not be collected by java GC, either.
There is my code below, is there any objects holding the references of the two classes?
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Map;
import org.apache.http.HttpStatus;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class HttpUtil {
private static Logger logger = LoggerFactory.getLogger(HttpUtil.class);
/**
* Http post请求
*/
public static String doHttpPost(String postUrl, Map<String, String> headers,
Map<String,String> params,String filePath) {
try {
CloseableHttpClient httpClient = HttpClients.createDefault();
try {
HttpPost post = getHttpPost(postUrl, headers, params, filePath);
CloseableHttpResponse response = httpClient.execute(post);
try {
InputStream inputStream = null;
InputStreamReader inputStreamReader = null;
BufferedReader bufferedReader = null;
if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
try {
inputStream = response.getEntity().getContent();
inputStreamReader = new InputStreamReader(inputStream);
bufferedReader = new BufferedReader(inputStreamReader);
return bufferedReader.readLine();
}catch (Exception e) {
logger.warn("do http post fail : ", e);
}finally {
if (bufferedReader != null) {
logger.info("release bufferedReader: " + filePath);
bufferedReader.close();
}
if (inputStreamReader != null) {
logger.info("release inputStreamReader: " +filePath);
inputStreamReader.close();
}
if (inputStream != null) {
logger.info("release inputStream: " +filePath);
inputStream.close();
}
}
}
} catch (Exception e) {
logger.warn("do http post fail: ", e);
} finally {
if (response != null) {
logger.info("release response: " + filePath);
response.close();
}
if (post != null) {
logger.info("release HttpPost: " + filePath);
post.releaseConnection();
}
}
} catch (Exception e) {
logger.warn("do http post fail: ", e);
} finally {
if (httpClient != null) {
logger.info("release httpClient: " + filePath);
httpClient.close();
logger.info("release connectionManager: " + filePath);
httpClient.getConnectionManager().shutdown();
}
}
} catch (Exception e) {
logger.warn("do http post fail: ", e);
}
return "";
}
private static HttpPost getHttpPost(String postUrl, Map<String, String> headers,
Map<String, String> params, String filePath) {
HttpPost post = new HttpPost(postUrl);
String[] headerKeys = headers.keySet().toArray(new String[headers.keySet().size()]);
for (String key : headerKeys) {
post.setHeader(key, headers.get(key));
}
FileBody fileBody = new FileBody(new File(filePath));
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
multipartEntityBuilder.addPart("express_image", fileBody);
String[] paramKeys = params.keySet().toArray(new String[params.keySet().size()]);
for (String key : paramKeys) {
StringBody valueBody = new StringBody(params.get(key), ContentType.TEXT_PLAIN);
multipartEntityBuilder.addPart(key, valueBody);
}
post.setEntity(multipartEntityBuilder.build());
return post;
}
}
I can't tell you why you run out of memory, but I can tell you, that it's no good idea to do it this way (despite you using a "recommended way").
I'm using CloseableHttpClient and PoolingHttpClientConnectionManager from version 4.5.3 and I never close them (they're both allocated once in a singleton and used for all requests). My application runs currently since maybe one month doing about one request per second (not really much).
Note that creating a new client for each connection is about as efficient as buying a new car for each trip.
I'm new android development and I have been researching this for some time however I can't seem to correct the error. I have a "The Method is undefined for the type object" error on both getStatusCode() and getReasonPhrase(). Any help would be appreciated.
My code as fallows
package com.javapapers.java.io;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.BasicHttpParams;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
public class HttpUtil {
public String getHttpResponse(HttpRequestBase request) {
String result = null;
try {
DefaultHttpClient httpClient = new DefaultHttpClient(new BasicHttpParams());
HttpResponse httpResponse = httpClient.execute(request);
int statusCode = httpResponse.getStatusLine().getStatusCode();
String reason = httpResponse.getStatusLine().getReasonPhrase();
StringBuilder sb = new StringBuilder();
if (statusCode == 200) {
HttpEntity entity = httpResponse.getEntity();
InputStream inputStream = entity.getContent();
BufferedReader bReader = new BufferedReader(
new InputStreamReader(inputStream, "UTF-8"), 8);
String line = null;
while ((line = bReader.readLine()) != null) {
sb.append(line);
}
} else {
sb.append(reason);
}
result = sb.toString();
} catch (UnsupportedEncodingException ex) {
} catch (ClientProtocolException ex1) {
} catch (IOException ex2) {
}
return result;
}
}
sorry if the question is too easy, but I do not know the answer..
What I have to do is to invoke a method of a web service using a java app.
Here you can find a web service:
http://muovi.roma.it/ws/xml/autenticazione/1
And I want Invoke the method called "autenticazione.Accedi:"
I have a python example that is doing this:
from xmlrpclib import Server
from pprint import pprint
DEV_KEY = 'Inserisci qui la tua chiave'
s1 = Server('http://muovi.roma.it/ws/xml/autenticazione/1')
s2 = Server('http://muovi.roma.it/ws/xml/paline/7')
token = s1.autenticazione.Accedi(DEV_KEY, '')
res = s2.paline.Previsioni(token, '70101', 'it')
pprint(res)
But I need the same operation in Java... Can anyone help me in this problem?
thank you
I recommend you using this project as a Library.
https://github.com/matessoftwaresolutions/AndroidHttpRestService
It makes you easy deal with apis, control network problems etc.
You can find a sample of use there.
You only have to:
Build your URL
Tell the component to execute in POST/GET etc. mode
Build your JSON
I hope it helps!!!
package com.example.jojo.gridview;
import android.util.Log;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
/**
* Created by jojo on 12/10/15.
*/
public class WebService {
String url="http://192.168.1.15/Travel_Dairy/";
String invokeGetWebservice(String webUrl)
{
String result = "";
webUrl=webUrl.replace(" ","%20");
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(webUrl);
HttpResponse response;
try {
response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream inputstream= entity.getContent();
BufferedReader bufferedreader = new BufferedReader(
new InputStreamReader(inputstream), 2 * 1024);
StringBuilder stringbuilder = new StringBuilder();
String currentline = null;
try {
while ((currentline = bufferedreader.readLine()) != null) {
stringbuilder.append(currentline + "\n");
}
} catch (IOException e) {
e.printStackTrace();
}
result = stringbuilder.toString();
Log.e("Result", result);
inputstream.close();
return result;
}
} catch (ClientProtocolException e1) {
Log.e("ClientProtocolException", e1.toString());
return result;
} catch (IOException e1) {
Log.e("IOException", e1.toString());
return result;
}
return result;
}
public List<DataModel> getTrips() {
String getname="view_details.php?";
String completeurlforget=url+getname;
//String seturl= "ur_id="+userid;
//String finalurl=completeurlforget+seturl;
String result=invokeGetWebservice(completeurlforget);
try {
JSONArray jsonarry=new JSONArray(result);
List<DataModel> ar=new ArrayList();
for(int i=0;i<jsonarry.length();i++)
{
JSONObject jsonobj=jsonarry.getJSONObject(i);
DataModel user=new DataModel();
user.setName(jsonobj.getString("name"));
user.setImage(jsonobj.getString("image"));
ar.add(user);
}
return ar;
} catch (JSONException e) {
e.printStackTrace();
}
return null;
}
}
I'm using Eclipse, for an application where I have to read a JSON file from an URL.
The code I'm using is this one:
http://collegewires.com/android/2012/06/json-parsing-in-android/
Ok, the CLASS which I'm using for reading JSON files is called Parser.java:
package com.cw.json;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.json.JSONException;
import org.json.JSONObject;
public class Parser {
static InputStream is = null;
static JSONObject jsonObject = null;
static String json = "";
// class constructor
public Parser() {
}
public JSONObject getJSONFromUrl(String url) {
// Making HTTP request
try {
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url);
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
is = httpEntity.getContent();
} catch (UnsupportedEncodingException exception) {
exception.printStackTrace();
} catch (ClientProtocolException exception) {
exception.printStackTrace();
} catch (IOException exception) {
exception.printStackTrace();
}
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, "iso-8859-1"), 8);
StringBuilder sBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sBuilder.append(line + "\n");
}
is.close();
json = sBuilder.toString();
} catch (Exception exception) {
exception.printStackTrace();
}
// Parsing the string to a JSON object
try {
jsonObject = new JSONObject(json);
} catch (JSONException exception) {
exception.printStackTrace();
}
// JSON String
return jsonObject;
}
}
I was using this code for reading a JSON file which is directly an ARRAY, but the code gives me an error.
My question is: is it possible to read an Array instead always reading an Object??
maybe using another class?
Try to modify your Parser.java to this, so you gat a Array not a Object. And please buy you a JAVA Book for beginners, so you do learn how to speak JAVA
public JSONArray getJSONFromUrl(String url) {
....
static JSONArray jsonArray = null;
....
// Parsing the string to a JSON Array
try {
jsonArray = new JSONArray(json);
} catch (JSONException exception) {
exception.printStackTrace();
}
return jsonArray;
}
The type Enum is not generic; it cannot be parameterized with arguments <RestClient.RequestMethod>
I've this error in the following code ..
package ayanoo.utility;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Vector;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.HTTP;
import android.util.Log;
public class RestClient {
public enum RequestMethod
{
GET,
POST
}
private Vector <NameValuePair> params;
private String url;
private int responseCode;
private String message;
private String response;
public String getResponse() {
return response;
}
public String getErrorMessage() {
return message;
}
public int getResponseCode() {
return responseCode;
}
public RestClient(String url)
{
this.url = url;
params = new Vector<NameValuePair>();
}
public void AddParam(String name, String value)
{
params.add(new BasicNameValuePair(name, value));
}
public void Execute(RequestMethod method) throws IOException
{
switch(method) {
case GET:
{
//add parameters
String combinedParams = "";
if(!params.isEmpty()){
combinedParams += "/";
for(NameValuePair p : params)
{
//String paramString = p.getName() + "=" + p.getValue();
String paramString = p.getValue();
if(combinedParams.length() > 1)
{
combinedParams += "&" + paramString;
}
else
{
combinedParams += paramString;
}
}
}
Log.d("URL See:",url + combinedParams);
URL urlObject = new URL(url + combinedParams);
//URL urlObject = new URL("http://www.aydeena.com/Services/Search.svc/JSON/SearchByText/1");
executeRequest(urlObject);
break;
}
case POST:
{
HttpPost request = new HttpPost(url);
//add headers
if(!params.isEmpty()){
request.setEntity(new UrlEncodedFormEntity(params, HTTP.UTF_8));
}
executeRequest(request, url);
break;
}
}
}
private void executeRequest(URL urlObject) throws IOException{
HttpURLConnection con = null;
con = (HttpURLConnection) urlObject.openConnection();
con.setReadTimeout(10000 /* milliseconds */);
con.setConnectTimeout(15000 /* milliseconds */);
con.setRequestMethod("GET");
//con.addRequestProperty("Referer",
// "http://www.pragprog.com/titles/eband/hello-android");
con.setDoInput(true);
// Start the query
con.connect();
response = convertStreamToString(con.getInputStream());
Log.d("Response:", response);
}
private void executeRequest(HttpUriRequest request, String url)
{
HttpClient client = new DefaultHttpClient();
Log.d("Test URL:", url);
HttpResponse httpResponse;
try {
httpResponse = client.execute(request);
responseCode = httpResponse.getStatusLine().getStatusCode();
message = httpResponse.getStatusLine().getReasonPhrase();
HttpEntity entity = httpResponse.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
response = convertStreamToString(instream);
// Closing the input stream will trigger connection release
instream.close();
}
} catch (ClientProtocolException e) {
client.getConnectionManager().shutdown();
e.printStackTrace();
} catch (IOException e) {
client.getConnectionManager().shutdown();
e.printStackTrace();
}
}
private static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
}
what's the problem ??? !
I had the same problem, and it turned out that it was because the standard lib was not in the eclipse class path for the project. Just go into Build Path -> Add Libraries and add the JRE System Library
Are you sure the Java compiler is set to 1.5 (default for android) or better? If you are using Eclipse you can see that from the preferences.
I had the same problem.
I only had one error in my project which was the "is not generic one'.
After I commented out the Enum code I found a lot more errors.
There seemed to be some kind of hold-up. Only after fixing the other errors and then removing the comments did it work.
Yes I also saw this error message for a project that was previously working fine.
I checked the compiler version (I am using 1.6) as well as the system library (it is already being used) to no avail.
Finally I just closed the project and then re-opened it, and then the problem went away. Sounds like an Eclipse bug to me.