This question already has answers here:
How to use Jsoup with Volley?
(3 answers)
Closed 6 years ago.
I'm trying to parse data from my server in Java with jsoup. I wrote a new function and it should return data in string format, but it returns blank string. Here is my code:
public String doc;
public String pare(final String url){
Thread downloadThread = new Thread() {
public void run() {
try {
doc = Jsoup.connect(url).get().toString();
}
catch (IOException e) {
e.printStackTrace();
}
}
};
downloadThread.start();
return doc;
}
You're returning the doc object immediately, before the thread has had a chance to add any data to it, so it should be no surprise that this returns empty. You can't return threaded information in this way, and instead will need to use some type of call-back mechanism, one that notifies you when the thread is done and when data is ready to be consumed.
On android platform, you shouldn't ask Jsoup to download anything for you. Under the hood, Jsoup make use of HttpUrlConnection. This class is notoriously slow and has some known issues.
Use a faster alternative instead: Volley.
Here is the function in your post taking advantage of Volley. In the following sample code, I'm using a CountDownLatch to wait for the data.
private static RequestQueue myRequestQueue = null;
public String pare(final String url) throws Exception {
final String[] doc = new String[1];
final CountDownLatch cdl = new CountDownLatch(1);
StringRequest documentRequest = new StringRequest( //
Request.Method.GET, //
url, //
new Response.Listener<String>() {
#Override
public void onResponse(String response) {
doc[0] = Jsoup.parse(response).html();
cdl.coutDown();
}
}, //
new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
Log.e("MyActivity", "Error while fetching " + url, error);
}
} //
);
if (myRequestQueue == null) {
myRequestQueue = Volley.newRequestQueue(this);
}
// Add the request to the queue...
myRequestQueue.add(documentRequest);
// ... and wait for the document.
// NOTA: User experience can be a concern here. We shouldn't freeze the app...
cdl.await();
return doc[0];
}
I totally agree with the above answer. You can follow any of the below tutorials for fetching data from server
http://www.androidhive.info/2014/05/android-working-with-volley-library-1/
http://www.vogella.com/tutorials/Retrofit/article.html
These two are the best libraries for Network calls in android
Before the return statement add a downloadThread.join(). This will wait until the thread has finished and put the response into doc. But: Doing so you will loose all benefit from the asynchronous execution, it's behaving the same as you just would code:
public String pare(final String url){
return Jsoup.connect(url).get().toString();
}
Related
I wrote a program in Java to find all pages of a website, starting with the URL of the startpage (using Jsoup as webcrawler). It is ok for small websites but too slow for sites with 200 or more pages:
public class SiteInspector {
private ObservableSet<String> allUrlsOfDomain; // all URLS found for site
private Set<String> toVisit; // pages that were found but not visited yet
private Set<String> visited; // URLS that were visited
private List<String> invalid; // broken URLs
public SiteInspector() {...}
public void getAllWebPagesOfSite(String entry) //entry must be startpage of a site
{
toVisit.add(entry);
allUrlsOfDomain.add(entry);
while(!toVisit.isEmpty())
{
String next = popElement(toVisit);
getAllLinksOfPage(next); //expensive
toVisit.remove(next);
}
}
public void getAllLinksOfPage(String pageURL) {
try {
if (urlIsValid(pageURL)) {
visited.add(pageURL);
Document document = Jsoup.connect(pageURL).get(); //connect to pageURL (expensive network operation)
Elements links = document.select("a"); //get all links from page
for(Element link : links)
{
String nextUrl = link.attr("abs:href"); // "http://..."
if(nextUrl.contains(new URL(pageURL).getHost())) //ignore URLs to external hosts
{
if(!isForbiddenForCrawlers(nextUrl)) // URLS forbidden by robots.txt
{
if(!visited.contains(nextUrl))
{
toVisit.add(nextUrl);
}
}
allUrlsOfDomain.add(nextUrl);
}
}
}
else
{
invalid.add(pageURL); //URL-validation fails
}
}
catch (IOException e) {
e.printStackTrace();
}
}
private boolean isForbiddenForCrawlers(String url){...}
private boolean urlIsValid(String url) {...}
public String popElement(Set<String> set) {...}
I know I have to run the expensive network-operation in extra threads.
Document document = Jsoup.connect(pageURL).get(); //connect to pageURL
My problem is that I have no idea how to properly outsource this operation while keeping the sets consistent (how to synchronize?). If possible I want to use a ThreadPoolExecutor to control the amount of threads that is getting started during the process. Do you guys have an idea how to solve this? Thanks in advance.
To use threads and also keep the sets consistent, you just need to create a thread that receives the variable you want to add to the Set but created empty, so the thread fills it when done and then adds it to the Set.
A simple example of that could be:
Main.class
for (String link : links) {
String validUrl = null;
taskThread = new Thread( new WebDownloadThreadHanlder(link, validUrl, barrier));
taskThread.start();
if (validUrl != null) {
allUrlsOfDomain.add(validUrl);
}
}
barrier.acquireUninterruptibly(links.size());
WebDownloadThreadHandler.class
public class WebDownloadThreadHandler implements Runnable {
private String link;
private String validUrl;
private Semaphore barrier;
public ScopusThreadHandler(String link, String validUrl, Semaphore barrier) {
this.link = link;
this.validUrl = null;
this.barrier = barrier;
}
public void run () {
try {
Document document = Jsoup.connect(this.link).userAgent("Mozilla/5.0");
Elements elements = document.select(YOUR CSS QUERY);
/*
YOUR JSOUP CODE GOES HERE, AND STORE THE VALID URL IN: this.validUrl = THE VALUE YOU GET;
*/
} catch (IOException) {
e.printStackTrace();
}
this.barrier.release();
}
}
What you are doing here is creating a thread for every web you want to get all the links from, and storing them into variables, if you want to retrieve more than one lvalid link from every page, you can do it using a Set and adding it a to a global set (appending it). The thing is that to keep your code consistent you need to store the retrieved values in the variable you pass the thread as argument using THIS keyword.
Hope it helps! If you need anything else feel free to ask me!
I have been trying for awhile to figure out an issue with Asynchronous i/o in an android application that I am working on.
This application is required to download data to from a series of tables from Microsoft Dynamics CRM.
Once the data has been down it must preform a series of operations on the data to fill out some forms.
My problem is that I must wait for the downloads to be complete in order to start the update process.
If I add a any form of wait to my code it seems that it blocks indefinitely and never executes the callback.
I have tried methods using AtomicBooleans, AtomicIntegers, and CountDownLatchs with no success.
Here is an example using an AtomicInteger.
final CountDownLatch latch = new CountDownLatch(1);
OrganizationServiceProxy orgService;
orgService = new OrganizationServiceProxy(Constant.ENDPOINT, CRMLogin.getRequestInterceptor());
ColumnSet columnSet = new ColumnSet();
columnSet.AddColumns(AccountEntry.FETCH_COLS);
orgService.Retrieve(AccountEntry.ENTITY, UUID.fromString(accountid), columnSet, new Callback<Entity>() {
#Override
public void success(Entity entity, Response response) {
Account account = new Account();
//Load the existing fields for the account
account.load(index);
String activityid = account.getValue(AccountEntry.ACTIVITY_ID);
String recordid = account.getValue(AccountEntry.RECORD_ID);
String name = account.getValue(AccountEntry.ACCOUNT_NAME);
//Overload the fields for the account
account.load(entity);
//Reset overloaded fields on the account.
account.setValue(AccountEntry.ACTIVITY_ID, activityid);
account.setValue(AccountEntry.RECORD_ID, recordid);
account.setValue(AccountEntry.ACCOUNT_NAME, name);
//overwrite the record in the database.
account.setValue(AccountEntry.SYNCED, "1");
account.update();
Log.d("pullAccount>>>", accountid + " " + "pulled.");
latch.countDown();
}
#Override
public void failure(RetrofitError error) {
Log.d("pullAccount>>>", accountid + " " + error.getMessage());
latch.countDown();
}
});
try{
latch.await(); //THIS BLOCKS FOREVER AND EVER
}
catch (Exception e){
}
Of note is the CallBack is implemented using Retrofit.
Any guidance would be greatly appreciated.
Thank you.
Look at AsyncTask it will handle what you want in a way that Android is optimized for. There is example usage here
http://developer.android.com/reference/android/os/AsyncTask.html
EDIT:
I kinda threw this together, let me know if it works as you would expect
public class AsyncOrganizationService extends AsyncTask<Void, Void, Entity> {
#Override
protected Entity doInBackground(Void... params) {
final CountDownLatch blocker = new CountDownLatch(1);
OrganizationServiceProxy orgService;
orgService = new OrganizationServiceProxy(Constant.ENDPOINT, CRMLogin.getRequestInterceptor());
ColumnSet columnSet = new ColumnSet();
columnSet.AddColumns(AccountEntry.FETCH_COLS);
final SettableFuture<Entity> result = SettableFuture.create();
orgService.Retrieve(AccountEntry.ENTITY, UUID.fromString(accountid), columnSet, new SortedList.Callback<Entity>() {
#Override
public void success(Entity entity, HttpHelper.Response response) {
result.set(entity);
blocker.countDown();
}
});
try {
blocker.await();
return result.get();
} catch (InterruptedException | ExecutionException e) {
throw new RuntimeException(e);
}
}
#Override
protected void onPostExecute(Entity entity) {
Account account = new Account();
//Load the existing fields for the account
account.load(index);
String activityid = account.getValue(AccountEntry.ACTIVITY_ID);
String recordid = account.getValue(AccountEntry.RECORD_ID);
String name = account.getValue(AccountEntry.ACCOUNT_NAME);
//Overload the fields for the account
account.load(entity);
//Reset overloaded fields on the account.
account.setValue(AccountEntry.ACTIVITY_ID, activityid);
account.setValue(AccountEntry.RECORD_ID, recordid);
account.setValue(AccountEntry.ACCOUNT_NAME, name);
//overwrite the record in the database.
account.setValue(AccountEntry.SYNCED, "1");
account.update();
Log.d("pullAccount>>>", accountid + " " + "pulled.");
}
Im using Guava's SettableFuture class (http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/SettableFuture.html). Guava is quite an amazing library - if you're not using it you should consider doing so. Otherwise, you could whip something up really quick
So I have been trying to return the content grabbed from httpResponse from a class, however, im not very successful at the moment.
String requestContent = null;
Net.HttpRequest httpRequest;
httpRequest = new Net.HttpRequest(Net.HttpMethods.GET);
httpRequest.setUrl("https://api.parse.com/1/classes/gamerooms/");
System.out.println(Parse.getRestAPIKey() + Parse.getApplicationId());
httpRequest.setHeader("X-Parse-Application-Id", Parse.getApplicationId());
httpRequest.setHeader("X-Parse-REST-API-Key", Parse.getRestAPIKey());
httpRequest.setContent(requestContent);
Gdx.net.sendHttpRequest(httpRequest, new Net.HttpResponseListener() {
#Override
public void handleHttpResponse(Net.HttpResponse httpResponse) {
content = httpResponse.getResultAsString();
}
#Override
public void failed(Throwable t) {
}
#Override
public void cancelled() {
}
});
return content;
}
When I return content, it is empty but when i print the content in handleHttpResponse, i can see it there. Any solutions?
The problem is, that it is handled like a Thread. So when you're returning the content, the answer maybe not already arrived. So you must handle the result in the handleHttpResponse method. If you don't want all the code in that method, you could also call a function.
Also because you're in a Thread I think you should call Gdx.app.postRunnable(Runnable runnable) if you want to change something in the code, which you shouldn't while you are in the render method or need OpenGL context. The Runnable will be called directly before the render method. Threading libGdx
I have the following code for FitBit integration into Android, it is used from this library https://github.com/manishsri01/FitbitIntegration, I can get the response.getBody() to show the JSON body in the webview but I would like the application to be able to automatically update the code without having to login and grab the PIN for OAuth everytime I run the app. What can I do to fix this? I would also like to parse the JSON .getBody() into separate string variables. How can I accomplish this?
MainActivity
public class MainActivity extends Activity {
OAuthService service;
Token requestToken;
// Replace these with your own api key and secret
private String apiKey = "************************";
private String apiSecret = "*************************";
private String accessToken;
private String tokenSecret;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
final WebView wvAuthorize = (WebView) findViewById(R.id.wvAuthorize);
final EditText etPIN = (EditText) findViewById(R.id.etPIN);
service = new ServiceBuilder().provider(FitbitApi.class).apiKey(apiKey)
.apiSecret(apiSecret).build();
// network operation shouldn't run on main thread
new Thread(new Runnable() {
public void run() {
requestToken = service.getRequestToken();
final String authURL = service
.getAuthorizationUrl(requestToken);
// Webview nagivation should run on main thread again...
wvAuthorize.post(new Runnable() {
#Override
public void run() {
wvAuthorize.loadUrl(authURL);
}
});
}
}).start();
}
public void btnRetrieveData(View view) {
EditText etPIN = (EditText) findViewById(R.id.etPIN);
String gotPIN = etPIN.getText().toString();
final Verifier v = new Verifier(gotPIN);
// network operation shouldn't run on main thread
new Thread(new Runnable() {
public void run() {
Token accessToken = service.getAccessToken(requestToken, v);
OAuthRequest request = new OAuthRequest(Verb.GET,
"http://api.fitbit.com/1/user/-/profile.json");
service.signRequest(accessToken, request); // the access token from step
// 4
final Response response = request.send();
final TextView tvOutput = (TextView) findViewById(R.id.tvOutput);
// Visual output should run on main thread again...
tvOutput.post(new Runnable() {
#Override
public void run() {
tvOutput.setText(response.getBody());
}
});
}
}).start();
}
}
FitBitApi
public class FitbitApi extends DefaultApi10a {
private static final String AUTHORIZE_URL = "https://www.fitbit.com/oauth/authorize?oauth_token=%s";
public String getAccessTokenEndpoint() {
return "https://api.fitbit.com/oauth/access_token";
}
public String getRequestTokenEndpoint() {
return "https://api.fitbit.com/oauth/request_token";
}
public String getAuthorizationUrl(Token token) {
return String.format(AUTHORIZE_URL, token.getToken());
}
}
It sounds like you have two separate questions here. Firstly, in regards to saving credentials, there are a number of ways you can do this, the easiest is probably by saving the user/pin details in Android's SharedPreferences. However, you'll still need to make the request for an access token. You should also save the access token (in a cache or DB) and re-use it until it is expired. You may want to read up on ways to secure these credentials if they're considered private.
Your second question regarding parsing JSON is quite common, and if your intention is to map JSON objects to Java objects, you should consider using Google's GSON or Jackson JSON Processor.
If you're intending for Fitbit's API to be a large part of your app, consider using Spring Social and make a Fitbit endpoint (there is a Spring for Android which uses Spring Social). It might be a bit overkill though, but I generally like the structure.
I'm trying to implement search autocompletion with android-query library
I have callback instance in my activity:
Callback:
class SearchCompleteCallback extends AjaxCallback<ItemSearchResult> {
public void callback(String url, ItemSearchResult searchResult, AjaxStatus status) {
Log.d("SCB", String.format("Url:%s\n Msg: %s\n Code: %s\n Error: %s",
url,
status.getMessage(),
status.getCode(),
status.getError()));
if (searchResult != null) {
Log.d("SCB", String.format("Status: %s\n Val: %s",
searchResult.getStatus(),
searchResult.getInnGroup().getItems()));
updateSearchResult(searchResult);
}
else {
Log.w("SCB", "Ajax failed");
}
}
}
Search routine, that called on text change:
private void doSearch(String query) {
ppApi.getSearchResult(query, searchCompleteListener);
}
and
APIClass
public class PPServerApi {
private AQuery aq;
private GsonTransformer transformer;
private static class GsonTransformer implements Transformer{
public <T> T transform(String url, Class<T> type, String encoding, byte[] data, AjaxStatus status) {
Gson g = new Gson();
return g.fromJson(new String(data), type);
}
}
public PPServerApi(AQuery newAq){
aq = newAq;
transformer = new GsonTransformer();
AQUtility.setDebug(true);
AjaxCallback.setTransformer(transformer);
}
public void getSearchResult(String itemName, AjaxCallback<ItemSearchResult> cb){
String url = "http://my.api.server/search?q=" + itemName;
aq.ajax(url, ItemSearchResult.class, cb.header("content-type", "application/json"));
}
}
So, the question is how to abort old queries before sending new one ?
(I don't need result of old queries if text in search field changed)
I've tried to call searchCompleteListener.abort() in doSearch(), but it causes exception in next going query:
08-09 20:59:10.551: W/AQuery(6854): get:http://my.api.server/search?q=abc
08-09 20:59:10.551: W/AQuery(6854): creating http client
08-09 20:59:10.561: W/AQuery(6854): java.io.IOException: Aborted
08-09 20:59:10.561: W/AQuery(6854): at com.androidquery.callback.AbstractAjaxCallback.httpDo(AbstractAjaxCallback.java:1569)
...
so, i can't perform even single query in this case.
There is no way of making android-query cancel an AJAX request once it has been started.
You'll have to use another library, sorry.
What you can do is to check if the request has become obsolete when it finishes.
You could do that by checking if the URL matches the latest URL you requested for
if (searchResult != null && url.equals(latestRequestUrl)) {
(note, you'd have to let getSearchResult return the URL)
You can use the droidQuery library instead. Using droidQuery, you can cancel all Ajax tasks using the call:
$.ajaxKillAll();
You can also perform your request with this:
$.ajax(new AjaxOptions().url(url).header("content-type", "application/json").type("json").dataType("GET").dataType("json").success(new Function() {
#Override
public void invoke($ droidQuery, Object... params) {
JSONObject json = (JSONObject) params[0];
//TODO handle json
}
}).error(new Function() {
#Override
public void invoke($ droidQuery, Object... params) {
AjaxError error = (AjaxError) params[0];
Log.e("Ajax", "Error " + error.status + ": " + error.reason);
}
}));
You can abort any aquery processing using this.
private AjaxCallback<String> ajaxCallback = new AjaxCallback<String>(){
#Override
public void callback(String url, String object, AjaxStatus status) {
//do your processing with server response
processInformation(result);
};
};
//on our previous code
query.ajax(remoteUrl,String.class,ajaxCallback);
public void cancelAquery(){
//this statement does cancel the request i.e. we won't receive any information on callback method
//ajaxCallback.async(null);
ajaxCallback.abort();
}
For more info, you can see this link https://laaptu.wordpress.com/tag/android-cancelling-aquery/
just call aq.ajaxCancel() for more details refer to the API docs
http://android-query.googlecode.com/svn/trunk/javadoc/com/androidquery/AbstractAQuery.html#ajaxCancel()