I am trying to pull about 20,000 users from my Google domain. However, i know that Google only has a limit of about 500 users for a pull request. I know about the pageToken stuff, but the documentation for it online is terrible. Can someone show me how to use the pageToken? Please keep in mind i am using the google client libraries. This is what my code looks like so far:
#Test
public void paginationTest() throws IOException, NullPointerException, GeneralSecurityException {
try {
Directory directory = GCAuthentication.getDirectoryService("xxx", "vvv", dddd);
Directory.Users.List list = directory.users().list().setOrderBy("email").setMaxResults(500).setDomain("dev.royallepage.ca");
do {
com.google.api.services.admin.directory.model.Users users = list.execute();
java.util.List<User> uL = users.getUsers();
//uL.addAll(users.getUsers());
//list.setPageToken(list.getPageToken());
System.out.println(uL.size());
}while (list.getPageToken() != null && list.getPageToken().length() > 0);
}catch(NullPointerException e) {
}
}
Please advise what i am doing wrong! Thanks,
Mesam
You will have to create a function that will get the pageToken variable then call another request including the nextPageToken.
Use the pageToken query string for responses with large number of groups. In the case of pagination, the response returns the nextPageToken property which gives a token for the next page of response results. Your next request uses this token as the pageToken query string value.
Sample Code Request:
GET https://www.googleapis.com/admin/directory/v1/users
?domain=primary domain name&pageToken=token for next results page
&maxResults=max number of results per page
&orderBy=email, givenName, or familyName
&sortOrder=ascending or descending
&query=email, givenName, or familyName:the query's value*
Hope this helps!
Related
I am working with Google Ads API. Based on my use case, I need to get the impressions, clicks, and other statistics about the products based on location(Without creating the campaign).
I contacted the Google Ads API team to address the issue but the doc reference they provided me didn't work in my case. (I'm not using keywords also)
generate forecast metrics
And also the query I used to access the data is...
public void queryTest(long customerId,String campaignId) {
try (GoogleAdsServiceClient googleAdsServiceClient =
googleAdsClient.getLatestVersion().createGoogleAdsServiceClient()) {
SearchGoogleAdsStreamRequest request =
SearchGoogleAdsStreamRequest.newBuilder()
.setCustomerId(Long.toString(customerId))
.setQuery("SELECT metrics.clicks, metrics.impressions,geographic_view.country_criterion_id FROM geographic_view WHERE geographic_view.country_criterion_id = 1009919")
.build();
ServerStream<SearchGoogleAdsStreamResponse> stream =
googleAdsServiceClient.searchStreamCallable().call(request);
for (SearchGoogleAdsStreamResponse response : stream) {
for (GoogleAdsRow googleAdsRow : response.getResultsList()) {
System.out.println("outPut"+googleAdsRow.getGeographicView());
}
}
}
}
Can someone please help me to resolve this issue??
Thank you!!
public void conectUrl() throws IOException, InterruptedException {
product= new ArrayList<>();
String url = "https://www.continente.pt/stores/continente/pt-pt/public/pages/category.aspx?cat=campanhas#/?page=1&sf=Revelance";
page = Jsoup.connect(url).userAgent("JSoup scraper").get();
//get actual page
Elements paginaAtu=page.getElementsByClass("_actualPage");
paginaAtual=Integer.parseInt(paginaAtu.attr("value"));
//get Total Pages
Elements nextPage=page.getElementsByClass("_actualTotalPages");
numPaginas =Integer.parseInt(nextPage.attr("value"));
for(paginaAtual=1;paginaAtual<numPaginas;paginaAtual++) {
getProductInfo("https://www.continente.pt/stores/continente/pt-pt/public/pages/category.aspx?cat=campanhas#/?page="+paginaAtual+"&sf=Revelance");
}
}
Always return the same resul with different URL. I already searched about jsoup cache , i am not the first person doing this question, however nobody says how to resolve the situation. In theory, JSoup doesn't cache url pages...
I already did the code "sleep" during 30 seconds to load the new URL however still not working, return always the same result.
Anybody can help me? Thank you in advance.
I was wondering what the best way is for me to refresh an access token that is obtained through the client credentials flow within OAuth 2.0. I've read over the spec, but I can't seem to be able to find an answer for my particular situation.
For my specific case, I am using the Spotify Web API for my Android app in order to search for content (tracks, albums, and/or artists). In order to perform a search, I need an access token. Since I'm not interested in a Spotify user's data, I can use the client credentials flow to obtain the access token, which is explain in Spotify's terms here.
Because the access token can eventually expire, I had to figure out a way to refresh it once expiration occurred. What I'm ultimately wondering is if my approach is valid and if there's any concern with how I've approached this.
First and foremost, I stored my access token within SharedPreferences. Within onCreate(), I have the following:
#Override
protected void onCreate(Bundle savedInstanceState) {
// A bunch of other stuff, views being initialized, etc.
mAccessToken = getAccessToken();
// If the access token is expired, then we will attempt to retrieve a new one
if (accessTokenExpired()) {
retrieveAccessToken();
}
}
I've defined accessTokenExpired() and retrieveAccessToken() as follows:
private boolean accessTokenExpired() {
// If mAccessToken hasn't yet been initialized, that means that we need to try to retrieve
// an access token. In this case, we will return true;
if (mAccessToken == null) {
return true;
}
SharedPreferences preferences = getPreferences(Context.MODE_PRIVATE);
long timeSaved = preferences.getLong(PREFERENCES_KEY_TOKEN_RESPONSE_TIME_SAVED, 0L);
long expiration = preferences.getLong(PREFERENCES_KEY_TOKEN_RESPONSE_EXPIRATION, 0L);
long now = System.currentTimeMillis()/1000;
long timePassed = Math.abs(now - timeSaved);
if (timePassed >= expiration) {
return true;
} else {
return false;
}
}
One thing worth noting about retrieveAccessToken() is that I'm using Retrofit for my HTTP request:
private void retrieveAccessToken() {
// First, we obtain an instance of SearchClient through our ClientGenerator class
mClient = ClientGenerator.createClient(SearchClient.class);
// We then obtain the client ID and client secret encoded in Base64.
String encodedString = encodeClientIDAndSecret();
// Finally, we initiate the HTTP request and hope to get the access token as a response
Call<TokenResponse> tokenResponseCall = mClient.getAccessToken(encodedString, "client_credentials");
tokenResponseCall.enqueue(new Callback<TokenResponse>() {
#Override
public void onResponse(Call<TokenResponse> call, Response<TokenResponse> response) {
Log.d(TAG, "on Response: response toString(): " + response.toString());
TokenResponse tokenResponse = null;
if (response.isSuccessful()) {
tokenResponse = response.body();
Log.d(TAG, tokenResponse.toString());
mAccessToken = tokenResponse.getAccessToken();
saveAccessToken(tokenResponse);
}
}
#Override
public void onFailure(Call<TokenResponse> call, Throwable t) {
Log.d(TAG, "onFailure: request toString():" + call.request().toString());
mAccessToken = "";
}
});
}
Finally, saveAccessToken(tokenResponse) is sort of the complement of accessTokenExpired(), where I'm saving the values from the token response into SharedPreferences rather than retrieving them.
Are there any concerns with how I'm doing this? I got the idea from this SO post and slightly modified it. Spotify doesn't provide a refresh token in their access token response. Therefore, I can't make use of it here to reduce the number of access token requests I make.
Any input on this would be greatly appreciated!
Two considerations are:
you probably want some error handling around the requests you make using the access token that can handle the token expiring and do retries. The two situations where this will help are
when the token expires between checking if it's valid and your usage of it
when in the cycle of check the token is valid -> make some requests with the token -> repeat, you spend over an hour using the token. Another way you can do it is to calculate now + expected_api_request_time > token_expiration_time where expected_api_request_time would be a constant you set, but I think handling token expiry as an exception is better practice (you probably want to be able to make retries anyway in cases of network instability).
you can perform the calculations to work out when the token expires either when you retrieve the timeSaved and expiration from your local storage, or just calculate the time the token will expire initially and save that. This is relatively minor, both this and the way you've done it are fine I think.
According to Facebook Docs
If your app is making enough calls to be considered for rate limiting by our system, we return an X-App-Usage HTTP header. [...] When any of these metrics exceed 100 the app will be rate limited.
I am using Facebook4J to connect my application to the Facebook API. But I could not find any documentation about how I can get the X-App-Usage HTTP header after a Facebook call, in order to avoid being rate limited. I want to use this header to know dinamically if I need to increase or decrease the time between each API call.
So, my question is: using Facebook4J, is possible to check if Facebook returned the X-App-Usage HTTP header and get it? How?
There is a getResponseHeader method for the response of BatchRequests in facebook4j see Facebook4j code examples
You could try getResponseHeader("X-App-Usage")
// Executing "me" and "me/friends?limit=50" endpoints
BatchRequests<BatchRequest> batch = new BatchRequests<BatchRequest>();
batch.add(new BatchRequest(RequestMethod.GET, "me"));
batch.add(new BatchRequest(RequestMethod.GET, "me/friends?limit=50"));
List<BatchResponse> results = facebook.executeBatch(batch);
BatchResponse result1 = results.get(0);
BatchResponse result2 = results.get(1);
// You can get http status code or headers
int statusCode1 = result1.getStatusCode();
String contentType = result1.getResponseHeader("Content-Type");
// You can get body content via as****() method
String jsonString = result1.asString();
JSONObject jsonObject = result1.asJSONObject();
ResponseList<JSONObject> responseList = result2.asResponseList();
// You can map json to java object using DataObjectFactory#create****()
User user = DataObjectFactory.createUser(jsonString);
Friend friend1 = DataObjectFactory.createFriend(responseList.get(0).toString());
Friend friend2 = DataObjectFactory.createFriend(responseList.get(1).toString());
I have a code that fetches conversations and the messages inside them (a specific number of pages). It works most of the time, but for certain conversations it throws an exception, such as:
Exception in thread "main" com.restfb.exception.FacebookOAuthException: Received Facebook error response of type OAuthException: Unknown path components: /[id of the message]/messages (code 2500, subcode null)
at com.restfb.DefaultFacebookClient$DefaultGraphFacebookExceptionMapper.exceptionForTypeAndMessage(DefaultFacebookClient.java:1192)
at com.restfb.DefaultFacebookClient.throwFacebookResponseStatusExceptionIfNecessary(DefaultFacebookClient.java:1118)
at com.restfb.DefaultFacebookClient.makeRequestAndProcessResponse(DefaultFacebookClient.java:1059)
at com.restfb.DefaultFacebookClient.makeRequest(DefaultFacebookClient.java:970)
at com.restfb.DefaultFacebookClient.makeRequest(DefaultFacebookClient.java:932)
at com.restfb.DefaultFacebookClient.fetchConnection(DefaultFacebookClient.java:356)
at test.Test.main(Test.java:40)
After debugging I found the ID that doesn't work and tried to access it from graph-api, which results in an "unknown path components" error. I also attempted to manually find the conversation in me/conversations and click the next page link in the graph api explorer which also lead to the same error.
Is there a different way to retrieve a conversation than by ID? And if not, could someone show me an example to verify first if the conversation ID is valid, so if there are conversations I can't retrieve I could skip them instead of getting an error. Here's my current code:
Connection<Conversation> fetchedConversations = fbClient.fetchConnection("me/Conversations", Conversation.class);
int pageCnt = 2;
for (List<Conversation> conversationPage : fetchedConversations) {
for (Conversation aConversation : conversationPage) {
String id = aConversation.getId();
//The line of code which causes the exception
Connection<Message> messages = fbClient.fetchConnection(id + "/messages", Message.class, Parameter.with("fields", "message,created_time,from,id"));
int tempCnt = 0;
for (List<Message> messagePage : messages) {
for (Message msg : messagePage) {
System.out.println(msg.getFrom().getName());
System.out.println(msg.getMessage());
}
if (tempCnt == pageCnt) {
break;
}
tempCnt++;
}
}
}
Thanks in advance!
Update: Surrounded the problematic part with a try catch as a temporary solution, also counted the number of occurrences and it only effects 3 out of 53 conversations. I also printed all the IDs, and it seems that these 3 IDs are the only ones that contain a "/" symbol, I'm guessing it has something to do with the exception.
The IDs that work look something like this: t_[text] (sometimes a "." or a ":" symbol) and the ones that cause an exception are always t_[text]/[text]
conv_id/messages is not a valid graph api call.
messages is a field of conversation.
Here is what you do (single call to api):
Connection<Conversation> conversations = facebookClient.fetchConnection("me/conversations", Conversation.class);
for (Conversation conv : conversations.getData()) {
// To get list of messages for given conversation
LinkedList<Message> allConvMessagesStorage = new LinkedList<Message>();
Connection<Message> messages25 = facebookClient.fetchConnection(conv.getId()+"/messages", Message.class);
//Add messages returned
allConvMessagesStorage.addAll(messages25.getData());
//Check if there is next page to fetch
boolean progress = messages25.hasNext();
while(progress){
messages25 = facebookClient.fetchConnectionPage(messages25.getNextPageUrl(), Message.class);
//Append next page of messages
allConvMessagesStorage.addAll(messages25.getData());
progress = messages25.hasNext();
}
}