This question already has answers here:
How to retrieve more than 100 results using Twitter4j
(4 answers)
Closed 6 years ago.
I am trying to create a program that searches a query from twitter. The problem I am having is that the API returns only a 100 result queries and when I try to retrieve more it keeps giving me the same results again.
User user = twitter.showUser("johnny");
Query query = new Query("football");
query.setCount(100);
query.lang("en");
int i=0;
try {
QueryResult result = twitter.search(query);
for(int z = 0;z<2;z++){
for( Status status : result.getTweets()){
System.out.println("#" + status.getUser().getScreenName() + ":" + status.getText());
i++;
}
}
The program will print me 200 results relating to the query "football", but instead of giving me 200 different results it prints a 100 results twice. My end results should be that I can print as many different results as the rate limit allows. I have seen programs that return more than 100 responses for a specific user, but I haven't seen something that can return more than a 100 responses for a unique query like "football".
To get more than 100 results on a search Query you need to call to the next iteration of the Query.
Query query = new Query("football");
QueryResult result;
int Count=0;
do {
result = twitter.search(query);
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
System.out.println("#" + tweet.getUser().getScreenName() + ":" + tweet.getText());
Count++;
}
try {
Thread.sleep(500);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
while ((query = result.nextQuery()) != null);
System.out.println(Count);
System.exit(0);
I just tested it and got 275 tweets, keep in mind this from the documentation:
The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.
And:
Before getting involved, it’s important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.
Related
Hello i am trying to Just retrieve 1 tweet using Twitter4j but the setCount() method is doing what ever it wants ( maybe its just me doing something wrong ) i leave my code below.
I have tried with multiple options "1", "2","0" and regardless the number it retrieves any amount of tweets from 3 to 10.
ConfigurationBuilder cf = new ConfigurationBuilder();
cf.setDebugEnabled(true)
.setOAuthConsumerKey("xxx")
.setOAuthConsumerSecret("xxxx")
.setOAuthAccessToken("xxxx")
.setOAuthAccessTokenSecret("xxxx");
TwitterFactory tf = new TwitterFactory(cf.build());
twitter4j.Twitter twitter = tf.getInstance();
try {
Query query = new Query("sverige");
QueryResult result;
do {
query.setCount(2);
result = twitter.search(query);
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
System.out.println("#" + tweet.getUser().getScreenName() + tweet.getFavoriteCount() + tweet.getUser().getName());
}
} while ((query = result.nextQuery()) != null);
System.exit(0);
} catch (TwitterException te) {
te.printStackTrace();
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
You are getting all the tweets from that user, at 2 results per page. So if they had 22 tweets, you would get 11 pages of 2 tweets per page.
query.setCount(2); // set the number of tweets per page
// get the next page until there are no more pages
} while ((query = result.nextQuery()) != null);
The loop will continue to query for tweets while more tweets are available.
This is because
(query = result.nextQuery()) != null is true until all of the tweets (matching the query) are read.
setCount only limits the amount of tweets each search operation returns.
WHen debugging a situation like this (if you don't want to look into the source code / documentation) you can test how many times the outer loop occurs.
I am working on accessing data from Google BigQuery, the data is 500MB which I need to transform as part of the requirement. I am setting Allow Large Results, setting a destination table etc.
I have written a java job in Google's new cloud library since that is recommended now - com.google.cloud:google-cloud-bigquery:0.21.1-beta (I have tried 0.20 beta as well without any fruitful results)
I am having problem with pagination of this data, the library is inconsistent in fetching results page wise. Here is my code snippet,
Code Snippet
System.out.println("Accessing Handle of Response");
QueryResponse response = bigquery.getQueryResults(jobId, QueryResultsOption.pageSize(10000));
System.out.println("Got Handle of Response");
System.out.println("Accessing results");
QueryResult result = response.getResult();
System.out.println("Got handle of Result. Total Rows: "+result.getTotalRows());
System.out.println("Reading the results");
int pageIndex = 0;
int rowId = 0;
while (result != null) {
System.out.println("Reading Page: "+ pageIndex);
if(result.hasNextPage())
{
System.out.println("There is Next Page");
}
else
{
System.out.println("No Next Page");
}
for (List<FieldValue> row : result.iterateAll()) {
System.out.println("Row: " + rowId);
rowId++;
}
System.out.println("Getting Next Page: ");
pageIndex++;
result = result.getNextPage();
}
Output print statements
Accessing Handle of Response
Got Handle of Response
Accessing results
Got handle of Result. Total Rows: 9617008
Reading the results
Reading Page: 0
There is Next Page
Row: 0
Row: 1
Row: 2
Row: 3
:
:
Row: 9999
Row: 10000
Row: 10001
:
:
Row: 19999
:
:
Please note that it never hits/prints - "Getting Next Page: ".
My expectation was that I would get data in chunks of 10000 rows at a time. Please note that if I run the same code on a query which returns 10-15K rows and set the pageSize to be 100 records, I do get the "Getting Next Page:" after every 100 rows. Is this a known issue with this beta library?
This looks very close to a problem I have been struggling with for hours. And I just found the solution, so I will share it here, even though you probably found a solution yourself a long time ago.
I did exactly like the documentation and tutorials said, but my page size were not respected and I kept getting all rows every time, no matter what I did. Eventually I found another example, official I think, right here.
What I learned from that example is that you should only use iterateAll() to get the rest of the rows. To get the current page rows you need to use getValues() instead.
The title is confusing, I know, but I do not know how else to phrase this question.
Using twitter4j, I am able to get tweets and the list of users who have retweeted that tweet, like this
However, if the tweet is actually a retweet then I am not able to get the list of retweeters. Example
This is the code I am using to get the list of retweeters:
if(tweet.getId() > 0 && tweet.getRetweetCount() > 0) {
try {
List<Status> statuses = twitter.getRetweets(tweet.getId());
for (Status status : statuses) {
System.out.println("\n" + "\t" + "Retweeter ID:" + status.getUser().getId() + "\n" + "\t" + "Retweeter Name:" + status.getUser().getScreenName());
}
} catch (TwitterException e) {
//twitter.getRetweeterIds(tweet.getId(), 2, -1);
e.printStackTrace();
}
}
How do I get the retweeters of a retweet?
Twitter4J is a Java client library that interfaces with the Twitter REST API.
To understand the right call to use it's best to understand the underlying REST API.
Looking at the Twitter Rest API we can see an API that returns a list of users who have retweeted a particular tweet, GET statuses/retweeters/ids.
In your code the Twitter4J API you're using, getRetweets(), does not return the IDs of users who retweeted.
Looking at the Twitter4J Twitter4J API docs we find getRetweeterIds(statusId) that returns the list of user IDs that retweed a particular tweet indicated by statusId.
i am using this code i get an error TwitterException 429
trying to fecth the tweets.
i want to fecth whole tweets of twitter account from starting.
how to solve twitter rate limit issue.
int limitRateCounter=0;
int countOfTweets=0; int numberOfTweets = 3500; long lastID = Long.MAX_VALUE; ArrayList<Status> status = new ArrayList<Status>();while (status.size () < numberOfTweets) { try {.out.print("\nlimit counter = "+limitRateCounter);.out.print("\t tweetsCounter = "+countOfTweets);
List<Status> listOfStatus=
twitter.getUserTimeline(tweeterHandle,pg);
/* making twitter request */
countOfTweets=countOfTweets+listOfStatus.size();
status.addAll(listOfStatus);
limitRateCounter++;
// println("Gathered " + tweets.size() + " tweets");
for (Status t: status)
if(t.getId() < lastID) lastID = t.getId();
}
catch (TwitterException te) {
System.out.println("Couldn't connect: " + te);
//twitter=getTwitterDetails2();
break;
};
pg.setMaxId(lastID-1); /* add pagging max id */
}
after 120 request .getUserTimeline(tweeterHandle,pg); methode not fetching
new tweets after some time get exception.
Twitter4j uses Twitter API to access Twitter data. This API has limits in the number of invocations as is specified here:
API Rate Limits
Rate Limits Charts
You seem to be facing this problem right now. Basically your code needs to wait before making another call until you are again inside the API rate limits. Take into account that the rate limit is per "access token" as the documentation specifies, so you could increase the number of calls your code can make if you provide it with more access tokens, but in the end you will have a (bigger) limit.
create several accounts on twitter (10) then put access tokens in array
I want to extract all the tweets of the week using twitter4j lib in java. I tried doing it like below
try {
Query query = new Query("since:2013-04-01&until:2013-04-08");
QueryResult result;
do {
result = twitter.search(query);
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
System.out.println("#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
}
} while ((query = result.nextQuery()) != null);
System.exit(0);
But, I ended up with an error message
403:The request is understood, but it has been refused. An accompanying error message will explain why. This code is used when requests are being denied due to update limits (https://support.twitter.com/articles/15364-about-twitter-limits-update-api-dm-and-following).
message - Missing or invalid url parameter
code - 195
So, how do i get the tweets by giving dates as parameters? Thanks
I think the below snippet should answer your question (Date in YYYY-MM-DD format)
Query query = new Query("#sad");
query.lang("en");
query.setSince("2006-01-01");
query.setUntil("2013-12-28");
QueryResult result = twitter.search(query);