Develop Reference

Trying to get more than 20 followers to display, how can i loop this to give me more? [duplicate]

I have been wondering if there is a way to access all the twitter followers list.
We have tried using call to the REST API via twitter4j:
public List<User> getFriendList() {
List<User> friendList = null;
try {
friendList = mTwitter.getFollowersList(mTwitter.getId(), -1);
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (TwitterException e) {
e.printStackTrace();
}
return friendList;
}
But it returns only a list of 20 followers.
I tried using the same call in loop, but it cause a rate limit exception - says we are not allowed to make too many requests in a small interval of time.
Do we have a way around this?

You should definitely use getFollowersIDs. As the documentation says, this returns an array (list) of IDs objects. Note that it causes the list to be broken into pages of around 5000 IDs at a time. To begin paging provide a value of -1 as the cursor. The response from the API will include a previous_cursor and next_cursor to allow paging back and forth.
The tricky part is to handle the cursor. If you can do this, then you will not have the problem of getting only 20 followers.
The first call to getFollowersIDs will need to be given a cursor of -1. For subsequent calls, you need to update the cursor value, by getting the next cursor, as done in the while part of the loop.
long cursor =-1L;
IDs ids;
do {
ids = twitter.getFollowersIDs(cursor);
for(long userID : ids.getIDs()){
friendList.add(userID);
}
} while((cursor = ids.getNextCursor())!=0 );
Here is a very good reference:
https://github.com/yusuke/twitter4j/blob/master/twitter4j-examples/src/main/java/twitter4j/examples/friendsandfollowers/GetFriendsIDs.java
Now, if the user has more than around 75000 followers, you will have to do some waiting (see Vishal's answer).
The first 15 calls will yield you around 75000 IDs. Then you will have to sleep for 15 minutes. Then make another 15 calls, and so on till you get all the followers. This can be done using a simple Thread.sleep(time_in_milliseconds) outside the for loop.

Just Change like this and try, this is working for me
try {
Log.i("act twitter...........", "ModifiedCustomTabBarActivity.class");
// final JSONArray twitterFriendsIDsJsonArray = new JSONArray();
IDs ids = mTwitter.mTwitter.getFriendsIDs(-1);// ids
// for (long id : ids.getIDs()) {
do {
for (long id : ids.getIDs()) {
String ID = "followers ID #" + id;
String[] firstname = ID.split("#");
String first_Name = firstname[0];
String Id = firstname[1];
Log.i("split...........", first_Name + Id);
String Name = mTwitter.mTwitter.showUser(id).getName();
String screenname = mTwitter.mTwitter.showUser(id).getScreenName();
// Log.i("id.......", "followers ID #" + id);
// Log.i("Name..", mTwitter.mTwitter.showUser(id).getName());
// Log.i("Screen_Name...", mTwitter.mTwitter.showUser(id).getScreenName());
// Log.i("image...", mTwitter.mTwitter.showUser(id).getProfileImageURL());
}
} while (ids.hasNext());
} catch (Exception e) {
e.printStackTrace();
}

Try This...
ConfigurationBuilder confbuilder = new ConfigurationBuilder();
confbuilder.setOAuthAccessToken(accessToken)
.setOAuthAccessTokenSecret(secretToken)
.setOAuthConsumerKey(TwitterOAuthActivity.CONSUMER_KEY)
.setOAuthConsumerSecret(TwitterOAuthActivity.CONSUMER_SECRET);
Twitter twitter = new TwitterFactory(confbuilder.build()).getInstance();
PagableResponseList<User> followersList;
ArrayList<String> list = new ArrayList<String>();
try
{
followersList = twitter.getFollowersList(screenName, cursor);
for (int i = 0; i < followersList.size(); i++)
{
User user = followersList.get(i);
String name = user.getName();
list.add(name);
System.out.println("Name" + i + ":" + name);
}
listView.setAdapter(new ArrayAdapter<String>(this, android.R.layout.simple_list_item_1 , list));
listView.setVisibility(View.VISIBLE);
friend_list.setVisibility(View.INVISIBLE);
post_feeds.setVisibility(View.INVISIBLE);
twit.setVisibility(View.INVISIBLE);
}

This is a tricky one.
You should specify whether you're using application or per user tokens and the number of users you're fetching followers_ids for.
You get just 15 calls per 15 minutes in case of an application token. You can fetch a maximum of 5000 followers_ids per call. That gives you a maximum of 75K followers_ids per 15 minutes.
If any of the users you're fetching followers_ids for has over 75K followers, you'll get the rate_limit error immediately. If you're fetching for more than 1 user, you'll need to build strong rate_limit handling in your code with sleeps and be very patient.
The same applies for friends_ids.
I've not had to deal with fetching more than 75K followers/friends for a given user but come to think of it, I don't know if it's even possible anymore.

Creating a new object

At the moment my program has a Jpanel with about 15 different inputs. These inputs are created to one long string called "searchInput".
"searchInput" is then put into a query named which then gets metadata about the tweet result one at a time, such as createdAt, user and text. These results are printed to a textArea named tweetsResult.
createdAt, user and text are only some of the metadata behind each tweet, more being favouriteCount and retweetCount.
Because Twitter has a maximum amount of Tweets it can deliver to you using the API, I feel it would be best if I created a new class which uses searchInput, and then get the desired amount of Tweets with all of the metadata into an object. And then from the main class, I would only call the results with the specific meta data that I would like.
Is this the right way to do it? If so how would I create this new object, presumably using getters and setters?
I hope this is clear to you, and many thanks for the help!
List<Status> tweets = new ArrayList<Status>();
//Standard Twitter and Twitter4j authentication
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.setOAuthConsumerKey(consumerKey);
builder.setOAuthConsumerSecret(consumerSecret);
Configuration configuration = builder.build();
TwitterFactory twitterFactory = new TwitterFactory(configuration);
Twitter twitter = twitterFactory.getInstance();
twitter.setOAuthAccessToken(new AccessToken(accessToken, accessTokenSecret));
//The query is split up into 5 sections
//All the text boxes are read in the seperate methods
String searchWordInput = getWordSearchQuery();
String searchPeopleInput = getPeopleSearchQuery();
String filtersSearchInput = getFiltersSearchQuery();
String dateAndLocationSearchInput = getDateAndLocationSearchQuery();
String radiusAndLocationSearchInput = getRadiusAndLocationSearchQuery();
String searchInput = searchWordInput + searchPeopleInput + filtersSearchInput + dateAndLocationSearchInput + radiusAndLocationSearchInput;
//One big search String called "searchInput" is taken to be sent to Twitter
//Increases maximum amount of tweets in one search
int wantedTweets = 100; //CHANGE THIS FOR DIFFERENT AMOUNT OF TWEETS
long lastSearchID = Long.MAX_VALUE;
int remainingTweets = wantedTweets;
Query query = new Query(searchInput);
tweetsResultInput.append(searchInput);
try {
while (remainingTweets > 0) {
remainingTweets = wantedTweets - tweets.size();
if (remainingTweets > 100) {
query.count(100);
} else {
query.count(remainingTweets);
}
QueryResult result = twitter.search(query);
tweets.addAll(result.getTweets());
Status s = tweets.get(tweets.size() - 1);
lastSearchID = s.getId();
query.setMaxId(lastSearchID);
remainingTweets = wantedTweets - tweets.size();
}
//Increases maximum amount of tweets in ones search
tweetsResult.append(searchInput);
tweetsResult.append("\n");
//For the query tweets, using "searchInput", get every tweet in the format; when Tweet was sent, who it was sent by, and the Tweets text and write to tweetsResults
for (Status tweet : tweets) {
tweetsResult.append(tweet.getCreatedAt() + ":\t#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
tweetsResult.append("\n");
}
} catch (TwitterException te) {
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}

Avoiding duplicate entries in a mongoDB with Java and JSON objects

I´m developing an analyzing program for Twitter Data.
I´m using mongoDB and at the moment. I try to write a Java program to get tweets from the Twitter API and put them in the database.
Getting the Tweets already works very well, but I have a problem when I want to put them in the database. As the Twitter API often returns just the same Tweets, I have to place some kind of index in the database.
First of all, I connect to the database and get the collection related to the search-term, or create this collection if this doesn´t exist.
public void connectdb(String keyword)
{
try {
// on constructor load initialize MongoDB and load collection
initMongoDB();
items = db.getCollection(keyword);
BasicDBObject index = new BasicDBObject("tweet_ID", 1);
items.ensureIndex(index);
} catch (MongoException ex) {
System.out.println("MongoException :" + ex.getMessage());
}
}
Then I get the tweets and put them in the database:
public void getTweetByQuery(boolean loadRecords, String keyword) {
if (cb != null) {
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
try {
Query query = new Query(keyword);
query.setCount(50);
QueryResult result;
result = twitter.search(query);
System.out.println("Getting Tweets...");
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
BasicDBObject basicObj = new BasicDBObject();
basicObj.put("user_name", tweet.getUser().getScreenName());
basicObj.put("retweet_count", tweet.getRetweetCount());
basicObj.put("tweet_followers_count", tweet.getUser().getFollowersCount());
UserMentionEntity[] mentioned = tweet.getUserMentionEntities();
basicObj.put("tweet_mentioned_count", mentioned.length);
basicObj.put("tweet_ID", tweet.getId());
basicObj.put("tweet_text", tweet.getText());
if (mentioned.length > 0) {
// System.out.println("Mentioned length " + mentioned.length + " Mentioned: " + mentioned[0].getName());
}
try {
items.insert(basicObj);
} catch (Exception e) {
System.out.println("MongoDB Connection Error : " + e.getMessage());
loadMenu();
}
}
// Printing fetched records from DB.
if (loadRecords) {
getTweetsRecords();
}
} catch (TwitterException te) {
System.out.println("te.getErrorCode() " + te.getErrorCode());
System.out.println("te.getExceptionCode() " + te.getExceptionCode());
System.out.println("te.getStatusCode() " + te.getStatusCode());
if (te.getStatusCode() == 401) {
System.out.println("Twitter Error : \nAuthentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect.\nEnsure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.");
} else {
System.out.println("Twitter Error : " + te.getMessage());
}
loadMenu();
}
} else {
System.out.println("MongoDB is not Connected! Please check mongoDB intance running..");
}
}
But as I mentioned before, there are often the same tweets, and they have duplicates in the database.
I think the tweet_ID field is a good field for an index and should be unique in the collection.

Set the unique option on your index to have MongoDb enforce uniqueness:
items.ensureIndex(index, new BasicDBObject("unique", true));
Note that you'll need to manually drop the existing index and remove all duplicates or you won't be able to create the unique index.

This question is already answered but I would like to contribute a bit since MongoDB API 2.11 offers a method which receives unique option as a parameter:
public void ensureIndex(DBObject keys, String name, boolean unique)
A minor remind to someone who would like to store json documents on MongoDBNote is that uniqueness must be applied to a BasicObject key and not over values. For example:
BasicDBObject basicObj = new BasicDBObject();
basicObj.put("user_name", tweet.getUser().getScreenName());
basicObj.put("retweet_count", tweet.getRetweetCount());
basicObj.put("tweet_ID", tweet.getId());
basicObj.put("tweet_text", tweet.getText());
basicObj.put("a_json_text", "{"info_details":{"info_id":"1234"},"info_date":{"year":"2012"}, {"month":"12"}, {"day":"10"}}");
On this case, you can create unique index only to basic object keys:
BasicDBObject index = new BasicDBObject();
int directionOrder = 1;
index.put("tweet_ID", directionOrder);
boolean isUnique = true;
items.ensureIndex(index, "unique_tweet_ID", isUnique);
Any index regarding JSON value like "info_id" would not work since it´s not a BasicObject key.
Using indexes on MongDB is not as easy as it sounds. You may also check MongoDB docs for more details here Mongo Indexing Tutorials and Mongo Index Concepts. Direction order might be pretty important to understand once you need a composed index which is well explained here Why Direction order matter.

Twitter4J: Get more than 1 tweet from a user

I'm trying to get a list of recent statuses from each user on a persons list of followers. I've got the following to get the users...
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()){
twitter4j.User TW_user = twitter.showUser(ID);
}
All I can get from this is getStatus() which is their most recent status. getHomeTimeline() is also insufficient as I need a list of recent tweets from each user. Is there anyway I can achieve this using Twitter4J?

I was just trying to find this answer myself. I had decent success using the getUserTimeline method. Looks like you're trying to look up a list of friend IDs, so this method below should take the long[] and spit out all the user statuses. lookupUsers also accepts a String[] of screen names if you want to look users up that way instead.
public static void lookupUsers(long[] usersList) {
try {
Twitter twitter = new TwitterFactory().getInstance();
ResponseList<User> users = twitter.lookupUsers(usersList);
Paging paging = new Paging(1, 100);
List<Status> statuses;
for (User user : users) {
statuses = twitter.getUserTimeline(user.getScreenName(), paging);
System.out.println("\nUser: #" + user.getScreenName());
for (Status s : statuses) {
System.out.println(s.getText());
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
}

Alex's answer is close, but will only get you 100 tweets per user. The following will get you all (or at least the API's max limit):
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()) {
Status[] tweets = getAllTweets(twitter, ID);
System.out.println(ID + ": " + tweets.length);
}
Status[] getAllTweets(Twitter twitter, long userId)
{
int pageno = 1;
List statuses = new ArrayList();
while (true)
{
try
{
int size = statuses.size();
Paging page = new Paging(pageno++, 100);
statuses.addAll(twitter.getUserTimeline(userId, page));
if (statuses.size() == size)
break;
}
catch (TwitterException e)
{
e.printStackTrace();
}
}
return (Status[]) statuses.toArray(new Status[0]);
}

How to search tweets in real time using a recursive fashion?

I have used to search a tweets using search method with passing a keyword in twitter4j. This is my code
String query = "Cricket";
Query searchQuery = new Query(query);
try {
QueryResult queryResult = twitter.search(searchQuery);
} catch (TwitterException e) {
log.error("Unable to search query = {}", query, e);
}
When i test this code, its showing only top 20 tweets. But i need to search a tweets in recursive fashion. So, how can i search a tweets in real time using a recursive fashion?

public static void main(String[] args) throws TwitterException {
Twitter twitter = new TwitterFactory().getInstance();
Query query = new Query("**your query**");
query.setRpp(100); // here you show 100 tweets
QueryResult result = twitter.search(query);
for (Tweet tweet : result.getTweets()) {
System.out.println(tweet.getFromUser() + ":" + tweet.getText());
}
}
Why do you want to use recursive function?