I have used to search a tweets using search method with passing a keyword in twitter4j. This is my code
String query = "Cricket";
Query searchQuery = new Query(query);
try {
QueryResult queryResult = twitter.search(searchQuery);
} catch (TwitterException e) {
log.error("Unable to search query = {}", query, e);
}
When i test this code, its showing only top 20 tweets. But i need to search a tweets in recursive fashion. So, how can i search a tweets in real time using a recursive fashion?
public static void main(String[] args) throws TwitterException {
Twitter twitter = new TwitterFactory().getInstance();
Query query = new Query("**your query**");
query.setRpp(100); // here you show 100 tweets
QueryResult result = twitter.search(query);
for (Tweet tweet : result.getTweets()) {
System.out.println(tweet.getFromUser() + ":" + tweet.getText());
}
}
Why do you want to use recursive function?
Related
I am making a project which requires me to fetch tweets from a user's twitter handle and collect the tweets related to a particular hashtag ,eg-#IphoneX .
But I need a larger number of tweets ,(close to 2000 would be enough) and I am able to fetch a little more than a 100. I have used twitter4j .Can someone let me know how do I do this? This is the code that I used:-
public class TwitterScrapper {
/**
* #param args the command line arguments
* #throws twitter4j.TwitterException
*/
public static void main(String[] args) throws TwitterException {
ConfigurationBuilder cf = new ConfigurationBuilder();
cf.setDebugEnabled(true)
.setOAuthConsumerKey("-------------------")
.setOAuthConsumerSecret("----------------------")
.setOAuthAccessToken("------------------------")
.setOAuthAccessTokenSecret("-------------------");
TwitterFactory tf = new TwitterFactory(cf.build());
twitter4j.Twitter twitter = tf.getInstance();
try {
Query query = new Query("Iphone");
QueryResult result;
result = twitter.search(query);
List<Status> tweets = result.getTweets();
tweets.forEach((tweet) -> {
System.out.println("#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
});
System.exit(0);
} catch (TwitterException te) {
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
}
I am trying to search an expression in Twitter, my code:
public static List<Status> searchQuery(Twitter twitter, String search)
throws TwitterException, IOException {
Query query = new Query(search);
query.setCount(100);
query.setSince("2015-05-25");
QueryResult result;
List<Status> tweets = null;
do {
System.out.println("Write to File ...");
result = twitter.search(query);
List<Status> newTweets = result.getTweets();
if (tweets == null) {
tweets = newTweets;
} else {
tweets.addAll(newTweets);
}
WriteToFile.writeTweetsToFile(newTweets);
} while ((query = result.nextQuery()) != null);
return tweets;
}
But it's just return tweets of last month, when I'm using query.setUntil("2015-06-25"); nothing returned. What is the problem?
I have developed a code to search for tweets wihtout this time restriction, you can look at GitHub - GetOldTweets. It simple to use and you can search the deepest tweets.
We are using twitter4j userTimeLine for getting tweets from particular user. How do I use maxId and How do I fetch tweets from the last fetch of tweets???
My source code as follows,
public List<Status> userTimeLine(String keyWord, int page, int count) {
log.info("Showing user timeline.");
List<Status> statuses = new ArrayList<Status>(0);
Paging paging = new Paging(page, count);
try {
statuses = twitter.getUserTimeline(keyWord, paging);
} catch (TwitterException e) {
log.error("Unable to find user timeline", e);
}
return statuses;
}
This code returns 100 tweets for the first fetch. In the second fetch, it retrieves 102 [100(last fetched tweets)+2 (new tweets)] if there new tweets posted by the user. Otherwise it returns the same 100 tweets for each and every fetch.
How do I solve getting tweets from the last fetch of tweets?
You can specify tweets (by status id) using Paging to get the tweets that were posted in between using the sinceId and maxId methods.
since_id: returns elements which id are bigger than the specified id
max_id: returns elements which id are smaller than the specified id
For example:
Paging paging = new Paging(1, 10).sinceId(258347905419730944L).maxId(258348815243960320L);
List<Status> statuses = twitter.getHomeTimeline(paging);
You can find lots of things here : , and also you can use sthg like that;
Query query = new Query("from:somebody").since("2011-01-02");
Twitter twitter = new TwitterFactory().getInstance();
QueryResult result = twitter.search(query);
I´m developing an analyzing program for Twitter Data.
I´m using mongoDB and at the moment. I try to write a Java program to get tweets from the Twitter API and put them in the database.
Getting the Tweets already works very well, but I have a problem when I want to put them in the database. As the Twitter API often returns just the same Tweets, I have to place some kind of index in the database.
First of all, I connect to the database and get the collection related to the search-term, or create this collection if this doesn´t exist.
public void connectdb(String keyword)
{
try {
// on constructor load initialize MongoDB and load collection
initMongoDB();
items = db.getCollection(keyword);
BasicDBObject index = new BasicDBObject("tweet_ID", 1);
items.ensureIndex(index);
} catch (MongoException ex) {
System.out.println("MongoException :" + ex.getMessage());
}
}
Then I get the tweets and put them in the database:
public void getTweetByQuery(boolean loadRecords, String keyword) {
if (cb != null) {
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
try {
Query query = new Query(keyword);
query.setCount(50);
QueryResult result;
result = twitter.search(query);
System.out.println("Getting Tweets...");
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
BasicDBObject basicObj = new BasicDBObject();
basicObj.put("user_name", tweet.getUser().getScreenName());
basicObj.put("retweet_count", tweet.getRetweetCount());
basicObj.put("tweet_followers_count", tweet.getUser().getFollowersCount());
UserMentionEntity[] mentioned = tweet.getUserMentionEntities();
basicObj.put("tweet_mentioned_count", mentioned.length);
basicObj.put("tweet_ID", tweet.getId());
basicObj.put("tweet_text", tweet.getText());
if (mentioned.length > 0) {
// System.out.println("Mentioned length " + mentioned.length + " Mentioned: " + mentioned[0].getName());
}
try {
items.insert(basicObj);
} catch (Exception e) {
System.out.println("MongoDB Connection Error : " + e.getMessage());
loadMenu();
}
}
// Printing fetched records from DB.
if (loadRecords) {
getTweetsRecords();
}
} catch (TwitterException te) {
System.out.println("te.getErrorCode() " + te.getErrorCode());
System.out.println("te.getExceptionCode() " + te.getExceptionCode());
System.out.println("te.getStatusCode() " + te.getStatusCode());
if (te.getStatusCode() == 401) {
System.out.println("Twitter Error : \nAuthentication credentials (https://dev.twitter.com/pages/auth) were missing or incorrect.\nEnsure that you have set valid consumer key/secret, access token/secret, and the system clock is in sync.");
} else {
System.out.println("Twitter Error : " + te.getMessage());
}
loadMenu();
}
} else {
System.out.println("MongoDB is not Connected! Please check mongoDB intance running..");
}
}
But as I mentioned before, there are often the same tweets, and they have duplicates in the database.
I think the tweet_ID field is a good field for an index and should be unique in the collection.
Set the unique option on your index to have MongoDb enforce uniqueness:
items.ensureIndex(index, new BasicDBObject("unique", true));
Note that you'll need to manually drop the existing index and remove all duplicates or you won't be able to create the unique index.
This question is already answered but I would like to contribute a bit since MongoDB API 2.11 offers a method which receives unique option as a parameter:
public void ensureIndex(DBObject keys, String name, boolean unique)
A minor remind to someone who would like to store json documents on MongoDBNote is that uniqueness must be applied to a BasicObject key and not over values. For example:
BasicDBObject basicObj = new BasicDBObject();
basicObj.put("user_name", tweet.getUser().getScreenName());
basicObj.put("retweet_count", tweet.getRetweetCount());
basicObj.put("tweet_ID", tweet.getId());
basicObj.put("tweet_text", tweet.getText());
basicObj.put("a_json_text", "{"info_details":{"info_id":"1234"},"info_date":{"year":"2012"}, {"month":"12"}, {"day":"10"}}");
On this case, you can create unique index only to basic object keys:
BasicDBObject index = new BasicDBObject();
int directionOrder = 1;
index.put("tweet_ID", directionOrder);
boolean isUnique = true;
items.ensureIndex(index, "unique_tweet_ID", isUnique);
Any index regarding JSON value like "info_id" would not work since it´s not a BasicObject key.
Using indexes on MongDB is not as easy as it sounds. You may also check MongoDB docs for more details here Mongo Indexing Tutorials and Mongo Index Concepts. Direction order might be pretty important to understand once you need a composed index which is well explained here Why Direction order matter.
I'm trying to get a list of recent statuses from each user on a persons list of followers. I've got the following to get the users...
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()){
twitter4j.User TW_user = twitter.showUser(ID);
}
All I can get from this is getStatus() which is their most recent status. getHomeTimeline() is also insufficient as I need a list of recent tweets from each user. Is there anyway I can achieve this using Twitter4J?
I was just trying to find this answer myself. I had decent success using the getUserTimeline method. Looks like you're trying to look up a list of friend IDs, so this method below should take the long[] and spit out all the user statuses. lookupUsers also accepts a String[] of screen names if you want to look users up that way instead.
public static void lookupUsers(long[] usersList) {
try {
Twitter twitter = new TwitterFactory().getInstance();
ResponseList<User> users = twitter.lookupUsers(usersList);
Paging paging = new Paging(1, 100);
List<Status> statuses;
for (User user : users) {
statuses = twitter.getUserTimeline(user.getScreenName(), paging);
System.out.println("\nUser: #" + user.getScreenName());
for (Status s : statuses) {
System.out.println(s.getText());
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
}
Alex's answer is close, but will only get you 100 tweets per user. The following will get you all (or at least the API's max limit):
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()) {
Status[] tweets = getAllTweets(twitter, ID);
System.out.println(ID + ": " + tweets.length);
}
Status[] getAllTweets(Twitter twitter, long userId)
{
int pageno = 1;
List statuses = new ArrayList();
while (true)
{
try
{
int size = statuses.size();
Paging page = new Paging(pageno++, 100);
statuses.addAll(twitter.getUserTimeline(userId, page));
if (statuses.size() == size)
break;
}
catch (TwitterException e)
{
e.printStackTrace();
}
}
return (Status[]) statuses.toArray(new Status[0]);
}