How can I fetch more than 2000 tweets using twitter4j? - java

I am making a project which requires me to fetch tweets from a user's twitter handle and collect the tweets related to a particular hashtag ,eg-#IphoneX .
But I need a larger number of tweets ,(close to 2000 would be enough) and I am able to fetch a little more than a 100. I have used twitter4j .Can someone let me know how do I do this? This is the code that I used:-
public class TwitterScrapper {
/**
* #param args the command line arguments
* #throws twitter4j.TwitterException
*/
public static void main(String[] args) throws TwitterException {
ConfigurationBuilder cf = new ConfigurationBuilder();
cf.setDebugEnabled(true)
.setOAuthConsumerKey("-------------------")
.setOAuthConsumerSecret("----------------------")
.setOAuthAccessToken("------------------------")
.setOAuthAccessTokenSecret("-------------------");
TwitterFactory tf = new TwitterFactory(cf.build());
twitter4j.Twitter twitter = tf.getInstance();
try {
Query query = new Query("Iphone");
QueryResult result;
result = twitter.search(query);
List<Status> tweets = result.getTweets();
tweets.forEach((tweet) -> {
System.out.println("#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
});
System.exit(0);
} catch (TwitterException te) {
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
}

Related

Creating a new object

At the moment my program has a Jpanel with about 15 different inputs. These inputs are created to one long string called "searchInput".
"searchInput" is then put into a query named which then gets metadata about the tweet result one at a time, such as createdAt, user and text. These results are printed to a textArea named tweetsResult.
createdAt, user and text are only some of the metadata behind each tweet, more being favouriteCount and retweetCount.
Because Twitter has a maximum amount of Tweets it can deliver to you using the API, I feel it would be best if I created a new class which uses searchInput, and then get the desired amount of Tweets with all of the metadata into an object. And then from the main class, I would only call the results with the specific meta data that I would like.
Is this the right way to do it? If so how would I create this new object, presumably using getters and setters?
I hope this is clear to you, and many thanks for the help!
List<Status> tweets = new ArrayList<Status>();
//Standard Twitter and Twitter4j authentication
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.setOAuthConsumerKey(consumerKey);
builder.setOAuthConsumerSecret(consumerSecret);
Configuration configuration = builder.build();
TwitterFactory twitterFactory = new TwitterFactory(configuration);
Twitter twitter = twitterFactory.getInstance();
twitter.setOAuthAccessToken(new AccessToken(accessToken, accessTokenSecret));
//The query is split up into 5 sections
//All the text boxes are read in the seperate methods
String searchWordInput = getWordSearchQuery();
String searchPeopleInput = getPeopleSearchQuery();
String filtersSearchInput = getFiltersSearchQuery();
String dateAndLocationSearchInput = getDateAndLocationSearchQuery();
String radiusAndLocationSearchInput = getRadiusAndLocationSearchQuery();
String searchInput = searchWordInput + searchPeopleInput + filtersSearchInput + dateAndLocationSearchInput + radiusAndLocationSearchInput;
//One big search String called "searchInput" is taken to be sent to Twitter
//Increases maximum amount of tweets in one search
int wantedTweets = 100; //CHANGE THIS FOR DIFFERENT AMOUNT OF TWEETS
long lastSearchID = Long.MAX_VALUE;
int remainingTweets = wantedTweets;
Query query = new Query(searchInput);
tweetsResultInput.append(searchInput);
try {
while (remainingTweets > 0) {
remainingTweets = wantedTweets - tweets.size();
if (remainingTweets > 100) {
query.count(100);
} else {
query.count(remainingTweets);
}
QueryResult result = twitter.search(query);
tweets.addAll(result.getTweets());
Status s = tweets.get(tweets.size() - 1);
lastSearchID = s.getId();
query.setMaxId(lastSearchID);
remainingTweets = wantedTweets - tweets.size();
}
//Increases maximum amount of tweets in ones search
tweetsResult.append(searchInput);
tweetsResult.append("\n");
//For the query tweets, using "searchInput", get every tweet in the format; when Tweet was sent, who it was sent by, and the Tweets text and write to tweetsResults
for (Status tweet : tweets) {
tweetsResult.append(tweet.getCreatedAt() + ":\t#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
tweetsResult.append("\n");
}
} catch (TwitterException te) {
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}

Trouble with twitter4j and processing

I'm trying to get processing to display the most recent tweet containing my keyword. I'm stuck. I'm not sure how I would check to see if it's updating. I know that it's able to pull the most recent tweet, but as of now, the only way I can view the most recent tweet is by restarting my program. Sorry it's messy.
import twitter4j.conf.*;
import twitter4j.*;
import twitter4j.auth.*;
import twitter4j.api.*;
import java.util.*;
Twitter twitter; // creating the twitter object
String searchString = "newborn"; // twitter search query
List<Status> tweets;// List of Status object to hold tweets
List<Status> tweets2;
int currentTweet;//
int nextTweet;
void setup ()
{
size(800,600);
ConfigurationBuilder cb = new ConfigurationBuilder(); // New config obj, auth.
cb.setOAuthConsumerKey("");
cb.setOAuthConsumerSecret("");
cb.setOAuthAccessToken("");
cb.setOAuthAccessTokenSecret("");
TwitterFactory tf = new TwitterFactory(cb.build()); // init. twit obj by retr inst from twit fact
twitter = tf.getInstance(); //here, twit gets isnt is the inst
getTweets2();
getNewTweets(); // func to get the tweets
currentTweet = 0;
nextTweet = 1;
//Status status = tweets.get(currentTweet);
thread("refreshTweets");
}
void draw()
{
background(0);
//currentTweet = currentTweet + 1;
//nextTweet = nextTweet + 1;
if (currentTweet >= tweets.size())
{
currentTweet = 0;
}
if (nextTweet >= tweets.size())
{
nextTweet = 1;
}
Status status = tweets.get(currentTweet); // retrieve current tweet from list of tweets
Status status2 = tweets.get(nextTweet);
Status status3 = tweets2.get(0);
Status status4 = status;
fill(200);
if(tweets.get(0).getCreatedAt() != tweets2.get(0).getCreatedAt()) {
//text(status.getText(),random(width),random(height), 300, 200);
text("current tweet:" + status.getCreatedAt() + ".....if" ,250,250, 300, 200);
//text("next tweet:" + status2.getCreatedAt(),250,350, 300, 200);
//text("10th tweet:" + status3.getCreatedAt(),250,450, 300, 200);
}else {
text("current tweet:" + status2.getCreatedAt() + ".....else",250,250, 300, 200);
}
delay(2500);
getNewTweets();
//System.out.println(status2.getText());
}
void getNewTweets()
{
try
{
//we try to get the tweetuses!
Query query = new Query(searchString);
QueryResult result = twitter.search(query);
tweets = result.getTweets();
}
catch (TwitterException te)
{
//tweetus aborted?
System.out.println("failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
void getTweets2() //this neat thing gets those tweets
{
try
{
//we try to get the tweetuses!
Query query = new Query(searchString);
QueryResult result = twitter.search(query);
tweets2 = result.getTweets();
}
catch (TwitterException te)
{
//tweetus aborted?
System.out.println("failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
void refreshTweets()
{
while(true)
{
getNewTweets();
println("Updated Tweets");
delay(5000);
}
}
Note: You've posted your secret tokens publicly to the internet, so you should change them as soon as possible! Just editing your question to not include them won't be enough, since they'll still be in the revision history.
You've got a lot of extra stuff going on in this program. You can simplify this a great deal. All you need to do is this:
Step 1: Write a function that gets a tweet (or an ArrayList of tweets). Keep track of that tweet in a sketch-level variable. You've already done this with the getNewTweets() function, but you can get rid of the other functions.
Step 2: Call that function at the beginning of the sketch. This is so you have something to show. Alternatively, you could just show some dummy text like "waiting to fetch tweets" until the next step.
Step 3: Every X seconds, call that function. You can do this by using the frameCount variable from the draw() function. Make sure you limit this so you don't go over twitter's rate limit.
Step 4: Use the sketch-level tweet variable to draw the current tweet however you want.
Putting it all together, it looks like this:
import twitter4j.conf.*;
import twitter4j.*;
import twitter4j.auth.*;
import twitter4j.api.*;
Twitter twitter;
String searchString = "cat";
Status currentTweet;
void setup ()
{
size(800, 600);
ConfigurationBuilder cb = new ConfigurationBuilder(); // New config obj, auth.
cb.setOAuthConsumerKey("");
cb.setOAuthConsumerSecret("");
cb.setOAuthAccessToken("");
cb.setOAuthAccessTokenSecret("");
TwitterFactory tf = new TwitterFactory(cb.build());
twitter = tf.getInstance();
getNewTweets();
}
void draw()
{
//get new tweet once every 10 seconds
if (frameCount % (60*10) == 0) {
getNewTweets();
}
background(0);
fill(200);
text("current tweet:" + currentTweet.getText(), 250, 250, 300, 200);
}
void getNewTweets()
{
println("Getting new tweet...");
try
{
Query query = new Query(searchString);
QueryResult result = twitter.search(query);
currentTweet = result.getTweets().get(0);
}
catch (TwitterException te)
{
te.printStackTrace();
}
}

Get Old Tweets on Twitter4j search

I am trying to search an expression in Twitter, my code:
public static List<Status> searchQuery(Twitter twitter, String search)
throws TwitterException, IOException {
Query query = new Query(search);
query.setCount(100);
query.setSince("2015-05-25");
QueryResult result;
List<Status> tweets = null;
do {
System.out.println("Write to File ...");
result = twitter.search(query);
List<Status> newTweets = result.getTweets();
if (tweets == null) {
tweets = newTweets;
} else {
tweets.addAll(newTweets);
}
WriteToFile.writeTweetsToFile(newTweets);
} while ((query = result.nextQuery()) != null);
return tweets;
}
But it's just return tweets of last month, when I'm using query.setUntil("2015-06-25"); nothing returned. What is the problem?
I have developed a code to search for tweets wihtout this time restriction, you can look at GitHub - GetOldTweets. It simple to use and you can search the deepest tweets.

Twitter4J: Get more than 1 tweet from a user

I'm trying to get a list of recent statuses from each user on a persons list of followers. I've got the following to get the users...
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()){
twitter4j.User TW_user = twitter.showUser(ID);
}
All I can get from this is getStatus() which is their most recent status. getHomeTimeline() is also insufficient as I need a list of recent tweets from each user. Is there anyway I can achieve this using Twitter4J?
I was just trying to find this answer myself. I had decent success using the getUserTimeline method. Looks like you're trying to look up a list of friend IDs, so this method below should take the long[] and spit out all the user statuses. lookupUsers also accepts a String[] of screen names if you want to look users up that way instead.
public static void lookupUsers(long[] usersList) {
try {
Twitter twitter = new TwitterFactory().getInstance();
ResponseList<User> users = twitter.lookupUsers(usersList);
Paging paging = new Paging(1, 100);
List<Status> statuses;
for (User user : users) {
statuses = twitter.getUserTimeline(user.getScreenName(), paging);
System.out.println("\nUser: #" + user.getScreenName());
for (Status s : statuses) {
System.out.println(s.getText());
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
}
Alex's answer is close, but will only get you 100 tweets per user. The following will get you all (or at least the API's max limit):
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()) {
Status[] tweets = getAllTweets(twitter, ID);
System.out.println(ID + ": " + tweets.length);
}
Status[] getAllTweets(Twitter twitter, long userId)
{
int pageno = 1;
List statuses = new ArrayList();
while (true)
{
try
{
int size = statuses.size();
Paging page = new Paging(pageno++, 100);
statuses.addAll(twitter.getUserTimeline(userId, page));
if (statuses.size() == size)
break;
}
catch (TwitterException e)
{
e.printStackTrace();
}
}
return (Status[]) statuses.toArray(new Status[0]);
}

How to search tweets in real time using a recursive fashion?

I have used to search a tweets using search method with passing a keyword in twitter4j. This is my code
String query = "Cricket";
Query searchQuery = new Query(query);
try {
QueryResult queryResult = twitter.search(searchQuery);
} catch (TwitterException e) {
log.error("Unable to search query = {}", query, e);
}
When i test this code, its showing only top 20 tweets. But i need to search a tweets in recursive fashion. So, how can i search a tweets in real time using a recursive fashion?
public static void main(String[] args) throws TwitterException {
Twitter twitter = new TwitterFactory().getInstance();
Query query = new Query("**your query**");
query.setRpp(100); // here you show 100 tweets
QueryResult result = twitter.search(query);
for (Tweet tweet : result.getTweets()) {
System.out.println(tweet.getFromUser() + ":" + tweet.getText());
}
}
Why do you want to use recursive function?

Categories

Resources