Develop Reference

How can I fetch more than 2000 tweets using twitter4j?

I am making a project which requires me to fetch tweets from a user's twitter handle and collect the tweets related to a particular hashtag ,eg-#IphoneX .
But I need a larger number of tweets ,(close to 2000 would be enough) and I am able to fetch a little more than a 100. I have used twitter4j .Can someone let me know how do I do this? This is the code that I used:-
public class TwitterScrapper {
/**
* #param args the command line arguments
* #throws twitter4j.TwitterException
*/
public static void main(String[] args) throws TwitterException {
ConfigurationBuilder cf = new ConfigurationBuilder();
cf.setDebugEnabled(true)
.setOAuthConsumerKey("-------------------")
.setOAuthConsumerSecret("----------------------")
.setOAuthAccessToken("------------------------")
.setOAuthAccessTokenSecret("-------------------");
TwitterFactory tf = new TwitterFactory(cf.build());
twitter4j.Twitter twitter = tf.getInstance();
try {
Query query = new Query("Iphone");
QueryResult result;
result = twitter.search(query);
List<Status> tweets = result.getTweets();
tweets.forEach((tweet) -> {
System.out.println("#" + tweet.getUser().getScreenName() + " - " + tweet.getText());
});
System.exit(0);
} catch (TwitterException te) {
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
}
}
}

how to disable page query in Spring-data-elasticsearch

I use spring-data-elasticsearch framework to get query result from elasticsearch server, the java code like this:
public void testQuery() {
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withFields("createDate","updateDate").withQuery(matchAllQuery()).withPageable(new PageRequest(0,Integer.MAX_VALUE)).build();
List<Entity> list = template.queryForList(searchQuery, Entity.class);
for (Entity e : list) {
System.out.println(e.getCreateDate());
System.out.println(e.getUpdateDate());
}
}
I get the raw query log in server, like this:
{"from":0,"size":10,"query":{"match_all":{}},"fields":["createDate","updateDate"]}
As per the query log, spring-data-elasticsearch will add size limit to the query. "from":0, "size":10, How can I avoid it to add the size limit?

You don't want to do this, you could use the findAll functionality on a repository that returns an Iterable. I think the best way to obtain all items is to use the scan/scroll functionality. Maybe the following code block can put you in the right direction:
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(QueryBuilders.matchAllQuery())
.withIndices("customer")
.withTypes("customermodel")
.withSearchType(SearchType.SCAN)
.withPageable(new PageRequest(0, NUM_ITEMS_PER_SCROLL))
.build();
String scrollId = elasticsearchTemplate.scan(searchQuery, SCROLL_TIME_IN_MILLIS, false);
boolean hasRecords = true;
while (hasRecords) {
Page<CustomerModel> page = elasticsearchTemplate.scroll(scrollId, SCROLL_TIME_IN_MILLIS, CustomerModel.class);
if (page != null) {
// DO something with the records
hasRecords = (page.getContent().size() == NUM_ITEMS_PER_SCROLL);
} else {
hasRecords = false;
}
}

Getting Tweets From Particular User From The Last Fetch Using MaxId

We are using twitter4j userTimeLine for getting tweets from particular user. How do I use maxId and How do I fetch tweets from the last fetch of tweets???
My source code as follows,
public List<Status> userTimeLine(String keyWord, int page, int count) {
log.info("Showing user timeline.");
List<Status> statuses = new ArrayList<Status>(0);
Paging paging = new Paging(page, count);
try {
statuses = twitter.getUserTimeline(keyWord, paging);
} catch (TwitterException e) {
log.error("Unable to find user timeline", e);
}
return statuses;
}
This code returns 100 tweets for the first fetch. In the second fetch, it retrieves 102 [100(last fetched tweets)+2 (new tweets)] if there new tweets posted by the user. Otherwise it returns the same 100 tweets for each and every fetch.
How do I solve getting tweets from the last fetch of tweets?

You can specify tweets (by status id) using Paging to get the tweets that were posted in between using the sinceId and maxId methods.
since_id: returns elements which id are bigger than the specified id
max_id: returns elements which id are smaller than the specified id
For example:
Paging paging = new Paging(1, 10).sinceId(258347905419730944L).maxId(258348815243960320L);
List<Status> statuses = twitter.getHomeTimeline(paging);
You can find lots of things here : , and also you can use sthg like that;
Query query = new Query("from:somebody").since("2011-01-02");
Twitter twitter = new TwitterFactory().getInstance();
QueryResult result = twitter.search(query);

Twitter4J: Get more than 1 tweet from a user

I'm trying to get a list of recent statuses from each user on a persons list of followers. I've got the following to get the users...
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()){
twitter4j.User TW_user = twitter.showUser(ID);
}
All I can get from this is getStatus() which is their most recent status. getHomeTimeline() is also insufficient as I need a list of recent tweets from each user. Is there anyway I can achieve this using Twitter4J?

I was just trying to find this answer myself. I had decent success using the getUserTimeline method. Looks like you're trying to look up a list of friend IDs, so this method below should take the long[] and spit out all the user statuses. lookupUsers also accepts a String[] of screen names if you want to look users up that way instead.
public static void lookupUsers(long[] usersList) {
try {
Twitter twitter = new TwitterFactory().getInstance();
ResponseList<User> users = twitter.lookupUsers(usersList);
Paging paging = new Paging(1, 100);
List<Status> statuses;
for (User user : users) {
statuses = twitter.getUserTimeline(user.getScreenName(), paging);
System.out.println("\nUser: #" + user.getScreenName());
for (Status s : statuses) {
System.out.println(s.getText());
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
}

Alex's answer is close, but will only get you 100 tweets per user. The following will get you all (or at least the API's max limit):
IDs list = twitter.getFriendsIDs(0);
for(long ID : list.getIDs()) {
Status[] tweets = getAllTweets(twitter, ID);
System.out.println(ID + ": " + tweets.length);
}
Status[] getAllTweets(Twitter twitter, long userId)
{
int pageno = 1;
List statuses = new ArrayList();
while (true)
{
try
{
int size = statuses.size();
Paging page = new Paging(pageno++, 100);
statuses.addAll(twitter.getUserTimeline(userId, page));
if (statuses.size() == size)
break;
}
catch (TwitterException e)
{
e.printStackTrace();
}
}
return (Status[]) statuses.toArray(new Status[0]);
}

How to search tweets in real time using a recursive fashion?

I have used to search a tweets using search method with passing a keyword in twitter4j. This is my code
String query = "Cricket";
Query searchQuery = new Query(query);
try {
QueryResult queryResult = twitter.search(searchQuery);
} catch (TwitterException e) {
log.error("Unable to search query = {}", query, e);
}
When i test this code, its showing only top 20 tweets. But i need to search a tweets in recursive fashion. So, how can i search a tweets in real time using a recursive fashion?

public static void main(String[] args) throws TwitterException {
Twitter twitter = new TwitterFactory().getInstance();
Query query = new Query("**your query**");
query.setRpp(100); // here you show 100 tweets
QueryResult result = twitter.search(query);
for (Tweet tweet : result.getTweets()) {
System.out.println(tweet.getFromUser() + ":" + tweet.getText());
}
}
Why do you want to use recursive function?