I need to implement a search over multiple entities (which works) and i need to implement pagination over that search (which doesn't work).
So what i do is:
//creates the initial BooleanJunction from the first QueryBuilder
boolJunc = allQueryBuilders.get(0).bool();
for (QueryBuilder buildeForCurrentEntity: allQueryBuilders) {
boolJunc.should(buildeForCurrentEntity.keyword().onFields(searchedFields).matching(searchText).createQuery());
}
//after all BolleanJunctions are set, creates the final lucene query
org.apache.lucene.search.Query mainLuceneQuery = boolJunc.createQuery();
FullTextQuery jpaQuery = fullTextSession.createFullTextQuery(mainLuceneQuery, classes); //classes is array of all classes used to make those QueryBuilder's
//set pagination parameters
jpaQuery.setFirstResult(firstResultIndex);
jpaQuery.setMaxResults(maxNumberOfResults);
List<?> result = jpaQuery.list();
For a search term that finds 10 results:
if i set my paging parameters as firstResultIndex = 0 and maxNumberOfResults = 50, then it works normally
if its firstResultIndex = 0 and maxNumberOfResults = 5 then it dosn't return anything
if its firstResultIndex = 4 and maxNumberOfResults = 50 then it ignores firstResultIndex and just returns all 10 results
Is there something i am missing? How do i make pagination work in my case?
Related
I'm developing an application in Quarkus that integrates with the DynamoDB database. I have a query method that returns a list and I'd like this list to be paginated, but it would have to be done manually by passing the parameters.
I chose to use DynamoDBMapper because it gives more possibilities to work with lists of objects and the level of complexity is lower.
Does anyone have any idea how to do this pagination manually in the function?
DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
.withLimit(pageSize)
.withExclusiveStartKey(paginationToken);
PaginatedScanList<YourModel> result = mapper.scan(YourModel.class, scanExpression);
String nextPaginationToken = result.getLastEvaluatedKey();
You can pass the pageSize and paginationToken as parameters to your query method. The nextPaginationToken can be returned along with the results, to be used for the next page.
DynamoDB Mapper paginates by iterating over the results, by lazily loading the dataset:
By default, the scan method returns a "lazy-loaded" collection. It initially returns only one page of results, and then makes a service call for the next page if needed. To obtain all the matching items, iterate over the result collection.
Ref
For example:
List<Customer> result = mapper.scan(Customer.class, scanExpression);
for ( Customer cust : result ) {
System.out.println(cust.getId());
}
To Scan manually page by page you can use ScanPage
final DynamoDBScanExpression scanPageExpression = new DynamoDBScanExpression()
.withLimit(limit);
do {
ScanResultPage<MyClass> scanPage = mapper.scanPage(MyClass.class, scanPageExpression);
scanPage.getResults().forEach(System.out::println);
System.out.println("LastEvaluatedKey=" + scanPage.getLastEvaluatedKey());
scanPageExpression.setExclusiveStartKey(scanPage.getLastEvaluatedKey());
} while (scanPageExpression.getExclusiveStartKey() != null);
Ref
Ref
I am having a bit of trouble with projecting and paging with mongooperations. I always get empty result. The query criteria working fine without aggregation. I tryed the same code without paging(skip&limit) and without sort but I still get the empty result.
My code:
public List<ProfileBasic> findAllActiveUsersByGenderAndAgeBetweenProjectedPage(Gender gender, int fromAge, int toAge, int pageSize, int page) {
Criteria criteria = Criteria.where("gender").is(gender).and("confirmed").is(true)
.and("dateOfBirth").lte(LocalDate.now().minusYears(fromAge))
.gte(LocalDate.now().minusYears(toAge));
MatchOperation match = Aggregation.match(criteria);
ProjectionOperation project = Aggregation.project()
.and("id").as("id")
.and("name").as("name")
.and("lastName").as("lastName")
.and("gender").as("gender")
.and("dateOfBirth").as("dateOfBirth")
.and("lastVisit").as("lastVisit");
SkipOperation skip = new SkipOperation(pageSize*(page-1));
LimitOperation limit = new LimitOperation(pageSize);
SortOperation sort = new SortOperation(new Sort(Sort.Direction.DESC, "lastVisit"));
Aggregation aggregate = Aggregation.newAggregation(project, match, skip, limit, sort);
return operations.aggregate(aggregate, User.class, ProfileBasic.class).getMappedResults();
}
I would appreciate any help.
When we do Aggregation in MongoDB data is processed in stages and the output of one stage is provided as input to the next stage.
In my code first stage was project and not match(query):
Aggregation aggregate = Aggregation.newAggregation(project, match, skip, limit, sort);
Thats why it didn't work.
I changed it to:
Aggregation aggregate = Aggregation.newAggregation(match, skip, limit, project, sort);
And now it works fine.
The official Spring documentation provides the following example (slightly simplified here) for grouping results from a Solr query:
Field field = new SimpleField("popularity");
Query query = new SimpleQuery("inStock:true");
SimpleQuery groupQuery = new SimpleQuery(new SimpleStringCriteria("*:*"));
GroupOptions groupOptions = new GroupOptions()
.addGroupByField(field)
.addGroupByQuery(query);
groupQuery.setGroupOptions(groupOptions);
GroupPage<Product> page = solrTemplate.queryForGroupPage("collection-1", query, Product.class);
However, when I try this, it ignores the search conditions (inStock:true) and just performs the grouping on all results ("*:*")
There are several things about this example I had to modify to get it working. First of all, as you may have noticed, the argument query should actually be groupQuery in the queryForGroupPage call. I also removed the Query and put the actual search conditions in groupQuery. The working version is as follows, where groupByField is a SimpleField:
SimpleQuery groupQuery = new SimpleQuery(conditions);
GroupOptions groupOptions = new GroupOptions()
.addGroupByField(groupByField);
groupQuery.setGroupOptions(groupOptions);
return solrTemplate.queryForGroupPage(groupQuery, YourObjectHere.class);
private ElasticsearchTemplate elasticsearchTemplate = return new ElasticsearchTemplate(NodeBuilder.nodeBuilder().local(true).node().client());
FilterBuilder[] fbs = {
FilterBuilders.geoDistanceFilter("location").point(LAT, LON).distance(10.0, DistanceUnit.KILOMETERS).optimizeBbox("memory").ge oDistance(GeoDistance.ARC)
};
FilterBuilder fb = FilterBuilders.andFilter(fbs);
FilteredQueryBuilder builder = QueryBuilders.filteredQuery(QueryBuilders.termsQuery("interests", INTERESTS).minimumShouldMatch("2"), fb);
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(builder).build();
List<ESClass> results = this.elasticsearchTemplate.queryForList(searchQuery, ESClass.class);
System.out.println("Count: " + results.size());
The above code written in the Java Spring framework with elasticsearch classes, keeps returning 0 results even though that is not the case. For example, if I do the query without the termsQuery on "interests", it will return correct results for the geo distance filter. But if i do only termsQuery on "interests" (without geo distance filter), it also returns correct results that have atleast two matching interests (the results from both are exactly the same). So my question is, if I combine both of them why am i getting 0 results? And yes, I have checked it SHOULD NOT be zero.
I am trying to limit the document size returned by my query.i want lets say 10 documents back only,any my query normally displays 22,how would i go buy setting a limit for the returned output. i am aware i can just limit the list size by creating a list and adding to that list however i want to do it on the query level.
My Query: Thanks in advance :)
ueryBuilder raceGenderQuery = QueryBuilders.boolQuery()
.must(termQuery("lep_etg_desc", "indian"))
.must(termQuery("lep_gen_desc", "male"));
Set<String> suburbanLocationSet = new HashSet<String>();
suburbanLocationSet.add("queensburgh");
suburbanLocationSet.add("umhlanga");
suburbanLocationSet.add("tongaat");
suburbanLocationSet.add("phoenix");
suburbanLocationSet.add("shallcross");
suburbanLocationSet.add("balito");
//Build the necessary location query.
QueryBuilder locationQuery = QueryBuilders.boolQuery().must(termsQuery("lep_suburb_home", suburbanLocationSet));
//Combine all Queries so that its filtered to get exact results.
FilteredQueryBuilder finalSearchQuery = QueryBuilders.filteredQuery(QueryBuilders.boolQuery().must(raceGenderQuery).must(locationQuery), FilterBuilders.boolFilter().must(FilterBuilders.rangeFilter("lep_age").gte(25).lte(45)).must(FilterBuilders.rangeFilter("lep_max_income").gte(25000).lte(45000)));
//Run Query through elasticsearch iterating through documents in the traceps index for query matches.
List<Leads> finalLeadsList = new ArrayList<Leads>();
for (Leads leads : this.leadsRepository.search(finalSearchQuery)) {
finalLeadsList.add(leads);
}
I think this is what you want:
SearchResponse response = client.prepareSearch().setSearchType(SearchType.QUERY_THEN_FETCH).setSize(10).setQuery(finalSearchQuery).execute().get
You have to use QUERY_THEN_FETCH for it to return exactly size results because otherwise it gets size results from each shard.