I'm developing an application in Quarkus that integrates with the DynamoDB database. I have a query method that returns a list and I'd like this list to be paginated, but it would have to be done manually by passing the parameters.
I chose to use DynamoDBMapper because it gives more possibilities to work with lists of objects and the level of complexity is lower.
Does anyone have any idea how to do this pagination manually in the function?
DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
.withLimit(pageSize)
.withExclusiveStartKey(paginationToken);
PaginatedScanList<YourModel> result = mapper.scan(YourModel.class, scanExpression);
String nextPaginationToken = result.getLastEvaluatedKey();
You can pass the pageSize and paginationToken as parameters to your query method. The nextPaginationToken can be returned along with the results, to be used for the next page.
DynamoDB Mapper paginates by iterating over the results, by lazily loading the dataset:
By default, the scan method returns a "lazy-loaded" collection. It initially returns only one page of results, and then makes a service call for the next page if needed. To obtain all the matching items, iterate over the result collection.
Ref
For example:
List<Customer> result = mapper.scan(Customer.class, scanExpression);
for ( Customer cust : result ) {
System.out.println(cust.getId());
}
To Scan manually page by page you can use ScanPage
final DynamoDBScanExpression scanPageExpression = new DynamoDBScanExpression()
.withLimit(limit);
do {
ScanResultPage<MyClass> scanPage = mapper.scanPage(MyClass.class, scanPageExpression);
scanPage.getResults().forEach(System.out::println);
System.out.println("LastEvaluatedKey=" + scanPage.getLastEvaluatedKey());
scanPageExpression.setExclusiveStartKey(scanPage.getLastEvaluatedKey());
} while (scanPageExpression.getExclusiveStartKey() != null);
Ref
Ref
Related
I need to create a mongotemplate database query to get a specific number of elements into a list.
At the moment I just get all the elements with findAll(), and then I modify the obtained data using code that I have writen within the service class.
Initially, I have a Laptop class with fields price::BigDecimal and name::String and I use findAll() to get a list of them.
Then I put those in a HashMap, where key is the name field, sorted from most expensive to cheapest.
Map<String, List<Laptop>> laptopsMap = laptopsFrom.stream()
.collect(Collectors.groupingBy(Laptop::getName,
Collectors.collectingAndThen(Collectors.toList(),
l -> l.stream()
.sorted(Comparator.comparing(Laptop::getPrice).reversed())
.collect(Collectors.toList())
))
);
So the results are like below:
[{"MSI", [2200, 1100, 900]},
{"HP", [3200, 900, 800]},
{"Dell", [2500, 2000, 700]}]
Then, I use the code in the bottom of the question, to create a Laptop list with the following contents:
[{"HP", 3200}, {"Dell", 2500}, {"MSI", 2200},
{"Dell", 2000}, {"MSI", 1100}, {"HP", 900},
{"MSI", 900}, {"HP", 800}, {"Dell", 700}]
So basically, I iterate the map and from each key, I extract the next in line element of the list.
do {
for (Map.Entry<String, List<Laptop>> entry :
laptopsMap.entrySet()) {
String key = entry.getKey();
List<Laptop> value = entry.getValue();
finalResultsList.add(value.get(0));
value.remove(0);
if (value.size() == 0) {
laptopsMap.entrySet()
.removeIf(pr -> pr.getKey().equals(key));
} else {
laptopsMap.replace(key, value);
}
}
} while(!laptopsMap.isEmpty());
I instead of all this in-class code need to use a mongoTemplate database argument, but I cant seem to figure out how to create such a complex query. I have read material about Aggregation but I have not found anything helpful enough. At the moment, I have started putting a query together as shown below:
Query query = new Query();
query.limit(numOfLaptops);
query.addCriteria(Criteria.where(Laptop.PRICE).gte(minPrice));
The official Spring documentation provides the following example (slightly simplified here) for grouping results from a Solr query:
Field field = new SimpleField("popularity");
Query query = new SimpleQuery("inStock:true");
SimpleQuery groupQuery = new SimpleQuery(new SimpleStringCriteria("*:*"));
GroupOptions groupOptions = new GroupOptions()
.addGroupByField(field)
.addGroupByQuery(query);
groupQuery.setGroupOptions(groupOptions);
GroupPage<Product> page = solrTemplate.queryForGroupPage("collection-1", query, Product.class);
However, when I try this, it ignores the search conditions (inStock:true) and just performs the grouping on all results ("*:*")
There are several things about this example I had to modify to get it working. First of all, as you may have noticed, the argument query should actually be groupQuery in the queryForGroupPage call. I also removed the Query and put the actual search conditions in groupQuery. The working version is as follows, where groupByField is a SimpleField:
SimpleQuery groupQuery = new SimpleQuery(conditions);
GroupOptions groupOptions = new GroupOptions()
.addGroupByField(groupByField);
groupQuery.setGroupOptions(groupOptions);
return solrTemplate.queryForGroupPage(groupQuery, YourObjectHere.class);
I need to implement a Query operation on DynamoDB. Right now I'm doing it by giving the HashKey and then filtering out the results according to my conditions on non-key attributes.
This is what I'm doing :
MusicData hashKey = new MusicData();
hashKey.setID(singer);
DynamoDBQueryExpression<MusicData> queryExpression = new DynamoDBQueryExpression<MusicData>().withHashKeyValues(hashKey);
List<MusicData> queryResult = mapper.query(MusicData.class, queryExpression);
for (MusicData musicData : queryResult) {
if( my condtions ) {
do something;
}
}
What I'm trying to do is to be able to do something like this :
MusicData hashKey = new MusicData();
hashKey.setID(singer);
hashKey.setAlbum(sampleAlbum);
hashKey.setSinger(duration);
DynamoDBQueryExpression<MusicData> queryExpression = new DynamoDBQueryExpression<MusicData>().withHashKeyValues(hashKey);
List<MusicData> queryResult = mapper.query(MusicData.class, queryExpression);
for (MusicData musicData : queryResult) {
if( my condtions ) {
do something;
}
}
And get results already filtered out. Is there a way to do this in DynamoDB?
Yes, you can ask DynamoDB to perform filtering of queries before it returns results. However, you will still incur the 'cost' of reading these items even though they are not returned to your client. This is still a good practice as it will eliminate unnecessary transfer of items over the network.
To do this you will call additional methods on your DynamoDBQueryExpression object, specifically withFilterExpression and addExpressionAttributeNamesEntry / addExpressionAttributeValuesEntry to complete the expression.
Without the specific example of what type of conditions you want to apply it is hard to give an example, but depending on how simple your condition you want to apply is you could contain it in just the withFilterExpression method.
DynamoDBQueryExpression<MusicData> queryExpression = new DynamoDBQueryExpression<MusicData>().withHashKeyValues(hashKey).withFilterExpression("foo > 10");
I am trying to limit the document size returned by my query.i want lets say 10 documents back only,any my query normally displays 22,how would i go buy setting a limit for the returned output. i am aware i can just limit the list size by creating a list and adding to that list however i want to do it on the query level.
My Query: Thanks in advance :)
ueryBuilder raceGenderQuery = QueryBuilders.boolQuery()
.must(termQuery("lep_etg_desc", "indian"))
.must(termQuery("lep_gen_desc", "male"));
Set<String> suburbanLocationSet = new HashSet<String>();
suburbanLocationSet.add("queensburgh");
suburbanLocationSet.add("umhlanga");
suburbanLocationSet.add("tongaat");
suburbanLocationSet.add("phoenix");
suburbanLocationSet.add("shallcross");
suburbanLocationSet.add("balito");
//Build the necessary location query.
QueryBuilder locationQuery = QueryBuilders.boolQuery().must(termsQuery("lep_suburb_home", suburbanLocationSet));
//Combine all Queries so that its filtered to get exact results.
FilteredQueryBuilder finalSearchQuery = QueryBuilders.filteredQuery(QueryBuilders.boolQuery().must(raceGenderQuery).must(locationQuery), FilterBuilders.boolFilter().must(FilterBuilders.rangeFilter("lep_age").gte(25).lte(45)).must(FilterBuilders.rangeFilter("lep_max_income").gte(25000).lte(45000)));
//Run Query through elasticsearch iterating through documents in the traceps index for query matches.
List<Leads> finalLeadsList = new ArrayList<Leads>();
for (Leads leads : this.leadsRepository.search(finalSearchQuery)) {
finalLeadsList.add(leads);
}
I think this is what you want:
SearchResponse response = client.prepareSearch().setSearchType(SearchType.QUERY_THEN_FETCH).setSize(10).setQuery(finalSearchQuery).execute().get
You have to use QUERY_THEN_FETCH for it to return exactly size results because otherwise it gets size results from each shard.
I am running map reduce in mongoDB with morphia, this is my java code
String map = "function() { emit(this.id,this.cal.charge)}";
String reduce = "function(k, v) {var i, sum = 0;for (i in v) {sum += v[i];}return sum;}";
MapreduceResults<Results> mrRes = ds.mapReduce(MapreduceType.MERGE,ds.createQuery(MyTable.class).field("id").equal(5),map,reduce,null,null,Re.class);
This work fine and put results to 'Re' collection, but how can i get result as objects or list without inserting to a new Collection ?
Thanks
I couldn't commented this because it violates the length limit.
If it is not going to be too much of a fuss, you can do it by using java driver directly without using the morphia interface. just get the mongo object from morphia datastore and use java driver's map reduce command; it is something like this:
DBObject queryObject = new BasicDBObject("id", 5);
DBCollection collection = ds.getCollection(MyTable.class);
MapReduceCommand mrc = new MapReduceCommand(collection, // collection to do map-reduce on
map, // map function
reduce, // reduce function
null, // output collection for the result
MapReduceCommand.OutputType.INLINE, // output result type
queryObject); // query to use in map reduce function
btw morphia has a newer version in github https://github.com/jmkgreen/morphia maybe you wanna check that out too. I saw that the newer version also don't support inline operation on map-reduce.