Develop Reference

How to do pagination with DynamoDBMapper?

I'm developing an application in Quarkus that integrates with the DynamoDB database. I have a query method that returns a list and I'd like this list to be paginated, but it would have to be done manually by passing the parameters.
I chose to use DynamoDBMapper because it gives more possibilities to work with lists of objects and the level of complexity is lower.
Does anyone have any idea how to do this pagination manually in the function?

DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
.withLimit(pageSize)
.withExclusiveStartKey(paginationToken);
PaginatedScanList<YourModel> result = mapper.scan(YourModel.class, scanExpression);
String nextPaginationToken = result.getLastEvaluatedKey();
You can pass the pageSize and paginationToken as parameters to your query method. The nextPaginationToken can be returned along with the results, to be used for the next page.

DynamoDB Mapper paginates by iterating over the results, by lazily loading the dataset:
By default, the scan method returns a "lazy-loaded" collection. It initially returns only one page of results, and then makes a service call for the next page if needed. To obtain all the matching items, iterate over the result collection.
Ref
For example:
List<Customer> result = mapper.scan(Customer.class, scanExpression);
for ( Customer cust : result ) {
System.out.println(cust.getId());
}
To Scan manually page by page you can use ScanPage
final DynamoDBScanExpression scanPageExpression = new DynamoDBScanExpression()
.withLimit(limit);
do {
ScanResultPage<MyClass> scanPage = mapper.scanPage(MyClass.class, scanPageExpression);
scanPage.getResults().forEach(System.out::println);
System.out.println("LastEvaluatedKey=" + scanPage.getLastEvaluatedKey());
scanPageExpression.setExclusiveStartKey(scanPage.getLastEvaluatedKey());
} while (scanPageExpression.getExclusiveStartKey() != null);
Ref
Ref

Update field of a document inside an array of arrays in mongodb

I have a Document that has an array field called listsOfItems. Its contents are Documents and they all contain another array field called setOfItems, which is composed of Document.
So the structure is like listsOfItems(Array of Documents) > setOfItems(Array of Documents) > Items(Documents)
Each Document in listsOfItems has a listId field which is used to uniquely identify it's contents.
Each Document in setOfItems has a itemId field which is used to uniquely identify it's contents.
I want to change the value of a field inside a Document of setOfItems given the listId and itemId.
Currently, I'm trying to do it like this:
final var filter = Filters.and(Filters.eq("accessDetail.email", email),
Filters.eq("listsOfItems.listId", listId),
Filters.eq("listsOfItems.$.setOfItems.itemId", itemId));
final var updateQuery = Updates.set("listsOfItems.$.setOfItems.obtained", BsonBoolean.TRUE);
usersCollection.findOneAndUpdate(filter, updateQuery)
The query doesn't match any Document
I cannot access the elements in the setOfItems by directly accessing them by index because they are ordered by other indexes and not according to their itemId field.
What is the right query for performing the update given the condition?
EDIT1
I have replaced my filter to:
final var filter = Filters.and(Filters.eq("accessDetail.email", email),
Filters.elemMatch("listsOfItems",
Filters.eq("listId", listId)),
Filters.elemMatch("listsOfItems.setOfItems",
Filters.eq("itemId", itemId)));
Now I get the error message: 'Cannot create field 'obtained' with the correct sub-document that it should make changes in.
The issue boils down to the updateQuery.

To do this, you need to make use of arrayFilters.
Document filter = new Document("base-doc-field","<value>");//query for base doc
//fields if any
Document updateDoc = new Document("$inc",
new Document("listOfItems.$[elem].setOfItems.$[elem1].fieldName","value to
update"));
//prepare array filters.
List<Bson> arrayFilters = new ArrayList<>();
arrayFilters.add(new Document("elem.listId","<list_id>"));
arrayFilters.add(new Document("elem1.itemId","<item_id>"));
//prepare update options, and set these array filters to update options.
UpdateOptions options = new UpdateOptions().
arrayFilters(filterArr).bypassDocumentValidation(true);
//execute update method.
collection.updateOne(filter,updateDoc,options);
Visit https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/#update-nested-arrays-in-conjunction-with

Mongo spring query where two fields are equal

I want to execute a query in java where path and _id are two fields of the mongo document.
I want to get results list where these two fields are equal in the document.
I have tried using the following query.But could not retrieve the results properly.Received empty list which is not the case.
List<Metadata> MetadataList= ops.find(new Query(Criteria.where("path").is("_id")), Metadata.class);
How to get results where two field values are equal in mongo.

What you are looking for is the $where operator in MongoDB. Standard query operations do not compare the values of one field against another. In order to do this, you need to employ the JavaScript evaluation server side which can actually compare the two field values:
BasicQuery query = new BasicQuery(
new BasicDBObject("$where", "return this._id == this.path")
);
<Metadata> MetadataList = ops.find(query, Metadata.class);
Or you can do the same thing with native operators through the $redact pipeline stage available to the aggregation framework.
Pretty sure there is no $redact support in spring mongo as yet, but you can wrap the aggregation operation with a class to do so:
public class CustomAggregationOperation implements AggregationOperation {
private DBObject operation;
public CustomAggregattionOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
And use it like this:
Aggregation aggregation = newAggregation(
new CustomAggregationOperation(
new BasicDBObject(
"$redact",
new BasicDBObject("$cond",
new BasicDBObject()
.append("if", new BasicDBObject(
"$eq", Arrays.asList("$_id", "$path")
))
.append("then", "$$KEEP")
.append("else", "$$PRUNE")
)
)
)
);
AggregationResults<Metadata> results = ops.aggregate(
(TypedAggregation<Metadata>) aggregation, Metadata.class);
So basic MongoDB query operations do not compare field values against each other. To do this you need to follow one of the methods here.

You can use BasicDBObject to add condition.
Try something
BasicDBObject query = new BasicDBObject("path", new BasicDBObject("$eq", "_id");
collection.find(query);
Please refer the below link for more information
http://mongodb.github.io/mongo-java-driver/2.13/getting-started/quick-tour/

Check for duplicate records while BulkWriteOperation into mongo using hadoop reducer

I am using hadoop map-reduce for processing XML file. I am directly storing the JSON data into mongodb. How can I achieve that only non-duplicate records will be stored into database before executing BulkWriteOperation?
The duplicate records criteria will be based on product image and product name, I do not want to use a layer of morphia where we can assign indexes to the class members.
Here is my reducer class:
public class XMLReducer extends Reducer<Text, MapWritable, Text, NullWritable>{
private static final Logger LOGGER = Logger.getLogger(XMLReducer.class);
protected void reduce(Text key, Iterable<MapWritable> values, Context ctx) throws IOException, InterruptedException{
LOGGER.info("reduce()------Start for key>"+key);
Map<String,String> insertProductInfo = new HashMap<String,String>();
try{
MongoClient mongoClient = new MongoClient("localhost", 27017);
DB db = mongoClient.getDB("test");
BulkWriteOperation operation = db.getCollection("product").initializeOrderedBulkOperation();
for (MapWritable entry : values) {
for (Entry<Writable, Writable> extractProductInfo : entry.entrySet()) {
insertProductInfo.put(extractProductInfo.getKey().toString(), extractProductInfo.getValue().toString());
}
if(!insertProductInfo.isEmpty()){
BasicDBObject basicDBObject = new BasicDBObject(insertProductInfo);
operation.insert(basicDBObject);
}
}
//How can I check for duplicates before executing bulk operation
operation.execute();
LOGGER.info("reduce------end for key"+key);
}catch(Exception e){
LOGGER.error("General Exception in XMLReducer",e);
}
}
}
EDIT: After the suggested answer I have added :
BasicDBObject query = new BasicDBObject("product_image", basicDBObject.get("product_image"))
.append("product_name", basicDBObject.get("product_name"));
operation.find(query).upsert().updateOne(new BasicDBObject("$setOnInsert", basicDBObject));
operation.insert(basicDBObject);
I am getting error like: com.mongodb.MongoInternalException: no mapping found for index 0
Any help will be useful.Thanks.

I suppose it all depends on what you really want to do with the "duplicates" here as to how you handle it.
For one you can always use .initializeUnOrderedBulkOperation() which won't "error" on a duplicate key from your index ( which you need to stop duplicates ) but will report any such errors in the returned BulkWriteResult object. Which is returned from .execute()
BulkWriteResult result = operation.execute();
On the other hand, you can just use "upserts" instead and use operators such as $setOnInsert to only make changes where no duplicate existed:
BasicDBObject basicdbobject = new BasicDBObject(insertProductInfo);
BasicDBObject query = new BasicDBObject("key", basicdbobject.get("key"));
operation.find(query).upsert().updateOne(new BasicDBObject("$setOnInsert", basicdbobject));
So you basically look up the value of the field that holds the "key" to determine a duplicate with a query, then only actually change any data where that "key" was not found and therefore a new document and "inserted".
In either case the default behaviour here will be to "insert" the first unique "key" value and then ignore all other occurances. If you want to do other things like "overwrite" or "increment" values where the same key is found then the .update() "upsert" approach is the one you want, but you will use other update operators for those actions.

Passing more than one $Unwind object from java driver

The mongo java driver takes var args for aggregate method, I have an API in which $unwind objects get's created dynamically and its number is not fixed. how can I pass it through Mongo Java driver aggregate method, as it needs each object to be passed separately. I tried passing putting all the $unwind object in a BasicDBList and pass, but it fails. Can someone help me with some work around?
example:
db.foo.aggregate({$unwind:items},{$unwind:item2})
, but these unwind may vary as it is getting created at runtime.

you don't need to create a BasicDBList. This is how it works:
List<DBObject> unwindItems = new ArrayList<>();
if(<item2 is not null>){ //pseudo code
DBObject unwindItem1 = new BasicDBObject("$unwind", "$item1");
unwindItems.add(unwindItem1);
}
if(<item2 is not null>){ //pseudo code
DBObject unwindItem2 = new BasicDBObject("$unwind", "$item2");
unwindItems.add(unwindItem2);
}
//add any other dbObject in the list, it need not be an unwind operation, it could be match, project, group etc.
DBObject command = new BasicDBObject("aggregate", "foo");
command.put("pipeline", dbObjects);