I want to update an item in DynamoDB only if the new item has a more recent date than the existing item. Currently, I'm querying for the existing item, doing the comparison in my code, and then writing to db. I was wondering if there was a way to have DynamoDB do the checking for me. I've looked into using Expected, but its comparison operators need to take in a parameter, which defeats the purpose since it means having to query for the existing item anyway.
I'm working with Java 8.
The ConditionExpression can be used to check the condition and update the item if the condition is satisfied. This is similar to the WHERE condition in the SQL statement. The differences are:-
1) DynamoDB requires both Partition key and Range key to update the item. The non key attribute conditions can be given in the ConditionExpression
2) DynamoDB can update only one item at a time.
ConditionExpression — (String) A condition that must be satisfied in
order for a conditional update to succeed.
Expected — (map) This is a legacy parameter, for backward
compatibility. New applications should use ConditionExpression
instead. Do not combine legacy parameters and expression parameters in
a single API call; otherwise, DynamoDB will return a
ValidationException exception.
Sample code:-
The below code updates the item only if existing value "createdate" attribute is less than the new value (i.e. in other words new value is greater than the existing value in the table).
UpdateItemSpec updateItemSpec = new UpdateItemSpec().withPrimaryKey("yearkey", yearKey, "title", title)
.withReturnValues(ReturnValue.UPDATED_NEW).withUpdateExpression("set #createdate = :val1")
.withNameMap(new NameMap().with("#createdate", "createdate"))
.withValueMap(new ValueMap().withString(":val1", createDate))
.withConditionExpression("createdate < :val1");
UpdateItemOutcome outcome = null;
try {
outcome = table.updateItem(updateItemSpec);
} catch (ConditionalCheckFailedException ce) {
System.out.println("Conditional check failed." + ce.getMessage());
return false;
}
return true;
Related
I am using Dynamodb Item - getItem API to get records from DynamoDB table. But it returns Item object and I want to retrieve specific attribute value from the Item object. How can we do it in Java? I couldn't find references.
Table table = dynamoDB.getTable(tableName);
Item item = table.getItem(hashKeyFieldName, hashKeyFieldValue);
The item contains the following fields:
HashKey, TimeStamp, NumRetries
I want to get the specific NumRetries value from item above. Is it something that is possible? something like int numRetries = item.get("NumRetries");?
You should be able to do that with a Projection Expression:
GetItemSpec spec = new GetItemSpec().withPrimaryKey("primaryKey", primaryKey)
.withProjectionExpression("HashKey, TimeStamp, NumRetries");
Item outcome = table.getItem(spec);
A names map may be necessary.
You can use Projection Expressions to get certain attributes from an item but do keep in mind that using projection expressions does not reduce the usage and cost of RCUs that are used in retrieving the object.
Code example,
GetItemSpec spec = new GetItemSpec()
.withPrimaryKey("YourPrimaryKey", value)
.withProjectionExpression("NumRetries");
Item item = table.getItem(spec);
System.out.println(item.toJSONPretty());
More code examples can be found here.
I have a list of objects which I want to insert into a collection. The mongoTemplate.insert(list); works fine but now I want to modify it to upsert(); as my list can contain duplicate objects which are already inserted into a collection. So what I want is insert entire list and on the go check if the item is already present in the collection then skip it else insert it.
You can try out continueOnError or ordered flag like this:
db.collection.insert(myArray, {continueOnError: true})
OR,
db.collection.insert(myArray, {ordered: false})
You need to create a unique index field of your object's id(if there is no unique constraint). So that it will make error while you try to insert using same id.
Using the unique constraint you insert array or using BulkInsert
For using insert you can set a flag continueOnError: true which can continue insertion whenever error found in case of error because of unique constraint while inserting existing id of object.
The only way to do a bulk-upsert operation is the method MongoCollection.bulkWrite (or at least: the only way I know... ;-))
To use it, you have to convert your documents to the appropriate WriteModel: for upserts on a per-document basis, this is UpdateOneModel.
List<Document> toUpdate = ...;
MongoCollection coll = ...;
// Convert Document to UpdateOneModel<Document>
List<UpdateOneModel<Document>> bulkOperationList = toUpdate.stream()
.map(doc -> new UpdateOneModel<Document>(
Filters.eq("_id", doc.get("_id")), // identify by same _id
doc,
new UpdateOptions().upsert(true)))
.collect(Collectors.toList());
// Write to DB
coll.bulkWrite(bulkOperationList);
(Disclaimer: I only typed this code, I never ran it)
I'm using MongoDB 3.2 and want to avoid the duplicates in my collection. In order to do that I use createIndex() method (I tried different variants, none of them doesn't work):
dbColl.createIndex(new Document("guid", 1));
dbColl.createIndex(new BasicDBObject("guid", 1));
dbColl.createIndex(new Document("guid.content", 1));
dbColl.createIndex(new BasicDBObject("guid.content", 1));
Then I try to execute data insert with:
itemsArr.forEach(
item -> dbColl.insertOne(Document.parse(item.toString()))
);
I do it two times and anticipate that the second time MongoDB will not add any new row since the data has been already added and there is an index on the guid field. But that's not the case MongoDB adds duplicates despite index value.
Why does MongoDB add duplicates even if there is an index on a guid and/or guid.content field? And how to fix it? I want to be able to add the document with the same guid field only one time.
Here is a sample of documents structure:
In my data the guid field is a unique document identifier.
Regular indexes allow multiple documents with the same value.
What you need is not a regular index but an unique index. These are created by using the method createIndex(DBObject keys, DBObject options) with an options-object where unique is true.
collection.createIndex(new BasicDBObject("guid", 1), new BasicDBObject("unique", true));
With the help of Phillip, I composed a completely worked solution for the problem «How to avoid duplicates / skip duplicates on insert» in MongoDB 3.2 for Java Driver 3.2.0:
IndexOptions options = new IndexOptions();
// ensure the index is unique
options.unique(true);
// define the index
dbColl.createIndex(new BasicDBObject("guid", 1), options);
// add data to DB
for (Object item : itemsArr) {
// if there is a duplicate, skip it and write to a console (optionally)
try {
dbColl.insertOne(Document.parse(item.toString()));
} catch (com.mongodb.MongoWriteException ex) {
//System.err.println(ex.getMessage());
}
}
Feel free to use this ready-to-use solution.
EDIT:
I was actually incorrect. I was querying the table when I meant to query an index which explains my error. Vikdor's solution is a valid one though.
ORIGINAL:
I have a table with a Hash-Range key schema in DynamoDB. I need to be able to get all items associated with a specific hash key but it seems to require a range key condition. My issue is I want EVERY range key but there is no wildcard option. As of right now my range key is a string and the only way I could think to do this is by querying all range keys greater or equal to the smallest ascii characters I can use since the documentation says it sorts based on ascii character values.
I looked into scanning but it appears that simply will read the entire table which is NOT an option.
Is there any better way to query for all values of a hash key or can anyone confirm that using the method with the ascii character will work?
but it seems to require a range key condition.
This doesn't sound to be true.
I use DynamoDBMapper and use DynamoDBQueryExpression to query all the records with a given HashKey as follows:
DynamoDBQueryExpression<DomainObject> query =
new DynamoDBQueryExpression<DomainObject>();
DomainObject hashKeyValues = new DomainObject();
hashKeyValues.setHashKey(hashKeyValue);
query.setHashKeyValues(hashKeyValues);
// getMapper() returns a DynamoDBMapper object with the appropriate
// AmazonDynamoDBClient object.
List<DomainObject> results = getMapper().query(query);
HTH.
You can use DynamoDB's query API, which allows you to query the database based conditional expressions using the hash/range keys. You can see examples of the API here. Here is a relevant example:
ItemCollection<QueryOutcome> items = table.query("theHashFieldName", "theHashFieldToQuery");
You can also query using more complex expressions. E.g.:
DynamoDB dynamoDB = new DynamoDB(
new AmazonDynamoDBClient(new ProfileCredentialsProvider()));
Table table = dynamoDB.getTable("TableName");
QuerySpec spec = new QuerySpec()
.withKeyConditionExpression("Id = :v_id")
.withValueMap(new ValueMap()
.withString(":v_id", "TheId"));
ItemCollection<QueryOutcome> items = table.query(spec);
Iterator<Item> iterator = items.iterator();
Item item = null;
while (iterator.hasNext()) {
item = iterator.next();
System.out.println(item.toJSONPretty());
}
I am using DynamoDBMapper for a class, let's say "User" (username being the primary key) which has a field on it which says "Status". It is a Hash+Range key table, and everytime a user's status changes (changes are extremely infrequent), we add a new entry to the table alongwith the timestamp (which is the range key). To fetch the current status, this is what I am doing:
DynamoDBQueryExpression expr =
new DynamoDBQueryExpression(new AttributeValue().withS(userName))
.withScanIndexForward(false).withLimit(1);
PaginatedQueryList<User> result =
this.getMapper().query(User.class, expr);
if(result == null || result.size() == 0) {
return null;
}
for(final User user : result) {
System.out.println(user.getStatus());
}
This for some reason, is printing all the statuses a user has had till now. I have set scanIndexForward to false so that it is in descending order and I put limit of 1. I am expecting this to return the latest single entry in the table for that username.
However, when I even look into the wire logs of the same, I see a huge amount of entries being returned, much more than 1. For now, I am using:
final String currentStatus = result.get(0).getStatus();
What I am trying to understand here is, what is whole point of the withLimit clause in this case, or am I doing something wrong?
In March 2013 on the AWS forums a user complained about the same problem.
A representative from Amazon sent him to use the queryPage function.
It seems as if the limit is not preserved for elements but rather a limit on chunk of elements retrieved in a single API call, and the queryPage might help.
You could also look into the pagination loading strategy configuration
Also, you can always open a Github issue for the team.