Setup
I have the following table in java and DynamoDB (DynamoDB Mapper) annotations:
#DynamoDBTable(tableName="myentity")
public class MyEntity {
#DynamoDBHashKey
#DynamoDBAutoGeneratedKey
private String id;
#DynamoDBIndexHashKey()
//#DynamoDBIndexRangeKey
private String userId;
private Date eventDate;
The id is randomly generated when saved and the same userId can occur for multiple entities.
The tables where defined via Web GUI like this:
Primary partition key: id (String)
Primary sort key: userId (String)
Issue
I would like to have all the entities for one userId with a query instead of a scan.
Query - does not work like this:
public List<MyEntity> findByUserIdQuery(String userId) {
Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
eav.put(":val1", new AttributeValue().withS(userId));
DynamoDBQueryExpression<MyEntity> queryExpression = new DynamoDBQueryExpression<MyEntity>()
.withKeyConditionExpression("userId = :val1").withExpressionAttributeValues(eav);
List<MyEntity> list = mapper.query(MyEntity.class, queryExpression);
of course I get:
com.amazonaws.AmazonServiceException: Query condition missed key schema element: id (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: 123) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305)
because I don't know the generated id when I want to read data.
Workaround
Instead I use:
public List<MyEntity> findByOwner(String userId){
Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
eav.put(":val1", new AttributeValue().withS(userId));
DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
.withFilterExpression("userId = :val1").withExpressionAttributeValues(eav);
List<MyEntity> list = mapper.scan(MyEntity.class, scanExpression);
Possible solutions
The issue is that I never know the randomly generated key for the elements I have. When I want to read I only know the userId and I want to have all entities for them.
I watched the examples here:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.QueryScanExample.html
...but they know always the key for what they are searching. They are using a combinded key like this:
String partitionKey = forumName + "#" + threadSubject;*
eav.put(":val1", new AttributeValue().withS(partitionKey))
Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
DynamoDBQueryExpression<Reply> queryExpression = new DynamoDBQueryExpression<Reply>()
.withKeyConditionExpression("Id = :val1 and ReplyDateTime > :val2").withExpressionAttributeValues(eav);
Question
Is there something like a wildcard for this combinded partition key that I could use? (Something like: "* # myGreatUser")
Other suggestions for db setup and other annotations as key element?
Maybe an Index where I could use query instead of scan?
Maybe somebody has a working example for this kind of use case?
You just need to switch your partition key and sort key around.
#DynamoDBTable(tableName="myentity")
public class MyEntity {
#DynamoDBRangeKey
#DynamoDBAutoGeneratedKey
private String id;
#DynamoDBHashKey
private String userId;
private Date eventDate;
And then query using
public List<MyEntity> findByUserIdQuery(String userId) {
Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
eav.put(":val1", new AttributeValue().withS(userId));
DynamoDBQueryExpression<MyEntity> queryExpression = new DynamoDBQueryExpression<MyEntity>()
.withKeyConditionExpression("userId = :val1").withExpressionAttributeValues(eav);
List<MyEntity> list = mapper.query(MyEntity.class, queryExpression);
By the way, you mentioned that you thought your sort key was userId, but in the example supplied you have a primary hash key of id and a global secondary index hash key of userid. There is no sort key.
Related
I have a table in dynamo db.
Below is the key for this table.
partitionKey - campaignId
sortKey - email
I have created a local secondary index for this table.Below is the key for local secondary index
partitionKey - campaignId
sortKey - subStatus
As per the dynamo db you should include primary keys of the table and the index (as key), with last evaluated values (as attribute value) when you setting ExclusiveStartKey.
Below is the code snippet of code.
public QueryResultPage<Prospect> getPaginatedProspectsByCampaignIdAndSubStatus(String campaignId, ProspectSubStatus subStatus, Integer limit, Map<String, AttributeValue> lastEvaluatedKey, String email) {
Prospect prospect = new Prospect();
prospect.setCampaignId(campaignId);
Map<String, AttributeValue> exclusiveStartKey = new HashMap<String, AttributeValue>();
if(!lastEvaluatedKey.isEmpty()) {
exclusiveStartKey.put("campaingId", lastEvaluatedKey.get("campaingId"));
exclusiveStartKey.put("subStatus", lastEvaluatedKey.get("subStatus"));
exclusiveStartKey.put("email", new AttributeValue().withS(email));
}
Condition rangeKeyCondition = new Condition()
.withComparisonOperator(ComparisonOperator.EQ)
.withAttributeValueList(new AttributeValue().withS(subStatus.toString()));
DynamoDBQueryExpression queryExpression = null;
if(exclusiveStartKey.isEmpty()) {
queryExpression = new DynamoDBQueryExpression<Prospect>()
.withHashKeyValues(prospect)
.withIndexName("subStatus-index")
.withRangeKeyCondition("subStatus", rangeKeyCondition)
.withConsistentRead(false).withLimit(limit);
} else {
queryExpression = new DynamoDBQueryExpression<Prospect>()
.withHashKeyValues(prospect)
.withIndexName("subStatus-index")
.withExclusiveStartKey(exclusiveStartKey)
.withRangeKeyCondition("subStatus", rangeKeyCondition)
.withConsistentRead(false).withLimit(limit);
}
QueryResultPage<Prospect> prospects = mapper.queryPage(Prospect.class, queryExpression);
return prospects;
}
But when run it, I am getting below error. Can someone please help me out to understand what I am doing wrong and how can I achieve the pagination in it.
I have next table:
#DynamoDBTable(tableName = "Table")
#Getter
#Setter
public class Table {
#DynamoDBHashKey
private String id;
#DynamoDBAttribute
private List<String> list;
}
I need to add values to list from Java. I tried next approach:
UpdateItemRequest request = new UpdateItemRequest()
.withTableName("Table")
.withKey(Collections.singletonMap("id", new AttributeValue().withS("123")))
.withUpdateExpression("SET #ri = list_append(:vals, #ri)")
.withExpressionAttributeNames(Collections.singletonMap("#ri","list"))
.withExpressionAttributeValues(Collections.singletonMap(":vals",
new AttributeValue().withSS("1","2","3")));
dynamoDB.updateItem(request);
But it fails without providing exact reason:
Unable to unmarshall exception response with the unmarshallers provided
I got two questions:
What is wrong with my request ?
Why I get unmarshall exception and how to fix it to get real reason ?
P.S. Other update requests work fine e.g. I can update id with same approach without problem.
Updated:
I've found a way to update values from Java
UpdateItemRequest request = new UpdateItemRequest()
.withTableName("Table")
.withKey(Collections.singletonMap("id", new AttributeValue().withS("123")))
.withUpdateExpression("SET #ri = list_append(:vals, #ri)")
.withExpressionAttributeNames(Collections.singletonMap("#ri","list"))
.withExpressionAttributeValues(Collections.singletonMap(":vals",
new AttributeValue().withL(new AttributeValue().withS("1"),AttributeValue().withS("2"),AttributeValue().withS("3"))));
dynamoDB.updateItem(request);
But it only allows to add one by one values in list, but I need to add several.
private void updateList(String id, String field, List<AttributeValue> values) {
UpdateItemRequest request = new UpdateItemRequest()
.withTableName("Table")
.withKey(Collections.singletonMap("id", new AttributeValue().withS(id)))
.withUpdateExpression("SET #field = list_append(:values, #field)")
.withExpressionAttributeNames(Collections.singletonMap("#field", field))
.withExpressionAttributeValues(Collections.singletonMap(":values",
new AttributeValue().withL(values)));
dynamoDB.updateItem(request);
}
And use next method to convert to list of AttributeValue
private List<AttributeValue> buildAttributeValuesList(List<String> values) {
return values
.stream()
.map(it -> new AttributeValue().withS(it))
.collect(Collectors.toList());
}
Creation of table:
final ArrayList<KeySchemaElement> indexKeySchema = new ArrayList<KeySchemaElement>();
indexKeySchema.add(new KeySchemaElement().withAttributeName("expiresOn").withKeyType(KeyType.HASH));
indexKeySchema.add(new KeySchemaElement().withAttributeName("createdAt").withKeyType(KeyType.RANGE));
GlobalSecondaryIndex index = new GlobalSecondaryIndex()
.withIndexName("expiresOnIndex")
.withProvisionedThroughput(new ProvisionedThroughput()
.withWriteCapacityUnits((long) 10)
.withReadCapacityUnits((long) 1))
.withProjection(new Projection().withProjectionType(ProjectionType.ALL));
index.setKeySchema(indexKeySchema);
dynamoDbclient.createTable(new CreateTableRequest()
.withProvisionedThroughput(new ProvisionedThroughput(1000L, 1000L))
.withTableName(tableName)
.withAttributeDefinitions(Arrays.asList(new AttributeDefinition(hashColumn, ScalarAttributeType.S),
new AttributeDefinition("expiresOn", ScalarAttributeType.S),
new AttributeDefinition("createdAt", ScalarAttributeType.S)))
.withKeySchema(Arrays.asList(new KeySchemaElement(hashColumn, KeyType.HASH)))
.withGlobalSecondaryIndexes(index));
And this is the query:
final Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
eav.put(":val1", new AttributeValue().withS(startTime.toString()));
eav.put(":val2", new AttributeValue().withS(endTime.toString()));
final DynamoDBQueryExpression<MyEntity> queryExpression =
new DynamoDBQueryExpression<MyEntity>()
.withKeyConditionExpression("expiresOn BETWEEN :val1 AND :val2")
.withExpressionAttributeValues(eav)
.withConsistentRead(false)
.withIndexName("expiresOnIndex");
return dynamoDBMapper.query(MyEntity.class, queryExpression);
Always getting Query key condition not supported. I'm not sure what's wrong here. I read couple of stackoverflow answers which suggested to use GSI, but doesn't work for me.
When you Query a DDB table or index,
The BETWEEN key condition can only be used on the Sort (aka Range) Key, the hash key must be an always be an EQUAL condition...
I have a dynamodb Table and I want to do a conditional update operation on that using aws sdk java library. I have the table with hashkey named as "Id" and sortkey named as "Sk" . I have 5 other fields in the table and i have to update 2 fields based on the condition on the remaining fields. The condition is "total_record_count = record_passed + records_failed" , where total_record_count, record_passed and records_failed are the fields in the table.
Below is the code
DynamoDB dynamoDB = new DynamoDB(amazonDynamoDB);
Table table = dynamoDB.getTable("ciw_ocrResponse_upload");
Map<String, String> expressionAttributeNames = new HashMap<String, String>();
expressionAttributeNames.put("#updatedAt", "updatedAt");
expressionAttributeNames.put("#statusOfTracker","statusOfTracker");
Timestamp timestamp = new Timestamp(System.currentTimeMillis());
Map<String, Object> eav = new HashMap<>();
eav.put(":val1", value);
eav.put(":val2", timestamp.getTime());
UpdateItemSpec updateItemSpec = new UpdateItemSpec().withPrimaryKey("Id", requestId, "Sk", sortKey)
.withUpdateExpression("set #updatedAt = :val2, #statusOfTracker = :val1")
.withValueMap(eav)
.withNameMap(expressionAttributeNames)
.withConditionExpression(" total_record_count = record_passed + records_failed")
.withReturnValues(ReturnValue.UPDATED_NEW);
table.updateItem(updateItemSpec);
I am getting the error as Invalid ConditionExpression: Syntax error; token: "+", near: "record_passed + records_failed"
According to the docs this arithmetic operation is not supported.
Workaround is to create a total_record_count field which you need to calculate and add on every new item created. Also to add it to all existing items.
I have dynamodb table structure as follows:
{
"id": "1",
"skills": {
"skill1": "html",
"skill2": "css"
}
}
I have task to filter by skills value, In order to complete my task wrote java logic as follows:
AmazonDynamoDB client = dynamoDBService.getClient();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("dummy");
Map<String, String> attributeNames = new HashMap<String, String >();
attributeNames.put("#columnValue", "skills.skill1");
Map<String, AttributeValue> attributeValues = new HashMap<String, AttributeValue>();
attributeValues.put(":val1", new AttributeValue().withS("html"));
ScanSpec scanSpec = new ScanSpec().withProjectionExpression("skills.skill1")
.withFilterExpression("#columnValue = :val1 ").withNameMap(new NameMap().with("#columnValue", "skills.skill1"))
.withValueMap(new ValueMap().withString(":val1", "html"));
ItemCollection<ScanOutcome> items = table.scan(scanSpec);
Iterator<Item> iter = items.iterator();
while (iter.hasNext()) {
Item item = iter.next();
System.out.println("--------"+item.toString());
}
The mentioned code does not help me out. Any solution ?
You can use a ProjectionExpression to retrieve only specific attributes or elements, rather than an entire item. A ProjectionExpression can specify top-level or nested attributes, using document paths.
for example from AWS:
GetItemSpec spec = new GetItemSpec()
.withPrimaryKey("Id", 206)
.withProjectionExpression("Id, Title, RelatedItems[0], Reviews.FiveStar")
.withConsistentRead(true);
Item item = table.getItem(spec);
System.out.println(item.toJSONPretty());
Simple solution to this problem is:
First fetch all the records from the table.
Then iterate over the list of that object.
Extract the skills from each object.
Wrote your logic to do filtering.
Repeat the loop till the last record.
I found solution,scanSpec should be as follows:
ScanSpec scanSpec = new ScanSpec()
.withFilterExpression("#category.#uid = :categoryuid").withNameMap(new NameMap().with("#category","skills").with("#uid",queryString))
.withValueMap(new ValueMap().withString(":categoryuid", queryString));