Trying to delete multiple row of DynamoDB.
Querying only on one table of DynamoDB.On basis the of Partition key, its returning 2 values as output, which is save in list.
Now after deleting this value using BatchDelete only first element get deleted. Sometimes on random basis second value also get deleted but that was not happened every time.
DynamoDBQueryExpression<Abc> queryExpression = new DynamoDBQueryExpression<Abc>()
.withHashKeyValues(abc);
List<Abc> xyz = dynamoDBMapper.query(Abc.class,queryExpression);
//xyz has size 2
dynamoDBMapper.batchDelete(xyz);
Should I use sleep or is there any other way.
If you look at Java V1 here:
https://github.com/awsdocs/aws-doc-sdk-examples
You will see it's marked as deprecated.
I strongly recommend that you upgrade to the AWS SDK for Java v2 API.
When working with Java V2 and DynamoDB, the Enhanced Client offers a straightforward way to map client-side classes to DynamoDB tables. This is documented in the Java V2 Developer Guide here:
Mapping items in DynamoDB tables
To use the Enhanced Client to delete multiple items, you can use this Java code:
package com.example.dynamodb;
// snippet-start:[dynamodb.java2.mapping.batchdelete.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.Key;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.model.BatchWriteItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.DeleteItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.WriteBatch;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
// snippet-end:[dynamodb.java2.mapping.batchdelete.import]
/*
* Before running this code example, create an Amazon DynamoDB table named Customer with these columns:
* - id - the id of the record that is the key
* - custName - the customer name
* - email - the email value
* - registrationDate - an instant value when the item was added to the table
*
* Also, ensure that you have set up your development environment, including your credentials.
*
* For information, see this documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class EnhancedBatchDeleteItems {
public static void main(String[] args) {
ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
deleteBatchRecords(enhancedClient);
ddb.close();
}
// snippet-start:[dynamodb.java2.mapping.batchdelete.main]
public static void deleteBatchRecords(DynamoDbEnhancedClient enhancedClient) {
try {
DynamoDbTable<Customer> mappedTable = enhancedClient.table("Customer", TableSchema.fromBean(Customer.class));
Key key1 = Key.builder()
.partitionValue("id110")
.build();
Key key2 = Key.builder()
.partitionValue("id120")
.build();
BatchWriteItemEnhancedRequest request = BatchWriteItemEnhancedRequest.builder()
.writeBatches(WriteBatch.builder(Customer.class)
.mappedTableResource(mappedTable)
.addDeleteItem(DeleteItemEnhancedRequest.builder()
.key(key1)
.build())
.build(),
WriteBatch.builder(Customer.class)
.mappedTableResource(mappedTable)
.addDeleteItem(DeleteItemEnhancedRequest.builder()
.key(key2)
.build())
.build())
.build();
// Delete these two items from the table.
enhancedClient.batchWriteItem(request);
System.out.println("Records deleted");
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
// snippet-end:[dynamodb.java2.mapping.batchdelete.main]
}
You can find this example and other Java v2 DynamoDB examples in AWS Code Example Github.
I do not see any issue with your code, I have tested similar and works for me with no issue. My first suggestion would be to ensure you wrap your code in a try/catch block:
try {
DynamoDBMapper mapper = new DynamoDBMapper(client);
Reply key = new Reply();
key.setPk("1");
DynamoDBQueryExpression<Reply> queryExpression = new DynamoDBQueryExpression<Reply>()
.withHashKeyValues(key)
.withReturnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
.withScanIndexForward(false);
List<Reply> latestReplies = mapper.query(Reply.class, queryExpression);
// Log keys here to be sure you are deleting the correct item
for (Reply c: latestReplies) {
System.out.println(c.getPk());
}
mapper.batchDelete(latestReplies);
} catch (Throwable t) {
System.err.println("Error running: " + t);
t.printStackTrace();
}
I would suggest that you implement some logging, just to be sure that the items you read are the ones you expected to be deleted.
It may also be worth enabling CloudTrail Dataplane Logs which will allow you to see all the dataplane events being executed on the table.
You may also enable HTTP wire logging to provide you another level of logging, however, this is not advised for production workloads as the logging is quite verbose.
Related
We are using Google cloud service AUTOML TABLES for online prediction.
We have created, trained and deployed the model. The model is giving predictions using the Google console. We are trying to integrate this model in our java code.
We are not able to pass “values” attribute as array of strings in payload object in java code. We haven’t found anything for this in documentation.
Please find the links we are using for this:
https://cloud.google.com/automl-tables/docs/samples/automl-tables-predict
Please find the json object in the screenshot.
Please let us know how to pass “values” attribute as array of strings in payload object?
Thanks.
Based from the reference you are following, to be able to populate "values" you need to define it at the main(). You can refer to Class Value.Builder if you need to set Numbers, Null, etc. values.
List<Value> values = new ArrayList<>();
values.add(Value.newBuilder().setStringValue("This is test data.").build());
// add more elements in values as needed
This list values will be used in Row that accepts iterable protobuf value. See Row.newBuilder.addAllValues().
Row row = Row.newBuilder().addAllValues(values).build();
Using these, the payload is complete and a prediction request be built:
ExamplePayload payload = ExamplePayload.newBuilder().setRow(row).build();
PredictRequest request =
PredictRequest.newBuilder()
.setName(name.toString())
.setPayload(payload)
.putParams("feature_importance", "true")
.build();
PredictResponse response = client.predict(request);
Your full prediction code should look like this:
import com.google.cloud.automl.v1beta1.AnnotationPayload;
import com.google.cloud.automl.v1beta1.ExamplePayload;
import com.google.cloud.automl.v1beta1.ModelName;
import com.google.cloud.automl.v1beta1.PredictRequest;
import com.google.cloud.automl.v1beta1.PredictResponse;
import com.google.cloud.automl.v1beta1.PredictionServiceClient;
import com.google.cloud.automl.v1beta1.Row;
import com.google.cloud.automl.v1beta1.TablesAnnotation;
import com.google.protobuf.Value;
import com.google.protobuf.NullValue;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
class TablesPredict {
public static void main(String[] args) throws IOException {
// TODO(developer): Replace these variables before running the sample.
String projectId = "your-project-id";
String modelId = "TBL9999999999";
// Values should match the input expected by your model.
List<Value> values = new ArrayList<>();
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setStringValue("blue-colar").build());
values.add(Value.newBuilder().setStringValue("married").build());
values.add(Value.newBuilder().setStringValue("primary").build());
values.add(Value.newBuilder().setStringValue("no").build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setStringValue("yes").build());
values.add(Value.newBuilder().setStringValue("yes").build());
values.add(Value.newBuilder().setStringValue("cellular").build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setStringValue("unknown").build());
predict(projectId, modelId, values);
}
static void predict(String projectId, String modelId, List<Value> values) throws IOException {
// Initialize client that will be used to send requests. This client only needs to be created
// once, and can be reused for multiple requests. After completing all of your requests, call
// the "close" method on the client to safely clean up any remaining background resources.
try (PredictionServiceClient client = PredictionServiceClient.create()) {
// Get the full path of the model.
ModelName name = ModelName.of(projectId, "us-central1", modelId);
Row row = Row.newBuilder().addAllValues(values).build();
ExamplePayload payload = ExamplePayload.newBuilder().setRow(row).build();
// Feature importance gives you visibility into how the features in a specific prediction
// request informed the resulting prediction. For more info, see:
// https://cloud.google.com/automl-tables/docs/features#local
PredictRequest request =
PredictRequest.newBuilder()
.setName(name.toString())
.setPayload(payload)
.putParams("feature_importance", "true")
.build();
PredictResponse response = client.predict(request);
System.out.println("Prediction results:");
for (AnnotationPayload annotationPayload : response.getPayloadList()) {
TablesAnnotation tablesAnnotation = annotationPayload.getTables();
System.out.format(
"Classification label: %s%n", tablesAnnotation.getValue().getStringValue());
System.out.format("Classification score: %.3f%n", tablesAnnotation.getScore());
// Get features of top importance
tablesAnnotation
.getTablesModelColumnInfoList()
.forEach(
info ->
System.out.format(
"\tColumn: %s - Importance: %.2f%n",
info.getColumnDisplayName(), info.getFeatureImportance()));
}
}
}
}
For testing purposes I used Google's test dataset (gs://cloud-ml-tables-data/bank-marketing.csv) and used the code above to run send prediction.
See test prediction:
I am trying to debug the java code using HBaseTestingUtility library. I already have table created. I need to:
- Insert a value with a key in "myTable"
- Get the value from "myTable" with the key
- Verify the returned value is equal to the value I created
Here is the code that I filled out:
package HbaseUniteTest;
import jdk.nashorn.api.scripting.ScriptUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.*;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.io.compress.Compression;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.junit.Assert;
import static org.junit.Assert.assertEquals;
public class TestCreateTableClass
{
private final static String tableName = "myTable";
private static ScriptUtils HTableUtil;
public static void main( String[] args ) throws Exception {
//Start the "mini cluster"
HBaseTestingUtility testingUtility = new HBaseTestingUtility();
testingUtility.startMiniCluster();
//Get the configuration
//Configuration conf = ...
Configuration conf = testingUtility.getConfiguration();
//Instantiate a connection
Connection connection = ConnectionFactory.createConnection(conf);
//Define table "myTable"
HTableDescriptor table = new HTableDescriptor(TableName.valueOf(tableName));
table.addFamily(new HColumnDescriptor("cf1").setCompressionType(Compression.Algorithm.NONE));
//Create table "myTable"
connection.getAdmin().createTable(table);
//Get the first (and only) table name
String first_table = connection.getAdmin().listTableNames()[0].getNameAsString();
//Verify the returned Table name is equal to the table name we provided
assertEquals(tableName,first_table);
//Insert a value with a key in "myTable"
byte[] key = Bytes.toBytes("some-key");
Put put = new Put(key);
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.2"), System.currentTimeMillis(), Bytes.toBytes("val1.2"));
put.add(Bytes.toBytes("colfam2"), Bytes.toBytes("qual2.1"), System.currentTimeMillis(), Bytes.toBytes("val2.1"));
Result converted = HTableUtil.convert(put);
table.put(put);
Result readFromTable = table.get(new Get(key));
Assert.assertArrayEquals(readFromTable.raw(), converted.raw());
//Get the value from "myTable" with the key
//Verify the returned value is equal to the value you created
//Stop the mini cluster
testingUtility.shutdownMiniCluster();
System.out.println("END OF TEST");
}
public static void setHTableUtil(ScriptUtils HTableUtil) {
TestCreateTableClass.HTableUtil = HTableUtil;
}
}
However, I got the following error:
1. The error at this line of code with the function put.add()
put.add(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
The 2nd error on this line of code:
Result converted = HTableUtil.convert(put);
Java cannot find symbol for these 3 methods put(), get(), raw()
table.put(put);
Result readFromTable = table.get(new Get(key));
Assert.assertArrayEquals(readFromTable.raw(), converted.raw());
I also notice some warnings regarding the class HTableDescriptor, HColumnDescriptor have been deprecated. I checked on internet and they advice to use for example "TableDescriptorBuilder" instead but I am not sure how to use it. (Ref: https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java)
1. The error at this line of code with the function put.add().
I think you can use addColumn() like this for adding column.
put.addColumn(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.1"), System.currentTimeMillis(), Bytes.toBytes("val1.1"));
put.addColumn(Bytes.toBytes("colfam1"), Bytes.toBytes("qual1.2"), System.currentTimeMillis(), Bytes.toBytes("val1.2"));
put.addColumn(Bytes.toBytes("colfam2"), Bytes.toBytes("qual2.1"), System.currentTimeMillis(), Bytes.toBytes("val2.1"));
2. The 2nd error on this line of code:
I'm not familiar with 'ScriptUtils', But I think It works.
Result converted = (Result) HTableUtil.convert(put, Result.class);
3. Java cannot find symbol for these 3 methods put(), get(), raw()
It because you keep using 'HTableDescriptor' to put(), get(), or raw(). 'HTableDescriptor' is used to create table like DDL. You need to use Table class to manipulate using put(), get(), or raw().
Table createdTable = connection.getTable(TableName.valueOf(tableName));
createdTable.put(put);
Result readFromTable = createdTable.get(new Get(key));
Also, I believe class 'Result' doesn't provide raw(). So, you can compare both Results using Result.compareResults() like this.
Result.compareResults(readFromTable, converted);
4. How to use 'TableDescriptorBuilder'
Like I said above, 'Descriptor' is the class for defining your table, column family, column, and so on. So, you need to use it when you make/create them.
//Define table "myTable"
TableDescriptorBuilder table = TableDescriptorBuilder.newBuilder(TableName.valueOf(tableName));
table.setColumnFamily(ColumnFamilyDescriptorBuilder.newBuilder(Bytes.toBytes("cf1")).setCompressionType(Compression.Algorithm.NONE).build());
//Create table "myTable"
connection.getAdmin().createTable(table.build());
Hi I'm trying to create a java program which checks for a value present in the amazon dynamodb.
Well actually im trying to write a java program for a retail store where the program checks for a certain product(value) in the dynamodb and returns the quantity present in the store.I already created a table in dynamodb and also used the scan api to retrieve items.But now i want to search for a certain value is present or not and also return the quantity present.
It would be really helpful if anyone could help me with a snippet.
ps: i'm new to java
Thanks
Given following data modelling for a table ConsumerTable, with leaseKey as indexed field;
{
"checkpoint": "49572533067729099438069946639296561054199125138133221378",
"checkpointSubSequenceNumber": 0,
"leaseCounter": 1372794,
"leaseKey": "shardId-000000000000",
"leaseOwner": "pout_172.21.63.243",
"ownerSwitchesSinceCheckpoint": 0
}
Java API to query Amazon DynamoDB by the indexed field would be
import com.amazonaws.ClientConfiguration;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.retry.v2.RetryPolicy;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodbv2.document.DynamoDB;
public AmazonDynamoDB getDbConnection() {
AmazonDynamoDBClient dynamoDB = new AmazonDynamoDBClient( new ProfileCredentialsProvider("your-aws-auth-profile_in_~/.aws/credentials"));
dynamoDB.setRegion(Region.getRegion(Regions.US_WEST_2))
return dynamoDB;
}
public void getData() {
DynamoDB dynamoDB = new DynamoDB(getDbConnection());
Table consumerOffsetTable = dynamoDB.getTable("ConsumerTable");
Optional.ofNullable(consumerOffsetTable.getItem("leaseKey", "shardId-000000000000")).ifPresent(item -> {
Map<String, Object> itemMap = item.asMap();
System.out.println(itemMap.get("leaseKey").toString() + " -> " + itemMap.get("checkpoint").toString());
});
}
Maven dependency would be
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>1.11.115</version>
</dependency>
In your product, quantity example, index product attribute and query should be similar to above example.
Resources
For AWS access using SDK - http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html
For Dynamodb apis
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ScanJavaDocumentAPI.html
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryingJavaDocumentAPI.html
I have a collection with some documents in it. And in my application I am creating this collection first and then inserting documents. Also, based on the requirement I need to truncate (delete all documents) the collection as well. Using document db java api I have written the following code for my this purpose-
DocumentClient documentClient = getConnection(masterkey, server, portNo);
List<Database> databaseList = documentClient.queryDatabases("SELECT * FROM root r WHERE r.id='" + schemaName + "'", null).getQueryIterable().toList();
DocumentCollection collection = null;
Database databaseCache = (Database)databaseList.get(0);
List<DocumentCollection> collectionList = documentClient.queryCollections(databaseCache.getSelfLink(), "SELECT * FROM root r WHERE r.id='" + collectionName + "'", null).getQueryIterable().toList();
// truncate logic
if (collectionList.size() > 0) {
collection = ((DocumentCollection) collectionList.get(0));
if (truncate) {
try {
documentClient.deleteDocument(collection.getSelfLink(), null);
} catch (DocumentClientException e) {
e.printStackTrace();
}
}
} else { // create logic
RequestOptions requestOptions = new RequestOptions();
requestOptions.setOfferType("S1");
collection = new DocumentCollection();
collection.setId(collectionName);
try {
collection = documentClient.createCollection(databaseCache.getSelfLink(), collection, requestOptions).getResource();
} catch (DocumentClientException e) {
e.printStackTrace();
}
With the above code I am able to create a new collection successfully. Also, I am able to insert documents as well in this collection. But while truncating the collection I am getting below error-
com.microsoft.azure.documentdb.DocumentClientException: The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'delete
colls
eyckqjnw0ae=
I am using Azure Document DB Java API version 1.9.5.
It will be of great help if you can point out the error in my code or if there is any other better way of truncating collection. I would really appreciate any kind of help here.
According to your description & code, I think the issue was caused by the code below.
try {
documentClient.deleteDocument(collection.getSelfLink(), null);
} catch (DocumentClientException e) {
e.printStackTrace();
}
It seems that you want to delete a document via the code above, but pass the argument documentLink with a collection link.
So if your real intention is to delete a collection, please using the method DocumentClient.deleteCollection(collectionLink, options).
I was trying to print all the objects in a bucket but I am getting an error.
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 301, AWS Service: Amazon S3, AWS Request ID: 758A7CBF1A29FD74, AWS Error Code: PermanentRedirect, AWS Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., S3
At the moment I only have the following code :
public class S3Download {
/**
* #param args
*/
public static void main(String[] args) {
AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
Region usWest2 = Region.getRegion(Regions.US_WEST_2);
s3.setRegion(usWest2);
String bucketName = "apireleasecandidate1";
ListObjectsRequest listObjectRequest = new ListObjectsRequest().withBucketName(bucketName);
ObjectListing objectListing;
do{
objectListing = s3.listObjects(listObjectRequest);
for(S3ObjectSummary objectSummary : objectListing.getObjectSummaries()){
System.out.println(" - " + objectSummary.getKey() + " " + "(size = " +
objectSummary.getSize() + ")");
}
listObjectRequest.setMarker(objectListing.getNextMarker());
}while(objectListing.isTruncated());
}
}
I found this solution on amazon's website.
Does anyone know what I am missing?
For Scala developers, here it is recursive function to execute a full scan and map of the contents of an AmazonS3 bucket using the official AWS SDK for Java
import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.{S3ObjectSummary, ObjectListing, GetObjectRequest}
import scala.collection.JavaConversions.{collectionAsScalaIterable => asScala}
def map[T](s3: AmazonS3Client, bucket: String, prefix: String)(f: (S3ObjectSummary) => T) = {
def scan(acc:List[T], listing:ObjectListing): List[T] = {
val summaries = asScala[S3ObjectSummary](listing.getObjectSummaries())
val mapped = (for (summary <- summaries) yield f(summary)).toList
if (!listing.isTruncated) mapped.toList
else scan(acc ::: mapped, s3.listNextBatchOfObjects(listing))
}
scan(List(), s3.listObjects(bucket, prefix))
}
To invoke the above curried map() function, simply pass the already constructed (and properly initialized) AmazonS3Client object (refer to the official AWS SDK for Java API Reference), the bucket name and the prefix name in the first parameter list. Also pass the function f() you want to apply to map each object summary in the second parameter list.
For example
map(s3, bucket, prefix)(s => println(s))
will print all the files
val tuple = map(s3, bucket, prefix)(s => (s.getKey, s.getOwner, s.getSize))
will return the full list of (key, owner, size) tuples in that bucket/prefix
val totalSize = map(s3, "bucket", "prefix")(s => s.getSize).sum
will return the total size of its content (note the additional sum() folding function applied at the end of the expression ;-)
You can combine map() with many other functions as you would normally approach by Monads in Functional Programming
It appears that your bucket "apireleasecandidate1" is not in the us-west-1 region. I think it is in the us-classic region. You should modify your code to remove the setRegion() call.