Develop Reference

BatchDelete is not deleting all value in list

Trying to delete multiple row of DynamoDB.
Querying only on one table of DynamoDB.On basis the of Partition key, its returning 2 values as output, which is save in list.
Now after deleting this value using BatchDelete only first element get deleted. Sometimes on random basis second value also get deleted but that was not happened every time.
DynamoDBQueryExpression<Abc> queryExpression = new DynamoDBQueryExpression<Abc>()
.withHashKeyValues(abc);
List<Abc> xyz = dynamoDBMapper.query(Abc.class,queryExpression);
//xyz has size 2
dynamoDBMapper.batchDelete(xyz);
Should I use sleep or is there any other way.

If you look at Java V1 here:
https://github.com/awsdocs/aws-doc-sdk-examples
You will see it's marked as deprecated.
I strongly recommend that you upgrade to the AWS SDK for Java v2 API.
When working with Java V2 and DynamoDB, the Enhanced Client offers a straightforward way to map client-side classes to DynamoDB tables. This is documented in the Java V2 Developer Guide here:
Mapping items in DynamoDB tables
To use the Enhanced Client to delete multiple items, you can use this Java code:
package com.example.dynamodb;
// snippet-start:[dynamodb.java2.mapping.batchdelete.import]
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbEnhancedClient;
import software.amazon.awssdk.enhanced.dynamodb.DynamoDbTable;
import software.amazon.awssdk.enhanced.dynamodb.Key;
import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
import software.amazon.awssdk.enhanced.dynamodb.model.BatchWriteItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.DeleteItemEnhancedRequest;
import software.amazon.awssdk.enhanced.dynamodb.model.WriteBatch;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
// snippet-end:[dynamodb.java2.mapping.batchdelete.import]
/*
* Before running this code example, create an Amazon DynamoDB table named Customer with these columns:
* - id - the id of the record that is the key
* - custName - the customer name
* - email - the email value
* - registrationDate - an instant value when the item was added to the table
*
* Also, ensure that you have set up your development environment, including your credentials.
*
* For information, see this documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class EnhancedBatchDeleteItems {
public static void main(String[] args) {
ProfileCredentialsProvider credentialsProvider = ProfileCredentialsProvider.create();
Region region = Region.US_EAST_1;
DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();
DynamoDbEnhancedClient enhancedClient = DynamoDbEnhancedClient.builder()
.dynamoDbClient(ddb)
.build();
deleteBatchRecords(enhancedClient);
ddb.close();
}
// snippet-start:[dynamodb.java2.mapping.batchdelete.main]
public static void deleteBatchRecords(DynamoDbEnhancedClient enhancedClient) {
try {
DynamoDbTable<Customer> mappedTable = enhancedClient.table("Customer", TableSchema.fromBean(Customer.class));
Key key1 = Key.builder()
.partitionValue("id110")
.build();
Key key2 = Key.builder()
.partitionValue("id120")
.build();
BatchWriteItemEnhancedRequest request = BatchWriteItemEnhancedRequest.builder()
.writeBatches(WriteBatch.builder(Customer.class)
.mappedTableResource(mappedTable)
.addDeleteItem(DeleteItemEnhancedRequest.builder()
.key(key1)
.build())
.build(),
WriteBatch.builder(Customer.class)
.mappedTableResource(mappedTable)
.addDeleteItem(DeleteItemEnhancedRequest.builder()
.key(key2)
.build())
.build())
.build();
// Delete these two items from the table.
enhancedClient.batchWriteItem(request);
System.out.println("Records deleted");
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
}
// snippet-end:[dynamodb.java2.mapping.batchdelete.main]
}
You can find this example and other Java v2 DynamoDB examples in AWS Code Example Github.

I do not see any issue with your code, I have tested similar and works for me with no issue. My first suggestion would be to ensure you wrap your code in a try/catch block:
try {
DynamoDBMapper mapper = new DynamoDBMapper(client);
Reply key = new Reply();
key.setPk("1");
DynamoDBQueryExpression<Reply> queryExpression = new DynamoDBQueryExpression<Reply>()
.withHashKeyValues(key)
.withReturnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
.withScanIndexForward(false);
List<Reply> latestReplies = mapper.query(Reply.class, queryExpression);
// Log keys here to be sure you are deleting the correct item
for (Reply c: latestReplies) {
System.out.println(c.getPk());
}
mapper.batchDelete(latestReplies);
} catch (Throwable t) {
System.err.println("Error running: " + t);
t.printStackTrace();
}
I would suggest that you implement some logging, just to be sure that the items you read are the ones you expected to be deleted.
It may also be worth enabling CloudTrail Dataplane Logs which will allow you to see all the dataplane events being executed on the table.
You may also enable HTTP wire logging to provide you another level of logging, however, this is not advised for production workloads as the logging is quite verbose.

How to pass “values” attribute as array of strings in payload object in case of Googles AUTOML TABLE?

We are  using Google cloud service AUTOML TABLES for online prediction.
We have created, trained and deployed the model. The model is giving predictions using the Google console. We are trying to integrate this model in our java code.
We are not able to pass “values”  attribute as array of strings in payload object in java code. We haven’t found anything for this in documentation.
Please find the links we are using for this:
https://cloud.google.com/automl-tables/docs/samples/automl-tables-predict
Please find the json object in the screenshot.
Please let us know how to pass “values”  attribute as array of strings in payload object?
Thanks.

Based from the reference you are following, to be able to populate "values" you need to define it at the main(). You can refer to Class Value.Builder if you need to set Numbers, Null, etc. values.
List<Value> values = new ArrayList<>();
values.add(Value.newBuilder().setStringValue("This is test data.").build());
// add more elements in values as needed
This list values will be used in Row that accepts iterable protobuf value. See Row.newBuilder.addAllValues().
Row row = Row.newBuilder().addAllValues(values).build();
Using these, the payload is complete and a prediction request be built:
ExamplePayload payload = ExamplePayload.newBuilder().setRow(row).build();
PredictRequest request =
PredictRequest.newBuilder()
.setName(name.toString())
.setPayload(payload)
.putParams("feature_importance", "true")
.build();
PredictResponse response = client.predict(request);
Your full prediction code should look like this:
import com.google.cloud.automl.v1beta1.AnnotationPayload;
import com.google.cloud.automl.v1beta1.ExamplePayload;
import com.google.cloud.automl.v1beta1.ModelName;
import com.google.cloud.automl.v1beta1.PredictRequest;
import com.google.cloud.automl.v1beta1.PredictResponse;
import com.google.cloud.automl.v1beta1.PredictionServiceClient;
import com.google.cloud.automl.v1beta1.Row;
import com.google.cloud.automl.v1beta1.TablesAnnotation;
import com.google.protobuf.Value;
import com.google.protobuf.NullValue;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
class TablesPredict {
public static void main(String[] args) throws IOException {
// TODO(developer): Replace these variables before running the sample.
String projectId = "your-project-id";
String modelId = "TBL9999999999";
// Values should match the input expected by your model.
List<Value> values = new ArrayList<>();
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setStringValue("blue-colar").build());
values.add(Value.newBuilder().setStringValue("married").build());
values.add(Value.newBuilder().setStringValue("primary").build());
values.add(Value.newBuilder().setStringValue("no").build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setStringValue("yes").build());
values.add(Value.newBuilder().setStringValue("yes").build());
values.add(Value.newBuilder().setStringValue("cellular").build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setNullValue(NullValue.NULL_VALUE).build());
values.add(Value.newBuilder().setStringValue("unknown").build());
predict(projectId, modelId, values);
}
static void predict(String projectId, String modelId, List<Value> values) throws IOException {
// Initialize client that will be used to send requests. This client only needs to be created
// once, and can be reused for multiple requests. After completing all of your requests, call
// the "close" method on the client to safely clean up any remaining background resources.
try (PredictionServiceClient client = PredictionServiceClient.create()) {
// Get the full path of the model.
ModelName name = ModelName.of(projectId, "us-central1", modelId);
Row row = Row.newBuilder().addAllValues(values).build();
ExamplePayload payload = ExamplePayload.newBuilder().setRow(row).build();
// Feature importance gives you visibility into how the features in a specific prediction
// request informed the resulting prediction. For more info, see:
// https://cloud.google.com/automl-tables/docs/features#local
PredictRequest request =
PredictRequest.newBuilder()
.setName(name.toString())
.setPayload(payload)
.putParams("feature_importance", "true")
.build();
PredictResponse response = client.predict(request);
System.out.println("Prediction results:");
for (AnnotationPayload annotationPayload : response.getPayloadList()) {
TablesAnnotation tablesAnnotation = annotationPayload.getTables();
System.out.format(
"Classification label: %s%n", tablesAnnotation.getValue().getStringValue());
System.out.format("Classification score: %.3f%n", tablesAnnotation.getScore());
// Get features of top importance
tablesAnnotation
.getTablesModelColumnInfoList()
.forEach(
info ->
System.out.format(
"\tColumn: %s - Importance: %.2f%n",
info.getColumnDisplayName(), info.getFeatureImportance()));
}
}
}
}
For testing purposes I used Google's test dataset (gs://cloud-ml-tables-data/bank-marketing.csv) and used the code above to run send prediction.
See test prediction:

Neo4j ExecutionEngine does not return valid results

Trying to use a similar example from the sample code found here
My sample function is:
void query()
{
String nodeResult = "";
String rows = "";
String resultString;
String columnsString;
System.out.println("In query");
// START SNIPPET: execute
ExecutionEngine engine = new ExecutionEngine( graphDb );
ExecutionResult result;
try ( Transaction ignored = graphDb.beginTx() )
{
result = engine.execute( "start n=node(*) where n.Name =~ '.*79.*' return n, n.Name" );
// END SNIPPET: execute
// START SNIPPET: items
Iterator<Node> n_column = result.columnAs( "n" );
for ( Node node : IteratorUtil.asIterable( n_column ) )
{
// note: we're grabbing the name property from the node,
// not from the n.name in this case.
nodeResult = node + ": " + node.getProperty( "Name" );
System.out.println("In for loop");
System.out.println(nodeResult);
}
// END SNIPPET: items
// START SNIPPET: columns
List<String> columns = result.columns();
// END SNIPPET: columns
// the result is now empty, get a new one
result = engine.execute( "start n=node(*) where n.Name =~ '.*79.*' return n, n.Name" );
// START SNIPPET: rows
for ( Map<String, Object> row : result )
{
for ( Entry<String, Object> column : row.entrySet() )
{
rows += column.getKey() + ": " + column.getValue() + "; ";
System.out.println("nested");
}
rows += "\n";
}
// END SNIPPET: rows
resultString = engine.execute( "start n=node(*) where n.Name =~ '.*79.*' return n.Name" ).dumpToString();
columnsString = columns.toString();
System.out.println(rows);
System.out.println(resultString);
System.out.println(columnsString);
System.out.println("leaving");
}
}
When I run this in the web console I get many results (as there are multiple nodes that have an attribute of Name that contains the pattern 79. Yet running this code returns no results. The debug print statements 'in loop' and 'nested' never print either. Thus this must mean there are not results found in the Iterator, yet that doesn't make sense.
And yes, I already checked and made sure that the graphDb variable is the same as the path for the web console. I have other code earlier that uses the same variable to write to the database.
EDIT - More info
If I place the contents of query in the same function that creates my data, I get the correct results. If I run the query by itself it returns nothing. It's almost as the query works only in the instance where I add the data and not if I come back to the database cold in a separate instance.
EDIT2 -
Here is a snippet of code that shows the bigger context of how it is being called and sharing the same DBHandle
package ContextEngine;
import ContextEngine.NeoHandle;
import java.util.LinkedList;
/*
* Class to handle streaming data from any coded source
*/
public class Streamer {
private NeoHandle myHandle;
private String contextType;
Streamer()
{
}
public void openStream(String contextType)
{
myHandle = new NeoHandle();
myHandle.createDb();
}
public void streamInput(String dataLine)
{
Context context = new Context();
/*
* get database instance
* write to database
* check for errors
* report errors & success
*/
System.out.println(dataLine);
//apply rules to data (make ContextRules do this, send type and string of data)
ContextRules contextRules = new ContextRules();
context = contextRules.processContextRules("Calls", dataLine);
//write data (using linked list from contextRules)
NeoProcessor processor = new NeoProcessor(myHandle);
processor.processContextData(context);
}
public void runQuery()
{
NeoProcessor processor = new NeoProcessor(myHandle);
processor.query();
}
public void closeStream()
{
/*
* close database instance
*/
myHandle.shutDown();
}
}
Now, if I call streamInput AND query in in the same instance (parent calls) the query returns results. If I only call query and do not enter ANY data in that instance (yet web console shows data for same query) I get nothing. Why would I have to create the Nodes and enter them into the database at runtime just to return a valid query. Shouldn't I ALWAYS get the same results with such a query?

You mention that you are using the Neo4j Browser, which comes with Neo4j. However, the example you posted is for Neo4j Embedded, which is the in-process version of Neo4j. Are you sure you are talking to the same database when you try your query in the Browser?
In order to talk to Neo4j Server from Java, I'd recommend looking at the Neo4j JDBC driver, which has good support for connecting to the Neo4j server from Java.
http://www.neo4j.org/develop/tools/jdbc
You can set up a simple connection by adding the Neo4j JDBC jar to your classpath, available here: https://github.com/neo4j-contrib/neo4j-jdbc/releases Then just use Neo4j as any JDBC driver:
Connection conn = DriverManager.getConnection("jdbc:neo4j://localhost:7474/");
ResultSet rs = conn.executeQuery("start n=node({id}) return id(n) as id", map("id", id));
while(rs.next()) {
System.out.println(rs.getLong("id"));
}
Refer to the JDBC documentation for more advanced usage.
To answer your question on why the data is not durably stored, it may be one of many reasons. I would attempt to incrementally scale back the complexity of the code to try and locate the culprit. For instance, until you've found your problem, do these one at a time:
Instead of looping through the result, print it using System.out.println(result.dumpToString());
Instead of the regex query, try just MATCH (n) RETURN n, to return all data in the database
Make sure the data you are seeing in the browser is not "old" data inserted earlier on, but really is an insert from your latest run of the Java program. You can verify this by deleting the data via the browser before running the Java program using MATCH (n) OPTIONAL MATCH (n)-[r]->() DELETE n,r;
Make sure you are actually working against the same database directories. You can verify this by leaving the server running. If you can still start your java program, unless your Java program is using the Neo4j REST Bindings, you are not using the same directory. Two Neo4j databases cannot run against the same database directory simultaneously.

Retrieving value from objectdb database from program

I am facing some problems while I am trying to fetch data through the program. I am using objectDB as my database. Also, my database is already set up and I have dropped the laptop.odb file in the db folder of my objectDB installation. Also, when I go to the explorer and fire the query:
select this.modelName == "HP Pavillion"
correct results comes up. But, when I try to do the same thing with my code as in the following
public static void main(String argv[]) {
PersistenceManager pm = Utilities.getPersistenceManager("laptop.odb");
System.out.println("-- TEST --\n");
Query query = pm.newQuery(Laptop.class,"this.modelName == \"HP Pavillion\"");
Collection result = (Collection)query.execute();
System.out.println("Result is >> "+result);
Here no results are returned. My output is :
-- TEST find --
Result is >> []
My code for the class is the following.
package com.project;
import java.util.*;
import javax.annotation.processing.Processor;
import javax.jdo.*;
import com.objectdb.Utilities;
public class Laptop {
String modelName; // key
public static void main(String argv[]) {
PersistenceManager pm = Utilities.getPersistenceManager("laptop.odb");
System.out.println("-- TEST find --\n");
Query query = pm.newQuery(Laptop.class,"this.modelName == \"HP Pavillion\"");
Collection result = (Collection)query.execute();
System.out.println("Result is >> "+result);
}
Any suggestions ?

The reason could be that "laptop.odb" refers to a non existing ObjectDB database. In that case a new database is automatically created. Because the new database it is created empty, no results are returned from the query.
Try specifying an absolute path to the existing database.

Check if table exists

What is the fastest way to check if Hbase table exists? Looking at this api :
http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html
Which of these is the fastest :
tableExists
isTableEnabled
isTableAvailable
listTables
With #4 you get list of all tables and iterate trough it and compare if one of those tables matches your table name.
Or there is another, more smart way ?

Here is my sample code. (scala)
import org.apache.hadoop.hbase.HBaseConfiguration
var TableName = "sample"
val conf = HBaseConfiguration.create()
var hbaseAdmin = new HBaseAdmin(conf)
if (!hbaseAdmin.tableExists(TableName)) {
println(TableName + " Does Not Exist")
}
Here, you just need to use "tableExists" to check whether this TableName exists.

HBaseAdmin hba = new HBaseAdmin(hbaseTemplate.getConfiguration());
if (hba.tableExists(tableName) == false) {
HTableDescriptor tableDescriptor = new HTableDescriptor(tableName);
HColumnDescriptor columnDescriptor = new HColumnDescriptor(columnFamilyProfile);
tableDescriptor.addFamily(columnDescriptor);
hba.createTable(tableDescriptor);
}

Using HBaseAdmin.tableExists only takes about 500ms to check if the table exists. We only have two nodes in our cluster, so it might be dependent on the size of your cluster, but it doesn't seem unreasonably slow.

You could attempt to open an HTable to the table and (I think) it will throw an exception/error (not at work yet so can't do a quick test) if the table doesn't exist.
Not 100% this will work, just an off the top of the head idea. :)

I have to check if table exist every time i start my app. I have made this in a configuration class, with spring boot
Here is the code, hope it helps.
#Configuration
public class CustomHbaseConfiguration {
#Bean
public Connection hbaseConnection() throws IOException {
// Create connection
final org.apache.hadoop.conf.Configuration configuration = HBaseConfiguration.create();
// Validate that Hbase is available
HBaseAdmin.available(configuration);
// return the hbaseConnection Bean
return ConnectionFactory.createConnection(configuration);
}
#PostConstruct
public void hbaseTableLogic() throws IOException {
// With the hbaseConnection bean, get the HbaseAdmin instance
Admin admin = hbaseConnection().getAdmin();
// The name of my table
TableName YOUR_TABLE_NAME_HERE = TableName.valueOf("PUT_YOUR_TABLE_NAME_HERE");
// Check if the table already exists ? else : create table and colum family
if (!admin.tableExists(YOUR_TABLE_NAME_HERE)) {
HTableDescriptor hTableDescriptor = new HTableDescriptor(YOUR_TABLE_NAME_HERE);
hTableDescriptor.addFamily(new HColumnDescriptor("PUT_YOUR_COLUM_FAMILY_HERE"));
admin.createTable(hTableDescriptor);
}
}
}