Very slow save operation with DynamoDB on local machine - java

I am trying to use DynamoDB on my local pc.
Before I was using MongoDB and the performance of the DynamoDB compared to it is very poor.
The save operation to a table takes a very long time, about 13 seconds for 100 records.
The records are pretty small, example below.
Here is my full example and code which I use to run it:
public class dynamoTry {
private AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:8000", "us-east-2"))
.build();
private DynamoDB dynamoDB = new DynamoDB(client);
private DynamoDBMapper mapper = new DynamoDBMapper(client);
public static void main(String[] args) {
dynamoTry dt = new dynamoTry ();
dt .deleteTable();
dt .buildGrid();
dt .demoFill();
dt .scanTable();
}
public void buildGrid() {
System.out.println("Attempting to create table; please wait...");
String tableName = "Grid";
List<AttributeDefinition> attributeDefinitions = new ArrayList<AttributeDefinition>();
attributeDefinitions.add(new AttributeDefinition().withAttributeName("name").withAttributeType(ScalarAttributeType.S));
attributeDefinitions.add(new AttributeDefinition().withAttributeName("country").withAttributeType(ScalarAttributeType.S));
List<KeySchemaElement> keySchema = new ArrayList<KeySchemaElement>();
keySchema.add(new KeySchemaElement().withAttributeName("name").withKeyType(KeyType.HASH));
keySchema.add(new KeySchemaElement().withAttributeName("country").withKeyType(KeyType.RANGE));
CreateTableRequest request = new CreateTableRequest().withTableName(tableName).withKeySchema(keySchema)
.withAttributeDefinitions(attributeDefinitions).withProvisionedThroughput(
new ProvisionedThroughput().withReadCapacityUnits(500L).withWriteCapacityUnits(500L));
Table table = dynamoDB.createTable(request);
try {
table.waitForActive();
System.out.println("Success.");
} catch (InterruptedException e) {e.printStackTrace();}
}
public void demoFill() {
final List<GridPoint> gpl = new ArrayList<GridPoint>();
int count = 0;
while (count < 100) {
final String point = "point" + count;
gpl.add(makeGP(point, count, "continent", "country", new HashSet<Double>(Arrays.asList(22.23435, 37.89746))));
count++;
}
long startTime = System.nanoTime();
addBatch(gpl);
long endTime = System.nanoTime();
long duration = (endTime - startTime)/1000000;
System.out.println(duration + " [ms]");
}
public void addBatch(List<GridPoint> gpl) {
mapper.batchSave(gpl);
}
public GridPoint makeGP(String name, int sqNum, String continent, String country, HashSet<Double> cords) {
GridPoint item = new GridPoint();
item.setName(name);
item.setSqNum(sqNum);
item.setContinent(continent);
item.setCountry(country);
item.setCoordinates(cords);
return item;
}
public void scanTable() {
Map<String, AttributeValue> eav = new HashMap<String, AttributeValue>();
eav.put(":val", new AttributeValue().withN("0"));
DynamoDBScanExpression scanExpression = new DynamoDBScanExpression().withFilterExpression("sqNum >= :val").withExpressionAttributeValues(eav);
List<GridPoint> scanResult = mapper.scan(GridPoint.class, scanExpression);
for (GridPoint gp : scanResult) {
System.out.println(gp);
}
}
public void deleteTable() {
Table table = dynamoDB.getTable("Grid");
try {
System.out.println("Attempting to delete table 'Grid', please wait...");
table.delete();
table.waitForDelete();
System.out.print("Success.");
}
catch (Exception e) {
System.err.println("Unable to delete table: ");
System.err.println(e.getMessage());
}
}
}
Here is the code for the GridPoint class:
#DynamoDBTable(tableName = "Grid")
public class GridPoint {
private String name;
private int sqNum;
private String continent;
private String country;
private HashSet<Double> coordinates; // [longitude, latitude]
// Partition key
#DynamoDBHashKey(attributeName = "name")
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
#DynamoDBAttribute(attributeName = "sqNum")
public int getSqNum() {
return sqNum;
}
public void setSqNum(int sqNum) {
this.sqNum = sqNum;
}
#DynamoDBAttribute(attributeName = "continent")
public String getContinent() {
return continent;
}
public void setContinent(String continent) {
this.continent = continent;
}
#DynamoDBAttribute(attributeName = "country")
public String getCountry() {
return country;
}
public void setCountry(String country) {
this.country = country;
}
#DynamoDBAttribute(attributeName = "coordinates")
public HashSet<Double> getCoordinates() {
return coordinates;
}
public void setCoordinates(HashSet<Double> coordinates) {
this.coordinates = coordinates;
}
#Override
public String toString() {
return "GP {name = " + name + ", sqNum = " + sqNum + ", continent = " + continent + ", country = " + country
+ ", coordinates = " + coordinates.toString() + "}";
}
}
Why is it so slow? is there any way of speeding the writing process?
In MongoDB the same operations would take less than a second.
When I was running it about 3000 points it took several minutes to finish, seems not reasonable.
Is it possible to make the batch save process parallel? would it speed things up?
I also tried to set the ProvisionedThroughput parameter to a higher value but that did not help.
I am lost, any help would be appreciated, thank you.

It's is slow because it is not DynamoDB. There is no Local DynamoDB!
DynamoDB is a managed service provided by AWS and it is really fast (milliseconds for the first bytes), highly scalable and durable. It is a really good product with a lot of performance for a small amount of money. But it is a managed service. It only works on AWS environment. There is no way to you or anyone else get a copy and install DynamoDB in Azure, GCP or even in your local environment.
What are you using is a facade, probably developed by AWS Team to help developers test their applications. There are other DynamoDB facades, not developed by AWS Team but everyone of then just respect a protocol that accepts all api calls from the original product. As a facade, his objective is just provide a endpoint that can receive your calls and respond like the original product. If you make a call that the original DynamoDB would respond with an Ok the facade will respond with an Ok. If you make a call that the original DynamoDB would respond with a failure the facade will send you a failure.
There is no compromisse with performance or even data durability. If you need a durable database, with good performance, you must go with MongoDB. DynamoDB was created to be used on AWS environment only.
Again: There is no such thing like DynamoDB local.

DynamoDB has predefined limits. It is possible that you are running into these limits. Consider increasing WriteCapacityUnits for the table to increase performance. You may also want to increase ReadCapacityUnits for the scan.

Related

Java Stream and foreach

I have a list of maps. The map holds two values "key" and "value" I have to filter out specific values from that list. So I am iterating through the list and if the map has the key that I want, then I take that value and set it in another pojo.
{
teams=["
{key=NAME, value="ANKIT"},
{key=START_DATE, value=2016-09-01}
}
String START_DATE = "START_DATE";
STRING NAME = "NAME";
I have multiple conditions to check. I am doing this using foreach. Can this be done using Java 8 stream().
teamList.forEach(
team -> {
if (NAME.equals(team.get("key"))) {
team.setName(team.get("value"));
} else if (START_DATE.equals(team.get("key"))) {
team.setEndDate(LocalDate.parse(team.get("value")));
}
});
You can't use Java 8 new features to replace your old Java code. I think as everyone was trying to point out that fact what you are trying is not a recommended use of Java 8 streams.
Best practice in Java 8 is to avoid state-full operations. (That means, you are modifying an existing object. Here in this case, you are setting data to a object that resides in memory which had created earlier)
If you still wants to write this simple logic using Java 8 streams, below you can see the POJO class and the main method that sets the Data to that object.
public class POJO {
private Integer age;
private LocalDate birthday;
public LocalDate getBirthday() {
return birthday;
}
public void setBirthday(LocalDate birthday) {
this.birthday = birthday;
}
public Integer getAge() {
return age;
}
public void setAge(Integer age) {
this.age = age;
}
#Override
public String toString() {
return "POJO{" +
"age=" + age +
", birthday=" + birthday +
'}';
}
}
public class MapIteration {
public static Map<String,Integer> ageMap = new HashMap<>();
private static final String NAME = "Doki";
private static final String START_DATE = "31/10/2016";
private static POJO myDataObject = new POJO();
private static BiConsumer<String,Integer> integerConsumer = (k, v) -> {
try {
if(NAME.equals(k.toString()))
myDataObject.setAge(Integer.valueOf(v.toString()));
else if (START_DATE.equals(LocalDate.parse(k.toString()).toString()))
myDataObject.setBirthday(LocalDate.parse(k.toString()));
} catch (NumberFormatException | DateTimeParseException e) {
System.out.println("exception occurred: Because sometimes names cannot be parsed as Date formats");
}
};
static{
ageMap.put("John", 23);
ageMap.put("Norman", 26);
ageMap.put("Micheal", 25);
ageMap.put("Doki", 22);
}
public static void main(String[] args){
ageMap.forEach(integerConsumer);
System.out.println(myDataObject);
}
}

Multiple products in a single client(arrayList)

I've been asked to program a small online sales aplication.
It sounds very simple in theory (but it's been a hell for me). I'm just supposed to have an arrayList with about 5 products and then have a client buy 1 to 5 products and print the sales total.
public class Product {
private String name;
private double price;
public Product(String name, double price) {
this.name = name;
this.price = price;
}
public String getName() {
return name;
}
public double getPrice() {
return price;
}
public String printInfo() {
return "Product: " + name + " Cost: " + price;
}
}
Then I have a client class:
public class Cliente {
private String name;
private int numPedido;
ArrayList<Producto> products = new ArrayList<Producto>();
public void listBuilder() {
Producto shirt = new Producto("Shirt", 30);
Producto tshirt = new Producto("T-Shirt", 40);
Producto sweater = new Producto("Sweater", 50);
}
public Cliente(String name, int numPedido) {
this.name = name;
this.numPedido = numPedido;
}
public Cliente() {
}
public String getName() {
return name;
}
public int getNumPedido() {
return (int) (Math.random() * 100);
}
public void addNewClient() {
name = JOptionPane.showInputDialog("Nombre: ");
}
public String printInfo() {
return "Nombre: " + name;
}
}
Right now I'm stuck thinking on how to make a client select a product and get that attached to him. I was thinking on making an arrayList of an arrayList but I'm sure that would complicate things. I know there is probably an easier way to connect them but I can't think of any. The option I have in mind is a method which shows numbers from 1 to 3(corresponding to each product) and when the user picks one it should return the price of the item.
Still not sure how to implement it in a way that the user can pick multiple products.
EDIT:
I also have an admin class that goes like this:
public class Admin {
private Client[] clientList;
public AdminPedidos() {
clientList = new Client[2];
}
public void AddContact() {
clienteList[0] = addProduct();
clienteList[1] = addProduct();
fillList();
}
public Cliente addProduct() {
String contactoString = JOptionPane.showInputDialog("Are you a new client? Press 1 if yes.");
if (contactoString.equals("1")) {
return new Cliente();
} else {
return new Cliente(); //just for testing
}
}
private void fillList() {
for (Client i : clientList) {
i.addNewClient();
}
}
public void printContact() {
for (Client i : clientList) {
System.out.println(i.printInfo());
}
}
}
You can have some purchaseProduct method attached to each Client.
public void purchaseProduct(Product product) { this.products.add(product); }
Then each Client you instantiate (Client client = new Client(name, id);) can add Products to his/her cart with the purchaseProduct method.
I'm assuming you are using some kind of user input method (Scanner). With that you can read the user's input of which Product they want and accordingly call the function with the right Product.
The listBuilder function doesn't quite make sense to me btw (and after your edit, it's really hard to make sense of what the Admin class should be/represent).
Edit: You would probably want to create an ArrayList<Product> which will be attached to each client, which you already have. I sense that you have a difficulty deciding where to put your actual Products. You should not put them inside your Client class for sure.
You should think about who/where they are going to be used. Probably in main right? So just instantiate them there first and then the Client could choose which one to purchase (via the method I introduced before):
client.purchaseProduct(product);

Cannot Save Multiple Objects To DynamoDB?

I'm attempting to add multiple entries to my DynamoDB database through a service, but when the code is executed, Only one entry is saved. This is what I have done so far:
public int onStartCommand(Intent intent, int flags, int startId) {
CognitoCachingCredentialsProvider credentialsProvider = new CognitoCachingCredentialsProvider(
getApplicationContext(), // Context
"us-east-1:851c121e-326d-4c9e-be47-fec40eb7b693", // Identity Pool ID
Regions.US_EAST_1 // Region
);
AmazonDynamoDB client = new AmazonDynamoDBClient(credentialsProvider);
Toast.makeText(this, "Adding To Database...", Toast.LENGTH_LONG).show();
CognitoCachingCredentialsProvider cognitoProvider = new CognitoCachingCredentialsProvider(
getApplicationContext(), // Context
"us-east-1:851c121e-326d-4c9e-be47-fec40eb7b693", // Identity Pool ID
Regions.US_EAST_1 // Region
);
AmazonDynamoDBClient ddbClient = new AmazonDynamoDBClient(cognitoProvider);
DynamoDBMapper mapper = new DynamoDBMapper(ddbClient);
String streamName = intent.getStringExtra("streamName");
String userCreated = intent.getStringExtra("userCreated");
ArrayList<String> selectedFriend = intent.getStringArrayListExtra("userInvolved");
intent.getStringExtra("comment");
for(int i = 0; i < selectedFriend.size(); i++){
Streams streams = new Streams();
streams.setStreamname(streamName);
streams.setUsercreated(userCreated);
streams.setUserinvolved(selectedFriend.get(i));
streams.setUnread(true);
streams.setDeleted(false);
mapper.save(streams);
}
}
And like i said, only one entry will be saved. I was suggested to use an arraylist, but I'm not sure how to go about doing that. Can anyone help?
All help is appreciated.
EDIT: The Streams.java class:
#DynamoDBTable(tableName = "Streams")
public class Streams{
private String streamname;
private String usercreated;
private String userinvolved;
private boolean unread;
private boolean deleted;
#DynamoDBAttribute(attributeName = "Stream_Name")
public String getStreamname() {
return streamname;
}
public void setStreamname(String streamname) {
this.streamname = streamname;
}
#DynamoDBHashKey(attributeName = "User_Created")
public String getUsercreated() {
return usercreated;
}
public void setUsercreated(String usercreated) {
this.usercreated = usercreated;
}
#DynamoDBAttribute(attributeName = "User_Involved")
public String getUserinvolved() {
return userinvolved;
}
public void setUserinvolved(String userinvolved) {
this.userinvolved = userinvolved;
}
#DynamoDBAttribute(attributeName = "Unread")
public boolean isUnread() {
return unread;
}
public void setUnread(boolean unread) {
this.unread = unread;
}
#DynamoDBAttribute(attributeName = "Deleted")
public boolean isDeleted() {
return deleted;
}
public void setDeleted(boolean deleted) {
this.deleted = deleted;
}
}
It seems your DynamoDB hash key is the user created field in the Stream class. This is the DynamoDB primary key which uniquely identifies a single item in the DynamoDB table.
Each time you call mapper.save(streams), you are overwriting the item you wrote in the previous iteration of the loop since it seems you only set the user created field once here:
String userCreated = intent.getStringExtra("userCreated");.
Try using a unique identifier (e.g. stream name as the hash key). If multiple items have the same hash key you can also use a schema with a range key to create a (hash key, range key) primary key to uniquely identify an item in the DynamoDB table.
DynamodbMapper has a batch save method that allows you to save multiple objects in a batch. Check out the DynamoDBMapper.batchSave.

How To Pass a JRBeanCollectionDataSource to iReport?

I'm currently trying to use jasper to help me create reports. I have the information and data that I want displayed in this method:
private void writeToFile(final List<ScenarioLoadModel> sceneLoadModel) throws Exception {
final BufferedWriter bw = new BufferedWriter(new FileWriter("/Uma/nft/result.psv"));
for (final ScenarioLoadModel slm : sceneLoadModel) {
bw.write(slm.getScenarioId() + PSP + slm.getScenarioId() + PSP + slm.getScenarioConfig().getName() + PSP + slm.getLoad() + PSP + "" + EOL);
if (!slm.getScenarios().isEmpty()) {
final int tempCount = slm.getScenarios().get(0).getTemplates().size();
final int sceneCount = slm.getScenarios().size();
for (int tempIdx = 0; tempIdx < tempCount; tempIdx++) {
String id = null;
int pass = 0;
int fail = 0;
final Map<String, BigDecimal> metricMap = new HashMap<String, BigDecimal>();
final DefaultStatisticalCategoryDataset dataset = new DefaultStatisticalCategoryDataset();
for (int sceneIdx = 0; sceneIdx < sceneCount; sceneIdx++) {
final Template temp = slm.getScenarios().get(sceneIdx).getTemplates().get(tempIdx);
if (temp.isError()) {
fail++;
} else {
pass++;
}
if (sceneIdx == 0) {
id = temp.getId();
}
final MetricGroupModel mgm = slm.getScenarios().get(sceneIdx).getMetricGroupModel().get(tempIdx);
if (mgm != null) {
for (final MetricModel mm : mgm.getMetricModel()) {
for (final MetricValue mv : mm.getMetricValue()) {
dataset.add(mv.getValue(), new BigDecimal(0.0), mv.getType(), id);
}
}
}
}
final TemplateConfig tc = TemplateManager.getTemplateConfig(id);
bw.write(slm.getScenarioId() + PSP);
bw.write(id + PSP + tc.getName() + PSP + 1 + PSP + pass + "/" + fail);
for (final Object row : dataset.getRowKeys()) {
final Number mean = dataset.getValue((String) row, id);
bw.write(PSP + row + PSP + mean);
}
bw.write(EOL);
}
}
}
bw.close();
}
From my understanding I create Beans and then put them all in a Bean Factory, to create my object that will be ready to be passed to iReport.
How can I put all this information into a Bean? I essentially want the bean to include the scenario/test case and whether or not it passed. (This is for test automation)
I tried to read your code to make a a best guess at what columns you would want, but with no context, I have no clue. All the bean is a pojo, with private fields and public getters and setters.
Assuming there is no grouping and essentially each ScenarioLoadModel will correspond to one row in the report you would end up with a bean like this:
public class ScenariaResults {
private String id;
private String name;
private String load;
private int passCount;
private int failCount;
public ScenariaResults(String id, String name, String load, int passCount,
int failCount) {
super();
this.id = id;
this.name = name;
this.load = load;
this.passCount = passCount;
this.failCount = failCount;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getLoad() {
return load;
}
public void setLoad(String load) {
this.load = load;
}
public int getPassCount() {
return passCount;
}
public void setPassCount(int passCount) {
this.passCount = passCount;
}
public int getFailCount() {
return failCount;
}
public void setFailCount(int failCount) {
this.failCount = failCount;
}
#Override
public String toString() {
return "ScenariaResults [id=" + id + ", name=" + name + ", load="
+ load + ", passCount=" + passCount + ", failCount="
+ failCount + "]";
}
}
So basically in the code you have above you build instances of ScenarioResults and add them to a list. Once you have the list, all you need to do is create a JRDataSource:
List<ScenarioResults> dataBeanList = ...call your method to get the list of results
//create the datasource
JRDataSource dataSource = new JRBeanCollectionDataSource(dataBeanList);
Now when designing the report in iReport it can be a little tricky to get the fields imported automatically. Basically first add your project with the bean to the classpath in iReports (could just point it to the bin folder or jar file`): Tools -> options -> classpath tab. Now follow these steps to add the fields.
Click the following icon:
Select the JavaBean Datasource tab.
Enter the classname of your bean. (ex. ScenarioResults)
Click Read attributes
Highlight the fields you want in the report and click Add Selected Field(s).
Click OK.
Now if you want to test what the report looks like with data, and not just an empty datasource, this is where the Factory comes in. It is only for testing while using iReport. You need to create a class that will essentially create a dummy data set for you. It should look something like:
import java.util.ArrayList;
import java.util.List;
public class ScenarioResultsFactory {
public static List<ScenarioResults> createBeanCollection() {
List<ScenarioResults> list = new ArrayList<ScenarioResults>();
list.add(new ScenarioResults("1", "test", "load", 10, 5));
//add as many as you want
return list;
}
}
Now you need to create a Datasource pointing to it in iReport.
Next to the Datasource dropdown in the toolbar click the icon with the tooltip `Report Datasources.
Click New.
Select JavaBeans set datasource. Click Next.
For name enter ScenarioResultsFactory.
For the Factory class you need to put the classname including package. So if the class is in the com package you should have com.ScenarioResultsFactory here.
For the static method put createBeanCollection if not already there.
Check the Use field description check box. Click Test to make sure it worked.
Click Save.

Why getStamp() return the same value?

I read this code in Thinking in Java and get puzzled:
package generics;
//: generics/Mixins.java
import java.util.*;
interface TimeStamped { long getStamp(); }
class TimeStampedImp implements TimeStamped {
private final long timeStamp;
public TimeStampedImp() {
timeStamp = new Date().getTime();
}
public long getStamp() { return timeStamp; }
}
interface SerialNumbered { long getSerialNumber(); }
class SerialNumberedImp implements SerialNumbered {
private static long counter = 1;
private final long serialNumber = counter++;
public long getSerialNumber() { return serialNumber; }
}
interface Basic {
public void set(String val);
public String get();
}
class BasicImp implements Basic {
private String value;
public void set(String val) { value = val; }
public String get() { return value; }
}
class Mixin extends BasicImp
implements TimeStamped, SerialNumbered {
private TimeStamped timeStamp = new TimeStampedImp();
private SerialNumbered serialNumber =
new SerialNumberedImp();
public long getStamp() { return timeStamp.getStamp(); }
public long getSerialNumber() {
return serialNumber.getSerialNumber();
}
}
public class Mixins {
public static void main(String[] args) {
Mixin mixin1 = new Mixin(), mixin2 = new Mixin();
mixin1.set("test string 1");
mixin2.set("test string 2");
System.out.println(mixin1.get() + " " +
mixin1.getStamp() + " " + mixin1.getSerialNumber());
System.out.println(mixin2.get() + " " +
mixin2.getStamp() + " " + mixin2.getSerialNumber());
while(true)System.out.println(new Date().getTime());
}
} /* Output: (Sample)
test string 1 1132437151359 1
test string 2 1132437151359 2
*///:~
Why are the values returned of getStamp() the same? (1132437151359 == 1132437151359)?
Two objects are created and they have different propoties created in different time, so Why?
The expression new Date().getTime() is a slow way of doing System.currentTimeMillis() which has a minimum resolution of one milli-seconds (but can be as much as 16 ms on some OSes)
This means if the method is called less than one milli-second apart it can give the same result.
A better option is to use AtomicLong.getAndIncrement() for ids.
Using time for serial numbers is not a good idea. The reason you're getting the same time is probably because the code runs rather quickly and enough time doesn't elapse between instantiation of the first object and the second. The time stamp is returned in milliseconds and so if the instantiation of both objects is within 1ms of each other, you won't see a difference.
If you increase load on the system, you might see a difference, or if you use Thread.sleep(5) to cause your program to pause. Both approaches aren't very good.
Instead of using the time for a unique id, use UUID.
Try something like this:
Mixin mixin1 = new Mixin();
Thread.sleep(10);
Mixin mixin2 = new Mixin();
Now you got 10 ms pause in the process of creating those 2 objects.
Your class is simple and you have fast computer so distance in time between two instantations is so small that Java can't see it.

Categories

Resources