I am trying to see if the next token in a string is equivalent to map key, and adding a message to an array list.
ArrayList<String> trace = new ArrayList<String>();
if(!element.startsWith("PRINT")) {
while(st.hasMoreTokens()) {
tokens.add(st.nextToken());
}
for(String key: expression.keySet())
if(st.nextToken() == key)
trace.add(key + "Changed from " + expression.get(key) + " to " + tokens.get(2));
expression.put(tokens.get(0),Integer.parseInt(tokens.get(2)));
tokens.clear();
}
ERROR MESSAGE
Exception in thread "main" java.util.NoSuchElementException
at java.util.StringTokenizer.nextToken(StringTokenizer.java:349)
at Commander.main(Commander.java:45)
The problem is with st.nextToken() call.
You are checking st.hasMoreTokens() only once but calling st.nextToken() twice. And it is throwing the error.
During 1st while loop you are reading all the tokens:
while(st.hasMoreTokens()) {
tokens.add(st.nextToken());
}
Since 'all' the tokens are already read, now when you try to read the token again inside the for loop you will get the error.
if(st.nextToken() == key) {
trace.add(key + "Changed from " + expression.get(key) + " to " + tokens.get(2));
}
There is also another potential problem. You are trying to convert the 3rd token to an Integer. It might fail there as well:
// Value of tokens.get(2) is "string"
Integer.parseInt(tokens.get(2))
You will get error here also. It would be good to check if tokens.get(2) is of numeric type or not. like:
int value = 0;
if (StringUtils.isNumeric(tokens.get(2))) {
value = Integer.parseInt(tokens.get(2));
}
// Default value will be 0.
expression.put(tokens.get(0), value);
Reference: StringUtils.isNumeric()
So, the modified code will be like:
ArrayList<String> trace = new ArrayList<String>();
if(!element.startsWith("PRINT")) {
List<String> tokens = new ArrayList<String>();
while(st.hasMoreTokens()) {
tokens.add(st.nextToken());
}
// Size of tokens has to be more then 3, otherwise you will get another error.
if (tokens.size() >= 3) {
for(String key: expression.keySet()) {
if(tokens.get(0).equals(key)) {
// Do your other operations...
trace.add(key + "Changed from " + expression.get(key) + " to " + tokens.get(2));
}
}
// Do some more operations ...
int value = 0;
if (StringUtils.isNumeric(tokens.get(2))) {
value = Integer.parseInt(tokens.get(2));
}
// Default value will be 0.
expression.put(tokens.get(0), value);
}
tokens.clear();
}
Related
My problem is in the part where I'm doing the "if/else" conditions, when I call the function that will perform the comparisons and will define if the test passed or not and will send some information, I'm receiving null.
Problems are among the asterisks. If anyone can help me
This is my code :
public static void fxSpot_GBP_JPY(TradeData data, TradeData output) throws Exception {
if (data == null) {
fail("The input data object was not correctly filled");
}
if (output == null) {
fail("The output data object was not correctly filled");
}
//Used to set the comment, the status and update to JIRA
FieldsJSON fields = new FieldsJSON();
String assertionError = "";
List<String> inputs = new ArrayList<String>();
List<String> outputs = new ArrayList<String>();
String newDate = Utils.formatTimeZoneMinute(data.getTradeDate());
String asOfTrade = Utils.formatTimeZoneMinute(data.getAsOfTradeDate());
String executionDate = Utils.formatTimeZoneMinute(output.getExecutionDateTime());
try {
//Add the data in the list
inputs.add(data.getTransactionNumber()); outputs.add(output.getBloombergId());
inputs.add(newDate); outputs.add(output.getTradeDate());
inputs.add(asOfTrade); outputs.add(executionDate);
inputs.add(data.getSettlementDate()); outputs.add(output.getValueDate());
inputs.add(data.getTradeAmount()); outputs.add(output.getAmount2());
inputs.add(data.getCustomerAccountCounterparty()); outputs.add(output.getMiPartyId());
inputs.add(data.getPrincipalLoanAmount()); outputs.add(output.getAmount());
inputs.add(data.getSecurityPrice()); outputs.add(output.getRate());
inputs.add(data.getISOCodeOf1stCurrency()); outputs.add("BRL");//output.getCurrency2()
inputs.add(data.getISOCodeOf2ndCurrency()); outputs.add(output.getCurrency1());
//Compare values
System.out.println("-------------------");
int y = 0;
int x = 0;
for(String input : inputs) {
for(String out : outputs) {
if(y == x) {
if(input.equals(out)) {
WriterCSV.setOk("Ok");
**String comment = input + " = " + out;
fields.setComment(comment);
fields.setStatus("PASS");**
System.out.println("ok - " + input + " = " + out);
}else {
WriterCSV.setOk("not Ok");
**String comment = input + " = " + out;
fields.setComment(comment);
fields.setStatus("FAIL");**
System.out.println("not Ok - " + input + " = " + out);
}
}
x = x+1; // count of the list of output
}
y = y+1; // count of the list of inputs
x = 0; // reset to 0 the count of outputs
}
// evidence with the name and value of fields compared
WriterCSV.reportSpot_CSV(data,output);
}
Here is my test:
#Test
#Tag("compare")
public void CompareSpot() throws Exception {
//Create a list to read the CSVfile
List<DTOTradeData> dto;
//Used to get the TradeData of list dto.
DTOTradeData dtd = new DTOTradeData();
// Read a csvFile and return a list with the values to new xml
dto = CSVReader.readCSV("spot.csv");
//The xpath of xml
FileDriverSpot spot = new FileDriverSpot();
FileDriver output = new FileDriverSpotOutput();
FieldsJSON fields = new FieldsJSON();
//new xml = dataInput and the outputFile = dataOutput
TradeData dataInput = new TradeData();
TradeData dataOutput = new TradeData();
for (int i = 0; i < dto.size(); i++) {
dtd = dto.get(i); // get TradeData
dtd.getTradeData().setDriver(spot); // set the driver
if (fileExist(Setup.xmlPath + dtd.getInputFile() + ".xml")) {
dataInput = Reader.read(spot, Setup.xmlPath + dtd.getInputFile() + ".xml");
dataOutput = Reader.read(output, Setup.spotPath + dtd.getOutputFile());
try {
// make the comparison
**FunctionalTest.fxSpot_GBP_JPY(dataInput, dataOutput);**
}
catch(AssertionError e) {
String comment = e.toString();
fields.setComment(comment);
}
} else {
fail("The file: " + dtd.getTemplateFile()
+ " needs to go through the writing process before being compared.");
}
//Convert the file to base64
String inputData = UpdateTestStatus.convertToBase64(Setup.xmlPath + dtd.getInputFile() + ".xml");
String outputData = UpdateTestStatus.convertToBase64(Setup.spotPath + dtd.getOutputFile());
String evidenceCompared = UpdateTestStatus.convertToBase64(Setup.reportPath+"ReportSpot.csv");
System.out.println(UpdateTestStatus.updateTestRun(**fields.getStatus(),fields.getComment()**,
inputData,dtd.getInputFile()+ ".xml", //data of the XML and the name of the file
outputData,dtd.getOutputFile(),
evidenceCompared,"ReportSpot.csv",
Setup.testExec, dtd.getJiraId()).asString()); // ID testExecution and ID of
}
}
The test and the code under test each create a separate instance of FieldsJSON. Data set in one instance will not be visible in the other (unless the data is declared static, in which case there's no need to create instances).
You can fix this by using a single instance, either passed to the fxSpot_GBP_JPY method from the test, or returned from that method to the test.
The end result of the code is to be able to parse the string array to integer array but as you can see I get an exception. I tried to split the [ and ] signs but when I call the first (0) element splited_game_result[0] of the array, it's returns nothing. What is the solution here? I tried using trim method.
String[] ca_po_pe = {"4:7","PAUSE","2:0","1:3 PINE","PAUSE","CANCEL","2:4","0:5 PINE","PAUSE","CANCEL"};
String canc_postp_pen_games = "";
boolean confirming = true;
for(String looped_games : ca_po_pe) {
if(confirming) { canc_postp_pen_games+=looped_games; confirming=false; }
else { canc_postp_pen_games+=", "+looped_games; } }
System.out.println("LOOPED FINAL RESULT GAMES TO INSERT COMMAS: " + "\n" + canc_postp_pen_games + "\n");
String[] fin_games_res = canc_postp_pen_games.split("[,]");
ArrayList<String> arraylist= new ArrayList<String>();
Collections.addAll(arraylist, fin_games_res);
for (String str: arraylist) {
}
System.out.println("ELEMENTS ADDED TO ARRAYLIST: " + arraylist + "\n");
int noItems = arraylist.size();
for (int i = 0; i < noItems; i++) {
String currItem = arraylist.get(i);
if (currItem.contains("PAUSE")) {
arraylist.add(new String("400"));
noItems++; }
if (currItem.contains("CANCEL")) {
arraylist.add(new String("300"));
noItems++; }
if (currItem.contains(" PINE")) {
arraylist.add(new String("500"));
noItems++; }
}
System.out.println("ELEMENTS BEFORE REMOVAL: " + "\n" + arraylist + "\n");
Iterator<String> iter_getting = arraylist.iterator();
while(iter_getting.hasNext()) {
if(iter_getting.next().contains("PAUSE")){
iter_getting.remove(); }}
Iterator<String> iter_getting1 = arraylist.iterator();
while(iter_getting1.hasNext()) {
if(iter_getting1.next().contains("CANCEL")){
iter_getting1.remove(); }}
Iterator<String> iter_getting2 = arraylist.iterator();
while(iter_getting2.hasNext()) {
if(iter_getting2.next().contains(" PINE")){
iter_getting2.remove(); }}
System.out.println("ELEMENTS AFTER REMOVAL: " + "\n" + arraylist);
System.out.println("ELEMENT ON INDEX 3 BEFORE CONVERTION TO ARRAY: " + "\n" + arraylist.get(3));
String convert = arraylist.toString();
System.out.println("CONVERTED STRING: " + convert);
String[]splited_game_result = convert.trim().split("[\\[:,\\]]");
System.out.println(" AFTER SPLITING TO ARRAY: " + "\n" + splited_game_result[0]);
Integer[] final_result = new Integer[splited_game_result.length];
int g = 0;
for(String fsgr : splited_game_result)
{ final_result[g] = Integer.parseInt(fsgr); g++;}
System.out.println("INTEGER ELEMENT ON INDEX 0: " + "\n" + final_result[1]);
For everybody who wonders, exception is raised in the very end of the source. It says NumberFormatException: For input string: "". See source below:
for(String fsgr : splited_game_result)
{ final_result[g] = Integer.parseInt(fsgr); g++;}
The reason is in the regular expession author used to split the array:
String[]splited_game_result = convert.trim().split("[\\[:,\\]]");
It produces the following array of strings:
["", "4", "7", " 2", "0", " 2", "4", " 400", " 500", " 400", " 300", " 500", " 400", " 300"]
Issue is that Integer.parseInt requires string that contain parsable integer
Possible workaround is to add if condition to ensure your fsgr is not an empty string and doesn't contain whitespaces:
for(String fsgr : splited_game_result) {
String fsgrTrim = fsgr.trim();
if (!fsgrTrim.isEmpty()) {
final_result[g] = Integer.parseInt(fsgrTrim);
g++;
}
}
Or to add try-catch clause:
for(String fsgr : splited_game_result) {
try {
final_result[g] = Integer.parseInt(fsgr);
g++;
} catch (Exception e) {
e.printStackTrace();
}
}
I still don't know what you're trying to do, but here is your exception. You are parsing an empty string and trying to parse a space before the number. Put comments in for your logic.
for (String fsgr : splited_game_result) {
try {
final_result[g] = Integer.parseInt(fsgr);
g++;
} catch(Exception e2) {
System.out.println("Exception = ["+ fsgr + "]");
}
}
Output (in middle of put. Use first and last printlns as location)
CONVERTED STRING: [4:7, 2:0, 2:4, 400, 500, 400, 300, 500, 400, 300]
AFTER SPLITING TO ARRAY:
Exception = []
Exception = [ 2]
Exception = [ 2]
Exception = [ 400]
Exception = [ 500]
Exception = [ 400]
Exception = [ 300]
Exception = [ 500]
Exception = [ 400]
Exception = [ 300]
INTEGER ELEMENT ON INDEX 0:
7
public String generateDataPDF() {
System.out.println("Inside generate PDF");
String filePath = "";
HttpSession sess = ServletActionContext.getRequest().getSession();
try {
sess.setAttribute("msg", "");
if (getCrnListType().equalsIgnoreCase("F")) {
try {
filePath = getModulePath("CRNLIST_BASE_LOCATION") + File.separator + getCrnFileFileName();
System.out.println("File stored path : " + filePath);
target = new File(filePath);
FileUtils.copyFile(crnFile, target);
} catch (Exception e) {
System.out.println("File path Exception " + e);
}
}
System.out.println("Values from jsp are : 1)Mode of Generation : " + getCrnListType() + " 2)Policy Number : " + getCrnNumber() + " 3)Uploaded File Name : " + getCrnFileFileName() + " 4)LogoType : " + getLogoType()
+ " 5)Output Path : " + getOutputPath() + " 6)Type of Generation : " + getOptionId() + " 7)PDF Name : " + getPdfName());
String srtVAL = "";
String arrayVaue[] = new String[]{getCrnListType(), getCrnListType().equalsIgnoreCase("S") ? getCrnNumber() : filePath, getLogoType().equalsIgnoreCase("WL") ? "0" : "1",
getOutputPath(), getGenMode(), getRenType()};
//INS DB Connection
con = getInsjdbcConnection();
ArrayList selectedCRNList = new ArrayList();
String selectedCRNStr = "";
selectedCRNStr = getSelectedVal(selectedCRNStr, arrayVaue[1]);
String[] fileRes = selectedCRNStr.split("\\,");
if (fileRes[0].equalsIgnoreCase("FAIL")) {
System.out.println("fileRes is FAIL beacause of other extension file.");
sess.setAttribute("pr", "Please upload xls or csv file.");
return SUCCESS;
}
System.out.println("List file is : " + selectedCRNStr);
String st[] = srtVAL.split("[*]");
String billDateStr = DateUtil.getStrDateProc(new Date());
Timestamp strtPasrsingTm = new Timestamp(new Date().getTime());
String minAMPM = DateUtil.getTimeDate(new Date());
String str = "";
String batchID = callSequence();
try {
System.out.println("Inside Multiple policy Generation.");
String userName=sess.getAttribute("loginName").toString();
String list = getProcessesdList(userName);
if (list != null) {
System.out.println("list is not null Users previous data is processing.....");
//setTotalPDFgNERATEDmSG("Data is processing please wait.");
sess.setAttribute("pr","Batch Id "+list+" for User " + userName + " is currently running.Please wait till this Process complete.");
return SUCCESS;
}
String[] policyNo = selectedCRNStr.split("\\,");
int l = 0, f = 0,counter=1;
for (int j = 0; j < policyNo.length; j++,counter++) {
String pdfFileName = "";
int uniqueId=counter;
globUniqueId=uniqueId;
insertData(batchID, new Date(), policyNo[j], getOptionId(), userName,uniqueId);
System.out.println("Executing Proc one by one.");
System.out.println("policyNo[j]" + policyNo[j]);
System.out.println("getOptionId()" + getOptionId());
System.out.println("seqValue i.e batchId : " + batchID);
}
str = callProcedure(policyNo[j], getOptionId(), batchID);
String[] procResponse = str.split("\\|");
for (int i = 0; i < procResponse.length; i++) {
System.out.println("Response is : " + procResponse[i]);
}
if (procResponse[0].equals("SUCCESS")) {
Generator gen = new Generator();
if (getPdfName().equalsIgnoreCase("true")) {
System.out.println("Checkbox is click i.e true");
pdfFileName = procResponse[1];
} else {
System.out.println("Checkbox is not click i.e false");
String POLICY_SCH_GEN_PSS = getDetailsForFileName(userName, policyNo[j], batchID);
String[] fileName = POLICY_SCH_GEN_PSS.split("\\|");
if (getLogoType().equals("0") || getLogoType().equals("2")) {
System.out.println("If logo is O or 1");
pdfFileName = fileName[1];
} else if (getLogoType().equals("1")) {
System.out.println("If logo is 2");
pdfFileName = fileName[0];
}
}
b1 = gen.genStmt(procResponse[1], procResponse[2], "2", getLogoType(), "0", pdfFileName,"1",userName,batchID);
l++;
updateData(uniqueId,batchID, "Y");
} else {
f++;
updateData(uniqueId,batchID, "F");
}
}
sess.setAttribute("pr","Total "+l+" "+getGenericModulePath("PDF_RES1") + " " + " " + getGenericModulePath("PDF_RES2") + " " + f);
}catch (Exception e) {
updateData(globUniqueId,batchID, "F");
System.out.println("Exception in procedure call");
setTotalPDFgNERATEDmSG("Fail");
e.printStackTrace();
sess.setAttribute("pr", "Server Error.");
return SUCCESS;
}
}catch (Exception ex) {
ex.printStackTrace();
sess.setAttribute("pr", "Server Error.");
return SUCCESS;
}
System.out.println("Above second return");
return SUCCESS;
}
GenerateDataPDf method generates PDF based on the parameters i.e ProductType(GenMode),CrnList(uploaded in excel file...)Code works fine when only single user generates PDF. But If two different User(User and roles are assigned in application) start the process same time request paraeters are overridden then! Suppose first user request pdf for 50 customers for product 1. User1's process is still running and second user request for product2. Now User1's pdf are generated but for product2.....! Here batchId is unique for every single request.One table is maintained where batch_id,all pdf,generation flags are mainained there. How do I solve this?
As per your comment, this is what I would do, It's probably not the best way to do !
Firstly : Create a function to collet all your data at the beginning. You should not modify/update/create anything when you are generating a PDF. IE : array/list collectPDFData() wich should retourn an array/list.
Secondly : Use a synchronized methods like synchronized boolean generatePDF(array/list)
"Synchronized" methods use monitor lock or intrinsic lock in order to manage synchronization so when using synchronized, each method share the same monitor of the corresponding object.
NB : If you use Synchronize, it's probably useless to collect all your data in a separate way, but I think it's a good practice to make small function dedicated to a specific task.
Thus, your code should be refactored a little bit.
CODE
ArrayList<String> tokens = new ArrayList<String>();
ArrayList<String> PRINT = new ArrayList<String>();
String tok = "";
for(String element : list) {
StringTokenizer st = new StringTokenizer(element);
if(!element.startsWith("PRINT")) {
while(st.hasMoreTokens()) {
tok = st.nextToken();
tokens.add(tok);
for(String key : expression.keySet())
if(tok.equals(key))
System.out.println(tokens.get(0) + " changed from " + expression.get(key) + " to " + Integer.parseInt(tokens.get(2)));
}
expression.put(tokens.get(0),Integer.parseInt(tokens.get(2)));
tokens.clear();
}
I am getting an array out of bounds exception at below line
System.out.println(tokens.get(0) + " changed from " + expression.get(key) + " to " + Integer.parseInt(tokens.get(2))
I know that tokens.get(0) = "A", tokens.get(1) = "=" and tokens.get(2) = "7"
Instead of code
for(String element : list) {
StringTokenizer st = new StringTokenizer(element);
if(!element.startsWith("PRINT")) {
while(st.hasMoreTokens()) {
tok = st.nextToken();
tokens.add(tok);
Use below code
for(String element : list) {
for(int i=0;i<element.length();i++){
tokens.add(element.charAt(i)+"");
}
The problem is that the first time you add a token to tokens there will be only one item, however your for loop requires 3 (at indexes 0, 1 and 2).
I'm trying to use the DynamoDB Parallel Scan Example:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelJavaScanning.html
I have 200,000 items, and I've taken the sequential code scan, and modified it slightly for my usage:
Map<String, AttributeValue> lastKeyEvaluated = null;
do
{
ScanRequest scanRequest = new ScanRequest()
.withTableName(tableName)
.withExclusiveStartKey(lastKeyEvaluated);
ScanResult result = client.scan(scanRequest);
double counter = 0;
for(Map<String, AttributeValue> item : result.getItems())
{
itemSerialize.add("Set:"+counter);
for (Map.Entry<String, AttributeValue> getItem : item.entrySet())
{
String attributeName = getItem.getKey();
AttributeValue value = getItem.getValue();
itemSerialize.add(attributeName
+ (value.getS() == null ? "" : ":" + value.getS())
+ (value.getN() == null ? "" : ":" + value.getN())
+ (value.getB() == null ? "" : ":" + value.getB())
+ (value.getSS() == null ? "" : ":" + value.getSS())
+ (value.getNS() == null ? "" : ":" + value.getNS())
+ (value.getBS() == null ? "" : ":" + value.getBS()));
}
counter += 1;
}
lastKeyEvaluated = result.getLastEvaluatedKey();
}
while(lastKeyEvaluated != null);
The counter gives exactly 200,000 when this code has finished, however, I also wanted to try the parallel scan.
Function Call:
ScanSegmentTask task = null;
ArrayList<String> list = new ArrayList<String>();
try
{
ExecutorService executor = Executors.newFixedThreadPool(numberOfThreads);
int totalSegments = numberOfThreads;
for (int segment = 0; segment < totalSegments; segment++)
{
// Runnable task that will only scan one segment
task = new ScanSegmentTask(tableName, itemLimit, totalSegments, segment, list);
// Execute the task
executor.execute(task);
}
shutDownExecutorService(executor);
}
.......Catches something if error
return list;
Class:
I have a static list that the data is shared with all the threads. I was able to retrieve the lists, and output the amount of data.
// Runnable task for scanning a single segment of a DynamoDB table
private static class ScanSegmentTask implements Runnable
{
// DynamoDB table to scan
private String tableName;
// number of items each scan request should return
private int itemLimit;
// Total number of segments
// Equals to total number of threads scanning the table in parallel
private int totalSegments;
// Segment that will be scanned with by this task
private int segment;
static ArrayList<String> list_2;
Object lock = new Object();
public ScanSegmentTask(String tableName, int itemLimit, int totalSegments, int segment, ArrayList<String> list)
{
this.tableName = tableName;
this.itemLimit = itemLimit;
this.totalSegments = totalSegments;
this.segment = segment;
list_2 = list;
}
public void run()
{
System.out.println("Scanning " + tableName + " segment " + segment + " out of " + totalSegments + " segments " + itemLimit + " items at a time...");
Map<String, AttributeValue> exclusiveStartKey = null;
int totalScannedItemCount = 0;
int totalScanRequestCount = 0;
int counter = 0;
try
{
while(true)
{
ScanRequest scanRequest = new ScanRequest()
.withTableName(tableName)
.withLimit(itemLimit)
.withExclusiveStartKey(exclusiveStartKey)
.withTotalSegments(totalSegments)
.withSegment(segment);
ScanResult result = client.scan(scanRequest);
totalScanRequestCount++;
totalScannedItemCount += result.getScannedCount();
synchronized(lock)
{
for(Map<String, AttributeValue> item : result.getItems())
{
list_2.add("Set:"+counter);
for (Map.Entry<String, AttributeValue> getItem : item.entrySet())
{
String attributeName = getItem.getKey();
AttributeValue value = getItem.getValue();
list_2.add(attributeName
+ (value.getS() == null ? "" : ":" + value.getS())
+ (value.getN() == null ? "" : ":" + value.getN())
+ (value.getB() == null ? "" : ":" + value.getB())
+ (value.getSS() == null ? "" : ":" + value.getSS())
+ (value.getNS() == null ? "" : ":" + value.getNS())
+ (value.getBS() == null ? "" : ":" + value.getBS()));
}
counter += 1;
}
}
exclusiveStartKey = result.getLastEvaluatedKey();
if (exclusiveStartKey == null)
{
break;
}
}
}
catch (AmazonServiceException ase)
{
System.err.println(ase.getMessage());
}
finally
{
System.out.println("Scanned " + totalScannedItemCount + " items from segment " + segment + " out of " + totalSegments + " of " + tableName + " with " + totalScanRequestCount + " scan requests");
}
}
}
Executor Service Shut Down:
public static void shutDownExecutorService(ExecutorService executor)
{
executor.shutdown();
try
{
if (!executor.awaitTermination(10, TimeUnit.SECONDS))
{
executor.shutdownNow();
}
}
catch (InterruptedException e)
{
executor.shutdownNow();
Thread.currentThread().interrupt();
}
}
However, the amount of items changes every time I run this piece of code (Varies around 60000 in total, 6000 per threads, with 10 created threads). Removing synchronization does not change the result too.
Is there a bug with the synchronization or with the Amazon AWS API?
Thanks All
EDIT:
The new function call:
ScanSegmentTask task = null;
ArrayList<String> list = new ArrayList<String>();
try
{
ExecutorService executor = Executors.newFixedThreadPool(numberOfThreads);
int totalSegments = numberOfThreads;
for (int segment = 0; segment < totalSegments; segment++)
{
// Runnable task that will only scan one segment
task = new ScanSegmentTask(tableName, itemLimit, totalSegments, segment);
// Execute the task
Future<ArrayList<String>> future = executor.submit(task);
list.addAll(future.get());
}
shutDownExecutorService(executor);
}
The new class:
// Runnable task for scanning a single segment of a DynamoDB table
private static class ScanSegmentTask implements Callable<ArrayList<String>>
{
// DynamoDB table to scan
private String tableName;
// number of items each scan request should return
private int itemLimit;
// Total number of segments
// Equals to total number of threads scanning the table in parallel
private int totalSegments;
// Segment that will be scanned with by this task
private int segment;
ArrayList<String> list_2 = new ArrayList<String>();
static int counter = 0;
public ScanSegmentTask(String tableName, int itemLimit, int totalSegments, int segment)
{
this.tableName = tableName;
this.itemLimit = itemLimit;
this.totalSegments = totalSegments;
this.segment = segment;
}
#SuppressWarnings("finally")
public ArrayList<String> call()
{
System.out.println("Scanning " + tableName + " segment " + segment + " out of " + totalSegments + " segments " + itemLimit + " items at a time...");
Map<String, AttributeValue> exclusiveStartKey = null;
try
{
while(true)
{
ScanRequest scanRequest = new ScanRequest()
.withTableName(tableName)
.withLimit(itemLimit)
.withExclusiveStartKey(exclusiveStartKey)
.withTotalSegments(totalSegments)
.withSegment(segment);
ScanResult result = client.scan(scanRequest);
for(Map<String, AttributeValue> item : result.getItems())
{
list_2.add("Set:"+counter);
for (Map.Entry<String, AttributeValue> getItem : item.entrySet())
{
String attributeName = getItem.getKey();
AttributeValue value = getItem.getValue();
list_2.add(attributeName
+ (value.getS() == null ? "" : ":" + value.getS())
+ (value.getN() == null ? "" : ":" + value.getN())
+ (value.getB() == null ? "" : ":" + value.getB())
+ (value.getSS() == null ? "" : ":" + value.getSS())
+ (value.getNS() == null ? "" : ":" + value.getNS())
+ (value.getBS() == null ? "" : ":" + value.getBS()));
}
counter += 1;
}
exclusiveStartKey = result.getLastEvaluatedKey();
if (exclusiveStartKey == null)
{
break;
}
}
}
catch (AmazonServiceException ase)
{
System.err.println(ase.getMessage());
}
finally
{
return list_2;
}
}
}
Final EDIT:
Function Call:
ScanSegmentTask task = null;
ArrayList<String> list = new ArrayList<String>();
ArrayList<Future<ArrayList<String>>> holdFuture = new ArrayList<Future<ArrayList<String>>>();
try
{
ExecutorService executor = Executors.newFixedThreadPool(numberOfThreads);
int totalSegments = numberOfThreads;
for (int segment = 0; segment < totalSegments; segment++)
{
// Runnable task that will only scan one segment
task = new ScanSegmentTask(tableName, itemLimit, totalSegments, segment);
// Execute the task
Future<ArrayList<String>> future = executor.submit(task);
holdFuture.add(future);
}
for (int i = 0 ; i < holdFuture.size(); i++)
{
boolean flag = false;
while(flag == false)
{
Thread.sleep(1000);
if(holdFuture.get(i).isDone())
{
list.addAll(holdFuture.get(i).get());
flag = true;
}
}
}
shutDownExecutorService(executor);
}
Class:
private static class ScanSegmentTask implements Callable>
{
// DynamoDB table to scan
private String tableName;
// number of items each scan request should return
private int itemLimit;
// Total number of segments
// Equals to total number of threads scanning the table in parallel
private int totalSegments;
// Segment that will be scanned with by this task
private int segment;
ArrayList<String> list_2 = new ArrayList<String>();
static AtomicInteger counter = new AtomicInteger(0);
public ScanSegmentTask(String tableName, int itemLimit, int totalSegments, int segment)
{
this.tableName = tableName;
this.itemLimit = itemLimit;
this.totalSegments = totalSegments;
this.segment = segment;
}
#SuppressWarnings("finally")
public ArrayList<String> call()
{
System.out.println("Scanning " + tableName + " segment " + segment + " out of " + totalSegments + " segments " + itemLimit + " items at a time...");
Map<String, AttributeValue> exclusiveStartKey = null;
try
{
while(true)
{
ScanRequest scanRequest = new ScanRequest()
.withTableName(tableName)
.withLimit(itemLimit)
.withExclusiveStartKey(exclusiveStartKey)
.withTotalSegments(totalSegments)
.withSegment(segment);
ScanResult result = client.scan(scanRequest);
for(Map<String, AttributeValue> item : result.getItems())
{
list_2.add("Set:"+counter);
for (Map.Entry<String, AttributeValue> getItem : item.entrySet())
{
String attributeName = getItem.getKey();
AttributeValue value = getItem.getValue();
list_2.add(attributeName
+ (value.getS() == null ? "" : ":" + value.getS())
+ (value.getN() == null ? "" : ":" + value.getN())
+ (value.getB() == null ? "" : ":" + value.getB())
+ (value.getSS() == null ? "" : ":" + value.getSS())
+ (value.getNS() == null ? "" : ":" + value.getNS())
+ (value.getBS() == null ? "" : ":" + value.getBS()));
}
counter.addAndGet(1);
}
exclusiveStartKey = result.getLastEvaluatedKey();
if (exclusiveStartKey == null)
{
break;
}
}
}
catch (AmazonServiceException ase)
{
System.err.println(ase.getMessage());
}
finally
{
return list_2;
}
}
}
OK, I believe the issue is in the way you synchronized.
In your case, your lock is pretty much pointless, as each thread has its own lock, and so synchronizing never actually blocks one thread from running the same piece of code. I believe that this is the reason that removing synchronization does not change the result -- because it never would have had an effect in the first place.
I believe your issue is in fact due to the static ArrayList<String> that's shared by your threads. This is because ArrayList is actually not thread-safe, and so operations on it are not guaranteed to succeed; as a result, you have to synchronize operations to/from it. Without proper synchronization, it could be possible to have two threads add something to an empty ArrayList, yet have the resulting ArrayList have a size of 1! (or at least if my memory hasn't failed me. I believe this is the case for non-thread-safe objects, though)
As I said before, while you do have a synchronized block, it really isn't doing anything. You could synchronize on list_2, but all that would do is effectively make all your threads run in sequence, as the lock on the ArrayList wouldn't be released until one of your threads was done.
There are a few solutions to this. You can use Collections.synchronizedList(list_2) to create a synchronized wrapper to your ArrayList. This way, adding to the list is guaranteed to succeed. However, this induces a synchronization cost per operations, and so isn't ideal.
What I would do is actually have ScanSegmentTask implement Callable (technically Callable<ArrayList<String>>. The Callable interface is almost exactly like the Runnable interface, except its method is call(), which returns a value.
Why is this important? I think that what would produce the best results for you is this:
Make list_2 an instance variable, initialized to a blank list
Have each thread add to this list exactly as you have done
Return list_2 when you are done
Concatenate each resulting ArrayList<String> to the original ArrayList using addAll()
This way, you have no synchronization overhead to deal with!
This will require a few changes to your executor code. Instead of calling execute(), you'll need to call submit(). This returns a Future object (Future<ArrayList<String>> in your case) that holds the results of the call() method. You'll need to store this into some collection -- an array, ArrayList, doesn't matter.
To retrieve the results, simply loop through the collection of Future objects and call get() (I think). This call will block until the thread that the Future object corresponds to is complete.
I think that's it. While this is more complicated, I think that this is be best performance you're going to get, as with enough threads either CPU contention or your network link will become the bottleneck. Please ask if you have any questions, and I'll update as needed.