DynamoDB scan seems to not do anything - java

Basically, trying to scan a DynamoDB table to correct a data problem (which is why it is a scan):
var tableName = dynamoDbMapperProvider.getTableName(DynamoSnapshot.class, address);
log("Starting re-encryption for table " + tableName);
int migratedSnapshotsCount = 0;
for (var snapshot : dynamoDBMapper.scan(DynamoSnapshot.class, snapshotScanExpression)) {
if (snapshot.getEncryptedKey() != null) {
try {
encryptionService.decrypt(
EnvelopeMessage.of(snapshot.getBinaryData(), snapshot.getEncryptedKey(), snapshot.getCmkId())
);
// decryption okay so just set Cmk to configured value as a side-effect...
log("Updating CmkId for " + snapshot.getRootId());
dynamoDBMapper.save(snapshot.withCmkId(cmkId));
} catch (EncryptDecryptException ex) {
if (ex.isKmsError()) {
// ... reencrypt and log...
migratedSnapshotsCount++;
}
}
}
}
log(
"Re-encrypt encrypted key process completed for table " + tableName + ", " + migratedSnapshotsCount +
" Snapshots migrated and re-encrypted."
);
This is running in the cloud so I cannot debug this locally. I would expect at least 2 log messages, but we're only getting the 1st from before the for-each loop and then nothing, no exceptions, just nothing. How is that even possible?

Scan is lazily loaded by default. Try setting the scan to EAGER_LOADING so that the data is returned on execution.
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapperConfig.PaginationLoadingStrategy.html

Related

Proper way to query for a row that may not exist - Java8

I'm trying to write a method that will retrieve a row in a database. Around 50% of the time, this row will not exist. I tried using jdbcTemplate.queryForObject, but this resulted in exceptions being thrown if the row didn't exist in the DB. I don't think its good practice to try catch the method.
I read that if your data may be null, jdbcTemplate.query was a better choice. But I can only get it working if I return a list. Returning a list doesn't really make sense here, since I'm only selecting 1 row, that may or may not exist.
public List<LoanDetailsBean> getLoanDetailsByInsaddr(String insaddr) {
String SQL_GET_LOAN_DETAILS_BY_INSADDR =
"SELECT TOP 1 cip.tranchesize, " +
" cip.maturitydate, " +
" cip.moodysissuerrating, " +
" cip.snpissuerrating, " +
" cip.moodysassetrating, " +
" cip.snpassetrating " +
"FROM cloinstrumentproperty cip " +
"WHERE insaddr = ? " +
"ORDER BY updatedtime DESC ";
return isdbJdbcTemplate.query(SQL_GET_LOAN_DETAILS_BY_INSADDR, new Object[] { insaddr }, (rs, rowNum) ->
new LoanDetailsBean(
rs.getDouble("tranchesize"),
rs.getString("maturitydate"),
rs.getString("moodysissuerrating"),
rs.getString("snpissuerrating"),
rs.getString("moodysassetrating"),
rs.getString("snpassetrating")
)
);
}
How can I re-write this to make a little more sense?
Is there an option to use 3rd-party libraries in your project?
this could be done musch simpler with my own:
<dependency>
<groupId>com.github.buckelieg</groupId>
<artifactId>db-fn</artifactId>
<version>0.3.4</version>
</dependency>
and then:
public List<LoanDetailsBean> getLoanDetailsByInsaddr(String insaddr) {
try (DB db = new DB("jdbc:postgresql://host:port/database?user=user&password=pass")) {
return db.select("SELECT TOP 1 cip.tranchesize, cip.maturitydate, cip.moodysissuerrating, cip.snpissuerrating, cip.moodysassetrating, cip.snpassetrating FROM cloinstrumentproperty cip WHERE insaddr=? ORDER BY updatedtime DESC", insaddr)
.list(rs -> new LoanDetailsBean(rs.getDouble("tranchesize"), rs.getString("maturitydate"), rs.getString("moodysissuerrating"), rs.getString("snpissuerrating"), rs.getString("moodysassetrating"), rs.getString("snpassetrating"))));
}
}

Java IBM Lotus Notes retrive data from view fails in the middle due to null object

I am trying to retrieve data from lotus notes database view using a java program. Below is my code:
int resultsCount = view.getEntryCount();
print("Results found in view = " + resultsCount);
Document doc = view.getFirstDocument();
if (doc != null) {
int count = 1;
while (count <= resultsCount) {
count++;
try {
doc = view.getNextDocument(doc);
if (doc == null) {
print("Record " + count + " error. Null object.");
}
} catch (NotesException e) {
print("Record " + count + " error. Exception.");
}
}
}
else {
print("Record " + count + " error. Null object.");
}
I get below results:
Results found in view = 1567
Record 866 error. Null object.
Why is there a null document found when actually 1567 records present in the db view?
How can I resume to get rest of the records, because view.getNextDocument(doc) fails with Notes Exception after this happens.
Fixed by using
int resultsCount = view.getAllEntries().getCount();
instead of
int resultsCount = view.getEntryCount();
Using view.getAllEntries().getCount() returns the actual entry count which is 866. I am not sure what view.getEntryCount() returns. But it is definitely not the actual document count.
Edit:
As mentioned in XPages getEntryCount vs getAllEntries().getCount() view.getEntryCount() includes replications and save conflicts. Therefore to get actual record count needs to use view.getAllEntries().getCount()

No file BLOB, through JSON from ORACLE to SQL Server

i am a student and right now I'm doing an internship working with a local library, and in this case i have the following problem:
In the project i´m making, i need to retrieve image data from a temporal table, constructed in ORACLE that receives its data from some triggers in an INFORMIX DB and parse it through a monitor made in JAVA, in a JSON format to a web service published in C# and insert that image in a SQL Server DB.
I looked around and i found that it was possible to parse images through JSON using Base64 encoding and whatnot but when they talk about it they say that you must have the image path file and encode it. as you may have realized by now, i cant use that route because i don't have those images, best case scenario, the triggers are able to feed some BLOB data (by what I've been told). but i have to insert them in the SQL Server DB as Varbinary(MAX).
To summarize:
-->Informix DB has images -->triggers feed an ORACLE Temp_table (images sent probably as BLOB or CLOB at most)-->monitor made in JAVA must read those BLOBS or CLOBS and send them through JSON
-->Web Service made in C# must receive that JSON, and insert the images in a SQL Server DB (where they need to be visible, without having the physical file to refer to).
the schema i´m using (it has been IMPOSED to me, i didn't had a saying in this) is something similar to this: (it´s really long and tedious code so i´ll try to make it as neat and clean as possible)
This is the part of the java monitor that specifies which fields from the temp_table are feeding what fields in the JSON structure
public static BookRecordList viewBookRecordTable(Connection connection) throws ExceptionToOracleConcurrent
{
BookRecordList bookRecordList = new BookRecordList();
BookRecord bookRecord = new BookRecord();
Statement stmt = null;
String query = "SELECT operacion,"
+ "UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(img_logo,32670,1))"
+ "x_logo,"
+ "UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(img_logoGris,32760,1))"
+ "UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(r_firma,32670,1)),"
+ " FROM "
+ dataBaseConnectionData.getDB_SHCHEMA() + "."+ dataBaseConnectionData.getDB_TABLE_COLA()
+ " WHERE (some condition)";
try
{
stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(query);
while(rs.next())
{
try
{
bookRecord = new BookRecord();
bookRecord.setOperacion(rs.getInt("operacion"));
bookRecord.setImg_logo(rs.getString("img_logo"));
bookRecord.setImg_logoGris(rs.getString("img_logoGris"))
bookRecord.setR_firma(rs.getString("r_firma"));
bookRecord.print();
bookRecordList.getBookRecordList().add(bookRecord);
}
catch (Exception e)
{
logger.error("Some exception " + dataBaseConnectionData.getDB_TABLE_COLA() + ": " + e.toString());
e.printStackTrace();
//Process next order
continue;
}
}
}
catch (SQLException e )
{
logger.fatal("Some exception " + dataBaseConnectionData.getDB_TABLE_COLA() + ": " + e.toString());
throw new ExceptionToOracleConcurrent("exception definition " + dataBaseConnectionData.getDB_TABLE_COLA() + ": " + e.toString());
}
finally
{
if (stmt != null)
{
try
{
stmt.close();
}
catch (SQLException e)
{
logger.fatal("another exception " + e.toString());
}
}
}
return bookRecordList;
}
This is the part of the java monitor that generates the JSON (the empty cases contain another stuff that goes into the JSON but i sorted that out)
private static String GenerateJSON(SomeClass someClass) throws IOException
{
int operation = someClass.getOperation();
JSONObject obj = new JSONObject();
String jsonText = "";
switch (operation)
{
case 0:
//obligatory fields
obj.put("img_logo",someClass.getImg_logo());
break;
case 1:
break;
case 2:
//Obligatory fields
obj.put("img_foto",someClass.getC_empleado());
obj.put("img_firma",someClass.getC_empleado());
break;
case 3:
obj.put("r_firma",someClass.getR_firma());
break;
case 4:
break;
case 5:
break;
}
StringWriter out = new StringWriter();
obj.writeJSONString(out);
jsonText = out.toString();
String newJson = jsonText.replace("\\/", "/");
logger.info("JSON a enviar: " + newJson);
return newJson;
}
The web service is made in C#, it´s another case based program, structured accordingly to the operation number received in the JSON, it calls a number of function and, in the end, it comes down to these two:
this part of the WS receive the parameters of the parsed JSON
public int ActualizarFichaLibro( String img_foto, String r_firma)
{
try
{
//Define query to insert
Cmd.CommandText = QueryCFA.ActualizarFicha();
//Define parameters types to insert
Cmd.Parameters.Add("#img_foto", SqlDbType.VarBinary, -1);
Cmd.Parameters.Add("#r_firma", SqlDbType.VarBinary, -1);
//Define parameters values to insert
Cmd.Parameters["#img_foto"].Value = img_foto;
Cmd.Parameters["#r_firma"].Value = r_firma;
int rowCount = Cmd.ExecuteNonQuery();
CerrarConexionBd();
return rowCount;
}
catch (Exception)
{
return 0;
}
}
and finally that invokes a simple query, in this particular case, to this one:
public string ActualizarFicha()
{
Query = "UPDATE dbo.fichaEmpleado SET( CASE WHEN #img_foto = '' THEN NULL ELSE img_foto = CONVERT(VARBINARY(MAX), #img_foto, 2) END,"
+ "CASE WHEN #r_firma = '' THEN NULL ELSE img_firma = CONVERT(VARBINARY(MAX), #r_firma, 2) END,"
+"WHERE (some conditions)";
return Query;
}
my questions are:
is there a way to do this (sending images from one DB to anther) through JSON, specifically with this massive schema this people got going on? if not is there a way to do it?
the querys for reading a BLOB (possible BLOB) and inserting a Varbinary are well implemented?
I´m sorry for the extremely long explanation, I've been working on this for a week and i cant seem to find a proper way to do it (at least not with this schema, but the bosses don't want to change it)

Database insertion synchronization

I have a java code that generates a request number based on the data received from database, and then updates the database for newly generated
synchronized (this.getClass()) {
counter++;
System.out.println(counter);
System.out.println("start " + System.identityHashCode(this));
certRequest
.setRequestNbr(generateRequestNumber(certInsuranceRequestAddRq
.getAccountInfo().getAccountNumberId()));
System.out.println("outside funcvtion"+certRequest.getRequestNbr());
reqId = Utils.getUniqueId();
certRequest.setRequestId(reqId);
System.out.println(reqId);
ItemIdInfo itemIdInfo = new ItemIdInfo();
itemIdInfo.setInsurerId(certRequest.getRequestId());
certRequest.setItemIdInfo(itemIdInfo);
dao.insert(certRequest);
addAccountRel();
counter++;
System.out.println(counter);
System.out.println("end");
}
the output for System.out.println() statements is `
1
start 27907101
com.csc.exceed.certificate.domain.CertRequest#a042cb
inside function request number66
outside funcvtion66
AF88172D-C8B0-4DCD-9AC6-12296EF8728D
2
end
3
start 21695531
com.csc.exceed.certificate.domain.CertRequest#f98690
inside function request number66
outside funcvtion66
F3200106-6033-4AEC-8DC3-B23FCD3CA380
4
end
In my case I get a call from two threads for this code.
If you observe both the threads run independently. However the data for request number is same in both the cases.
is it possible that before the database updation for first thread completes the second thread starts execution.
`
the code for generateRequestNumber() is as follows:
public String generateRequestNumber(String accNumber) throws Exception {
String requestNumber = null;
if (accNumber != null) {
String SQL_QUERY = "select CERTREQUEST.requestNbr from CertRequest as CERTREQUEST, "
+ "CertActObjRel as certActObjRel where certActObjRel.certificateObjkeyId=CERTREQUEST.requestId "
+ " and certActObjRel.certObjTypeCd=:certObjTypeCd "
+ " and certActObjRel.certAccountId=:accNumber ";
String[] parameterNames = { "certObjTypeCd", "accNumber" };
Object[] parameterVaues = new Object[] {
Constants.REQUEST_RELATION_CODE, accNumber };
List<?> resultSet = dao.executeNamedQuery(SQL_QUERY,
parameterNames, parameterVaues);
// List<?> resultSet = dao.retrieveTableData(SQL_QUERY);
if (resultSet != null && resultSet.size() > 0) {
requestNumber = (String) resultSet.get(0);
}
int maxRequestNumber = -1;
if (requestNumber != null && requestNumber.length() > 0) {
maxRequestNumber = maxValue(resultSet.toArray());
requestNumber = Integer.toString(maxRequestNumber + 1);
} else {
requestNumber = Integer.toString(1);
}
System.out.println("inside function request number"+requestNumber);
return requestNumber;
}
return null;
}
Databases allow multiple simultaneous connections, so unless you write your code properly you can mess up the data.
Since you only seem to require a unique growing integer, you can easily generate one safely inside the database with for example a sequence (if supported by the database). Databases not supporting sequences usually provide some other way (such as auto increment columns in MySQL).

error in dataSearch using berkley DB

I've created a database using Berkley DB that stores N records where a record is a key/value pair. I originally populated it with only 20 records. With 20 records I managed to do a Key Search, and a Data Search (where I search through the database record by record for a data value that matches the string data inputted by the user).
public String dataSearch (String dataInput) {
String foundKey = null;
String foundData = null;
Cursor cursor = null;
try {
cursor = myDb.openCursor(null, null);
DatabaseEntry theKey = new DatabaseEntry();
DatabaseEntry theData = new DatabaseEntry();
while (cursor.getNext(theKey, theData, LockMode.DEFAULT) == OperationStatus.SUCCESS) {
foundKey = new String(theKey.getData(), "UTF-8");
foundData = new String(theData.getData(), "UTF-8");
// this is to see each key - data - inputdata as I was having an issue
System.out.println("KEY: " + foundKey +
"\nDATA: " + foundData +
"\nINPUT_DATA: " + dataInput + "\n\n");
if (foundData.equals(dataInput)) {
System.out.println("-----------------------------------\n\n");
System.out.println("Found record: " + foundKey +
"\nwith data: " + foundData);
System.out.println("\n\n-----------------------------------");
}
}
/* I then close the cursor and catch exceptions and such */
this works fine when I have less than (or equal to) 20 records... but when I use a bigger number I seem to have some funny behaviour. I set the number of records to 1000... the last key/data values to be inserted into the database are:
KEY: zghxnbujnsztazmnrmrlhjsjfeexohxqotjafliiktlptsquncuejcrebaohblfsqazznheurdqbqbxjmyqr
DATA: jzpqaymwwnoqzvxykowdhxvfbuhrsfojivugrmvmybbvurxmdvmrclalzfscmeknyzkqmrcflzdooyupwznvxikermrbicapynwspbbritjyeltywmmslpeuzsmh
I had it print out the last values to be inserted into the database then did a key search on the above key to ensure that the data above was infact the data associated with that key in the database. However, when I do a data search on the data listed above I get no found matching record (whereas the same process found a record when there was 20 records). I looked into it a bit more and got each my data search to print each key/data pair that it returned and found the following result:
KEY: zghxnbujnsztazmnrmrlhjsjfeexohxqotjafliiktlptsquncuejcrebaohblfsqazznheurdqbqbxjmyqrpzlyvnmdlvgyvzhbceeftcqssbeckxkuepxyphsgdzd
DATA: jzpqaymwwnoqzvxykowdhxvfbuhrsfojivugrmvmybbvurxmdvmrclalzfscmeknyzkqmrcflzdooyupwznvxikermrbicapynwspbbritjyeltywmmslpeuzsmhozy
INPUT DATA: jzpqaymwwnoqzvxykowdhxvfbuhrsfojivugrmvmybbvurxmdvmrclalzfscmeknyzkqmrcflzdooyupwznvxikermrbicapynwspbbritjyeltywmmslpeuzsmh
as you can see it seems to have randomly appended some extra bytes to the data value. however if I do a key search these extra bytes don't show up. So I think the problem is in the dataSearch function. The same results occur if I use b+tree or hash.
Any Ideas?
Thanks
After a long time looking at this I realized my error was that I was not reinitializing the theKey & theData variables.
the fix is in the while loop
while (cursor.getNext(theKey, theData, LockMode.DEFAULT) == OperationStatus.SUCCESS) {
foundKey = new String(theKey.getData(), "UTF-8");
foundData = new String(theData.getData(), "UTF-8");
// this is to see each key - data - inputdata as I was having an issue
System.out.println("KEY: " + foundKey +
"\nDATA: " + foundData +
"\nINPUT_DATA: " + dataInput + "\n\n");
if (foundData.equals(dataInput)) {
System.out.println("-----------------------------------\n\n");
System.out.println("Found record: " + foundKey +
"\nwith data: " + foundData);
System.out.println("\n\n-----------------------------------");
}
// THIS IS THE FIX
theKey = new DatabaseEntry();
theData = new DatabaseEntry();
// ----------------------------
}

Categories

Resources