Develop Reference

spring repository - repeated updates

Im developing an application using spring boot, and spring JPA repositories.
The application logic is simple:
Get records from an external service
Check if record exists in my mysql database
If exists do an update, if not do an insert
This works, but the performance decreases as records are processed.
I enabled mysql logs to watch incoming queries, and I saw lot of repeated updates.
The log entries follows this pattern
lot of updates
other sentences
the same updates in (1) plus more updates
other sentences
the same updates in (3) plus more updates
Im using a Repository that extends CrudRepository and I do the updates using repository.save() method.
Im trying to save objects of a subclass of Event (using discriminator), those objects have a Hashmap that maps to a JSON column in mysql.
I observed too, that if I only change values in HashMap (without invoking repository.save() method), the database updates are still being invoked.
This is the recurrent code invoked when I get a record from external service:
else if (event.getType().equals(EventType.PAYMENT)) {
PaymentEvent paymentEvent = (PaymentEvent)event;
PaymentEvent oldPaymentEvent = eventRepository.getPaymentEvent(paymentEvent.getPaymentId(), paymentEvent.getTimestamp());
if (oldPaymentEvent == null) {
// save new event
eventRepository.save(paymentEvent);
} else {
// this method updates the internal Hashmap that maps to JSON Column in Mysql
oldPaymentEvent.updateCustomPropsFrom(paymentEvent);
// update old event, if I comment this line database updates are still being triggered (modifying internal hashamp in the line above trigger the udpates)
eventRepository.save(oldPaymentEvent);
}
This is the Repository definiton:
public interface EventRepository extends CrudRepository<Event, Long> {
}
And finally this is the Hashmap merge:
public void updateCustomPropsFrom(PaymentEvent event) {
if (event.getCouponAmount() != null ) {
this.customProps.put(KEY_COUPON_AMOUNT, event.getCouponAmount());
}
if (event.getCurrencyId() != null ) {
this.customProps.put(KEY_CURRENCY_ID, event.getCurrencyId());
}
if (event.getPaymentId() != null ) {
this.customProps.put(KEY_PAYMENT_ID, event.getPaymentId());
}
if (event.getReceivedAmount() != null ) {
this.customProps.put(KEY_RECEIVED_AMOUNT, event.getReceivedAmount());
}
if (event.getPaymentType() != null ) {
this.customProps.put(KEY_PAYMENT_TYPE, event.getPaymentType());
}
if (event.getPaymentReason() != null ) {
this.customProps.put(KEY_PAYMENT_REASON, event.getPaymentReason());
}
if (event.getTimestamp() != null)
this.setTimestamp(event.getTimestamp());
if (event.getStoreId()!=null)
this.setStoreId(event.getStoreId());
if (event.getUserId()!=null)
this.setUserId(event.getUserId());
if (event.getUserId()!=null)
this.setUserId(event.getUserId());
}
I've never faced this problem, can anyone help me?
Thanks in advance. Regards

How Do You Check For DELETE/UPDATE Without A WHERE Clause

I currently have a listener that we use to do a few different monitoring-type activities (like log a warning if a query takes more than 5 seconds), but it also watches for and kills "silly bugs" -- especially UPDATE and DELETE queries that are missing a WHERE clause.
In the past we did the following (note that we are using com.foundationdb.sql):
/**
* Hook into the query execution lifecycle before rendering queries. We are checking for silly mistakes,
* pure SQL, etc.
*/
#Override
public void renderStart(final #NotNull ExecuteContext ctx) {
if (ctx.type() != ExecuteType.WRITE)
return;
String queryString = ctx.sql();
try (final Query query = ctx.query()) {
// Is our Query object empty? If not, let's run through it
if (!ValidationUtils.isEmpty(query)) {
queryString = query.getSQL(ParamType.INLINED);
final SQLParser parser = new SQLParser();
try {
final StatementNode tokens = parser.parseStatement(query.getSQL());
final Method method = tokens.getClass().getDeclaredMethod("getStatementType");
method.setAccessible(true);
switch (((Integer) method.invoke(tokens)).intValue()) {
case StatementType.UPDATE:
SelectNode snode = ConversionUtils.as(SelectNode.class,
((DMLStatementNode) tokens).getResultSetNode());
// check if we are a mass delete/update (which we don't allow)
if ((Objects.isNull(snode)) || (Objects.isNull(snode.getWhereClause())))
throw new RuntimeException("A mass update has been detected (and prevented): "
+ DatabaseManager.getBuilder().renderInlined(ctx.query()));
break;
case StatementType.DELETE:
snode = ConversionUtils.as(SelectNode.class,
((DMLStatementNode) tokens).getResultSetNode());
// check if we are a mass delete/update (which we don't allow)
if ((Objects.isNull(snode)) || (Objects.isNull(snode.getWhereClause())))
throw new RuntimeException("A mass delete has been detected (and prevented): "
+ DatabaseManager.getBuilder().renderInlined(ctx.query()));
break;
default:
if (__logger.isDebugEnabled()) {
__logger
.debug("Skipping query because we don't need to do anything with it :-): {}", queryString);
}
}
} catch (#NotNull StandardException | IllegalAccessException
| IllegalArgumentException | InvocationTargetException | NoSuchMethodException
| SecurityException e) {
// logger.error(e.getMessage(), e);
}
}
// If the query object is empty AND the SQL string is empty, there's something wrong
else if (ValidationUtils.isEmpty(queryString)) {
__logger.error(
"The ctx.sql and ctx.query.getSQL were empty");
} else
throw new RuntimeException(
"Someone is trying to send pure SQL queries... we don't allow that anymore (use jOOQ): "
+ queryString);
}
}
I really don't want to use yet another tool -- especially since most SQL parsers can't handle UPSERTs or the wide variety of queries that jOOQ can, so a lot just get cut out -- and would love to use jOOQ's constructs, but I'm having trouble. Ideally I could just check the query class and if it's an Update or Delete (or a subclass), I would just scream if it isn't an instance of UpdateConditionStep or DeleteConditionStep, but that doesn't work because the queries are coming back as UpdateQueryImpl... and without crazy reflection, I can't see if there is a condition in use.
So... right now I'm doing:
/**
* Hook into the query execution lifecycle before rendering queries. We are checking for silly mistakes, pure SQL,
* etc.
*/
#Override
public void renderStart(final #NotNull ExecuteContext ctx) {
if (ctx.type() != ExecuteType.WRITE)
return;
try (final Query query = ctx.query()) {
// Is our Query object empty? If not, let's run through it
if (!ValidationUtils.isEmpty(query)) {
// Get rid of nulls
query.getParams().entrySet().stream().filter(entry -> Objects.nonNull(entry.getValue()))
.filter(entry -> CharSequence.class.isAssignableFrom(entry.getValue().getDataType().getType()))
.filter(entry -> NULL_CHARACTER.matcher((CharSequence) entry.getValue().getValue()).find())
.forEach(entry -> query.bind(entry.getKey(),
NULL_CHARACTER.matcher((CharSequence) entry.getValue().getValue()).replaceAll("")));
if (Update.class.isInstance(query)) {
if (!UpdateConditionStep.class.isInstance(query)) {
if (!WHERE_CLAUSE.matcher(query.getSQL(ParamType.INDEXED)).find()) {
final String queryString = query.getSQL(ParamType.INLINED);
throw new RuntimeException(
"Someone is trying to run an UPDATE query without a WHERE clause: " + queryString);
}
}
} else if (Delete.class.isInstance(query)) {
if (!DeleteConditionStep.class.isInstance(query)) {
if (!WHERE_CLAUSE.matcher(query.getSQL(ParamType.INDEXED)).find()) {
final String queryString = query.getSQL(ParamType.INLINED);
throw new RuntimeException(
"Someone is trying to run a DELETE query without a WHERE clause: " + queryString);
}
}
}
} else
throw new RuntimeException(
"Someone is trying to send pure SQL queries... we don't allow that anymore (use jOOQ): "
+ ctx.sql());
}
}
This let's me get rid of the third party SQL parser, but now I'm using a regular expression on the non-inlined query looking for \\s[wW][hH][eE][rR][eE]\\s, which isn't ideal, either.
Is there a way to use jOOQ to tell me if an UPDATE, DELETE, has a WHERE clause?
Similarly, is there a way that let's me see what table the query is acting against (so that I can limit the tables someone can perform mutable actions against -- obviously that one wouldn't check if it's UPDATE or DELETE, instead using the ExecuteType)?

That's an interesting idea and approach. One problem I can see with it is performance. Rendering the SQL string a second time and then parsing it again sounds like a bit of overhead. Perhaps, this ExecuteListener should be active in development and integration test environments only, not in production.
Regarding your questions
Is there a way to use jOOQ to tell me if an UPDATE, DELETE, has a WHERE clause?
Since you seem to be open to use reflection to access a third party library's internals, well of course, you could check if the ctx.query() is of type org.jooq.impl.UpdateQueryImpl or org.jooq.impl.DeleteQueryImpl. In version 3.10.1, both of them have a private condition member, which you could check.
This will obviously break any time the internals are changed, but it might be a pragmatic solution for now.
Similarly, is there a way that let's me see what table the query is acting against
A more general and more robust approach would be to implement a VisitListener, which is jOOQ's callback that is called during expression tree traversal. You can hook into the generation of the SQL string and the collection of bind variables, and throw your errors as soon as you encounter:
An UPDATE or DELETE statement
... without a WHERE clause
... updating a table from a specific set of tables
You "just" have to implement a stack machine that remembers all of the above things prior to throwing the exception. An example of how VisitListener can be implemented is given here:
https://blog.jooq.org/2015/06/17/implementing-client-side-row-level-security-with-jooq
New feature in the future
This kind of feature has been discussed a couple of times on the mailing list as well. It's a low hanging fruit to support by jOOQ natively. I've created a feature request for jOOQ 3.11, for this:
https://github.com/jOOQ/jOOQ/issues/6771

Creating managed RealmList outside RealmObject

In my app, I have a method that accepts an ArrayList of ID's and returns a RealmList of Machines belonging to these IDs.
public RealmList<Machine> getMachinesById(ArrayList<Long> machineIds) {
RealmList<Machine> machines = new RealmList<Machine>();
for (int i = 0; i < machineIds.size(); i++){
Machine m = getMachineById(machineIds.get(i));
if (m != null) {
machines.add(m);
}
}
return machines;
}
The getMachineById() function just finds the correct machine for a specific id.
I want to filter this output some more, however, when I try to get the RealmQuery by doing .where(), I get an Exception telling me I should put this RealmList in 'managed mode'.
Caused by: io.realm.exceptions.RealmException: This method is only available in managed mode
at io.realm.RealmList.where(RealmList.java:425)
I'm aware that I get this error because this list is standalone, and not managed by Realm.
It is probably important to add that this function will be called quite a lot, since it is triggered every time some list in my app refreshes. This would mean that (if possible) every time I'm creating a new managed RealmList.
My questions:
Is there any way to let this RealmList be managed by Realm?
If this is possible, is it a problem that this function is being called pretty often
Is there any other (preferred) way to achieve this (List of IDs > RealmResults/RealmQuery)

Is there any way to let this RealmList be managed by Realm?
Yes. There is. But the point of having RealmList is it should be a field of an RealmObjects. eg.:
public class Factory {
RealmList<Machine> machineList;
// setter & getters
}
Factory factory = new Factory();
RealmList<Machine> machineList = new RealmList<>();
// Add something to the list
factory.setMachines(machineList);
realm.beginTransaction();
Factory managedFactory = realm.copyToRealmOrUpdate(factory);
realm.commitTransaction();
Managed means it has be persisted Realm.
If this is possible, is it a problem that this function is being called pretty often
Depends, if you don't need to persist them again, see answer 3.
Is there any other (preferred) way to achieve this (List of IDs > RealmResults/RealmQuery)
In your case, maybe you can use ReaulResults instead? eg.:
RealmQuery<Machine> query = realm.where(Machine.class);
for (int i = 0; i < machineIds.size(); i++){
if (i != 0) query = query.or();
query = query.equalTo("id", machineIds.get(i));
}
RealmResults<Machine> machines = query.findAll();

Modifying a large set of objects using JPA / EclipseLink

I need to iterate 50k objects and change some fields in them.
I'm limited in memory so I don't want to bring all 50k objects into memory at once.
I thought doing it with the following code using cursor, but I was wondering whether all the objects I've processes using the cursor are left in the Entity Manager cache.
The reason I don't want to do it with offset and limit is because the database needs to work much harder since each page is a complete new query.
From previous experience once the Entity manager cache gets bigger, updates become real slow.
So usually I call flush and clear after every few hundreds of updates.
The problem here is that flushing / clearing will break the cursor.
I will be happy to learn the best approach of updating a large set of objects without loading them all into memory.
Additional information on how EclipseLink cursor works in such scenraio will be valuable too.
JpaQuery<T> jQuery = (JpaQuery<T>) query;
jQuery.setHint(QueryHints.RESULT_SET_TYPE, ResultSetType.ForwardOnly)
.setHint(QueryHints.SCROLLABLE_CURSOR, true);
Cursor cursor = jQuery.getResultCursor();
Iterator<MyObj> cursorIterator = cursor.iterator();
while (cursorIterator.hasNext()) {
MyObj myObj = cursorIterator.next();
ChangeMyObj(myObj);
}
cursor.close();

Use pagination + entityManager.clear() after each page. Also execute every page in a single transaction OR you will have to create/get a new EntityManager after an exception occurs (ar least with Hibernate: the EntityManager instance could be in an inconsistent state after an exception).

Try this sample code:
List results;
int index= 0;
int max = 100;
do {
Query query= manager.createQuery("JPQL QUERY");
query.setMaxResults(max).
setFirstResult(index);
results = query.getResultList( );
Iterator it = results.iterator( );
while (it.hasNext( )) {
Object c = (Object)it.next( );
}
entityManager.clear( );
index = index + results.getSize( );
} while (results.size( ) > 0);

How to protect against SQL injection when the WHERE clause is built dynamically from search form?

I know that the only really correct way to protect SQL queries against SQL injection in Java is using PreparedStatements.
However, such a statement requires that the basic structure (selected attributes, joined tables, the structure of the WHERE condition) will not vary.
I have here a JSP application that contains a search form with about a dozen fields. But the user does not have to fill in all of them - just the one he needs. Thus my WHERE condition is different every time.
What should I do to still prevent SQL injection?
Escape the user-supplied values? Write a wrapper class that builds a PreparedStatement each time? Or something else?
The database is PostgreSQL 8.4, but I would prefer a general solution.
Thanks a lot in advance.

Have you seen the JDBC NamedParameterJDBCTemplate ?
The NamedParameterJdbcTemplate class
adds support for programming JDBC
statements using named parameters (as
opposed to programming JDBC statements
using only classic placeholder ('?')
arguments.
You can do stuff like:
String sql = "select count(0) from T_ACTOR where first_name = :first_name";
SqlParameterSource namedParameters = new MapSqlParameterSource("first_name", firstName);
return namedParameterJdbcTemplate.queryForInt(sql, namedParameters);
and build your query string dynamically, and then build your SqlParameterSource similarly.

I think that fundamentally, this question is the same as the other questions that I referred to in my comment above, but I do see why you disagree — you're changing what's in your where clause based on what the user supplied.
That still isn't the same as using user-supplied data in the SQL query, though, which you definitely want to use PreparedStatement for. It's actually very similar to the standard problem of needing to use an in statement with PreparedStatement (e.g., where fieldName in (?, ?, ?) but you don't know in advance how many ? you'll need). You just need to build the query dynamically, and add the parameters dynamically, based on information the user supplied (but not directly including that information in the query).
Here's an example of what I mean:
// You'd have just the one instance of this map somewhere:
Map<String,String> fieldNameToColumnName = new HashMap<String,String>();
// You'd actually load these from configuration somewhere rather than hard-coding them
fieldNameToColumnName.put("title", "TITLE");
fieldNameToColumnName.put("firstname", "FNAME");
fieldNameToColumnName.put("lastname", "LNAME");
// ...etc.
// Then in a class somewhere that's used by the JSP, have the code that
// processes requests from users:
public AppropriateResultBean[] doSearch(Map<String,String> parameters)
throws SQLException, IllegalArgumentException
{
StringBuilder sql;
String columnName;
List<String> paramValues;
AppropriateResultBean[] rv;
// Start the SQL statement; again you'd probably load the prefix SQL
// from configuration somewhere rather than hard-coding it here.
sql = new StringBuilder(2000);
sql.append("select appropriate,fields from mytable where ");
// Loop through the given parameters.
// This loop assumes you don't need to preserve some sort of order
// in the params, but is easily adjusted if you do.
paramValues = new ArrayList<String>(parameters.size());
for (Map.Entry<String,String> entry : parameters.entrySet())
{
// Only process fields that aren't blank.
if (entry.getValue().length() > 0)
{
// Get the DB column name that corresponds to this form
// field name.
columnName = fieldNameToColumnName.get(entry.getKey());
// ^-- You'll probably need to prefix this with something, it's not likely to be part of this instance
if (columnName == null)
{
// Somehow, the user got an unknown field into the request
// and that got past the code calling us (perhaps the code
// calling us just used `request.getParameterMap` directly).
// We don't allow unknown fields.
throw new IllegalArgumentException(/* ... */);
}
if (paramValues.size() > 0)
{
sql.append("and ");
}
sql.append(columnName);
sql.append(" = ? ");
paramValues.add(entry.getValue());
}
}
// I'll assume no parameters is an invalid case, but you can adjust the
// below if that's not correct.
if (paramValues.size() == 0)
{
// My read of the problem being solved suggests this is not an
// exceptional condition (users frequently forget to fill things
// in), and so I'd use a flag value (null) for this case. But you
// might go with an exception (you'd know best), either way.
rv = null;
}
else
{
// Do the DB work (below)
rv = this.buildBeansFor(sql.toString(), paramValues);
}
// Done
return rv;
}
private AppropriateResultBean[] buildBeansFor(
String sql,
List<String> paramValues
)
throws SQLException
{
PreparedStatement ps = null;
Connection con = null;
int index;
AppropriateResultBean[] rv;
assert sql != null && sql.length() > 0);
assert paramValues != null && paramValues.size() > 0;
try
{
// Get a connection
con = /* ...however you get connections, whether it's JNDI or some conn pool or ... */;
// Prepare the statement
ps = con.prepareStatement(sql);
// Fill in the values
index = 0;
for (String value : paramValues)
{
ps.setString(++index, value);
}
// Execute the query
rs = ps.executeQuery();
/* ...loop through results, creating AppropriateResultBean instances
* and filling in your array/list/whatever...
*/
rv = /* ...convert the result to what we'll return */;
// Close the DB resources (you probably have utility code for this)
rs.close();
rs = null;
ps.close();
ps = null;
con.close(); // ...assuming pool overrides `close` and expects it to mean "release back to pool", most good pools do
con = null;
// Done
return rv;
}
finally
{
/* If `rs`, `ps`, or `con` is !null, we're processing an exception.
* Clean up the DB resources *without* allowing any exception to be
* thrown, as we don't want to hide the original exception.
*/
}
}
Note how we use information the user supplied us (the fields they filled in), but we didn't ever put anything they actually supplied directly in the SQL we executed, we always ran it through PreparedStatement.

The best solution is to use a middle that does data validation and binding and acts as an intermediary between the JSP and the database.
There might be a list of column names, but it's finite and countable. Let the JSP worry about making the user's selection known to the middle tier; let the middle tier bind and validate before sending it on to the database.

Here is a useful technique for this particular case, where you have a number of clauses in your WHERE but you don't know in advance which ones you need to apply.
Will your user search by title?
select id, title, author from book where title = :title
Or by author?
select id, title, author from book where author = :author
Or both?
select id, title, author from book where title = :title and author = :author
Bad enough with only 2 fields. The number of combinations (and therefore of distinct PreparedStatements) goes up exponentially with the number of conditions. True, chances are you have enough room in your PreparedStatement pool for all those combinations, and to build the clauses programatically in Java, you just need one if branch per condition. Still, it's not that pretty.
You can fix this in a neat way by simply composing a SELECT that looks the same regardless of whether each individual condition is needed.
I hardly need mention that you use a PreparedStatement as suggested by the other answers, and a NamedParameterJdbcTemplate is nice if you're using Spring.
Here it is:
select id, title, author
from book
where coalesce(:title, title) = title
and coalesce(:author, author) = author
Then you supply NULL for each unused condition. coalesce() is a function that returns its first non-null argument. Thus if you pass NULL for :title, the first clause is where coalesce(NULL, title) = title which evaluates to where title = title which, being always true, has no effect on the results.
Depending on how the optimiser handles such queries, you may take a performance hit. But probably not in a modern database.
(Though similar, this problem is not the same as the IN (?, ?, ?) clause problem where you don't know the number of values in the list, since here you do have a fixed number of possible clauses and you just need to activate/disactivate them individually.)

I'm not confident if there is a quote() method, which was widely used in PHP's PDO. This would allow you a more flexible query building approach.
Also, one of the possible ideas could be creating special class, which would process filter criterias and would save into a stack all placeholders and their values.