I am trying to save a bunch of SQL transaction. I am in a context of ESB routes, transferring from a SQL source to a SQL target, and the order of the SQL transactions is not guaranteed, so you can have a SQL update before the object was inserted.
Due to the architecture, I'm saving those SQL transactions 1000 by 1000 (I'm using a messageQueue). So some of these can fail, and I re-route them in order to retry or reject them. To improve efficiency, I'm willing to improve the older system, where if the 1000 fail, you save 1 by 1, to implements dichotomia (if the save fail, you split the list and try again), via recursivity. I am also tracking an attribute of my objects, thanks to another list (objectsNo) for further operations.
However I am getting a ConcurrentModificationException when in my first recursivity, when calling objectsList.size(). How can I avoid it ? I'm also opened, and would be very thankful to any solutions which would provide another way than dichotomia to improve efficiency (and would by such bypass my issue).
Suppressed: java.util.ConcurrentModificationException: null
at java.util.ArrayList$SubList.checkForComodification(ArrayList.java:1231)
at java.util.ArrayList$SubList.size(ArrayList.java:1040)
at fr.company.project.esbname.mariadb.MariaDbDatabase.saveObjectWithDichotomie(MariaDbDatabase.java:398)
at fr.company.project.esbname.mariadb.MariaDbDatabase.saveObjectWithDichotomie(MariaDbDatabase.java:404)
at fr.company.project.esbname.mariadb.MariaDbDatabase.saveObject(MariaDbDatabase.java:350)
at sun.reflect.GeneratedMethodAccessor324.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:472)
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:291)
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:264)
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:178)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
... 22 common frames omitted
I tried to understand, but there should not be any mistake. Even if I used recursivity, it stays single-threaded. I considered that the issue could be with hibernate (some requests from the save which failed could stay in the cache, and lock modification), but the issue is with the size, which is on a sublist of the original list.
private List<String> saveObjectWithDichotomie(List<Object> objects,
List<String> objectsNo,
Exchange exchange) throws JsonProcessingException {
try {
objectRepository.save(objects);
return objectsNo;
} catch (DataIntegrityViolationException e) {
if (objects.size() == 1) {
objectsNo.clear();
errorProcessor.sendErrorToRejets(objects.get(0), exchange, e);
return objectsNo;
} else {
List<Object> objectsFirstHalf = objects.subList(0, objects.size()/2);
List<Object> objectsSecondHalf = objects.subList(objects.size()/2, objects.size());
List<String> objectsNoFirstHalf = objectsNo.subList(0, objectsNo.size()/2);
List<String> objectsNoSecondHalf = objectsNo.subList(objectsNo.size()/2, objectsNo.size());
objectsNo.clear();
objectsNo.addAll(
saveObjectWithDichotomie(objects, objectsNoFirstHalf, exchange)
);
objectsNo.addAll(
saveObjectWithDichotomie(objects, objectsNoSecondHalf, exchange)
);
return objectsNo;
}
}
}
If you would read the documentation of sublist is clearly says:
The returned list is backed by this list, so non-structural changes in the returned list are reflected in this list, and vice-versa.
That being the reason for your Exception (no need for multiple threads for this to happen). Thus when you create a new List, create it via:
List<Object> objectsFirstHalf = new ArrayList<>(objects.subList(0, objects.size()/2));
Two things:
ConcurrentModificationException does not mean that the list was modified by another thread, but that something is trying to access the list in an expected state but it was changed in the meantime.
subList does not create an actual new list, it creates a view on the original list. That means that you cannot change the original list without making the retrieved sublist invalid.
So,
objectsNo.clear();
is your problem.
See this MCVE:
public class Sublist {
public static void main(String[] args) {
List<String> list = new ArrayList<>(
IntStream.range(0, 100).mapToObj(Integer::toString).collect(Collectors.toList()));
List<String> sublist = list.subList(10, 20);
// outputs "15"
System.out.println(sublist.get(5));
list.clear();
// throws ConcurrentModificationException
System.out.println(sublist.get(5));
}
}
Related
I have a specific use case for data processing where I am returning a future of type Future<List<SamplePOJO>>. I have multiple such futures which I am adding to a List.
But CompositeFuture.join() doesn't work on this list as it is asking for a List<Future> instead of a List<Future<List<SamplePOJO>>>. Is there any workaround for this?
You can collect all those Future<List<SamplePOJO>> in the List<Future> instead of List<Future<List<SamplePOJO>>>.
That will make CompositeFuture.all method accept it.
Future<List<String>> f = getFuture();
List<Future> futures = new ArrayList<>();
futures.add(f);
CompositeFuture.all(futures);
Here's an expanded set of example code (that I mistakenly wrote for another question and moved here).
So there exists a bug in Vert.x that causes issues with CompositeFuture.all(listoffutures), etc., at least in JDK 17, if listoffutures is of type List<Future<SomeType>> (or List<Future<?>>).
This bug might get fixed in Vert.x 5.
I got some success with the code below. The contrived example here is that I want to turn a List<Future<File>> into a Future<List<File>>.
#SuppressWarnings("rawtypes")
static List<Future> ltol(List<Future<File>> sa) {
List<Future> l = new ArrayList<>();
l.addAll(sa);
return l;
}
// A contrived example of what I was doing, which uses .compose and returns
// a Future of the list of results (File objects in my case)
Future<List<File>> mymethodcall(List<Future<File>> attachments) {
return CompositeFuture.all(ltol(attachments)).compose(files -> {
// Note we're reading the result of the .all call in the compose
List<File> mb = new ArrayList<>();
files.list().stream().forEach(o -> {
// Do whatever you need to do here with the results but they'll likely
// need to be cast (to File, in this case).
mb.add((File) o);
});
return Future.succeededFuture(mb);
});
}
The important step is getting your List<Future<T> into a List<Future>, if you need to. I did it by gross brute force in the static method above.
I'm facing a weird behavior in my Java code using List.
The code is very simple, I have a List of Object called AccessRequest which comes from a database and I'm using this first List to create a new one but with a filter to select only a few objects.
Here is the code :
private void updateCommentIfNeeded() {
List<AccessRequest> accessRequestList = getAllRequest();
List<AccessRequest> commentsList = getCommentsListProcessedManually(accessRequestList);
}
public List<AccessRequest> getCommentsListProcessedManually(List<AccessRequest> accessRequests) {
accessRequests.removeIf(ar -> !ar.getComment().equals("To be processed manually"));
if (accessRequests.size() != 0) {
SQLServerConnection sqlServerConnection = new SQLServerConnection(sqlServerUrl);
accessRequests.removeIf(ar -> !sqlServerConnection.emailExists(ar.getEmail()));
}
return accessRequests;
}
I'm supposed to get a second List only containing the objects that has their comments to To be processed manually, which I do. But the weird part is that the first List also takes the value of the second as if I wrote accessRequestList = commentsList but there is no such thing and I'm using local variable.
Ex :
I have 3 objects in my first List, but only one containing the required comment
Both list ends with containing the only objects containing the comment
I'm kind of lost here if anyone has an idea !
Your method getCommentsListProcessedManually modifies the list you're passing. I believe you're operating under the assumption that passing the list as a parameter somehow creates a copy of the list, whereas what is actually happening is that a reference to the list is passed by value.
There are several ways to solve this, but the easiest is to simply create a copy of your input list at the start of your method:
public List<AccessRequest> getCommentsListProcessedManually(List<AccessRequest> input) {
List<AccessRequest> accessRequests = new ArrayList<>(input);
accessRequests.removeIf(ar -> !ar.getComment().equals("To be processed manually"));
if (accessRequests.size() != 0) {
SQLServerConnection sqlServerConnection = new SQLServerConnection(sqlServerUrl);
accessRequests.removeIf(ar -> !sqlServerConnection.emailExists(ar.getEmail()));
}
return accessRequests;
}
You could also use the Stream API for this (using the filter operation), but that's quite a bit trickier in this situation.
You are passing a reference of the list to the method getCommentsListProcessedManually.
So accessRequestList and the one passed as a parameter are the same, hence any operation done to the list is done to the same list.
You can create a copy of the list before passing it as a parameter:
List<AccessRequest> newList = new ArrayList<AccessRequest>(accessRequestList);
Here is the situation - I need to maintain a collection(unbounded/single writer) of ids(string) in Java 7.
As new records come in with a particular flag set (immaterial)- I attempt to insert.
If I find a pre-existing record in the collection - I alert and over-write anyways.
If new records come in with the flag unset - I attempt to remove the record,if one exists.
All lookups(2-step) to be avoided for performance.
Insert/update/remove to be as close to O(1) as possible.
Would HashSet be the most apt collection for this?
If your code is single-threaded, HashSet should be a good match. Sample implementation:
Set<String> ids = new HashSet<>();
void processRecord(Record record) {
if (record.hasFlag()) {
if (!ids.add(record.getId())) {
alertDuplicate(record);
}
} else {
ids.remove(record.getId());
}
}
I have extended the EmptyInterceptor provided by hibernate to perform some logic on post flush. The overwritten post flush method is provided with an iterator. When I tried to iterate, I received ConcurrentModificationException.
Below is my code snippet,
#Override
public void postFlush(Iterator entities) throws CallbackException
{
while (entities.hasNext())
{
Object entity;
try
{
entity = entities.next();
}
catch(ConcurrentModificationException e)
{
// I get concurrent modification exception while iterating.
return;
}
}
}
I am getting the below exception,
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$ValueIterator.next(HashMap.java:950)
at org.hibernate.internal.util.collections.LazyIterator.next(LazyIterator.java:51)
at com.mycompany.MyInterceptor.postFlush(MyInterceptor.java:55)
at org.hibernate.event.internal.AbstractFlushingEventListener.postPostFlush(AbstractFlushingEventListener.java:401)
at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:70)
at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1130)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1580)
at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:374)
From Hibernate Forum we can understand that the iterator passed to the postFlush() method is not thread safe causing ConcurrentModificationException.
Suggestions and solution to avoid the exception is appreciated.
If it's synchronization issue try using a ConcurrentHashMap instead of a plain HashMap
See also this answer i think it might help
Manually copy it in a List
#Override
public void postFlush(Iterator entities) {
super.postFlush(entities);
List<Object> objects = new ArrayList<>();
while (entities.hasNext()) {
objects.add(entities.next());
}
.
.
.
now you can use objects list
If you look at the implementation of IteratorUtils.toList, it just does:
List list = new ArrayList(estimatedSize);
while (iterator.hasNext()) {
list.add(iterator.next());
}
which isn't any faster than doing it that way, except... perhaps by allocating the list with an estimated size of 10, it is faster because it isn't necessarily having to re-allocate...
Copying iterator into a list via org.apache.commons.collections.IteratorUtils before iterating worked for me :
#Override
public void preFlush(Iterator entities) {
List list= IteratorUtils.toList(entities);
for(Object o : list){...}
}
However i can't explain why it is working when using IteratorUtils...
I'm having issues with getting an iteration done (and modification) through the Set, which contains Objects. I've tried so many ways of iteration (4), but none of them seem to work and still throw me the Error java.util.ConcurrentModificationException.
[Code is written in Groovy]
private void replaceRock() {
ObjectNodeManager.OBJECTS.each {
System.out.println("Going...");
if(it.getPosition().withinDistance(player.getPosition(), 30)) {
System.out.println("Found...");
Position position = it.getPosition();
ObjectNode newRock = new ObjectNode(439, position, ObjectDirection.NORTH, ObjectType.DEFAULT);
ObjectNodeManager.unregister(it);
ObjectNodeManager.register(newRock);
it.remove();
}
}
}
I've tried synchronization to prevent access from other Threads, but this also didn't work. Please help me, I'm very desperate.
First find them (this will give you basically a list of refs) and then deal with them:
ObjectNodeManager.OBJECTS.findAll {
it.getPosition().withinDistance(player.getPosition(), 30))
}.each{
ObjectNode newRock = new ObjectNode(439, it.position, ObjectDirection.NORTH, ObjectType.DEFAULT)
ObjectNodeManager.unregister(it)
ObjectNodeManager.register(newRock)
it.remove()
}
On a random site note: i'd add a replace method in the ObjectNodeManager to combine unregister, register, remove. Also working with class methods and properties is not the best thing to do (but since it looks like a game...)
The problem is that you are modifying the list of objects while you are looping through the objects.
Try iterating through a copy of the objects instead.
ArrayList<YourType> copy = new ArrayList<YourType>(ObjectNodeManager.OBJECTS);
copy.each(...)