I am using myBatis 3.2.x and have run into a scenario where I need to do multiple table inserts in one database trip,
I was wondering if I can create a master INSERT sql mapper file which would call these multi table inserts and save me network trips
I am consuming JSON objects from a EMS server and my Turn around time is a bit higher then required.
All suggestions and hints are welcome.
Thanks
VR
Use Collections.sort() to sort and use a simple for cycle to catch doubles, e.g.:
Collections.sort(myList);
A previous = null;
for (A elem: myList) {
if (elem.compareTo(previous) == 0) {
System.err.println("Duplicate: "+elem);
}
previous = elem;
}
Assuming that the Comparable is consistent with the equals implementation, you can use a Set. You can add each element to a Set using Set.add(..) and use the return value of add to determine if the value was already present in the Set and create either a Set or a List to return.
Note: If you need each duplicate returned only once, you can change the return list to a set.
List<A> duplicates(List<A> myList) {
Set<A> s = new HashSet<A>();
List<A> duplicates = new ArrayList<A>(); // change to using a Set if you want to report each duplicate item only once.
for (A item: myList) {
if (!s.add(item)) {
duplicates.add(item);
}
}
return duplicates;
}
An improved version using sorting (to report duplicate elements only once, I assume there are no null values in the list):
Collections.sort(myList);
A previous = null, elem = null;
for (java.util.Iterator<A> it = myList.iterator; it.hasNext(); elem = it.next()) {
if (elem.compareTo(previous) == 0) {
System.err.println("Duplicate: "+elem);
while (it.hasNext() && (elem = it.next()).compareTo(previous)) {
//loop through other values
}
}
previous = elem;
}
A version using SortedSet (probably this is faster a bit): and corrected the same
SortedSet<A> set = new TreeSet<>(), duplicates = new TreeSet<>();
for (A a: myList) {
if (!set.add(a)) {
duplicates.add(a);
}
}
return duplicates;//or for (A a: duplicates) System.println("Duplicate: " + a);
Related
I have an ArrayList of objects that have a version number as a field. I want to do some work on that ArrayList, but I only want the most recent version of the object. I was thinking of coding as such:
ArrayList<ObjectVO> ObjectList = getObjectList();
for(ObjectVO myVO : ObjectList) {
Iterator<ObjectVO> iter = ObjectList.iterator();
while(iter.hasNext()) {
ObjectVO checkVO = iter.next();
if(myVO.getID().equals(checkVO.getID()) {
//they are the same object ID. Check the version number, remove it lower
if(myVO.getVersion() > checkVO.getVersion()) {
iter.remove();
}
}
}
}
Is this valid? I don't know if the fact that we are in a for loop originally would break the mutability of the ArrayList at runtime.
No, this won't work. iter.remove() will cause the out for loop to fail with ConcurrentModificationException.
Instead of doing this, you can do this with indexed for loops, and a BitSet to keep track of things you want to remove:
BitSet toRemove = new BitSet();
for (int m = 0; m < ObjectList.size(); ++m) {
if (toRemove.get(m)) continue;
ObjectVO myVO = ObjectList.get(m);
for (int c = 0; c < ObjectList.size(); ++c) {
if (toRemove.get(c)) continue;
ObjectVO checkVO = ObjectList.get(c);
if(myVO.getID().equals(checkVO.getID()) {
//they are the same object ID. Check the version number, remove it lower
if(myVO.getVersion() > checkVO.getVersion()) {
toRemove.set(c);
}
}
}
}
This is basically your code, but it doesn't do the removal yet. Then you can sweep through the list after and remove them:
int dst = 0;
for (int src = 0; src < ObjectList.size(); ++src) {
if (!toRemove.get(src)) {
ObjectList.set(dst++, ObjectList.get(src));
}
}
ObjectList.subList(dst, ObjectList.size()).clear();
The point of using a BitSet like this is that removal from an ArrayList is inefficient if you are removing from anywhere other than the end, because it requires all of the elements "to the right" of the element you remove to be shuffled along by one position. The loop with the set/get and clear allows you to only move each of the retained elements once.
You can do a bit better than the quadratic loop, though, if you group the list elements by things with the same ID: then you don't need to keep on checking the entire list:
BitSet toKeep = new BitSet();
IntStream.range(0, ObjectList.size())
.mapToObj(a -> a)
.collect(
groupingBy(a -> ObjectList.get(a).getID(),
maxBy(comparingInt(a -> ObjectList.get(a).getVersion()))))
.values()
.forEach(a -> toKeep.set(a));
int dst = 0;
for (int src = 0; src < ObjectList.size(); ++src) {
if (toKeep.get(src)) {
ObjectList.set(dst++, ObjectList.get(src));
}
}
ObjectList.subList(dst, ObjectList.size()).clear();
Assuming you have the memory, rather than do an O(N^2) operation, you could do this more efficiently (O(N)) by using a Map to track the newest Version for each Id. One pass tracks the newest version for each Id, and the second removes elements which are not the latest.
Map<Integer, Thing> newestById = new HashMap<>();
for (Thing thing : list) {
newestById.merge(thing.id, thing, (a,b) -> a.version > b.version ? a : b);
}
list.removeIf(thing -> thing != newestById.get(thing.id)); }
Depending on your use case, you might even be able to store your data in a Map instead of a List, and check if the version is the latest before adding it to the Map.
As the other answers have discussed this won't work. You have three options as I see them, trading memory for CPU cycles/flexibility. I've used Integer instead of ObjectVO in my examples, but it'll be trivial to swap them.
Option 1 - moderate memory, single-pass of the array
Track the highest ID you've seen and populate an ArrayList with new items as they meet the criteria. When you encounter a new higher ID, throw away the ArrayList and create a new one:
ArrayList<Integer> objectList = getObjectList();
Integer bestId = -1;
ArrayList<Integer> allObjectsMatchingId = new ArrayList<>();
for(Integer currentObject : objectList) {
if(currentObject > bestId) {
bestId = currentObject;
allObjectsMatchingId = new ArrayList<>();
} else if(currentObject == bestId) {
allObjectsMatchingId.add(currentObject);
}
}
return allObjectsMatchingId;
Option 2 - more expensive memory, single-pass of the array, most flexible.
For each ID you see, create an ArrayList and store it against a map. This allows you to easily change the criteria about what ID you want to keep.
ArrayList<Integer> objectList = getObjectList();
Map<Integer, ArrayList<Integer>> objectsById = new HashMap<>();
for(Integer currentObject : objectList) {
ArrayList<Integer> listForId = objectsById.get(currentObject);
if(listForId == null) {
listForId = new ArrayList<Integer>();
}
listForId.add(currentObject);
objectsById.put(currentObject, listForId);
}
Integer bestId = -1;
for(Integer i : objectsById.keySet()) {
if(i > bestId) {
bestId = i;
}
}
return objectsById.get(bestId);
Option 3 - no additional memory aside from id, two-passes of the array.
Search through the ArrayList for the highest ID, then filter the array to only elements that pass that filter.
This is the closest to your current implementation, the difference being that you do them in separate steps. This reduces complexity from O(N^2) to O(N), and is valid as you aren't modifying the ArrayList while iterating it. You could use a Stream here to filter instead of an iterator if you're Java 8 compatible. See Java: Efficient ArrayList filtering?
ArrayList<Integer> objectList = getObjectList();
Integer bestId = -1;
for(Integer currentObject : objectList) {
if(currentObject > bestId) {
bestId = currentObject;
}
}
Iterator<Integer> iter = objectList.iterator();
while(iter.hasNext()) {
if(iter.next() != bestId) {
iter.remove();
}
}
Why not use Java Streams to solve this:
Collection<ObjectVO> result = objectList.stream()
.collect(Collectors.toMap(ObjectVO::getID, Function.identity(),
BinaryOperator.maxBy(Comparator.comparing(ObjectVO::getVersion))))
.values();
This creates a map which contains the max version for each id. Then you can just use Map.values() to get the object list.
If you need a List or an ArrayList you can just use new ArrayList<>(result).
When I wrote this piece of code due to the pnValue.clear(); the output I was getting was null values for the keys. So I read somewhere that adding values of one map to the other is a mere reference to the original map and one has to use the clone() method to ensure the two maps are separate. Now the issue I am facing after cloning my map is that if I have multiple values for a particular key then they are being over written. E.g. The output I am expecting from processing a goldSentence is:
{PERSON = [James Fisher],ORGANIZATION=[American League, Chicago Bulls]}
but what I get is:
{PERSON = [James Fisher],ORGANIZATION=[Chicago Bulls]}
I wonder where I am going wrong considering I am declaring my values as a Vector<String>
for(WSDSentence goldSentence : goldSentences)
{
for (WSDElement word : goldSentence.getWsdElements()){
if (word.getPN()!=null){
if (word.getPN().equals("group")){
String newPNTag = word.getPN().replace("group", "organization");
pnValue.add(word.getToken().replaceAll("_", " "));
newPNValue = (Vector<String>) pnValue.clone();
annotationMap.put(newPNTag.toUpperCase(),newPNValue);
}
else{
pnValue.add(word.getToken().replaceAll("_", " "));
newPNValue = (Vector<String>) pnValue.clone();
annotationMap.put(word.getPN().toUpperCase(),newPNValue);
}
}
sentenceAnnotationMap = (LinkedHashMap<String, Vector<String>>) annotationMap.clone();
pnValue.clear();
}
EDITED CODE
Replaced Vector with List and removed cloning. However this still doesn't solve my problem. This takes me back to square one where my output is : {PERSON=[], ORGANIZATION=[]}
for(WSDSentence goldSentence : goldSentences)
{
for (WSDElement word : goldSentence.getWsdElements()){
if (word.getPN()!=null){
if (word.getPN().equals("group")){
String newPNTag = word.getPN().replace("group", "organization");
pnValue.add(word.getToken().replaceAll("_", " "));
newPNValue = (List<String>) pnValue;
annotationMap.put(newPNTag.toUpperCase(),newPNValue);
}
else{
pnValue.add(word.getToken().replaceAll("_", " "));
newPNValue = pnValue;
annotationMap.put(word.getPN().toUpperCase(),newPNValue);
}
}
sentenceAnnotationMap = annotationMap;
}
pnValue.clear();
You're trying a bunch of stuff without really thinking through the logic behind it. There's no need to clear or clone anything, you just need to manage separate lists for separate keys. Here's the basic process for each new value:
If the map contains our key, get the list and add our value
Otherwise, create a new list, add our value, and add the list to the map
You've left out most of your variable declarations, so I won't try to show you the exact solution, but here's the general formula:
List<String> list = map.get(key); // try to get the list
if (list == null) { // list doesn't exist?
list = new ArrayList<>(); // create an empty list
map.put(key, list); // insert it into the map
}
list.add(value); // update the list
It has been established that when you use Hibernate's Restrictions.in(String property, List list), you have to limit the size of list.
This is because the database server might not be able to handle long queries. Aside from adjusting the configuration of the database server.
Here are the solutions I found:
SOLUTION 1: Split the list into smaller ones and then add the smaller lists separately into several Restrictions.in
public List<Something> findSomething(List<String> subCdList) {
Criteria criteria = getSession().createCriteria(getEntityClass());
//if size of list is greater than 1000, split it into smaller lists. See List<List<String>> cdList
if(subCdList.size() > 1000) {
List<List<String>> cdList = new ArrayList<List<String>>();
List<String> tempList = new ArrayList<String>();
Integer counter = 0;
for(Integer i = 0; i < subCdList.size(); i++) {
tempList.add(subCdList.get(i));
counter++;
if(counter == 1000) {
counter = 0;
cdList.add(tempList);
tempList = new ArrayList<String>();
}
}
if(tempList.size() > 0) {
cdList.add(tempList);
}
Criterion criterion = null;
//Iterate the list of lists, add the restriction for smaller list
for(List<String> cds : cdList) {
if (criterion == null) {
criterion = Restrictions.in("subCd", cds);
} else {
criterion = Restrictions.or(criterion, Restrictions.in("subCd", cds));
}
}
criteria.add(criterion);
} else {
criteria.add(Restrictions.in("subCd", subCdList));
}
return criteria.list();
}
This is an okay solution since you will only have one select statement. However, I think it's a bad idea to have for loops on the DAO layer because we do not want the connection to be open for a long time.
SOLUTION 2: Use DetachedCriteria. Instead of passing the list, query it on the WHERE clause.
public List<Something> findSomething() {
Criteria criteria = getSession().createCriteria(getEntityClass());
DetachedCriteria detached = DetachedCriteria.forClass(DifferentClass.class);
detached.setProjection(Projections.property("cd"));
criteria.add(Property.forName("subCd").in(detached));
return criteria.list();
}
The problem in this solution is on the technical usage of DetachedCriteria. You usually use it when you want to create a query to a another class that is totally not connected (or does not have relationship) on your current class. On the example, Something.class has a property subCd that is a foreign key from DifferentClass. Another, this produces a subquery on the where clause.
When you look at the code:
1. SOLUTION 2 is simpler and concise.
2. But SOLUTION 1 offers a query with only one select.
Please help me decide which one is more efficient.
Thanks.
For Solution 1 : Instead of using for loops, you can try as below
To avoid this use an utility method to build the Criterion Query IN clause if the number of parameter values passed has a size more than 1000.
class HibernateBuildCriteria {
private static final int PARAMETER_LIMIT = 800;
public static Criterion buildInCriterion(String propertyName, List<?> values) {
Criterion criterion = null;
int listSize = values.size();
for (int i = 0; i < listSize; i += PARAMETER_LIMIT) {
List<?> subList;
if (listSize > i + PARAMETER_LIMIT) {
subList = values.subList(i, (i + PARAMETER_LIMIT));
} else {
subList = values.subList(i, listSize);
}
if (criterion != null) {
criterion = Restrictions.or(criterion, Restrictions.in(propertyName, subList));
} else {
criterion = Restrictions.in(propertyName, subList);
}
}
return criterion;
}
}
Using the Method :
criteria.add(HibernateBuildCriteria.buildInCriterion(propertyName, list));
hope this helps.
Solution 1 has one major drawback: you may end up with a lot of different prepared statements which would need to be parsed and for which execution plan would need to be calculated and cached. This process may be much more expensive than the actual execution of the query for which the statement has already been cached by the database. Please see this question for more details.
The way how I solve this is to utilize the algorithm used by Hibernate for batch fetching of lazy loaded associated entities. Basically, I use ArrayHelper.getBatchSizes to get the sublists of ids and then I execute a separate query for each sublist.
Solution 2 is appropriate only if you can project ids in a subquery. But if you can't, then you can't use it. For example, the user of your app edited 20 entities on a screen and now they are saving the changes. You have to read the entities by ids to merge the changes and you cannot express it in a subquery.
However, an alternative approach to solution 2 could be to use temporary tables. For example Hibernate does it sometimes for bulk operations. You can store your ids in the temporary table and then use them in the subquery. I personally consider this to be an unnecessary complication compared to the solution 1 (for this use case of course; Hibernate's reasoning is good for their use case), but it is a valid alternative.
I am using Spring-WS to populate a List<SomeBean> w.r.t. a web-service. The final list that I get after consuming the service contains null entries for some of its properties. For example, consider the following list of MangaBean which has title, author and genre as properties.
MangaBean [title=Bleach, author=Kubo Tite, genre=fantasy]
MangaBean [title=Naruto, author=Masashi Kishimoto, genre=action]
MangaBean [title=One Piece, author=Eiichiro Oda, genre=null]
MangaBean [title=Fairy Tail, author=Mashima Hiro, genre=null]
MangaBean [title=Rurouni Kenshin, author=Watsuki Nobuhiro, genre=Shounen]
MangaBean [title=Takamagahara, author=KAWAI Juuzou, genre=Supernatural]
MangaBean [title=Historys Strongest Disciple Kenichi, author=Matsuena Syun, genre=Martial arts]
MangaBean [title=Hajime no Ippo, author=Jyoji Morikawa, genre=null]
The genre entry for some beans are null. I am trying to remove the objects with null properties in the list.
My questions are:
Is it proper/good practice to alter the original list by removing specific entries? Or should I populate another list w.r.t. the previous list?
If I should alter the list and remove those null entries, how should I do it?
I tried this but it fails as the object-refs themselves aren't null. I tried the traditional for loop but it fails to remove adjucent objects with null properties. Cannot use the enhanced-for-loop for removing items while accessing the list. Kindly render me some brain. :)
You can use an Iterator and remove the one having a null genre manually:
for (Iterator<MangaBean> it = mangas.iterator(); it.hasNext();) {
if (it.next().getGenre() == null) {
it.remove();
}
}
Or you could filter the list using some common libraries like Apache Commons CollectionUtils, or Guava Iterables.filter():
// apache commons
mangas = CollectionUtils.filter(mangas, new Predicate() {
#Override public boolean evaluate(Object object) {
return ((Manga) manga).getGenre() != null;
}
});
// Guava
Iterables.filter(mangas, new Predicate<Manga>() {
#Override public boolean apply(Manga m) {
return manga.getGenre() != null;
}
});
There is nothing about removing from a List that is bad practice, unless you cannot trust the source of the List to give you a mutable reference.
If it is mutable (not read-only), then you can use an Iterator:
List<MangaBean> list = ...;
Iterator<MangaBean> iterator = list.iterator();
while (iterator.hasNext())
{
MangaBean bean = iterator.next();
if ( ! isValidMangaBean(bean))
{
iterator.remove();
}
}
The limitation presents itself with removing from the List if it is a read-only List. By creating a copy that only contains the non-null elements, you can guarantee that you will not hit any roadblocks, but using the above Iterator will work if the List is not read-only.
List<MangaBean> list = ...;
List<MangaBean> cleaned = new ArrayList<MangaBean>();
for (MangaBean bean : list)
{
if (isValidMangaBean(bean))
{
cleaned.add(bean);
}
}
Reference function that could be expanded upon:
boolean isValidMangaBean(MangaBean bean)
{
return bean.getGenre() != null;
}
If you continuously run into the above situation, then consider writing a Filter interface that would allow you to perform the above filtering much more generically.
I have a list of IDs: List<Integer> updatedIds.
I have a master list (say, taken from the DB): List<Records> masterList.
I want to do the following:
For each ID in updatedIds, check if it's in masterList. If not, add the record to the masterList.
For each Record in masterList, check if it's in updatedIds. If not, it is obsolete, so remove it from masterList.
The straightforward code for this is as follows:
for (Integer updId : updatedIds) {
boolean hasMapping = false;
for (Record rec : masterList) {
if (rec.getId() == updId) { hasMapping = true; break; }
}
if (!hasMapping) {
//TODO add updId to masterList
}
}
for (Record rec : masterList) {
boolean isObsolete = true;
for (Integer updId : updatedIds) {
if (rec.getId() == updId) { isObsolete = false; break; }
}
if (isObsolete) {
//TODO remove rec from masterList
}
}
The first loop takes care of requirement 1, the second takes care of requirement 2. It looks very inefficient, and I think I may be using the wrong data structure for this kind of task.
Is there a more efficient way of implementing the algorithm above?
If you sort both lists (e.g. using Collections.sort), the updatedIDs in natural order and the masterList ordered by ID, you can set up a single loop to go through both of them. You could possibly retrieve the records in sorted order, if they come from a DB, and skip that step.
Collections.sort(masterList, myComparator);
Collections.sort(updatedIDs);
Iterator m_it = masterList.iterator();
Iterator u_it = updatedIDs.iterator();
// * Some code here to deal with the possibility that either list is empty
Record rec = m_it.next();
int u = u_it.next();
bool done = false;
while (! done) {
if (rec.getID() < u) {
// rec's ID was missing from updatedIDs
m_it.remove();
if (m_it.hasNext()) {
rec = m_it.next();
} else {
done = true;
// * add u and all subsequent updated IDs to master list
}
} else if (rec.getID() > u) {
// u is new - doesn't occur in masterList
// * add u to masterList (or probably to a separate list that you
// later append to masterList)
if (u_it.hasNext()) {
u = u_it.next();
} else {
done = true;
// * remove rec and all remaining records from the master list
}
} else {
// rec's ID matches u: proceed to next pair of items
bool m_nx = m_it.hasNext(), u_nx = u_it.hasNext();
if (m_nx && u_nx) {
rec = m_it.next();
u = u_it.next();
} else if ((! m_nx) && (! u_nx)) {
done = true;
} else if (m_nx && (! u_nx)) {
done = true;
// * remove all subsequent records from the master list
} else if ((! m_nx) && u_nx) {
done = true;
// * add all subsequent integers in updatedIDs to the master list
}
}
}
use HashSet. that will give you a constant time look up. however, each item in your set should be unique. then you can use that number as hashcode as well and you have O(1) lookup whereas in List you have O(n) lookup time.
You can HashMap<Integer,Records> instead of List<Records>. Where you will get constant look up O(1).
HashMap -> Integer - id and Records - corresponding record.