Develop Reference

Iterating and comparing big data set

Basically I receive a 2 big data lists from 2 different database, the list looks like this:
List 1:
=============
A000001
A000002
A000003
.
.
A999999
List 2:
=============
121111
000111
000003
000001
.
.
I need to compare two list and find out each data which is in List 1 is available in List 2 (after appending some standard key to it), so that and if it is available put it in 3rd list for further manipulation. As an example A000001 is available in List 1 as well as in List 2 (after appending some standard key to it) so I need to put it in 3rd list.
Basically I have this code, it does like this for each row in List 1, I'm iterating through all data in List 2 and doing comparison. (Both are array list)
List<String> list1 = //Data of list 1 from db
List<String> list2 = //Data of list 2 from db
for(String list1Item:list1) {
for(String list2Item:list2) {
String list2ItemAfterAppend = "A" + list2Item;
if(list1Item.equalsIgnoreCase(list2ItemAfterAppend)) {
//Add it to 3rd list
}
}
}
Yes, this logic works fine, but I feel this is not efficient way to iterate list. After putting timers, it's taking 13444 milliseconds on average for 2000x5000 list of data. My question is, is there any other logic you people can think of or suggest me to improve the performance of this code?
I hope I'm clear, if not please let me know if I can improve question.

You can order both list, then using only one loop iterate on both value, switching which index increments depending on which value is the biggest. Something like:
boolean isWorking = true;
Collections.sort(list1);
Collections.sort(list2);
int index1 = 0;
int index2 = 0;
while(isWorking){
String val1 = list1.get(index1);
String val2 = "A" + list2.get(index2);
int compare = val1.compareTo(val2)
if(compare == 0){
list3.add(val1);
index1++;
index2++;
}else if (compare > 0){
val2++;
}else{ // if(compare < 0)
val1++;
}
isWorking = !(index1 == list1.size() || index2 == list2.size() );
}
Be carefull about what kind of List you're using. The get(int i) on LinkedList is expensive, whereas it is not on an ArrayList. Also, you might want to save list1.size() and list2.size(), I dont't think it calcluates it everytime, but chek it. I'm not sure if it's really usefull/efficient, but you can initialise list3 with the size of the smallest of both list (taking into acount the loadFactor, look up for it), so list3 doesnt have to resize everytime.
The code above is not tested (maybe switch val1++ and val2++), but you get the idea. I believe it's faster than yours (because it's O(n+m) rather than O(n*m) but I'll let you see (both sort() and compareTo() will add some time compared to your method, but normally it shouldn't be too much). If you can, use your RDBMS to sort both list when you get them (so you don't have to do it in the Java code)

I think the problem is how big the list is and how much memory you have.
For me for under 1 million records, I will use a HashSet to make it faster.
Code may like:
Set<String> set1 = //Data of list 1 from db, when you get the data you make it a Set instead of a List. HashSet is enough for you to use.
List<String> list2 = //Data of list 2 from db
Then you just need to:
for(String list2Item:list2) {
if(set1.contains("A" + list2Item) {
}
}
Hope this can help you.

You can use intersection method from apache commons. Example:
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.apache.commons.collections4.CollectionUtils;
public class NewClass {
public static void main(String[] args) {
List<String> list1 = Arrays.asList("A000001","A000002","A000003");
List<String> list2 = Arrays.asList("121111","000111","000001");
List<String> list3 = new ArrayList<>();
list2.stream().forEach((s) -> {list3.add("A"+s);});
Collection<String> common = CollectionUtils.intersection(list1, list3);
}
}

You could try to use the Stream API for this, the code to create the new list with Streams is very concise and straightforward and probably very similar in performance:
List<String> list3 = list2.stream()
.map(s->"A"+s)
.filter(list1::contains)
.collect(Collectors.toList());
If the list are big, you could try to process the list in parallel and use multiple threads to process the list. This may or may not improve the performance. Doing some measures its important to check if processing the list in parallel is actually improving the performance.
To process the stream in parallel, you only need to call the method parallel on the stream:
List<String> list3 = list2.stream()
.parallel()
.map(s->"A"+s)
.filter(list1::contains)
.collect(Collectors.toList());

Your code is doing a lot of String manipulation, 'equalsIgnoreCase' convert the Characters to upper/lower case. This is being executed in your inner loop and the size of your list is 5000x2000, so the String manipulation is being done millions of times.
Ideally, get your Strings in either upper or lower case from the database and avoid the conversion inside the inner loop. If this is not possible, probably converting the case of the String at the beginning improves the performance.
Then, you could create a new list with the elements of one of the lists and keep all the elements present in the other list, the code with the uppercase conversion could be:
list1.replaceAll(String::toUpperCase);
List<String> list3 = new ArrayList<>(list2);
list3.replaceAll(s->"A"+s.toUpperCase());
list3.retainAll(list1);

Java for each statement

I'm really good with VB and I have a project where I need to check an array. If the same item in an array exists twice or more it needs to be changed to an item that doesn't exist. Now I'm in a class where they're making us use Java for this project.
I was wondering what is the equivalent of a for each loop in Java? I checked the JavaDocs and it only had info for the regular for loop, I didn't notice any section that said anything about a for each loop.

It's more subtle in Java than VB. You can find the official docs in the Oracle documentation here (towards the bottom):
Java For Loops
The provided example is:
// Returns the sum of the elements of a
int sum(int[] a) {
int result = 0;
for (int i : a)
result += i;
return result;
}
Hope that helps. Be careful not to remove or add elements inside the loop or you will get a Concurrent Modification Exception.

try
String arr [] = // you decide how this gets initialized
for (String obj: arr) {
}

This is called "iterating over collections". An array can be implicitly converted to a collection, so you can iterate over an array in the same way, using the "enhanced for-loop".
List<String> names = new LinkedList<>();
// ... add some names to the collection
for(name:names) {
System.out.println(name);
}
I'm not sure if VB has collections - they are a big part of Java and I recommend you look into them.
Of course this changes a bit in Java 8, although you'll notice a collection is still the backbone of forEach().
List<String> names = new LinkedList<>();
// ... add some names to the collection
names.forEach(name -> System.out.println(name));

A for each loop (also known as the enhanced for loop) is as follows:
for (String name : names) {
// here, the loop will work over each element of 'names',
// with the variable name with which to access each element
// being 'name', and output it
System.out.println(name);
}
A normal for loop is as follows:
for (int i = 0; i < max; i++) {
// here, i will iterate until max, then the loop will stop.
// any array access here has to be done manually using i, which increments.
}
If insertion order from the names array is important, keep adding the objects to a LinkedHashSet<String>, then with either a for loop or enhanced for loop or iterator, go over your list of names and add each of them to the LinkedHashSet. If the add method, passing in your name, returns false, generate a new name and add that.
If insertion order is not important, use a HashSet<String> instead.
At the end, convert back to an array if it is important (String[] bla = map.toArray(new String[0])), or output the toString() of the map.

How to convert Vector elements to comma separated String

I am having a Vector<String> containing some items. I want to search those items in database.
For that purpose I need to make a String consisting of comma separated items out of that Vector so that I can pass it to SQL Query.
I am doing something like this.
StringBuilder list = new StringBuilder("");
for (String items : itemList) {
if(its last element then) //How to check here
list.append(items);
else
list.append(items+",");
}
but that will return me the output like "item1,item2,item3,"
where as I want to omit the last comma (,) in case if it is last element of Vector.
I have checked Vector API and found this method lastElement() (which returns last element not boolean).
How can I check it in the loop or is there any smarter/efficient way to do it?

I would go with the String.join approach.
final List<String> itemList = Arrays.asList("item1", "item2", "item3");
final String commaSeparated = String.join(",", itemList);
System.out.println(commaSeparated); // -> item1,item2,item3
A nice and clean solution. Only available for Java 8.

This is a possible duplicate of Java: function for arrays like PHP's join()?
Don't reinvent the wheel! This has been done many many times, unit tested and maintained. so use them, don't make your own.
You can use Apache Commons StringUtils and use the join method on Iterable (which Vector is)
StringUtils.join(itemList, ", ")
Also, as pointed out by Tom in the comments of your question, in Java 8 you can use String.join()

Have a look at libraries like Google Guava
Joiner.on(",").useForNull("null").join(itemList)

Since you're using a StringBuilder, it's easier to just delete the last comma:
StringBuilder list = new StringBuilder("");
for (String items : itemList) {
list.append(items).append(',');
}
if ( list.length() > 0 ) {
list.setLength(list.length() - 1 );
}

Change your loop as below:
for (int i=0; i<itemList.size(); i++) {
list.append(itemList.get(i));
if(i != itemList.size() - 1) {
list.append(",");
}
Internally vector uses array and you are accessing element by index now instead of using advance for each loop.

There are several third party libraries that do the heavy lifting for you, like Apache Commons Lang's StringUtils:
String list = StringUtils.join(items, ',');
If you absolutely must do this yourself, I'd start with the first element and append a comma and an element for each succeeding element:
StringBuilder list = new StringBuilder();
Iterator<String> iter = items.iterator();
if (iter.hasNext()) {
// Special treatment for the first item
list.append(iter.next());
// The rest of the items
while (iter.hasNext()) {
list.append(',').append(iter.next());
}
}

Simply use,
StringUtils.join(itemList, ", ");

How to make a Java array behave like a JavaScript array?

In JavaScript, I can write code like this:
var a = new Array();
a[2] = 'a';
a[20] = 'b';
and this would not work on Java, the point is I don't want to specific the exact length for it.
How could I keep this happy style when writing java?

If you don't want to specific length you can use List like this:
List<Character> list = new ArrayList<>();
list.add('a');
list.add('b');

You cannot. Java is to Javascript as ham is to hamster. There is no reason to believe they have the same syntax.
If you want a sparse array, use a Map:
final Map<Integer, Character> a = new LinkedHashMap<>();
a.put(2, 'a');
a.put(20, 'b');

When you want an array of characters you can do it like this:
char[] array = new char[30];
array[2] = 'a';
array[20] = 'b';

Java and JavaScript are two deferent languages. you can't do same thing in both
In Java you can write
char[] arr=new char[2];
arr[0]='a';
arr[1]='b';
If you don't want to specific the length, you can use List in Java
List<Character> list=new ArrayList<>();
list.add('a');
list.add('b');

As others already pointed out: Java and JavaScript are two different things. For containers with variable size there is the Java collections framework.
But wich to choose? That depends on what you need. From your question I can imagine two cases:
a variable sized, indexed container:
basically an array-like list. In Java there's besides other list implementations the ArrayList used as follows:
List<Character> myList = new ArrayList<Character>();
// insert element at the end of the list
myList.add('a');
// insert element at specific position in list
myList.add(1, 'b');
// this will fail, because there's no element at position 2!!!
myList.add(3, 'c')
a container for mappings from integer to character:
In java there's lots of map implementations, I propose the HashMap, used like this:
Map<Integer,Character> myMap = new HashMap<Integer,Character>();
// insert mappings int -> char
myMap.put(0, 'a');
myMap.put(1, 'b');
myMap.put(20, 'c');
Each Container serves a different purpose. I advise reading the Java collections tutorial to be able to choose the best fitting one. Also take a look at tucuxi's answer, as he presented a solution wich simulates the desired beahviour but consider that (as he said himself) this is not the java way of doing things!

You can always write your own. You will not get the syntax, but most of the flavor will still be there.
Note that, efficiency-wise, this is a terrible idea. You can write much better code by learning "the Java way" of doing things. This is true of all languages: programming against the grain of the language is sure to cause you pain.
But here is the code:
class MyArray<T> extends ArrayList<T> {
public MyArray<T>() { super(); }
public void add(int i, T value) {
if (size() < i) {
ensureCapacity(i+1); // grow at most once instead of multiple times
while (size() < i)) {
add(null); // extend with a null object
}
add(value);
} else {
add(i, value);
}
}
}
Now you can compare a garden-variety ArrayList with an instance of MyArray:
ArrayList<Character> a = new ArrayList<>();
MyArray<Character> b = new MyArray<>()
a.add(10, 'X'); // IndexOutOfBoundsException, size is 0
b.add(10, 'X'); // no exception - you get [10 x null, 'X']
b.get(10); // returns 'X'
Bear in mind that JavaScript arrays can be indexed by arbitrary objects and not only integers -- but that the JavaScript VM tries to use numerically-indexed arrays if at all possible. For arbitrary indexing, you would need to use a Java HashMap:
class MyArray2<T> extends HashMap<Object, T> {
public MyArray2<T>() { super(); }
public void add(Object o, T value) { set(o, value); }
}
You would then use as:
MyArray2<Character> c = new MyArray2<>()
c.add("anything", '?');
c.get("anything"); // returns '?'

It depends, if you are going to use an array of a fixed size you can use:
char myarray[]=new char[50];
myarray[2]='a';
myarray[20]='b';
If you are going to change the size of the array dynamically you can use a Collection like an ArrayList (look at the doc) and insert chars in the positions you want

like this
char arr[]=new char[30]; //declares an array which can hold 30 characters
arr[2]='a';
arr[20]='b';
but if you don't want to specify the length,than arraylist is something which will help you to accomplish your task because array's size is always fixed in Java

Is there a way to use an enhanced for loop to copy an array?

I just started using the enhance for-loop. I want to know if I can use this loop to copy an array. I want to iterate through every element of a certain array and copy it to a new one. It would also be nice to use the enhanced for-loop to instantiate my new array (instead of a typical for-loop). In my current implementation I do know how big I want the array to be, but for future reference I would like to know if I can do this, and if so, how.
My specific plans for what I'm doing might help to answer my question. What I am doing is retrieving a line of text from a text file then calling split( "," ) on that string - this returns an array of Strings. I want to store this array in memory so I can play with it later.
The way I understand the enhanced for-loop to work is that the first value is assigned the current position in the array and the second value is the array that is to be traversed.
I was wondering if there are other formats for for-loops, besides: for ( initialization; termination; iterate ) and for ( Object o : list[] ).

If you want to keep to the enhanced for loop for copying an array, there is one mayor problem: the enhanced for loop doesn't have a counter. Inserting elements into an array however requires a counter. So you could of course do this manually like so:
String[] array = {"A", "Bb", "c", "dD"};
String[] newArray = new String[array.length];
int i=0;
for(String stuff : array) {
newArray[i++] = stuff;
}
This is entirely possible but not really the idea behind the enhanced for loop.
More in line with the intention would be something like this:
String[] array = {"A", "Bb", "c", "dD"};
List<String> list = new ArrayList<String>();
for(String stuff : array) {
list.add(stuff);
}
String[] newArray = new String[list.size()];
list.toArray(newArray);
That way not only do you follow the idea behind the enhanced for loop, you also allow for the possibility that array.length() != newArray.length() (because, say, you filtered the array).
EDIT: as of Java 7, there are indeed only the two for loops you mentioned. This may change in future versions though if it seems sensible; after all, the enhanced for loop was only added in Java 5 (as can be seen here).

To my knowledge, there are only standard for(init; termination; iterataion) loops and for-each for(type o : iterable) loops.
First, knowing the size ahead of time shouldn't be a concern. For instantiating the new array use the original array's .length field: new String[original.length]; as shown below.
Moving along, for what you are doing, the standard for loop is appropriate for two reasons:
You would need to nest two for-each loops in order to iterate both
loops, making it more hassle than a standard for loop. (or add an externally defined counter, as in blalasaadri's solution)
More importantly, in the case of a primitive data type or a String, the variable declared before the : in the for-each loop represents the value of each successive element, and is not a reference to the actual element. As such, any changes to the variable are gone once the loop iterates. I'm not sure if this holds for 'normal' objects (ie: not String), as I've not tried, though I want to find out now.
To illustrate:
String[] sArr = {"foobar"};
for(String s : sArr){
s = "openbar";
}
is the equivalent of:
String[] sArr = {"foobar"};
String s = sArr[0];
s = "openbar";
Sadly, for sArr[0], there is no open bar, same old foobar.
As for solutions, if you can import java.util.Arrays; then try:
String[] copyStrings = Arrays.copyOf(arrStrings, arrStrings.length);
Or, if you need to roll your own:
public String[] copyArray(String[] original){
String[] dupe = new String[original.length]; //I assume you want equal length
for(int i = 0; i < original.length; i++){
dupe[i] = original[i]; //single iterator traverses both arrays
}
return dupe;
}

// copy contents of Object[] arr1 into Object[] arr2
arr2 = new int[arr1.length];
int i=0;
for(Object c:arr1){ //store an element of arr1 in c iteratively
arr2[i] = c;
i++;
}