Develop Reference

Iterating and comparing big data set

Basically I receive a 2 big data lists from 2 different database, the list looks like this:
List 1:
=============
A000001
A000002
A000003
.
.
A999999
List 2:
=============
121111
000111
000003
000001
.
.
I need to compare two list and find out each data which is in List 1 is available in List 2 (after appending some standard key to it), so that and if it is available put it in 3rd list for further manipulation. As an example A000001 is available in List 1 as well as in List 2 (after appending some standard key to it) so I need to put it in 3rd list.
Basically I have this code, it does like this for each row in List 1, I'm iterating through all data in List 2 and doing comparison. (Both are array list)
List<String> list1 = //Data of list 1 from db
List<String> list2 = //Data of list 2 from db
for(String list1Item:list1) {
for(String list2Item:list2) {
String list2ItemAfterAppend = "A" + list2Item;
if(list1Item.equalsIgnoreCase(list2ItemAfterAppend)) {
//Add it to 3rd list
}
}
}
Yes, this logic works fine, but I feel this is not efficient way to iterate list. After putting timers, it's taking 13444 milliseconds on average for 2000x5000 list of data. My question is, is there any other logic you people can think of or suggest me to improve the performance of this code?
I hope I'm clear, if not please let me know if I can improve question.

You can order both list, then using only one loop iterate on both value, switching which index increments depending on which value is the biggest. Something like:
boolean isWorking = true;
Collections.sort(list1);
Collections.sort(list2);
int index1 = 0;
int index2 = 0;
while(isWorking){
String val1 = list1.get(index1);
String val2 = "A" + list2.get(index2);
int compare = val1.compareTo(val2)
if(compare == 0){
list3.add(val1);
index1++;
index2++;
}else if (compare > 0){
val2++;
}else{ // if(compare < 0)
val1++;
}
isWorking = !(index1 == list1.size() || index2 == list2.size() );
}
Be carefull about what kind of List you're using. The get(int i) on LinkedList is expensive, whereas it is not on an ArrayList. Also, you might want to save list1.size() and list2.size(), I dont't think it calcluates it everytime, but chek it. I'm not sure if it's really usefull/efficient, but you can initialise list3 with the size of the smallest of both list (taking into acount the loadFactor, look up for it), so list3 doesnt have to resize everytime.
The code above is not tested (maybe switch val1++ and val2++), but you get the idea. I believe it's faster than yours (because it's O(n+m) rather than O(n*m) but I'll let you see (both sort() and compareTo() will add some time compared to your method, but normally it shouldn't be too much). If you can, use your RDBMS to sort both list when you get them (so you don't have to do it in the Java code)

I think the problem is how big the list is and how much memory you have.
For me for under 1 million records, I will use a HashSet to make it faster.
Code may like:
Set<String> set1 = //Data of list 1 from db, when you get the data you make it a Set instead of a List. HashSet is enough for you to use.
List<String> list2 = //Data of list 2 from db
Then you just need to:
for(String list2Item:list2) {
if(set1.contains("A" + list2Item) {
}
}
Hope this can help you.

You can use intersection method from apache commons. Example:
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.apache.commons.collections4.CollectionUtils;
public class NewClass {
public static void main(String[] args) {
List<String> list1 = Arrays.asList("A000001","A000002","A000003");
List<String> list2 = Arrays.asList("121111","000111","000001");
List<String> list3 = new ArrayList<>();
list2.stream().forEach((s) -> {list3.add("A"+s);});
Collection<String> common = CollectionUtils.intersection(list1, list3);
}
}

You could try to use the Stream API for this, the code to create the new list with Streams is very concise and straightforward and probably very similar in performance:
List<String> list3 = list2.stream()
.map(s->"A"+s)
.filter(list1::contains)
.collect(Collectors.toList());
If the list are big, you could try to process the list in parallel and use multiple threads to process the list. This may or may not improve the performance. Doing some measures its important to check if processing the list in parallel is actually improving the performance.
To process the stream in parallel, you only need to call the method parallel on the stream:
List<String> list3 = list2.stream()
.parallel()
.map(s->"A"+s)
.filter(list1::contains)
.collect(Collectors.toList());

Your code is doing a lot of String manipulation, 'equalsIgnoreCase' convert the Characters to upper/lower case. This is being executed in your inner loop and the size of your list is 5000x2000, so the String manipulation is being done millions of times.
Ideally, get your Strings in either upper or lower case from the database and avoid the conversion inside the inner loop. If this is not possible, probably converting the case of the String at the beginning improves the performance.
Then, you could create a new list with the elements of one of the lists and keep all the elements present in the other list, the code with the uppercase conversion could be:
list1.replaceAll(String::toUpperCase);
List<String> list3 = new ArrayList<>(list2);
list3.replaceAll(s->"A"+s.toUpperCase());
list3.retainAll(list1);

Java for each statement

I'm really good with VB and I have a project where I need to check an array. If the same item in an array exists twice or more it needs to be changed to an item that doesn't exist. Now I'm in a class where they're making us use Java for this project.
I was wondering what is the equivalent of a for each loop in Java? I checked the JavaDocs and it only had info for the regular for loop, I didn't notice any section that said anything about a for each loop.

It's more subtle in Java than VB. You can find the official docs in the Oracle documentation here (towards the bottom):
Java For Loops
The provided example is:
// Returns the sum of the elements of a
int sum(int[] a) {
int result = 0;
for (int i : a)
result += i;
return result;
}
Hope that helps. Be careful not to remove or add elements inside the loop or you will get a Concurrent Modification Exception.

try
String arr [] = // you decide how this gets initialized
for (String obj: arr) {
}

This is called "iterating over collections". An array can be implicitly converted to a collection, so you can iterate over an array in the same way, using the "enhanced for-loop".
List<String> names = new LinkedList<>();
// ... add some names to the collection
for(name:names) {
System.out.println(name);
}
I'm not sure if VB has collections - they are a big part of Java and I recommend you look into them.
Of course this changes a bit in Java 8, although you'll notice a collection is still the backbone of forEach().
List<String> names = new LinkedList<>();
// ... add some names to the collection
names.forEach(name -> System.out.println(name));

A for each loop (also known as the enhanced for loop) is as follows:
for (String name : names) {
// here, the loop will work over each element of 'names',
// with the variable name with which to access each element
// being 'name', and output it
System.out.println(name);
}
A normal for loop is as follows:
for (int i = 0; i < max; i++) {
// here, i will iterate until max, then the loop will stop.
// any array access here has to be done manually using i, which increments.
}
If insertion order from the names array is important, keep adding the objects to a LinkedHashSet<String>, then with either a for loop or enhanced for loop or iterator, go over your list of names and add each of them to the LinkedHashSet. If the add method, passing in your name, returns false, generate a new name and add that.
If insertion order is not important, use a HashSet<String> instead.
At the end, convert back to an array if it is important (String[] bla = map.toArray(new String[0])), or output the toString() of the map.

How to convert a List of String arrays to a List of objects

I want to convert a List to a List so that each object on my new list includes the first element of each String[].
Do you know if this is possible to do in java?
for example:
public List<String[]> readFile(){
String[]array1={"A","1.3","2.4","2.3"};
String[]array2={"B","1.43","3.4","2.8"};
String[]array3={"C","5.23","2.45","2.9"};
List<String[]>ReadFile= new ArrayList<String[]>();
ReadFile.add(array1);
ReadFile.add(array2);
ReadFile.add(array3);
return ReadFile;
}
Now I want a method which will take the List ReadFile from above to somehow split the arrays of strings into an ID which will be the first element "A", "B", "C" and another part which would be the string array of numbers which I will put through another method to convert numbers from String to type Double. I have already got the method to convert to double but I need to be able to keep track of the ID field because the ID field will be used to identify the array of numbers.
A friend suggested that I create an Object where each objects has one part as a String ID and the other part as an array. That is the part which I do not know how to do.
Can anybody help please?
below is the method declaration which I believe I should have so the return type will be List where each array has been converted to an Object with two parts.
public List<Object> CreateObject(List<String[]>ReadFile){
}
Thanks,
Jetnori.
Hi all, Thank you for taking your time to help.
I can see the benefit of using HashTables. I am as of now trying to implement it. I know i might be sidetracking a little but just to explain what I am trying to do:
In my project I have CSV file with data about gene expression levels. The method that I use from OpenCSV to read the file returns a List(String[]) where each String[] is one row in the file. The first element of each row is variable name (recA, ybjE etc). The rest of the row will be numbers data related to that variable. I want to calculate Pearson's correlation between each of the number arrays. The method which I have got implemented already does that for me but the problem that I have now is that I had to remove the string values from my arrays before I could convert to double by iterating over the array. After I have managed to calculate the correlation between each array of doubles by still keeping the ID linked to the row, I want to be able to draw an undirected node graph between the genes that have a correlation higher than a threshold which I will set (for example correlation higher than 0.80). I don't know if i am biting more than i can chew but I have 30 days to do it and I believe that with the help of people like you guys I will get through it.
Sorry for going on for a bit.
thanks,
Jetnori.

I agree with the answer Alb provided, however this is what your friend has suggested, first you need a class to represent the data. I have included a constructor that parses the data and one that accepts already parsed data, depending on how you like to think of things.
public class NumberList {
private double[] numbers;
private String key;
public NumberList(Strig key, double[] numbers){
this.ley = key;
this.numbers = numbers;
}
public NumberList(String[] inputList) {
key = inputList[0];
numbers = new double[inputList.length-1];
for(int i=1;i<inputList.length;i++){
numers[i-1] = Double.parseDouble(inputList[i]);
}
}
public String getKey() {
return key;
}
public double[] getNumbers() {
return numbers;
}
}
Then you need your function to generate the list:
public List<NumberList> CreateObject(List<String[]> ReadFile){
ArrayList<NumberList> returnList = new ArrayList<NumberList>(ReadFile.size());
for (String[] input : ReadFile) {
returnList.add(new NumberList(input));
}
return returnList;
}
Note this uses the constructor that parses the data, if you use the other constructor then the "CreateObject" function would need to include the parsing logic.
Finally on a side note the standard convention in java is that the only thing that is capitalized are class names and final static fields (which appear in all caps sepearted by underscores), so conventionally the method signature would be:
public List<NumberList> createObject(List<String[]> readFile){
...
}

Sounds like you need a Map instead of a List, it lets you index things by a key (in your case ID).
Map<String, String[]> map = new Hashmap<String, String[]>();
for( String[] array : ReadFile ){
map.put( array[0], array );
}
then to get the array of values for 'A' you would do:
String[] values = map.get( "a" );
If you want the values to be doubles instead of strings you'll want to change the array before putting it (the map.put call) I'd advise using a list or other collections instead of an array also. You also will probably also want to remove the ID part from these values, which my code does not do.

public class Split_ListwithIDs {
Hashtable<String, String[]> table = new Hashtable<String, String[]>();
Splitter spl ;
public Split_ListwithIDs(Splitter split){
spl = split;
}
private void addEntry(String key , String[] vals){
table.put(key, vals);
}
public void parseList(List<String[]> list){
for(String[] entry : list){
String[] temp = new String[entry.length - 1];
System.arraycopy(entry, 1, temp, 0,entry.length - 1);
addEntry(entry[0], spl.GetStringArrayOfNumbers(temp));
}
}
class SplittingHelper implements Splitter{
#Override
public String[] GetStringArrayOfNumbers(String[] arr) {
String[] strArray = null ;
// implementation here
return arr;
}
}
interface Splitter {
String[] GetStringArrayOfNumbers(String[] arr);
}
}
You will have to use a Hashtable instead of a list of objects.( I am assuming that you will need to search through the list for a given entry using the First alphabet as key - This will be very laborious if you want to use a List ).
In the method SplittingHelper , provide your custom logic to parse the string of numbers and return another string[] of numbers.

I don't understand your goal, but for 'an object with 2 parts' you might consider storing them in a Hashtable: http://download.oracle.com/javase/6/docs/api/java/util/Hashtable.html

How to lowercase every element of a collection efficiently?

What's the most efficient way to lower case every element of a List or Set?
My idea for a List:
final List<String> strings = new ArrayList<String>();
strings.add("HELLO");
strings.add("WORLD");
for(int i=0,l=strings.size();i<l;++i)
{
strings.add(strings.remove(0).toLowerCase());
}
Is there a better, faster way? How would this example look like for a Set? As there is currently no method for applying an operation to each element of a Set (or List) can it be done without creating an additional temporary Set?
Something like this would be nice:
Set<String> strings = new HashSet<String>();
strings.apply(
function (element)
{ this.replace(element, element.toLowerCase();) }
);
Thanks,

Yet another solution, but with Java 8 and above:
List<String> result = strings.stream()
.map(String::toLowerCase)
.collect(Collectors.toList());

This seems like a fairly clean solution for lists. It should allow for the particular List implementation being used to provide an implementation that is optimal for both the traversal of the list--in linear time--and the replacing of the string--in constant time.
public static void replace(List<String> strings)
{
ListIterator<String> iterator = strings.listIterator();
while (iterator.hasNext())
{
iterator.set(iterator.next().toLowerCase());
}
}
This is the best that I can come up with for sets. As others have said, the operation cannot be performed in-place in the set for a number of reasons. The lower-case string may need to be placed in a different location in the set than the string it is replacing. Moreover, the lower-case string may not be added to the set at all if it is identical to another lower-case string that has already been added (e.g., "HELLO" and "Hello" will both yield "hello", which will only be added to the set once).
public static void replace(Set<String> strings)
{
String[] stringsArray = strings.toArray(new String[0]);
for (int i=0; i<stringsArray.length; ++i)
{
stringsArray[i] = stringsArray[i].toLowerCase();
}
strings.clear();
strings.addAll(Arrays.asList(stringsArray));
}

You can do this with Google Collections:
Collection<String> lowerCaseStrings = Collections2.transform(strings,
new Function<String, String>() {
public String apply(String str) {
return str.toLowerCase();
}
}
);

If you are fine with changing the input list here is one more way to achieve it.
strings.replaceAll(String::toLowerCase)

Well, there is no real elegant solution due to two facts:
Strings in Java are immutable
Java gives you no real nice map(f, list) function as you have in functional languages.
Asymptotically speaking, you can't get a better run time than your current method. You will have to create a new string using toLowerCase() and you will need to iterate by yourself over the list and generate each new lower-case string, replacing it with the existing one.

Try CollectionUtils#transform in Commons Collections for an in-place solution, or Collections2#transform in Guava if you need a live view.

This is probably faster:
for(int i=0,l=strings.size();i<l;++i)
{
strings.set(i, strings.get(i).toLowerCase());
}

I don't believe it is possible to do the manipulation in place (without creating another Collection) if you change strings to be a Set. This is because you can only iterate over the Set using an iterator or a for each loop, and cannot insert new objects whilst doing so (it throws an exception)

Referring to the ListIterator method in the accepted (Matthew T. Staebler's) solution. How is using the ListIterator better than the method here?
public static Set<String> replace(List<String> strings) {
Set<String> set = new HashSet<>();
for (String s: strings)
set.add(s.toLowerCase());
return set;
}

I was looking for similar stuff, but was stuck because my ArrayList object was not declared as GENERIC and it was available as raw List type object from somewhere. I was just getting an ArrayList object "_products". So, what I did is mentioned below and it worked for me perfectly ::
List<String> dbProducts = _products;
for(int i = 0; i<dbProducts.size(); i++) {
dbProducts.add(dbProducts.get(i).toLowerCase());
}
That is, I first took my available _products and made a GENERIC list object (As I were getting only strings in same) then I applied the toLowerCase() method on list elements which was not working previously because of non-generic ArrayList object.
And the method toLowerCase() we are using here is of String class.
String java.lang.String.toLowerCase()
not of ArrayList or Object class.
Please correct if m wrong. Newbie in JAVA seeks guidance. :)

Using JAVA 8 parallel stream it becomes faster
List<String> output= new ArrayList<>();
List<String> input= new ArrayList<>();
input.add("A");
input.add("B");
input.add("C");
input.add("D");
input.stream().parallel().map((item) -> item.toLowerCase())
.collect(Collectors.toCollection(() -> output));

For-Each and Pointers in Java [duplicate]

This question already has answers here:
Why does the foreach statement not change the element value?
(6 answers)
Closed 5 years ago.
Ok, so I'm tyring to iterate through an ArrayList and remove a specefic element. However, I am having some trouble using the For-Each like structure. When I run the following code:
ArrayList<String> arr = new ArrayList<String>();
//... fill with some values (doesn't really matter)
for(String t : arr)
{
t = " some other value "; //hoping this would change the actual array
}
for(String t : arr)
{
System.out.println(t); //however, I still get the same array here
}
My question in, how can I make 't' a pointer to 'arr' so that I am able to change the values in a for-each loop? I know I could loop through the ArrayList using a different structure, but this one looks so clean and readable, it would just be nice to be able to make 't' a pointer.
All comments are appreciated! Even if you say I should just suck it up and use a different construct.

I think the best approach may be to use a for loop.
ArrayList<String> arr = new ArrayList<String>();
for (int i = 0; i < arr.size(); i++) {
String t = arr.get(i);
if (// your condition is met) {
arr.set(i, "your new value");
}
}

The problem is that you're trying to change the loop-scoped reference t to let it point to a new String instance. This ain't going to work. It does not refer the actual entry in the arraylist. You need to change the actual value of the reference. If String was mutable and provided a fictive set() method for that, you could in theory do
for (String t : arr) {
t.set("some other value");
}
or so, but that's not possible as it is immutable. Better get a handle of the entrypoint in the array itself using the normal for loop:
for (int i = 0; i < arr.size(); i++) {
arr.set(i, "some other value");
}
If you insist in using the enhanced for loop, then you need to replace String by StringBuilder, which is mutable:
for (StringBuilder t : arr) {
t.delete(0, t.length()).append("some other value");
}
Remember, Java is pass-by-value, not pass-by-reference.

For-each doesn't give you an index pointer, so you just can't use it to change an immutable value.
Either use a for-loop with an index or use a mutable type (like StringBuffer, not String)

An array of objects (like strings) in Java is a contiguous block containing an ordered series of references. So, when you have an array of 4 strings, what you really have is 4 references stored IN the array, and 4 string objects that are outside of the array but are referenced by its 4 elements.
What the for-each construct in Java does is create a local variable and, for each iteration, copy into that local variable the reference from the array cell that corresponds to that iteration. When you set the loop variable (t = " some other value") you are putting a reference to a new string, "some other value", into the local variable t, not into the array.
The contrasts with some other languages (like Perl) where the loop variable acts like an alias to the array/list element itself.

Your code is re-written by the compiler as something like this:
ArrayList<String> arr = new ArrayList<String>();
//... fill with some values (doesn't really matter)
for (final Iterator <String> i = arr.iterator(); i.hasNext();) {
String t;
t = i.next();
t = " some other value "; // just changes where t is pointing
}
To do what you want you would have to write the for loop like this:
for (final ListIterator<String> i = arr.iterator(); i.hasNext();) {
final String t;
t = i.next();
i.set("some other value");
}
Iterator does not have the set method, only ListIterator does.

Basically you want to remove the String t from the list arr. Just do a arr.remove(t) and you could be done. But you can't do it while iterating over the same list. You'll get an Exception if you try to modify the list this way.
You have two options:
clone your list, iterate through the clone and remove the 'specific' String from the original list
create a list for delete candidates, add all 'specific' Strings to that list and, after iterating through the original list, iterate through the wastebin and remove everything you've collected here from the original list.
Option 1 is the easist, the clone can be made like:
List<String> clone = new ArrayList<String>(arr);

You seem to misunderstand how objects/references work in Java, which is pretty fundamental to using the language effectively. However, this code here should do what you want (apologies for the lack of explanation):
ArrayList<String> arr = new ArrayList<String>();
//... fill with some values (doesn't really matter)
for(int i = 0; i < arr.size(); i++)
{
arr.set(i, " some other value "); // change the contents of the array
}
for(String t : arr)
{
System.out.println(t);
}

I believe, this is not related to immutable or mutable.
t = " some other value "; //hoping this would change the actual array
t does not hold the reference to actual object. Java copies the value from arraylist and puts that value into t so array list value does not get affect.
HTH

This has been answered well. Still here is my suggestion. The var t inside loop is only visible there. It will not be seen outside the loop. You could do t.set() if it was not String.

Use a StringBuffer rather than plain strings. This way the string within is mutable.

Strings are immutable. If you had a mutable type like StringBuilder/Buffer, you could change the string in your iteration. You do have references, remember.