Develop Reference

Java Multithreading Implementation for generating unique codes

My question is how I would implement multithreading to this task correctly.
I have a program that takes quite a long time to finish executing. About an hour and a half. I need to generate about 10,000 random and unique number codes. The code below is how I first implemented it and have it right now.
import java.util.Random;
import java.util.ArrayList;
public class Main
{
public static void main(String[] args) {
Random random = new Random();
// This holds all the codes
ArrayList<String> database = new ArrayList<>();
int counter = 0;
while(counter < 10000){
// Generate a 10 digit long code and append to sb
StringBuilder sb = new StringBuilder();
for(int i = 0; i < 10; i++){
sb.append(random.nextInt(10));
}
String code = String.valueOf(sb);
sb.setLength(0);
// Check if this code already exists in the database
// If not, then add the code and update counter
if(!database.contains(code)){
database.add(code);
counter++;
}
}
System.out.println("Done");
}
}
This of course is incredibly inefficient. So my question is: Is there is a way to implement multithreading that can work on this single piece of code? Best way I can word it is to give two cores/ threads the same code but have them both check the a single ArrayList? Both cores/ threads will generate codes but check to make sure the code it just made doesn't already exist either from the other core/ thread or from itself. I drew a rough diagram below. Any insight, advice, or pointers is greatly appreciated.

Using a more appropriate data structure and a more appropriate representation of the data, this should be a lot faster and easier to read, too:
Set<Long> database = new HashSet<>(10000);
while(database.size() < 10000){
database.add(ThreadLocalRandom.current().nextLong(10_000_000_000L);
}

Start with more obvious optimizations:
Do not use ArrayList, use HashSet. ArrayList contains() time complexity is O(n), while HashSet is O(1). Read this question about Big O summary for java collections framework. Read about Big O notation.
Initialize your collection with appropriate initial capacity. For your case that would be:
new HashSet<>(10000);
Like this underlying arrays won't be copied to increase their capacity. I would suggest to look/debug implementations of java collections to better understand how they work under the hood. Even try to implement them on your own.
Before you delve into complex multithreading optimizations, fix the simple problems - like bad collection choices.
Edit: As per suggestion from #Thomas in comments, you can directly generate a number(long) in the range you need - 0 to 9_999_999_999. You can see in this question how to do it. Stringify the resulting number and if length is less than 10, pad with leading zeroes.

Example:
(use ConcurrentHashMap, use threads, use random.nextLong())
public class Main {
static Map<String,Object> hashMapCache = new ConcurrentHashMap<String,Object>();
public static void main(String[] args) {
Random random = new Random();
// This holds all the codes
ArrayList<String> database = new ArrayList<>();
int counter = 0;
int NumOfThreads = 20;
int total = 10000;
int numberOfCreationsForThread = total/NumOfThreads;
int leftOver = total%NumOfThreads;
List<Thread> threadList = new ArrayList<>();
for(int i=0;i<NumOfThreads;i++){
if(i==0){
threadList.add(new Thread(new OneThread(numberOfCreationsForThread+leftOver,hashMapCache)));
}else {
threadList.add(new Thread(new OneThread(numberOfCreationsForThread,hashMapCache)));
}
}
for(int i=0;i<NumOfThreads;i++){
threadList.get(i).start();;
}
for(int i=0;i<NumOfThreads;i++){
try {
threadList.get(i).join();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
for(String key : hashMapCache.keySet()){
database.add(key);
}
System.out.println("Done");
}}
OneThread:
public class OneThread implements Runnable{
int numberOfCreations;
Map<String,Object> hashMapCache;
public OneThread(int numberOfCreations,Map<String,Object> hashMapCache){
this.numberOfCreations = numberOfCreations;
this.hashMapCache = hashMapCache;
}
#Override
public void run() {
int counter = 0;
Random random = new Random();
System.out.println("thread "+ Thread.currentThread().getId() + " Start with " +numberOfCreations);
while(counter < numberOfCreations){
String code = generateRandom(random);
while (code.length()!=10){
code = generateRandom(random);
}
// Check if this code already exists in the database
// If not, then add the code and update counter
if(hashMapCache.get(code)==null){
hashMapCache.put(code,new Object());
counter++;
}
}
System.out.println("thread "+ Thread.currentThread().getId() + " end with " +numberOfCreations);
}
private static String generateRandom(Random random){
return String.valueOf(digits(random.nextLong(),10));
}
/** Returns val represented by the specified number of hex digits. */
private static String digits(long val, int digits) {
val = val > 0 ? val : val*-1;
return Long.toString(val).substring(0,digits);
}
}

Compartmentalizing loops over a large iteration

The Goal of my question is to enhance the performance of my algorithm by splitting the range of my loop iterations over a large array list.
For example: I have an Array list with a size of about 10 billion entries of long values, the goal I am trying to achieve is to start the loop from 0 to 100 million entries, output the result for the 100 million entries of whatever calculations inside the loop; then begin and 100 million to 200 million doing the previous and outputting the result, then 300-400million,400-500million and so on and so forth.
after I get all the 100 billion/100 million results, then I can sum them up outside of the loop collecting the results from the loop outputs parallel.
I have tried to use a range that might be able to achieve something similar by trying to use a dynamic range shift method but I cant seem to have the logic fully implemented like I would like to.
public static void tt4() {
long essir2 = 0;
long essir3 = 0;
List cc = new ArrayList<>();
List<Long> range = new ArrayList<>();
// break point is a method that returns list values, it was converted to
// string because of some concatenations and would be converted back to long here
for (String ari1 : Breakpoint()) {
cc.add(Long.valueOf(ari1));
}
// the size of the List is huge about 1 trillion entries at the minimum
long hy = cc.size() - 1;
for (long k = 0; k < hy; k++) {
long t1 = (long) cc.get((int) k);
long t2 = (long) cc.get((int) (k + 1));
// My main question: I am trying to iterate the entire list in a dynamic way
// which would exclude repeated endpoints on each iteration.
range = LongStream.rangeClosed(t1 + 1, t2)
.boxed()
.collect(Collectors.toList());
for (long i : range) {
// Hard is another method call on the iteration
// complexcalc is a method as well
essir2 = complexcalc((int) i, (int) Hard(i));
essir3 += essir2;
}
}
System.out.println("\n" + essir3);
}
I don't have any errors, I am just looking for a way to enhance performance and time. I can do a million entries in under a second directly, but when I put the size I require it runs forever. The size I'm giving are abstracts to illustrate size magnitudes, I don't want opinions like a 100 billion is not much, if I can do a million under a second, I'm talking massively huge numbers I need to iterate over doing complex tasks and calls, I just need help with the logic I'm trying to achieve if I can.

One thing I would suggest right off the bat would be to store your Breakpoint return value inside a simple array rather then using a List. This should improve your execution time significantly:
List<Long> cc = new ArrayList<>();
for (String ari1 : Breakpoint()) {
cc.add(Long.valueOf(ari1));
}
Long[] ccArray = cc.toArray(new Long[0]);
I believe what you're looking for is to split your tasks across multiple threads. You can do this with ExecutorService "which simplifies the execution of tasks in asynchronous mode".
Note that I am not overly familiar with this whole concept but have experimented with it a bit recently and give you a quick draft of how you could implement this.
I welcome those more experienced with multi-threading to either correct this post or provide additional information in the comments to help improve this answer.
Runnable Task class
public class CompartmentalizationTask implements Runnable {
private final ArrayList<Long> cc;
private final long index;
public CompartmentalizationTask(ArrayList<Long> list, long index) {
this.cc = list;
this.index = index;
}
#Override
public void run() {
Main.compartmentalize(cc, index);
}
}
Main class
private static ExecutorService exeService = Executors.newCachedThreadPool();
private static List<Future> futureTasks = new ArrayList<>();
public static void tt4() throws ExecutionException, InterruptedException
{
long essir2 = 0;
long essir3 = 0;
ArrayList<Long> cc = new ArrayList<>();
List<Long> range = new ArrayList<>();
// break point is a method that returns list values, it was converted to
// string because of some concatenations and would be converted back to long here
for (String ari1 : Breakpoint()) {
cc.add(Long.valueOf(ari1));
}
// the size of the List is huge about 1 trillion entries at the minimum
long hy = cc.size() - 1;
for (long k = 0; k < hy; k++) {
futureTasks.add(Main.exeService.submit(new CompartmentalizationTask(cc, k)));
}
for (int i = 0; i < futureTasks.size(); i++) {
futureTasks.get(i).get();
}
exeService.shutdown();
}
public static void compartmentalize(ArrayList<Long> cc, long index)
{
long t1 = (long) cc.get((int) index);
long t2 = (long) cc.get((int) (index + 1));
// My main question: I am trying to iterate the entire list in a dynamic way
// which would exclude repeated endpoints on each iteration.
range = LongStream.rangeClosed(t1 + 1, t2)
.boxed()
.collect(Collectors.toList());
for (long i : range) {
// Hard is another method call on the iteration
// complexcalc is a method as well
essir2 = complexcalc((int) i, (int) Hard(i));
essir3 += essir2;
}
}

MultiThreaded Fibonacci

public class Fibonacci {
public static class PFibo extends Thread {
private int x;
public long answer;
public PFibo(int x) {
this.x = x;
}
public void run() {
if (x <= 2)
answer = 1;
else {
try {
PFibo t = new PFibo(x - 1);
t.start();
long y = RFibo(x - 2);
t.join();
answer = t.answer + y;
} catch (InterruptedException ex) {
}
}
}
}
public static long RFibo(int no) {
if (no == 1 || no == 2) {
return 1;
}
return RFibo(no - 1) + RFibo(no - 2);
}
public static void main(String[] args) throws Exception {
try {
long start = System.currentTimeMillis();
PFibo f = new PFibo(30);
f.start();
f.join();
long end = System.currentTimeMillis();
System.out.println("Parallel-Fibonacci:" + f.answer + "\tTime:" + (end - start));
start = System.currentTimeMillis();
long result = RFibo(30);
end = System.currentTimeMillis();
System.out.println("Normal-Fibonacci:" + result + "\tTime:" + (end - start));
} catch (Exception e) {
}
}
}
I am currently reading 'Multithreaded Algorithms' from 'Introduction to Algorithms'. I tried implementing a basic multithreaded program for calculating the n-th fibonacci number. For n=30 the program gave the following output :
Parallel-Fibonacci:832040 Time:10
Normal-Fibonacci:832040 Time:3
Why is the parallel version slower that the non-parallel version. Has thread-switching or 'too-many-number-of-threads' slowed it down ?
What approach has to followed to speed-up the parallel version ?

Has thread-switching or 'too-many-number-of-threads' slowed it down ?
Yes of course. In a number of ways-
As already been pointed out in comments
You are creating a new thread per call i.e.
PFibo t = new PFibo(x - 1);
t.start();
Effectively you have created around 28 threads for PFibo(30) which means one context switch for evaluating each term
Secondly, as the evaluation of PFibo(x) depends on PFibo(x - 1) which required you to put a call to join() method there, each time you are creating/starting a new thread i.e. eventually it has become serial.
So the final cost = cost of actual serial method RFibo(n) + around n context switches + sync time (time taken by join())
What approach has to followed to speed-up the parallel version ?
Well I would say, don't do it. Fibonacci series' solution pattern does not suit to be optimized by parallelism. Just rely on serial version(you can implement an iterative version for more efficiency).

your input is too small to gain any benefit from parallelism. Nevertheless, it makes sense to parallelize this version of the Fibonacci algorithm. Your algorithm is exponential. By creating new threads, you split exponential work among the threads. Notice, however, that there is, indeed, a linear-time algorithm to compute the Fibonacci numbers, which, as people here have already said, it is better to run sequentially. So, using larger inputs with your implementation, I get, on an Intel 2.3GHz:
$ java Fib 30
Parallel-Fib:832040 Time:0.026805616
Sequential-Fib:832040 Time:0.002786453
$ java Fib 33
Parallel-Fib:3524578 Time:0.012451416
Sequential-Fib:3524578 Time:0.012420652
$ java Fib 36
Parallel-Fib:14930352 Time:0.035997556
Sequential-Fib:14930352 Time:0.056066557
$ java Fib 44
Parallel-Fib:701408733 Time:2.037292083
Sequential-Fib:701408733 Time:3.050315551

Threadprogramming, Heap OutOfMemory

i've been programming for the first time with Threads in Java, so here's a pretty much beginners question in terms of Threads.
My Code:
import java.util.ArrayList;
import java.util.List;
public class CollatzRunner implements Runnable {
private int lastNumber = 0;
private int highestCounter = 0;
private int highestValue = 0;
public void run() {
while(this.lastNumber < 1000000) {
this.lastNumber++;
Collatz c = new Collatz(lastNumber);
List<Integer> values = new ArrayList<Integer>();
while(c.hasNext()) {
values.add(c.next());
}
if(this.highestCounter < values.size()) {
this.highestCounter = values.size();
this.highestValue = values.get(0);
}
//System.out.println(Thread.currentThread().getName() + ": " + this.lastNumber);
System.out.println(Thread.currentThread().getName());
}
}
}
and:
public class CollatzSimulator {
public static void main(String[] args) {
CollatzRunner runner = new CollatzRunner();
Thread t1 = new Thread(runner, "Thread-1");
Thread t2 = new Thread(runner, "Thread-2");
Thread t3 = new Thread(runner, "Thread-3");
Thread t4 = new Thread(runner, "Thread-4");
//Thread t5 = new Thread(runner, "Thread-5");
//Thread t6 = new Thread(runner, "Thread-6");
//Thread t7 = new Thread(runner, "Thread-7");
//Thread t8 = new Thread(runner, "Thread-8");
t1.start();
t2.start();
t3.start();
t4.start();
//t5.start();
//t6.start();
//t7.start();
//t8.start();
System.out.println("bla");
}
}
When running this code if almost immediately get OutOfMemoryError: Heap space. So i suspect i got a pretty major memory leak here. Problem is that i have no experience in this field, therefore i ask on this site.
What i've tried so far:
- Did a heap space dump file (800MB in 5 sec generated)
- tried to set the Collatz instance to null after using it to kill the reference in hope that the garbage collector will free the heap space.
My program is just a little class Collatz, that generates the collatzsequence for a given number and i want to use threads to generate all collatzsequenzes for 0 < n < 1000000.
Thanks for any help!

I am pretty sure the problem can be found here:
List<Integer> values = new ArrayList<Integer>();
while(c.hasNext()) {
values.add(c.next());
}
From my understanding a Collatz-sequence can get very long. What c.next() does is to compute the next member of the sequence, this uses nearly no space at all. But concatenating all sequence members into a list takes a huge amount of space. And the more threads you create, the sooner you'll hit that barrier.
If you don't have the option to allocate a huge amount of space for the heap, I'm pretty sure the only option would be to write a part of the values list out to a file as soon as it exceeds a specific length (you'll have to output it at some point anyway). However, with all that I/O the usage of multithreading will be pretty pointless

In order to determine the length of a Collatz sequence for a given starting value: do NOT store the sequence, simply count the number of values Collatz returns:
Collatz c = new Collatz(lastNumber);
int length = 0;
while(c.hasNext()) {
length++;
}
if(this.highestCounter < length) {
this.highestCounter = length;
this.highestValue = lastNumber;
}
Later
After reading about the "thread" challenge: If you have a computer with more than one core, it makes sense to try and search for the longest Collatz sequence up to 1M by running several threads (number of cores) in parallel, but not using the very same code. Add arguments startValue and increment to Collatz runner and add this for each iteration, so that the first thread of, say, four, computes 1, 5, 9,... the second 2, 6, 10,... the third 3, 7, 11,... and the fourth 4, 8, 12,....

Unique random number for a particular timestamp

I am kind of learning concepts of Random number generation & Multithreading in java.
The idea is to not generating a repeated random number of range 1000 in a particular millisecond (Considering, not more than 50 data, in a multithreaded way will be processed in a millisecond). So that list of generated random number at the specific time is unique. Can you give me any idea as i am ending up generating couple of repeated random numbers (also, there is a considerable probability) in a particular milli second.
I have tried the following things where i failed.
Random random = new Random(System.nanoTime());
double randomNum = random.nextInt(999);
//
int min=1; int max=999;
double randomId = (int)Math.abs(math.Random()* (max - min + 1) + min);
//
Random random = new Random(System.nanoTime()); // also tried new Random();
double randomId = (int)Math.abs(random.nextDouble()* (max - min + 1) + min);
As I am appending the timestamp that is being generated, in a multithreaded environment i see the same ids (around 8-10) that is being generated (2-4 times) for 5000+ unique data.

First, you should use new Random(), since it looks like this (details depend on Java version):
public Random() { this(++seedUniquifier + System.nanoTime()); }
private static volatile long seedUniquifier = 8682522807148012L;
I.e. it already makes use of nanoTime() and makes sure different threads with the same nanoTime() result get different seeds, which new Random(System.nanoTime()) doesn't.
(EDIT: Pyranja pointed out this is a bug in Java 6, but it's fixed in Java 7:
public Random() {
this(seedUniquifier() ^ System.nanoTime());
}
private static long seedUniquifier() {
// L'Ecuyer, "Tables of Linear Congruential Generators of
// Different Sizes and Good Lattice Structure", 1999
for (;;) {
long current = seedUniquifier.get();
long next = current * 181783497276652981L;
if (seedUniquifier.compareAndSet(current, next))
return next;
}
}
private static final AtomicLong seedUniquifier
= new AtomicLong(8682522807148012L);
)
Second, if you generate 50 random numbers from 1 to 1000, the probability some numbers will be the same is quite high thanks to the birthday paradox.
Third, if you just want unique ids, you could just use AtomicInteger counter instead of random numbers. Or if you want a random part to start with, append a counter as well to guarantee uniqueness.

This class will allow you to get nonrepeating values from a certain range until the whole range has been used. Once the range is used, it will be reinitialized.
Class comes along with a simple test.
If you want to make the class thread safe, just add synchronized to nextInt() declaration.
Then you can use the singleton pattern or just a static variable to access the generator from multiple threads. That way all your threads will use the same object and the same unique id pool.
public class NotRepeatingRandom {
int size;
int index;
List<Integer> vals;
Random gen = new Random();
public NotRepeatingRandom(int rangeMax) {
size = rangeMax;
index = rangeMax; // to force initial shuffle
vals = new ArrayList<Integer>(size);
fillBaseList();
}
private void fillBaseList() {
for (int a=0; a<size; a++) {
vals.add(a);
}
}
public int nextInt() {
if (index == vals.size()) {
Collections.shuffle(vals);
index = 0;
}
int val = vals.get(index);
index++;
return val;
}
public static void main(String[] args) {
NotRepeatingRandom gen = new NotRepeatingRandom(10);
for (int a=0; a<30; a++) {
System.out.println(gen.nextInt());
}
}
}

If I understand your question correctly, multiple threads are creating their own instances of Random class at the same time and all threads generate the same random number?
Same number is generated, because all random instances where created at the same time, i.e. with the same seed.
To fix this, create only one instance of Random class, which is shared by all threads so that all your threads call nextDouble() on the same instance. Random.nextDouble() class is thread safe and will implicitly update its seed with every call.
//create only one Random instance, seed is based on current time
public static final Random generator= new Random();
Now all threads should use the same instance:
double random=generator.nextDouble()