Develop Reference

How to process contents from large text file using multiple threads?

I have to read a huge file contains text, around 3GB (and 40 Million lines). Just reading it happens really fast:
try (BufferedReader br = new BufferedReader(new FileReader("file.txt"))) {
while ((line = br.readLine()) != null) {
//Nothing here
}
}
With each read line from above code i do some parsing on the string and process it further.(a huge task). I try to do that multiple threads.
A) I have tried BlockingQueue like this
try (BufferedReader br = new BufferedReader(new FileReader("file.txt"))) {
String line;
BlockingQueue<String> queue = new ArrayBlockingQueue<>(100);
int numThreads = 5;
Consumer[] consumer = new Consumer[numThreads];
for (int i = 0; i < consumer.length; i++) {
consumer[i] = new Consumer(queue);
consumer[i].start();
}
while ((line = br.readLine()) != null) {
queue.put(line);
}
queue.put("exit");
} catch (FileNotFoundException ex) {
Logger.getLogger(ReadFileTest.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException | InterruptedException ex) {
Logger.getLogger(ReadFileTest.class.getName()).log(Level.SEVERE, null, ex);
}
class Consumer extends Thread {
private final BlockingQueue<String> queue;
Consumer(BlockingQueue q) {
queue = q;
}
public void run() {
while (true) {
try {
String result = queue.take();
if (result.equals("exit")) {
queue.put("exit");
break;
}
System.out.println(result);
} catch (InterruptedException ex) {
Logger.getLogger(ReadFileTest.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
This approach took more time than normal single threaded processing.
I am not sure why - what am I doing wrong?
B) I have tried ExecutorService:
try (BufferedReader br = new BufferedReader(new FileReader("file.txt"))) {
String line;
ExecutorService pool = Executors.newFixedThreadPool(10);
while ((line = br.readLine()) != null) {
pool.execute(getRunnable(line));
}
pool.shutdown();
} catch (FileNotFoundException ex) {
Logger.getLogger(ReadFileTest.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(ReadFileTest.class.getName()).log(Level.SEVERE, null, ex);
}
private static Runnable getRunnable(String run){
Runnable task = () -> {
System.out.println(run);
};
return task;
}
This approach also takes more time than printing directly inside while loop. What am I doing wrong?
What is the correct way to do it?
How can I efficiently process the read line with multiple threads?

Answering one part here - why is the BlockingQueue option slower.
It is important to understand that threads don't come for "free". There is always certain overhead required to get them up and "manage" them.
And of course, when you are actually using more threads than your hardware can support "natively" then context switching is added to the bill.
Beyond that, also the BlockingQueue doesn't come free either. You see, in order to preserve order, that ArrayBlockingQueue probably has to synchronize somewhere. Worst case, that means locking and waiting. Yes, the JVM and JIT are usually pretty good about such things; but again, a certain "percentage" gets added to the bill.
But just for the record, that shouldn't matter. From the javadoc:
This class supports an optional fairness policy for ordering waiting producer and consumer threads. By default, this ordering is not guaranteed. However, a queue constructed with fairness set to true grants threads access in FIFO order. Fairness generally decreases throughput but reduces variability and avoids starvation.
As you are not setting "fairness"
BlockingQueue queue = new ArrayBlockingQueue<>(100);
that shouldn't affect you. On the other hand: I am pretty sure you expected those lines to be processed in order, so you would actually want to go for
BlockingQueue<String> queue = new ArrayBlockingQueue<>(100, true);
and thereby further slowing down the whole thing.
Finally: I agree with the comments given so far. Benchmarking such things is a complex undertaking; and many aspects influence the results. The most important question is definitely: where is your bottle neck?! Is it IO performance (then more threads don't help much!) - or is it really overall processing time (and then the "correct" number of threads for processing should definitely speed up things).
And regarding "how to do this in the correct way" - I suggest to check out this question on softwareengineering.SE.

How to process contents from large text file using multiple threads?
If your computer has enough RAM, I would do the following:
read the entire file into a variable (an ArrayList for example) - using only one thread to read the whole file.
then launch one ExecutorService (with a thread pool that uses no more than the maximum number of cores that your computer can run simultaneously)
int cores = Runtime.getRuntime().availableProcessors();
ExecutorService executorService = Executors.newFixedThreadPool(cores);
finally, divide the lines read, among a limited number of callables/runnables and submit those callables/runnables to your ExecutorService (so all of them can execute simultaneously in your ExecutorService).
unless your processing of lines uses I/O, I assume that you will reach near 100% CPU utilization, and none of your threads will be in waiting state.
do you want even faster processing?
scale vertically is the easiest option: buy even more RAM, better CPU (with more cores), use a Solid State Drive

May be all thread accessing same shared resource concurrently so result more contentious.
One thing you can try reader thread put all line in single key do submit in partition way so it will less contentious.
public void execute(Runnable command) {
final int key= command.getKey();
//Some code to check if it is runing
final int index = key != Integer.MIN_VALUE ? Math.abs(key) % size : 0;
workers[index].execute(command);
}
Create worker with queue so that if you want some task required sequentially then run.
private final AtomicBoolean scheduled = new AtomicBoolean(false);
private final BlockingQueue<Runnable> workQueue = new LinkedBlockingQueue<Runnable>(maximumQueueSize);
public void execute(Runnable command) {
long timeout = 0;
TimeUnit timeUnit = TimeUnit.SECONDS;
if (command instanceof TimeoutRunnable) {
TimeoutRunnable timeoutRunnable = ((TimeoutRunnable) command);
timeout = timeoutRunnable.getTimeout();
timeUnit = timeoutRunnable.getTimeUnit();
}
boolean offered;
try {
if (timeout == 0) {
offered = workQueue.offer(command);
} else {
offered = workQueue.offer(command, timeout, timeUnit);
}
} catch (InterruptedException e) {
throw new RejectedExecutionException("Thread is interrupted while offering work");
}
if (!offered) {
throw new RejectedExecutionException("Worker queue is full!");
}
schedule();
}
private void schedule() {
//if it is already scheduled, we don't need to schedule it again.
if (scheduled.get()) {
return;
}
if (!workQueue.isEmpty() && scheduled.compareAndSet(false, true)) {
try {
executor.execute(this);
} catch (RejectedExecutionException e) {
scheduled.set(false);
throw e;
}
}
}
public void run() {
try {
Runnable r;
do {
r = workQueue.poll();
if (r != null) {
r.run();
}
}
while (r != null);
} finally {
scheduled.set(false);
schedule();
}
}
As suggested above there is no fixed rule for thread pool size .But there is some suggestion or best practice available can be used depending upon your use case.
CPU Bound Tasks
For CPU bound tasks, Goetz (2002, 2006) recommends
threads = number of CPUs + 1
IO Bound Tasks
Working out the optimal number for IO bound tasks is less obvious. During an IO bound task, a CPU will be left idle (waiting or blocking). This idle time can be better used in initiating another IO bound request.

Java Problems of Multithreading

I have problem with synchronized reading of my file. The case is simple: to handle data in the file. My problem is synchronization. I try to make it on object reader of class BufferedReader, but I have two problems. If I initialise reader in "try with resourses" of my method read() I will get to different objects of BufReader for the same object when I use it in different theads. The second one I initialise it in class and get the same object for threads, but a great problem with exeptions and unpredictable behavior. I will write code for the second situation. Can you advice me how to solve. I am studying now, that's why I want advices.
class FileReaderClass {
private File file = new File("src\\exer1\\Transfers.txt");
private BufferedReader reader = null;
private FileReader fr = null;
StringBuilder sb = new StringBuilder();
void read() throws IOException {
try {
fr = new FileReader(file);
reader = new BufferedReader(fr);
String buftext;
while ((buftext = reader.readLine()) != null){
synchronized (reader) {
System.out.println(Thread.currentThread().getName());//for testing
sb.append(buftext).append("\n");
//System.out.println(buftext);
}
}
}
catch (IOException e) {
e.printStackTrace();
}
finally {
if (reader != null)
reader.close();
if (fr != null)
fr.close();
}
}
}
class Reader1 implements Runnable {
private FileReaderClass frc;
public Reader1(FileReaderClass frc) {
this.frc = frc;
}
#Override
public void run() {
try {
frc.read();
} catch (IOException e) {
e.printStackTrace();
}
}
}
public class Ex1 {
public static void main(String[] args) throws InterruptedException {
FileReaderClass frc = new FileReaderClass();
Thread t1 = new Thread(new Reader1(frc));
Thread t2 = new Thread(new Reader2(frc));
t1.start();
t2.start();
t1.join();
t2.join();
System.out.println("---------------------");
System.out.println(frc.sb);
}
}

You are sharing the same FileReaderClass instance (that is frc) in two threads. Your FileReaderClass is not thread safe, that causes the problems. There are two alternatives here:
Create another instance of FileReaderClass, so that both threads will use different instances.
Mark the method read as snychronized.
You are synchronizing on the wrong level. Your sync guard is useless, as there are actually two readers. That's because each time read is called, a reader will be created, which will be a separate instance. We can't even know for sure, which reader instance was the subject of syncing.

If you want to parallelize reading of your file you should use the same reader but use proper synchronization for each read by any thread, so that it stays consistent, the easiest way is to make the whole read method synchronized and instantiate your readers in your class constructor.
As you may want to read and write to your string builder in the correct order, the thread that reads a line must only release the lock after it writes to StringBuilder, making your whole method to be pretty much guarded by a lock. It will work but you won't gain any performance, and for small files it may cause too much overhead, but as an exercise it may suffice.
Your code as it is has the following problems
You are always creating new readers, thus every thread calling this method will start from the beginning.
As previously stated by another answear you are synchronizing at the wrong level, any thread may read and the thread scheduler can suspend it right after, causing your writes to StringBuilder object to be out of order (i think it is not intended).

Best queue for sliding windows in Java

I'm writing sliding windows protocol:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class ABC {
static boolean status_1 = true;
public static void main(String[] args) {
BlockingQueue<String> block1 = new LinkedBlockingQueue<String>(7); // size 7
Thread a1 = new Thread(new receive(block1));
Thread a2 = new Thread(new send(block1));
a2.start();
a1.start();
}
}
class receive implements Runnable {
BlockingQueue<String> block;
public receive(BlockingQueue<String> block) {
this.block = block;
}
#Override
public void run() {
while (true) {
try {
System.out.println("out: " + block.size() + " " + block.take());
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
class send implements Runnable {
BlockingQueue<String> block;
public send(BlockingQueue<String> block) {
this.block = block;
}
#Override
public void run() {
InputStreamReader in = new InputStreamReader(System.in);
BufferedReader bufferedReader = new BufferedReader(in);
int i = 0;
String e;
while (true) {
try {
e = "" + i++;
System.out.println(e);
block.put(e);
if (i == 1000) {
break; //Test 1000 number
}
} catch (InterruptedException f) {
// TODO Auto-generated catch block
f.printStackTrace();
}
}
}
In my example I used BlockingQueue to do the task but it delayed alot. The receive thread keep full size.
Is there any queue in Java could do the task with better performance in real time UDP?

The thing is, that you have no guarnatees that "sending" thread will work with the "same speed" as the "receiving" thread. Parralel thread execution is non-deterministic. You assume otherwise.
You have introduced logical synchonization between threads by assumption that both threads will work with the same speed - 1 item put, 1 item taken, 1 item put, 1 item taken and so on.
So by accident this works for you in case of put but not with offer it is because according to docs https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/BlockingQueue.html put will block until queue will have space to accept new element - that synchronization is missing when u use offer, resuylting in dropped packets.
So basicly what is happening here, is that sending thread occupy more CPU time thus production more data than receiver can consume resulting in dropped data.
It is not about queue performance at all.
Basicly your code is not the great example as there is no natural network latency etc. This code might work to some point , if we would introduce network latency. You can emulate that by adding Thread.sleep(ms) in producer thread between put call.
As a side not, stick to Java's naming convetion

I think your problem is the messuring method. You use block.size() to determine the queue fill grade. The size() method is a relatively long running operation on the queue, which leads to the send thread running away.
If you remove the size output you will see a quite fairly distributed output between send and receive.
By the way, using System.out also disturbs your experiment because of the synchronization to the console output stream. A better approach would be using independent output streams with some timing information.

Reading Really big Files With Java

I am reading a 77MB file inside a Servlet, in future this will be 150GB. This file is not written using any kind of nio package thing, it is just written using BufferedWriter.
Now this is what I need to do.
Read the file line by line. Each line is a "hash code" of a text. Separate it into pieces of 3 chars (3 chars represents 1 word) It could be long, it could be short, I don't know.
After reading the line, convert it into real words. We have a Map of words and Hashes so we can find the words.
Up to now, I used BufferedReader to read the file. It is slow and not good for Huge files like 150GB. It took hours to complete the entire process even for this 77MB file. Because we can't keep the user waiting for hours, it should be within seconds. So, we decided to load the file into the memory. First we thought about loadng every single line into a LinkedList, so the memory coulkd save it. But you know, memory cannot save such a big amount. After a Big Search, I decided Mapping Files to the memory would be the answer. Memory is super faster than the Disk, so we could read the files super fast too.
Code:
public class MapRead {
public MapRead()
{
try {
File file = new File("E:/Amazon HashFile/Hash.txt");
FileChannel c = new RandomAccessFile(file,"r").getChannel();
MappedByteBuffer buffer = c.map(FileChannel.MapMode.READ_ONLY, 0,c.size()).load();
for(int i=0;i<buffer.limit();i++)
{
System.out.println((char)buffer.get());
}
System.out.println(buffer.isLoaded());
System.out.println(buffer.capacity());
} catch (IOException ex) {
Logger.getLogger(MapRead.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
But I could not see any "super fast" thing. And I need line by line. I have few questions to ask.
You read my description and you know what I need to do. I have done the first step for that, so is that correct?
The way I Map is correct? I mean, this is no difference than reading it in normal way. So does this hold the "entire" file in memory first? (lets say using a technique called Mapping) Then we have to write another code to access that memory?
How to read line by line, in super "fast" ? (If I have to load/map the entire file to the memory first for hours, then access it in super speed in seconds, I am totally fine with it too)
Reading files in Servlets is good ? (Because it is being accessed by number of people, and only one IO stream will be opened at once. In this case this servlet will be accessed by thousands at once)
Update
This is how my code look when I updated it with SO user Luiggi Mendoza's answer.
public class BigFileProcessor implements Runnable {
private final BlockingQueue<String> linesToProcess;
public BigFileProcessor (BlockingQueue<String> linesToProcess) {
this.linesToProcess = linesToProcess;
}
#Override
public void run() {
String line = "";
try {
while ( (line = linesToProcess.take()) != null) {
System.out.println(line); //This is not happening
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public class BigFileReader implements Runnable {
private final String fileName;
int a = 0;
private final BlockingQueue<String> linesRead;
public BigFileReader(String fileName, BlockingQueue<String> linesRead) {
this.fileName = fileName;
this.linesRead = linesRead;
}
#Override
public void run() {
try {
//Scanner do not work. I had to use BufferedReader
BufferedReader br = new BufferedReader(new FileReader(new File("E:/Amazon HashFile/Hash.txt")));
String str = "";
while((str=br.readLine())!=null)
{
// System.out.println(a);
a++;
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
public class BigFileWholeProcessor {
private static final int NUMBER_OF_THREADS = 2;
public void processFile(String fileName) {
BlockingQueue<String> fileContent = new LinkedBlockingQueue<String>();
BigFileReader bigFileReader = new BigFileReader(fileName, fileContent);
BigFileProcessor bigFileProcessor = new BigFileProcessor(fileContent);
ExecutorService es = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
es.execute(bigFileReader);
es.execute(bigFileProcessor);
es.shutdown();
}
}
public class Main {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
// TODO code application logic here
BigFileWholeProcessor b = new BigFileWholeProcessor ();
b.processFile("E:/Amazon HashFile/Hash.txt");
}
}
I am trying to print the file in BigFileProcessor. What I understood is this;
User enter file name
That file get read by BigFileReader, line by line
After each line, the BigFileProcessor get called. Which means, assume BigFileReader read the first line. Now the BigFileProcessor is called. Now once the BigFileProcessor completes the processing for that line, now the BigFileReader reads the line 2. Then again the BigFileProcessor get called for that line, and so on.
May be my understanding about this code is incorrect. How should I process the line anyway?

I would suggest using multi thread here:
One thread will take care to read every line of the file and insert it into a BlockingQueue in order to be processed.
Another thread(s) will take the elements from this queue and process them.
To implement this multi thread work, it would be better using ExecutorService interface and passing Runnable instances, each should implement each task. Remember to have only a single task to read the file.
You could also manage a way to stop reading if the queue has a specific size e.g. if the queue has 10000 elements then wait until its size is down to 8000, then continue reading and filling the queue.
Reading files in Servlets is good ?
I would recommend never do heavy work in servlet. Instead, fire an asynchronous task e.g. via JMS call, then in this external agent you will process your file.
A brief sample of the above explanation to solve the problem:
public class BigFileReader implements Runnable {
private final String fileName;
private final BlockingQueue<String> linesRead;
public BigFileReader(String fileName, BlockingQueue<String> linesRead) {
this.fileName = fileName;
this.linesRead = linesRead;
}
#Override
public void run() {
//since it is a sample, I avoid the manage of how many lines you have read
//and that stuff, but it should not be complicated to accomplish
Scanner scanner = new Scanner(new File(fileName));
while (scanner.hasNext()) {
try {
linesRead.put(scanner.nextLine());
} catch (InterruptedException ie) {
//handle the exception...
ie.printStackTrace();
}
}
scanner.close();
}
}
public class BigFileProcessor implements Runnable {
private final BlockingQueue<String> linesToProcess;
public BigFileProcessor (BlockingQueue<String> linesToProcess) {
this.linesToProcess = linesToProcess;
}
#Override
public void run() {
String line = "";
try {
while ( (line = linesToProcess.take()) != null) {
//do what you want/need to process this line...
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public class BigFileWholeProcessor {
private static final int NUMBER_OF_THREADS = 2;
public void processFile(String fileName) {
BlockingQueue<String> fileContent = new LinkedBlockingQueue<String>();
BigFileReader bigFileReader = new BigFileReader(fileName, fileContent);
BigFileProcessor bigFileProcessor = new BigFileProcessor(fileContent);
ExecutorService es = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
es.execute(bigFileReader);
es.execute(bigFileProcessor);
es.shutdown();
}
}

NIO won't help you here. BufferedReader is not slow. If you're I/O bound, you're I/O bound -- get faster I/O.
Mapping the file in to memory can help, but only if you're actually using the memory in place, rather than just copying all of the data out of the big byte array that you get back. The primary advantage of mapping the file is that it keeps the data out of the java heap, and away from the garbage collector.
Your best performance will come from working on the data in place, and not copying it in to the heap if you can.
Some of your performance may be impacted by the object creation. For example, if you were trying to load your data in to the LinkedList, you're creating (likely) millions of nodes for the List itself, plus the object surrounding your data (even if they're just strings).
Creating Strings based on your memory mapped array can be quite efficient, as the String will simply wrap the data, not copy it. But you'll have to be UTF aware if you're working with something other than ASCII (as bytes are not characters in Java).
Also if you're loading in large things, with lots of objects, ensure that you have free space in your heap for them. And by free space, I mean actual room. You can have a 500MB heap, as specified by -Xmx, but the ACTUAL heap will not be that large initially, it will grow to that limit.
Assuming you have sufficient memory in the first place, you can do this via -Xms, which will pre-allocate the heap to a desired size, or you can simply do a quick byte[] buf = new byte[400 * 1024 * 1024], to make a huge allocation, force the GC, and stretch the heap.
What you don't want to be doing is allocating a million objects and have the VM GC every 10000 or so as it grows. Pre-allocating other data structures is also helpful (notably ArrayLists, LinkedLists not so much).

Divide the file into smaller parts. For this you'll need have access to seekable read so you can fast-forward to other parts of file.
For each part, spawn multiple worker threads, each with its own copy of the hash lookup table. Let completed threads join a collector thread, which will write completed chunks in order and signal the processing completion.
It will be better to stream file chunks rather than loading all of them in memory.

Collect and save value from threads into an array java socket

I am trying to create a java socket program with multithread concept.
Every single connection from client to the server is a thread.
Every client send their information to server.
I want to ask how to collect all value from the threads and put them all in an array of string?
Is needed to synchronized all this threads?
What the different if use only Thread t = new Thread() instead of Thread[] t = new Thread(client)?
Am i right for the beginning step with use Thread[] to collect all item from the threads?
Is it useful to use thread join()?
This is part which create thread,
public void listenSocket(int client){
int i=0;
Thread[] t = new Thread[client];
while(i<client){
ClientWorker w;
try{
w = new ClientWorker(server.accept());
t[i] = new Thread(w);
t[i].start();
System.out.println(t[i].getName());
} catch (IOException e) {
System.out.println("Accept failed: 4444");
System.exit(-1);
}
i++;
}
}
This is the part of what the thread doing
class ClientWorker implements Runnable{
Socket client;
String ip;
String load;
String data[]=new String[7];
ClientWorker(Socket client){
this.client = client;
}
public void setIP(String ip){
this.ip = ip;
}
public void setData(String load){
this.load = load;
}
public void getIP(){
System.out.println(ip);
}
public void getData(){
System.out.println(load);
}
public void run(){
BufferedReader in = null;
PrintWriter out = null;
XmlGenerator xml = new XmlGenerator();
try{
in = new BufferedReader(new InputStreamReader(client.getInputStream()));
out = new PrintWriter(client.getOutputStream(), true);
} catch (IOException e) {
System.out.println("in or out failed");
}
while(true){
try{
String a = in.readLine();
setData(a);
String b = client.getRemoteSocketAddress().toString();
setIP(b);
out.println(a);
getData();
getIP();
}
catch (IOException e) {
System.out.println("Read failed");
}
}
}
}`
i've read the topic about sync,callable thread but i found nothing.
sorry for repost.

There are two ways to answer this:
First way:
I want to ask how to collect all value from the threads and put them all in an array of string? Is needed to synchronized all this threads?
If you are going to collect the results that way, then yes it is necessary to explicitly synchronize the threads' use of that array.
What the different if use only Thread t = new Thread() instead of Thread[] t = new Thread(client)?
It is not clear what you mean, but either way if you have two threads accessing / updating the same data structure then they need to synchronize for the application to be reliable / portable.
Am i right for the beginning step with use Thread[] to collect all item from the threads?
It is one approach. But there are better approaches.
Is it useful to use thread join()?
If you are going to try to address this problem at this level, then it could be useful.
The problem is that your code doesn't give me much confidence that you have a clear idea of what you are doing. Or to put it another way, there is no evidence of a design. And it is not really possible to give you specific advice if we can't figure out how you think your code should work. The code certainly doesn't seem to line up with the things you are saying at the start of your question ...
The Second way to answer this is that from Java 5 onwards, the "java.util.concurrent" package provides a broad range of "off the shelf" components for implementing multi-threaded systems. What you seem to be trying to implement sounds like a good match for an ExecutorService. It can deal with all of the low-level details of creating and managing threads, and allows the application to get the results of each computation back via a Future. This takes away the need for explicit synchronization ... the ExecutorService and Future implementations take care of that.
So, my recommendation would be to read the Javadocs for ExecutorService, etcetera and figure out how to do what you are trying to do using that. Your code is likely to be better if you do it that way.