I thought I had a fairly decent understanding of locks and basic multi-threading concepts, but I'm royally messing something up here.
All the program is supposed to do is receive a filename for a text file and the number of threads you want to use from the user, and then count the number of "http" links in that file before returning that number to the user.
For the life of me, however, I can't get my threads to count properly. I've tried making the "count" variable an atomic integer and using the "incrementAndGet()" function, I've tried placing locks in the code where I thought they should be, and I've tried specifying the necessary functions are synchronized, all to no avail.
I've been doing the reading that I can on locks, concurrency, and whatnot, but I'm apparently doing something very wrong. Could someone explain to me where/why I need to be placing locks in my code (or how to properly use atomic integers, if that would be more effective)?
I originally had a lock for when the "count" variable was used (outside of any loops so it actually did something), but I'm receiving all kinds of numbers when I run the program.
My code looks like this:
import java.io.*;
import java.util.*;
import java.util.concurrent.locks.*;
import java.util.concurrent.atomic.AtomicInteger;
public class Count
{
static String fname;
static int count = 0;
public static void main(String[] arg)
{
Scanner in = new Scanner(System.in);
int numthreads;
if( arg.length > 0 )
{
fname = arg[0];
numthreads = Integer.parseInt(arg[1]);
}
else
{
System.out.println("File?");
fname = in.nextLine();
System.out.println("Num threads?");
numthreads = in.nextInt();
}
RandomAccessFile raf;
try
{
raf = new RandomAccessFile(fname,"r");
Thread[] T = new Thread[numthreads];
for(int i=0;i<numthreads;++i)
{
T[i] = new Thread(new Task(i,numthreads,raf));
T[i].start();
}
for(Thread t : T)
{
try
{
t.join();
}
catch (InterruptedException e)
{
System.out.println("Thread join failed!");
e.printStackTrace();
System.exit(0);
}
}
System.out.println("The total is: "+count);
}
catch(IOException e)
{
System.out.println("Could not read file");
}
}
}
class Task implements Runnable
{
int myid;
int totaltasks;
RandomAccessFile raf;
ReentrantLock L = new ReentrantLock();
public Task(int myid, int totaltasks, RandomAccessFile raf)
{
this.myid=myid;
this.totaltasks=totaltasks;
this.raf = raf;
}
#Override
public void run()
{
try
{
long filesize = raf.length(); //length of the entire file
long sz = filesize / totaltasks; //length of entire file divided by number of threads gives thread size
long start = myid*sz; // thread's number * size of thread. How we find start of thread in the file
raf.seek(start);
if(start != 0 )
{
raf.readLine();
}
while(raf.getFilePointer() < (start+sz))
{
String s = raf.readLine();
Scanner sc = new Scanner(s);
while(sc.hasNext())
{
String w = sc.next();
if( w.startsWith("http://"))
{
Count.count++;
}
}
}
}
catch(IOException e)
{
System.out.println("IO error!");
}
}
}
If there's bigger issues at work here than me needing to use locks in the proper places, please let me know and I'll at least have something to work with, but as far as I'm aware it should just be a matter of performing locks properly and I'm either not understanding how they work correctly or just putting them in all the wrong places.
Related
Suppose I have the following code, where one thread generates squares and writes them to a buffer while another thread prints them:
import java.util.*;
public class Something {
public static Buffer buffer = new Buffer();
public static class Buffer {
private int[] buffer;
private static final int size = 10;
//Indexes for putting and taking element form buffer
private int in, out;
//Number of elements in buffer
private int k;
public Buffer() {
buffer = new int[size];
in = 0;
out = 0;
k = 0;
}
public synchronized void put(int e) {
try {
while (k == buffer.length) {
wait();
}
} catch (InterruptedException ex) {
}
buffer[in] = e;
k++;
in = ++in % size;
notifyAll();
}
public synchronized int take() {
try {
while (k == 0) {
wait();
}
} catch (InterruptedException ex) {
}
int e = buffer[out];
buffer[out] = 0;
out = ++out % size;
k--;
notifyAll();
return e;
}
public synchronized boolean notEmpty() {
return k != 0;
}
}
public static class Generator implements Runnable {
int limit;
public Generator(int lim) {
limit= lim;
}
#Override
public void run() {
for (int i = 1; i < limit; i++) {
buffer.put(i * i);
}
}
}
public static class Printer implements Runnable {
private Thread[] generators;
public Printer(Thread[] gen) {
generators = gen;
}
public synchronized boolean nobody() {
for (Thread th : generators) {
if (th.isAlive()) {
return false;
}
}
return true;
}
#Override
public void run() {
int x = 0;
while (!nobody() || buffer.notEmpty()) {
x = buffer.take();
System.out.println(x);
}
}
}
public static void main(String[] args) throws InterruptedException {
Thread generator = new Thread(new Generator(69));
Thread printer = new Thread(new Printer(new Thread[]{generator}));
generator.start();
printer.start();
generator.join();
printer.join();
}
}
Generator should generate squares of numbers until it reaches some limit (limit = 69, in this case). Printer should print all values generated by Generator. Buffer works somewhat like ring buffer. Indexes for putting (in) and taking (out) element are cycling in bounds of buffer size. Buffer has methods for putting and taking elements from buffer. Generator thread cannot put elements in buffer if it is full (that is, there are no zero elements; zero element is 0, for precision sake...). Printer works this way: first it checks if there are any alive generator threads and then checks if buffer contains only zero elements. If neither of these conditions is true, printer thread terminates.
Now, to the problem. I always get printed all squares from 1 to 68, which is expected output of this program. However, on very rare occasion after all numbers had been output I get a deadlock. How rarely? Well, maybe in 1 out of 100 executions of program. I had to keep hitting "F6" on NetBeans like crazy to get a deadlock. And yes, I know that I can test this simply putting all main code in for loop.
Conversely, if I comment out print line in Printers' run method, deadlock happens almost all the time. Here:
#Override
public void run() {
int x = 0;
while (!nobody() || buffer.notEmpty()) {
x = buffer.take();
//System.out.println(x);
}
}
I do not expect this behavior, because element still gets taken from buffer and generator should be awoken.
Why does this happen? And how do I fix it?
Sorry if question isn't clear enough, I'll try to clarify it as best I can if needed.
I think I fount the problem. Here is what I got: There is a very short moment in time, where the Generator thread is still alive (i.e. Thread.isAlive() will return true), but the Generator has left the for-loop within run(). If the Printer queries its while-condition within its run() at this point in time, it will try to take() something, that is not there (and never will be). Indeed, you can verify, that the Generator always finishes, meaning termination detection on the side of the Printer is faulty. For a hot fix, you can simply add a magic constant is Printers while condition:
#Override
public void run() {
int x = 0;
int count = 0;
while (++count < 69) {
x = buffer.take();
System.out.println(x);
}
}
For a clean termination detection, you could set some common flag-variable to false, signaling that the Generator has finished work and the Printer can stop working. But this has to be done in a synchronized manner, meaning the Printer is not allowed to query this condition, while the Generator is after its last push, but before it sets this common flag.
I think I've almost figured out my java program. It is designed to read a text file and find the largest integer by using 10 different threads. I'm getting this error though:
Error:(1, 8) java: class Worker is public, should be declared in a file named Worker.java
I feel my code may be more complex than it needs to be so I'm trying to figure out how to shrink it down in size while also fixing the error above. Any assistance in this matter would be greatly appreciated and please let me know if I can clarify anything. Also, does the "worker" class have to be a seperate file? I added it to the same file but getting the error above.
import java.io.BufferedReader;
import java.io.FileReader;
public class datafile {
public static void main(String[] args) {
int[] array = new int[100000];
int count;
int index = 0;
String datafile = "dataset529.txt"; //string which contains datafile
String line; //current line of text file
try (BufferedReader br = new BufferedReader(new FileReader(datafile))) { //reads in the datafile
while ((line = br.readLine()) != null) { //reads through each line
array[index++] = Integer.parseInt(line); //pulls out the number of each line and puts it in numbers[]
}
}
Thread[] threads = new Thread[10];
worker[] workers = new worker[10];
int range = array.length / 10;
for (count = 0; count < 10; count++) {
int startAt = count * range;
int endAt = startAt + range;
workers[count] = new worker(startAt, endAt, array);
}
for (count = 0; count < 10; count++) {
threads[count] = new Thread(workers[count]);
threads[count].start();
}
boolean isProcessing = false;
do {
isProcessing = false;
for (Thread t : threads) {
if (t.isAlive()) {
isProcessing = true;
break;
}
}
} while (isProcessing);
for (worker worker : workers) {
System.out.println("Max = " + worker.getMax());
}
}
}
public class worker implements Runnable {
private int startAt;
private int endAt;
private int randomNumbers[];
int max = Integer.MIN_VALUE;
public worker(int startAt, int endAt, int[] randomNumbers) {
this.startAt = startAt;
this.endAt = endAt;
this.randomNumbers = randomNumbers;
}
#Override
public void run() {
for (int index = startAt; index < endAt; index++) {
if (randomNumbers != null && randomNumbers[index] > max)
max = randomNumbers[index];
}
}
public int getMax() {
return max;
}
}
I've written a few comments but I'm going to gather them all in an answer so anyone in future can see the aggregate info:
At the end of your source for the readtextfile class (which should be ReadTextile per java naming conventions) you have too many closing braces,
} while (isProcessing);
for (Worker worker : workers) {
System.out.println("Max = " + worker.getMax());
}
}
}
}
}
The above should end on the first brace that hits the leftmost column. This is a good rule of thumb when making any Java class, if you have more than one far-left brace or your last brace isn't far-left you've probably made a mistake somewhere and should go through checking your braces.
As for your file issues You should have all your classes named following Java conventions and each class should be stored in a file called ClassName.java (case sensitive). EG:
public class ReadTextFileshould be stored in ReadTextFile.java
You can also have Worker be an inner class. To do this you could pretty much just copy the source code into the ReadTextFile class (make sure it's outside of the main method). See this tutorial on inner classes for a quick overview.
As for the rest of your question Code Review SE is the proper place to ask that, and the smart folks over there probably will provide better answers than I could. However I'd also suggest using 10 threads is probably not the most efficient way in to find the largest int in a text file (both in development and execution times).
I am making a program that reads a file of mixed values (int and string), prints only the integer values and keeps a running total of the amount of integer values within the file. Everything is working except for my running total of integer values within a given file and i am very confused on why it keeps printing 0 when i know there are more then 0 integer values within the file.
Here is my code:
package davi0030_a03;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class MyFile {
private String fileName; // name of the file
private int count = 0; // number of valid integers in the file
private final int MAX_SIZE = 10; // the size of the array
private Scanner inputStream = null;
private int[] theArray = new int[MAX_SIZE];
private boolean strangeInt = false;
private int total = 0;
// constructor to set the file name
public MyFile(String theName) { // constructor to set the file name
this.fileName = new String(theName);
// you may or may not want to do other stuffs here
}
public void openFile() {
System.out.println("opening file: " + fileName);
try {
inputStream = new Scanner(new FileInputStream("src/davi0030_a03/"
+ fileName));
} catch (FileNotFoundException e) {
System.out.println("File was not found or could not be opened");
}
}
// log a message on whether two ints in the file add to target
public void findPair(int target) {
openFile();
fileToArray();
findStrangeInt();
findTotal();
}
public void findTotal(){
inputStream.reset();
while(inputStream.hasNext()){
if(inputStream.hasNextInt()){
total +=1;
}
inputStream.next();
}
System.out.println(total);
}
public void findStrangeInt() {
inputStream.reset();
while (inputStream.hasNext()) {
try {
Integer.parseInt(inputStream.next());
} catch (NumberFormatException nfe) {
strangeInt = true;
}
}
if (strangeInt = true) {
System.out.println("File contains an incorrectly written int");
}
}
public void fileToArray() {
inputStream.reset();
while (inputStream.hasNext() && count < MAX_SIZE) {
if (inputStream.hasNextInt()) {
theArray[count] = inputStream.nextInt();
count++;
}
}
}
// print the content of the file
public void printFile() {
openFile();
inputStream.reset();
System.out.println("Printing content of file " + fileName);
while (inputStream.hasNext()) {
try {
int convert = Integer.parseInt(inputStream.next());
System.out.println(convert);
} catch (NumberFormatException nfe) {
System.out.println("xxx");
}
}
}
}
content of file:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
a
b
d
EDIT: My solution below most likely solves the logic issue regarding your searching the file. However, I suggest you read some of the other comments the question has gotten. The IO handling here especially needs some improvement.
I believe your issue lies in your while statement. According to java documentation on the Scanner class,here, the method .hasNextInt() is returning based on the next element scanned. I think you are assuming it will return true as long as there are ints in the file. This is not the case. If the first thing the scanner hits is not an int, it is returning false and exiting your while loop. This would explain the returned 0, the value you initialize your variable 'total' to. I would suggest you do something like the following:
while(scanner.hasNext()){
if(scanner.hasNextInt()){
total +=1;
}
scanner.next();
}
On compiling my code below it seems to be in a state of deadlock, and i don't know how i can fix it. I am attempting to write a pipeline as a sequence of threads linked together as a buffer, and each thread can read the preceding node in the pipeline, and consequentially write to the next one. The overall goal is to spilt a randomly generated arraylist of data over 10 threads and sort it.
class Buffer{
// x is the current node
private int x;
private boolean item;
private Lock lock = new ReentrantLock();
private Condition full = lock.newCondition();
private Condition empty = lock.newCondition();
public Buffer(){item = false;}
public int read(){
lock.lock();
try{
while(!item)
try{full.await();}
catch(InterruptedException e){}
item = false;
empty.signal();
return x;
}finally{lock.unlock();}
}
public void write(int k){
lock.lock();
try{
while(item)
try{empty.await();}
catch(InterruptedException e){}
x = k; item = true;
full.signal();
}finally{lock.unlock();}
}
}
class Pipeline extends Thread {
private Buffer b;
//private Sorted s;
private ArrayList<Integer> pipe; // array pipeline
private int ub; // upper bounds
private int lb; // lower bounds
public Pipeline(Buffer bf, ArrayList<Integer> p, int u, int l) {
pipe = p;ub = u;lb = l;b = bf;//s = ss;
}
public void run() {
while(lb < ub) {
if(b.read() > pipe.get(lb+1)) {
b.write(pipe.get(lb+1));
}
lb++;
}
if(lb == ub) {
// store sorted array segment
Collections.sort(pipe);
new Sorted(pipe, this.lb, this.ub);
}
}
}
class Sorted {
private volatile ArrayList<Integer> shared;
private int ub;
private int lb;
public Sorted(ArrayList<Integer> s, int u, int l) {
ub = u;lb = l;shared = s;
// merge data to array from given bounds
}
}
class Test1 {
public static void main(String[] args) {
int N = 1000000;
ArrayList<Integer> list = new ArrayList<Integer>();
for(int i=0;i<N;i++) {
int k = (int)(Math.random()*N);
list.add(k);
}
// write to buffer
Buffer b = new Buffer();
b.write(list.get(0));
//Sorted s = new Sorted();
int maxBuffer = 10;
int index[] = new int[maxBuffer+1];
Thread workers[] = new Pipeline[maxBuffer];
// Distribute data evenly over threads
for(int i=0;i<maxBuffer;i++)
index[i] = (i*N) / maxBuffer;
for(int i=0;i<maxBuffer;i++) {
// create instacen of pipeline
workers[i] = new Pipeline(b,list,index[i],index[i+1]);
workers[i].start();
}
// join threads
try {
for(int i=0;i<maxBuffer;i++) {
workers[i].join();
}
} catch(InterruptedException e) {}
boolean sorted = true;
System.out.println();
for(int i=0;i<list.size()-1;i++) {
if(list.get(i) > list.get(i+1)) {
sorted = false;
}
}
System.out.println(sorted);
}
}
When you start the run methods, all threads will block until the first thread hits full.await(). then one after the other, all threads will end up hitting full.await(). they will wait for this signal.
However the only place where full.signal occurs is after one of the read methods finishes.
As this code is never reached (because the signal is never fired) you end up with all threads waiting.
in short, only after 1 read finishes, will the writes trigger.
if you reverse the logic, you start empty, you write to the buffer (with signal, etc, etc) and then the threads try to read, I expect it will work.
generally speaking you want to write to a pipeline before reading from it. (or there's nothing to read).
I hope i'm not misreading your code but that's what I see on first scan.
Your Buffer class it flipping between read and write mode. Each read must be followed by a write, that by a read and so on.
You write the buffer initially in your main method.
Now one of your threads reaches if(b.read() > pipe.get(lb+1)) { in Pipeline#run. If that condition evaluates to false, then nothing gets written. And since every other thread must still be the very same if(b.read(), you end up with all reading threads that can't progress. You will either have to write in the else branch or allow multiple reads.
I have very similar multithreaded code elsewhere in my codebase that works fine, but I can't see quite what's going wrong here.
This is a simple multi-threaded process to generate some result XML for a search query. The output of running this method is:
Returning from threads
The line System.out.println("Finished multithreading loop");" is never reached.
Modifying the number of threads doesn't help.
private void fillAllResults() {
int threads = 2;
final FutureTask[] tasks = new FutureTask[threads];
final ExecutorService executor = Executors.newCachedThreadPool();
for (int i = 0; i < allResults.size(); i++) {
tasks[i] = new FutureTask<Integer>(new Callable<Integer>() {
public Integer call() throws Exception {
int index;
while ((index = getResultsIndex()) < allResults.size()) {
System.out.println("Processing result " + index);
Result result = allResults.get(index);
fillResultXML(result);
}
System.out.println("Returning from threads");
return 1;
}
});
executor.execute(tasks[i]);
}
for (int i = 0; i < threads; i++) {
try {
tasks[i].get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
executor.shutdown();
System.out.println("Finished multithreading loop");
}
Edit, thanks all for the quick replies! Here's the answers:
It shows 'processing result' as many times as I have results. If allResults.size() is 25, it shows processing result 1, processing result 2 ... processing result 24.
Here's the extra code that's missing:
private List<Result> allResults = new ArrayList<Result>();
private int resultsIndex = 0;
private synchronized int getResultsIndex() {
return resultsIndex++;
}
And in case anyone's wondering, I can guarantee that none of the code within the loop increases the size of allResults.
I suppose it is related to the fact, that your array tasks has a length of threads (i.e. two in your case) but you assign more values to it within the lines
for (int i = 0; i < allResults.size(); i++) {
tasks[i] = ...
....
}
If your list allResults has more than two entries your thread will be stopped by an ArrayIndexOutOfBoundsException. Maybe you catch this one but do not handle it properly outside the code you presented.
It looks like getResultsIndex() isn't updating after every loop resulting in an infinite loop.
It's not clear from your code what allResults and getResultsIndex are, but you never seem to update whatever getResultsIndex returns.