Develop Reference

unable to find while loops behavior in java

In my application, I have used two while loops for reading the sockets. while loop-1 takes 1% of CPU and while loop-2 takes 100% CPU. I am confused. Please share your valuable thoughts. For more information, I have added two while loops below:
While loop-1:
while(true) {
String message = onReadFinal(in);
if (message != null) {
synchronized (message) {
System.out.println("Message size:" + message);
}
}
}
private String onReadFinal(DataInputStream in) {
String mess = null;
try{
StringBuilder builder = new StringBuilder();
do {
builder.append(in.readInt());
} while(in.available() > 0);
mess = builder.toString();
} catch(IOException e){ e.printStackTrace(); }
return mess;
}
While loop-2:
while(true) {
String message = onRead(in);
if (message != null) {
synchronized (message) {
System.out.println("Message size:" + message);
}
}
}
public String onRead(DataInputStream input) {
String socketmessage = null;
int length;
try {
if(input != null) {
length = input.available();
if(length>0) {
byte[] message = new byte[length];
input.readFully(message, 0, message.length);
socketmessage = new String(message);
}
}
} catch (IOException ioException) {
ioException.printStackTrace();
}
return socketmessage;
}
Also, if I execute infinite while loop with no statements, it takes 100% of CPU. Someone can you please explain.

if a loop does not contain blocking operations, it always would take 100% CPU.
Your first loop contains blocking operation in.readInt(). Your second loop also contains potentially blocking operation input.readFully(), but calls it only when the stream buffer has enough bytes to read, and so in fact this operation does not block. When there is not enough bytes, the loop immediately repeats non-blocking operation input.available(), without much sense.
You better tell us what you want.

Java socket time out does not work

I've a class which is responsible for listening two other machines which have exactly the same classes, so it's a network of three computers having the same code. The connection is there and I can see them passing data to each other. Everything until there works OK.
Things get tricky when I take out one of the machines and observe how the other two behave. Expectedly, when one of the machines stops working for some reason, other two should continue. And if two of them stop, the remaining should go on.
I tried to implement this mechanism below. However, when I take out one of the machines, the program keeps waiting, so it does not switch to "two-way comparison mode".
public void listen() {
try {
logger.info("Creating listener sockets");
while (isRunning) {
final byte[] buf = new byte[bufferSize];
final DatagramPacket packetOne = new DatagramPacket(buf, buf.length);
final DatagramPacket packetTwo = new DatagramPacket(buf, buf.length);
MediatorMessageMsg mediatorMessageOne = null;
MediatorMessageMsg mediatorMessageTwo = null;
try {
socketReceiverOne.receive(packetOne);
ByteArrayInputStream firstInput = new ByteArrayInputStream(buf);
mediatorMessageOne = MediatorMessageMsg.parseDelimitedFrom(firstInput);
socketReceiverTwo.receive(packetTwo);
ByteArrayInputStream secondInput = new ByteArrayInputStream(buf);
mediatorMessageTwo = MediatorMessageMsg.parseDelimitedFrom(secondInput);
logger.trace("Received packets");
} catch (final SocketTimeoutException e) {
logger.trace(e.getMessage());
continue;
} catch (final SocketException e) {
logger.warn(e);
logger.warn("Ignore the error and go on.");
continue;
} catch (final IOException e) {
logger.error("Incoming communication stopped!");
logger.error(e);
stop();
}
// if two mediators sent the data, it's OK
if (packetOne.getLength() > 0 && packetTwo.getLength() > 0) {
handlePackets(mediatorMessageOne, mediatorMessageTwo);
logger.info("Number of active mediators: 2. Comparison style: 1v1v1");
}
// if only one sent the data, compare it with our own
else if (packetOne.getLength() > 0 || packetTwo.getLength() > 0) {
// whicehever sent the data, compare its data with our own
logger.info("Number of active mediators: 1. Comparison style: 1v1");
if (packetOne.getLength() > 0) {
handlePackets(mediatorMessageOne);
} else {
handlePackets(mediatorMessageTwo);
}
}
// if no data is sent, then pass our own directly
else {
logger.info("Number of active mediators: 0. Comparison style: No Comparison");
// our datamodel to retrieve data on our own
DataModel modelOwn = DataModel.getInstance();
MediatorMessageMsg newMessage = MediatorMessageMsg.newBuilder().setHeading(modelOwn.getHeading()).setSpeed(modelOwn.getSpeed()).setSender(getId()).build();
// publish(topicName, newMessage);
}
Thread.sleep(1);
}
socketReceiverOne.close();
socketReceiverTwo.close();
logger.info("stopped");
} catch (final IllegalArgumentException e) {
logger.error("Illegal argument received: " + e);
} catch (final Exception e) {
logger.error("Unexpected error occured: " + e);
} finally {
if (socketReceiverOne instanceof DatagramSocket && socketReceiverTwo instanceof DatagramSocket) {
if (!socketReceiverOne.isClosed() || !socketReceiverTwo.isClosed()) {
socketReceiverOne.close();
socketReceiverTwo.close();
}
}
}
}
To save your time, let me share my opinion on the matter. I suspect the problem to be in this part:
socketReceiverOne.receive(packetOne);
ByteArrayInputStream firstInput = new ByteArrayInputStream(buf);
mediatorMessageOne = MediatorMessageMsg.parseDelimitedFrom(firstInput);
socketReceiverTwo.receive(packetTwo);
ByteArrayInputStream secondInput = new ByteArrayInputStream(buf);
mediatorMessageTwo = MediatorMessageMsg.parseDelimitedFrom(secondInput);
To me it seems like the program expects a package and when it cannot receive it, it keeps waiting. Although I have time out exception condition, I cannot get this done.
private int socketTimeout = 1000 * 2;// 2sec
socketReceiverOne.setSoTimeout(socketTimeout);
socketReceiverTwo.setSoTimeout(socketTimeout);
Any thoughts?

Okay I found where I was mistaken. I needed more ports (for in and out). Once I incorporated those ports, the problem did not occur again.

Read the 30Million user id's one by one from the big file

I am trying to read a very big file using Java. That big file will have data like this, meaning each line will have an user id.
149905320
1165665384
66969324
886633368
1145241312
286585320
1008665352
And in that big file there will be around 30Million user id's. Now I am trying to read all the user id's one by one from that big file only once. Meaning each user id should be selected only once from that big file. For example, if I have 30Million user id's then it should print 30 Million user id only once with the use of Multithreading code.
Below is the code I have which is a multithreaded code running with 10 threads but with the below program, I am not able to make sure that each user id is selected only once.
public class ReadingFile {
public static void main(String[] args) {
// create thread pool with given size
ExecutorService service = Executors.newFixedThreadPool(10);
for (int i = 0; i < 10; i++) {
service.submit(new FileTask());
}
}
}
class FileTask implements Runnable {
#Override
public void run() {
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader("D:/abc.txt"));
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
//do things with line
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Can anybody help me with this? What wrong I am doing? And what is the fastest way to do this?

You really can't improve on having one thread reading the file sequentially, assuming that you haven't done anything like stripe the file across multiple disks. With one thread, you do one seek and then one long sequential read; with multiple threads you're going to have the threads causing multiple seeks as each gains control of the disk head.
Edit: This is a way to parallelize the line processing while still using serial I/O to read the lines. It uses a BlockingQueue to communicate between threads; the FileTask adds lines to the queue, and the CPUTask reads them and processes them. This is a thread-safe data structure, so no need to add any synchronization to it. You're using put(E e) to add strings to the queue, so if the queue is full (it can hold up to 200 strings, as defined in the declaration in ReadingFile) the FileTask blocks until space frees up; likewise you're using take() to remove items from the queue, so the CPUTask will block until an item is available.
public class ReadingFile {
public static void main(String[] args) {
final int threadCount = 10;
// BlockingQueue with a capacity of 200
BlockingQueue<String> queue = new ArrayBlockingQueue<>(200);
// create thread pool with given size
ExecutorService service = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < (threadCount - 1); i++) {
service.submit(new CPUTask(queue));
}
// Wait til FileTask completes
service.submit(new FileTask(queue)).get();
service.shutdownNow(); // interrupt CPUTasks
// Wait til CPUTasks terminate
service.awaitTermination(365, TimeUnit.DAYS);
}
}
class FileTask implements Runnable {
private final BlockingQueue<String> queue;
public FileTask(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader("D:/abc.txt"));
String line;
while ((line = br.readLine()) != null) {
// block if the queue is full
queue.put(line);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
class CPUTask implements Runnable {
private final BlockingQueue<String> queue;
public CPUTask(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
String line;
while(true) {
try {
// block if the queue is empty
line = queue.take();
// do things with line
} catch (InterruptedException ex) {
break; // FileTask has completed
}
}
// poll() returns null if the queue is empty
while((line = queue.poll()) != null) {
// do things with line;
}
}
}

We are talking about an average of a 315 MB file with lines separated by new line. I presume this easily fits into memory. It is implied that there is no particular order in the user names that has to be conserved. So I would recommend the following algorithm:
Get the file length
Copy each a 10th of the file into a byte buffer (binary copy should be fast)
Start a thread for processing each of these buffers
Each thread processes all lines in his area except the first and last one.
Each thread must return the first and last partitial line in its data when done,
the “last” of each thread must be recombined with the “first” one of the one working on the next file block because you may have cut through a line. And these tokens must then be processed afterwards.

Fork Join API introduced in 1.7 is a great fit for this use case. Check out http://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html. If you search, you are going to find lots of examples out there.

How do I solve this indexOutOfBoundsException in my server send/receive thread?

I am creating a multiplayer game in Java with a server and multiple clients. Everything runs perfectly, until I press the Kick-button in the server to kick a client.
Error at receive thread of server, after kicking the first person who joined out of three:
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck(ArrayList.java:604)
at java.util.ArrayList.get(ArrayList.java:382)
> at networktest.Server$3.run(Server.java:186)
at java.lang.Thread.run(Thread.java:722)
The pointed line is the ois = new ObjectInputStream where I receive datatype in the server receive thread. The server kicks the first person perfectly, but removes the second one in the list too, with an error of java.lang.ClassCastException.
server receive:
private static Thread receive = new Thread()
{
#Override
public void run()
{
ObjectInputStream ois;
while (true)
{
for (int i = 0; i < list_sockets.size(); i++)
{
try
{
ois = new ObjectInputStream(list_sockets.get(i).getInputStream());
int receive_state = (Integer) ois.readObject(); // receive state
ois = new ObjectInputStream(list_sockets.get(i).getInputStream());
byte datatype = (byte) ois.readObject(); // receive datatype
if(datatype == 2){
ois = new ObjectInputStream(list_sockets.get(i).getInputStream());
ChatLine chatLine = (ChatLine) ois.readObject(); // receive ChatLine
} else if (datatype == 0){
ois = new ObjectInputStream(list_sockets.get(i).getInputStream());
DataPackage dp = (DataPackage) ois.readObject(); // receive dp
list_data.set(i, dp);
}
if (receive_state == 1) // Client Disconnected by User
{
disconnectClient(i);
i--;
}
}
catch (Exception ex) // Client Disconnected (Client Didn't Notify Server About Disconnecting)
{
System.err.println("Error # receive:");
ex.printStackTrace();
disconnectClient(i);
i--;
}
}
try {
this.sleep(3);
} catch (InterruptedException ex) {
Logger.getLogger(Server.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
};
user send:
Thread send = new Thread()
{
public void run()
{
ObjectOutputStream oos;
byte datatype = 0;
while (connected){
if (socket != null){
try {
DataPackage dp = new DataPackage();
dp.x = Client.player.x;
dp.y = Client.player.y;
dp.username = username;
dp.charType = charType;
dp.walking = (byte)Client.player.walking;
if (Client.outputChatLine.line != null)
datatype = 2;
else {
datatype = 0;
}
oos = new ObjectOutputStream(socket.getOutputStream());
oos.writeObject(Integer.valueOf(Client.this.state)); // send state
oos = new ObjectOutputStream(socket.getOutputStream());
oos.writeObject(Byte.valueOf(datatype)); // send datatype
if (datatype == 2)
{
oos.reset();
oos.writeObject(Client.outputChatLine);
Client.outputChatLine = new ChatLine();
} else {
oos = new ObjectOutputStream(socket.getOutputStream());
oos.writeObject(dp);
}
if (Client.this.state == 1) {
connected = false;
socket = null;
JOptionPane.showMessageDialog(null, "Client Disconnected", "Info", 1);
System.exit(0);
}
}
catch (Exception ex){}
}
try {
this.sleep(2);
} catch (InterruptedException ex) {
Logger.getLogger(Client.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
};
disconnect client method:
public static void disconnectClient(int index)
{
try
{
list_clients_model.removeElementAt(index);
list_client_states.remove(index);
list_data.remove(index);
list_sockets.remove(index);
}
catch (Exception ex) {}
}
Does anyone know how to solve this?

It looks like you're expecting some other thread to fill some data into list_sockets while you're sleep(3)ing. But you have no synchronization to ensure that this happens only while you're sleeping.
It could equally well happen that the other thread is updating list_sockets simultaneously with your own thread calling list_sockets.get(i). And the ArrayList implementation it is almost certainly not written to have two different of its methods executing simultaneously in two different threads. For example, the other thread could be in the middle of resizing the backing array when you're trying to read an element, and then any crazy thing can go wrong, including the error you're seeing.
You need to learn about inter-thread synchronization. At the very least you need synchronized blocks to protect access to shared data structures. And while you're at it; look at wait/notify or some higher-level concurrency tools to get rid of that horrible 3-millisecond polling loop -- instead have the thread that drops work into the list explicitly wake up the worker thread.

Processing log files, distribute work among worker threads, to find a simple sum

I want to distribute work among threads. Load parts of a log file and then distribute the work to process parts of the file.
In my simple example, I wrote 800,000 lines of data and had a number in each line. And then I sum the number.
When I run this example, I get totals that are slightly off. Do you see in this threading code where threads might not complete properly and hence won't total the numbers?
public void process() {
final String d = FILE;
FileInputStream stream = null;
try {
stream = new FileInputStream(d);
final BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String data = "";
do {
final Stack<List<String>> allWork = new Stack<List<String>>();
final Stack<ParserWorkerAtLineThread> threadPool = new Stack<ParserWorkerAtLineThread>();
do {
if (data != null) {
final List<String> currentWorkToDo = new ArrayList<String>();
do {
data = reader.readLine();
if (data != null) {
currentWorkToDo.add(data);
} // End of the if //
} while(data != null && (currentWorkToDo.size() < thresholdLinesToAdd));
// Hand out future work
allWork.push(currentWorkToDo);
} // End of the if //
} while(data != null && (allWork.size() < numberOfThreadsAllowedInPool));
// Process the lines from the work to do //
// Hand out the work
for (final List<String> theCurrentTaskWork : allWork) {
final ParserWorkerAtLineThread t = new ParserWorkerAtLineThread();
t.data = theCurrentTaskWork;
threadPool.push(t);
}
for (final Thread workerAboutToDoWork : threadPool) {
workerAboutToDoWork.start();
System.out.println(" -> Starting my work... My name is : " + workerAboutToDoWork.getName());
} // End of the for //
// Waiting on threads to finish //
System.out.println("Waiting for all work to complete ... ");
for (final Thread waiting : threadPool) {
waiting.join();
} // End of the for //
System.out.println("Done waiting ... ");
} while(data != null); // End of outer parse file loop //
} catch(Exception e) {
e.printStackTrace();
} finally {
if (stream != null) {
try {
stream.close();
} catch (final IOException e) {
e.printStackTrace();
}
} // End of the stream //
} // End of the try - catch finally //
}

While you're at it, why not use a bounded BlockingQueue (ArrayBlockingQueue) of size thresholdLinesToAdd. This would be your producer code where you read the lines and use the method put on that queue to block until space is available.
As Chris mentionned before, use the Executors.newFixedThreadPool() to submit your work items on it. Your consumers would call take() to block until an element is available.
This is not a map/reduce. If you wanted a map/reduce, you would need another queue in the mix where you would publish keys to it. As an example, if you were to count the number of INFO and DEBUG occurances in your logs, your mapper would queue the extracted words every time it encounters it. The reducer would dequeue the mapper's output and increment the counter of each words. The result of your reducer would the word count for DEBUG and INFO.