Develop Reference

Java [Peer-To-Peer]: Runnable stops/blocks unexpectedly

I am developing a simple distributed ledger. I want to be able to start nodes on different ports, that could communicate with eachother. Each programme then would have a file, where it would write newly discovered nodes.
At first, only the most reliable nodes are hardcoded into that file.
Here is procedurally what happens:
1) I start a new node, which starts a HTTP server (I use com.sun.HttpServer). The server has a GetAddress handler, which listens to requests that go to the specified URI. It then gets the IP and PORT (which is specified in URI query params), acquires a semaphore for a known_nodes.txt file, and writes the newly received peer address to that file, if it's not already there, and send the contents of newly updated file as a json list back to the requester.
2) Within my Node class (which, as mentioned earlier, starts a HTTPServer on a separate thread), I create a ScheduledExecutorService and give it a runnable to be run every few seconds, whose job will be to connect to the URLs present in the known_nodes.txt file, and ask them for their known_nodes. If we received nodes that were not previously present in our known_nodes file, we overwrite our file.
NOW!
If I start a node, and try to request it from the browser, everything goes as planned - we receive a request, write it to our file, then our runnable will try to connect to the address specified in request. If we caught a SocketTimeoutException, we remove the address from our known_nodes.txt file.
The problem arises, when I start two nodes, running let's say on port 8001 and 8002. Please note, that each node has its own known_nodes file.
What happens, is that one of the nodes will stop running the DiscoverAddresses task, the other one won't. So effectively, one nodes stopped receiving requests.
NB! The node that will stop its scheduledtask will STILL send at least ONE discovering request, and then will die/block(?).
Here is the code for the runnable task:
#Override
public void run() {
log.info("still running ");
PeerAddressesHolder inactiveNodes = new PeerAddressesHolder();
ApplicationConfiguration appConf = ApplicationConfiguration.getInstance();
for (PeerAddress peerAddress : knownNodes.getAddresses()) {
if (isSameNode(peerAddress)) {
continue;
}
String urlString = String.format("http://%s:%s%s?myport=%d", peerAddress.getIP(), peerAddress.getPort(), Constants.GET_ADDRESS, myPort);
try {
StringBuilder result = new StringBuilder();
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setConnectTimeout(5000);
conn.setRequestMethod("GET");
try (InputStream connInputStream = conn.getInputStream();
InputStreamReader ir = new InputStreamReader(connInputStream);
BufferedReader br = new BufferedReader(ir)){
String line;
while ((line = br.readLine()) != null) {
result.append(line).append('\n');
}
} catch (Exception e) {
log.warn("Couldn't read from connection input stream",e);
}
PeerAddressesHolder peerAddressesHolder = gson.fromJson(result.toString(), PeerAddressesHolder.class);
boolean fetchedNew = false;
for (PeerAddress fetchedAddress : peerAddressesHolder.getAddresses()) {
if (!isValidAddress(peerAddress)) {
log.warn("Peer has sent us a null-address. It will be ignored.");
return;
}
if (!knownNodes.contains(fetchedAddress)) {
knownNodes.addAddress(fetchedAddress);
fetchedNew = true;
}
}
if (fetchedNew) {
FileUtils.writeToFile(appConf.getKnownNodesFilePath(), gson.toJson(knownNodes), false);
}
} catch (SocketTimeoutException e) {
if (appConf.getMostReliableNodes().contains(peerAddress)) {
log.warn("Most reliable node not available: " + peerAddress);
} else {
inactiveNodes.addAddress(peerAddress);
log.warn("Connection timeout from " + peerAddress + ". It will be removed.");
}
} catch (Exception e) {
log.warn("Couldn't discover new addresses." + e);
}
}
try {
knownNodes.removeAll(inactiveNodes.getAddresses());
FileUtils.writeToFile(appConf.getKnownNodesFilePath(), gson.toJson(knownNodes), false);
} catch (IOException ioe) {
log.warn("Couldn't write to file after deleting dead node", ioe);
}
}
And here is how I start it upon Node creation.
public NetworkNode(int port) {
this.appConf = ApplicationConfiguration.getInstance();
this.port = port;
log.info("Starting a new node on port " + port);
try {
this.knownNodes = FileUtils.createPeerAddressesList(appConf.getKnownNodesFilePath());
} catch (Exception e) {
log.error("Error while trying to construct a list of peer addresses from file content on path: " + appConf.getKnownNodesFilePath());
}
scheduledExecutorService = Executors.newScheduledThreadPool(4);
scheduledExecutorService.scheduleAtFixedRate(new DiscoverAddressesTask(knownNodes, this.port), 3, 4, TimeUnit.SECONDS);
Methods dealing with file reading/writing are all done using try-with-resources construct, so my initial idea that the runnable stops because of some unclosed streams is probably not valid.

IBM MQ CLIENT java.lang.UnsatisfiedLinkError: no mqjbnd05 in java.library.path exception [duplicate]

This question already has answers here:
java.lang.UnsatisfiedLinkError
(6 answers)
Closed 6 years ago.
I am tring to connect to IBM websphere Client with a Java Programme
the Following are the code:=
import com.ibm.mq.MQException;
import com.ibm.mq.MQGetMessageOptions;
import com.ibm.mq.MQMessage;
import com.ibm.mq.MQPutMessageOptions;
import com.ibm.mq.MQQueue;
import com.ibm.mq.MQQueueManager;
/**
* Simple example program
*/
public class MQSample {
// code identifier
static final String sccsid = "#(#) MQMBID sn=p750-002-131001_DE su=_FswqMCqGEeOZ3ui-rZDONA pn=MQJavaSamples/wmqjava/MQSample.java";
// define the name of the QueueManager
private static final String qManager = "QM_ORANGE";
// and define the name of the Queue
private static final String qName = "SYSTEM.DEFAULT.LOCAL.QUEUE";
// private static final String qName = "QM_APPLE";
public static void main(String args[]) {
try {
System.out.println("Connecting to queue manager: " + qManager);
MQQueueManager qMgr = new MQQueueManager(qManager);
int openOptions =1| 16;
System.out.println("Accessing queue: " + qName);
MQQueue queue = qMgr.accessQueue(qName, openOptions);
MQMessage msg = new MQMessage();
msg.writeUTF("Hello, World!");
MQPutMessageOptions pmo = new MQPutMessageOptions();
System.out.println("Sending a message...");
queue.put(msg, pmo);
// Now get the message back again. First define a WebSphere MQ
// message
// to receive the data
// MQMessage rcvMessage = new MQMessage();
// Specify default get message options
// MQGetMessageOptions gmo = new MQGetMessageOptions();
// Get the message off the queue.
// System.out.println("...and getting the message back again");
// queue.get(rcvMessage, gmo);
// And display the message text...
//String msgText = rcvMessage.readUTF();
// System.out.println("The message is: " + msgText);
// Close the queue
System.out.println("Closing the queue");
queue.close();
// Disconnect from the QueueManager
System.out.println("Disconnecting from the Queue Manager");
qMgr.disconnect();
System.out.println("Done!");
}
catch (MQException ex) {
System.out.println("A WebSphere MQ Error occured : Completion Code " + ex.completionCode
+ " Reason Code " + ex.reasonCode);
ex.printStackTrace();
for (Throwable t = ex.getCause(); t != null; t = t.getCause()) {
System.out.println("... Caused by ");
t.printStackTrace();
}
}
catch (java.io.IOException ex) {
System.out.println("An IOException occured whilst writing to the message buffer: " + ex);
}
return;
}
}
but right now i am getting following error
Exception in thread "main" java.lang.UnsatisfiedLinkError: no mqjbnd05 in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at com.ibm.mq.MQSESSION.loadLib(MQSESSION.java:872)
at com.ibm.mq.server.MQSESSION$1.run(MQSESSION.java:228)
at java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.server.MQSESSION.<clinit>(MQSESSION.java:222)
at com.ibm.mq.MQSESSIONServer.getMQSESSION(MQSESSIONServer.java:70)
at com.ibm.mq.MQSESSION.getSession(MQSESSION.java:492)
at com.ibm.mq.MQManagedConnectionJ11.<init>(MQManagedConnectionJ11.java:168)
at com.ibm.mq.MQBindingsManagedConnectionFactoryJ11._createManagedConnection(MQBindingsManagedConnectionFactoryJ11.java:179)
at com.ibm.mq.MQBindingsManagedConnectionFactoryJ11.createManagedConnection(MQBindingsManagedConnectionFactoryJ11.java:215)
at com.ibm.mq.StoredManagedConnection.<init>(StoredManagedConnection.java:84)
at com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java:168)
at com.ibm.mq.MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueueManagerFactory.java:772)
at com.ibm.mq.MQQueueManagerFactory.procure(MQQueueManagerFactory.java:697)
at com.ibm.mq.MQQueueManagerFactory.constructQueueManager(MQQueueManagerFactory.java:657)
at com.ibm.mq.MQQueueManagerFactory.createQueueManager(MQQueueManagerFactory.java:153)
at com.ibm.mq.MQQueueManager.<init>(MQQueueManager.java:451)
at MQSample.main(MQSample.java:30)
when i refered about this everyone saying to put mqjbnd.dll in java.library i put that file in that path also still not working

IBM MQ CLIENT java.lang.UnsatisfiedLinkError: no mqjbnd05 in
java.library.path exception
Your title pretty much says it all. 'MQ Client' generally means that the queue manager is REMOTE to where you are running your application. But 'no mqjbnd05' means that you are attempting to connect to the queue manager in bindings mode, queue manager is running on the same server as your queue manager.
99% of the time an application gets that errors is because the application and queue manager are running on separate servers and the application is not specifying: channel name, hostname/IP address & port #.
Note: An application can connect in 2 ways to a queue manager:
(1) client mode - meaning the application and queue manager are running on separate servers and the application is not specifying: channel name, hostname/IP address & port #.
(2) bindings mode - meaning the application and queue manager are running on the same servers (no networking information is specified).
Note: Don't use MQEnvironment class but rather put the connection information in a Hashtable and pass it to the MQQueueManager class. MQEnvironment is NOT thread safe.
Here's a working sample MQ application that will connect (client mode) to a remote queue manager:
import java.io.IOException;
import java.util.Hashtable;
import com.ibm.mq.*;
import com.ibm.mq.constants.CMQC;
/**
* Java class to connect to MQ. Post and Retrieve messages.
*
* Sample Command Line Parameters
* -h 127.0.0.1 -p 1414 -c TEST.CHL -m MQA1 -q TEST.Q1 -u userid -x password
*/
public class MQClientTest
{
private Hashtable<String, String> params = null;
private Hashtable<String, Object> mqht = null;
private String qManager;
private String inputQName;
/**
* The constructor
*/
public MQClientTest()
{
super();
}
/**
* Make sure the required parameters are present.
*
* #return true/false
*/
private boolean allParamsPresent()
{
boolean b = params.containsKey("-h") && params.containsKey("-p") &&
params.containsKey("-c") && params.containsKey("-m") &&
params.containsKey("-u") && params.containsKey("-x") &&
params.containsKey("-q");
if (b)
{
try
{
Integer.parseInt((String) params.get("-p"));
}
catch (NumberFormatException e)
{
b = false;
}
}
return b;
}
/**
* Extract the command-line parameters and initialize the MQ variables.
*
* #param args
* #throws IllegalArgumentException
*/
private void init(String[] args) throws IllegalArgumentException
{
params = new Hashtable<String, String>();
if (args.length > 0 && (args.length % 2) == 0)
{
for (int i = 0; i < args.length; i += 2)
{
params.put(args[i], args[i + 1]);
}
}
else
{
throw new IllegalArgumentException();
}
if (allParamsPresent())
{
qManager = (String) params.get("-m");
inputQName = (String) params.get("-q");
mqht = new Hashtable<String, Object>();
mqht.put(CMQC.CHANNEL_PROPERTY, params.get("-c"));
mqht.put(CMQC.HOST_NAME_PROPERTY, params.get("-h"));
try
{
mqht.put(CMQC.PORT_PROPERTY, new Integer(params.get("-p")));
}
catch (NumberFormatException e)
{
mqht.put(CMQC.PORT_PROPERTY, new Integer(1414));
}
mqht.put(CMQC.USER_ID_PROPERTY, params.get("-u"));
mqht.put(CMQC.PASSWORD_PROPERTY, params.get("-x"));
// I don't want to see MQ exceptions at the console.
MQException.log = null;
}
else
{
throw new IllegalArgumentException();
}
}
/**
* Method to put then get a message to/from a queue.
*/
public void putAndGetMessage()
{
MQQueueManager qMgr = null;
MQQueue queue = null;
MQMessage putMessage = null;
MQMessage getMessage = null;
int openOptions = CMQC.MQOO_INPUT_AS_Q_DEF | CMQC.MQOO_OUTPUT + CMQC.MQOO_FAIL_IF_QUIESCING;
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = CMQC.MQGMO_NO_WAIT + CMQC.MQGMO_FAIL_IF_QUIESCING;
MQPutMessageOptions pmo = new MQPutMessageOptions();
pmo.options = CMQC.MQPMO_FAIL_IF_QUIESCING;
String msg = "Hello World, WelCome to MQ.";
try
{
qMgr = new MQQueueManager(qManager, mqht);
queue = qMgr.accessQueue(inputQName, openOptions);
putMessage = new MQMessage();
putMessage.writeUTF(msg);
// put the message on the queue
queue.put(putMessage, pmo);
System.out.println("Message is put on MQ.");
// get message from MQ.
getMessage = new MQMessage();
// assign message id to get message.
getMessage.messageId = putMessage.messageId;
/*
* Tell the queue manager that we want a message with a specific MsgID.
*/
gmo.matchOptions = CMQC.MQMO_MATCH_MSG_ID;
// get message options.
queue.get(getMessage, gmo);
String retreivedMsg = getMessage.readUTF();
System.out.println("Message got from MQ: " + retreivedMsg);
}
catch (MQException e)
{
System.err.println("CC=" + e.completionCode + " : RC=" + e.reasonCode);
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
try
{
if (queue != null)
queue.close();
}
catch (MQException e)
{
System.err.println("MQCLOSE CC=" + e.completionCode + " : RC="
+ e.reasonCode);
}
try
{
if (qMgr != null)
qMgr.disconnect();
}
catch (MQException e2)
{
System.err.println("MQDISC CC=" + e2.completionCode + " : RC="
+ e2.reasonCode);
}
}
}
public static void main(String[] args)
{
System.out.println("Processing Main...");
MQClientTest clientTest = new MQClientTest();
try
{
// initialize MQ.
clientTest.init(args);
// put and retrieve message from MQ.
clientTest.putAndGetMessage();
}
catch (IllegalArgumentException e)
{
System.out.println("Usage: java MQClientTest -h host -p port -c channel -m QueueManagerName -q QueueName -u userid -x password");
System.exit(1);
}
System.out.println("Done!");
}
}

There is nothing in the question to indicate which version of MQ client is being used, how it was installed, whether the IBM-provided code works, or whether the environment was set correctly. That makes this more of a "how do I configure and test MQ client?" kind of question so I'll answer it that way.
Back-level MQ clients were always installed using IBM's full-client install media. Newer clients can be installed using IBM's Java-only install media. It is always recommended to use one of these methods.
The one thing to not do (which coincidentally is the thing done most often) is to simply grab the jar files from the MQ Server installation. The reasons for this include:
The IBM installer lays down a known-good set of files.
Maintenance can be applied to an installation made using IBM's installer.
When using IBM's install methods, things like trace directories and the location of the MQClient.ini file are predictable.
So first thing is to make sure you are running from the latest IBM-provided full client or Java-only install media. Alternatively, install IBM MQ Advanced for Developers which delivers a full MQ install to the desktop, including all the client support. MQ Advanced for Developers is free for individual use.
Prior to launching code, set the environment. See:
Environment variables relevant to IBM MQ classes for Java
Environment variables used by IBM MQ classes for JMS
Per the docs:
On Windows, all the environment variables are set automatically during
installation. On any other platform, you must set them yourself. On a
UNIX system, you can use the script setjmsenv (if you are using a
32-bit JVM) or setjmsenv64 (if you are using a 64-bit JVM) to set the
environment variables. On AIX, HP-UX, Linux, and Solaris, these
scripts are in the MQ_INSTALLATION_PATH/java/bin directory.
IBM provides lots of sample code. On *nix systems this is in /opt/mqm/samp/. On windows it's in the [MQ install directory]\tools. If the full client is installed try the compiled C code first, like amqsgetc. This establishes whether basic connectivity is in place. Once you know the channel connectivity works, try the Java or JMS samples.
Getting back to the original post, before we can help we'd need to know which of the above steps had already been completed and something about the configuration. Otherwise what you get back generally leads you down a path to a setup in which configuration is almost guaranteed to be hosed: "Try adding this library here," or "try mucking about with your CLASSPATH like this..." Such trial-and-error approaches often work but
are unsupportable and lead to problems over time.

Mulitple clients with a server and sockets? [duplicate]

I did some different tutorials but nothing works, can someone see what i'm doing wrong?
private volatile boolean keepRunning = true;
public FileSharedServer() {
}
#Override
public void run() {
try {
System.out.println("Binding to Port " + PORT + "...");
// Bind to PORT used by clients to request a socket connection to
// this server.
ServerSocket serverSocket = new ServerSocket(PORT);
System.out.println("\tBound.");
System.out.println("Waiting for Client...");
socket = serverSocket.accept();
System.out.println("\tClient Connected.\n\n");
if (socket.isConnected()) {
System.out.println("Writing to client serverId " + serverId
+ ".");
// Write the serverId plus the END character to the client thru
// the socket
// outStream
socket.getOutputStream().write(serverId.getBytes());
socket.getOutputStream().write(END);
}
while (keepRunning) {
System.out.println("Ready");
// Receive a command form the client
int command = socket.getInputStream().read();
// disconnect if class closes connection
if (command == -1) {
break;
}
System.out.println("Received command '" + (char) command + "'");
// decide what to do.
switch (command) {
case LIST_FILES:
sendFileList();
break;
case SEND_FILE:
sendFile();
break;
default:
break;
}
}
} catch (IOException e) {
System.out.println(e.getMessage());
} finally {
// Do not close the socket here because the readFromClient() method
// still needs to
// be called.
if (socket != null && !socket.isClosed()) {
try {
System.out.println("Closing socket.");
socket.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/**
* This method sends the names of all of the files in the share directory.
*
* #throws IOException
*/
private void sendFileList() throws IOException {
File serverFilesDir = new File("serverFiles/");
if (!serverFilesDir.exists() || serverFilesDir.isFile()) {
System.out.println("'serverfiles' is not an existing directory");
throw new IOException("'serverfiles' directory does not exist.");
}
File[] files = serverFilesDir.listFiles();
for (File file : files) {
socket.getOutputStream().write(file.getName().getBytes());
// Even the last one must end with END and then finally with
// END_OF_LIST.
socket.getOutputStream().write(END);
}
socket.getOutputStream().write(END_OF_LIST);
}
/**
* this methods sends a particular file to the client.
*
* #throws IOException
*/
private void sendFile() throws IOException {
StringBuilder filename = new StringBuilder();
int character = -1;
while ((character = socket.getInputStream().read()) > -1
&& character != END && (char) character != END_OF_LIST) {
filename.append((char) character);
}
System.out.println(filename);
File file = new File(System.getProperty("user.dir")
+ System.getProperty("file.separator") + "serverfiles",
filename.toString());
String totalLength = String.valueOf(file.length());
socket.getOutputStream().write(totalLength.getBytes());
socket.getOutputStream().write(END);
FileInputStream fileInputStream = new FileInputStream(file);
int nbrBytesRead = 0;
byte[] buffer = new byte[1024 * 2];
try {
while ((nbrBytesRead = fileInputStream.read(buffer)) > -1) {
socket.getOutputStream().write(buffer, 0, nbrBytesRead);
}
} finally {
fileInputStream.close();
}
}
public static void main(String[] args) throws InterruptedException {
// Create the server which waits for a client to request a connection.
FileSharedServer server = new FileSharedServer();
System.out.println("new thread");
Thread thread = new Thread(server);
thread.start();
}
}
Do I need another class or just a couple of lines in main? on the very bottom.
It's over a wifi network and all I need is two clients at once, or more :)

The problem here is that you are running only a single thread on the server. This thread accepts a connection, writes the server ID to the connection, then reads from the connection. The thread then continues to read from the connection until a -1 is received, at which point the thread exits. At no point does the thread try to accept a second connection; ServerSocket.accept() is called only once. As a result, you can only handle one client.
What you need is to split your class into two separate classes. In the first class, the run() method goes into a loop, calling ServerSocket.accept(), and each time that method returns a socket, creates an instance of the second class, hands it the socket, and starts it, after which it loops back to the ServerSocket.accept() call.
The second class is almost identical to the class you've already written, except that it doesn't contain the ServerSocket.accept() call. Instead, socket is a member variable which is initialized, by the first class, before it is started. It can do all the handling of the socket, sending the server ID, receiving and handling commands, etc., just as your existing code does.

parallel file downloads using nio without creating threads per file download

I have tried a program which download files parallely using java.nio by creating a thread per file download.
package com.java.tftp.nio;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
import java.nio.ByteBuffer;
import java.nio.channels.DatagramChannel;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
/**
* This class is used to download files concurrently from tftp server by
* configuring the filenames, no of files.
*
* #author SHRIRAM
*
*/
public class TFTP_NIO_Client {
/**
* destination folder
* */
private String destinationFolder;
/**
* list of files names to download
* */
private List<String> fileNames;
/**
* integer indicates the number of files to download concurrently
* */
private int noOfFilesToDownload;
public TFTP_NIO_Client(List<String> fileNames, String destinationFolder,
int noOfFilesToDownload) {
this.destinationFolder = destinationFolder;
this.fileNames = fileNames;
this.noOfFilesToDownload = noOfFilesToDownload;
initializeHandlers();
}
/**
* This method creates threads to register the channel to process download
* files concurrently.
*
* #param noOfFilesToDownload
* - no of files to download
*/
private void initializeHandlers() {
for (int i = 0; i < noOfFilesToDownload; i++) {
try {
Selector aSelector = Selector.open();
SelectorHandler theSelectionHandler = new SelectorHandler(
aSelector, fileNames.get(i));
theSelectionHandler.start();
} catch (IOException e) {
e.printStackTrace();
}
}
}
/**
* Setup RRQ/WRQ packet Packet : | Opcode | FileName | 0 | mode | 0 |
* Filename -> Filename in array of bytes. 0 -> indicates end of file mode
* -> string in byte array 'netascii' or 'octet'
*
* #param aOpcode
* #param aMode
* #param aFileName
* #throws IOException
*/
private void sendRequest(int aOpcode, int aMode, String aFileName,
DatagramChannel aChannel, InetSocketAddress aAddress)
throws IOException {
// Read request packet
TFTPReadRequestPacket theRequestPacket = new TFTPReadRequestPacket();
aChannel.send(
theRequestPacket.constructReadRequestPacket(aFileName, aMode),
aAddress);
}
/**
* sends TFTP ACK Packet Packet : | opcode | Block# | opcode -> 4 -> 2 bytes
* Block -> block number -> 2bytes
*
* #param aBlock
*/
private ByteBuffer sendAckPacket(int aBlockNumber) {
// acknowledge packet
TFTPAckPacket theAckPacket = new TFTPAckPacket();
return theAckPacket.getTFTPAckPacket(aBlockNumber);
}
/**
* This class is used to handle concurrent downloads from the server.
*
* */
public class SelectorHandler extends Thread {
private Selector selector;
private String fileName;
/**
* flag to indicate the file completion.
* */
private boolean isFileReadFinished = false;
public SelectorHandler(Selector aSelector, String aFileName)
throws IOException {
this.selector = aSelector;
this.fileName = aFileName;
registerChannel();
}
private void registerChannel() throws IOException {
DatagramChannel theChannel = DatagramChannel.open();
theChannel.configureBlocking(false);
selector.wakeup();
theChannel.register(selector, SelectionKey.OP_READ);
sendRequest(Constants.OP_READ, Constants.ASCII_MODE, fileName,
theChannel, new InetSocketAddress(Constants.HOST,
Constants.TFTP_PORT));
}
#Override
public void run() {
process();
}
private void process() {
System.out.println("Download started for " + fileName + " ");
File theFile = new File(destinationFolder
+ fileName.substring(fileName.lastIndexOf("/")));
FileOutputStream theFout = null;
try {
theFout = new FileOutputStream(theFile);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
while (!isFileReadFinished) {
try {
if (selector.select() == 0) {
try {
// sleep 2sec was introduced because selector is
// thread safe but keys are not thread safe
Thread.sleep(2000);
} catch (InterruptedException e) {
continue;
}
continue;
}
Set<SelectionKey> theSet = selector.selectedKeys();
Iterator<SelectionKey> theSelectedKeys = theSet.iterator();
synchronized (theSelectedKeys) {
while (theSelectedKeys.hasNext()) {
SelectionKey theKey = theSelectedKeys.next();
theSelectedKeys.remove();
if (theKey.isReadable()) {
isFileReadFinished = read(theKey, theFout,
fileName);
if (!isFileReadFinished) {
theKey.interestOps(SelectionKey.OP_READ);
}
} else if (theKey.isWritable()) {
// there is no implementation for file write to
// server.
theKey.interestOps(SelectionKey.OP_READ);
}
}
}
} catch (IOException ie) {
ie.printStackTrace();
}
}
System.out.println("Download finished for " + fileName);
try {
if (selector.isOpen()) {
selector.close();
}
if (theFout != null) {
theFout.close();
}
} catch (IOException ie) {
}
}
}
/**
* #param aKey
* registered key for the selector
* #param aOutStream
* - file output stream to write the file contents.
* #return boolean
* #throws IOException
*/
private boolean read(SelectionKey aKey, OutputStream aOutStream,
String aFileName) throws IOException {
DatagramChannel theChannel = (DatagramChannel) aKey.channel();
// data packet
TFTPDataPacket theDataPacket = new TFTPDataPacket();
ByteBuffer theReceivedBuffer = theDataPacket.constructTFTPDataPacket();
SocketAddress theSocketAddress = theChannel.receive(theReceivedBuffer);
theReceivedBuffer.flip();
byte[] theBuffer = theReceivedBuffer.array();
byte[] theDataBuffer = theDataPacket.getDataBlock();
if (theDataPacket.getOpCode() == Constants.OP_DATA) {
int theLimit = theDataPacket.getLimit();
// checks the limit of the buffer because a packet with data less
// than 512 bytes of content signals that it is the last packet in
// transmission for this particular file
if (theLimit != Constants.MAX_BUFFER_SIZE
&& theLimit < Constants.MAX_BUFFER_SIZE) {
byte[] theLastBlock = new byte[theLimit];
System.arraycopy(theBuffer, 0, theLastBlock, 0, theLimit);
// writes the lastblock
aOutStream.write(theLastBlock);
// sends an acknowledgment to the server using TFTP packet
// block number
theChannel
.send(sendAckPacket((((theBuffer[2] & 0xff) << 8) | (theBuffer[3] & 0xff))),
theSocketAddress);
if (theChannel.isOpen()) {
theChannel.close();
}
return true;
} else {
aOutStream.write(theDataBuffer);
// sends an acknowledgment to the server using TFTP packet
// block number
theChannel
.send(sendAckPacket((((theBuffer[2] & 0xff) << 8) | (theBuffer[3] & 0xff))),
theSocketAddress);
return false;
}
} else if (Integer.valueOf(theBuffer[1]) == Constants.OP_ERROR) {
System.out.println("File : " + aFileName + " not found ");
handleError(theReceivedBuffer);
}
return false;
}
/**
* This method handles the error packet received from Server.
*
* #param aBuffer
*/
private void handleError(ByteBuffer aBuffer) {
// Error packet
new TFTPErrorPacket(aBuffer);
}
}
Is it possible to download multiple files in parallel using java.nio by not creating a thread per file download? If yes can anybody suggest a solution to proceed further.

I would provide an approach to achieve what you are aiming for :
Let L the list of files to be downloaded.
Create a Map M which will hold the mapping of File name to be downloaded and the corresponding Selector instance.
For each file F in L
Get Selector SK from M corresponding to F
Process the state of the Selector by checking for any of the events being ready.
If processing is complete then set the Selector corresponding to F as null. This will help in identifying files
whose
processing is completed.Alternatively, you can remove F from
L; so that the next time you are looping you only process files that are not yet completely downloaded.
The above being said, I am curious to understand why you would want to attempt such a feat? If the thought process behind this requirement is to reduce the number of threads to 1 then it is not correct. Remember, you would end up really taxing the single thread running and for sure your throughput would not necessarily be optimal since the single thread would be dealing with both network as well as disk I/O. Also, consider the case of encountering an exception while writing one of the several files to the disk - you would end up aborting the transfer for all the files; something I am sure you do not want.
A better and more scalable approach would be to poll selectors on a single thread, but hand off any I/O activity to a worker thread. A better approach still would be to read the techniques presented in Doug Lea's paper and implement them. In fact Netty library already implements this pattern and is widely used in production.

Java blocking on multiple blockers

I'm not a very experienced Java programmer, so forgive me if this is a bit of a newbie question.
I'm designing a system that consists broadly of 3 modules. A client, a server and an application. The idea is the client sends a message to the server. The server triggers a use case in the application. The result of the use case is returned to the server, and the server sends the results to the client. I opted for this architecture because I'm expecting to need to support multiple clients at once, I want to be able to reuse the server module in other applications, I want to keep the code responsible for managing client connections as uncoupled from the code that implements the domain logic as possible, and because of the opportunity to learn some more advanced java.
I'm planning to tie the various modules together with queues. The client is simple enough. Issue a message and block until a response arrives (it may be oversimplifying but it's a reasonable model for now). The application is equally not a problem. It blocks on its input queue, executes a use case when it receives a valid message and pushes the results to an output queue. Having multiple clients makes things a bit more tricky but still within my grasp with my experience level. The server maintains threads for every open connection, and the server outbound/application inbound queue is synchronised, so if 2 messages arrive at once the second thread will just have to wait a moment for the first thread to deliver its payload into the queue.
The problem is the part in the middle, the server, which will have to block on two independent things. The server is watching both the client, and the application's output queue (which serves as an input queue for the server). The server needs to block until either a message comes in from the client (which it then forwards to the application), or until the application completes a task and pushes the results into the application outbound/server inbound queue.
As far as I can tell, Java can only block on one thing.
Is it possible to have the server block until either the client sends a message or the server inbound queue ceases to be empty?
UPDATE:
I've had a bit of time to work on this, and have managed to pare the problem down to the bare minimum that illustrates the problem. There's a somewhat bulky code dump to follow, even with the trimming, so apologies for that. I'll try to break it up as much as possible.
This is the code for the Server:
public class Server implements Runnable {
private int listenPort = 0;
private ServerSocket serverSocket = null;
private BlockingQueue<Message> upstreamMessaes = null;
private BlockingQueue<Message> downstreamMessages = null;
private Map<Integer, Session> sessions = new ConcurrentHashMap ();
private AtomicInteger lastId = new AtomicInteger ();
/**
* Start listening for clients to process
*
* #throws IOException
*/
#Override
public void run () {
int newSessionId;
Session newSession;
Thread newThread;
System.out.println (this.getClass () + " running");
// Client listen loop
while (true) {
newSessionId = this.lastId.incrementAndGet ();
try {
newSession = new Session (this, newSessionId);
newThread = new Thread (newSession);
newThread.setName ("ServerSession_" + newSessionId);
this.sessions.put (newSessionId, newSession);
newThread.start ();
} catch (IOException ex) {
Logger.getLogger (Server.class.getName ()).log (Level.SEVERE, null, ex);
}
}
}
/**
* Accept a connection from a new client
*
* #return The accepted Socket
* #throws IOException
*/
public Socket accept () throws IOException {
return this.getSocket().accept ();
}
/**
* Delete the specified Session
*
* #param sessionId ID of the Session to remove
*/
public void deleteSession (int sessionId) {
this.sessions.remove (sessionId);
}
/**
* Forward an incoming message from the Client to the application
*
* #param msg
* #return
* #throws InterruptedException
*/
public Server messageFromClient (Message msg) throws InterruptedException {
this.upstreamMessaes.put (msg);
return this;
}
/**
* Set the port to listen to
*
* We can only use ports in the range 1024-65535 (ports below 1024 are
* reserved for common protocols such as HTTP and ports above 65535 don't
* exist)
*
* #param listenPort
* #return Returns itself so further methods can be called
* #throws IllegalArgumentException
*/
public final Server setListenPort (int listenPort) throws IllegalArgumentException {
if ((listenPort > 1023) && (listenPort <= 65535)) {
this.listenPort = listenPort;
} else {
throw new IllegalArgumentException ("Port number " + listenPort + " not valid");
}
return this;
}
/**
* Get the server socket, initialize it if it isn't already started.
*
* #return The object's ServerSocket
* #throws IOException
*/
private ServerSocket getSocket () throws IOException {
if (null == this.serverSocket) {
this.serverSocket = new ServerSocket (this.listenPort);
}
return this.serverSocket;
}
/**
* Instantiate the server
*
* #param listenPort
* #throws IllegalArgumentException
*/
public Server ( int listenPort,
BlockingQueue<Message> incomingMessages,
BlockingQueue<Message> outgoingMessages) throws IllegalArgumentException {
this.setListenPort (listenPort);
this.upstreamMessaes = incomingMessages;
this.downstreamMessages = outgoingMessages;
System.out.println (this.getClass () + " created");
System.out.println ("Listening on port " + listenPort);
}
}
I believe the following method belongs in the Server but is currently commented out.
/**
* Notify a Session of a message for it
*
* #param sessionMessage
*/
public void notifySession () throws InterruptedException, IOException {
Message sessionMessage = this.downstreamMessages.take ();
Session targetSession = this.sessions.get (sessionMessage.getClientID ());
targetSession.waitForServer (sessionMessage);
}
This is my Session class
public class Session implements Runnable {
private Socket clientSocket = null;
private OutputStreamWriter streamWriter = null;
private StringBuffer outputBuffer = null;
private Server server = null;
private int sessionId = 0;
/**
* Session main loop
*/
#Override
public void run () {
StringBuffer inputBuffer = new StringBuffer ();
BufferedReader inReader;
try {
// Connect message
this.sendMessageToClient ("Hello, you are client " + this.getId ());
inReader = new BufferedReader (new InputStreamReader (this.clientSocket.getInputStream (), "UTF8"));
do {
// Parse whatever was in the input buffer
inputBuffer.delete (0, inputBuffer.length ());
inputBuffer.append (inReader.readLine ());
System.out.println ("Input message was: " + inputBuffer);
this.server.messageFromClient (new Message (this.sessionId, inputBuffer.toString ()));
} while (!"QUIT".equals (inputBuffer.toString ()));
// Disconnect message
this.sendMessageToClient ("Goodbye, client " + this.getId ());
} catch (IOException | InterruptedException e) {
Logger.getLogger (Session.class.getName ()).log (Level.SEVERE, null, e);
} finally {
this.terminate ();
this.server.deleteSession (this.getId ());
}
}
/**
*
* #param msg
* #return
* #throws IOException
*/
public Session waitForServer (Message msg) throws IOException {
// Generate a response for the input
String output = this.buildResponse (msg.getPayload ()).toString ();
System.out.println ("Output message will be: " + output);
// Output to client
this.sendMessageToClient (output);
return this;
}
/**
*
* #param request
* #return
*/
private StringBuffer buildResponse (CharSequence request) {
StringBuffer ob = this.outputBuffer;
ob.delete (0, this.outputBuffer.length ());
ob.append ("Server repsonded at ")
.append (new java.util.Date ().toString () )
.append (" (You said '" )
.append (request)
.append ("')");
return this.outputBuffer;
}
/**
* Send the given message to the client
*
* #param message
* #throws IOException
*/
private void sendMessageToClient (CharSequence message) throws IOException {
// Output to client
OutputStreamWriter osw = this.getStreamWriter ();
osw.write ((String) message);
osw.write ("\r\n");
osw.flush ();
}
/**
* Get an output stream writer, initialize it if it's not active
*
* #return A configured OutputStreamWriter object
* #throws IOException
*/
private OutputStreamWriter getStreamWriter () throws IOException {
if (null == this.streamWriter) {
BufferedOutputStream os = new BufferedOutputStream (this.clientSocket.getOutputStream ());
this.streamWriter = new OutputStreamWriter (os, "UTF8");
}
return this.streamWriter;
}
/**
* Terminate the client connection
*/
private void terminate () {
try {
this.streamWriter = null;
this.clientSocket.close ();
} catch (IOException e) {
Logger.getLogger (Session.class.getName ()).log (Level.SEVERE, null, e);
}
}
/**
* Get this Session's ID
*
* #return The ID of this session
*/
public int getId () {
return this.sessionId;
}
/**
* Session constructor
*
* #param owner The Server object that owns this session
* #param sessionId The unique ID this session will be given
* #throws IOException
*/
public Session (Server owner, int sessionId) throws IOException {
System.out.println ("Class " + this.getClass () + " created");
this.server = owner;
this.sessionId = sessionId;
this.clientSocket = this.server.accept ();
System.out.println ("Session ID is " + this.sessionId);
}
}
The test application is fairly basic, it just echoes a modified version of the original request message back. The real application will do work on receipt of a message and returning a meaningful response to the Server.
public class TestApp implements Runnable {
private BlockingQueue <Message> inputMessages, outputMessages;
#Override
public void run () {
Message lastMessage;
StringBuilder returnMessage = new StringBuilder ();
while (true) {
try {
lastMessage = this.inputMessages.take ();
// Construct a response
returnMessage.delete (0, returnMessage.length ());
returnMessage.append ("Server repsonded at ")
.append (new java.util.Date ().toString () )
.append (" (You said '" )
.append (lastMessage.getPayload ())
.append ("')");
// Pretend we're doing some work that takes a while
Thread.sleep (1000);
this.outputMessages.put (new Message (lastMessage.getClientID (), lastMessage.toString ()));
} catch (InterruptedException ex) {
Logger.getLogger (TestApp.class.getName ()).log (Level.SEVERE, null, ex);
}
}
}
/**
* Initialize the application
*
* #param inputMessages Where input messages come from
* #param outputMessages Where output messages go to
*/
public TestApp (BlockingQueue<Message> inputMessages, BlockingQueue<Message> outputMessages) {
this.inputMessages = inputMessages;
this.outputMessages = outputMessages;
System.out.println (this.getClass () + " created");
}
}
The Message class is extremely simple and just consists of an originating client ID and a payload string, so I've left it out.
Finally the main class looks like this.
public class Runner {
/**
*
* #param args The first argument is the port to listen on.
* #throws Exception
*/
public static void main (String[] args) throws Exception {
BlockingQueue<Message> clientBuffer = new LinkedBlockingQueue ();
BlockingQueue<Message> appBuffer = new LinkedBlockingQueue ();
TestApp appInstance = new TestApp (clientBuffer, appBuffer);
Server serverInstance = new Server (Integer.parseInt (args [0]), clientBuffer, appBuffer);
Thread appThread = new Thread (appInstance);
Thread serverThread = new Thread (serverInstance);
appThread.setName("Application");
serverThread.setName ("Server");
appThread.start ();
serverThread.start ();
appThread.join ();
serverThread.join ();
System.exit (0);
}
}
While the real application will be more complex the TestApp illustrates the basic pattern of use. It blocks on its input queue until there's something there, processes it, then pushes the result onto its output queue.
Session classes manage a live connection between a particular client and the server. It takes input from the client and converts it to Message objects, and it takes Message objects from the Server and converts them to output to send to the client.
The Server listens for new incoming connections and sets up a Session object for each incoming connection it has. When a Session passes it a Message, it puts it into its upstream queue for the application to deal with.
The difficulty I'm having is getting return messages to travel back down from the TestApp to the various clients. When a message from a client comes in, the Session generates a Message and sends it to the Server, which then puts it into its upstream queue, which is also the input queue for the TestApp. In response, the TestApp generates a response Message and puts it into the output queue, which is also the downstream queue for the Server.
This means that Sessions need to wait for two unrelated events. They should block until
Input arrives from the client (the BufferedReader on the client socket has input to process),
OR a message is sent to it by the Server (the server calls the WaitForServer () method on the session)
As for the Server itself, it also has to wait for two unrelated events.
a Session calls messageFromClient() with a message to pass to the TestApp,
OR the TestApp pushes a message onto the output/downstream queue to be passed onto a Session.
What on the face of it looked like a simple task to achieve is proving a lot more difficult than I first imagined. I expect I'm overlooking something obvious, as I'm still quite new to concurrent programming, but if you can help point out where I'm going wrong I'd appreciate instruction.

Because your implementation is using methods to pass data between client-session-server, you've actually already solved your immediate problem. However, this may not have been your intention. Here's what's happening:
Session's run method is running in its own thread, blocking on the socket. When the server calls waitForServer, this method immediately executes in the server's thread - in Java, if a thread calls a method then that method executes in that thread, and so in this case the Session did not need to unblock. In order to create the problem you are trying to solve, you would need to remove the waitForServer method and replace it with a BlockingQueue messagesFromServer queue - then the Server would place messages in this queue and Session would would need to block on it, resulting in Session needing to block on two different objects (the socket and the queue).
Assuming that you switch to the implementation where the Session will need to block on two objects, I think you can solve this with a hybrid of the two approaches I described in the comments:
Each Session's socket will need a thread to block on it - I don't see any way around this, unless you're willing to replace this with a fixed thread pool (say, 4 threads) that poll the sockets and sleep for a few dozen milliseconds if there's nothing to read from them.
You can manage all Server -> Sessions traffic with a single queue and a single thread that blocks on it - the Server includes the Session "address" in its payload so that the thread blocking on it knows what to do with the message. If you find that this doesn't scale when you have a lot of sessions, then you can always increase the thread/queue count, e.g. with 32 sessions you can have 4 threads/queues, 8 sessions per thread/queue.

I may have misunderstood but it seems that where you have the code "listening" for a message, you should be able to use a simple OR statement to solve this.
One other thing that might be useful is to add a unique id to every client so you can tell which client the message is intended for.