I need to managed a pool of agents from my application. All are written in Java but the agents need to run in their own JVM. I wrote a proof of concept that starts the subprocesses and uses the stdout/stdin to send commands and keep-alive information. I also open a socket connection for data transfer.
I guess that some connection pooling libraries should be able to help in the management of the agents.
What about the communication between the agents and the main process ? Using TCP with XML messages (JAXB) is not really as reliable or convenient as I would like. Any suggestion for a better library to assist here ?
I could very well write what I need myself but I'm sure other people have done that way better already.
For messaging could try something like ZeroMQ, it's a messaging tool and has local transports for communicationg between processes, then you could just serialised objects between the process.
The alternative is to go back to traditionally rmi, probably the simplest.
You could try Hessian:
http://hessian.caucho.com/
or Preon:
http://preon.sourceforge.net/
I've actually found two ways that would have been of great help when I developed this:
WebSockets. I used simple sockets but then I needed to reinvent signaling to check who's sending and done sending things. I used a line-based approach but it's really ugly. WebSockets offer the message-based communication and that's great.
Hazelcast. This is a "distributed system" and offers great things like distributed executors (I schedule a message to be sent in the app server and let any available out-of-jvm agent handle it, atomically), shared and thread safe hashmaps (to keep track of who is running) etc. Many of the similar tools I had seen were either in native code (like ZeroMQ btw) or with per-CPU licenses and such. Hazelcast is community edition and can be bundled into my apps.
Actually, I had started using vert.x to handle websocket-based communication and realized it was itself using hazelcast.
Related
Firstly Cheers to all PROGRAMMERS [ Today = Programmers day :) ]
Secondly,
I'm working on a project where the specifications require using a server as a front end and an application in the back end. The project is an advanced smart home system. The server will handle commands coming from the client through the internet (let's say like a remote control from outside the house) and send them (through a channel of communication) to the application (planning on using JAVA application) which will handle the main logic like controlling hardware stuff (lights ...) , reading from a microphone (local mic) and accessing a database to act as a speech recognition system (offline).
Now I'm still in the planning phase and I'm not sure which technologies are the best for this project. I'm thinking to use Node.js or Apache as the server and a JAVA application as the back end and any SQL database for the application's SRS.
I hope this illustration demonstrates clearly how the system works:
The main question is:
What is the best way to make the Java application communicate with the server (communication channel [must be bidirectional]) ?
and Do you recommend a specific server other than the mentioned ones for this job ?
What crossed my mind so far:
1- JSP and servlets (making the server is the application too). But I don't want a server to handle the offline stuff and I'm not sure if java servlets can access hardware interface. I also want the server to be separate from making critical decisions (different layer for security reasons and since it won't be used as frequently as the local [offline] system).
2- Communication channel :
A- A shared file, but it's a bad idea since I don't want the application to check if the file contents changed (command received) or not from time to time (excessive operations).
B- A an inter-process-communication through a port (socket communication) seems the best solution but I don't know how that would turn in terms of operation cost and communication errors.
OS used : Linux Raspbian
EDIT:
I'm sure ZMQ+Apache is good enough for this task, but how is it in comparison to WebServices (like SOAP) ? Would WebServices be a better solution in terms of standard implementation and security ?
All related suggestions are welcomed, TQ
ZeroMQ is great for internal communications, or any other similar communication solutions.
For specifically your case, I can see that ZeroMQ would be a best fit.
Reasons:
You offline server have to be agnostic to web solution.
Communication can be reliable and bi-directional, possibly another patterns like (pub>sub, req<>res, etc).
Restarting any of sides would not require to restart the sockets (connection) on other side, as messages are queued.
Possibility to scale not just on same hardware, but as well to local area network or even through internet.
Big community of support. It might look a bit hard to get into, but in reality it is dead simple, just go to examples and once concept understood - it is very easy and neat to work with.
ZeroMQ has lots drivers for most popular languages, that includes Java and Node.js.
Considerations:
You need to think over packets and data will be sent. So some popular data protocols like XML or JSON is good way of thinking.
Responsibilities over different services - make sure they are not dependant on each other too much. Or if main offline server - is a core of system, make sure it does not depend on web facing service, so that web face can be removed/replaced/improved etc.
Few more points to think about:
Why Java, and what about modular approach? For example if you want to expand and scale - add more sensors into smart home solutions, then having one giant application would require to change it, it is harder to maintain as well as maintain different clients with own needs. Think modular way - some core functionality for offline stuff, but many aggregator processes that would talk to different sensors. This makes easier to support different setups and environments, as well maintain the system as a whole by improving independent components.
I'm developing a distributed application, and I need to connect a client Java based to a server C++ based. Both of them will need to send information to each other, but I need them to be able to do things while waiting for the information, and they don't know when they are gonna get new information, or send information.
How can I achieve this? Now I'm trying to implement a basic communication with Sockets, but I don't really get to communicate them. I have read that using sockets + threads is usually a good approach for client-server apps.
Could you please recommend me some web or book to read about this, or send me some example code to learn?
Do you think that i should use other approach, better than sockets? maybe a higher level library (i would need it for c++ and java) or a totally different way?
EDIT:
I will add some extra information.
What I would love to achieve is the following:
My C++ program has a main loop, where I would like to have a call like GetUpdatedDataFromRemoteDevice() where I read the new values of some numerical variables that previously got updated from the net (the socket, for example).
Eventually, the C++ program will need to send a message to the remote device, to tell him to send other kind of data, and after that, keep getting the updated values.
From the Java program (remote device) the application running is an interactive touchable screen, that cant get blocked by the network transmissions, because it must keep working for the user, so all the networking should be done in a separated thread.
That thread, should connect to the server, and when a button is pushed, start to send the data (4 changing numerical values) in a loop until another event happens.
It would be nice also to be easily re-connectable to the server.
ICE is a modern and good library for distributed applications:
many languages as C++ and Java
many platforms
GNU GPL
good performance
easy to use
First, you define the messages you want to exchange between server and client.
Then, you implement the C++ and Java source code to handle these messages.
More info at http://zeroc.com/ice.html
Have fun ;-)
EDIT: I have to use ACE in some projects. I can tell ACE is very old, maybe mature, but uses outdated C++ coding rules :-(
Therefore ACE is not as easy to use as STL or BOOST. Moreover, ACE is not really efficient... I prefer ICE ;-)
I don't know what your application is but robust client server socket programming is pretty hairy task to do properly. Hardware byte order, String encoding, Network errors, retries, duplicate messages, acks etc.. require lots of good design and careful programming. You need to get it work well as single-threaded before even thinking using multiple threads.
Unless you need instant notifications from server to client I suggest that you use HTTP as protocol between client and server. Client can poll server occasionally for new messages.
Anyway the problem has been solved multiple times already.
http://activemq.apache.org/
http://www.rabbitmq.com/devtools.html
http://www.cs.wustl.edu/~schmidt/ACE-overview.html
I did something of this sort once. In my case it was easier to connect my C++ app to a local Java app using JNI and then have the two Java apps talk to each other.
I have a piece of C library that I don't trust (in the sense that it might crash frequently). I am calling this from a Java process.
To prevent the crash in C library bringing the whole Java app. down, I figured it will be best if I spawn a dedicated java processes for this library, and let it interface with the Java app. through socket programming or RMI. Then, if a crash happens, I can just spawn another one and continue processing.
Is ProcessBuilder the way to go? Or are there any other easier ways?
Thanks!
Yes, hosting the native code in a separate Java process is the only way to protect your application from native code.
As for easier ways, just minor implementation differences. For example, not spawning the code from your Java application and wrapping the native code in a native wrapper that is configured to auto-start. This would simplify the solution, if you have knowledge of C and sockets. In this approach, RMI wouldn't be the best choice.
Even if you wrap the native code in Java, I still wouldn't pick RMI. I have run into networking problems with Windows on WANs. I would keep the communication simple if possible. If the data is too complicated, maybe a basic serialization library. There are a few choices if you go down the XML route. It's overkill, but you could also embed an http server and web services layer. I don't know your system requirements, bu
Recovery is going to create a variety of challenges. If it stops responding, do you just spawn another process...how many times are you willing to do that... Process management from Java, leaves a lot to be desired.
I don't know of an easier way.
For the interaction between the parent and the child, i wouldn't use RMI or sockets - i'd use the child's standard input and output streams, accessible through the Process object. This is simple, efficient, and private. You can use the streams exactly as you would socket streams, although without any considerations of identity, addresses, authentication, and so on. You can write a protocol yourself, or use something like Thrift or Protocol Buffers to build a protocol from entity definitions.
If performance isn't an issue and if there is a possibility of other applications hitting your "native" service, I'd go the RESTful or some other sort of web service oriented way. As far as re-spawning on crashes are concerned, as others have mentioned, just spawn the process as a service and you should be good to go.
If your application is the only entity which would be hitting this native service, then I'd prefer to go the RMI way as opposed to the pure socket way. IMO, RMI is a natural fit for inter-process communication (where the processes are Java processes). RMI has the concept of an "activatable" remote object which would be a natural fit given your requirements (auto-spawn on crash). Also, if using RMI, your application would speak with the native process through well defined Java interfaces rather than ad-hoc protocol contracts (which can be achieved using other high level solutions like web services but a real pain when it comes to raw sockets).
BTW, JFTR, we are using this strategy with our production app and it is working out quite well, YMMV. :-)
I need to establish a communication model between C++ layer and Java layer in my application. Initially, I planned to use SOAP with XML, but my clients are interested in setting up a database communication channel.
I am new to DB and not sure how to proceed. I would like to take your sincere suggestions on the implementation of communication in terms of objects between C++ and Java layer using database.
Thanks,
Geet
Database as communication? shudder
http://en.wikipedia.org/wiki/Database-as-IPC
This is an anti-pattern. Can you change your clients' minds?
Sockets are easier than a full blown SOAP interface.
If you have 2 different applications communicating, sockets is the way to go. If your C++ layer is more like a library, you could also use JNI (http://en.wikipedia.org/wiki/JNI, google for tutorials).
The choice of communication channel and blocking model is largely application dependent but sockets will probably work best if you don't need to worry about security. SSL/Mutual auth is your next step up.
I rolled my own, but I would use google protocol buffers if I had to do it all again.
http://code.google.com/p/protobuf/
They seem to capture much of what people wanted out of ASN1 (but not all messed up) and let you do what people often try to do with serializing java Properties.
We have a C++ application on Windows that starts a java process. These two apps need to communicate with each other (via snippets of xml).
What interprocess communication method would you choose, and why?
Methods on the table for us are: a shared file(s), pipes and sockets (although I think this has some security concerns). I'm open to other methods.
I'm not sure why you think socket-based communication would have security concerns (use SSL). It is often a very good approach as it is language agnostic, assuming that you have a well-defined communication protocol. Have a look at Google's protocol buffers, for example - they generate the required Java classes and streams.
In my experience, file systems (especially network file systems) are not well suited to such communication as they are not necessarily tuned for messaging (I've seen caching issues result in files being not picked up by the target process for example).
Another option is a messaging layer (AMQ or Tibco for example) although this will likely involve a greater administrative overhead (plus expertise) to set up.
Personally I would opt for a pure-socket approach because of its flexibility and simplicity. You will be in complete control.
I've used named pipes for communication between C# and a cross-platform c++ app and had nothing but good results. Barring that sockets is definitely the way to go.
Sockets are nice. They give you the ability to very easily create a blackbox testing layer around each component, as well as run each component on its own machine.
Security is definitely a concern, but there are a good range of options depending on how important it is. You can use SSL, custom handshaking, password protected logins and firewalls to help secure it.
Edit:
Not something I'd recommend, but there's also shared memory using JNI. Just thought I'd mention it because it's not on your list.
Ice is pretty cool :)