It is not entirely obvious how to go about using RProtoBuf for communicating between R and other languages (Java, in my case).
The RprotoBuf Developers developed something that is still here - https://r-forge.r-project.org/scm/viewvc.php/java/?root=rprotobuf, but it seems very outdated. I am not sure if this is the way to go. Here are two conversations between the authors of RProtoBuf that might help with understanding the code -
http://lists.r-forge.r-project.org/pipermail/rprotobuf-yada/2009-December/000116.html
http://lists.r-forge.r-project.org/pipermail/rprotobuf-yada/2009-December/000119.html
It seems that they started work with Java and then abandoned it in C++'s favour!
Is there anyone using R-RProtoBuf-Java combination? How do you do it? Is there a tutorial or example available?
My exposure to Java is very very limited. I want to use a few programs written in Java.
Edit :
To clarify, I suppose I want to see an example of an R rpc client being used with RProtobuf. Pointers towards Java RPC servers would be welcome.
Edit2 :
The first link actually points to some documentation generator code, as Dirk pointed out.
I don't know about RProtoBuf, but if you just want to call Java from R, then you might be interested in rJava.
It is not outdated and has examples (right on the main page, see the link).
We have recently published a preprint on arXiv of a JSS paper we wrote with more examples of using RProtoBuf, including sending RPC requests to remote web services. For more exposition of sharing data between R and other languages with RProtoBuf, see RProtoBuf: Efficient Cross-Language Data Serialization in R.
You can use RProtoBuf with any transport mechanism, as explained in the article -- You can save serialized protocol buffers to files to be read by other applications written in other languages, or you can send them over connections/sockets or other higher level RPC systems. Protocol Buffers are widely used in everything from Sony Playstations to large scale web services, but they do not include an RPC system -- you use them as your serialization format with whatever transport system you are already using.
Related
I'm looking for a way to implement RPC between Java and C. What are the options to do this?
Best Wishes
p.s I have web java application which is hosted on Glassfish server and C daemon. I need to directly call functions from bought sides.
The whole point of RPC is to let two opaque processes on different systems talk to each other over a network. The languages used are irrelevant, except that you have to learn the corresponding RPC libraries for both languages.
Google Protocol Buffers address some of the difficulties with serialization, and provide a "RPC service" abstraction. You'll need to implement the "remote" part -- sending the data across, etc. -- but it'll give you cross-language compatible serialization.
The Google implementation doesn't natively support C -- only C++ -- but it looks like C is one of the languages for which there is an add-on.
Another option is the open source "thrift" library (originally from facebook). It also supports generating local stubs in a multiple of languages. Though I suspect the protocol buffer library as suggested by #Louis Wasserman is higher quality than thrift.
There are couple of java libraries implementing oncrpc:
http://code.google.com/p/nio-jrpc/ and
http://sourceforge.net/apps/trac/remotetea/wiki/WikiStart.
We have two code bases, one written in C++ (MS VS 6) and another in Java (JDK 6).
Looking for creative ways to make the two talk to each other.
More Details:
Both applications are GUI applications.
Major rewrites or translations are not an option.
Communications needs to be two-way.
Try to avoid anything involving writing files to disk.
So far the options considered are:
zero MG
RPC
CORBA
JNI
Compiling Java to native code, and then linking
Essentially, apart from the last item, this boils down to a choice between various ways to achieve interprocess communication between a Java application and a C++ application. Still open to other creative suggestions!
If you have attempted this, or something similar before please chime in with your suggestions, lessons learnt, pitfalls to avoid, etc.
Someone will no doubt point out shortly, that there is no one correct answer to this question. I thought I would tap on the collective expertise of the SO community anyway, and hope to get many excellent answers.
Well, it depends on how tightly integrated you want these applications to be and how you see them evolving in the future. If you just want to communicate data between the two of them (e.g. you want one to be able to open a file written by the other, or read a stream directly from the other), then I would say that protocol buffers are your best bet. If you want the window rendered by one of these GUI apps to actually be embedded in a panel of the other GUI app, then you probably want to use the JNI approach. With the JNI approach, you can use SWIG to automate a great deal of it, though it is dangerously magical and comes with a number of caveats (e.g. it doesn't do so well with function overloading).
I strongly recommend against CORBA, RMI, and similarly remote-procedure-call implementations, mostly because, in my experience, they tend to be very heavy-weight and consume a lot of resources. If you do want something similar to RMI, I would recommend something lighter weight where you pass messages, but not actual objects (as is the case with RMI). For example, you could use protocol buffers as your message format, and then simply serialize these back and forth across normal sockets.
Kit Ho mentioned XML or JSON, but protocol buffers are significantly more efficient than either of those formats and also have notions of backwards-compatibility built directly into the definition language.
Use Jacob ( http://sourceforge.net/projects/jacob-project ), JCom ( http://sourceforge.net/projects/jcom ), or j-Interop ( http://j-interop.org ) and use COM for communication.
Since you're using Windows, I'd suggest using DDE (Dynamic Data Exchange). There's a Java library available from Java Parts.
Dont' know how much data and what type of data you wanna transfer and communicate.
But to simplify the way, I suggest using XML or Json based on HTTP protocol.
Since there are lots of library for both applications and you won't spend too much effort to implement and understand.
More, if you have additional applications to talk with, it is not hard since both tech. are cross-languages.
correct me if i am wrong
What is the "official" Java API for client/server or P2P communication? Java RMI? Some other networking API??
Is this official networking API the standard for both SE and EE?
I'm sure the answer is very context-specific, so let's take a look at a few instances:
You have 2 swing clients installed on 2 machines and connected to the same network (or the Internet), and you want either of them to send the other a primitive, such as the integer 4, or some POJO, like a "Widget" object
Same as #1 above, but between a Swing client and a fully-compliant Java EE back-end (implementing managed beans, app servers, the whole nine yards)
I don't have a specific application in mind, I'm just wondering what are the "norms" for client-client and client-server communication in the world of Java.
If being bound by Java isn't a problem, RMI is a pretty abstracted solution when it comes to the client and server solution "exchanging" data (especially when the data is Java classes which might be difficult/too much effort to represent as textual data). Just make sure your object implements Serializable and almost anything can be transmitted over the wire.
If this doesn't fit your bill and you want to drop down the raw networking stuff, the client-server socket framework Netty is a pretty good choice.
There's no such thing as the most official networking API in J2SE, all J2SE APIs are official in the sense they are supported by Sun (now Oracle).
That said, you should choose your API based on following criteria:
Do you (or your team) know how to use particular API;
How simple/complex is this API to use;
What throughput are you aiming for? For performance-sensitive applications you may be forced to use binary protocol. For the rest of cases, you can use text-based protocol.
For example, between two clients simple text-based protocol will suffice for passing POJOs, for example using Apache MINA or Google protocol buffers.
This will work between client and server as well.
Response to Zac's questions in comment:
Binary protocols performance gain comes from the fact you don't need to convert everything to text form and back -- you just can pass binary presentation of you application memory with minimal changes, like, in case of BSD Sockets API, converting from host byte-order to network byte-order. Unfortunately, I don't know details about how RMI/Java serialization processes objects, but I'm sure, it still much faster than passing all data in readable form;
Yes, MINA and protocol buffers have Java APIs. They just not part of Java SE bundle, you have to download them separately. By the way, MINA can use both binary and readable serialization, depending on how you use it.
You should define notion of 'good' somehow, for example, answering to questions I mentioned above. If you want to use objects over network, use RMI. If you don't, Netty or MINA will suffice, whatever you'll find easier to master.
For P2P, Sun at one point pushed JXTA pretty hard.
I wouldn't dare to use RMI for P2P communication.
rmi is pretty much the standard java to java protocol. it's built in and very simple to use. most j2ee backends also communicate using rmi, although that's not the only possibility.
J2SE the most common is probably RMI or raw sockets.
J2EE uses a messaging bus that everyone (servers and clients) subscribes to which is quite different from rmi style solutions (although at the lowest level an implementation may still rely on RMI). It helps automate redundancy and failover. If you need this functionality I believe it can be used in SE as well.
I haven't used J2EE for quite a while now, so this may have changed, but I doubt it. The messaging system was a core component of J2EE.
I know of at least one post which has same words like this. But this is not exactly same as that post. I'm trying to work a way to "share" data between a .NET and Java application. I'm not concerned about objects, but just plain strings if u like.
I have a .NET application capturing real-time data and a Java application which has capability to analyze and work on this data. I'm looking for ways to re-use this same java app without coding it entirely in .NET.
My problem is that the data is "fairly" REAL-Time (.NET), and so has to be the analysis (Java). I can live with microsecond delays but I can't afford one second delay. WebServices, Queues (as in Messaging Queues), RDBMS are some of the options I can think of. Is there any better way?
Or has anybody got some real performance numbers for the solutions I mentioned above to select one of them? And just to get started: RDBMSs' are not "THAT" good for concurrent (connections doing) insertion/updation/reading, at least with the crude way of doing DBMS stuff. (Deadlocks?)
What are "objects" if not a mechanism for describing "data"? But I digress - I suspect I would look at a TCP socket between the two. If the data is very basic, then fine - just write directly to the stream; if there is any complexity, perhaps use something like "protocol buffers" to provide an easy way of reading/writing dense data to a stream without having to write every last byte yourself.
I think microsecond delays are going to be a challenge for any approach here... will millisecond delays do?
For completeness:
Another possible is to use Named pipes, it should be pretty quick, and I'd imagine (being a java guy I can only imagine) that .NET has native support for them. The down side is that on windows you'll have to either write a JNI extension or use a library like JNA to poke around at the Win32 API from Java.
Sounds like a local socket could do. The latency should be in low ms or less.
Depending on your program you may get some milage out of what #Cowan reports in answer to 'Any Concept of shared memory in java', his answer is: Any concept of shared memory in Java
In summary: he say's that you can use memory mapped files between two processes on the same machine. This in theory could work between .NET and java assuming .NET has some memory mapped file support.
Different machines communicate with each other by sending messages into sockets. Please check the below link for example.
Socket programming in the real world
Answers provided here are great. One idea that might be of interest, but is probably asking for more trouble than it's worth is to load both VMs in a single process (both the JVM and the CLR can be loaded within a native Windows application) and give them access to native code. Java via JNI and .Net via the mapping functions to native code that they allow.
You could also leverage native queue semaphores to wake up a thread on one side or the other when data is updated.
While JNI transitions are expense, they would probably still be faster than the native local socket implementation.
How is your Java application currently deployed? It sounds to me like you're willing to make some modification to it, so I'm assuming you have access to the source code.
I know this is a little out there, but could you compile the Java application in the J# compiler, so that your .NET app has native access to it?
You can convert your compiled java application to .NET by IKVM. After that you can change logic of your .NET application so it will not make data transfers to Java application, but just call data processing code written in Java as it were written and compiled for .NET.
There are a number of JMS servers which support .NET and Java clients. These can perform messages in under a millisecond.
However you might like to try an RPC solution like Hessian RPC or Protobuf RPC. These can achieve lower latencies and can give the appearance of direct calls between platforms. These support .NET and Java as well.
I am implementing a website using PHP for the front end and a Java service as the back end. The two parts are as follows:
PHP front end listens to http requests and interacts with the database.
The Java back end run continuously and responds to calls from the front end.
More specifically, the back end is a daemon that connects and maintain the link to several IM services (AOL, MSN, Yahoo, Jabber...).
Both of the layers will be deployed on the same system (a CentOS box, I suppose) and introducing a middle layer (for instance: using XML-RPC) will reduce the performance (the resource is also rather limited).
Question: Is there a way to link the two layers directly? (no more web services in between)
Since this is communication between two separate running processes, a "direct" call (as in JNI) is not possible. The easiest ways to do such interprocess communcation are probably named pipes and network sockets. In both cases, you'll have to define a communication protocol and implement it on both sides. Using a standard protocol such as XML-RPC makes this easier, but is not strictly necessary.
There are generally four patterns for application integration:
via Filesystem, ie. one producers writes data to a directory monitored by the consumer
via Database, ie. two applications share a schema or table and use it to swap data
via RMI/RPC/web service/any blocking, sync call from one app to another. For PHP to Java you can pick from the various integration libraries listed above, or use some web services standards like SOAP.
via messaging/any non-blocking, async operation where one app sends a message to another app.
Each of these patterns has pros and cons, but a good rule of thumb is to pick the one with the loosest coupling that you can get away with. For example, if you selected #4 your Java app could crash without also taking down your PHP app.
I'd suggest before looking at specific libraries or technologies listed in the answers here that you pick the right pattern for you, then investigate your specific options.
I have tried PHP-Java bridge(php-java-bridge.sourceforge.net/pjb/) and it works quite well. Basically, we need to run a jar file (JavaBridge.jar) which listens on port(there are several options available like Local socket, 8080 port and so on). Your java class files must be availabe to the JavaBridge in the classpath. You need to include a file Java.inc in your php and you can access the Java classes.
Sure, there are lots of ways, but you said about the limited resource...
IMHO define your own lightweight RPC-like protocol and use sockets on TCP/IP to communicate. Actually in this case there's no need to use full advantages of RPC etc... You need only to define API for this particular case and implement it on both sides. In this case you can serialize your packets to quite small. You can even assign a kind of GUIDs to your remote methods and use them to save the traffic and speed-up your intercommunication.
The advantage of sockets usage is that your solution will be pretty scalable.
You could try the PHP/Java integration.
Also, if the communication is one-way (something like "sendmail for IM"), you could write out the PHP requests to a file and monitor that in your Java app.
I was also faced with this problem recently. The Resin solution above is actually a complete re-write of PHP in Java along the lines of JRuby, Jython and Rhino. It is called Quercus. But I'm guessing for you as it was for me, tossing out your Apache/PHP setup isn't really an option.
And there are more problems with Quercus besides: the free version is GPL, which is tricky if you're developing commercial software (though not as tricky as Resin would like you to believe (but IANAL)) and on top of that the free version doesn't support compiling to byte code, so its basically an interpreter written in Java.
What I decided on in the end was to just exchange simple messages over HTTP. I used PHP's json_encode()/json_decode() and Java's json-lib to encode the messages in JSON (simple, text-based, good match for data model).
Another interesting and light-weight option would be to have Java generate PHP code and then use PHP include() directive to fetch that over HTTP and execute it. I haven't tried this though.
If its the actual HTTP calls you're concerned about (for performance), neither of these solutions will help there. All I can say is that I haven't had problems with the PHP and Java on the same LAN. My feeling is that it won't be a problem for the vast majority of applications as long as you keep your RPC calls fairly course-grained (which you really should do anyway).
Sorry, this is a bit of a quick answer but: i heard the Resin app server has support for integrating java and PHP.
They claim they can smash php and java together: http://www.caucho.com/resin-3.0/quercus/
I've used resin for serving J2ee applications, but not for its PHP support.
I'd be interested to hear of such adventures.
Why not use web service?
Make a Java layer and put a ws access(Axis, SpringWS, etc...) and the Php access the Java layer using one ws client.
I think it's simple and useful.
I've come across this page which introduces a means to link the two layers. However, it still requires a middle layer (TCP/IP). Moreover, other services may exploit the Java service as well because it accepts all incoming connections.
http://www.devx.com/Java/Article/20509
[Researching...]