Develop Reference

Using IPC to combine multiple languages

This is a general "noob" question about software design, so I apologise if it seems vague,
but I would really appreciate the advice. Note the system described below is purely an example, not a specific product I have in mind.
I often have a need to combine the functionality of several libraries or utilities, written in different languages. For example, if I want to code a high-performance audio processing application for the desktop, I will write it in C / C++. Then, I want to add a nice GUI. But I don't want to learn Qt. I like the look and feel of Adobe Air, and would like to use that. Later, I have a need to access a USB device. But the USB library I have only has an API in Java. How can I combine all these elements together, to take advantage of their relative strengths?
Clearly, I cannot compile these various elements into one single executable. So I need to create and run them seperately, and give them a means to communicate. The most common way to do this seems to be using IPC (Inter Process Communication), eg shared memory or sockets. I prefer the idea of sockets, as the programs could potentially run on seperate machines on a network.
So I decide to create a local client / server system, with a custom API, to allow these elements to communicate. For example, the Air application will receive a message from the C application, telling it to update it's UI. The USB application running in Java will use the sockets to stream audio from the USB hardware, into the C application.
My question : is using local sockets in this way a typical way to design such a system?
Will the performance be much worse than a truly native application (e.g. everything in Java or C, in a single executable) ? It also seems likely that such an approach would be prone to bugs, and difficult to maintain?
I frequently find myself coming up against the limits of existing software libraries (e.g. a graphics library with a pretty, flexible UI but no way to access low-level hardware, or a media library that can mix many audio streams, but has no support for video playback), and find it very frustrating. If anyone could advise the best way to combine arbitrary software libraries like this, I would really appreciate it.
Thanks in advance!

As you have correctly identified, combining libraries from different language or platforms is hard. There are several ways to do it, but none are ideal. Examples:
Native call interfaces (e.g. JNI / JNA) - very fast but tricky to make work correctly, and you have the problem that the data types used typically don't map cleanly across different platforms. Adds native dependencies.
Socket based IPC with text protocol (XML, JSON, etc) - works OK and common formats are likely to be supported at both ends, but adds a lot of overhead. Can be a pain to maintain custom schema mappings etc.
Socket based IPC with binary protocol (e.g. Google protocol buffers) - quite efficient, needs a lot of work to get a custom protocol working correctly on both ends
Communication via a 3rd system (e.g. database, message queue, filesystem) - lots of overhead, can get fragile, introduces a major dependency on a 3rd system.
In my experience, it usually isn't worth integrating a new language / platform just to get one specific library or feature. Take your user interface example - no matter how nice Adobe Air looks, I doubt it is worth trying to integrate it with an existing C/C++ application.
Even if you get it to work, it will significantly complicate the future maintenance and devlopment of your application. Builds become more complex. You need to maintain additional communication / "glue" code. You need to manage more dependencies. Your users will get hit by many more configuration issues. Testing becomes more difficult. It becomes harder to teach someone new about how the whole system works. You need to maintain your skills in more languages / frameworks etc.
I'd recommend the following strategy:
Pick a primary platform
Whenever you need a new library or feature, look for something on your primary platform first. Hopefully (usually?) there is something good available - but even if not then it might be worth coding something yourself if the requirement is quite small.
Only if there is no reasonable option on the primary platform, then you can start to think about integrating a new language/platform
In terms of primary platform, I'd normally suggest a JVM language like Java, Scala or Clojure since the JVM is very well engineered, offers great performance, is highly portable and has the largest / most cohesive library ecosystem (most of which is open source). The JVM is therefore probably the best "general purpose" choice unless you have some very specific requirement which is unlikely to be possible on the JVM, e.g.:
If you are doing lots of embedded / realtime / systems programming wthat requires hardware access you probably need to go for C/C++
If you are coding purely for web-based clients, you probably want to use JavaScript (if you are also writing code on the server side you can consider JavaScript code generation frameworks/libraries that can work on the JVM, e.g. Vaadin or ClojureScript)

the answer is pretty much depends on the technologies you're using and there is no silver-bullet solution for this.
In general, this solutions will fall into one of the following categories:
Some interprocess communication techniques
Integrations provided by the language/platform itself
Database/some common storage (even files :) )
Example of the first:
Sockets/pipes/whatever you operating system allows.
CORBA - allows to write distributed code in different languages.
Google protobuf - allows serialization/deserialization of data-objects and its language agnostic
For the second it really depends on language/ecosystem you're using.
Examples for java:
JNI - Java Native interface - allow to execute code (dlls/so) outside the JVM.
JCA - if you're in the enterprise environment - you can write the integration with the legacy systems in this.
For languages that are compiled into the native code its less tricky - you can write and compile some code, say in Pascal, and then use the DLL in C.
Sometimes when we're talking about Java there is a plethora of languages that have their own syntax and compiler, but their compiler compiles into java binary code that can be run inside the jvm. So if your solution is based on these languages the integration will be easier. Languages like Scala, Groovy, Closure, Jython and so on are falling into this category.
The last but not the least technology to be mentioned is Web Services. This is a very popular tool for integration of different system, although its more used in enterprise environment.
Basically its an abstraction over the sockets layer that allows to send data objects in XML/JSON format between the processes/servers. Both of XML and JSON are language agnostic, so its not an issue to create an XML in a program written in C++ and then consume it in JAVA.
Hope this helps

Java app & C++ app integration / communication

We have two code bases, one written in C++ (MS VS 6) and another in Java (JDK 6).
Looking for creative ways to make the two talk to each other.
More Details:
Both applications are GUI applications.
Major rewrites or translations are not an option.
Communications needs to be two-way.
Try to avoid anything involving writing files to disk.
So far the options considered are:
zero MG
RPC
CORBA
JNI
Compiling Java to native code, and then linking
Essentially, apart from the last item, this boils down to a choice between various ways to achieve interprocess communication between a Java application and a C++ application. Still open to other creative suggestions!
If you have attempted this, or something similar before please chime in with your suggestions, lessons learnt, pitfalls to avoid, etc.
Someone will no doubt point out shortly, that there is no one correct answer to this question. I thought I would tap on the collective expertise of the SO community anyway, and hope to get many excellent answers.

Well, it depends on how tightly integrated you want these applications to be and how you see them evolving in the future. If you just want to communicate data between the two of them (e.g. you want one to be able to open a file written by the other, or read a stream directly from the other), then I would say that protocol buffers are your best bet. If you want the window rendered by one of these GUI apps to actually be embedded in a panel of the other GUI app, then you probably want to use the JNI approach. With the JNI approach, you can use SWIG to automate a great deal of it, though it is dangerously magical and comes with a number of caveats (e.g. it doesn't do so well with function overloading).
I strongly recommend against CORBA, RMI, and similarly remote-procedure-call implementations, mostly because, in my experience, they tend to be very heavy-weight and consume a lot of resources. If you do want something similar to RMI, I would recommend something lighter weight where you pass messages, but not actual objects (as is the case with RMI). For example, you could use protocol buffers as your message format, and then simply serialize these back and forth across normal sockets.
Kit Ho mentioned XML or JSON, but protocol buffers are significantly more efficient than either of those formats and also have notions of backwards-compatibility built directly into the definition language.

Use Jacob ( http://sourceforge.net/projects/jacob-project ), JCom ( http://sourceforge.net/projects/jcom ), or j-Interop ( http://j-interop.org ) and use COM for communication.

Since you're using Windows, I'd suggest using DDE (Dynamic Data Exchange). There's a Java library available from Java Parts.

Dont' know how much data and what type of data you wanna transfer and communicate.
But to simplify the way, I suggest using XML or Json based on HTTP protocol.
Since there are lots of library for both applications and you won't spend too much effort to implement and understand.
More, if you have additional applications to talk with, it is not hard since both tech. are cross-languages.
correct me if i am wrong

Considerations for a highly-interoperable SOA implementation with Java?

It's been more than two years since someone asked "Which SOAP XML object serialization library for Java would you recommend?" and assuming a lot of things changed since then, I believe that's enough of a good reason to revamp the topic.
So in lieu of implementing SOA in Java that will be as interoperable as possible with the .NET Framework (or any other enterprise platform for that matter), in a first analysis I'm considering going with Apache Tuscany but I was wondering if there's any reason I should not? Any feedback will be definitely appreciated.

Actually, the first consideration I'd make is "do I really need SOAP/XMl?".
In particular:
WSDL is not nearly as cross-platform as is suggested; it is very easy to get cases that don't work well (there was one here on SO re PayPal earlier this week)
XML is human readable... But as a result is relatively slow and bloated
There are other ways of sharing a definition - for example, we all manage just fine with an API document that says "this API returns JSON in this shape {...}". WSDL mainly seems to help drag-and-drop developers; of which I am not one.
As such, I suggest looking at simpler (think: YAGNI) protocols. For payload, I'm a big fan of protobuf, which deliberately is a simpler schema and therefore allows for much less uncertainty over what can/can't be handled by different implementations, while still covering every sane use-case. It is also very efficient or both CPU/bandwidth. Very portable, and has a schema definition language (.proto) for defining the data, but MUCH simpler than WSDL. Shock horror! It isn't human readable... Well, there are tools for that :) why impose the failings of us week and slow flesh-based units on our APIs?

We use the metro framework and so far it has proven to be highly interoperable with WCF including complex message level security using mutual certificate authentication and username token. In our case WCF is server and Java is client.

Java Backend and Rails Frontend

I have a startup considering building a Java backend and a Rails frontend. The Java backend will take care of creating a caching layer for the database and offer other additional services. The Rails frontend will mostly be for creating the webapp and monitoring tools.
What startups/companies out there are using this kind of setup? What are some gotchas in terms of development speed, deployment, scalability, and integration?
(What would be helpful for me is personal experience or informal case studies. I'd like to de-priorities answers addressing alternatives like Grails or JRuby unless it turns out to be a big part of the equation)
Thanks!

I've never done any rails development but here are my thoughts. Why not just use Grails? I've done a fair amount of Grails development and it works well for rapid prototyping. It also offers all the power of Java, Spring, and Hibernate. Instead of dealing with communication between two different technologies you could take advantage of the fact that Grails uses Spring and Hibernate under the covers to deal with caching as well as any other requirements these technologies support. If you have Java developers Grails should not be to hard to pick up. The grails plugin story is decent. All of them are stored in a central place and are easy to obtain but quality differs depending on the plugin author. You also have to remember that since Grails uses Groovy which is syntactically similar to Java and runs on the JVM it is very easy to use existing java code including the multitude of available Java libraries. I don't know this for certain but I assume there are a lot more libraries out there for use with Java and there for with Grails then are available for the Ruby language and Rails. I can't make a resource usage call for you but my question would be how many people do you have with experience designing Rails systems that use Java on the back end with all the gotchas that will go along with that? You may find that you have no one who is proficient at debugging when something goes wrong in communication between the two technologies.

I have used similar pair for one of my projects, but instead of RoR I used Python. I don't think there's large difference.
In general, there's nothing specific in this kind of programming. Two most important things you must care about:
good modularity;
well thought-out protocol between RoR and Java.
First is about exact functionality decomposition between parts of the system. We've got some troubles because of not understanding what part of a job must be done by Java, and what part by Python. In general, you must tie together all functions that are close to each other, and thing that are far must be connected in a very few places. I guess you know rules of good modularity, but in case of composing different languages this must be thought-out much more carefully. You can also be interested in creating several distinct Java services (e.g. one for database caching and another one for all the rest) to be able freely combine them or even use in other projects later.
Second is about communication between your parts. I can see two ways to communicate: through database and through pure network protocol. The former need some network communication anyway, so we used pure network protocol, without any other way to connect parts.
We had experimented a lot with SOAP, but it gave hundreds of errors: this protocol is quite ok to connect services written in one language (i.d. Java to Java), but terrible for connecting services in different languages - automatic tools for generating WSDLs gave different results for Java and Python, and manual creating of scheme was hard and labor-intensive.
So we used to REST. It uses all the features of HTTP protocol such as all four main HTTP methods (POST, GET, PUT, DELETE), error codes and many other things, so it covers almost everything you may want. The only restriction with REST is that it can't hold state, so you may need to implement your own sessions mechanism.
If you're not very comfortable with REST and seek some real example, see Facebook Graph API and for implementing REST services in Java you can use Restlets.

Perhaps I'm stating the obvious here, but the biggest gotcha here is the actual fact you're combining two technologies. Not only will development be slower - Rails and all Java frameworks are expecting to have a full Ruby/Java application - but for the other issues you mention (deployment, scalability, integration), the existing tools and solutions will not work, or at least not very well.
If you have a compelling reason to combine the two, go ahead but expect to spend more time on these issues than with a single technology solution. If you don't, pick either Rails or Java.

I would strongly urge you to consider having the frontend and backend in the same JVM for performance reasons.
For Rails, JRuby can run Rails, and you can have it in a Java EE container providing fast communication between components.
Having multiple architechtures which do not run in the same process requires you to do network based communication which takes much, much longer than just passing a reference to an object (I do not expect you to want to use shared memory).

Twitter.com is supposedly using a Java backend, and Rails for their web interface. Here are some resources:
http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne/
http://blog.adsdevshop.com/2008/05/02/twitter-is-not-abandoning-rails/
http://twitter.com/#!/ev/status/801530348
More specifically, they use Scala on the backend (which compiles to Java byte code):
http://www.artima.com/scalazine/articles/twitter_on_scala.html
I don't think there is anything wrong with this approach. However, you will be supporting two different platforms/VMs. I have seen this kind of situtation result in a sort of fracturing of expertise within an organization. And you end up paying to a lot to have two sets of expertise in house.
If the problem of segmenting expertise concerns you, I would suggest running JRuby of Rails. It is very mature now - and runs better than Rails on Ruby 1.8.x.

We have used Ruby on Rails for front-end and Core-Java SOAP Api for backend. It worked very well.
The main issue is not with performance but with the people. It is very difficult to hire expert people in Ruby on Rails for everything, instead you hire few or you can easily mould to just learn Rails UI.
As startup point of view, it is best strategy. So many believe ROR is best for startups... but very few knew it because in most of MNC's we use Java, so there is pressure to learn Rails and Code, or hire very few ROR developers. So instead you can use ROR for frontend (only few people do that, or it is easy to learn) and use experienced Java developers for DB updates and business logic.

Java/JRuby on Rails vs .NET 3.5... feel like I'm doing it wrong

I'm planning to develop a web-services (SOAP to C++ client) in Java with Metro/Hibernate and front it up with a web-site written in JRuby on Rails, connecting Java modules through JRuby and generally through a database. I already wrote some code and set up everything.
Now, I've heard that .NET 3.5 is really powerful and after some reading I acknowledged that to be true. However, would it make sense to drop this monstrous yet curious hybrid of Java and RoR to switch everything to .NET? Everything: services, database connection (linq for objects), web-front, ajax stuff -- everything in one huge .NET 3.5 solution.
I am also looking to have fun but not as much fun as, say, in C++ for server-side :)
I know that with the power of RoR it doesn't really matter whether it's either side. However, whether I'm all set up for .NET or Java, I still have the feeling I'm doing it wrong and I should just switch to the not-side-I'm-currently-on.
I have a very limited experience with web development, I've only written an ASP.NET web-service once and supported Java web-service so I have a real OCD with the choice of platforms here.

.NET 3.5 is loads of fun (among mainstream platforms), but in my personal opinion Ruby is as much fun as if not more fun than .NET 3.5.
About the capabilities, they are also roughly equivalent, you certainly can make a SOAP web services application in both without problems.
About the "monstrous hybrid", there's no such thing, it's just a (very active) port, like IronRuby or IronPython in .NET land.
There will be more of a coherent whole if you stick in .NET and write in, say, C# than if you mix and match Ruby and Java, of course. But you certainly can use only JRuby.
Given that the fun and capability factors are more or less equal, I'd pick JRuby, just in case I happen to have to deploy on Linux, Solaris or something that's not Windows in the future.
If you are certain you'll want to stick to Windows forever, then pick whatever's most fun for you.

I'm sorry, but what you are doing sounds like a big ball o' mud. Doesn't matter to me what path you take, but I'd simplify the architecture by reducing the number of technologies/frameworks that you are using. If you can find a way to do it in Ruby/Rails without reference to Java (although you may need to keep the C++ client-side), that would work. Obviously, working in .NET (my preferred platform) is another choice. Windows Communication Foundation makes it much easier to write web services that are lighter weight than the old ASP.NET web services -- including RESTful web services using JSON.
If I were doing it today I'd use ASP.NET MVC for the web front-end, WCF (for web services), and either eliminate the C++ client or consider writing it in C# with .NET or Mono.

It sounds to me like you are over planning or over engineering a little bit here. I don't quite have a clear picture of your architecture by reading, but I see many words like .NET, Java, SOAP, C++, JRuby on Rails, RoR, web services, Hibernate, ajax, linq, etc, are being thrown around, so I just wonder how they are all going to fit together. Here are some simplified architectures that might or not fit your needs since I don't quite understand your problem. I also do not see that there is any question from your post, so given the circumstance, I would suggest:
Web front end: RoR with Ajax.
Web services: .NET.
Data access: Linq.
What this means is your .NET web services would use Linq for accessing the data and expose via SOAP which would be consumed by the front end Ror.
Alternatively, replace .NET by Java and Linq by Hibernate.
If the above is useless, then my other suggestion is you don't have to combine all those technologies in one solution.

I'd rather move to FreeBSD or Debian in the future... I sure don't want to stay on Windows forever. But there is Mono, right?