Develop Reference

Using IPC to combine multiple languages

This is a general "noob" question about software design, so I apologise if it seems vague,
but I would really appreciate the advice. Note the system described below is purely an example, not a specific product I have in mind.
I often have a need to combine the functionality of several libraries or utilities, written in different languages. For example, if I want to code a high-performance audio processing application for the desktop, I will write it in C / C++. Then, I want to add a nice GUI. But I don't want to learn Qt. I like the look and feel of Adobe Air, and would like to use that. Later, I have a need to access a USB device. But the USB library I have only has an API in Java. How can I combine all these elements together, to take advantage of their relative strengths?
Clearly, I cannot compile these various elements into one single executable. So I need to create and run them seperately, and give them a means to communicate. The most common way to do this seems to be using IPC (Inter Process Communication), eg shared memory or sockets. I prefer the idea of sockets, as the programs could potentially run on seperate machines on a network.
So I decide to create a local client / server system, with a custom API, to allow these elements to communicate. For example, the Air application will receive a message from the C application, telling it to update it's UI. The USB application running in Java will use the sockets to stream audio from the USB hardware, into the C application.
My question : is using local sockets in this way a typical way to design such a system?
Will the performance be much worse than a truly native application (e.g. everything in Java or C, in a single executable) ? It also seems likely that such an approach would be prone to bugs, and difficult to maintain?
I frequently find myself coming up against the limits of existing software libraries (e.g. a graphics library with a pretty, flexible UI but no way to access low-level hardware, or a media library that can mix many audio streams, but has no support for video playback), and find it very frustrating. If anyone could advise the best way to combine arbitrary software libraries like this, I would really appreciate it.
Thanks in advance!

As you have correctly identified, combining libraries from different language or platforms is hard. There are several ways to do it, but none are ideal. Examples:
Native call interfaces (e.g. JNI / JNA) - very fast but tricky to make work correctly, and you have the problem that the data types used typically don't map cleanly across different platforms. Adds native dependencies.
Socket based IPC with text protocol (XML, JSON, etc) - works OK and common formats are likely to be supported at both ends, but adds a lot of overhead. Can be a pain to maintain custom schema mappings etc.
Socket based IPC with binary protocol (e.g. Google protocol buffers) - quite efficient, needs a lot of work to get a custom protocol working correctly on both ends
Communication via a 3rd system (e.g. database, message queue, filesystem) - lots of overhead, can get fragile, introduces a major dependency on a 3rd system.
In my experience, it usually isn't worth integrating a new language / platform just to get one specific library or feature. Take your user interface example - no matter how nice Adobe Air looks, I doubt it is worth trying to integrate it with an existing C/C++ application.
Even if you get it to work, it will significantly complicate the future maintenance and devlopment of your application. Builds become more complex. You need to maintain additional communication / "glue" code. You need to manage more dependencies. Your users will get hit by many more configuration issues. Testing becomes more difficult. It becomes harder to teach someone new about how the whole system works. You need to maintain your skills in more languages / frameworks etc.
I'd recommend the following strategy:
Pick a primary platform
Whenever you need a new library or feature, look for something on your primary platform first. Hopefully (usually?) there is something good available - but even if not then it might be worth coding something yourself if the requirement is quite small.
Only if there is no reasonable option on the primary platform, then you can start to think about integrating a new language/platform
In terms of primary platform, I'd normally suggest a JVM language like Java, Scala or Clojure since the JVM is very well engineered, offers great performance, is highly portable and has the largest / most cohesive library ecosystem (most of which is open source). The JVM is therefore probably the best "general purpose" choice unless you have some very specific requirement which is unlikely to be possible on the JVM, e.g.:
If you are doing lots of embedded / realtime / systems programming wthat requires hardware access you probably need to go for C/C++
If you are coding purely for web-based clients, you probably want to use JavaScript (if you are also writing code on the server side you can consider JavaScript code generation frameworks/libraries that can work on the JVM, e.g. Vaadin or ClojureScript)

the answer is pretty much depends on the technologies you're using and there is no silver-bullet solution for this.
In general, this solutions will fall into one of the following categories:
Some interprocess communication techniques
Integrations provided by the language/platform itself
Database/some common storage (even files :) )
Example of the first:
Sockets/pipes/whatever you operating system allows.
CORBA - allows to write distributed code in different languages.
Google protobuf - allows serialization/deserialization of data-objects and its language agnostic
For the second it really depends on language/ecosystem you're using.
Examples for java:
JNI - Java Native interface - allow to execute code (dlls/so) outside the JVM.
JCA - if you're in the enterprise environment - you can write the integration with the legacy systems in this.
For languages that are compiled into the native code its less tricky - you can write and compile some code, say in Pascal, and then use the DLL in C.
Sometimes when we're talking about Java there is a plethora of languages that have their own syntax and compiler, but their compiler compiles into java binary code that can be run inside the jvm. So if your solution is based on these languages the integration will be easier. Languages like Scala, Groovy, Closure, Jython and so on are falling into this category.
The last but not the least technology to be mentioned is Web Services. This is a very popular tool for integration of different system, although its more used in enterprise environment.
Basically its an abstraction over the sockets layer that allows to send data objects in XML/JSON format between the processes/servers. Both of XML and JSON are language agnostic, so its not an issue to create an XML in a program written in C++ and then consume it in JAVA.
Hope this helps

Java app & C++ app integration / communication

We have two code bases, one written in C++ (MS VS 6) and another in Java (JDK 6).
Looking for creative ways to make the two talk to each other.
More Details:
Both applications are GUI applications.
Major rewrites or translations are not an option.
Communications needs to be two-way.
Try to avoid anything involving writing files to disk.
So far the options considered are:
zero MG
RPC
CORBA
JNI
Compiling Java to native code, and then linking
Essentially, apart from the last item, this boils down to a choice between various ways to achieve interprocess communication between a Java application and a C++ application. Still open to other creative suggestions!
If you have attempted this, or something similar before please chime in with your suggestions, lessons learnt, pitfalls to avoid, etc.
Someone will no doubt point out shortly, that there is no one correct answer to this question. I thought I would tap on the collective expertise of the SO community anyway, and hope to get many excellent answers.

Well, it depends on how tightly integrated you want these applications to be and how you see them evolving in the future. If you just want to communicate data between the two of them (e.g. you want one to be able to open a file written by the other, or read a stream directly from the other), then I would say that protocol buffers are your best bet. If you want the window rendered by one of these GUI apps to actually be embedded in a panel of the other GUI app, then you probably want to use the JNI approach. With the JNI approach, you can use SWIG to automate a great deal of it, though it is dangerously magical and comes with a number of caveats (e.g. it doesn't do so well with function overloading).
I strongly recommend against CORBA, RMI, and similarly remote-procedure-call implementations, mostly because, in my experience, they tend to be very heavy-weight and consume a lot of resources. If you do want something similar to RMI, I would recommend something lighter weight where you pass messages, but not actual objects (as is the case with RMI). For example, you could use protocol buffers as your message format, and then simply serialize these back and forth across normal sockets.
Kit Ho mentioned XML or JSON, but protocol buffers are significantly more efficient than either of those formats and also have notions of backwards-compatibility built directly into the definition language.

Use Jacob ( http://sourceforge.net/projects/jacob-project ), JCom ( http://sourceforge.net/projects/jcom ), or j-Interop ( http://j-interop.org ) and use COM for communication.

Since you're using Windows, I'd suggest using DDE (Dynamic Data Exchange). There's a Java library available from Java Parts.

Dont' know how much data and what type of data you wanna transfer and communicate.
But to simplify the way, I suggest using XML or Json based on HTTP protocol.
Since there are lots of library for both applications and you won't spend too much effort to implement and understand.
More, if you have additional applications to talk with, it is not hard since both tech. are cross-languages.
correct me if i am wrong

JVM/CLR Source-compatible Language Options

I have an open source Java database migration tool (http://www.liquibase.org) which I am considering porting to .Net.
The majority of the tool (at least from a complexity side) is around logic like "if you are adding a primary key and the database is Oracle use this SQL. If database is MySQL use this SQL. If the primary key is named and the database is Postgres use this SQL".
I could fork the Java codebase and covert it (manually and/or automatically), but as updates and bug fixes to the above logic come in I do not want to have to apply it to both versions. What I would like to do is move all that logic into a form that can be compiled and used by both Java and .Net versions naively.
The code I am looking to convert does not contain any advanced library usage (JDBC, System.out, etc) that would vary significantly from Java to .Net, so I don't think that will be an issue (at worst it can be designed around).
So what I am looking for is:
A language in which I can code common parts of my app in and compile it into classes usable by the "standard" languages on the target platform
Does not add any runtime requirements to the system
Nothing so strange that it scares away potential contributors
I know Python and Ruby both have implementations on for the JVM and CLR. How well do they fit my requirements? Has anyone been successful (or unsuccesful) using this technique for cross-platform applications? Are there any gotcha's I need to worry about?

Check out the Fantom programming language. It has its own Java-like/C#-like syntax but can target either the Java VM or .NET CLR.
Their "Why Fantom" page gives a high-level overview of their approach to portability versus dynamic languages running on a VM.

You might have some luck using IKVM.NET. I'm not sure on its exact status, but it's worth a try if you're insistent on running Java code on the .NET Framework. It includes a .NET implementation of the Java base class library, so it seems reasonably complete.
The only other option I might suggest is porting the code to the J# language, a full .NET language (although not first class in the sense that C# or VB.NET is). The language was designed so that the differences with Java were minimal.

If you are thinking about an emdedded approach, you might look at Lua.

How to deal with the most common classes missing on J2ME

I'm trying to code an application which runs un different java platforms like J2SE, J2ME, Android, etc. I already know that I'll have to rewrite most of the UI for each platform, but want to reuse the core logic.
Keeping this core portable involves three drawbacks that I know of:
Keeping to the old Java 1.4 syntax, not using any of the nice language features of Java 5.0
only using external libraries that are known to work on those platforms (that is: don't use JNI and don't have dependencies to other libs which violate this rules)
only using the classes which are present on all those platforms
I know of ways to overcome (1): code in 5.0 style and automatically convert it to 1.4 (retroweaver - haven't tried it yet, but seems ok).
I think (2) is a problem that I just have to accept.
Now I'd like to know what's the best workarround for (3), especially collection classes, which I miss the most. I can think of those:
Most programmers I know just don't use Set, Map, List, etc. and fallback to Vector and plain Arrays. I think this makes code ugly in the first place. But I also know that the right choice between TreeSet/Hashset or LinkedList/ArrayList is crucial for performance, and always using Vector and Arrays can't be right.
I could code my own implementations of that classes. This seems to be reinventing the wheel, and I think I could not do it as good as others have done.
Since Java is open source, I could grab the sourcecode of the J2SE Collections framework and include into my application when building for J2ME. I don't know if this is a good idea, though. Pherhaps there are good reasons not to do this.
Maybe there already are libraries out there, which rebuild the most important features of the collections framework, but are optimized for low end systems, pherhaps by not implementing functionality that is used infrequently. Do you know any?
Thanks for your answers and opinions!
Edit: I finally found a (complex, but nice) solution, and I thought by providing my own answer and accepting it, the solution would become visible at the top. But to the contrary, my answer is still at the very bottom.

J2ME is brutal, and you're just going to have to resign yourself to doing without some of the niceties of other platforms. Get used to Hashtable and Vector, and writing your own wrappers on top of those. Also, don't make the mistake of assuming that J2ME is standard either, as each manufacturer's JVM can do things in profoundly different ways. I wouldn't worry much about performance initially, as just getting correctness on J2ME is enough of a challenge. It is possible to write an app that runs across J2ME, J2SE and Android, as I've done it, but it takes a lot of work. One suggestion that I'd have is that you write the core of your application logic and keep it strictly to java.lang, java.util and java.io. Anywhere where you're going to be doing something that might interact with the platform, such as the file system or network, you can create an interface that your core application code interacts with, that you have different implementations for the different environments. For example, you can have an interface that wraps up HTTP stuff, and uses javax.microedition.io.HttpConnection with J2ME and java.net.HttpURLConnection on Android. It's a pain, but if you want to maintain an app running on all three of those environments, it can get you there. Good luck.

It's been a while since I asked this question, and I while since I found a nice, working solution for the problem, but I had since forgotton to tell you.
My main focus was the Java Collections Framework, which is part of the java.util package.
I've finally taken the source code of Suns Java 6.0 and copied all the classes that belong to the Collections framework into a project of my own. This was a Java 6.0 project, but I used the jars from J2ME as classpath. Most of those classes that I copied depend on other J2SE classes, so there are broken dependencies. Anyway, it was quite easy to cut those depensencies by leaving out everything that deals with serialization (which is not a priority for me) and some minor adjustments.
I compiled the whole thing with a Java 6 compiler, and retrotranslator was used to port the resulting bytecode back to Java 1.2.
Next problem is the package name, because you can't deliver classes from java.util with a J2ME application and load them - the bootstrap class loader won't look into the applications jar file, the other bootloaders aren't allowed to load something with that package name, and on J2ME you can't define custom classloaders. Retrotranslator not only converts bytecode, it also helps to change name references in existing bytecode. I had to move and rename all classes in my project, e.g. java.util.TreeMap became my.company.backport.java.util.TreeMap_.
I was than able to write actual J2ME application in a second Java 6.0 project which referenced the usual java.util.TreeMap, using the generic syntax to create type-safe collections, compile that app to Java 6.0 byte code, and run it through retrotranslator to create Java 1.2 code that now references my.company.backport.java.util.TreeMap_. Note that TreeMap is just an example, it actually works for the whole collections framework and even for 3rd party J2SE Jars that reference that framework.
The resulting app can be packaged as a jar and jad file, and runs fine on both J2ME emulators and actual devices (tested on a Sony Ericsson W880i).
The whole process seems rather complex, but since I used Ant for build automation, and I needed retranslator anyway, there only was a one-time overhead to setup the collection framework backport.
As stated above, I've done this nearly a year ago, and writing this mostly from the top of my head, so I hope there are no errors in it. If you are interested in more details, leave me a comment. I've got a few pages of German documentation about that process, which I could provide if there is any demand.

We faced exactly this situation in developing zxing. If J2ME is in your list of targets, this is your limiting factor by far. We targeted MIDP 2.0 / CLDC 1.1. If you have a similar requirement, you need to stick to Java 1.2. Java 1.4 language features are definitely not present (like assert) and in general you won't find anything after 1.2 in J2ME.
We did not use external libraries, but, you could package them into your deployed .jar file with little trouble. It would make the resulting .jar bigger, and that could be an issue. (Then you can try optimizers/shrinkers like ProGuard to mitigate that.)
I did end up reimplementing something like Collections.sort() and Comparator since we needed them and they are not in J2ME. So yeah you might consider doing this in cases, though only where necessary.
We used Vector and Hashtable and arrays since there is no other choice, really, in J2ME. I would just use them unless you have a reason not to, and that would be performance I guess. In theory JVM makers are already optimizing their implementation but that doesn't mean you couldn't make a better one... I guess I would be surprised if it is worth it in the vast majority of cases. Just make sure you really need to do this before putting in the effort.

To answer part of your question another collections library would be Javolution which can be built for j2me.

Accessing .NET/dll libraries/components from Java?

Are there inexpensive or free gateways from .NET to Java? I'm looking at some data acquisition hardware which has drivers for C/C++ and .NET -- I really don't want to do any programming in .NET.
Update: I haven't done what I originally wanted to do, but I've done something similar, using JNA to encapsulate some functions from a DLL, in order to control a USB hardware device from Java. (the DLL comes from the device manufacturer) It works really nicely. Thanks!

You could also try to use JNA for accessing the native library. JNA provides Java programs easy access to native shared libraries (DLLs on Windows) without writing anything but Java code—no JNI or native code is required. If their API is fairly straight foward, this might be the path of least resistance.
See their getting started guide where they call some native code (printf and GetSystemTime).

Well, there's JNBridge and EZ JCom, just from a Google search.
You could also use IKVM which is a slightly different approach.
(Any reason for not wanting to learn .NET, out of interest? It's a nice platform, and C# is a lovely language...)

If they have C++ versions of the drivers then you could write a wrapper around it using JNI and then load that in Java. JNI can be a bit of a pain, but it would let you use the C++ version of their drivers and not have to deal with .Net at all if you don't want.

I am partial to the recommendation to jump in the deep end with C# since it is so similar to Java. I did this and used IKVM to compile my favorite Java libs. to .NET assemblies and you get [nearly] all the core java runtime classes to boot, so if you tire of trying to find just the right C# collection type, you can always go back to java.util. (No generic collections though. Not sure why.)
Depending on what platform you're on, you have several choices for free IDEs too. For windows you can get Visual Studio Express for free but I also use SharpDevelop. You can also get the Mono IDE on Linux (and a few flavours of Unix, I think ?).
The C# learning curve is shallow if you already know Java. I only blew off 1.5 limbs on landmines that came out of nowhere for reasons I still don't understand, but workarounds were easy to come by. The worst thing about it was the darn developer docs which are AWFUL on account of being so slow. I really miss the snappiness of JavaDoc. Not only are the online docs incredibly slow, the problem is compounded by someones's iffy decision to put class summaries, constructors and methods/properties all on seperate pages so it just takes forever. Someone said to get the docs installer and install docs locally for a slightly improved experience. Not a bad idea I suppose.

I am author of jni4net, open source interprocess bridge between JVM and CLR. It's build on top of JNI and PInvoke. No C/C++ code needed. I hope it will help you.

If you have a Java application, the JNI mentioned by the others will be the way to go. You write some wrapper classes, and that's it.
If writing the wrappes is a too big task (depending on the number of methods you have to wrap), have a look at SWIG . I think it generates wrappers automatically, but I never actually used it.
If you want to code in the Java language, but you don't care if your program will run on the JRE/JVM, then you might as well use Microsoft J#. Basically, it's writing Java-Code wich is compiled to .NET-Bytecode and can use the .NET classes of the driver as well as your existing Java classes. With J# you will run into problems if your existing Java-code is newer than Java 1.4, look at this question on how to solve them.
From that point on, you could later add code in J#, C# or any other .NET language. However, you won't get back to the JRE/JVM easily.