Develop Reference

Using IPC to combine multiple languages

This is a general "noob" question about software design, so I apologise if it seems vague,
but I would really appreciate the advice. Note the system described below is purely an example, not a specific product I have in mind.
I often have a need to combine the functionality of several libraries or utilities, written in different languages. For example, if I want to code a high-performance audio processing application for the desktop, I will write it in C / C++. Then, I want to add a nice GUI. But I don't want to learn Qt. I like the look and feel of Adobe Air, and would like to use that. Later, I have a need to access a USB device. But the USB library I have only has an API in Java. How can I combine all these elements together, to take advantage of their relative strengths?
Clearly, I cannot compile these various elements into one single executable. So I need to create and run them seperately, and give them a means to communicate. The most common way to do this seems to be using IPC (Inter Process Communication), eg shared memory or sockets. I prefer the idea of sockets, as the programs could potentially run on seperate machines on a network.
So I decide to create a local client / server system, with a custom API, to allow these elements to communicate. For example, the Air application will receive a message from the C application, telling it to update it's UI. The USB application running in Java will use the sockets to stream audio from the USB hardware, into the C application.
My question : is using local sockets in this way a typical way to design such a system?
Will the performance be much worse than a truly native application (e.g. everything in Java or C, in a single executable) ? It also seems likely that such an approach would be prone to bugs, and difficult to maintain?
I frequently find myself coming up against the limits of existing software libraries (e.g. a graphics library with a pretty, flexible UI but no way to access low-level hardware, or a media library that can mix many audio streams, but has no support for video playback), and find it very frustrating. If anyone could advise the best way to combine arbitrary software libraries like this, I would really appreciate it.
Thanks in advance!

As you have correctly identified, combining libraries from different language or platforms is hard. There are several ways to do it, but none are ideal. Examples:
Native call interfaces (e.g. JNI / JNA) - very fast but tricky to make work correctly, and you have the problem that the data types used typically don't map cleanly across different platforms. Adds native dependencies.
Socket based IPC with text protocol (XML, JSON, etc) - works OK and common formats are likely to be supported at both ends, but adds a lot of overhead. Can be a pain to maintain custom schema mappings etc.
Socket based IPC with binary protocol (e.g. Google protocol buffers) - quite efficient, needs a lot of work to get a custom protocol working correctly on both ends
Communication via a 3rd system (e.g. database, message queue, filesystem) - lots of overhead, can get fragile, introduces a major dependency on a 3rd system.
In my experience, it usually isn't worth integrating a new language / platform just to get one specific library or feature. Take your user interface example - no matter how nice Adobe Air looks, I doubt it is worth trying to integrate it with an existing C/C++ application.
Even if you get it to work, it will significantly complicate the future maintenance and devlopment of your application. Builds become more complex. You need to maintain additional communication / "glue" code. You need to manage more dependencies. Your users will get hit by many more configuration issues. Testing becomes more difficult. It becomes harder to teach someone new about how the whole system works. You need to maintain your skills in more languages / frameworks etc.
I'd recommend the following strategy:
Pick a primary platform
Whenever you need a new library or feature, look for something on your primary platform first. Hopefully (usually?) there is something good available - but even if not then it might be worth coding something yourself if the requirement is quite small.
Only if there is no reasonable option on the primary platform, then you can start to think about integrating a new language/platform
In terms of primary platform, I'd normally suggest a JVM language like Java, Scala or Clojure since the JVM is very well engineered, offers great performance, is highly portable and has the largest / most cohesive library ecosystem (most of which is open source). The JVM is therefore probably the best "general purpose" choice unless you have some very specific requirement which is unlikely to be possible on the JVM, e.g.:
If you are doing lots of embedded / realtime / systems programming wthat requires hardware access you probably need to go for C/C++
If you are coding purely for web-based clients, you probably want to use JavaScript (if you are also writing code on the server side you can consider JavaScript code generation frameworks/libraries that can work on the JVM, e.g. Vaadin or ClojureScript)

the answer is pretty much depends on the technologies you're using and there is no silver-bullet solution for this.
In general, this solutions will fall into one of the following categories:
Some interprocess communication techniques
Integrations provided by the language/platform itself
Database/some common storage (even files :) )
Example of the first:
Sockets/pipes/whatever you operating system allows.
CORBA - allows to write distributed code in different languages.
Google protobuf - allows serialization/deserialization of data-objects and its language agnostic
For the second it really depends on language/ecosystem you're using.
Examples for java:
JNI - Java Native interface - allow to execute code (dlls/so) outside the JVM.
JCA - if you're in the enterprise environment - you can write the integration with the legacy systems in this.
For languages that are compiled into the native code its less tricky - you can write and compile some code, say in Pascal, and then use the DLL in C.
Sometimes when we're talking about Java there is a plethora of languages that have their own syntax and compiler, but their compiler compiles into java binary code that can be run inside the jvm. So if your solution is based on these languages the integration will be easier. Languages like Scala, Groovy, Closure, Jython and so on are falling into this category.
The last but not the least technology to be mentioned is Web Services. This is a very popular tool for integration of different system, although its more used in enterprise environment.
Basically its an abstraction over the sockets layer that allows to send data objects in XML/JSON format between the processes/servers. Both of XML and JSON are language agnostic, so its not an issue to create an XML in a program written in C++ and then consume it in JAVA.
Hope this helps

Advantages of using php in frontend and java at the backend

I want to use PHP in the frontend(for UI and calling services) and java at the backend(for database interaction and creating REST services) for my web application.
I want to know the advantages and disadvantages of this kind of approach.

If thats the case, I will strongly recommend to look at Groovy with Grails.
Assuming that the time is the main concern. By using Groovy with Grails, you can get faster development, and with all the power of Java, seamlessly.
Ease of development
Higher productivity
No need to have two teams one for PHP and other for Java. Java folks will get Groovy very fast
Get the power of popular and state-of-the-art frameworks like, Spring and Hibernate
Run it on JVM
Everything of Java is available
Grails unit tests
[Edited]
From your comments to the question, it seems like you are looking for some powerful web templates, as available in Joomla or Drupal CMSs.

Ug, that sounds messy. Things like security have to either be implemented cleverly or duplicated (think form validation vs server side validation, you probably want both). Pure Java or pure PHP solutions will have things like that built in.
Next, it becomes a bit tricky to debug certain things. If you have an issues you have two logical "stacks" to sieve through.
Finally, I personally don't know of web server that runs both Java and PHP so you I bet you'll have to have two webservers which means twice the configuration and twice as many fail points.

Advantages:
Site could scale better in java, because PHP does not have a proper thread model.
Java is a strongly typed language with a lot of good IDEs to help you write code properly. It also has a very good testing framework support.
Disadvantages:
Two codebases(PHP and java). Can add extra complexity.

Java Backend and Rails Frontend

I have a startup considering building a Java backend and a Rails frontend. The Java backend will take care of creating a caching layer for the database and offer other additional services. The Rails frontend will mostly be for creating the webapp and monitoring tools.
What startups/companies out there are using this kind of setup? What are some gotchas in terms of development speed, deployment, scalability, and integration?
(What would be helpful for me is personal experience or informal case studies. I'd like to de-priorities answers addressing alternatives like Grails or JRuby unless it turns out to be a big part of the equation)
Thanks!

I've never done any rails development but here are my thoughts. Why not just use Grails? I've done a fair amount of Grails development and it works well for rapid prototyping. It also offers all the power of Java, Spring, and Hibernate. Instead of dealing with communication between two different technologies you could take advantage of the fact that Grails uses Spring and Hibernate under the covers to deal with caching as well as any other requirements these technologies support. If you have Java developers Grails should not be to hard to pick up. The grails plugin story is decent. All of them are stored in a central place and are easy to obtain but quality differs depending on the plugin author. You also have to remember that since Grails uses Groovy which is syntactically similar to Java and runs on the JVM it is very easy to use existing java code including the multitude of available Java libraries. I don't know this for certain but I assume there are a lot more libraries out there for use with Java and there for with Grails then are available for the Ruby language and Rails. I can't make a resource usage call for you but my question would be how many people do you have with experience designing Rails systems that use Java on the back end with all the gotchas that will go along with that? You may find that you have no one who is proficient at debugging when something goes wrong in communication between the two technologies.

I have used similar pair for one of my projects, but instead of RoR I used Python. I don't think there's large difference.
In general, there's nothing specific in this kind of programming. Two most important things you must care about:
good modularity;
well thought-out protocol between RoR and Java.
First is about exact functionality decomposition between parts of the system. We've got some troubles because of not understanding what part of a job must be done by Java, and what part by Python. In general, you must tie together all functions that are close to each other, and thing that are far must be connected in a very few places. I guess you know rules of good modularity, but in case of composing different languages this must be thought-out much more carefully. You can also be interested in creating several distinct Java services (e.g. one for database caching and another one for all the rest) to be able freely combine them or even use in other projects later.
Second is about communication between your parts. I can see two ways to communicate: through database and through pure network protocol. The former need some network communication anyway, so we used pure network protocol, without any other way to connect parts.
We had experimented a lot with SOAP, but it gave hundreds of errors: this protocol is quite ok to connect services written in one language (i.d. Java to Java), but terrible for connecting services in different languages - automatic tools for generating WSDLs gave different results for Java and Python, and manual creating of scheme was hard and labor-intensive.
So we used to REST. It uses all the features of HTTP protocol such as all four main HTTP methods (POST, GET, PUT, DELETE), error codes and many other things, so it covers almost everything you may want. The only restriction with REST is that it can't hold state, so you may need to implement your own sessions mechanism.
If you're not very comfortable with REST and seek some real example, see Facebook Graph API and for implementing REST services in Java you can use Restlets.

Perhaps I'm stating the obvious here, but the biggest gotcha here is the actual fact you're combining two technologies. Not only will development be slower - Rails and all Java frameworks are expecting to have a full Ruby/Java application - but for the other issues you mention (deployment, scalability, integration), the existing tools and solutions will not work, or at least not very well.
If you have a compelling reason to combine the two, go ahead but expect to spend more time on these issues than with a single technology solution. If you don't, pick either Rails or Java.

I would strongly urge you to consider having the frontend and backend in the same JVM for performance reasons.
For Rails, JRuby can run Rails, and you can have it in a Java EE container providing fast communication between components.
Having multiple architechtures which do not run in the same process requires you to do network based communication which takes much, much longer than just passing a reference to an object (I do not expect you to want to use shared memory).

Twitter.com is supposedly using a Java backend, and Rails for their web interface. Here are some resources:
http://www.radicalbehavior.com/5-question-interview-with-twitter-developer-alex-payne/
http://blog.adsdevshop.com/2008/05/02/twitter-is-not-abandoning-rails/
http://twitter.com/#!/ev/status/801530348
More specifically, they use Scala on the backend (which compiles to Java byte code):
http://www.artima.com/scalazine/articles/twitter_on_scala.html
I don't think there is anything wrong with this approach. However, you will be supporting two different platforms/VMs. I have seen this kind of situtation result in a sort of fracturing of expertise within an organization. And you end up paying to a lot to have two sets of expertise in house.
If the problem of segmenting expertise concerns you, I would suggest running JRuby of Rails. It is very mature now - and runs better than Rails on Ruby 1.8.x.

We have used Ruby on Rails for front-end and Core-Java SOAP Api for backend. It worked very well.
The main issue is not with performance but with the people. It is very difficult to hire expert people in Ruby on Rails for everything, instead you hire few or you can easily mould to just learn Rails UI.
As startup point of view, it is best strategy. So many believe ROR is best for startups... but very few knew it because in most of MNC's we use Java, so there is pressure to learn Rails and Code, or hire very few ROR developers. So instead you can use ROR for frontend (only few people do that, or it is easy to learn) and use experienced Java developers for DB updates and business logic.

Should I pick Java or Javascript for my webapp's backend?

Should I use Javascript or Java for my webapp's background? For Javascript, I could use sproutcore/gianduia (as Apple recommends.)
Is it possible to develop mid-scale webapps with Javascript despite the fact that Javascript is used mostly on the client-side?
Is Java better for server-side and object-oriented programming?

You can use either. Which you choose will bring with it a number of competing advantages and disadvantages. I'm not too familiar with Java frameworks though many exist. JavaScript server side frameworks such as Rhino, node.js, and platforms that make development on those easier are becoming more and more popular. Look up any of the many JavaScript conference talks to find out more.
If you are just getting started or have a history with JavaScript, that might be easiest to pick up, though you won't find a lot of documentation outside of the web. Java will give you a ton of books, media, etc, but you'll have a greater learning curve if you haven't done a lot of development.
You mentioned OOP specifically. Java follows OOP, but JavaScript uses prototype inheritance and can be difficult to use in an OO style without a framework. JavaScript lends itself to a more functional style, though, so if you are interested in that, you might gain some advantages there.

The question is a bit confusing as to what you're comparing java to, so I can't answer whether java is better or not. But Javascript is for client-side code only and should not be used for server-side for security reasons and you could use ajax/jquery to retrieve and send data to server but that depends on your needs and application. If the webapp is highly client-side then java, ajax and jquery are good ways to go.
Hope this is a start, let us know more a little about what you're trying to make/do.
Best

Java is a good server-side language. As user pst mentioned, JavaScript can apparently also be used as a server-side language, but this is extremely rare I would say. In regards to OOP, my opinion is that OOP is easier, more straight-forward, and better supported in Java compared to JavaScript. So my recommendation would be to use Java as your server-side language.

Typically a language like Java is been mainly for the server side and Javascript on the client side. On the client side you can write Java Applets, but for various reasons, they aren't used very often on websites. All modern web browsers support Javascript, so it is a natural choice for the client side.
These days Javascript can be run on the server. This would be a great solution for someone who is really good with Javascript. Though, java may be a better choice if you want the app to be highly performant and scalable.
Java can be used on the client side if you write your app with GWT. GWT takes Java code and compiles it into Javascript which will run natively in the browser. GWT does a lot of optimization of your code so it runs very efficiently in the browser. It also allows you to use the many development tools available for Java such such as IDE's which offer code completion and unit testing frameworks.
Java is statically typed and object oriented, javascript is dynamically typed and based functional language. They use very different paradigms despite having fairly similar syntax. Comparing them is a little bit like comparing apples and oranges. They both have their strengths and weaknesses.
It's possible to write an entire javascript that only lives on the client side but you sacrifice the ability to interact with extremely large datasources, share data with other users, or allow the user to save data and access it from another computer. You could also write an entire Java app that only lives on the server side, but you lose interactivity.

JavaScript is used for the server-side, and very successfully: see http://nodejs.org.
And whether "Java is better for OOP", what do you mean by OOP? JavaScript is actually much better for OOP, if you don't define it too loosely (something like Java/C#/C++, with static types and inheritance trees and polymorphism by inheritance) but broadly (using objects; where object is entity with identity, behaviour and state). Why is it better - because it is more powerful - it is duck-typed and it's model of OOP is prototypical, which is model of OOP broader than the classical class-based (you can have classes, but you can have more than that).

Recommended language for writing a web service

I am an open source developer, and I need to write a highly-scalable web service. For a variety of reasons (basically poor support for concurrency), I am using Ruby on Rails for my app front-end but not for the back-end (read: http://www.ddj.com/go-parallel/article/showArticle.jhtml?articleID=212903282). My web service will need to call several partner APIs for each request, so it needs to be smart about threading to have high throughput.
What language / stack does the community recommend for writing such a web service? I haven't written Java in a while (and it seems really old school :D). I need to be able to prototype this service quickly.

What language / stack does the community recommend for writing such a web service? I haven't written Java in a while (and it seems really old school :D). I need to be able to prototype this service quickly.
Then I would definitely recommend to use Java and the JAX-WS API which, like other Java EE 5 APIs, is annotation-based to ease the development. In other words, you annotate a Java class and voilà, you're done and can deploy it as web service. This is really perfect for prototyping (even if I would switch later to contract-first approach). For an implementation of this API, you can use the reference implementation (JAX-WS RI) which is included in Java 6 (since Java 6u4) and also used in Metro, the web service stack of GlassFish (Metro = JAX-WS RI + WSIT). Another (very good) stack which also implements JAX-WS is Apache CXF.
If you want some numbers about the performances of JAX-WS RI, have a look at this benchmark, it will give you an idea of its capabilities (and/or of the hardware you'll need).
And to get an idea of what this API looks like, have a look at Introducing JAX-WS 2.0 With the Java SE 6 Platform, Part 1 and Introducing JAX-WS 2.0 With the Java SE 6 Platform, Part 2 (the complexity may not be very representative of real life though).
Update: I've mentioned JAX-WS RI, Metro, Apache CXF and I would understand if you tell me that this is confusing. The problem is that if you google for JAX-WS, you'll meet all these terms/acronyms. So I tried to give the minimal informations to clarify what they are. If you need any clarification or more details, just let me know.

A little known but good option is C++. There are a number of frameworks ranging from high level AJAX abstractions (Wt: http://webtoolkit.eu/) to simple CGI libraries (e.g. http://cgi.sourceforge.net/). The performance is very good and development is fast otherwise, but the need for recompiling all the time may obstruct your development. Another issue to consider is the smaller number of readily available webdev libraries and APIs, but I have found everything that I need (e.g. SOCI, LibXML++ and Boost.ASIO).
C# and Java are also high performance options but development in them might be slower than C++, largely depending on what you are used with. On the other hand you get plenty of libraries and support.
While most scripting languages (most notably Python and Ruby) lack proper support for multithreading due to them using global interpreter locks, this is not really a problem for web services. The problem can be worked around by running a separate process for each user. This has somewhat larger memory requirements, but in general it works well and it certainly is safer.
The bigger problem with Python and Ruby is that they execute the code extremely slowly. To make the problem even worse, Rails is a very slow framework, compared to other Ruby libraries. I have successfully created high-load web services with Ruby (using Ramaze web framework). Manual optimization on source code level is required for reaching acceptable performance levels, but for what I worked on this was enough and we could keep using Ruby.

If you choose to go on the Java path, then I recommend using jax-ws (metro implementation, already included by default in Java 6) for two reasons:
It is easy and more intuitive than the alternatives.
It is faster, and you mentioned that performance is important. See here a very recent article about performance.

To prototype quickly:
Python (scripting lauange like Ruby which may help you learn quickly)
or JSP since you've learned Java before.

It depends on the rest of your requirements.
For example, if you want to use .NET then you may want to use F# for your backend. If you use
Expert F#
the author will show how to use F# in web pages.
If you want to use something like Java then look at Scala as an option.
Or, if you want very scalable, where, if you add new machines you don't have to change any code, then you can look at Erlang.
You can get threading to work with OOP languages but it won't tend to be as efficient, as the test and set way of locking may not be as efficient as desired.

There is nothing wrong with spawning Ruby processes for this. As long as you design them to be independent (from each other and from the database), your scalability will be ok. You can get higher performance from the same hardware with other languages and environments, but introducing a new language in an open source project is a big no-no. It will basically reduce the number of available developers too much.