Develop Reference

When would you use Java Native Interface, excluding interfacing with libraries in other languages or to access platform dependent functionality?

I understand using JNI for language bindings to libraries written in C/C++ or to access platform specific functionality unavailable to Java. I'm thinking along the lines of making a GUI or using Java for file management, basically the things that are difficult to do in a cross platform manner, and then send complicated data crunching as a batch job to functions written in C/C++ using only standard libraries. This would allow the same program to be used across Windows, Linux, and Mac while having the speed of a native compiled language.
Is this something that is actually done or is the Java Virtual Machine and JIT compiling good enough to make any performance improvement from using JNI negligible?
The project I had in mind was to make a printed circuit board (PCB) design software and sometimes tasks in it are computationally intense. This includes auto-routing the traces if you don't want to run them yourself and design rule checks which basically ensure your PCB is electrically identical to your schematic and also compliant with what the PCB manufacturer can do. These seem like batch tasks that you could send to a C/C++ function for processing.

Using IPC to combine multiple languages

This is a general "noob" question about software design, so I apologise if it seems vague,
but I would really appreciate the advice. Note the system described below is purely an example, not a specific product I have in mind.
I often have a need to combine the functionality of several libraries or utilities, written in different languages. For example, if I want to code a high-performance audio processing application for the desktop, I will write it in C / C++. Then, I want to add a nice GUI. But I don't want to learn Qt. I like the look and feel of Adobe Air, and would like to use that. Later, I have a need to access a USB device. But the USB library I have only has an API in Java. How can I combine all these elements together, to take advantage of their relative strengths?
Clearly, I cannot compile these various elements into one single executable. So I need to create and run them seperately, and give them a means to communicate. The most common way to do this seems to be using IPC (Inter Process Communication), eg shared memory or sockets. I prefer the idea of sockets, as the programs could potentially run on seperate machines on a network.
So I decide to create a local client / server system, with a custom API, to allow these elements to communicate. For example, the Air application will receive a message from the C application, telling it to update it's UI. The USB application running in Java will use the sockets to stream audio from the USB hardware, into the C application.
My question : is using local sockets in this way a typical way to design such a system?
Will the performance be much worse than a truly native application (e.g. everything in Java or C, in a single executable) ? It also seems likely that such an approach would be prone to bugs, and difficult to maintain?
I frequently find myself coming up against the limits of existing software libraries (e.g. a graphics library with a pretty, flexible UI but no way to access low-level hardware, or a media library that can mix many audio streams, but has no support for video playback), and find it very frustrating. If anyone could advise the best way to combine arbitrary software libraries like this, I would really appreciate it.
Thanks in advance!

As you have correctly identified, combining libraries from different language or platforms is hard. There are several ways to do it, but none are ideal. Examples:
Native call interfaces (e.g. JNI / JNA) - very fast but tricky to make work correctly, and you have the problem that the data types used typically don't map cleanly across different platforms. Adds native dependencies.
Socket based IPC with text protocol (XML, JSON, etc) - works OK and common formats are likely to be supported at both ends, but adds a lot of overhead. Can be a pain to maintain custom schema mappings etc.
Socket based IPC with binary protocol (e.g. Google protocol buffers) - quite efficient, needs a lot of work to get a custom protocol working correctly on both ends
Communication via a 3rd system (e.g. database, message queue, filesystem) - lots of overhead, can get fragile, introduces a major dependency on a 3rd system.
In my experience, it usually isn't worth integrating a new language / platform just to get one specific library or feature. Take your user interface example - no matter how nice Adobe Air looks, I doubt it is worth trying to integrate it with an existing C/C++ application.
Even if you get it to work, it will significantly complicate the future maintenance and devlopment of your application. Builds become more complex. You need to maintain additional communication / "glue" code. You need to manage more dependencies. Your users will get hit by many more configuration issues. Testing becomes more difficult. It becomes harder to teach someone new about how the whole system works. You need to maintain your skills in more languages / frameworks etc.
I'd recommend the following strategy:
Pick a primary platform
Whenever you need a new library or feature, look for something on your primary platform first. Hopefully (usually?) there is something good available - but even if not then it might be worth coding something yourself if the requirement is quite small.
Only if there is no reasonable option on the primary platform, then you can start to think about integrating a new language/platform
In terms of primary platform, I'd normally suggest a JVM language like Java, Scala or Clojure since the JVM is very well engineered, offers great performance, is highly portable and has the largest / most cohesive library ecosystem (most of which is open source). The JVM is therefore probably the best "general purpose" choice unless you have some very specific requirement which is unlikely to be possible on the JVM, e.g.:
If you are doing lots of embedded / realtime / systems programming wthat requires hardware access you probably need to go for C/C++
If you are coding purely for web-based clients, you probably want to use JavaScript (if you are also writing code on the server side you can consider JavaScript code generation frameworks/libraries that can work on the JVM, e.g. Vaadin or ClojureScript)

the answer is pretty much depends on the technologies you're using and there is no silver-bullet solution for this.
In general, this solutions will fall into one of the following categories:
Some interprocess communication techniques
Integrations provided by the language/platform itself
Database/some common storage (even files :) )
Example of the first:
Sockets/pipes/whatever you operating system allows.
CORBA - allows to write distributed code in different languages.
Google protobuf - allows serialization/deserialization of data-objects and its language agnostic
For the second it really depends on language/ecosystem you're using.
Examples for java:
JNI - Java Native interface - allow to execute code (dlls/so) outside the JVM.
JCA - if you're in the enterprise environment - you can write the integration with the legacy systems in this.
For languages that are compiled into the native code its less tricky - you can write and compile some code, say in Pascal, and then use the DLL in C.
Sometimes when we're talking about Java there is a plethora of languages that have their own syntax and compiler, but their compiler compiles into java binary code that can be run inside the jvm. So if your solution is based on these languages the integration will be easier. Languages like Scala, Groovy, Closure, Jython and so on are falling into this category.
The last but not the least technology to be mentioned is Web Services. This is a very popular tool for integration of different system, although its more used in enterprise environment.
Basically its an abstraction over the sockets layer that allows to send data objects in XML/JSON format between the processes/servers. Both of XML and JSON are language agnostic, so its not an issue to create an XML in a program written in C++ and then consume it in JAVA.
Hope this helps

Find an efficient way to integrate different language libraries into one project using Python as the "glue"

I am about to get involved in a NLP-related project and I need to use various libraries. Some are in java, others in C/C++ (for tasks that require more speed) and finally some are in Python. I was thinking of using Python as the "glue" and create wrapper-classes for every task that I want to do that relies on a different language. In order to do that, the wrapper class, for example, would execute the java program and communicate with it using pipes.
My questions are:
Do you think that would work for cpu-demanding and highly repetitive tasks? Or would the overhead added by the pipe-communication be too heavy?
Is there any other (preferably simple) architecture that you would suggest?

I would simply advise not doing this.
Don't implement stuff in C/C++ "for speed". The performance benefit is not likely to be as great as you expect; e.g. compared with implementing in Java using "best practice" design and performance techniques.
Don't try and glue lots of languages together. You are setting yourself up for lots of portability issues, difficulties in debugging, and reliability issues; e.g. due to C / C++ bugs crashing the JVM. In addition, there are performance overheads in bridging between languages, and there can be unexpected bottlenecks. (For instance, you may find that your C/C++ has to be run single-threaded due to threading issues, and that you therefore can't get the benefit of Java multi-threading on a typically multi-core system.)
Instead, I advise you to look for libraries that allow you to implement the entire application in one language. If that is not possible, design it so that the different language components are different executables / processes, communicating via some kind of RPC, messaging, or whatever.

Whether or not you'd have problems communicating over pipes / sockets has nothing to do with how CPU intensive the tasks are, but how frequently you'd need to send information between the processes and how much data they need to send. Setting up threads to do your communication will have little processing overhead.
You can probably automatically wrap the C/C++ code with Python (SWIG, ctypesgen, Boost.Python), so the only glue you'll have to write yourself would then be talking to Java.
You could also do it the other way -- run the Python code in the JVM with Jython so the Python and Java code are together, then talk to the C/C++ from there.

You should take a look at Apache UIMA. It is designed exactly for this. From the project website:
The Frameworks run the components, and are available for both Java and C++. The Java Framework supports running both Java and non-Java components (using the C++ framework). The C++ framework, besides supporting annotators written in C/C++, also supports Perl, Python, and TCL annotators.
UIMA can manage pipes and annotators and is built to scale.

I would look at Jepp or JPype instead of using IPC for this. I would avoid Jython since loading the C/C++ libraries into Java would probably be harder than into CPython.

1) Do you think that would work for cpu-demanding and highly repetitive tasks? Or would the overhead added by the pipe-communication be too heavy?
Depends on your task. If this is a typical NLP app where you have a large model loaded in memory and you only communicate relatively small pieces of data (strings in, label sequences/parse trees out), it may work. Pipe communication is hard to get right, though, since there's a lot of buffering and synchronization issues you have to tackle. Python is a very good glue language, but it doesn't solve everything.
2) Is there any other (preferably simple) architecture that you would suggest?
Make your NLP components services and connect to them via REST interfaces. There are off-the-shelf tools that do this, e.g. CLAM. Pyro and SPIRO make communication between Java and Python even more direct and might be easier to use than HTTP/REST (but YMMV).
The parts that are written in C/C++ can also be integrated with CPython using Cython. Don't start implementing things in C or C++ because you think they'll be faster, though; you can also implement them in Python first, then see if you can get the desired performance with NumPy and/or Cython.

How to create large deployable programs with multiple languages?

Hi all this is my post on stackoverflow.
I am normally a lurker and find everything I need without posting but was drawing some blanks when trying to picture how this works. Maybe if someone can provide some insight I would greatly appreciate it.
So I understand the software development cycle and know a decent amount about code
(c++ and Java, and visual basic, I am a CIS major with a CS minor in my last year). I can write some programs in elipse using Java and I can even make some basic GUI components in eclipse using Java.
Now onto my 3 part question: I understand one could make a .jar from several .java files and distribute it, but while thinking about a common program like say yahoo messenger (just as an example) I came to the following three part question:
First, being could you write a program like yahoo messenger in just eclipse using Java or would you need other tools like other languages and other development environments, how does the project go from start to finish? (I think I am kind of lost here because I have only wrote command line and simple GUI applications).
Second, Say it was easier to create a feature in yahoo messenger using a different language, how can one write one program in more than one language? Say for example you wanted to write some code in python or c++ and your the majority of your code was in Java, ie your main method is java and you are compiling using JVM. (assuming you cant just stick python or c++ code in a java program) I did some Googling around and saw some things about linking the compiler and including native code to include other language code in a Java project. Links to other reading material is acceptable too if the explanation is too long.
Third- How does deployment work? Say I am done writing the code for my program and want to turn it into an .exe (for windows users) and stick it on my site for people to download. I know windows comes built in with an iexpress utility to create .exe's. Besides distributing a .jar how would one go about turning source into an exe? Thanks again for all your input and time. I am a beginner and trying to wrap my head around these concepts. The answers can be provided in a technical realm or just conceptual either is greatly appreciated.
-Mark

1) I most cases you can write your program in java without needing any other programming language. There are rather rare cases where you need to call a dll from java to interface with some proprietary program, for this you would need to use JNI and C or C++. A perhaps more common case for using multiple languages is for adding scriptability to your application. For example, my company offers a server/client application that is scriptable by users using Groovy, but the server and the rich client itself are written in Java only.
2) The integration of java and another programming language depends on the other language. Integrating Groovy is easy, and I think integrating Python (using JPython) or Ruby (using JRuby) is fairly easy. But it is an effort (not to mention the mental stress of programming in different languages) and I would not recommend doing that unless there's a specific requirement for this.
3) As always, there are several options. See how-can-i-convert-my-java-program-to-an-exe-file for creating a windows executable. Or you can create windows installer using e.g. NSIS. Or use Java Web Start.
If the intent behind this question is getting an idea how some big java rich-client (desktop) applications are written and deployed, I recommend the Eclipse RCP book. This book will walk you through the development and deployment of an XMPP/Jabber messenger client using the Eclipse RCP framework. Be aware though that there is no one true way of creating a big application and other java application frameworks do things differently.

Several IM clients are written in Java, though I would hazard a guess that the mainstream ones would be mostly written in C, C++ or (on the Mac) Objective-C.
Writing the one program in multiple languages has numerous challenges, and the nature of the challenges varies depending on the combination of languages you want to use. In many cases, you will probably not have much luck combining more than two languages. One set of impedance problems is bad enough, three is an almost guaranteed disaster.
You can avoid these problems by splitting a single application across multiple programs, each of which is written in a single language and communicates with the other programs via some kind of IPC mechanism.
Creating an "exe" is also a very language-specific concern. For instance, Java, C#, C/C++, and Python all have radically different deployment stories.

Say I am done writing the code for my program and want to turn it into an .exe (for windows users) and stick it on my site for people to download.
In that case, I'd say you were foolish. ;)
Java Web Start is a better option for deploying a Java based rich client app. from a web site. JWS works for any platform with Java.

Emulator Framework

Are there any good open source frameworks for developing computer system emulators? I am particularly interested in something written in Python or Java that can reduce the effort involved in developing emulators for 8-bit processors (e.g. 6502, 6510, etc.).

Isn't the 6510 in the C64?
You might be able to make use of the java libraries that emulate c64 code
http://www.dreamfabric.com/c64/
http://www.jac64.com/jac64-java-based-c64-emulator.html
If you aren't afraid of C++ try this general purpose one:
http://cef.sourceforge.net/index.php

You may want to check out VICE, which can emulates a variety of Commodore 8-bit computers: "the C64, the C64DTV, the C128, the VIC20, almost all PET models, the PLUS4 and the CBM-II (aka C610)". That includes 6502, 6510 and 8502 processors. VICE is released under GPL and is written in C.

I've developed a complete emulator for the MIX machine (Knuth's imaginary computer from TAOCP) in Perl a few years ago. The source code is well documented and the simulator is runnable, so one can practice with examples. It wasn't too difficult and I don't recall needing any special framework. The machine's registers are just state variables in the simulator, and the rest is interpreting instructions and changing this internal state.
Do you have more specific questions? Perhaps it will then be easier to point you in the right direction.