Develop Reference

Wrapping C++ modules in Java

I've inherited a project, originally written in C++.
Due to implementation requirements, I need to either re-write the project in a JVM based language, like Java or Kotlin, or simply wrap the existing code in a Java or Kotlin (RESTful) API.
The existing code base is also entangled with an very old network simulation framework.
I'm therefore leaning heavily towards untangling the simulation framework and wrapping the C++ code using something like JNI and SWIG to implement in a non simulated environment.
I'm wondering if JNI and SWIG are still the best options available?
Any advice will be greatly appreciated!

Wrapping with JNI (or SWIG) requires a clever definition of the API. See a nice explanation here. There exist some C++ frameworks that make JNI operations easier on the C++ side, consider them early, before you get too deeply invested in reference management. From what you write, it may be necessary also to provide some Java interface below the C++ layer, e.g. we use the the Java network APIs on Android, even from C++ code.

Running algorithms in compiled C/C++ code within a Java/PHP/Python framework?

Occasionally, I have come across programming techniques that involve creating application frameworks or websites in Java, PHP or Python, but when complex algorithms are needed, writing those out in C or C++ and running them as API-like function calls within your Java/PHP/Python code.
I have been googling and searching around the net for this, and unless I don't know the name of the practice, I can't seem to find anything on it.
To put simply, how can I:
Create functions or classes in C or C++
Compile them into a DLL/binary/some form
Run the functions from -
Java
PHP
Python
I suspect JSON/XML like output and input must be created between the Java/PHP/Python and the C/C++ function so the data can be easily bridged, but that is okay.
I'm just not sure how to approach this technique, but it seems like a very smart way to take advantage of the great features of Java, PHP, and Python while at the same time utilizing the very fast programming languages for large, complex tasks.
The other thought going through my head is if I am creating functions using only literals in Java/PHP/Python, will it go nearly as fast as C anyway?
The specific tasks I'm looking to work with C/C++ on is massive loops, pinging a database, and analyzing maps. No work has started yet, its all theory now.

You can easily extend a python script with custom C++ code using Boost.Python, see this website for more details: http://www.boost.org/doc/libs/1_50_0/libs/python/doc/
This is how you can use it:
char const* greet()
{
return "hello, world";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
You need to compile this into a shared library. You will get a .dll on windows and a .so on Linux. The library will include the necessary code to make it available to python. Example using it:
>>> import hello_ext
>>> print hello_ext.greet()
hello, world
Here are some more examples: http://www.boost.org/doc/libs/1_50_0/libs/python/doc/tutorial/doc/html/index.html
When using Boost.Python remember to link your shared object to python if you are not using weak dynamic linking. There are similar things for PHP and Java.
As for other languages, I never used a custom shared library with Java but did so with PHP and it was a pain using the native Api. I found using swig way more pleasant.

Altough I agree with the comments (you might do it for fun, for business it's a bad idea) you might be interested in this similar question. The mentioned SWIG framework supports
all the languages you mentioned. I worked with it in a project with tons of legacy C code. Not really simple, but very powerful.

For Java, you can search JNI (Java Native Interface), there're a lot of guides telling how to use it.

On a slightly different take from the other proposed solutions, you could look into Gearman
Basically, it's a broker system. You have workers, which can be written in C in your case, to which you can delegate tasks from your python / php / java / w/e code.
Strong point is that you decouple both applications (if you rewrite your app in another language, you'll probably have less work as you only need to get the app to talk to Gearman).
Bad thing is that I think you'll be adding overhead which could make the performance boost irrelevant.

Use JNI instead of JNA to call native code?

JNA seems a fair bit easier to use to call native code compared to JNI. In what cases would you use JNI over JNA?

JNA does not support mapping of c++ classes, so if you're using c++ library you will need a jni wrapper
If you need a lot of memory copying. For example, you call one method which returns you a large byte buffer, you change something in it, then you need to call another method which uses this byte buffer. This would require you to copy this buffer from c to java, then copy it back from java to c. In this case jni will win in performance because you can keep and modify this buffer in c, without copying.
These are the problems I've encountered. Maybe there's more. But in general performance is not that different between jna and jni, so wherever you can use JNA, use it.
EDIT
This answer seems to be quite popular. So here are some additions:
If you need to map C++ or COM, there is a library by Oliver Chafic, creator of JNAerator, called BridJ. It is still a young library, but it has many interesting features:
Dynamic C / C++ / COM interop : call C++ methods, create C++ objects (and subclass C++ classes from Java!)
Straightforward type mappings with good use of generics (including much nicer model for Pointers)
Full JNAerator support
works on Windows, Linux, MacOS X, Solaris, Android
As for memory copying, I believe JNA supports direct ByteBuffers, so memory copying can be avoided.
So, I still believe that wherever possible, it is better to use JNA or BridJ, and revert to jni if performance is critical, because if you need to call native functions frequently, performance hit is noticeable.

It's difficult to answer such a generic question. I suppose the most obvious difference is that with JNI, the type conversion is implemented on the native side of the Java/native border, while with JNA, the type conversion is implemented in Java. If you already feel quite comfortable with programming in C and have to implement some native code yourself, I would assume that JNI won't seem too complex. If you are a Java programmer and only need to invoke a third party native library, using JNA is probably the easiest path to avoid the perhaps not so obvious problems with JNI.
Although I've never benchmarked any differences, I would because of the design, at least suppose that type conversion with JNA in some situations will perform worse than with JNI. For example when passing arrays, JNA will convert these from Java to native at the beginning of each function call and back at the end of the function call. With JNI, you can control yourself when a native "view" of the array is generated, potentially only creating a view of a part of the array, keep the view across several function calls and at the end release the view and decide if you want to keep the changes (potentially requiring to copy the data back) or discard the changes (no copy required). I know you can use a native array across function calls with JNA using the Memory class, but this will also require memory copying, which may be unnecessary with JNI. The difference may not be relevant, but if your original goal is to increase application performance by implementing parts of it in native code, using a worse performing bridge technology seems not to be the most obvious choice.

You are writing code a few years ago before there was JNA or are targeting a pre 1.4 JRE.
The code you are working with is not in a DLL\SO.
You are working on code that is incompatible with LGPL.
That is only what I can come up with off the top of my head, though I am not a heavy user of either. It also seems like you might avoid JNA if you wanted a better interface than the one they provide but you could code around that in java.

By the way, in one of our projects, we kept a very small JNI foot print. We used protocol buffers for representing our domain objects and thus had only one native function to bridge Java and C (then of course that C function would call a bunch of other functions).

It's not a direct answer and I have no experience with JNA but, when I look at the Projects Using JNA and see names like SVNKit, IntelliJ IDEA, NetBeans IDE, etc, I'm tend to believe it's a pretty decent library.
Actually, I definitely think I would have used JNA instead of JNI when I had to as it indeed looks simpler than JNI (which has a boring development process). Too bad, JNA wasn't released at this time.

I actually did some simple benchmarks with JNI and JNA.
As others already pointed out, JNA is for convenience. You don't need to compile or write native code when using JNA. JNA's native library loader is also one of the best/easiest to use I've ever seen. Sadly, you can't use it for JNI it seems. (That's why I wrote an alternative for System.loadLibrary() that uses the path convention of JNA and supports seamless loading from the classpath (ie jars).)
The performance of JNA however, can be much worse than that of JNI. I made a very simple test that called a simple native integer increment function "return arg + 1;". Benchmarks done with jmh showed that JNI calls to that function are 15 times faster than JNA.
A more "complex" example where the native function sums up an integer array of 4 values still showed that JNI performance is 3 times faster than JNA. The reduced advantage was probably because of how you access arrays in JNI: my example created some stuff and released it again during each summing operation.
Code and test results can be found at github.

If you want JNI performance but are daunted by its complexity, you may consider using tools that generate JNI bindings automatically. For example, JANET (disclaimer: I wrote it) allows you to mix Java and C++ code in a single source file, and e.g. make calls from C++ to Java using standard Java syntax. For example, here's how you'd print a C string to the Java standard output:
native "C++" void printHello() {
const char* helloWorld = "Hello, World!";
`System.out.println(#$(helloWorld));`
}
JANET then translates the backtick-embedded Java into the appropriate JNI calls.

I investigated JNI and JNA for performance comparison because we needed to decide one of them to call a dll in project and we had a real time constraint. The results have showed that JNI has greater performance than JNA(approximately 40 times). Maybe there is a trick for better performance in JNA but it is very slow for a simple example.

Unless I'm missing something, isn't the main difference between JNA vs JNI that with JNA you can't call Java code from native (C) code?

In my specific application, JNI proved far easier to use. I needed to read and write continuous streams to and from a serial port -- and nothing else. Rather than try to learn the very involved infrastructure in JNA, I found it much easier to prototype the native interface in Windows with a special-purpose DLL that exported just six functions:
DllMain (required to interface with Windows)
OnLoad (just does an OutputDebugString so I can know when Java code attaches)
OnUnload (ditto)
Open (opens the port, starts read and write threads)
QueueMessage (queues data for output by the write thread)
GetMessage (waits for and returns data received by the read thread since the last call)

Is there a way to call the STL Libraries of C++ from Java using JNI?

Is there a way to call STL libraries from JNI, I believe JNI provides a C like interface for native calls, how do we achieve this for the C++ template libraries?

I agree that if you're looking for just the plain STL, you could probably use a Java library instead. However, if you insist on wrapping STL, SWIG provides some STL wrapping in JNI out of the box (see this for the basic mechanism), which should produce relatively stable, tested code.

I know this is possible, although I haven't done it myself. But JNI code can crash the JVM if you are not VERY CAREFUL. Also, JNI code is much more difficult to maintain than Java code. I was on one project where the Java -> JNI -> STL and COM code was thrown away and we replaced it with C# accessing the same STL and COM and a socket. We never looked back.
If you are doing only a small amount of JNI, it may be worth it. If you are creating a large interface via JNI, I strongly recommend instead writing the component that interacts with STL in C# instead of Java, and using a socket to communicate between the C# and Java components. It will be far easier to write, test, and maintain.

I have to ask why you find it necessary to use the STL libraries from Java. Is there something in them that's not provided by Java's own massive set of libraries?
I understand why you'd have to call your own code (and I'm assuming you've profiled the Java code and found it wanting, otherwise your effort is wasted) but you seem to be asking how to call STL stuff directly.
We've often had trouble with JNI and it makes our code less portable, so you have to have a very good reason for wanting to use it in our shop, and you have to prove with hard data, why the Java-supplied stuff is not adequate.

Yes. You required to use the swig typemap to convert C++ STL classes to Java objects... and vice versa.
For more information, please visit,
http://www.swig.org/Doc2.0/Android.html

Linking languages

I asked a question earlier about which language to use for an AI prototype. The consensus seemed to be that if I want it to be fast, I need to use a language like Java or C++, but that Python / Perl / Ruby would be good for the interface bits.
So, this leads me on to another question. How easy is it to link these languages together? And which combination works best? So, if I wanted to have a Ruby CGI-type program calling C++ or Java AI functions, is that easy to do? Any pointers for where I look for information on doing that kind of thing? Or would a different combination be better?
My main experience with writing web applications started with C++ CGI and then moved on to Java servlets (about 10 years ago) and then after a long gap away from programming I did some PHP. But I've not had experience of writing a web application in a scripting language which then calls out to a compiled language for the speed-critical bits. So any advice will be welcome!

Boost.Python provides an easy way to turn C++ code into Python modules. It's rather mature and works well in my experience.
For example, the inevitable Hello World...
char const* greet()
{
return "hello, world";
}
can be exposed to Python by writing a Boost.Python wrapper:
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
That's it. We're done. We can now build this as a shared library. The resulting DLL is now visible to Python. Here's a sample Python session:
>>> import hello_ext
>>> print hello.greet()
hello, world
(example taken from boost.org)

First, a meta comment: I would highly recommend coding the entire thing in a high-level language, profiling like mad, and optimizing only where profiling shows it's necessary. First optimize the algorithm, then the code, then think about bringing in the heavy iron. Having an optimum algorithm and clean code will make things much easier when/if you need to reimplement in a lower-level language.
Speaking for Python, IronPython/C# is probably the easiest optimization path.
CPython with C++ is doable, but I find C a lot easier to handle (but not all that easy, being C). Two tools that ease this are cython/pyrex (for C) and shedskin (for C++). These compile Python into C/C++, and from there you can access C/C++ libraries without too much ado.
I've never used jython, but I hear that the jython/Java optimization path isn't all that bad.

I agree with the Idea of coding first in a high level language such as Python, Profiling and then Implementing any code that needs speeding up in C / C++ and wrapping it for use in the high level language.
As an alternative to boost I would like to suggest SWIG for creating Python callable code from C. Its reasonably painless to use, and will compile callable modules for a wide range of languages. (Python, Ruby, Java, Lua. to name a few) from C code.
The wrapping process is semi automated, so there is no need to add new functions to the base C code, making a smoother work flow.

If you choose Perl there are plenty of resources for interfacing other languages.
Inline::C
Inline::CPP
Inline::Java
From Inline::C-Cookbook:
use Inline C => <<'END_C';
void greet() {
printf("Hello, world\n");
}
END_C
greet;
With Perl 6 it gets even easier to import subroutine from native library code using NativeCall.
use v6.c;
sub c-print ( Str() $s ){
use NativeCall;
# restrict the function to inside of this subroutine because printf is
# vararg based, and we only handle '%s' based inputs here
# it should be possible to handle more but it requires generating
# a Signature object based on the format string and then do a
# nativecast with that Signature, and a pointer to printf
sub printf ( str, str --> int32 ) is native('libc:6') {}
printf '%s', $s
}
c-print 'Hello World';
This is just a simple example, you can create a class that has a representation of a Pointer, and have some of the methods be C code from the library you are using. ( only works if the first argument of the C code is the pointer, otherwise you would have to wrap it )
If you need the Perl 6 subroutine/method name to be different you can use the is symbol trait modifier.
There are also Inline modules for Perl 6 as well.

Perl has several ways to use other languages. Look at the Inline:: family of modules on CPAN. Following the advice from others in this question, I'd write the whole thing in a single dynamic language (Perl, Python, Ruby, etc) and then optimize the bits that need it. With Perl and Inline:: you can optimize in C, C++, or Java. Or you could look at AI::Prolog which allows you to embed Prolog for AI/Logic programming.

It may be a good approach to start with a script, and call a compilation-based language from that script only for more advanced needs.
For instance, calling java from ruby script works quite well.
require "java"
# The next line exposes Java's String as JString
include_class("java.lang.String") { |pkg, name| "J" + name }
s = JString.new("f")

You can build your program in one of the higher level languages for example Python or Ruby and then call modules that are compiled in the lower level language for the parts you need performance. You can choose a platform depending on the lower level language you want.
For example if you want to do C++ for the speedy stuff you can just use plain Python or Ruby and call DLLs compiled in C++. If you want to use Java you can use Jython or one of the other dynamic languages on the Java platform to call the Java code this is easier than the C++ route because you've got a common virtual machine so a Java object can be used directly in Jython or JRuby. The same can be done on the .Net platform with the Iron-languages and C# although you seem to have more experience with C++ and Java so those would be better options.

I have a different perspective, having had lots of luck with integrating C++ and Python for some real time live video image processing.
I would say you should match the language to the task for each module. If you're responding to a network, do it in Python, Python can keep up with network traffic just fine. UI: Python, People are slow, and Python is great for UIs using wxPython or PyObjC on Mac, or PyGTK. If you're doing math on lots of data, or signal processing, or image processing... code it in C or C++ with unit tests, then use SWIG to create the binding to any higher level language.
I used the image libraries in wxWidgets in my C++, which are already exposed to Python through wxPython, so it was extremely powerful and quick. SCONS is a build tool (like make) which knows what to do with swig's .i files.
The topmost level can be in C or Python, you'll have more control and fewer packaging and deployment issues if the top level is in C or C++... but it will take a really long time to duplicate what Py2EXE or Py2App gives you on Windows or Mac (or freeze on Linux.)
Enjoy the power of hybrid programming! (I call using multiple languages in a tightly coupled way 'hybrid' but it's just a quirk of mine.)

If the problem domain is hard (and AI problems can often be hard), then I'd choose a language which is expressive or suited to the domain first, and then worry about speeding it up second. For example, Ruby has meta-programming primitives (ability to easily examine and modify the running program) which can make it very easy/interesting to implement certain types of algorithms.
If you implement it in that way and then later need to speed it up, then you can use benchmarking/profiling to locate the bottleneck and either link to a compiled language for that, or optimise the algorithm. In my experience, the biggest performance gain is from tweaking the algorithm, not from using a different implementation language.