I asked a question earlier about which language to use for an AI prototype. The consensus seemed to be that if I want it to be fast, I need to use a language like Java or C++, but that Python / Perl / Ruby would be good for the interface bits.
So, this leads me on to another question. How easy is it to link these languages together? And which combination works best? So, if I wanted to have a Ruby CGI-type program calling C++ or Java AI functions, is that easy to do? Any pointers for where I look for information on doing that kind of thing? Or would a different combination be better?
My main experience with writing web applications started with C++ CGI and then moved on to Java servlets (about 10 years ago) and then after a long gap away from programming I did some PHP. But I've not had experience of writing a web application in a scripting language which then calls out to a compiled language for the speed-critical bits. So any advice will be welcome!
Boost.Python provides an easy way to turn C++ code into Python modules. It's rather mature and works well in my experience.
For example, the inevitable Hello World...
char const* greet()
{
return "hello, world";
}
can be exposed to Python by writing a Boost.Python wrapper:
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
That's it. We're done. We can now build this as a shared library. The resulting DLL is now visible to Python. Here's a sample Python session:
>>> import hello_ext
>>> print hello.greet()
hello, world
(example taken from boost.org)
First, a meta comment: I would highly recommend coding the entire thing in a high-level language, profiling like mad, and optimizing only where profiling shows it's necessary. First optimize the algorithm, then the code, then think about bringing in the heavy iron. Having an optimum algorithm and clean code will make things much easier when/if you need to reimplement in a lower-level language.
Speaking for Python, IronPython/C# is probably the easiest optimization path.
CPython with C++ is doable, but I find C a lot easier to handle (but not all that easy, being C). Two tools that ease this are cython/pyrex (for C) and shedskin (for C++). These compile Python into C/C++, and from there you can access C/C++ libraries without too much ado.
I've never used jython, but I hear that the jython/Java optimization path isn't all that bad.
I agree with the Idea of coding first in a high level language such as Python, Profiling and then Implementing any code that needs speeding up in C / C++ and wrapping it for use in the high level language.
As an alternative to boost I would like to suggest SWIG for creating Python callable code from C. Its reasonably painless to use, and will compile callable modules for a wide range of languages. (Python, Ruby, Java, Lua. to name a few) from C code.
The wrapping process is semi automated, so there is no need to add new functions to the base C code, making a smoother work flow.
If you choose Perl there are plenty of resources for interfacing other languages.
Inline::C
Inline::CPP
Inline::Java
From Inline::C-Cookbook:
use Inline C => <<'END_C';
void greet() {
printf("Hello, world\n");
}
END_C
greet;
With Perl 6 it gets even easier to import subroutine from native library code using NativeCall.
use v6.c;
sub c-print ( Str() $s ){
use NativeCall;
# restrict the function to inside of this subroutine because printf is
# vararg based, and we only handle '%s' based inputs here
# it should be possible to handle more but it requires generating
# a Signature object based on the format string and then do a
# nativecast with that Signature, and a pointer to printf
sub printf ( str, str --> int32 ) is native('libc:6') {}
printf '%s', $s
}
c-print 'Hello World';
This is just a simple example, you can create a class that has a representation of a Pointer, and have some of the methods be C code from the library you are using. ( only works if the first argument of the C code is the pointer, otherwise you would have to wrap it )
If you need the Perl 6 subroutine/method name to be different you can use the is symbol trait modifier.
There are also Inline modules for Perl 6 as well.
Perl has several ways to use other languages. Look at the Inline:: family of modules on CPAN. Following the advice from others in this question, I'd write the whole thing in a single dynamic language (Perl, Python, Ruby, etc) and then optimize the bits that need it. With Perl and Inline:: you can optimize in C, C++, or Java. Or you could look at AI::Prolog which allows you to embed Prolog for AI/Logic programming.
It may be a good approach to start with a script, and call a compilation-based language from that script only for more advanced needs.
For instance, calling java from ruby script works quite well.
require "java"
# The next line exposes Java's String as JString
include_class("java.lang.String") { |pkg, name| "J" + name }
s = JString.new("f")
You can build your program in one of the higher level languages for example Python or Ruby and then call modules that are compiled in the lower level language for the parts you need performance. You can choose a platform depending on the lower level language you want.
For example if you want to do C++ for the speedy stuff you can just use plain Python or Ruby and call DLLs compiled in C++. If you want to use Java you can use Jython or one of the other dynamic languages on the Java platform to call the Java code this is easier than the C++ route because you've got a common virtual machine so a Java object can be used directly in Jython or JRuby. The same can be done on the .Net platform with the Iron-languages and C# although you seem to have more experience with C++ and Java so those would be better options.
I have a different perspective, having had lots of luck with integrating C++ and Python for some real time live video image processing.
I would say you should match the language to the task for each module. If you're responding to a network, do it in Python, Python can keep up with network traffic just fine. UI: Python, People are slow, and Python is great for UIs using wxPython or PyObjC on Mac, or PyGTK. If you're doing math on lots of data, or signal processing, or image processing... code it in C or C++ with unit tests, then use SWIG to create the binding to any higher level language.
I used the image libraries in wxWidgets in my C++, which are already exposed to Python through wxPython, so it was extremely powerful and quick. SCONS is a build tool (like make) which knows what to do with swig's .i files.
The topmost level can be in C or Python, you'll have more control and fewer packaging and deployment issues if the top level is in C or C++... but it will take a really long time to duplicate what Py2EXE or Py2App gives you on Windows or Mac (or freeze on Linux.)
Enjoy the power of hybrid programming! (I call using multiple languages in a tightly coupled way 'hybrid' but it's just a quirk of mine.)
If the problem domain is hard (and AI problems can often be hard), then I'd choose a language which is expressive or suited to the domain first, and then worry about speeding it up second. For example, Ruby has meta-programming primitives (ability to easily examine and modify the running program) which can make it very easy/interesting to implement certain types of algorithms.
If you implement it in that way and then later need to speed it up, then you can use benchmarking/profiling to locate the bottleneck and either link to a compiled language for that, or optimise the algorithm. In my experience, the biggest performance gain is from tweaking the algorithm, not from using a different implementation language.
Related
I am in the process of developing a library that will be written in Go and compiled down to a C shared library so it can be called by languages such as Python, Node.JS, Java and Ruby
I have just realised that there will be an issue when it comes to callbacks. How can I callback into the calling code which will be at least one of the above languages? Is there a single way I can do it or will I need to implement something specific on the Go side for each language?
Update for clarity:
I am already able to build Go as a library and execute the code from other languages such as Java and Python.
My question specifically relates to a situation where go is running something asynchronously and needs to call back into the caller (i.e. Java, Python).
The lingua franca of interfacing between different programming languages is to go through C. You can use something like cgo to make your Go code accessible through C, then use the C or "native" bindings in Python, Java, etc to invoke it.
Usually developers only go through this much trouble for large and well-maintained projects. If you have a personal project and if performance is less of a concern, I would suggest communicating via json (for web languages) or stdout/stderr pipes (probably more what you want). You can print commands to stdout from Ruby and have your Go code process the request and report back on its stdout. It's usually possible although sometimes tricky to pipe both input and output to another program. CGI programs work this way, spawning off interpreters for other languages and piping data to that interpreter.
Update: Calling back into the Java/Python/whatever language runtime from Go is difficult if you want to pass objects; I imagine that will need to be crafted differently depending on the binding API (Java, Python). Maybe you have already tackled this problem if you have the bindings in place? It might be easier if you can get away with not passing any parameters to the callback (i.e. it's basically a timer callback). Minimize the data which needs to be generated in Go and read in the other language.
Well, this is gonna be a mostly link-only answer, however since Go 1.5 you can produce shared libraries that you can call from C, or any language that supports that (aka python, ruby, php, etc)
http://blog.ralch.com/tutorial/golang-sharing-libraries/
this is not my link, I've briefly followed it before to make a dummy python test.
I'm implementing an algorithm for pathfinding that is a variant of A* (HPA*) but with a triangulated search space. I've been making a Java application and have written a good amount of code, but I recently found a C++ library that already takes care of the triangulations for the project. I have several options but I'm not sure what I should do: convert the library to Java or integrate the C++ code into my Java library. I could also rewrite my code in C++ but I'm not very familiar with the language.
This application tests the performance of an algorithm for a paper I'm writing so it doesn't need to be portable. What do you think my best option is?
If you really want to call C++ code from Java, read up on the Java Native Interface (JNI) coding conventions.
But generally, unless you have an extremely good reason to cross languages, it's more trouble than it's worth. How complicated is the triangulation code? If it's less than a few hundred lines, I'd suggest reimplementing; if it's more than a few thousand, using a library may be justified but you should probably check whether there's a Java library available or if someone has already written the appropriate JNI glue code.
Occasionally, I have come across programming techniques that involve creating application frameworks or websites in Java, PHP or Python, but when complex algorithms are needed, writing those out in C or C++ and running them as API-like function calls within your Java/PHP/Python code.
I have been googling and searching around the net for this, and unless I don't know the name of the practice, I can't seem to find anything on it.
To put simply, how can I:
Create functions or classes in C or C++
Compile them into a DLL/binary/some form
Run the functions from -
Java
PHP
Python
I suspect JSON/XML like output and input must be created between the Java/PHP/Python and the C/C++ function so the data can be easily bridged, but that is okay.
I'm just not sure how to approach this technique, but it seems like a very smart way to take advantage of the great features of Java, PHP, and Python while at the same time utilizing the very fast programming languages for large, complex tasks.
The other thought going through my head is if I am creating functions using only literals in Java/PHP/Python, will it go nearly as fast as C anyway?
The specific tasks I'm looking to work with C/C++ on is massive loops, pinging a database, and analyzing maps. No work has started yet, its all theory now.
You can easily extend a python script with custom C++ code using Boost.Python, see this website for more details: http://www.boost.org/doc/libs/1_50_0/libs/python/doc/
This is how you can use it:
char const* greet()
{
return "hello, world";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
You need to compile this into a shared library. You will get a .dll on windows and a .so on Linux. The library will include the necessary code to make it available to python. Example using it:
>>> import hello_ext
>>> print hello_ext.greet()
hello, world
Here are some more examples: http://www.boost.org/doc/libs/1_50_0/libs/python/doc/tutorial/doc/html/index.html
When using Boost.Python remember to link your shared object to python if you are not using weak dynamic linking. There are similar things for PHP and Java.
As for other languages, I never used a custom shared library with Java but did so with PHP and it was a pain using the native Api. I found using swig way more pleasant.
Altough I agree with the comments (you might do it for fun, for business it's a bad idea) you might be interested in this similar question. The mentioned SWIG framework supports
all the languages you mentioned. I worked with it in a project with tons of legacy C code. Not really simple, but very powerful.
For Java, you can search JNI (Java Native Interface), there're a lot of guides telling how to use it.
On a slightly different take from the other proposed solutions, you could look into Gearman
Basically, it's a broker system. You have workers, which can be written in C in your case, to which you can delegate tasks from your python / php / java / w/e code.
Strong point is that you decouple both applications (if you rewrite your app in another language, you'll probably have less work as you only need to get the app to talk to Gearman).
Bad thing is that I think you'll be adding overhead which could make the performance boost irrelevant.
This question already has answers here:
Closed 13 years ago.
Possible Duplicate:
Java Python Integration
I have a large existing codebase written in 100% Java, but I would like to use Python for some new sections of it. I need to do some text and language processing, and I'd much rather use Python and a library like NLTK to do this.
I'm aware of the Jython project, but it looks like this represents a way to use Java and its libraries from within Python, rather than the other way round - am I wrong about this?
If not, what would be the best method to interface between Java and Python, such that (ideally) I can call a method in Python and have the result returned to Java?
I'm aware of the Jython project, but
it looks like this represents a way to
use Java and its libraries from within
Python, rather than the other way
round - am I wrong about this?
Yes, you are wrong. You can either call a command line interpreter to run python code using Jyton or use python code from Java. In the past there was also a python-to-Java compiler, but it got discontinued with Jython 2.2
I would write a Python module to handle the text and language processing, and then build a small bridge in jython that your java program can interact with. The jython bridge will be a very simple one, that's really only responsible for forwarding calls to the python module, and return the answer from the python module to the java module. Jython is really easy to use, and setup shouldn't take you more than 15 minutes.
Best of luck!
I don't think you could use NLTK from Jython, since it depends on Numpy which isn't ported to the JVM. If you need NLTK or any other native CPython extension, you might consider using some IPC mechanism to communicate between CPython and the JVM. That being said, there is a project to allow calling CPython from Java, called Jepp:
http://jepp.sourceforge.net/
The reverse (calling Java code from CPython) is the goal of JPype and javaclass:
sourceforge.net/projects/jpype/
pypi.python.org/pypi/javaclass/0.1
I've never used any of these project, so I cant't vow for their quality.
Jython is a Python implementation running on the JVM. You can find information about embedding Python in an existing Java app in the user guide.
I don't know the environment that you're working in, but be aware that mixing languages in the same app can quickly lead to a mess. I recommend creating Java interfaces to represent the operations that you plan to use, along with separately-packaged implementation classes that wrap the Python code.
IN my opinion, Jython is exactly what you are looking at.
It is an implementation of Python within the JVM; as such, you can freely exchange objects and, for instance, inherit from a Java class (with some limitations).
Note that, its major strength point (being on top of of JVM) is also its major drawback, because it cannot use all (C)Python extension written in C (or in any other compiled language); this may have an impact on what you are willing to do with your text processing.
For more information about what is Jython, its potential and its limitations, I suggest you reading the Jython FAQ.
Simply run the Python interpreter as a subprocess from within Java.
Write your Python functionality as a proper script, which reads from stdin and writes to stdout.
Use the Java Runtime class to spawn a subprocess that runs your Python script. This is very simple to do and provides a very clean interface.
Edit
import simplejson
import sys
for request in sys.stdin.readlines():
args = simplejson.loads( request )
result = myFunction( args['this'], args['that'] )
sys.stdout.writeline( simplejson.dumps( result ) + "\n" )
The interface is simple, structured and very low overhead.
Remember to first check from those paying for the development that they're OK with the codebase needing a developer who knows both Python and Java from now on, and other cost and maintainability effects you've undoubtedly already accounted for.
See: http://www.acm.org/about/se-code 1.06, 2.03, 2.09, 4.03, 4.05, 6.07
Has anyone successfully "wrapped up" a C++ API in Java or .NET? I have an application that provides a C++ API for writing plug-ins. What I'd like to do is access that API from .NET or Java.
Would I need to use COM, or are there simpler/better alternatives?
If you have a very straight forward method signatures (i.e. methods that takes and returns primitive types such as int, char[], void* ... etc), it is reasonably easy to do so in .NET and still possible but a bit harder in Java with JNI.
However, if your class methods uses modern C++ programming techniques such as boost shared pointers and STL containers then it is a very different story. You will need to be really careful with memory management in this case.
EDIT:
It is gonna be even more interesting if the method has C++ template arguments because C++ template system is very different from C# or Java generics in that it is only a compile time mechanism. Basically this means the signature of the method or class is different every time you pass a different data type to the template argument. This made the method impossible to wrap in C# or Java.
I think that you may be asking to something similar to the Java Native Inferface It allows you to call code written in languages such as C++. I've never worked with it though. But you may want to check it out. Hope it helps.
On the Java side there are many options here .. this question actually is fairly close to what you are looking for provided your API can be boxed in a DLL, com or activex objects.
I personnaly used JIntegra to wrap API calls to office (Word) and used it directly witin Java. It did take some hacking to get the desired functionnality but we ended up making it work. The fidling was actually on the Word side, the actual integration was relatively easy.
If you are going to use .NET, I would recommend creating a C++/CLI wrapper. C++/CLI lets you mix the managed .NET code with native C++ code. There are some tricks to this approach, but it works very well in majority of cases.
Wikipedia entry has also some good links.
Using SWIG to handle the JNI stuff makes wrapping a cpp API and using it from Java quite painless.
Here's a SWIG tutorial
I'm the lead author on SlimDX. It may not be the largest open source interoperability project out there, but at 150K+ lines across the project, it's fairly substantial. It's written using C++/CLI, which is a Microsoft developed language, largely compatible with C++, that is designed to allow you to build wrappers. It works very well, with the caveat that it is Windows only. Mono will not take it. I don't know how much of a concern that is for you, but it's probably a better alternative than COM. Unfortunately there's a lot of tips and tricks that you have to basically come up with yourself. I've meant to blog about a lot of the ones we use on SlimDX, but somehow I never quite get around to it.
On the Java side, I believe you have to use JNI. Good luck with that. I've never talked to anyone who's worked with JNI without both of us laughing at it, in a bad "ouch that hurt" sort of way.
I maintain software that is implemented in C++ but must have interfaces in multiple languages including Java and .NET, but also including delphi and VB6. Plus, it has to work on multiple platforms (so Java needs to work on unix).
The way it was done was to use a single DLL exporting plain C functions using primitive types. For example, given an class Foo:
long MY_EXPORT_FLAG FooCreate()
{
return (long)new Foo();
}
void MY_EXPORT_FLAG FooDestroy(long Handle)
{
delete (Foo*)Handle;
}
void MY_EXPORT_FLAG BarMethod(long Handle, const char* pStr1, long* pReturnValue)
{
*pReturnValue = ((Foo*)Handle)->BarMethod( pStr1 );
}
Then your JNI/.NET/VB6/Delphi code implements language specific class wrappers, but call these C dll functions. Each class wrapper would contain a Handle, and pass it into the C functions.
This works quite well because most languages tend to use C as the lowest common denominator, so as long as you can export your interfaces through a thin C api you can build interfaces for other languages.
We export a C (and not C++) api in the DLL because function signatures are much more standardized across platforms.
Don't forget, that on C# and Java, you'll have to deal with multibyte string encoding, so you should figure out how to transcode your strings to/from your C++ code. This typically involves knowing about locales, or you can cheap out and just support UTF-8.