Develop Reference

apache PIG with datafu: Cannot resolve UDF's

I'm trying the quickstart from here: http://datafu.incubator.apache.org/docs/datafu/getting-started.html
I tried nearly everything, but I'm sure it must be my fault somewhere. I tried already:
exporting PIG_HOME, CLASSPATH, PIG_CLASSPATH
starting pig with -cpdatafu-pig-incubating-1.3.0.jar
registering datafu-pig-incubating-1.3.0.jar locally and in hdfs => both succesful (at least no error shown)
nothing helped
Trying this on pig:
register datafu-pig-incubating-1.3.0.jar
DEFINE Median datafu.pig.stats.StreamingMedian();
data = load '/user/hduser/numbers.txt' using PigStorage() as (val:int);
data2 = FOREACH (GROUP data ALL) GENERATE Median(data);
or directly
data2 = FOREACH (GROUP data ALL) GENERATE datafu.pig.stats.StreamingMedian(data);
I get this name-resolve error:
2016-06-04 17:22:22,734 [main] ERROR org.apache.pig.tools.grunt.Grunt
- ERROR 1070: Could not resolve datafu.pig.stats.StreamingMedian using imports: [, java.lang., org.apache.pig.builtin.,
org.apache.pig.impl.builtin.] Details at logfile:
/home/hadoop/pig_1465053680252.log
When I look into the datafu-pig-incubating-1.3.0.jar it looks OK, everything in place. I also tried some Bag functions, same error then.
I think it's kind of a noob-error which I just don't see (as I did not find particular answers for datafu in SO or google), so thanks in advance for shedding some light on this.

Pig script is proper, the only thing that could break is that while registering datafu there were some class dependencies that coudn't been met.
Try to run locally (pig -x local) and see a detailed log.
Check also the version of pig - it should be newer than 0.14.0.

SURF and SIFT algorithms doesn't work in OpenCV 3.0 Java

I am using OpenCV 3.0 (the latest version) in Java, but when I use SURF algorithm or SIFT algorithm it doesn't work and throws Exception which says: OpenCV Error: Bad argument (Specified feature detector type is not supported.) in cv::javaFeatureDetector::create
I have googled, but the answers which was given to this kind of questions did not solve my problem. If anyone knows about this problem please let me know.
Thanks in advance!
Update: The code below in third line throws exception.
Mat img_object = Imgcodecs.imread("data/img_object.jpg");
Mat img_scene = Imgcodecs.imread("data/img_scene.jpg");
FeatureDetector detector = FeatureDetector.create(FeatureDetector.SURF);
MatOfKeyPoint keypoints_object = new MatOfKeyPoint();
MatOfKeyPoint keypoints_scene = new MatOfKeyPoint();
detector.detect(img_object, keypoints_object);
detector.detect(img_scene, keypoints_scene);

If you compile OpenCV from source, you can fix the missing bindings by editing opencv/modules/features2d/misc/java/src/cpp/features2d_manual.hpp yourself.
I fixed it by making the following changes:
(line 6)
#ifdef HAVE_OPENCV_FEATURES2D
#include "opencv2/features2d.hpp"
#include "opencv2/xfeatures2d.hpp"
#include "features2d_converters.hpp"
...(line 121)
case SIFT:
fd = xfeatures2d::SIFT::create();
break;
case SURF:
fd = xfeatures2d::SURF::create();
break;
...(line 353)
case SIFT:
de = xfeatures2d::SIFT::create();
break;
case SURF:
de = xfeatures2d::SURF::create();
break;
The only requirement is that you build opencv_contrib optional module along with your sources (you can download the git project from https://github.com/Itseez/opencv_contrib and just set its local path on opencv's ccmake settings.
Oh, and keep in mind that SIFT and SURF are non-free software ^^;

That is because they are not free in newer versions of OpenCV (3+). I faced that problem some time ago. You have to:
Download OpenCV (if you have not)
Download the nonfree part from opencv github repo
Generate the makefiles with cmake -DBUILD_SHARED_LIBS=OFF specifying the nonfree part with DOPENCV_EXTRA_MODULES_PATH=../opencv_contrib/modules option and build with make -j8 (or whatever Java version you use)
Edit features2d_manual.hpp file, including opencv2/xfeatures2d.hpp and including the necessary code for SIFT and SURF case, which are commented and not defined:
fd=xfeatures2d::SIFT::create(); for SIFT descriptor and de = xfeatures2d::SIFT::create(); for SIFT extractor. Do the same for SURF if you want to use it too.
I wrote this post explaining step by step how to compile the non-free OpenCV part in order to use privative tools like SIFT or SURF.
Compile OpenCV non-free part.

I believe changing features2d module (FeatureDetector class or any other classes from features2d_manual.hpp) to enable methods from OpenCV contrib modules is less attractive approach because it leads to circular dependency between the "core" OpenCV and extensions (which can be non-free or experimental).
There is another way to fix this issue without affecting feature2d classes. Making changes in xfeatures2d CMakeLists.txt as described here leads to generation of java wrappers for SIFT and SURF - opencv-310.jar has org.opencv.xfeatures2d package now. Some fix was required in /opencv/modules/java/generator/gen_java.py. Namely inserted 2 lines as shown below:
def addImports(self, ctype):
if ctype.startswith('vector_vector'):
self.imports.add("org.opencv.core.Mat")
self.imports.add("org.opencv.utils.Converters")
self.imports.add("java.util.List")
self.imports.add("java.util.ArrayList")
self.addImports(ctype.replace('vector_vector', 'vector'))
elif ctype.startswith('Feature2D'): #added
self.imports.add("org.opencv.features2d.Feature2D") #added
elif ctype.startswith('vector'):
self.imports.add("org.opencv.core.Mat")
self.imports.add('java.util.ArrayList')
if type_dict[ctype]['j_type'].startswith('MatOf'):
self.imports.add("org.opencv.core." + type_dict[ctype]['j_type'])
else:
self.imports.add("java.util.List")
self.imports.add("org.opencv.utils.Converters")
self.addImports(ctype.replace('vector_', ''))
After these changes wrappers are generated successfully. However the main problem still remains, how to use these wrappers from Java )). For example SIFT.create() gives the pointer to a new SIFT class but calling any class method (for instance detect()) crashes Java. I also noticed that using MSER.create() directly from Java leads to the same crash.
So it looks like the problem is isolated to the way how Feature2D.create() methods are wrapped in Java. The solution seems to be the following (again, changing /opencv/modules/java/generator/gen_java.py):
Find the string:
ret = "%(ctype)s* curval = new %(ctype)s(_retval_);return (jlong)curval->get();" % { 'ctype':fi.ctype }
Replace it with the following:
ret = "%(ctype)s* curval = new %(ctype)s(_retval_);return (jlong)curval;" % { 'ctype':fi.ctype }
Rebuild the opencv. That is it, all create() methods will start working properly for all children of Feature2D class including experimental and non-free methods. FeatureDescriptor/DescriptorExtractor wrappers can be deprecated I think as Feature2D is much easier to use.
BUT! I'm not sure if the suggested fix is safe for other OpenCV modules. Is there a scenario when (jlong)curval needs to be dereferenced? It looks like the same fix was suggested already here.

Run an ABCL code that uses cl-cppre

With reference to my previous question,
Executing a lisp function from Java
I was able to call lisp code from Java using ABCL.
But the problem is, the already existing lisp code uses CL-PPCRE package.
I can not compile the code as it says 'CL-PPCRE not found'.
I have tried different approaches to add that package,
including
1) how does one compile a clisp program which uses cl-ppcre?
2)https://groups.google.com/forum/#!topic/cl-ppcre/juSfOhEDa1k
Doesnot work!
Other thing is, that executing (compile-file aima.asd) works perfectly fine although it does also require cl-pprce
(defpackage #:aima-asd
(:use :cl :asdf))
(in-package :aima-asd)
(defsystem aima
:name "aima"
:version "0.1"
:components ((:file "defpackage")
(:file "main" :depends-on ("defpackage")))
:depends-on (:cl-ppcre))
The final java code is
interpreter.eval("(load \"aima/asdf.lisp\")");
interpreter.eval("(compile-file \"aima/aima.asd\")");
interpreter.eval("(compile-file \"aima/defpackage.lisp\")");
interpreter.eval("(in-package :aima)");
interpreter.eval("(load \"aima/aima.lisp\")");
interpreter.eval("(aima-load 'all)");
The error message is
Error loading C:/Users/Administrator.NUIG-1Z7HN12/workspace/aima/probability/domains/edit-nets.lisp at line 376 (offset 16389)
#<THREAD "main" {3A188AF2}>: Debugger invoked on condition of type READER-ERROR
The package "CL-PPCRE" can't be found.
[1] AIMA(1):
Can anyone help me?

You need to load cl-ppcre before you can use it. You can do that by using (asdf:load-system :aima), provided that you put both aima and cl-ppcre into locations that your ASDF searches.

I used QuickLisp to add cl-ppcre (because nothing else worked for me).
Here is what I did
(load \"~/QuickLisp.lisp\")")
(quicklisp-quickstart:install)
(load "~/quicklisp/setup.lisp")
(ql:quickload :cl-ppcre)
First 2 lines are only a one time things. Once quickLisp is installed you can start from line 3.

Starting JVM for Inline

I have a Perl script that uses Inline::Java and just has to fork (it is a server and I want it to handle multiple connections simultaneously).
So I wanted to implement this solution which makes use of a shared JVM with SHARED_JVM => 1. Since the JVM is not shutdown when the script exits, I want to reuse it with START_JVM => 0. But since it might just be the first time I start the server, I would also like to have a BEGIN block make sure a JVM is running before calling use Inline.
My question is very simple, but I couldn't find any answer on the web: How do I simply start a JVM? I've looked at man java and there seems to be just no option that means "start and just listen for connections".
Here is a simplified version of what I'm trying to do in Perl, if this helps:
BEGIN {
&start_jvm unless &jvm_is_running;
}
use Inline (
Java => 'STUDY',
SHARED_JVM => 1,
START_JVM => 0,
STUDY => ['JavaStuff'],
);
if (fork) {
JavaStuff->do_something;
wait;
}
else {
Inline::Java::reconnect_JVM();
JavaStuff->do_something;
}
What I need help with is writing the start_jvm subroutine.

If you've got a working jvm_is_running function, just use it to determine whether Inline::Java should start the JVM.
use Inline (
Java => 'STUDY',
SHARED_JVM => 1,
START_JVM => jvm_is_running() ? 0 : 1,
STUDY => ['JavaStuff'],
);

Thanks to details provided by tobyink, I am able to answer my own question, which was based on a the erroneous assumption that the JVM itself provides a server and a protocole.
As a matter of fact, one major component of Inline::Java is a server, written in Java, which handles request by the Inline::Java::JVM client, written in Perl.
Therefore, the command-line to launch the server is:
$ java org.perl.inline.java.InlineJavaServer <DEBUG> <HOST> <PORT> <SHARED_JVM> <PRIVATE> <NATIVE_DOUBLES>
where all parameters correspond to configuration options described in the Inline::Java documentation.
Therefore, in my case, the start_jvm subroutine would be:
sub start_jvm {
system
'java org.perl.inline.java.InlineJavaServer 0 localhost 7891 true false false';
}
(Not that it should be defined: tobyink's solution, while it did not directly address the question I asked, is much better.)
As for the jvm_is_running subroutine, this is how I defined it:
use Proc::ProcessTable;
use constant {
JAVA => 'java',
INLINE_SERVER => 'org.perl.inline.java.InlineJavaServer',
};
sub jvm_is_running {
my $pt = new Proc::ProcessTable;
return grep {
$_->fname eq JAVA && ( split /\s/, $_->cmndline )[1] eq INLINE_SERVER
} #{ $pt->table };
}

Human editable JSON-like or YAML-like program configuration in Java

Is there a Java library similar to libconfig for C++, where the config file is stored in a JSON-like format that can be edited by humans, and later read from the program?
I don't want to use Spring or any of the larger frameworks. What I'm looking for is a small, fast, self-contained library. I looked at java.util.Properties, but it doesn't seem to support hierarchical/nested config data.

I think https://github.com/typesafehub/config is exactly what you are looking for. The format is called HOCON for Human-Optimized Config Object Notation and it a super-set of JSON.
Examples of HOCON:
HOCON that is also valid JSON:
{
"foo" : {
"bar" : 10,
"baz" : 12
}
}
HOCON also supports standard properties format, so the following is valid as well:
foo.bar=10
foo.baz=12
One of the features I find very useful is inheritance, this allows you to layer configurations. For instance a library would have a reference.conf, and the application using the library would have an application.conf. The settings in the application.conf will override the defaults in reference.conf.
Standard Behavior for loading configs:
The convenience method ConfigFactory.load() loads the following
(first-listed are higher priority):
system properties application.conf (all resources on classpath with
this name)
application.json (all resources on classpath with this
name)
application.properties (all resources on classpath with this
name)
reference.conf (all resources on classpath with this name)

I found this HOCON example:
my.organization {
project {
name = "DeathStar"
description = ${my.organization.project.name} "is a tool to take control over whole world. By world I mean couch, computer and fridge ;)"
}
team {
members = [
"Aneta"
"Kamil"
"Lukasz"
"Marcin"
]
}
}
my.organization.team.avgAge = 26
to read values:
val config = ConfigFactory.load()
config.getString("my.organization.project.name") // => DeathStar
config.getString("my.organization.project.description") // => DeathStar is a tool to take control over whole world. By world I mean couch, computer and fridge ;)
config.getInt("my.organization.team.avgAge") // => 26
config.getStringList("my.organization.team.members") // => [Aneta, Kamil, Lukasz, Marcin]
Reference: marcinkubala.wordpress.com

Apache Commons Configuration API and Constretto seem to be somewhat popular and support multiple formats (no JSON mentioned, though). I've personally never tried either, so YMMV.

There's a Java library to handle JSON files if that's what you're looking for:
http://www.json.org/java/index.html
Check out other tools on the main page:
http://json.org/