Develop Reference

Java Weka prediction throws Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 22, Size: 22

I am trying to train a classifier on labeled data (data with the outcome vector included) and run predictions on unlabeled data using the Weka library in Java.
I've investigated every case I can find online of someone receiving the error
Exception in thread "main" java.lang.IndexOutOfBoundsException
when doing this and they seem to all be caused by either
a mis-match between the training and prediction data structures or
improperly handled missing values
The solution to the first cause is to make the data structures match and verify this with train.equalHeaders(test)). As far as I can tell the data structures are an exact match and the result of equalHeaders is true. There is no missing data in the data sets I've been using for development / testing.
My training data is the famous iris data set, which I produced by using the copy that's built into R by calling data(iris); write.csv(iris, "iris.csv", row.names = F). My prediction (test) data is the exact same data set, with the last column removed to simulate unlabeled test data. I've tried reading these files as .csv and from SQL Server tables and have encountered the same result.
I have tried 2 different ways of running the predictions; the way that's currently uncommented and the .evaluateModel method, both of which have the same error.
I have also tried changing the algorithm but this does not affect the error.
I have also printed the data to the screen and examined all of the available summary / diagnostic methods, all of which look as they should be.
The key part of my code is as follows. Originally I posted the entire code, so if you'd like to see that it's available in the edit history.
//Add dummy outcome attribute to make shape match training
Add filter1;
filter1 = new Add();
filter1.setAttributeIndex("last");
filter1.setNominalLabels("'\"setosa\"','\"versicolor\"','\"virginica\"'");
filter1.setAttributeName("\"Species\"");
filter1.setInputFormat(test);
test = Filter.useFilter(test, filter1);
Instances newTest = Filter.useFilter(test, filter); // create new test set
// set class attribute
newTest.setClassIndex(newTest.numAttributes() - 1);
// create copy
Instances labeled = new Instances(newTest);
System.out.println("check headers: " + newTrain.equalHeaders(newTest));
System.out.println(newTest); // throws the error if included
// label instances
for (int i = 0; i < newTest.numInstances(); i++) { // properly indexed
System.out.println(i);
double clsLabel = rf.classifyInstance(newTest.instance(i)); //throws the error if the earlier print is not included
labeled.instance(i).setClassValue(clsLabel);
}
}
}
The full error is:
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 22, Size: 22
java.util.ArrayList.rangeCheck(ArrayList.java:653)
java.util.ArrayList.get(ArrayList.java:429)
weka.core.Attribute.value(Attribute.java:735)
weka.core.AbstractInstance.stringValue(AbstractInstance.java:668)
weka.core.AbstractInstance.stringValue(AbstractInstance.java:644)
weka.core.AbstractInstance.toString(AbstractInstance.java:756)
weka.core.DenseInstance.toStringNoWeight(DenseInstance.java:330)
weka.core.AbstractInstance.toStringMaxDecimalDigits(AbstractInstance.java:692)
weka.core.AbstractInstance.toString(AbstractInstance.java:712)
java.lang.String.valueOf(String.java:2981)
java.lang.StringBuffer.append(StringBuffer.java:265)
weka.core.Instances.stringWithoutHeader(Instances.java:1734)
weka.core.Instances.toString(Instances.java:1718)
java.lang.String.valueOf(String.java:2981)
java.io.PrintStream.println(PrintStream.java:821)
weka.core.Tee.println(Tee.java:484)
myWeka.myWeka.main(myWeka.java:262)
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at weka.core.Attribute.value(Attribute.java:735)
at weka.core.AbstractInstance.stringValue(AbstractInstance.java:668)
at weka.core.AbstractInstance.stringValue(AbstractInstance.java:644)
at weka.core.AbstractInstance.toString(AbstractInstance.java:756)
at weka.core.DenseInstance.toStringNoWeight(DenseInstance.java:330)
at weka.core.AbstractInstance.toStringMaxDecimalDigits(AbstractInstance.java:692)
at weka.core.AbstractInstance.toString(AbstractInstance.java:712)
at java.lang.String.valueOf(String.java:2981)
at java.lang.StringBuffer.append(StringBuffer.java:265)
at weka.core.Instances.stringWithoutHeader(Instances.java:1734)
at weka.core.Instances.toString(Instances.java:1718)
at java.lang.String.valueOf(String.java:2981)
at java.io.PrintStream.println(PrintStream.java:821)
at weka.core.Tee.println(Tee.java:484)
at myWeka.myWeka.main(myWeka.java:262)
C:\Users\eggwhite\AppData\Local\NetBeans\Cache\8.1\executor-snippets\run.xml:53:
Java returned: 1
BUILD FAILED (total time: 23 seconds)
The error is thrown by double clsLabel = rf.classifyInstance(newTest.instance(i)); unless I include the line System.out.println(newTest); for diagnostic purposes, in which case the same error is thrown by that line.

Compiling my project with AOT throws a NullPointerException

I'm trying to build an uberjar using lein uberjar. During compiling, the following gets thrown:
Exception in thread "main" java.lang.NullPointerException, compiling:(/tmp/form-init8223412427040046857.clj:1:73)
at clojure.lang.Compiler.load(Compiler.java:7391)
at clojure.lang.Compiler.loadFile(Compiler.java:7317)
at clojure.main$load_script.invokeStatic(main.clj:275)
at clojure.main$init_opt.invokeStatic(main.clj:277)
at clojure.main$init_opt.invoke(main.clj:277)
at clojure.main$initialize.invokeStatic(main.clj:308)
at clojure.main$null_opt.invokeStatic(main.clj:342)
at clojure.main$null_opt.invoke(main.clj:339)
at clojure.main$main.invokeStatic(main.clj:421)
at clojure.main$main.doInvoke(main.clj:384)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
at clojure.string$lower_case.invokeStatic(string.clj:217)
at clojure.string$lower_case.invoke(string.clj:213)
at kappa.joke_classifier$is_jokable_QMARK_.invokeStatic(joke_classifier.clj:32)
at kappa.joke_classifier$is_jokable_QMARK_.invoke(joke_classifier.clj:29)
at kappa.core$maybe_joke.invokeStatic(core.clj:47)
at kappa.core$maybe_joke.invoke(core.clj:45)
at clojure.core$run_BANG_$fn__7276.invoke(core.clj:7393)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:341)
at clojure.core$reduce.invokeStatic(core.clj:6544)
at clojure.core$run_BANG_.invokeStatic(core.clj:7388)
at clojure.core$run_BANG_.invoke(core.clj:7388)
at kappa.core$run.invokeStatic(core.clj:58)
at kappa.core$run.invoke(core.clj:53)
at kappa.core$_main.invokeStatic(core.clj:66)
at kappa.core$_main.doInvoke(core.clj:61)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.Var.invoke(Var.java:375)
at user$eval5.invokeStatic(form-init8223412427040046857.clj:1)
at user$eval5.invoke(form-init8223412427040046857.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6927)
at clojure.lang.Compiler.eval(Compiler.java:6917)
at clojure.lang.Compiler.load(Compiler.java:7379)
... 14 more
The code using string/lower-case looks as follows:
(defn is-jokable? [msg]
(and
(> 30 (count msg))
(= :positive (.classify classifier (str/lower-case msg)))
(< 1 (:positive (.probabilities classifier (str/lower-case msg))))))
As can be seen from the stack trace, my -main function calls a function that eventually calls is-jokable, and str/lower-case seems to be called with nil. If I understand correctly this is because of the AOT compilation happening for uberjars, but I'm not quite sure why exactly this problem occurs... I've already tried to read up on AOT, but didn't find anything helpful. Can you explain the inner workings of AOT to me?

I'm almost embarrassed I have posted this, it must have been too late in the night... ;)
I got completely distracted by the compiling and invokeStatic hints in the stacktrace, and just now found that this was indeed an everyday bug where I expected a map to contain a key/value pair that wasn't existent, leading to nil being passed to the function in question.
Thanks for the helpful comments!

NullPointerException on return new[]

I have a very strange problem with NullPointerException. Code example is as follows:
...
... public String[] getParams(...) {
... ...
... ...
143 return new String[] {
144 getUserFullName(),
145 StringUtil.formatDate(sent),
. tiltu,
. StringUtil.initCap(user.getName()),
. vuosi.toString(),
. asiatyyppi[0] + " " + lisatiedot[0],
. asiatyyppi[1] + " " + lisatiedot[1],
. alaviitteet[0],
152 alaviitteet[1]};
153 }
Now, I have got an issue from production having a stack trace:
java.lang.NullPointerException
at package.EmailService.getParams(EmailService.java:143)
...
I am unable to produce that kind of stack trace myself. It maybe some environment issue that for some reason line numbers don't match. If I have null references on any variable stack trace points to that specific line but never to line 143.
But what I want to ask is: is it possible to produce NullPointerException at line 143 specifically?

The line number in the stack trace comes from the LineNumberTable attribute in the class file. (See JVM specification)
It would be no problem to output the right line number for a subexpression - all the compiler has to do is to say that from byte-code index x to y, there is a correspondence with source code line z.
But up to and including Java 1.7 there was a bug in the compiler, that was fixed in 1.8:
https://bugs.openjdk.java.net/browse/JDK-7024096
A DESCRIPTION OF THE PROBLEM : linenumbertable in compiled code only
has line numbers for starts of statements. When a statement is using
method chaining or other "fluent Interface" style API's a single
statement can span tens of lines and contain hundreds on method
invocations.
Currently an exception throw from within any of these methods will
have the linenumber of the first line of the enclosing statement which
makes it very hard to debug which method call is having a problem.
linnumbertable should have correct line numbers for every method
invocation.
--
BT2:EVALUATION
It seems simple enough to fix this, but its kinda risky at the end of
the jdk7 development cycle, targetting to jdk8.
So in 1.7, you would get the wrong reported line number for these kind of subexpressions (if they occurred within the same method though - if you invoked another method in a subexpression, and that other method caused a NullPointerException, you would see it reported there - this is probably why the bug isn't always a big problem)
One way you could work around this is by taking the Java 8 compiler to compile your source code, and use the flags javac -source 1.7 -target 1.7. But it would be better and safer to upgrade your prod environment to 1.8.

Consider your original code which defines a new String array:
return new String[] {
getUserFullName(),
StringUtil.formatDate(sent),
tiltu,
StringUtil.initCap(user.getName()),
vuosi.toString(),
asiatyyppi[0] + " " + lisatiedot[0],
asiatyyppi[1] + " " + lisatiedot[1],
alaviitteet[0],
alaviitteet[1]};
}
If any of the elements of the inline array should trigger a NullPointerException the JVM will interpret the Exception as having occurred on the line where the definition began. In other words, the JVM will view the above code as being the same as:
return new String[] { getUserFullName(), StringUtil.formatDate(sent), tiltu, StringUtil.initCap user.getName()), vuosi.toString(), asiatyyppi[0] + " " + lisatiedot[0], asiatyyppi[1] + " " + lisatiedot[1], alaviitteet[0], alaviitteet[1]}; }
where everything is on one line.
If you really want to handle NullPointerExceptions here, you should define the variables outside the instantiaition.

Scanner class in Java5 throw java.lang.NullPointerException

I am using scanner class in java5, and the following code will throw an exception:
Scanner scanner = new Scanner
(new File(args[0]));
int dealId;
while (scanner.hasNextLine()) {
dealId = scanner.nextInt();
System.out.println(dealId);
}
scanner.close();
The stacktrace is:
Exception in thread "main" java.lang.NullPointerException
at java.util.regex.Matcher.toMatchResult(libgcj.so.10)
at java.util.Scanner.myCoreNext(libgcj.so.10)
at java.util.Scanner.myPrepareForNext(libgcj.so.10)
at java.util.Scanner.myNextLine(libgcj.so.10)
at java.util.Scanner.hasNextLine(libgcj.so.10)
Does anybody knows what caused this exception?

The GCJ Home Page suggest it "supports most of the 1.4 libraries plus some 1.5 additions. "
Scanner was added in version 1.5 and I suspect you have hit a piece of functionality GCJ doesn't support. You need to try something different to see what you can get to work.
Is there any reason you are not using OpenJDK/Oracle Java 6 or 7? (Please don't say its for performance reasons ;)

I reproduced the error and found a work around
Here is the code, compiled on x86_64 GNU/Linux, Fedora with Java 1.5.0:
Scanner r = new Scanner(f, "ISO-8859-1");
while(r.hasNext()){
String line = r.nextLine(); //The guts of nextLine(), specifically:
//Matcher.toMatchResult bubbles up a
//nullPointerException
}
The file just contains two ascii words separated by a newline. The runtime Exception only occurs when nextLine processes the last line of the file, whether it has characters or not:
java.lang.NullPointerException
at java.util.regex.Matcher.toMatchResult(libgcj.so.10)
at java.util.Scanner.myCoreNext(libgcj.so.10)
at java.util.Scanner.myPrepareForNext(libgcj.so.10)
at java.util.Scanner.myNextLine(libgcj.so.10)
at java.util.Scanner.nextLine(libgcj.so.10)
at Main.parseFile(Main.java:1449)
at Main.construct(Main.java:1420)
at Main.populateBlogPosts(Main.java:1399)
at Main.main(Main.java:263)
Here is a bug report on this issue: https://bugs.openjdk.java.net/browse/JDK-6178785
Diagnosis
It's a bug in libgcj.so.10, perfectly legitimate ascii input as well as blankstring causes it to puke an NPE on the last line of a file.
Workaround
Since this bug only occurs on the very last line of the file, the hacky workaround is to initially make sure there is at least one newline at the end of the file, then catch and ignore the nullPointerException bubbled up from toMatchResult and exit the loop when that happens.

YUICompressor crashes - stackoverflow error

I frequently get what appears to be a stackoverflow error ;-) from YUICompressor. The following is the first part of thousands of error lines that come from attempting to compress a 24074 byte css stylesheet (not the "Caused by java.lang.StackOverflowError about 8 lines down):
iMac1:src jas$ min ../style2.min.css style2.css
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.yahoo.platform.yui.compressor.Bootstrap.main(Bootstrap.java:21)
Caused by: java.lang.StackOverflowError
at java.lang.Character.codePointAt(Character.java:2335)
at java.util.regex.Pattern$CharProperty.match(Pattern.java:3344)
at java.util.regex.Pattern$Branch.match(Pattern.java:4114)
... (plus 1021 more error lines)
The errors happen usually after adding a couple of lines to the file getting compressed. The css is fine, and works perfectly in the uncompressed format. I don't see a particular pattern to the types of selectors added to the file that cause the errors. In this case, adding the following selector to a previously compressible file resulted in the errors:
#thisisatest
{
margin-left:87px;
}
I am wondering if there is perhaps a flag to java to enlarge the stack that might help. Or if that is not the problem, what is?
EDIT:
As I was posting this question, it dawned on me that I should check the java command to see if there was a parameter to enlarge the stack. Turns out that it is -Xssn, where "n" is a parameter to indicate the stack size. Its default value is 512k. So I tried 1024k but that still led to the stackoverflow. Trying 2048k works however, and I think this could be the solution.
EDIT 2:
While I no longer use this method for minification any longer, to be more specific here is the full command (which I have set up as a shell alias), showing how the -Xss2048k parameter is used:
java -Xss2048k -jar ~/Documents/RepHunter/Website\ Materials/Code/Third\ Party\ Libraries/YUI\ Compressor/yuicompressor-2.4.8.jar --type css -o

As posted in my edit, the solution was to add the parameters to the java command. The clue was the error line at the 5-th "at" line, as follows:
at com.yahoo.platform.yui.compressor.Bootstrap.main(Bootstrap.java:21)
Caused by: java.lang.StackOverflowError
Seeing that the issue was a "StackOverlowError" ;-) gave the suggestion to try to increase the stack size. The default is 512k. My first try of 1024k did not work. However increasing it to 2048k did work, and I have had no further issues.