Develop Reference

JVM Error While Writing Data Frame to Oracle Database using parLapply

I want to parallelize my data writing process. I am writing a data frame to Oracle Database. This data has 4 million rows and 8 columns. It takes 6.5 hours without parallelizing.
When I try to go parallel, I get the error
Error in checkForRemoteErrors(val) :
7 nodes produced errors; first error: No running JVM detected. Maybe .jinit() would help.
I know this error. I can solve it when I work with single cluster. But I do not know how to tell other clusters the location of Java. Here is my code
Sys.setenv(JAVA_HOME='C:/Program Files/Java/jre1.8.0_181')
library(rJava)
library(RJDBC)
library(DBI)
library(compiler)
library(dplyr)
library(data.table)
jdbcDriver =JDBC("oracle.jdbc.OracleDriver",classPath="C:/Program Files/directory/ojdbc6.jar", identifier.quote = "\"")
jdbcConnection =dbConnect(jdbcDriver, "jdbc:oracle:thin:#//XXXXX", "YYYYY", "ZZZZZ")
By using Sys.setenv(JAVA_HOME='C:/Program Files/Java/jre1.8.0_181') I solve the same problem for single core. But when I go parallel
library(parallel)
no_cores <- detectCores() - 1
cl <- makeCluster(no_cores)
clusterExport(cl, varlist = list("jdbcConnection", "brand3.merge.u"))
clusterEvalQ(cl, .libPaths("C:/Users/onur.boyar/Documents/R/win-library/3.5"))
clusterEvalQ(cl, library(RJDBC))
clusterEvalQ(cl, library(rJava))
parLapply(cl, 1:length(brand3.merge.u$CELL_PH_NUM), function(x) dbSendUpdate(jdbcConnection, "INSERT INTO xxnvdw.an_cust_analytics VALUES(?,?,?,?,?,?,?,?)", brand3.merge.u[x, 1], brand3.merge.u[x,2], brand3.merge.u[x,3],brand3.merge.u[x,4],brand3.merge.u[x,5],brand3.merge.u[x,6],brand3.merge.u[x,7],brand3.merge.u[x,8]))
#brand3.merge.u is my data frame that I try to write.
I get the above error and I do not know how to set my Java location for other nodes.
I want to use parLapply since it is faster than foreach. Any help would be appreciated. Thanks!

JAVA_HOME environment variable
If the problem really is with the location of Java, you could set the environment variable in your .Renviron file. It is likely located in ~/.Renviron. Add a line to that file and this will be propagated to all R session that run via your user:
JAVA_HOME='C:/Program Files/Java/jre1.8.0_181'
Alternatively, you can just add that location to your PATH environment variable.
JVM Initialization via rJava
On the other hand the error message may point to just a JVM not being initialized, which you can solve with .jinit, a minimal example:
library(parallel)
cl <- makeCluster(detectCores())
parallel::parLapply(cl, 1:5, function(x) {
rJava::.jinit()
rJava::.jnew(class = "java/lang/Integer", x)$toString()
})
Working around Java use
This was not specifically asked, but you can also work around the need for Java dependency using ODBC drivers, which for Oracle should be accessible here:
con <- DBI::dbConnect(
odbc::odbc(),
Driver = "[your driver's name]",
...
)

Z3 Java Possible Memory Leak?

I have a strange memory problem with Java library of Z3 which I couldn't figure where the problem is. Oddly, I can't reproduce the problem on a Windows machine where I have Java 7 (I most probably have slightly older version of Z3 there though). The problem occurs on a MacOSx 10.6.8 with Java 6 and Z3 v4.3.2. I have an application that uses Z3 for analysis. I tracked the following piece of code as the (initial) source of the problem:
Symbol eNames = con.mkSymbol(domainName);
Symbol[] symbols = new Symbol[values.length];
for (int i = 0; i < values.length; i++) symbols[i] = con.mkSymbol(values[i]);
System.out.println("Before ENUMSORT");
//EnumSort eSort = con.mkEnumSort(domainName, values);
EnumSort eSort = con.mkEnumSort(eNames,symbols);
System.out.println("After ENUM SORT ...");
When I run the application I get the following after "Before ENUMSORT" is printed:
java(55938,0x100501000) malloc: *** error for object 0x10200f1b8: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
I know this is not a good way of debugging especially when there is a memory problem but it is very difficult to debug the code since it originates from JNI. When I look at the Z3 code here (https://github.com/Z3Prover/z3/blob/master/src/api/api_datatype.cpp) I couldn't figure what the source of the problem is. I assume that the method Z3_mk_enumeration_sort is called from mkEnumSort method in Java. When I change the call of mkEnumSort in my code to a form like
EnumSort eSort = con.mkEnumSort(domainName,new String[]{"X","Y"});
the problem seems gone. What do you think, what could be the source of the problem?
Any help is highly appreciated.

Run an ABCL code that uses cl-cppre

With reference to my previous question,
Executing a lisp function from Java
I was able to call lisp code from Java using ABCL.
But the problem is, the already existing lisp code uses CL-PPCRE package.
I can not compile the code as it says 'CL-PPCRE not found'.
I have tried different approaches to add that package,
including
1) how does one compile a clisp program which uses cl-ppcre?
2)https://groups.google.com/forum/#!topic/cl-ppcre/juSfOhEDa1k
Doesnot work!
Other thing is, that executing (compile-file aima.asd) works perfectly fine although it does also require cl-pprce
(defpackage #:aima-asd
(:use :cl :asdf))
(in-package :aima-asd)
(defsystem aima
:name "aima"
:version "0.1"
:components ((:file "defpackage")
(:file "main" :depends-on ("defpackage")))
:depends-on (:cl-ppcre))
The final java code is
interpreter.eval("(load \"aima/asdf.lisp\")");
interpreter.eval("(compile-file \"aima/aima.asd\")");
interpreter.eval("(compile-file \"aima/defpackage.lisp\")");
interpreter.eval("(in-package :aima)");
interpreter.eval("(load \"aima/aima.lisp\")");
interpreter.eval("(aima-load 'all)");
The error message is
Error loading C:/Users/Administrator.NUIG-1Z7HN12/workspace/aima/probability/domains/edit-nets.lisp at line 376 (offset 16389)
#<THREAD "main" {3A188AF2}>: Debugger invoked on condition of type READER-ERROR
The package "CL-PPCRE" can't be found.
[1] AIMA(1):
Can anyone help me?

You need to load cl-ppcre before you can use it. You can do that by using (asdf:load-system :aima), provided that you put both aima and cl-ppcre into locations that your ASDF searches.

I used QuickLisp to add cl-ppcre (because nothing else worked for me).
Here is what I did
(load \"~/QuickLisp.lisp\")")
(quicklisp-quickstart:install)
(load "~/quicklisp/setup.lisp")
(ql:quickload :cl-ppcre)
First 2 lines are only a one time things. Once quickLisp is installed you can start from line 3.

Error writing xlsx in R: Could not initialize class sun.java2d.Disposer

I'm using the xlsx package to write Excel files in R:
addPicture('trend_indirect.png' ,sheet1)
addDataFrame(df.ssis_duplmonth ,sheet1, startRow=22)
addDataFrame(df.ssis_dupltrans ,sheet1, startRow=35)
addDataFrame(df.ssis_duplmonth_dir, sheet2, startRow=22)
addDataFrame(df.ssis_dupltrans_dir, sheet2, startRow=55)
saveWorkbook(wb, file="SSIS_import_controls.xlsx")
At this point I get the following error:
> addDataFrame(df.ssis_duplmonth ,sheet1, startRow=22)
Error in .jcall("RJavaTools", "Z", "hasField", .jcast(x, "java/lang/Object"), :
java.lang.NoClassDefFoundError: Could not initialize class sun.java2d.Disposer
R version 2.15.2, 32bit.
Thanks
Edit: I can't really make it reproducible as probably the issue is in my settings but I get the error when I run this:
library('xlsx')
df.test <- iris[1:5, ]
wb <- createWorkbook()
sheet1 <- createSheet(wb, 'Indirect Sales')
addPicture('trend_indirect.png' ,sheet1)
addDataFrame(df.test ,sheet1, startRow=22)
saveWorkbook(wb, file="stack_test.xlsx")
The image is just a simple ggplot graph saved in png. Thanks

Try installing libxtst. That solved a similar problem for me.
I also installed fontconfig and libcups in the course of solving my issue, in case it wasn't libxtst that fixed it.

I was with the same exception but running a Java program using Ubuntu 12.
I've installed libxtst6 and add this java parameter to my JAVA_OPTS variable: -Djava.awt.headless=true
Then it works fine.

Trouble using java class within jruby

I'm trying to use Java Opencl from within jruby, but am encountering a problem which I can't solve, even with much google searching.
require 'java'
require 'JOCL-0.1.7.jar'
platforms = org.jocl.cl_platform_id.new
puts platforms.class
org.jocl.CL.clGetPlatformIDs(1, platforms, nil)
when I run this code using: jruby test.rb
I get the following error, when the last line is uncommented:
#<Class:0x10191777e>
TypeError: cannot convert instance of class org.jruby.java.proxies.ConcreteJavaP
roxy to class [Lorg.jocl.cl_platform_id;
LukeTest at test.rb:29
(root) at test.rb:4
Just wondering whether anyone has an idea on how to solve this problem?
EDIT:
ok so I think I've solved the first part of this problem by making platforms an array:
platforms = org.jocl.cl_platform_id[1].new
but that led to this error when adding the next couple of lines:
context_properties = org.jocl.cl_context_properties.new()
context_properties.addProperty(org.jocl.CL::CL_CONTEXT_PLATFORM, platforms[0])
CodegenUtils.java:98:in `human': java.lang.NullPointerException
from CodegenUtils.java:152:in `prettyParams'
from CallableSelector.java:462:in `argumentError'
from CallableSelector.java:436:in `argTypesDoNotMatch'
from RubyToJavaInvoker.java:248:in `findCallableArityTwo'
from InstanceMethodInvoker.java:66:in `call'
from CachingCallSite.java:332:in `cacheAndCall'
from CachingCallSite.java:203:in `call'
from test.rb:36:in `module__0$RUBY$LukeTest'
from test.rb:-1:in `module__0$RUBY$LukeTest'
from test.rb:4:in `__file__'
from test.rb:-1:in `load'
from Ruby.java:679:in `runScript'
from Ruby.java:672:in `runScript'
from Ruby.java:579:in `runNormally'
from Ruby.java:428:in `runFromMain'
from Main.java:278:in `doRunFromMain'
from Main.java:198:in `internalRun'
from Main.java:164:in `run'
from Main.java:148:in `run'
from Main.java:128:in `main'
for some reason when I print the class of platforms[0] it's listed as NilClass!?

You are overlooking a very simple mistake. You write
platforms = org.jocl.cl_platform_id.new
but that line creates a single instance of the class org.jocl.cl_platform_id. You then pass that as the second parameter to org.jocl.CL.clGetPlatformIDs in
org.jocl.CL.clGetPlatformIDs(1, platforms, nil)
and that doesn't work, because the second argument of the method requires an (empty) array of org.jocl.cl_platform_id objects.
What the error says is: "I have something that is a proxy for a Java object and I can't turn it into an an array of org.jocl.cl_platform_id objects, as you are asking me to do.
If you just say
platforms = []
and pass that in, it might just work :).