Develop Reference

JVM Error While Writing Data Frame to Oracle Database using parLapply

I want to parallelize my data writing process. I am writing a data frame to Oracle Database. This data has 4 million rows and 8 columns. It takes 6.5 hours without parallelizing.
When I try to go parallel, I get the error
Error in checkForRemoteErrors(val) :
7 nodes produced errors; first error: No running JVM detected. Maybe .jinit() would help.
I know this error. I can solve it when I work with single cluster. But I do not know how to tell other clusters the location of Java. Here is my code
Sys.setenv(JAVA_HOME='C:/Program Files/Java/jre1.8.0_181')
library(rJava)
library(RJDBC)
library(DBI)
library(compiler)
library(dplyr)
library(data.table)
jdbcDriver =JDBC("oracle.jdbc.OracleDriver",classPath="C:/Program Files/directory/ojdbc6.jar", identifier.quote = "\"")
jdbcConnection =dbConnect(jdbcDriver, "jdbc:oracle:thin:#//XXXXX", "YYYYY", "ZZZZZ")
By using Sys.setenv(JAVA_HOME='C:/Program Files/Java/jre1.8.0_181') I solve the same problem for single core. But when I go parallel
library(parallel)
no_cores <- detectCores() - 1
cl <- makeCluster(no_cores)
clusterExport(cl, varlist = list("jdbcConnection", "brand3.merge.u"))
clusterEvalQ(cl, .libPaths("C:/Users/onur.boyar/Documents/R/win-library/3.5"))
clusterEvalQ(cl, library(RJDBC))
clusterEvalQ(cl, library(rJava))
parLapply(cl, 1:length(brand3.merge.u$CELL_PH_NUM), function(x) dbSendUpdate(jdbcConnection, "INSERT INTO xxnvdw.an_cust_analytics VALUES(?,?,?,?,?,?,?,?)", brand3.merge.u[x, 1], brand3.merge.u[x,2], brand3.merge.u[x,3],brand3.merge.u[x,4],brand3.merge.u[x,5],brand3.merge.u[x,6],brand3.merge.u[x,7],brand3.merge.u[x,8]))
#brand3.merge.u is my data frame that I try to write.
I get the above error and I do not know how to set my Java location for other nodes.
I want to use parLapply since it is faster than foreach. Any help would be appreciated. Thanks!

JAVA_HOME environment variable
If the problem really is with the location of Java, you could set the environment variable in your .Renviron file. It is likely located in ~/.Renviron. Add a line to that file and this will be propagated to all R session that run via your user:
JAVA_HOME='C:/Program Files/Java/jre1.8.0_181'
Alternatively, you can just add that location to your PATH environment variable.
JVM Initialization via rJava
On the other hand the error message may point to just a JVM not being initialized, which you can solve with .jinit, a minimal example:
library(parallel)
cl <- makeCluster(detectCores())
parallel::parLapply(cl, 1:5, function(x) {
rJava::.jinit()
rJava::.jnew(class = "java/lang/Integer", x)$toString()
})
Working around Java use
This was not specifically asked, but you can also work around the need for Java dependency using ODBC drivers, which for Oracle should be accessible here:
con <- DBI::dbConnect(
odbc::odbc(),
Driver = "[your driver's name]",
...
)

java.lang.UnsatisfiedLinkError when writing using crunch MemPipeline

I am using com.cloudera.crunch version: '0.3.0-3-cdh-5.2.1'.
I have a small program that reads some AVROs and filters out invalid data based on some criteria. I am using pipeline.write(PCollection, AvroFileTarget) to write the invalid data output. It works fine in production run.
For unit testing this piece of code, I use MemPipeline instance.
But, it fails while writing the output in that case.
I get error:
java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)
at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86)
at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:428)
at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:197)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163)
at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:144)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:78)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
at java.io.DataOutputStream.writeBytes(DataOutputStream.java:276)
at com.cloudera.crunch.impl.mem.MemPipeline.write(MemPipeline.java:159)
Any idea what's wrong?

Hadoop environment variable should be configured properly along with hadoop.dll and winutils.exe.
Also pass the JVM argument while executing MR job/application
-Djava.library.path=HADOOP_HOME/lib/native

apache PIG with datafu: Cannot resolve UDF's

I'm trying the quickstart from here: http://datafu.incubator.apache.org/docs/datafu/getting-started.html
I tried nearly everything, but I'm sure it must be my fault somewhere. I tried already:
exporting PIG_HOME, CLASSPATH, PIG_CLASSPATH
starting pig with -cpdatafu-pig-incubating-1.3.0.jar
registering datafu-pig-incubating-1.3.0.jar locally and in hdfs => both succesful (at least no error shown)
nothing helped
Trying this on pig:
register datafu-pig-incubating-1.3.0.jar
DEFINE Median datafu.pig.stats.StreamingMedian();
data = load '/user/hduser/numbers.txt' using PigStorage() as (val:int);
data2 = FOREACH (GROUP data ALL) GENERATE Median(data);
or directly
data2 = FOREACH (GROUP data ALL) GENERATE datafu.pig.stats.StreamingMedian(data);
I get this name-resolve error:
2016-06-04 17:22:22,734 [main] ERROR org.apache.pig.tools.grunt.Grunt
- ERROR 1070: Could not resolve datafu.pig.stats.StreamingMedian using imports: [, java.lang., org.apache.pig.builtin.,
org.apache.pig.impl.builtin.] Details at logfile:
/home/hadoop/pig_1465053680252.log
When I look into the datafu-pig-incubating-1.3.0.jar it looks OK, everything in place. I also tried some Bag functions, same error then.
I think it's kind of a noob-error which I just don't see (as I did not find particular answers for datafu in SO or google), so thanks in advance for shedding some light on this.

Pig script is proper, the only thing that could break is that while registering datafu there were some class dependencies that coudn't been met.
Try to run locally (pig -x local) and see a detailed log.
Check also the version of pig - it should be newer than 0.14.0.

Postgres DB can't connect to R with RJDBC

I've been trying to query data from a PostgreSQL DB via R. I've tried skinning the cat with a few different packages (RODBC, RJDBC, DBI, RPostgres, etc), but I seem to keep getting driver errors. Oddly, I never have trouble using the same drivers/URL's and settings to connect to Postgres from SQLWorkbench/J.
I've tried using postgresql-9.2-1002.jdbc4.jar and postgresql-9.3-1100.jdbc41.jar, as well as the generic "PostgreSQL" driver in R. The two jar files are the (i) the driver I use all the time with SQLWorkbench/J and (ii) the slightly newer version of that same driver, respectively. Yet, when I try to use it...
drv_custom <- JDBC(driverClass = "org.postgresql.Driver", classPath="/Users/xxxx/postgresql-9.3-1100.jdbc41.jar")
I get an error:
Error in .jfindClass(as.character(driverClass)[1]) : class not found
OK, so next I try it with the generic driver:
drv_generic <- dbDriver("PostgreSQL")
and strangely, it doesn't want me to enter a username:
>con <- dbConnect(drv=drv_generic, "jdbc:postgresql://xxx.xxxxx.com", port=xxxx, uid="xxxx", password="xxxx")
>Error in postgresqlNewConnection(drv, ...) : unused argument (uid = "xxxx")
so I try it without user/uid:
con <- dbConnect(drv_generic, "jdbc:postgresql://padb-01.jiwiredev.com:5439", password="paraccel")
and get an error....
Error in postgresqlNewConnection(drv, ...) :
RS-DBI driver: (could not connect jdbc:postgresql://padb-01.xxx.com:5439#local on dbname "jdbc:postgresql://xxxx.xxxx.com:5439")
Apparently the syntax is wrong?
Then I circle back to trying the "custom" driver (either one of the .jar files from earlier) but without the driverClass specified.
drv_custom1 <- JDBC( classPath="/Users/xxxx/postgresql-9.2-1002.jdbc4.jar")
con <- dbConnect(drv=drv_custom1, "jdbc:postgresql://xxx.xxx.com", port=5439, uid="paraccel", pwd="paraccel")
and get this error:
Error in .jcall(drv#jdrv, "Ljava/sql/Connection;", "connect", as.character(url)[1], :
RcallMethod: attempt to call a method of a NULL object.
I tried it again with a slight alteration to the syntax:
con <- dbConnect(drv=drv_custom1, url="jdbc:postgresql://xxxx.xxxx.com", port=xxxx, uid="xxxx", pwd="xxxxx",dsn="xxxx")
and got the same error. I tried a number of other variations/approaches as well. I think part of my confusion comes from the fact that the documentation is handled in a very piecemeal way between packages like DBI and those that build upon it like RJDBC, so that when I look at documentation such as ?dbConnect many of the options I need to specify are not even mentioned, and I've been working based off of miscellaneous Google search results related to these packages/errors.
One thread I found suggested trying
.jaddClassPath( "xxxxx/postgresql-9.2-1002.jdbc4.jar" )
first, but that didn't seem to help.
I also tried using
x <- PostgreSQL(max.con = 16, fetch.default.rec = 500, force.reload = FALSE)
to no avail and I experimented with RODBC as the driver.
UPDATE:
I tried using an older version of the driver (jdbc3 instead of jdbc4), restarting R, and detaching all unnecessary packages.
I was able to load the driver
> drv_custom <- JDBC(driverClass = "org.postgresql.Driver", classPath="/xxxxx/xxxxx/postgresql-9.3-1100.jdbc3.jar")
but I still can't connect...
> con <- dbConnect(drv=drv_custom, "jdbc:postgresql://xxxxx.xxxxx.com", port=5439, uid="xxxxx", pwd="xxxxx")
Error in .verify.JDBC.result(jc, "Unable to connect JDBC to ", url) :
Unable to connect JDBC to jdbc:postgresql://xxxxx.xxxx.com

This works for me:
library(RJDBC)
drv <- JDBC("org.postgresql.Driver","C:/R/postgresql-9.4.1211.jar")
con <- dbConnect(drv, url="jdbc:postgresql://host:port/dbname", user="<user name>", password="<password>")
The trick was to include port and dbname in the url. For some reason, jdbc:postgresql does not seem to like reading those information from the dbConnect parameters.
If you are not sure what the dbname is, it is perhaps postgres.
If you are not sure what the port is, it is perhaps 5432.
So a typical call would look like:
con <- dbConnect(drv, url="jdbc:postgresql://10.10.10.10:5432/postgres", user="<user name>", password="<password>")
You can get the jar file from https://jdbc.postgresql.org/

It took some troubleshooting on IRC, but here's what had to happen:
I needed to clear the workspace, detach RODBC and RJDBC, then restart
Load RPostgreSQL
Use only the generic driver
Modify the syntax to
con <- dbConnect(drv=drv_generic, "xxx.xxx.com", port=vvvv, user="ffff", password="ffff", dbname="ggg")
Note: removing the jdbc:postgresql: part was key. Those would've been necessary with RJDBC, but RJDBC turned out to be an unnecessarily painful route.

findAll fails after play morphia is reinitialised

I have the following class which is persisted into a mongodb using morphia in a play! application. The class resides in a module which is a dependency of another play! application.
Its configuration is read from a file and persisted into the db on initial load (if its not already in the db) and then subsequent requests use the db version.
#Entity
public class Page extends Model {
#Id
public Long navigationId;
// etc ...
}
The initial load and subsequent query access works ok and i can see the page in mongo:
> db.Page.find({_id:20000})
{ "_id" : NumberLong(20000), "className" : "models.Page" etc }
However if i restart play! or make a code change that results in Morphia being reinitialised
(MorphiaPlugin-1.2.4> initialized appears in the logs) i get the following stack trace:
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: com.google.code.morphia.mapping.MappingException: Error setting value from converter (LongConverter) for models.Page.navigationId to 20000
at com.google.code.morphia.mapping.Mapper.fromDb(Mapper.java:487)
at com.google.code.morphia.mapping.Mapper.fromDBObject(Mapper.java:267)
at com.google.code.morphia.query.MorphiaIterator.convertItem(MorphiaIterator.java:66)
at com.google.code.morphia.query.MorphiaIterator.processItem(MorphiaIterator.java:53)
at com.google.code.morphia.query.MorphiaIterator.next(MorphiaIterator.java:48)
at com.google.code.morphia.query.QueryImpl.asList(QueryImpl.java:255)
at play.modules.morphia.Model$MorphiaQuery.asList(Model.java:1067)
at models.Page.findAll(Page.java)
at plugins.PageConfigLoadPlugin.loadPersistedPages(PageConfigLoadPlugin.java:62)
at plugins.PageConfigLoadPlugin.onApplicationStart(PageConfigLoadPlugin.java:51)
at play.plugins.PluginCollection.onApplicationStart(PluginCollection.java:425)
at play.Play.start(Play.java:495)
... 3 more
Caused by: java.lang.RuntimeException: com.google.code.morphia.mapping.MappingException: Error setting value from converter (LongConverter) for models.Page.navigationId to 20000
at com.google.code.morphia.mapping.ValueMapper.fromDBObject(ValueMapper.java:27)
at com.google.code.morphia.mapping.Mapper.readMappedField(Mapper.java:501)
at com.google.code.morphia.mapping.Mapper.fromDb(Mapper.java:484)
... 14 more
Caused by: com.google.code.morphia.mapping.MappingException: Error setting value from converter (LongConverter) for models.Page.navigationId to 20000
at com.google.code.morphia.converters.DefaultConverters.fromDBObject(DefaultConverters.java:133)
at com.google.code.morphia.mapping.ValueMapper.fromDBObject(ValueMapper.java:25)
... 16 more
If i drop the Collection from mongodb using the command line i can again load and query the Page obejct successfully from my play! web application
> db.Page.drop()
true
As I mentioned, this class is in a module. This problem only happens in one application that the module is a dependency of. The other demo application works fine.
any suggestions?

Does the dependent module have another alternate declaration of the Page or
Model classes?
From the top of the stack trace, ("com.google.code.morphia.mapping.MappingException: Error setting value from converter (LongConverter) for models.Page.navigationId to 20000"), it looks like there's a type mismatch between
the way a numeric value is being stored and retrieved. For example, in Java,
it is not uncommon to store something as a Double and then mistakenly try
to retrieve it as an Integer or a Long. So, can you verify that the
declaration of navigationId in the dependent module is the same as the one
you've shown here? They should share the definition, but its possible there's
an alternate.
Another thing to check is that your code is putting a Long into your
DBObject and not a float. For example, if you're inserting it via the shell,
you'll need to use one of the wrappers, e.g.
db.Page.save({_id:NumberLong(20000)});
Otherwise, this will also cause a mismatch when you retrieve it; by default
numbers in JavaScript are doubles.