I'm trying to follow the PredictionIO tutorial for a text classification enginge provided here. I'm using OSX and PredictionIO together with PostgreSQL.
When trying to import data into my naked App with the command
pio import --appid 1 --input data/stopwords.json
I get the following error:
Exception in thread "main" java.sql.SQLException: No suitable driver found for jdbc:postgresql://localhost/pio
So obviously the JDBC driver is missing or not linked appropriately. Here's where my trouble begins. Naively I downloaded the postgres jdbc driver from this source. I read that I need to add it to java's "CLASSPATH", so I tried this by adding
export CLASSPATH=$CLASSPATH:~/java-extensions/postgresql-9.4.1207.jre6.jar
to my bash configuration. Unfortunatly this does not work. A similar error situation is discussed here - but with my knowledge I'm not able to extract useful hints from it.
--
Version details:
psql (PostgreSQL) 9.4.5 installed via homebrew
OSX Yosemite
PredictionIO 0.9.5
pio status reports "Your system is all ready to go."
See this Google Group thread: https://groups.google.com/d/topic/predictionio-user/yJexMh5HpHc/discussion
tl;dr :
Fixed in next release of PredictionIO (v0.9.6), coming soon.
Workaround for earlier PIO versions :
pio train -- --driver-class-path WHEREVER/postgresql-9.4-1204-jdbc41.jar
As alex9311 pointed out, the pio import nowadays accepts a flag to specify the driver location:
pio import --appid 1 --input data/stopwords.json --driver-class-path WHEREVER/postgresql-9.4-1204-jdbc41.jar
See also this SO question
Related
Trying to run my embedded mysql based Unit tests I get an exception with this part:
Failed to instantiate [com.wix.mysql.EmbeddedMysql]: Factory method 'getEmbeddedMysql' threw exception; nested exception is com.wix.mysql.exceptions.CommandFailedException: Command 'CREATE USER 'sa'#'%' IDENTIFIED BY '';' on schema 'information_schema' failed with message 'Stream closed'
The same unit test and environment setup work on my MacBook
This machine with the error is an Ubuntu 20.04
Wix version is 4.6.2; Java 8, mysql.connector 8.0.24
I tried changing the dependencies versions and also tried with Java 11.
Run from within IntelliJ and on the command line. Same result.
Let me put the full comment I found in Github which helped me to fix this and I'm pretty sure most of the people seeing this in Ubuntu will find this as the solution:
I had this same issue with MySQL 5.7, while working on another open source project. I cloned the wix-embedded-mysql repository and ran the tests using the master branch, which also failed in the exact same way, except that I received a longer, more thorough message in the catch.
The issue was that I was on Ubuntu and did not have the ncurses 5 shared library installed. On ubuntu, I installed libncurses5 (apt install libncurses5) and everything started working (all tests on wix, and on my project).
I hope this helps resolve the issue.
Thanks to https://github.com/codesplode
I am trying to run some JUnit tests over cassandra. But I get the following error:
[08/12/19 10:48:40:411](main)([]) INFO - c.h.c.c.e.EmbeddedCassandra - Starting embedded Cassandra server.
8/12/19 10:48:41:497](main)([]) ERROR - o.a.c.u.NativeLibraryDarwin - Failed to link the C library against JNA. Native methods will be unavailable.
java.lang.UnsatisfiedLinkError: /private/var/folders/ty/wl4gxf352m328101m101hwh40000gn/T/jna--321969061/jna10641195286884112036.tmp: dlopen(/private/var/folders/ty/wl4gxf352m328101m101hwh40000gn/T/jna--321969061/jna10641195286884112036.tmp, 1): no suitable image found. Did find:
/private/var/folders/ty/wl4gxf352m328101m101hwh40000gn/T/jna--321969061/jna10641195286884112036.tmp: code signature in (/private/var/folders/ty/wl4gxf352m328101m101hwh40000gn/T/jna--321969061/jna10641195286884112036.tmp) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
It was running well until I changed to mac mojave and re-setup everything. I think it is an issue related to permissions may be or JNA?
IDE: IntelliJ
Java: AdoptOpenJDK 11.0.4
JNA: 4.2.2
Any kind of help will be highly appreciated!
This is the result of a bug in AdoptOpenJDK jdk-11.0.4+11 on macOS, persisting through 11.2.
It will be fixed in the jdk-11.0.4+11.3 release.
If you can't wait for the new release you can temporarily resolve by downgrading to 11.0.3+7
What version of cassandra unit? Support for java 11 (https://issues.apache.org/jira/browse/CASSANDRA-9608) isn't in until cassandra 4 and I dont think embedded cassandra is setup for that yet
Trying to connect to a HSQLDB Java database using python 3.7, jaydebeapi, and jpype. I tried the following
import jaydebeapi
UserName = "SA"
Password = ""
Java_Class = "org.hsqldb.jdbcDriver"
HSQL_Driver_Path = "/Hsqldb/driver/hsqldb.jar"
Database = "jdbc:hsqldb:/Hsqldb/database/OneDatabase"
jaydebeapi.connect(Java_Class,Database,[UserName,Password],jars=HSQL_Driver_Path)
and it resulted in the following error
java.lang.RuntimeExceptionPyRaisable: java.lang.RuntimeException: Class org.hsqldb.jdbcDriver not found
google states that this is a classpath error and I need to add a class path to fix.
edit: Details of setup: Mac, anaconda, python 3.7
I tried something similar in R using the RJDBC library and was able to connect just fine.
PathDriver = "/Hsqldb/driver/hsqldb.jar"
JDBCDriver = "org.hsqldb.jdbcDriver"
drv <- JDBC(JDBCDriver,PathDriver)
# Connect to Database
DatabaseP <- "jdbc:hsqldb:file:////Hsqldb/database/OneDatabase"
Con <- dbConnect(drv,DatabaseP,"SA","")
edit: Details of setup: Mac, R, Rstudio
I separated R from Anaconda as Anaconda was blocking many of the libraries I wanted to use.
This is probably apples to oranges, but why does python need a classpath set when R performs the function just fine? and how does one set a class path for HSQLDB for python?
When starting JPype it either takes the classpath from the environment variable CLASS_PATH or from a manually specified classpath defined using the jpype.startJVM(classpath="...") command. This command is likely part of the jaydebeapi api if you don't have to call it directly from JPype.
I assume that R defines its own classpath internally so there may just be differences in the what is being seen from the JVM. This could include a different JVM is being selected, the classpath including different locations, or in some cases if the driver has a native portion such as MS SQLServer the library path may also not be correct. In order for the JDBC driver to load all piece need to be found including the jar, support jars and native shared libraries. If all else fails manually try to load the java class using Class.forname. It may provide you with further diagnostics as to which piece is missing.
I have one Linux VPS dedicated just for running a jmeter. The tests run fine, but failed requests are not written to error.jtl with SimpleDataWriter. Java error is written to jmeter.log instead.
I run the tests in non-gui mode:
jmeter -n -t om5.jmx -j results-tmp1/t3-l1-jmeter.log
The error
2016/07/08 16:59:35 ERROR - jmeter.reporters.ResultCollector:
Error trying to record a sample java.lang.IllegalArgumentException: Failed marshalling:class:class
org.apache.jmeter.samplers.SampleResult,content:org.apache.jmeter.samplers.SampleResult#1f605bfa[saveConfig=org.apache.jmeter.samplers.SampleSaveConfiguration#b4a9237e,parent=<null>,
responseData={},responseCode=500,label=03 add to cart,resultFileName=,samplerData=<null>,threadName=Thread Group 1-149,responseMessage=Number of samples in transaction : 2, number of failing samples : 1,
responseHeaders=,contentType=,requestHeaders=,timeStamp=1467989884109,startTime=1467989884109,endTime=1467989975286,idleTime=1001,pauseTime=0,assertionResults=<null>,subResults=[reset basketItems, /some-page],
dataType=,success=false,files=[res-tmp/t400-l5-errors.jtl],dataEncoding=<null>,elapsedTime=90176,latency=0,connectTime=0,startNextThreadLoop=false,stopThread=false,stopTest=false,
stopTestNow=false,isMonitor=false,sampleCount=1,bytes=806,headersSize=192,bodySize=614,groupThreads=400,allThreads=400,nanoTimeOffset=1467988012523,useNanoTime=true,nanoThreadSleep=5000,location=<null>]
at org.apache.jmeter.save.SaveService.saveSampleResult(SaveService.java:345)
at org.apache.jmeter.reporters.ResultCollector.sampleOccurred(ResultCollector.java:557)
at org.apache.jmeter.threads.ListenerNotifier.notifyListeners(ListenerNotifier.java:67)
at org.apache.jmeter.threads.JMeterThread.notifyListeners(JMeterThread.java:819)
at org.apache.jmeter.threads.JMeterThread.doEndTransactionSampler(JMeterThread.java:534)
at org.apache.jmeter.threads.JMeterThread.triggerEndOfLoopOnParentControllers(JMeterThread.java:342)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:258)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
I cannot find what is wrong. It is only that one machine, that gives me this error. Everywhere else I try it, it works correctly.
The OS is Debian Jessie (8.5) - minimal from official repositories. I tried many different versions of java (1.7, 1.8) - currently running on the latest 1.8 (1.8.0_92-b14) and I have the latest apache-jmeter 3.0 r1743807, but previously I used 2.13 r1665067. I don't use any third-party plugins.
I didn't notice exactly when it stopped working, what change was done (some os update, maybe some change in jmx), but I have my tests in git, so I checked out older version from the date of my last full error.jtl and it is not writing the errors to jtl as well.
I reinstalled the Debian to Ubuntu 16.04 and nothing changed.
I don't know how to debug that problem, what should I do, because on any other machine I have an access to, it works fine.
I don't know if you've fixed this problem until now?
I had the same issue, and I fixed it just now.
I guess in your jmeter script, you checked the hostname to record sample result, in jmeter.log, you could find such message: jmeter couldn't get the linux Local IP, so hostname is Null, exception threw out.
So what you need to do is remove 'Save Hostname' from 'Sample Result Save Configuration' tab.
I downloaded sqljdbc4.jar. I'm invoking sqoop like so from the folder (where the jar is stored):
sqoop list-tables --driver com.microsoft.jdbc.sqlserver.SQLServerDriver --connect jdbc:sqlserver://localhost:1433;user=me;password=myPassword; -libjars=./sqljdbc4.jar
I'm getting the following warning & error:
13/10/25 18:38:13 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
13/10/25 18:38:13 INFO manager.SqlManager: Using default fetchSize of 1000
13/10/25 18:38:13 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.jdbc.sqlserver.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.jdbc.sqlserver.SQLServerDriver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:727)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.listTables(SqlManager.java:418)
at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
UPDATE
I changed the command line to reflect the comments below, I get the same error:
sqoop list-databases -libjars=<ABSOLUTE_PATH>/jars/sqljdbc4.jar --connect jdbc:sqlserver://localhost:1433;user=me;password=password
13/10/28 17:00:33 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:727)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
When I look at the listing of sqljdbc4.jar, I do see the class in that path... Is it possible that libjars option isn't doing what I think it is supposed to do?
In vast majority of cases using parameter --driver is not required and even more will lead to an undesirable behaviour. I would strongly recommend dropping this argument entirely from your command line. Check out Connectors vs Drivers blog post for more details.
Also in addition you are specifying a nonexistent JDBC Driver class. The correct one is:
com.microsoft.sqlserver.jdbc.SQLServerDriver
You can see it in the official docs, whereas you are specifying
com.microsoft.jdbc.sqlserver.SQLServerDriver
Notice the different order of jdbc and sqlserver packages. This is one of the reasons why it's recommended to not use the --driver option at all.
You need to put sqljdbc4.jar in $SQOOP_HOME/lib and also add sqoop-1.4.4.jar or whatever version you are using along with sqljdbc4.jar to $HADOOP_HOME/lib.
I'm using Hadoop-2.2.0, so i put it inside $HADOOP_HOME/share/hadoop/common/lib directory, and use the following command to do the import:
export HCAT_HOME=/home/Kuntal/BIG_DATA/hive-0.12.0/hcatalog
(sometimes HCatlog of Hive needs to be exported or set.)
./sqoop-import --connect "jdbc:sqlserver://IP\INSTANCE;port=1433;username=USERNAME;password=PASSWORD;database=DATABASE_NAME" --table TABLE_NAME --target-dir hdfs://localhost:50315/sqoop --m 1
Sometimes you have to specify the port, otherwise default works. Hope you find it useful.
According to this sqoop documentation, generic options like -libjars must come before tool-specific options:
Generic Hadoop command-line arguments:
(must preceed any tool-specific arguments)
...
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
Recently came across this same problem. Even though documentation says that it will pick up additional jar files. The problem is I believe propagated from Hadoop jar command line option. -libjars is not reliable option to set additional jar file path.
Instead, choose HADOOP_CLASSPATH option to setup additional jar files.
In my case, I had multiple different versions of driver JAR, but using -libjars was not correctly picking up file for me.
To resolve this, I specified
export HADOOP_CLASSPATH=/$SQOOP_HOME/<path_to_driver>.jar
This makes sure that correct JAR file gets loaded.