Apache Spark MySQL JavaRDD.foreachPartition - why I getting ClassNotFoundException - java

I would like to save data from each partition to MySQL Database. For doing that I created Class which implements VoidFunction<> :
public class DatabaseSaveFunction implements VoidFunction<Iterator<String>> {
/**
*
*/
private static final long serialVersionUID = -7039277486852158360L;
public void call(Iterator<String> it) {
Connection connect = null;
PreparedStatement preparedStatement = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connect = DriverManager.getConnection("jdbc:mysql://"
+ "xxx.us-west-2.rds.amazonaws.com" + "/"
+ "xxx", "xxx", "xxx");
preparedStatement = connect
.prepareStatement("insert into testdatabase.test values (default, ?)");
while (it.hasNext()) {
String outputElement = it.next();
preparedStatement.setString(1, "" + outputElement.length());
preparedStatement.executeUpdate();
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
try {
connect.close();
preparedStatement.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
And in my main method class I'm calling:
output.foreachPartition(new DatabaseSaveFunction());
I'm getting following error:
15/05/06 15:34:00 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 4, ip-172-31-36-44.us-west-2.compute.internal): java.lang.ClassNotFoundException: DatabaseSaveFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
Worker log:
15/05/06 15:34:00 ERROR executor.Executor: Exception in task 1.0 in stage 1.0 (TID 5)
java.lang.ClassNotFoundException: DatabaseSaveFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
Can anybody tell me what I'm doing wrong ? I would be very grateful for that.

Export the external class to jar and add that like sc.addJar("/path/x.jar") where sc is JavaSparkContext in your main. Then you wont get this error. The error is because you spark program is not able to find the class. Moreover in spark 1.3 and greater you can simple use a map options of jdbc and then use load("jdbc", options) to create a data frame and load data from any RDBMS. its really handy. I am not sure if this method works for connecting any RDBMS into spark. Please tell me if you have any other question.

Related

JavaFX application with JDBC in NetBeans don't start

I work on a small educational project - easy java app with database and gui. I work in NetBeans IDE.
For graphical interface I use JavaFX, and as I've testet this part work perfectly fine, but I have some problem with JDBC part (I use JDBC driver for sqlite database, created with DB browser for sqlite). My main application class looks like this:
public class BazaLyoko extends Application {
#Override
public void start(Stage stage) throws Exception {
Database baza = new Database();
String tytul = baza.getEpisodeName(1);
stage.setTitle(tytul);
Parent root = FXMLLoader.load(getClass().getResource("FXMLDocument.fxml"));
Scene scene = new Scene(root);
stage.setScene(scene);
stage.show();
}
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
launch(args);
}
}
And when I change those lines:
Database baza = new Database();
String tytul = baza.getEpisodeName(1);
To this (to replace value retrieved from database to a placeholder - I did this to ensure if the error is rather connected with JavaFX or JDBC part)
String tytul = "";
Program start without problem, but when I try to use my database application don't start. There isn't any compilation error or anything like that (or maybe I just don't see them, as I'm quite new to this IDE, though I think such things are usually blatantly visible) just no application window appear.
Here's the code for my database class(they both share the same package):
public class Database {
public static Connection conn;
public Database()
{
connect();
}
public static void connect()
{
String connectionString = "jdbc:sqlite:BazaLyoko.db";
try
{
conn = DriverManager.getConnection(connectionString);
if (conn != null)
{
DatabaseMetaData meta = conn.getMetaData();
//System.out.println("Nazwa sterownika to " + meta.getDriverName());
//System.out.println("Stworzono baze danych.");
}
}
catch (SQLException e)
{
System.out.println(e.getMessage());
}
}
private ResultSet doQuery(String query) throws SQLException
{
Statement stmt = null;
try
{
stmt = this.conn.createStatement();
ResultSet result = stmt.executeQuery(query);
return result;
}
catch (SQLException e )
{
throw new Error("Problem", e);
}
finally
{
if (stmt != null) { stmt.close(); }
}
}
public String getEpisodeName(int nr) throws SQLException
{
ResultSet result = doQuery("SELECT tytul FROM Odcinki WHERE numer="+String.valueOf(nr));
result.next();
return result.getString("tytul");
}
}
Database of that name is located inside my project directory (i also tried to copy it to package directory but nothing changed), the names of the table and column are correct to and exist a row with value 1 in column numer.
What I'm missing or doing wrong?
Edit:
I manage to get the following Stack Trace:
No suitable driver found for jdbc:sqlite:BazaLyoko.db
at com.sun.corba.se.impl.util.Utility.printStackTrace(Utility.java:933)
at bazalyoko.Database.connect(Database.java:46)
at bazalyoko.Database.<init>(Database.java:26)
at bazalyoko.BazaLyoko.start(BazaLyoko.java:23)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$8(LauncherImpl.java:863)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$7(PlatformImpl.java:326)
at com.sun.javafx.application.PlatformImpl.lambda$null$5(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$6(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$4(WinApplication.java:186)
at java.lang.Thread.run(Thread.java:748)
Exception in Application start method
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:389)
at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:917)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$1(LauncherImpl.java:182)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at bazalyoko.Database.doQuery(Database.java:57)
at bazalyoko.Database.getEpisodeName(Database.java:75)
at bazalyoko.BazaLyoko.start(BazaLyoko.java:24)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$8(LauncherImpl.java:863)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$7(PlatformImpl.java:326)
at com.sun.javafx.application.PlatformImpl.lambda$null$5(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$6(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$4(WinApplication.java:186)
... 1 more
Exception running application bazalyoko.BazaLyoko
Java Result: 1

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver exception in linux and text editor [duplicate]

This question already has answers here:
Connect Java to a MySQL database
(14 answers)
Closed 5 years ago.
My code looks like this-(working in normal text editor)
import java.sql.*;
import java.sql.DriverManager;
class JDBCTest {
private static final String url = "jdbc:mysql://localhost/learn";
private static final String user = "root";
private static final String password = "B!SHu12345";
public static void main(String args[]) {
try {
Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection(url, user, password);
Statement st=con.createStatement();
ResultSet rs=st.executeQuery("select * from bishu where id =101");
while(rs.next()){
String s=rs.getString("id");
System.out.println(s);
}
System.out.println("Success");
} catch (Exception e) {
e.printStackTrace();
System.out.println(e.getMessage());
}
}
}
At run time it gives an exception as follows-
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at JDBCTest.main(first.java:14)
Mysql connector was downloaded and saved in the path- Java/jre1.8.0_91/lib/ext/
I have gone through many similar solutions available, but wasn't able to find a valid one.
Add the MYSQL JDBC Driver from the library of your IDE.or place this JDBC driver file inside the lib directory of your project. Your question looks similar to this question

Trying to connect mySQL database with a java app keep getting stuck [duplicate]

This question already has answers here:
Connect Java to a MySQL database
(14 answers)
Closed 5 years ago.
ok so i have a java fx app that will display info from a mysql database runing locally on the same computer. i keep getting driver not found. here is the code:
String databaseURL = "jdbc:mysql://localhost:3307/mysql?zeroDateTimeBehavior=convertToNull";
String user = "user";
String password = "password";
Connection conn = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(databaseURL, user, password);
if (conn != null) {
System.out.println("Connected to the database");
}
} catch (ClassNotFoundException ex) {
System.out.println("Could not find database driver class");
ex.printStackTrace();
} catch (SQLException ex) {
System.out.println("An error occurred. Maybe user/password is invalid");
ex.printStackTrace();
} finally {
if (conn != null) {
try {
conn.close();
} catch (SQLException ex) {
ex.printStackTrace();
}
}
and here is the stack trace:
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:259)
at got_map.GOT_Map.start(GOT_Map.java:130)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$153(LauncherImpl.java:821)
at com.sun.javafx.application.LauncherImpl$$Lambda$51/1468396900.run(Unknown Source)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$166(PlatformImpl.java:323)
at com.sun.javafx.application.PlatformImpl$$Lambda$44/584634336.run(Unknown Source)
at com.sun.javafx.application.PlatformImpl.lambda$null$164(PlatformImpl.java:292)
at com.sun.javafx.application.PlatformImpl$$Lambda$47/1960854798.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$165(PlatformImpl.java:291)
at com.sun.javafx.application.PlatformImpl$$Lambda$45/501263526.run(Unknown Source)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$141(WinApplication.java:102)
at com.sun.glass.ui.win.WinApplication$$Lambda$37/96639997.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Apr 21, 2017 6:29:06 PM javafx.fxml.FXMLLoader$ValueElement processValue
WARNING: Loading FXML document with JavaFX API of version 8.0.111 by JavaFX runtime of version 8.0.20
Any help would be much appreciated
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
...
at java.lang.Class.forName(Class.java:259)
at got_map.GOT_Map.start(GOT_Map.java:130)
This : Class.forName("com.mysql.jdbc.Driver"); is not enough.
Indeed, it rises a ClassNotFoundException if the class cannot be retrieved from the runtime classpath.
You need just to add in the runtime classpath the MySQL JDBC driver matching with the installed MySQL version.
The problem is in the line, Class.forName("com.mysql.jdbc.Driver");
The reason for getting java.lang.ClassNotFoundException can be an issue in loading the driver.Add the *.jar driver to your project if you haven't added it.

Why driver is still missing?

I am trying to use Java DB for app only storage.
Environment Variables:
CLASSPATH:
%JAVA_HOME%\jre\lib;%DERBY_HOME%\lib\derby.jar
DERBY_HOME:
C:\Program Files\Java\jdk1.7.0_25\db
Path:
%DERBY_HOME%\bin
No changes in eclipse configurations was done and i cannot include it to build path(it somehow will be considered as third party library, which is not allowed)
When i type "sysinfo" in cmd it tells me Db is instaled and list packages and other info.
Code:
public class Main {
private static String dbURL = "jdbc:derby:AssertDB;user=me;password=mine;create=true";
private static Connection con = null;
public static void main(String[] args) {
setDBSystemDir();
createConnection();
CookiesTable pTable = new CookiesTable(con);
try {
pTable.createTable();
pTable.populateTable();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
// TODO Auto-generated method stub
// load files
// parse files to tables
// retrive SQL statment
// print result
}
private static void setDBSystemDir() {
// Decide on the db system directory: <userhome>/.addressbook/
String userHomeDir = System.getProperty("user.home", ".");
String systemDir = userHomeDir + "/.asertdb";
// Set the db system directory.
System.setProperty("derby.system.home", systemDir);
}
private static void createConnection() {
try {
Class.forName("org.apache.derby.jdbc.EmbeddedDriver").newInstance();
// Get a connection
con = DriverManager.getConnection(dbURL);
} catch (Exception except) {
System.out.println("DRIVER DRROOOOOP");
except.printStackTrace();
}
}
}
Exception:
java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at com.tuto.p4.Main.createConnection(Main.java:44)
at com.tuto.p4.Main.main(Main.java:13)
Exception in thread "main" java.lang.NullPointerException
at com.tuto.p4.CookiesTable.createTable(CookiesTable.java:29)
at com.tuto.p4.Main.main(Main.java:16)
Refering to the documentation there are two things to do in case you get a java.lang.ClassNotFoundException: org.apache.derby.jdbc.EmbeddedDriver if you run the SampleApp.
Configure Embedded Derby
Verify Derby
As you said that you checked sysinfo I would try it with adding derby.jar and derbytools.jar even if the tools are optional.
Or you can compare the SampleApp to your App....
Include derbytools.jar, derbyclient.jar as well.
Try putting System.out.println(System.getProperty("java.class.path"));
before Class.forName(...).newInstance()
The output must contain a valid path to derby.jar.

ClassNotFoundException in FreeTTS Java program

I am new to FreeTTS and this is my first program.The following program simply reads out a string passed.
I am using Linux Mint 32-bit if that matters.But I extracted jsapi.jar on my windows machine a day back.I believe it doesn't matter.
import javax.speech.*;
import java.util.*;
import javax.speech.synthesis.*;
public class Speak
{
String speaktext;
public void dospeak(String speak,String voicename)
{
speaktext=speak;
String voiceName =voicename;
try
{
SynthesizerModeDesc desc = new SynthesizerModeDesc(null,"general", Locale.US,null,null);
Synthesizer synthesizer = Central.createSynthesizer(desc);
synthesizer.allocate();
synthesizer.resume();
desc = (SynthesizerModeDesc) synthesizer.getEngineModeDesc();
Voice[] voices = desc.getVoices();
Voice voice = null;
for (int i = 0; i < voices.length; i++)
{
if (voices[i].getName().equals(voiceName))
{
voice = voices[i];
break;
}
}
synthesizer.getSynthesizerProperties().setVoice(voice);
System.out.print("Speaking : "+speaktext);
synthesizer.speakPlainText(speaktext, null);
synthesizer.waitEngineState(Synthesizer.QUEUE_EMPTY);
synthesizer.deallocate();
}
catch (Exception e)
{
String message = " missing speech.properties in " + System.getProperty("user.home") + "\n";
System.out.println(""+e);
System.out.println(message);
}
}
public static void main(String[] args)
{
Speak obj=new Speak();
obj.dospeak("Finally It speaks and speaks and speaks","kevin16");
}
}
I am getting this as a gift.
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/speech/freetts/ValidationException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at javax.speech.Central.registerEngineCentral(Central.java:703)
at javax.speech.Central.loadProps(Central.java:775)
at javax.speech.Central.availableSynthesizers(Central.java:651)
at javax.speech.Central.createSynthesizer(Central.java:553)
at Speak.dospeak(Speak.java:16)
at Speak.main(Speak.java:47)
Caused by: java.lang.ClassNotFoundException: com.sun.speech.freetts.ValidationException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more
What can I do not to accept this gift.
PS
I have added all the 4 archives viz en_us.jar,freetts-jsapi10.jar, jsapi.jar,mbrola.jar
1) Find in which jar this class is usually defined:
com.sun.speech.freetts.ValidationException
2) See if you have the jar
3) See if you have the right version of this jar
(open it, see if the class file is there)
4) See if the jar is in the classpath at runtime

Categories

Resources