How can I import a mysql database dump file (contains insert and create table statements) programmatically through a java program. I need this as the setup phase of a unit test.
Unfortunately this doesn't work:
Connection conn = dbConnectionSource.getConnection();
Statement stmt = conn.createStatement();
stmt.execute(FileUtils.readFileToString(new File("./some-sql-file")));
conn.close();
Thanks,
-A
PS - In Rails, I used fixtures for filling a test database. I made rails rails create the underlying tables through setting the environment to test, anything similar in Java.
You could start a new process from java and execute this command if you have access to the mysql executable wherever you are running the import. Something like this:
Runtime rt = Runtime.getRuntime();
Process pr = rt.exec("mysql -p -h ServerName DbName < dump.sql");
Backup:
/******************************************************/
//Database Properties
/******************************************************/
String dbName = “dbName”;
String dbUser = “dbUser”;
String dbPass = “dbPass”;
/***********************************************************/
// Execute Shell Command
/***********************************************************/
String executeCmd = “”;
executeCmd = “mysqldump -u “+dbUser+” -p”+dbPass+” “+dbName+” -r backup.sql”;
}
Process runtimeProcess =Runtime.getRuntime().exec(executeCmd);
int processComplete = runtimeProcess.waitFor();
if(processComplete == 0){
out.println(“Backup taken successfully”);
} else {
out.println(“Could not take mysql backup”);
}
Restore:
/******************************************************/
//Database Properties
/******************************************************/
String dbName = “dbName”;
String dbUser = “dbUser”;
String dbPass = “dbPass”;
/***********************************************************/
// Execute Shell Command
/***********************************************************/
String executeCmd = “”;
executeCmd = new String[]{“/bin/sh”, “-c”, “mysql -u” + dbUser+ ” -p”+dbPass+” ” + dbName+ ” < backup.sql” };
}
Process runtimeProcess =Runtime.getRuntime().exec(executeCmd);
int processComplete = runtimeProcess.waitFor();
if(processComplete == 0){
out.println(“success”);
} else {
out.println(“restore failure”);
}
Personally I would disrecommend loading a regular SQL dump in this way, because you would need non-trivial code to parse or at least tokenize SQL.
I would recommend using CSV data dumps, you can load these with a the LOAD DATA INFILE syntax. See: http://dev.mysql.com/doc/refman/5.1/en/load-data.html
Of course, you would still need to ensure the target tables exist, but if you know you only have to parse table creation DDL stattemnts, that will drastically simplify your java code.
Note that you can use mysqldump to extract CSV data from your database, see: http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html#option_mysqldump_tab
Effective solution can be found here:
https://stackoverflow.com/a/1044837
This explains how to run any sql script over jdbc.
I know this question is a bit old, but I think people are still looking for an answer like me.
Basically, I wanted to make two buttons in my GUI (one to import, and the other to export) and generate an SQL file, I tested the chosen solution to start a process from java and execute it with Runtime but it didn't work, I had an Access Denied error eventhought I am the only user in my computer. After some researches, I found this library (mysql-backup4j) and did this code :
EXPORT FUNCTION :
Properties properties = new Properties();
properties.setProperty(MysqlExportService.DB_NAME, "DATABASE_NAME");
properties.setProperty(MysqlExportService.DB_USERNAME, "DATABASE_USERNAME");
properties.setProperty(MysqlExportService.DB_PASSWORD, "DATABASE_PWD");
properties.setProperty(MysqlExportService.TEMP_DIR, new File(System.getProperty("user.dir") + "\\database_dump").getAbsolutePath());
properties.setProperty(MysqlExportService.PRESERVE_GENERATED_ZIP, "true");
MysqlExportService mysqlExportService = new MysqlExportService(properties); mysqlExportService.export();
IMPORT FUNCTION (the user chooses the SQL file) :
FileChooser fc = new FileChooser();
List<String> sqlExtensions = new ArrayList<>(List.of("*.sql", "*.SQL"));
fc.getExtensionFilters().add(new FileChooser.ExtensionFilter("Fichier SQL", sqlExtensions));
File f = fc.showOpenDialog(null);
if (f != null) {
System.out.println("database path : " + f.getAbsolutePath());
String sql = new String(Files.readAllBytes(Paths.get(f.getAbsolutePath())));
boolean res = MysqlImportService.builder()
.setDatabase("DATABASE_NAME").setSqlString(sql)
.setUsername("DATABASE_USERNAME").setPassword("DATABASE_PWD").setDeleteExisting(true)
.setDropExisting(true)
.importDatabase();
}
P.S.: I am using JavaFX for the GUI with JDK11.
Reference : How to backup your MySQL database programmatically using mysql-backup4j
Related
Recently, I'm trying to develop something use Bigtable emulator with java(Spring Boot) on IntelliJ IDEA tool.
What I have done:
Bigtable emulator works well on my computer (MacOs 10.15.6).
"cbt" works normally with Bigtable emulator running on my mac as somethings like this.
I've checked that running Bigtable emulator doesn't need real gcloud credential.
I write a unit test on IEDA like this works fine.
I have added environment variable in setting like this:
My unit test code:
I. Connect init:
Configuration conf;
Connection connection = null;
conf = BigtableConfiguration.configure("fake-project", "fake-instance");
String host = "localhost";
String port = "8086";
II. Constant data going to write into table.
final byte[] TABLE_NAME = Bytes.toBytes("Hello-Bigtable");
final byte[] COLUMN_FAMILY_NAME = Bytes.toBytes("cf1");
final byte[] COLUMN_NAME = Bytes.toBytes("greeting");
final String[] GREETINGS = {
"Hello World!", "Hello Cloud Bigtable!", "Hello!!"
};
III. Connecting: (Duplicated to I.Connect init.)
Configuration conf;
Connection connection = null;
conf = BigtableConfiguration.configure("fake-project", "fake-instance");
String host = "localhost";
String port = "8086";
III. Connecting: (Edited)
if(!Strings.isNullOrEmpty(host)){
conf.set(BigtableOptionsFactory.BIGTABLE_HOST_KEY, host);
conf.set(BigtableOptionsFactory.BIGTABLE_PORT_KEY,port);
conf.set(BigtableOptionsFactory.BIGTABLE_USE_PLAINTEXT_NEGOTIATION, "true");
}
connection = BigtableConfiguration.connect(conf);
IV. Write & Read data:
Admin admin = connection.getAdmin();
Table table = connection.getTable(TableName.valueOf(TABLE_NAME));
if(!admin.tableExists(TableName.valueOf(TABLE_NAME))){
HTableDescriptor descriptor = new HTableDescriptor(TableName.valueOf(TABLE_NAME));
descriptor.addFamily(new HColumnDescriptor(COLUMN_FAMILY_NAME));
System.out.print("Create table " + descriptor.getNameAsString());
admin.createTable(descriptor);
}
for (int i = 0; i < GREETINGS.length; i++) {
String rowKey = "greeting" + i;
Put put = new Put(Bytes.toBytes(rowKey));
put.addColumn(COLUMN_FAMILY_NAME, COLUMN_NAME, Bytes.toBytes(GREETINGS[i]));
table.put(put);
}
Scan scan = new Scan();
ResultScanner scanner = table.getScanner(scan);
for (Result row : scanner) {
byte[] valueBytes = row.getValue(COLUMN_FAMILY_NAME, COLUMN_NAME);
System.out.println('\t' + Bytes.toString(valueBytes));
}
V. Output
Hello World!
Hello Cloud Bigtable!
Hello!!
Problem came after I get this code to my project.
When I use 'debug' to run the code.
I get somethings like this
when it trying to connect bigtable:
Seems that it can't new instance base on the config i create.
Eventually, it shows me an error like
Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_x.BigtableConnection
P.S. I have tried to use command running IntelliJ IDEA. Reason I doing so is because I missing environment variable when I using unit test.
In my .zshrc:
My CMD tool is iTerm2 with oh-myzsh.
Anythings is help!!!!
Thanks lots.
It seems that you miss the constructor for the BigtableConnection: BigtableConnection(org.apache.hadoop.conf.Configuration conf)
I would suggest you trying to create a Connection object by following the steps mentioned in Google Documentation
private static Connection connection = null;
public static void connect() throws IOException {
Configuration config = BigtableConfiguration.configure(PROJECT_ID, INSTANCE_ID);
// Include the following line if you are using app profiles.
// If you do not include the following line, the connection uses the
// default app profile.
config.set(BigtableOptionsFactory.APP_PROFILE_ID_KEY, APP_PROFILE_ID);
connection = BigtableConfiguration.connect(config);
}
I'm asking because ALL examples I find in Google, are the same from the Fitnesse tutorial: a very simple query to a list or array in memory, NOT A REAL Database.
Yes, Fixtures never have to deal with that, but how am I supposed to test my fixtures if I can't even make the connection to the DB in a simulation of an "API"?
What I'm trying to simulate is the call from a FitNesse Fixture to query in Java into a PostgreSQL database/table. In this simple example I'm trying to obtain, at least one column from one row, in one table. When I execute the code, it runs perfectly by it's own. The problem is when trying to execute from Fitnesse through the fixture. It always fails with a ClassNotFoundException, when calling the JDBC driver. This doesn't happen by running the code by it's own.
Here is the code that does the query:
package queriespackage;
import java.sql.*;
public class testQuery01 {
public static Connection openDBConnection(){
Connection connectionString = null;
try {
String dbhost = "SOMEURL";//Redacted
String port = "SOMEPORT";//Redacted
String dbname = "THEDBNAME";//Redacted
String username = "SOMEUSER";//Redacted
String password = "SOMEPASSWORD";//Redacted
String driverJDBC = "org.postgresql.Driver";
Class.forName(driverJDBC);
connectionString = DriverManager.getConnection("jdbc:postgresql://" + dbhost + ":" + port + "/" + dbname,username,password);
connectionString.setAutoCommit(false);
} catch (SQLException e) {
e.printStackTrace(System.err);
System.exit(0);
} catch (Exception e) {
System.err.println( e.getClass().getName()+": "+ e.getMessage() );
System.exit(0);
};
return connectionString;
};
public static ResultSet executeQuery(Connection connectionString, int intAccountId) throws SQLException{
Statement querySession = connectionString.createStatement();
//The query string
String queryString = "SELECT DISTINCT "
+ "account_search.account_id,"
+ "account_search.account_name"
+ " FROM account_search "
+ " WHERE"
+ " account_search.account_id = "+ intAccountId
+ "LIMIT 1";
ResultSet queryResult = querySession.executeQuery(queryString);
return queryResult;
};
public static String processQueryResult(ResultSet queryResult) throws SQLException{
String strQueryValueReturned = null;
while (queryResult.next()) {
strQueryValueReturned = queryResult.getString("account_name");
};
return strQueryValueReturned;
};
public static boolean closeDBConnection(Connection connectionString){
try {
if(connectionString!=null){
connectionString.close();
}
} catch (SQLException e) {
System.err.println( e.getClass().getName()+": "+ e.getMessage() );
System.exit(0);
};
return true;
};
public static String testQuery(int intAccountId) throws SQLException, ClassNotFoundException{
boolean bolConnectionStatus = false;
String strValueReturned = null;
Connection connectionString = openDBConnection();
if(connectionString != null){
ResultSet qryQueryResult = executeQuery(connectionString, intAccountId);
strValueReturned = processQueryResult(qryQueryResult);
bolConnectionStatus = closeDBConnection(connectionString);
if(!bolConnectionStatus){
System.exit(0);
}
}else{
System.exit(0);
};
return strValueReturned;
};
};
If I add a Main method to that code, passing it the argument value for "intAccountId", it successfully returns the name of the account "account_name", just as expected.
Now here's the Fixture that should be called by the FitNesse test:
package fixturespackage;
import java.sql.SQLException;
import queriespackage.testQuery01;
public class testFixture01{
private int Int_AccountId;
//Fixture Constructor (setter)
public testFixture01(int Int_AccountId){
this.Int_AccountId = Int_AccountId;
};
public String query() throws SQLException, ClassNotFoundException{
return testQuery01.testQuery(Int_AccountId);
};
};
Just as the FitNesse guide says, there must be a "query" method, that does the actual call to the interface in the DB. I had to add a constructor instead of the "setter", because FitNesse actually demands it: "Could not invoke constructor for fixturepackage.testFixture01"
Here's the FitNesse page:
!***> System Variables
!define TEST_SYSTEM {slim}
!path C:\FitnessTest\bin
*!
|Query: fixturespackage.testFixture01|21 |
|Str_AccountName |
|SomeName |
Here's a Screenshot of my BuildPath, so you can see I have the JDBC Library for Java 8, JDK 1.8, JRE 1.8... and the "org.postgresql.Driver.class" is included in the project.
This is the error I receive, when running from FitNesse:
This is the error I get, when debugging the line where FitNesse failed by using Inspect tool:
... and YES, I also tried by hard coding the name of the JDBC:
I have searched a lot for a REAL LIFE example, both here, the FitNesse Guide and Google.
The FitNesse Guide might be extensive, but let's be sincere, it's full of "dirty word here", unrealistic and incomplete examples and missing a lot of information.
So, asking again, has anyone done a REAL LIFE test making queries, using FitNesse, that could help me find out what am I doing wrong?
I have to admit I've only done limited database tests with FitNesse, but I have used them (to query DB2).
I did not use query tables (or wrote my own fixtures to query), but instead used jdbcslim in combination with script tables and scenario's.
That fact that the driver class cannot be found suggests that although the jar is present on the classpath in your IDE it is not available when FitNesse is running your fixture code.
I notice you specify the classpath as a single directory in the wiki. In Java that means that all class files should be in that directory (as .class files, in the right subdirectory for their defined package). It will not pick up any jars (or zips) in that directory. Did you unpack your database driver's jar to that directory? If not, you need to add a !path line pointing to the jar (so the entire path including the filename) with the database driver.
Now listing every jar you need can quickly become cumbersome, so you can also use wildcards. I tend to copy all the jars I need to a single directory, that also contains my fixture .class files, and add a single !path line loading all jars in that directory.
So if you also copied your database driver to the directory in you question you could ensure it, and your own fixture, to be available via
!path C:\FitnessTest\bin
!path C:\FitnessTest\bin\*.jar
I have a working sqlite db which I have place in my /src folder.
I then went onto the Codename One website and followed their doc example
Database db = null;
Cursor cur = null;
try {
db = Display.getInstance().openOrCreate("MyDb.db");
if(query.getText().startsWith("select")) {
cur = db.executeQuery(query.getText());
int columns = cur.getColumnCount();
frmMain.removeAll();
if(columns > 0) {
boolean next = cur.next();
if(next) {
ArrayList<String[]> data = new ArrayList<>();
String[] columnNames = new String[columns];
for(int iter = 0 ; iter < columns ; iter++) {
columnNames[iter] = cur.getColumnName(iter);
}
while(next) {
Row currentRow = cur.getRow();
String[] currentRowArray = new String[columns];
for(int iter = 0 ; iter < columns ; iter++) {
currentRowArray[iter] = currentRow.getString(iter);
}
data.add(currentRowArray);
next = cur.next();
}
Object[][] arr = new Object[data.size()][];
data.toArray(arr);
frmMain.add(BorderLayout.CENTER, new Table(new DefaultTableModel(columnNames, arr)));
} else {
frmMain.add(BorderLayout.CENTER, "Query returned no results");
}
} else {
frmMain.add(BorderLayout.CENTER, "Query returned no results");
}
} else {
db.execute(query.getText());
frmMain.add(BorderLayout.CENTER, "Query completed successfully");
}
frmMain.revalidate();
} catch(IOException err) {
frmMain.removeAll();
frmMain.add(BorderLayout.CENTER, "Error: " + err);
frmMain.revalidate();
} finally {
Util.cleanup(db);
Util.cleanup(cur);
}
However when I run the example and try and execute a simple select query I get this error ...
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (no such table: MyTable)
So I have added the DB
I have used the 'openOrCreate' statement
Have I missed a step?
Thanks
Are you shure that you current working directory at execution is ./src ?
Try
db = Display.getInstance().openOrCreate("./src/MyDb.db");
or open with absolute filename:
db = Display.getInstance().openOrCreate("/path/to/src/MyDb.db");
You can try this cn1lib https://github.com/shannah/cn1-data-access-lib
I have used it and it works a charm, except doesn't work for 2 tables in the same query and can't perform delete operations.
Cheers
Thanks for all the input guys.
Unfortunately none of the advice worked for me.
However I did solve it in the end.
It turns out that there is a folder in my home directory called '.cn1/database'. Once I placed the DB into this folder it worked.
Two things:
1] If the db does not exist then it will create it and place it into this directory
2] The db does not show up anywhere in Netbeans (well not that I could see anyway)
Thanks again
From the developer guide:
Some SQLite apps ship with a "ready made" database. We allow you to
replace the DB file by using the code:
String path = Display.getInstance().getDatabasePath(“databaseName”);
You can then use the FileSystemStorage class to write the content of
your DB file into the path. Notice that it must be a valid SQLite
file!
Important: getDatabasePath() is not supported in the Javascript port. It
will always return null.
This is very useful for applications that need to synchronize with a
central server or applications that ship with a large database as part
of their core product.
You are relying on paths that make sense in the simulator, in the device you need to copy a resource into location. Check out the SQL demo where this is implemented: https://www.codenameone.com/blog/sql-demo-revisited.html
Use following code to copy database to Codenameone storage after put the database file in src folder, the database will copy to directory ".cn1/database" after running
String DB_NAME = "DBNAME.db";
Database temp_db = Database.openOrCreate(DB_NAME); //To create an empty file before copy, otherwise Android will throw file not found exception
temp_db.close();
String p = Database.getDatabasePath(DB_NAME);
OutputStream o = FileSystemStorage.getInstance().openOutputStream(p);
InputStream i = Display.getInstance().getResourceAsStream(getClass(), "/" + DB_NAME);
Util.copy(i, o);
This doesn't seem like a complete answer.
I'm going through the demo, and something seems to be missing:
Shouldn't a build task copy this db resource to the correct target folder? Otherwise how can a deployment ever work? (if the db doesnt get packaged up then how will it get deployed?) without it the app cant run
I am using jdbi to make connection to db and execute sql command.
dbi = new DBI("jdbc:mysql://"+dbHostName+"/"+dbName, "root", "");
dbi.withHandle(new HandleCallback<Object>() {
#Override
public Object withHandle(Handle handle) throws Exception {
handle.execute("Query to execute")
return null;
}
});
Now i want to run sql file using jdbi. I googled a lot but couldn't figure out how.
You should read your sql file to string and then execute it like
String script = ".. your sql file contents here ..";
try (Handle h = dbi.open()) {
h.createScript(script).execute();
}
I have created a web application(JSP) to extract data from mysql database to HDFS.In my Java code, I have used sqoop import command to achieve my requirement. The program executed successfully but the extracted data written to normal unix file system instead of hdfs.
Can anyone let me know how to provide hdfs file system path in sqoop import command?
package com.archival.da;
import java.sql.*;
public class DataImportSetup {
static int status=0;
public static int importsetup(String policy_id){
Connection con=GetCon.getCon();
PreparedStatement ps;
try {
ps = con.prepareStatement("SELECT
CON.SERVER,CON.PORT,CON.DB,CON.USER,
CON.PWD,POLICY.SQL_TEXT FROM POLICY
JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID WHERE POLICY.ID=?");
ps.setString(1,policy_id);
ResultSet rs=ps.executeQuery();
rs.next();
String ServerNm =
"jdbc:mysql://"+rs.getString(1)+":
"+rs.getString(2)+"/"
+rs.getString(3);
String ConUser=rs.getString(4);
String ConPass=rs.getString(5);
String SqlText=rs.getString(6);
String[] str={"import","--connect",ServerNm,"--hadoop-mapred- home","/ms/hadoop-1.2.0", "--query", SqlText , "--target-dir", "/user/root/city","--username", ConUser, "--password", ConPass,"--split-by","id"};
status=Sqoop.runTool(str);
System.out.println(status);
} catch (SQLException e) {
e.printStackTrace();
}
return status;
}
}
It's writing to the local file system instead of HDFS because the default file system is local unless otherwise configured. You can configure this to be HDFS using SqoopOptions - see this question / answer for an example:
How can I execute Sqoop in Java?
Specifically you need to locate and pass the location of your clusters core-site and hdfs-site xml files:
Configuration config = new Configuration();
config.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
config.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml"));