I want to overwrite 2 files.
Both files store information about my in memory database (HSQLDB):
db.data and db.script
My following code should do this:
public class DBReset {
public Path db_data = Paths.get("db_sep/db_backup/db.data");
public Path db_script = Paths.get("db_sep/db_backup/db.script");
public Path dest_data = Paths.get("db_sep/db.data");
public Path dest_script = Paths.get("db_sep/db.script");
public void discard() throws IOException {
Files.copy(this.db_data, this.dest_data, StandardCopyOption.REPLACE_EXISTING);
Files.copy(this.db_script, this.dest_script, StandardCopyOption.REPLACE_EXISTING);
}
}
However if I use
public Class anotherClass {
new DBReset.discard();
// do something with DB
new DBReset.discard();
// do something other with DB
}
The second discard() does not overwrites my files.
I use discard() to reset my database to its original state. Please don't ask / tell me there are other ways to reset the database, the actual problem is why it does not overwrite my files.
Sadly SHUTDOWN does not work.
public void discard() throws IOException, SQLException {
Connection c = DBConnectFactory.getDataSource.getConnection();
PreparedStatement ps = c.preparedStatement("SHUTDOWN");
ps.executeUpdate();
ps.close();
c.close();
Files.copy(this.db_data, this.dest_data, StandardCopyOption.REPLACE_EXISTING);
Files.copy(this.db_script, this.dest_script, StandardCopyOption.REPLACE_EXISTING);
}
Throws at first line of Connection c = DBConnectFactory.getDataSource.getConnection(); java.sql.Exception: error in script file line: 1 unknown token: ... lots of Unknown Sources.
Also tested in my runQuery(String query) - that opens a connection and then executes the given query via PreparedStatement - to force the SHUTDOWN, but it throws the same error as above, where I should get a Connection at getConnection().
What I want to do is:
Restore Original DB Instance.
Do some stuff with the DB, SELECT, INSERT and then do some assertions. Basically test some stuff against the Database.
Restore Original DB Instance.
Do some other tests against the Database.
PS: I'm using HSQLDB in file mode.
It seems you do not SHUTDOWN the database. If the database is open, the .data file cannot be overwritten. Execute the SQL statement below when you finish work with the database:
SHUTDOWN
This should be executed by the program that accesses the database :
new DBReset.discard();
// do something with DB -- then perform SHUTDOWN in the same java process
Related
What is the difference between those two queries:
SELECT my_fun(col_name) FROM my_table;
and
CREATE TABLE new_table AS SELECT my_fun(col_name) FROM my_table;
Where my_fun is a java UDF.
I'm asking, because when I create new table (second query) I receive a java error.
Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed
...
Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: Unable to instantiate UDF implementation class com.company_name.examples.ExampleUDF: java.lang.NullPointerException
I found that the source of error is line in my java file:
encoded = Files.readAllBytes(Paths.get(configPath));
But the question is why it works when table is not created and fails if table is created?
The problem might be with the way you read the file. Try to pass the file path as the second argument in the UDF, then read as follows
private BufferedReader getReaderFor(String filePath) throws HiveException {
try {
Path fullFilePath = FileSystems.getDefault().getPath(filePath);
Path fileName = fullFilePath.getFileName();
if (Files.exists(fileName)) {
return Files.newBufferedReader(fileName, Charset.defaultCharset());
}
else
if (Files.exists(fullFilePath)) {
return Files.newBufferedReader(fullFilePath, Charset.defaultCharset());
}
else {
throw new HiveException("Could not find \"" + fileName + "\" or \"" + fullFilePath + "\" in inersect_file() UDF.");
}
}
catch(IOException exception) {
throw new HiveException(exception);
}
}
private void loadFromFile(String filePath) throws HiveException {
set = new HashSet<String>();
try (BufferedReader reader = getReaderFor(filePath)) {
String line;
while((line = reader.readLine()) != null) {
set.add(line);
}
} catch (IOException e) {
throw new HiveException(e);
}
}
The full code for different generic UDF that utilizes file reader can be found here
I think there are several points unclear, so this answer is based on assumptions.
First of all, it is important to understand that hive currently optimize several simple queries and depending on the size of your data, the query that is working for you SELECT my_fun(col_name) FROM my_table; is most likely running locally from the client where you are executing the job, that is why you UDF can access your config file locally available, this "execution mode" is because the size of your data. CTAS trigger a job independent on the input data, this job runs distributed in the cluster where each worker fail accessing your config file.
It looks like you are trying to read your configuration file from the local file system, not from the HDSFS Files.readAllBytes(Paths.get(configPath)), this means that your configuration has to either be replicated in all the worker nodes or be added previously to the distributed cache (you can use add file from this, doc here. You can find another questions here about accessing files from the distributed cache from UDFs.
One additional problem is that you are passing the location of your config file through an environment variable which is not propagated to worker nodes as part of your hive job. You should pass this configuration as a hive config, there is an answer for accessing Hive Config from UDF here assuming that you are extending GenericUDF.
I'm asking because ALL examples I find in Google, are the same from the Fitnesse tutorial: a very simple query to a list or array in memory, NOT A REAL Database.
Yes, Fixtures never have to deal with that, but how am I supposed to test my fixtures if I can't even make the connection to the DB in a simulation of an "API"?
What I'm trying to simulate is the call from a FitNesse Fixture to query in Java into a PostgreSQL database/table. In this simple example I'm trying to obtain, at least one column from one row, in one table. When I execute the code, it runs perfectly by it's own. The problem is when trying to execute from Fitnesse through the fixture. It always fails with a ClassNotFoundException, when calling the JDBC driver. This doesn't happen by running the code by it's own.
Here is the code that does the query:
package queriespackage;
import java.sql.*;
public class testQuery01 {
public static Connection openDBConnection(){
Connection connectionString = null;
try {
String dbhost = "SOMEURL";//Redacted
String port = "SOMEPORT";//Redacted
String dbname = "THEDBNAME";//Redacted
String username = "SOMEUSER";//Redacted
String password = "SOMEPASSWORD";//Redacted
String driverJDBC = "org.postgresql.Driver";
Class.forName(driverJDBC);
connectionString = DriverManager.getConnection("jdbc:postgresql://" + dbhost + ":" + port + "/" + dbname,username,password);
connectionString.setAutoCommit(false);
} catch (SQLException e) {
e.printStackTrace(System.err);
System.exit(0);
} catch (Exception e) {
System.err.println( e.getClass().getName()+": "+ e.getMessage() );
System.exit(0);
};
return connectionString;
};
public static ResultSet executeQuery(Connection connectionString, int intAccountId) throws SQLException{
Statement querySession = connectionString.createStatement();
//The query string
String queryString = "SELECT DISTINCT "
+ "account_search.account_id,"
+ "account_search.account_name"
+ " FROM account_search "
+ " WHERE"
+ " account_search.account_id = "+ intAccountId
+ "LIMIT 1";
ResultSet queryResult = querySession.executeQuery(queryString);
return queryResult;
};
public static String processQueryResult(ResultSet queryResult) throws SQLException{
String strQueryValueReturned = null;
while (queryResult.next()) {
strQueryValueReturned = queryResult.getString("account_name");
};
return strQueryValueReturned;
};
public static boolean closeDBConnection(Connection connectionString){
try {
if(connectionString!=null){
connectionString.close();
}
} catch (SQLException e) {
System.err.println( e.getClass().getName()+": "+ e.getMessage() );
System.exit(0);
};
return true;
};
public static String testQuery(int intAccountId) throws SQLException, ClassNotFoundException{
boolean bolConnectionStatus = false;
String strValueReturned = null;
Connection connectionString = openDBConnection();
if(connectionString != null){
ResultSet qryQueryResult = executeQuery(connectionString, intAccountId);
strValueReturned = processQueryResult(qryQueryResult);
bolConnectionStatus = closeDBConnection(connectionString);
if(!bolConnectionStatus){
System.exit(0);
}
}else{
System.exit(0);
};
return strValueReturned;
};
};
If I add a Main method to that code, passing it the argument value for "intAccountId", it successfully returns the name of the account "account_name", just as expected.
Now here's the Fixture that should be called by the FitNesse test:
package fixturespackage;
import java.sql.SQLException;
import queriespackage.testQuery01;
public class testFixture01{
private int Int_AccountId;
//Fixture Constructor (setter)
public testFixture01(int Int_AccountId){
this.Int_AccountId = Int_AccountId;
};
public String query() throws SQLException, ClassNotFoundException{
return testQuery01.testQuery(Int_AccountId);
};
};
Just as the FitNesse guide says, there must be a "query" method, that does the actual call to the interface in the DB. I had to add a constructor instead of the "setter", because FitNesse actually demands it: "Could not invoke constructor for fixturepackage.testFixture01"
Here's the FitNesse page:
!***> System Variables
!define TEST_SYSTEM {slim}
!path C:\FitnessTest\bin
*!
|Query: fixturespackage.testFixture01|21 |
|Str_AccountName |
|SomeName |
Here's a Screenshot of my BuildPath, so you can see I have the JDBC Library for Java 8, JDK 1.8, JRE 1.8... and the "org.postgresql.Driver.class" is included in the project.
This is the error I receive, when running from FitNesse:
This is the error I get, when debugging the line where FitNesse failed by using Inspect tool:
... and YES, I also tried by hard coding the name of the JDBC:
I have searched a lot for a REAL LIFE example, both here, the FitNesse Guide and Google.
The FitNesse Guide might be extensive, but let's be sincere, it's full of "dirty word here", unrealistic and incomplete examples and missing a lot of information.
So, asking again, has anyone done a REAL LIFE test making queries, using FitNesse, that could help me find out what am I doing wrong?
I have to admit I've only done limited database tests with FitNesse, but I have used them (to query DB2).
I did not use query tables (or wrote my own fixtures to query), but instead used jdbcslim in combination with script tables and scenario's.
That fact that the driver class cannot be found suggests that although the jar is present on the classpath in your IDE it is not available when FitNesse is running your fixture code.
I notice you specify the classpath as a single directory in the wiki. In Java that means that all class files should be in that directory (as .class files, in the right subdirectory for their defined package). It will not pick up any jars (or zips) in that directory. Did you unpack your database driver's jar to that directory? If not, you need to add a !path line pointing to the jar (so the entire path including the filename) with the database driver.
Now listing every jar you need can quickly become cumbersome, so you can also use wildcards. I tend to copy all the jars I need to a single directory, that also contains my fixture .class files, and add a single !path line loading all jars in that directory.
So if you also copied your database driver to the directory in you question you could ensure it, and your own fixture, to be available via
!path C:\FitnessTest\bin
!path C:\FitnessTest\bin\*.jar
I am using jdbi to make connection to db and execute sql command.
dbi = new DBI("jdbc:mysql://"+dbHostName+"/"+dbName, "root", "");
dbi.withHandle(new HandleCallback<Object>() {
#Override
public Object withHandle(Handle handle) throws Exception {
handle.execute("Query to execute")
return null;
}
});
Now i want to run sql file using jdbi. I googled a lot but couldn't figure out how.
You should read your sql file to string and then execute it like
String script = ".. your sql file contents here ..";
try (Handle h = dbi.open()) {
h.createScript(script).execute();
}
I have created a web application(JSP) to extract data from mysql database to HDFS.In my Java code, I have used sqoop import command to achieve my requirement. The program executed successfully but the extracted data written to normal unix file system instead of hdfs.
Can anyone let me know how to provide hdfs file system path in sqoop import command?
package com.archival.da;
import java.sql.*;
public class DataImportSetup {
static int status=0;
public static int importsetup(String policy_id){
Connection con=GetCon.getCon();
PreparedStatement ps;
try {
ps = con.prepareStatement("SELECT
CON.SERVER,CON.PORT,CON.DB,CON.USER,
CON.PWD,POLICY.SQL_TEXT FROM POLICY
JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID WHERE POLICY.ID=?");
ps.setString(1,policy_id);
ResultSet rs=ps.executeQuery();
rs.next();
String ServerNm =
"jdbc:mysql://"+rs.getString(1)+":
"+rs.getString(2)+"/"
+rs.getString(3);
String ConUser=rs.getString(4);
String ConPass=rs.getString(5);
String SqlText=rs.getString(6);
String[] str={"import","--connect",ServerNm,"--hadoop-mapred- home","/ms/hadoop-1.2.0", "--query", SqlText , "--target-dir", "/user/root/city","--username", ConUser, "--password", ConPass,"--split-by","id"};
status=Sqoop.runTool(str);
System.out.println(status);
} catch (SQLException e) {
e.printStackTrace();
}
return status;
}
}
It's writing to the local file system instead of HDFS because the default file system is local unless otherwise configured. You can configure this to be HDFS using SqoopOptions - see this question / answer for an example:
How can I execute Sqoop in Java?
Specifically you need to locate and pass the location of your clusters core-site and hdfs-site xml files:
Configuration config = new Configuration();
config.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
config.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml"));
I am trying to get a connection to a SQLite database (using Eclipse on Windows 8). Everything workes fine as long as the path name doesn't contain any special characters (like "é"). I tried to convert it to UTF-8 (because I read on http://www.sqlite.org/c3ref/open.html that it should be), but it didn't work. I get an "out of memory" exception (SQLException) what means that no database file was found.
This is the code summary of what I did:
public static String DB_PATH = "jdbc:sqlite:" + System.getProperty("user.home") + "<Rest of the path><databasename>.sqlite";
public static void main(String[] args) throws ClassNotFoundException
{
// load the sqlite-JDBC driver using the current class loader
Class.forName("org.sqlite.JDBC");
Connection connection = null;
try
{
// create a database connection
connection = DriverManager.getConnection(DB_PATH);
Statement statement = connection.createStatement();
statement.setQueryTimeout(30); // set timeout to 30 sec.
// work with the database ...
}
}
catch(SQLException e)
{
// if the error message is "out of memory",
// it probably means no database file is found
System.err.println(e.getMessage());
}
finally
{
// try to disconnect
// ...
}
Thanks for your help!
Use the latest lib from http://www.xerial.org/maven/repository/artifact/org/xerial/sqlite-jdbc/