I'm trying to use EsperIO to load some information from database and use it in other queries with different conditions. To do it I'm using the following code:
ConfigurationDBRef dbConfig = new ConfigurationDBRef();
dbConfig.setDriverManagerConnection("org.postgresql.Driver",
"jdbc:postgresql://localhost:5432/myDatabase",
"myUser", "myPassword");
Configuration engineConfig = new Configuration();
engineConfig.addDatabaseReference("myDatabase", dbConfig);
// Custom class
engineConfig.addEventType("UserFromDB", UserDB.class);
EPServiceProvider esperEngine = EPServiceProviderManager.getDefaultProvider(engineConfig);
String statement = "insert into UserFromDB "
+ " select * from sql:myDatabase ['SELECT * from data.user']";
//Install this query in the engine
EPStatement queryEngineObject = esperEngine.getEPAdministrator().createEPL(statement);
// 1. At this point I can iterate over queryEngineObject without problems getting the information sent by database
// This query is only a 'dummy example', the 'final queries' are more complex
statement = "select * from UserFromDB";
EPStatement queryEngineObject2 = esperEngine.getEPAdministrator().createEPL(statement);
// 2. If I try to iterate over queryEngineObject2 I receive no data
How can I reuse UserFromDB stored information in other queries? (in the above example, in queryEngineObject2)
You don't have a stream since the database doesn't provide a stream. The database query provides rows only when its being iterated/pulled.
One option is to loop over each row and send it into the engine using "sendEvent":
// create other EPL statements before iterating
Iterator<EventBean> it = statement.iterator();
while(it.hasNext()) {
epService.getEPRuntime().sendEvent(event);
}
Related
I want to copy a table (10 million records) in originDB(sqlite3) into another database called targetDB.
The process of my method is:
read data from the origin table and generate a ResultSet, then generate corresponding insert sql about every record and execute commit to batch insert when the count of record reach 10000. The code as follow:
public void transfer() throws IOException, SQLException {
targetDBOperate.setCommit(false);//batch insert
int count = 0;
String[] cols = parser(propertyPath);//get fields of data table
String query = "select * from " + originTable;
ResultSet rs = originDBOperate.executeQuery(query);//get origin table
String base = "insert into " + targetTable;
while(rs.next()) {
count++;
String insertSql = buildInsertSql(base,rs,cols);//corresponding insert sql
targetDBOperate.executeSql(insertSql);
if(count%10000==0) {
targetDBOperate.commit();// batch insert
}
}
targetDBOperate.closeConnection();
}
The follow picture is the trend of using memory, and vertical axis represents memory usage
As we can say it will be bigger and bigger until out of memory. The stackoverflow has some relevant question such as Out of memory when inserting records in SQLite, FireDac, Delphi
, but I havent solve my problem for we use different implement method. My hypothesis is that when the count of record hasn't reach 10000, these corresponding insert sql will be cached in memory and they haven't been removed when execute commit by default? Every advice will be appreciate.
By moving a higher number of rows in SQLite or any other relational database you should follow some basic principles:
1) set autoCommit to false, i.e. do not commit each insert
2) use batch update, i.e. do not round trip for each row
3) use prepared statement, i.e. do not parse each insert.
Putting this together you get following code:
cn is the source connection, cn2 is the target connection.
For each inserted row you call addBatch, but only once per batchSize you call executeBatch which initiates a round trip.
Do not forget a last executeBatch at the end of the loop and the final commit.
cn2.setAutoCommit(false)
String SEL_STMT = "select id, col1,col2 from tab1"
String INS_STMT = "insert into tab2(id, col1,col2) values(?,?,?)"
def batchSize = 10000
def stmt = cn.prepareStatement(SEL_STMT)
def stmtIns = cn2.prepareStatement(INS_STMT)
rs = stmt.executeQuery()
while(rs.next())
{
stmtIns.setLong(1,rs.getLong(1))
stmtIns.setString(2,rs.getString(2))
stmtIns.setTimestamp(3,rs.getTimestamp(3))
stmtIns.addBatch();
i += 1
if (i == batchSize) {
def insRec = stmtIns.executeBatch();
i = 0
}
}
rs.close()
stmt.close()
def insRec = stmtIns.executeBatch();
stmtIns.close()
cn2.commit()
Sample test with your size with sqlite-jdbc-3.23.1:
inserted rows: 10000000
total time taken to insert the batch = 46848 ms
I do not observe any memory issues or problems with a large transaction
You are trying to fetch 10M records in one go by doing the following. This will definitely eat your memory like anything
String query = "select * from " + originTable;
ResultSet rs = originDBOperate.executeQuery(query);//get origin table
Use paginated queries to read the batches and do batch updates according.
You are not even doing a batch-update You are simply firing 10K queries one after the other by doing the following code
String insertSql = buildInsertSql(base,rs,cols);//corresponding insert sql
targetDBOperate.executeSql(insertSql);
if(count%10000==0) {
targetDBOperate.commit();// This simply means that you are commiting after 10K records
}
I have a problem with this query when I pass it to an oracle dbms
SELECT * FROM RD_RBF WHERE REQUEST_ID = 'S2N-F01-000000000001'
because of the dashes in the string the jvm return me this exception
java.sql.SQLException: Fail to convert to internal representation
How can I pass this query to oracle correctly? Thanks a lot
P.S. I'm not shure of the code because I'm using Talend software that generates automatically the code of components but I can post part of the code above
String dbquery_tOracleInput_1 = "SELECT * FROM RD_RBF WHERE REQUEST_ID = 'S2N-F01-000000000001'";
java.sql.ResultSet rs_tOracleInput_1 = null;
try{
rs_tOracleInput_1 = stmt_tOracleInput_1.executeQuery(dbquery_tOracleInput_1);
java.sql.ResultSetMetaData rsmd_tOracleInput_1 = rs_tOracleInput_1.getMetaData();
int colQtyInRs_tOracleInput_1 = rsmd_tOracleInput_1.getColumnCount();
Use toraclerow component.
Query as follows in component:
"SELECT * FROM RD_RBF WHERE REQUEST_ID = ?"
Go to advance setting, select use prepared statement and add 1 paarmeter index.
also select the propagate Query's record set.
tOracleRow_1 -----> tParseRecordSet----->
I am trying to make a simple program to fetch data from a table.
I am following http://www.avaje.org/ebean/getstarted_props.html#iud but am unable to get data. I have created new Entity Class from Database from Netbeans (which creates classes from relations). Here is what I am using:
ebean.properties
ebean.ddl.generate=true
ebean.ddl.run=true
ebean.debug.sql=true
ebean.debug.lazyload=false
ebean.logging=all
ebean.logging.logfilesharing=all
ebean.logging.directory=D:\\logs
ebean.logging.iud=sql
ebean.logging.query=sql
ebean.logging.sqlquery=sql
ebean.logging.txnCommit=none
datasource.default=h2
datasource.h2.username=sa
datasource.h2.password=
datasource.h2.databaseUrl=jdbc:h2:mem:tests;DB_CLOSE_DELAY=-1
datasource.h2.databaseDriver=org.h2.Driver
datasource.h2.minConnections=1
datasource.h2.maxConnections=25
datasource.h2.heartbeatsql=select 1
datasource.h2.isolationlevel=read_committed
datasource.mysql.username=root
datasource.mysql.password=kalsym#123
datasource.mysql.databaseUrl=jdbc:mysql://127.0.0.1:3306/wsp
datasource.mysql.databaseDriver=com.mysql.jdbc.Driver
datasource.mysql.minConnections=1
datasource.mysql.maxConnections=25
datasource.mysql.isolationlevel=read_committed
Table Data
Insert into routing_algo_type (name, description) values ('LCR', 'Least Cost Routing');
Code to fetch data
RoutingAlgoType routingObj = new RoutingAlgoType();
routingObj.setName("LCR");
RoutingAlgoType routingObj2 = Ebean.find(RoutingAlgoType.class, routingObj);
System.out.println("Got "+routingObj2.getDescription());
Now find returns null, which means it cant find the data?
I used following code to test connection
String sql = "select count(*) as count from dual";
SqlRow row = Ebean.createSqlQuery(sql).findUnique();
Integer i = row.getInteger("count");
System.out.println("Got " + i + " - DataSource good.");
Result from above code is
Got 1 - DataSource good.
Is there any way to check the connection?
I am new to neo4j and graph database, and I have to send a query to get some values.
I have food and category nodes, and the relationship type between the two is specified by another node categorized_as.
What I need to fetch is the pair of food_name and its category_name.
Thanks for your help in advance.
Here's the documentation on how to run cypher queries from java. Adapted for your example, it would look like this:
// Create a new graph DB at path DB_PATH
GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( DB_PATH );
// Create a new execution engine for running queries.
ExecutionEngine engine = new ExecutionEngine( db );
ExecutionResult result;
// Queries need to be run inside of transactions...
try ( Transaction ignored = db.beginTx() )
{
String query = "MATCH (f:food)-[:categorized_as]->(c:category) RETURN f.food_name as foodName, c.category_name as categoryName";
// Run that query we just defined.
result = engine.execute(query);
// Pull out the "foodNames" column from the result indicated by the query.
Iterator<String> foodNames = result.columnAs( "foodName" );
// Iterate through foodNames...
}
I am trying to implement PreparedStatement, which won't work with sql DB.
Suppose I have the following sql query:
String selectSqlQuery = "SELECT * FROM customer WHERE f1 = ? AND f2 =? AND f3 > ?";
and the following code:
//----
prest = con.prepareStatement(selectSqlQuery );
prest.setString(1, "val1");
prest.setString(2, "val2");
prest.setInt(3, 108);
ResultSet rs = prest.executeQuery();
//---
My question is how to implement setString and setInt methods for injecting params?
For now I save parameters' indexes and values into HashMap, but after it I can't make injection into sql query string.
implementation of sql's java interfaces are part of vendor specific jdbc driver. You probably just need to get the proper jdbc jar file for you database. writing implementations of such stuff is usually just needed if you intend to write your own database driver...
Since you're writing your own driver, you can play with your class a little. Let's change the approach. If you have a query like this one:
"SELECT * FROM table WHERE id = ? AND name = ?"
Replace the ? to turn it into
"SELECT * FROM table WHERE id = {0} AND name = {1}"
About your set methods, those will have to save your new parameters in an Object array, again matching against the index.
Object parameterArray = new Object[1];
public boolean setString(int paramIndex, String param) {
if(paramIndex < 0 || paramIndex > parameterArray.length)
throw new IllegalArgumentException("Can't set parameter " + paramIndex + ", The query only has " + parameterArray.length + " parameters.");
parameterArray[paramIndex - 1] = param;
}
Before executing the query, take advantage of your formatted string and set the parameters:
MessageFormat messageFormat = new MessageFormat(query);
String newQuery = messageFormat.format(parameterArray);
The format method will replace the {number} substrings for the corresponding element in the index represented by the number between brackets.