Develop Reference

JDBC optimize MySql request on Multithread

I'm building a webcrawler and I'm looking for the best way to handle my requests and connection between my threads and the database (MySql).
I've 2 types of threads :
Fetchers : They crawl websites. They produce url and add they into 2 tables : table_url and table_file. They select from table_url
to continue the crawl. And update table_url to set visited=1 when they
have read a url. Or visited=-1 when they are reading it. They can
delete row.
Downloaders : They download files. They select from table_file. They update table_file to change the Downloaded column. They never
insert anything.
Right now I'm working with this :
I've a pool of connection based on c3p0.
Every target (website) have thoses variables :
private Connection connection_downloader;
private Connection connection_fetcher;
I create both connection only once when I instanciate a website. Then every thread will use thoses connections based on their target.
Every thread have thoses variables :
private Statement statement;
private ResultSet resultSet;
Before every Query I open a SqlStatement :
public static Statement openSqlStatement(Connection connection){
try {
return connection.createStatement();
} catch (SQLException e) {
e.printStackTrace();
}
return null;
}
And after every Query I close sql statement and resultSet with :
public static void closeSqlStatement(ResultSet resultSet, Statement statement){
if (resultSet != null) try { resultSet.close(); } catch (SQLException e) {e.printStackTrace();}
if (statement != null) try { statement.close(); } catch (SQLException e) {e.printStackTrace();}
}
Right now my Select queries only work with one select (I never have to select more than one for now but this will change soon) and is defined like this :
public static String sqlSelect(String Query, Connection connection, Statement statement, ResultSet resultSet){
String result = null;
try {
resultSet = statement.executeQuery(Query);
resultSet.next();
result = resultSet.toString();
} catch (SQLException e) {
e.printStackTrace();
}
closeSqlStatement(resultSet, statement);
return result;
}
And Insert, Delete and Update queries use this function :
public static int sqlExec(String Query, Connection connection, Statement statement){
int ResultSet = -1;
try {
ResultSet = statement.executeUpdate(Query);
} catch (SQLException e) {
e.printStackTrace();
}
closeSqlStatement(resultSet, statement);
return ResultSet;
}
My question is simple : can this be improved to be faster ? And I'm concerned about mutual exclusion to prevent a thread to update a link while another is doing it.

I believe your design is flawed. Having one connection assigned full-time for one website will severly limit your overall workload.
As you already have setup a connection pool, it's perfectly okay to fetch before you use (and return afterwards).
Just the same, try-with-catch for closing all your ResultSets and Statements after will make code more readable - and using PreparedStatement instead of Statement would not hurt as well.
One Example (using a static dataSource() call to access your pool):
public static String sqlSelect(String id) throws SQLException {
try(Connection con = dataSource().getConnection();
PreparedStatement ps = con.prepareStatement("SELECT row FROM table WHERE key = ?")) {
ps.setString(1, id);
try(ResultSet resultSet = ps.executeQuery()) {
if(rs.next()) {
return rs.getString(1);
} else {
throw new SQLException("Nothing found");
}
}
} catch (SQLException e) {
e.printStackTrace();
throw e;
}
}
Following the same pattern I suggest you create methods for all the different Insert/Update/Selects your application uses as well - all using the connection only for the short time inside the DB logic.

I can not see a real advantage to have all the Database stuff in your webcrawler threads.
Why don't you use a static class with the sqlSelect and sqlExec method, but without the Connection and ResultSet parameters. Both connection objects are static as well. Make sure the connection objects are valid befor using them.

Can I put two Selects in a try?

try {
Connection lig = DriverManager.getConnection(
"jdbc:mysql://localhost/gym", "root", "0000");
PreparedStatement inst = lig
.prepareStatement("SELECT * FROM produtos_has_historico WHERE Produtos_idProdutos AND Historico_idHistorico");
ResultSet a = inst.executeQuery();
while (a.next()){
DefaultListModel model = new DefaultListModel();
model.addElement(a);
}
} catch( Exception e ){}
In this Select, I get the id historic and the id product, but I wanted to get also the name and price of the products that are in my other table "products" to add to my Jlist, can I use two selects ? Thank you.

It's not a problem to have two selects in a single try-catch block.

If you want to handle the generated exception the same way regardless of where it occurred, then I don't see a problem with enclosing multiple statements in the same try block.

Yes, it's fine to put multiple calls that may fail into a single try/catch block — in fact, that's a big part of why we have try/catch blocks (and exception handling in general), to move the code that handles exceptional things (failures) out of the way of the main code, so it's easier to understand the main code.
Compare these bits of pseudo-code
result = doThis();
if (result != success) {
handleTheFailure(result)
} else {
result = doThat();
if (result != success) {
handleThefailure(result);
} else {
result = doTheOther();
if (result != success) {
handleTheFailure(result);
}
}
}
vs.
try {
doThis();
doThat();
doTheOther();
}
catch (failure) {
handleFailure(failure);
}
More in the Java tutorial on exceptions.

No problem. I also recommend you use try-with-resources instead and be more specific about the exceptions you want to catch:
try (Connection conn = DriverManager.getConnection("...")) {
PreparedStatement ps1 = conn.prepareStatement("...");
try (ResultSet rs1 = ps1.executeQuery()) {
/* Parse first result */
}
PreparedStatement ps2 = conn.prepareStatement("...");
try (ResultSet rs2 = ps2.executeQuery()) {
/* Parse second result */
}
} catch (SQLException ex) {
for (Throwable t : ex) {
t.printStackTrace();
}
}

Instead of going for two selects - you can write a SQL query having a join in it .
You will be able to perform the join if you have common columns in your tables.
Then you can write a query using join and then the same code you can use to perform the operations and you can fetch all the values that you want.
For SQL joins basic - you can visit this site -
http://www.w3schools.com/sql/sql_join.asp

Execute a SQL query in a loop

I have this scenario. I will trigger a job in the server and as soon as the job is triggered an entry will be made into the job table with Execution_status_code as 1. I need to wait for some time say 5 mins and recheck the Execution_status_code value. As soon as the value is changed to 2, I need to proceed further.
I am using an existing connection for connecting to the database. I need to execute the SQL and if the SQL output is In progress, I need to wait for some time and then again execute the statement. Do this until the SQL output is success, until then keep waiting.
Below is the code I have tried.
Thread t = new Thread();
java.sql.Connection conn_javaComp = (java.sql.Connection)globalMap.get("conn_tNetezzaConnection_1");
java.sql.Statement st = null;
java.sql.ResultSet rs = null;
String check = null;
String dbquery_javaComp = "select case when EXECUTION_STATUS_CODE = 2 then 'Success' when EXECUTION_STATUS_CODE = 1 then 'In progress' else 'Failure' end as EXECUTION_STATUS_CODE from JOB_BKUP_NCR where JOB_TYPE_CODE="+context.JobTypeCode+" and Load_id = (select max(load_id) from JOB_BKUP_NCR where job_type_code="+context.JobTypeCode+") and START_DATETIME = (select max(START_DATETIME) from JOB_BKUP_NCR where job_type_Code="+context.JobTypeCode+")";
try
{
do
{
st = conn_javaComp.createStatement();
rs = st.executeQuery(dbquery_javaComp);
if(rs.next())
{
check = rs.getString(1);
System.out.println(check);
if (check.equalsIgnoreCase("In Progress"))
{
t.sleep(1000);
System.out.println("thread executed1");
System.out.println(dbquery_javaComp);
System.out.println(check);
}
}
else {
System.out.println(" No data found");
}
}while (!"Success".equals(check));
}
catch (Exception e) {
e.printStackTrace();
} finally {
try {
if( rs != null)
rs.close();
if( st!= null)
st.close();
}
catch (Exception e1) {
e1.printStackTrace();
}
}
The output i am getting is 'In Progress'. The loop is struck at In progress even after i change the value in the database. I am not sure where i am doing wrong. Any suggestions?

You are creating a new statement and a new resultset inside the loop, and so, they should be close inside the loop. I am thinking that your connection got corrupted with multiple statements and resultset without closing them. Please try to close them and see if that work.

The data that you are seeing will be cached.
Try closing and re-opening your DB connection. This may not even be good enough if you are using DB pooling.

There are many things I can foresee going wrong with your code. For once most DBMS will either lock the rows until you commit / close the connection or give you a snapshot of the data instead, hence you don't see the updated value or the transaction that supposed to update it wouldn't go through. Try comitting or close/reopen the transaction per loop iteration.
I would also doubt if this is a good code design as you are doing "polling". Consider if you can find other method of getting notified of the event.

try
{
//declare here your statement and resultset
st = conn_javaComp.createStatement();
rs = st.executeQuery(dbquery_javaComp);
do
{
if(rs.next())
{
check = rs.getString(1);
System.out.println(check);
if (check.equalsIgnoreCase("In Progress"))
{
t.sleep(1000);
System.out.println("thread executed1");
System.out.println(dbquery_javaComp);
System.out.println(check);
}
}
else {
System.out.println(" No data found");
}
}while (!"Success".equals(check));

jdbc performance

There are thee tables inside my database. One is employee, the second is employee_Project, and the third is employee_Reporting. Each table has a common employee_Number as its primary key, and there is a one to many relationship among them such that an employee has many projects and reporting dates.
I have run select * from employee, select * from employee_project, select * from employee_reporting in three data holder classes which have methods fillResultSet(Result set) and List<T> getData(). This is based on a SqlDbEngine class with a runQuery(PreparedStatement,DataHolder) method, and the implementation has been completed.
Now I have to design a getAllEmployee() method along with project and reporting detail with optimal code in java using JDBC. I have used an iterator but this solution is not acceptable; now I have to use a foreach loop.
This is what I have done:
public List<Employee> getAllEmployees() {
EmployeeDataHolderImpl empdataholder = new EmployeeDataHolderImpl();
List<Employee> list_Employee_Add = null;
try {
Connection connection = mySqlDbConnection.getConnection();
PreparedStatement preparedStatement = connection
.prepareStatement(GET_ALL_EMPLOYEE_DETAILS);
mySqlDBEngineImpl.runQuery(preparedStatement, empdataholder);
} catch (SQLException e) {
e.printStackTrace();
}
for (Employee employee : empdataholder.getData()) {
new EmployeeDAOImpl().getProject(employee);
new EmployeeDAOImpl.getReport(employee);
}
list_Employee_Add = empdataholder.getData();
return list_Employee_Add;
}
and make another method
public void getProject(Employee emp) {
EmployeeProjectDataHolderImpl employeeProjectHolder = new EmployeeProjectDataHolderImpl();
try {
Connection connection = mySqlDbConnection.getConnection();
PreparedStatement preparedStatement = connection
.prepareStatement(GET_ALL_PROJECT_DETAILS);
mySqlDBEngineImpl
.runQuery(preparedStatement, employeeProjectHolder);
} catch (SQLException e) {
e.printStackTrace();
}
for (EmployeeProject employee_Project : employeeProjectHolder.getData()) {
if (employee_Project.getEmployeeNumber() == emp.getEmpNumber()) {
emp.getProjects().add(employee_Project);
}
}
}
public void getReport(Employee emp) {
EmployeeReportDataHolderImpl employeeReportHolder = new EmployeeReportDataHolderImpl();
try {
Connection connection = mySqlDbConnection.getConnection();
PreparedStatement preparedStatement = connection
.prepareStatement(GET_ALL_REPORT_DETAILS);
mySqlDBEngineImpl
.runQuery(preparedStatement, employeeReportHolder);
} catch (SQLException e) {
e.printStackTrace();
}
for (EmployeeReport employee_Report : employeeReportHolder.getData()) {
if (employee_Report.getEmployeeNumber() == emp.getEmpNumber()) {
emp.getProjects().add(employee_Project);
}
}
}
}
and same for Employee Reporting but doing, this performance is going to decrease.no body worry about closing connection i will do it
Please tell me how I could improve my solution..

There are some issue with your code.
1.you are initializing EmployeeDAOImpl everytime, rather you can just keep one instance and call the operations over it.
new EmployeeDAOImpl().getProject(employee); new
EmployeeDAOImpl.getReport(employee);
2.I don't see where you close your connection after performing an SQL operation.

You should be having
try {
--code statements
}
catch(SQLException e){
e.printStackTrace();
}
finally{
-- close your connection and preparedStatement
}
Closing database connections is very vital.

If you use your actual code, you will have 3 impacts in your code:
You're opening a connection to get the employee's data.
For every employee, you open (and close) a new connection to get his projects.
For every employee, you open (and close) a new connection to get his reports.
Note that opening a new connection is a performance hit on your application. It doesn't matter if you use an enhanced for-loop or an Iterator, there would be many hits that can slow down your application.
Two ways to solve this problem:
Open a single connection where you run all your select statements. This will be better than opening/closing lot of connections.
Create a single SQL statement to retrieve the employees and the data you need for every employee. It will have better performance for different reasons:
A single connection to the database.
A single query instead of lot of queries to the database (a single I/O operation).
If your rdbms allows it, the query will be optimized for future requests (a single query instead of multiple queries).
I would prefer to go with the second option. For this, I tend to use a method that executes any SQL select statement and return a ResultSet. I'll post a basic example (note, the provided code can be improved depending on your needs), this method could be in your SqlDbEngine class:
public ResultSet executeSQL(Connection con, String sql, List<Object> arguments) {
PreparedStatement pstmt = null;
ResultSet rs = null;
try {
pstmt = con.prepareStatement(sql);
if (arguments != null) {
int i = 1;
for(Object o : arguments) {
pstmt.setObject(i++, o);
}
}
//method to execute insert, update, delete statements...
rs = pstmt.execute();
} catch(SQLException e) {
//handle the error...
}
return rs;
}
And this other method to handle all the query operation
public List<Employee> getAllEmployee() {
Connection con = null;
ResultSet rs = null;
List<Employee> lstEmployee = new ArrayList<Employee>();
try {
con = mySqlDbConnection.getConnection();
//write the sql to retrieve all the data
//I'm assuming these can be your columns, it's up to you
//this can be written using JOINs...
String sql = "SELECT E.EMPLOYEE_ID, E.EMPLOYEE_NAME, P.PROJECT_NAME, R.REPORT_NAME FROM EMPLOYEE E, PROJECT P, REPORT R WHERE E.EMPLOYEE_ID = P.EMPLOYEE_ID AND E.EMPLOYEE_ID = R.EMPLOYEE_ID";
//I guess you don't need parameters for this...
rs = SqlDbEngine.executeSQL(con, sql, null);
if (rs != null) {
Employee e;
int employeeId = -1, lastEmployeeId = -1;
while (rs.next()) {
//you need to make sure to create a new employee only when
//reading a new employee id
employeeId = rs.getInt("EMPLOYEE_ID");
if (lastEmployeeId != employeeId) {
e = new Employee();
lastEmployeeId = employeeId;
lstEmployee.add(e);
}
Project p = new Project();
Report r = new Report();
//fill values of p...
//fill values of r...
//you can fill the values taking advantage of the column name in the resultset
//at last, link the project and report to the employee
e.getProjects().add(p);
e.getReports().add(r);
}
}
} catch (Exception e) {
//handle the error...
} finally {
try {
if (rs != null) {
Statement stmt = rs.getStatement();
rs.close();
stmt.close();
}
if (con != null) {
con.close();
}
} catch (SQLException e) {
//handle the error...
}
}
return lstEmployee;
}
Note that the second way can be harder to code but it will give you the best performance. It's up to you to improve the provided methods, some advices:
Create a class that receives a ResultSet and builds a Project instance using the columns name of the ResultSet (similar for Report and Employee).
Create a method that handles the ResultSet and its Statement close.
As a best practice, never use select * from mytable, it's preferable to write the needed columns.

If I understand correctly, your code first loads all EmployeeReport rows and then filters them according to getEmployeeNumber(). You can let your database do this by modifying your SQL query.
Since you didn't show your SQL queries (I assume they're in GET_ALL_REPORT_DETAILS), I'll just make a guess... Try executing SQL like:
select *
from employee_reporting
where employeeNumber = ?
If you put this in a PreparedStatement, and then set the parameter value, your database will only return the data you need. For example:
PreparedStatement pstmt = con.prepareStatement(GET_ALL_REPORT_DETAILS);
pstmt.setInt(1, employee.getEmployeeNumber());
That should return only the EmployeeReport records having the desired employeeNumber. In case performance is still an issue, you could consider adding an index to your EmployeeReport table, but that's a different story...

Executing two Java PreparedStatements with one connection - style choice

Okay, I've realized that I really have asked way too many questions without contributing back to the community, but I want your opinions on this. Say if I have
private void closeAll(ResultSet rs, PreparedStatement ps, Connection con) {
if (rs != null)
try {
rs.close();
} catch (SQLException e) {
}
if (ps != null)
try {
ps.close();
} catch (SQLException e) {
}
if (con != null)
try {
con.close();
} catch (SQLException e) {
}
}
and I wanted to do several operations on my MySQL database using a single connection. Is it better to write
Connection con = ...;
PreparedStatement ps = null;
ResultSet rs = null;
try {
ps = con.prepareStatement(...);
rs = ps.executeQuery();
if (rs.next()) ...;
} catch (SQLException e) {
System.err.println("Error: " + e);
} finally {
closeAll(rs, ps, null);
}
try {
ps = con.prepareStatement(...);
rs = ps.executeQuery();
if (rs.next()) ...;
} catch (SQLException e) {
System.err.println("Error: " + e);
} finally {
closeAll(rs, ps, con);
}
or
Connection con = ...;
PreparedStatement ps = null;
ResultSet rs = null;
try {
ps = con.prepareStatement(...);
rs = ps.executeQuery();
if (rs.next()) ...;
rs.close();
ps.close();
ps = con.prepareStatement(...);
rs = ps.executeQuery();
if (rs.next()) ...;
} catch (SQLException e) {
System.err.println("Error: " + e);
} finally {
closeAll(rs, ps, con);
}
I consider better to mean either safer, clearer, more concise, or more robust. I'm not sure whether the latter will always close whichever prepared statements and result sets are open whenever it encounters an exception, while I believe it does look more concise. But the former looks nicer since it's more consistent, yet it puts more overhead since it uses more try finally blocks.
I realize that Java 7's automatic resource management part of Project Coin will force me to lean to the former since the resources used in the header are implicitly final in the body. However, I have quite some time before I have to worry about revising my code to adapt it to ARM and be able to remove the boilerplate code, so the question still stands: of the above two styles, which would be better practice? If they both do the expected behaviors, will the latter give me a noticeable performance boost that would excuse the "uglier" style?

It seems to me this is a case of personal preference. In my time I've written code that resembles both blocks. As regards performance I don't think there would be a particularly noticeable difference, only performance tests would tell.
It might be a case that the millisecond or so difference that one version delivers isn't half as important as the ten minutes or so that another person reading your code six months after you've written it would spend asking why.
My personal preference would be the second. You say you're holding a connection open to the database. With the first block of code if you get an exception thrown, that would be handled but then you'd drop down to the second try/catch and try again. You might not want that if the first one failed. With the second, an exception would cause you come out of the code and then close your connection.
I've programmed mainly in C#. It was about eight years ago when I last did any Java. But I think there is and have been plenty of C# programmers who've pondered this. I have at any rate.

If you look at it, you'll see that logic is somewhat different - first version will execute second query even if first query handling will fail. Second version will only proceed to executing second query, if handling first query+results will succeed.

The 2nd one is the way to go. The first one may fail to close the connection when an exception occurs in first try-block. When you just do printing to stderr in the catch-block then not, but you won't do that. Or should the 2nd statement be executed without respect to success/failure of first one?

I realise that this post is old, but if you are lucky enough to be running in a Java7 environment- I would suggest using the try-with-resource.
It will handle the connection closing for you- as the new versions of the class implement the Closable interface.
public static void viewTable(Connection con) throws SQLException {
String query = "select COF_NAME, SUP_ID, PRICE, SALES, TOTAL from COFFEES";
try (Statement stmt = con.createStatement()) {
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
String coffeeName = rs.getString("COF_NAME");
int supplierID = rs.getInt("SUP_ID");
float price = rs.getFloat("PRICE");
int sales = rs.getInt("SALES");
int total = rs.getInt("TOTAL");
System.out.println(coffeeName + ", " + supplierID + ", " +
price + ", " + sales + ", " + total);
}
} catch (SQLException e) {
JDBCTutorialUtilities.printSQLException(e);
}
}