I have a client / server application (local) where several clients perform operations concurrently on the server. The server creates a separate thread for each client in which the desired operations (register/login) are then processed. Each thread has its own connection to the database and communicates with the client via its own socket that i get from ServerSocket.accept(). The operations from the client require read and write access to a database (sqlite) on the server. The register op checks whether the entry already exists, if not it gets added. And the login op is just checking passwords. The thread will then tell the client whether the operation was successful or not.
When a client is finished, it sends a message to the server to close the connection. The thread on the server interrupts its loop and then closes its socket and its connection to the database.
When there is just one client, there are no problems with the program. If There are multiple clients concurrently, then i get 2 types of SQLExceptions, "database is locked" and "unique constraint failed". The former might happen, because there are concurrent transactions being performed on the database. And this is prevented, because of ACID. The latter might happen, because of concurrency (before inserting a new entry into the DB I check whether the primary key is already used, the primary key is a username). Correct me if im wrong.
Right now I am not using thread synchronization or disabling auto commit when doing the operations on the database.
I thought that the transactions on ACID DBs are automatically queued up if new ones are invoked while the current one has not finished yet. But this does not seem to be the case. So am I supposed to actively prevent these exceptions by using thread synchronization (e.g. by using Locks) when executing queries/updates or perhaps just retry until the operations succeed? What are common approaches to handle the exceptions? I know that its problematic that the thread wait for a message to end but this is another topic. This is just a program to learn about working with databases.
Here are some snippets of code that runs on the server's threads. When a thread using this runnable is created I create a new Socket from ServerSocket.accept() and a new Connection object from the driver and pass both to it.
public class ConnectionHandler implements Runnable {
private final Socket socket;
private final Connection dbConnection;
public ConnectionHandler(Socket socket, Connection databaseConnection) {
this.socket = socket;
this.dbConnection = databaseConnection;
}
#Override
public void run() {
try (socket; dbConnection;
ObjectInputStream ois = new ObjectInputStream(socket.getInputStream());
ObjectOutputStream oos = new ObjectOutputStream(socket.getOutputStream()); )
{
boolean shouldClose = false;
while (!shouldClose)
{
Actions action = (Actions) ois.readObject();
boolean success = false;
if (action == Actions.REGISTER)
{
success = performClientRegistration(ois, dbConnection);
}
else if (action == Actions.LOGIN)
{
success = performClientLogin(ois, dbConnection);
}
else if(action == Actions.SHUTDOWN)
{
shouldClose = true;
}
if (success) {
oos.writeObject(Response.SUCCESS);
} else {
oos.writeObject(Response.FAILURE);
}
}
}
catch (ClassNotFoundException | IOException | SQLException e) {
e.printStackTrace();
}
System.out.println("Socket to " + socket + " closed.");
}
private boolean performClientRegistration(ObjectInputStream ois, Connection dbConnection) throws IOException, ClassNotFoundException {
String userName = (String)ois.readObject();
String pwd = (String)ois.readObject();
String firstName = (String)ois.readObject();
String lastName = (String)ois.readObject();
try
{
boolean isClientRegistrationPossible = !DBHelper.existsEntry(dbConnection, userName);
if(isClientRegistrationPossible) {
DBHelper.addEntry(dbConnection, userName, pwd, firstName, lastName);
return true;
}
}
catch (SQLException e) {
e.printStackTrace();
}
return false;
}
} // END OF CLASS
public static void addEntry(Connection connection, String userName, String pwd, String firstName, String lastName) throws SQLException {
PreparedStatement statement = connection.prepareStatement("INSERT INTO users " +
"(userName, pwd, lastName, firstName) VALUES (?, ?, ?, ?)");
statement.setString(1, userName);
statement.setString(2, pwd);
statement.setString(3, lastName);
statement.setString(4, firstName);
statement.executeUpdate();
}
public static boolean existsEntry(Connection connection, String userName) throws SQLException {
PreparedStatement statement = connection.prepareStatement("SELECT 1 FROM users WHERE userName = ?");
statement.setString(1, userName);
ResultSet result = statement.executeQuery();
if(result.next()) {
return true;
}
return false;
}
No you should not thread synchronization for this. There are many reasons this will newer work.
Do you explicit start and commit each transactions with begin() and commit()?
But it sounds to me like you have transactions running for to long. You should newer have a transaction open, while communication with the client over the network. So what you should do is:
Get all the data you need from the client. Call Begin() on the database.
Do all needed sql on the database.
Then commit()
and then send the reply back to the client so the transaction is open as little time as possible.
Updated:
This also depend on the transaction isolation level. There is description of these is sqllite here:
Btw: You can set sqllite to work in exclusive mode, which might be a good test for you. If you do that and transactions begin to hang, then your problem is that you have open transactions which you don't close.
see:
https://dba.stackexchange.com/questions/258003/what-standard-sql-isolation-levels-do-the-sqlite-isolation-levels-correspond-to
Related
I'm currently working on a college project, and I'm creating a very simple e-commerce style website.
I'm using JDBC driver manager and connection pool for the connection to the db, while using Tomcat 9.0 as the container.
The problem is: when I modify some product through the website (let's say the amount available for example), the website doesn't always reflect the changes, while I can always see the data correctly in MySql Workbench.
It actually works one time out of two on the same query:
I run the query for the first time after the changes -> it shows the old value
I run the query for the second time after the changes -> it shows the new value
I run the query for the third time after the changes -> it shows the old value
And so on.
I've already tried to set caching off (from the query, using the SQL_NO_CACHE), but it didn't seem to solve the problem, I've tried to use Datasource instead, but it causes other problems that most likely I won't have the time to solve.
This is the connection pool file, which I think might be problem, I'm not that sure tho:
public class DriverManagerConnectionPool {
private static List<Connection> freeDbConnections;
static {
freeDbConnections = new LinkedList<Connection>();
try {
Class.forName("com.mysql.cj.jdbc.Driver");
} catch (ClassNotFoundException e) {
System.out.println("DB driver not found:"+ e.getMessage());
}
}
private static synchronized Connection createDBConnection() throws SQLException {
Connection newConnection = null;
String ip = "localhost";
String port = "3306";
String db = "storage";
String username = "root";
String password = "1234";
newConnection = DriverManager.getConnection("jdbc:mysql://"+ ip+":"+ port+"/"+db+"?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC", username, password);
newConnection.setAutoCommit(false);
return newConnection;
}
public static synchronized Connection getConnection() throws SQLException {
Connection connection;
if (!freeDbConnections.isEmpty()) {
connection = (Connection) freeDbConnections.get(0);
freeDbConnections.remove(0);
try {
if (connection.isClosed())
connection = getConnection();
} catch (SQLException e) {
connection.close();
connection = getConnection();
}
} else {
connection = createDBConnection();
}
return connection;
}
public static synchronized void releaseConnection(Connection connection) throws SQLException {
if(connection != null) freeDbConnections.add(connection);
}
}
I really hope you can help me, I haven't found any solution online!
I guess it is because of auto-commit is disabled. Please try using #Transactional or set auto-commit to true. You can also try to use db.commit after each statement.
As per your connection pool implementation, all connection in your pool seems to be auto committed false.
Please check you have properly committed the connection after executing the query or not.
So it might be the case that, when executing the query after changes with same connection it reflects those changes, done earlier and on other connections, old values are might get returned.
I am learning about MySQL database and I cannot quite understand one concept. Lets say there are two methods in the same class as the one shown below. Now, do i have to use Connection connect = dbConnection.getDBConnection(); in each method or is there a different way to declare one connection and use it across multiple methods?:
private void setUpdateButton(ActionEvent event) {
try{
Connection connect = dbConnection.getDBConnection();
Statement stmt = connect.createStatement();
if(txtID.getText().trim().isEmpty()||txtFirstName.getText().trim().isEmpty() || txtSecondName.getText().trim().isEmpty() ||
txtGender.getText().trim().isEmpty() || txtAge.getText().trim().isEmpty() || txtHomeAddress.getText().trim().isEmpty() || txtPhoneNumber.getText().trim().isEmpty()) {
showAlert("Invalid Input!", "All fields need to be populated before updating.");
}else {
String sqlQuery ="update student_information set Age ='"+Integer.parseInt(txtAge.getText())+"',Name ='"+txtFirstName.getText()+"',Surename='"+txtSecondName.getText()
+"',Gender='"+txtGender.getText()+"',Address='"+txtHomeAddress.getText()+"',PhoneNumber='"+txtPhoneNumber.getText()+"'where ID="+Integer.parseInt(txtID.getText());
stmt.executeLargeUpdate(sqlQuery);
setTxtArea();
showConfAlert("Update Completed!", "Record has been updated!");
Creating connections is a costly operation, so I think you should open the connection at the application startup, and close it on exit.
If your program is not multi thread you would be fine with a simple global object, otherwise other strategies should be used.
You can create a singleton for your app with a method returning the connection.
public class App {
private static App self;
private Connection connection;
private App(){
}
public synchronized App getInstance(){
if(self == null){
self = new App();
}
return self;
}
public synchronized Connection getConnection()throws SQLException {
if(connection==null || !isValid(connection)){
// Create a new connection
}
return connection;
}
private boolean isValid(Connection conn) {
// Check if the connection is valid otherwise return false
return false;
}
public static synchronized void close(){
try{
self.connection.close();
} catch (SQLException e){
// Swallow exception
} finally {
self = null;
}
}
}
You can get the connection anywhere like this:
Connection conn = App.getInstance().getConnection();
You should make sure to close the connection on exit, maybe with a shutdown hook.
You could also, create a wrapper around your connection that forwards all connection methods to the original connection , with the exception of close that marks the connection available , this way you create something like 1 connection pool.
If the connection is available you return it otherwise you either wait or throw or do what is most appropriate for your application.
We are developing a website using
Tomcat 7
JDBC
PostgreSQL 9.2
We've had some connection leaks and think we corrected them (the database no longer stops responding), but the behaviour of the connection pool still seems leaky, as we have a number of idle connections greater than the maxIdle set in context.xml. I'd like to be sure the problem is fixed.
For testing purposes, I'm using the following context.xml :
<Resource
auth="Container"
name="jdbc/postgres"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
type="javax.sql.DataSource"
username="admin"
password="..."
driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://127.0.0.1:5432/..."
initialSize="1"
maxActive="50"
minIdle="0"
maxIdle="3"
maxWait="-1"
minEvictableIdleTimeMillis="1000"
timeBetweenEvictionRunsMillis="1000"
/>
If I understand correctly, we should have 1 idle connection on startup and from 0 to 3 depending on the load, right ?
What is happening is : 1 connection on startup, up to 3 idle connections if the load is low, and more than 3 idle connections after a high load. Then these connections are not closed immediatly, and we don't know when/if they will be closed (sometime some of them are closed).
So the question is : is this behaviour normal, or not ?
Thanks for your help
EDIT : added factory attribute, didn't change the problem
EDIT 2 : using removeAbandoned & removeAbandonedTimeout make the idle connexions being closed every removeAbandonedTimeout. So we probably still have some connection leaks. Here are some pieces of code we are using to connect to the database and execute requests :
PostgreSQLConnectionProvider, just a static class to provide a connection :
public class PostgreSQLConnectionProvider {
public static Connection getConnection() throws NamingException, SQLException {
String dsString = "java:/comp/env/jdbc/postgres";
Context context = new InitialContext();
DataSource ds = (DataSource) context.lookup(dsString);
Connection connection = ds.getConnection();
return connection;
}
}
DAO abstract class :
public abstract class DAO implements java.lang.AutoCloseable {
// Private attributes :
private Connection _connection;
// Constructors :
public DAO() {
try { _connection = PostgreSQLConnectionProvider.getConnection(); }
catch (NamingException | SQLException ex) {
Logger.getLogger(DAO.class.getName()).log(Level.SEVERE, null, ex);
}
}
// Getters :
public Connection getConnection() { return _connection; }
// Closeable :
#Override
public void close() throws SQLException {
if(!_connection.getAutoCommit()) {
_connection.rollback();
_connection.setAutoCommit(true);
}
_connection.close();
}
}
UserDAO, a small DAO subclass (we have several DAO sublasses to request the database) :
public class UserDAO extends DAO {
public User getUserWithId(int id) throws SQLException {
PreparedStatement ps = null;
ResultSet rs = null;
User user = null;
try {
String sql = "select * from \"USER\" where id_user = ?;";
ps = getConnection().prepareStatement(sql);
ps.setInt(1, id);
rs = ps.executeQuery();
rs.next();
String login = rs.getString("login");
String password = rs.getString("password");
String firstName = rs.getString("first_name");
String lastName = rs.getString("last_name");
String email = rs.getString("email");
user = new User(id, login, password, firstName, lastName, email);
}
finally {
if(rs != null) rs.close();
if(ps != null) ps.close();
}
return user;
}
}
An example of a DAO subclass use :
try(UserDAO dao = new UserDAO()) {
try {
User user = dao.getUserWithId(52);
}
catch (SQLException ex) {
// Handle exeption during getUserWithId
}
}
catch (SQLException ex) {
// Handle exeption during dao.close()
}
Looking at the code it appears the connection is grabbed for the lifetime of the DAO, not the lifetime of the statement, which is the usual expectation. Normally, you would grab a connection from the pool just as you're about to execute the statement, and call close() on it when you're done in order to return it to the pool.
Additionally, in your finally clause, both rs.close() and ps.close() can throw exceptions resulting in missing the last call against the prepared statement.
In Java 7 you can also use a try with resources statement that will close both the prepared statement and the connection for you. According to the spec, the driver is supposed to close the result for you when the statement is closed.
I deployed my first Java web application a couple of days ago and realized a strange thing was happening. After a period of time all the dynamic content and functionality that relied on a connection to my database (testimonial submission, admin login) stopped working. It seems like this is happening every 24 hours or so. Every morning I realize it isn't working again.
I solve the issue by going in to the Tomcat web application manager and clicking "reload" on the web app in question. Immediately the dynamic features of the website work again.
My server is running Tomcat 7 and MySQL and the web app uses the JDBC driver to establish the connection to the database. I've made no alterations to Apache or Tomcat settings.
I have other web apps written in PHP that work persistently without fault it just seems to be this Java web app that has this problem.
What would cause this to happen and how can I make it so the web app doesn't need to be reloaded before it can establish a database connection again?
EDIT: attached some code for database connection
Database connection
public class DBConnection {
private static Connection conn;
private static final Configuration conf = new Configuration();
private static final String dbDriver = conf.getDbDriver();
private static final String dbHostName = conf.getDbHostname();
private static final String dbDatabaseName = conf.getDbDatabaseName();
private static final String dbUsername = conf.getDbUsername();
private static final String dbPassword = conf.getDbPassword();
public Connection getConnection(){
try{
Class.forName(dbDriver);
Connection conn = (Connection) DriverManager.getConnection(dbHostName + dbDatabaseName, dbUsername, dbPassword);
return conn;
} catch(Exception e){
e.printStackTrace();
}
return conn;
}
public void disconnect(){
try{
conn.close();
} catch (Exception e){}
}
}
Controller for login form:
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String form = request.getParameter("form");
// check login details
if(form.equals("loginForm")){
String username = request.getParameter("username").trim();
String password = request.getParameter("password").trim();
password = loginService.hashPassword(password);
boolean isValidUser = loginService.checkUser(username, password);
if(isValidUser){
Cookie loggedIn = new Cookie("loggedIn", "true");
loggedIn.setMaxAge(60*60*24);
response.addCookie(loggedIn);
out.print("success");
}else{
out.print("nope");
}
}
}
Login service checks login details are correct:
public boolean checkUser(String username, String password){
boolean isValid = false;
try{
sql = "SELECT username, password FROM morleys_user WHERE username=? AND password=? AND isActive=1 LIMIT 1";
prep = conn.prepareStatement(sql);
prep.setString(1, username);
prep.setString(2, password);
rs = prep.executeQuery();
if(rs.next()){
return true;
}
}catch(Exception e){
e.printStackTrace();
}finally{
connection.disconnect();
}
return isValid;
}
UPDATE
If I understand correctly I should not be handling a direct connection to a database and instead be using a service that will manage connections for me.
This is my example of establishing a DataSource connection to a MysQL database.
Establish a new DataSource instance of this class:
package uk.co.morleys;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
import javax.sql.DataSource;
import com.mysql.jdbc.jdbc2.optional.MysqlDataSource;
public class DataSourceFactory {
public static DataSource getMySQLDataSource() {
Properties props = new Properties();
FileInputStream fis = null;
MysqlDataSource mysqlDS = null;
try {
fis = new FileInputStream("db.properties");
props.load(fis);
mysqlDS = new MysqlDataSource();
mysqlDS.setURL(props.getProperty("MYSQL_DB_URL"));
mysqlDS.setUser(props.getProperty("MYSQL_DB_USERNAME"));
mysqlDS.setPassword(props.getProperty("MYSQL_DB_PASSWORD"));
} catch (IOException e) {
e.printStackTrace();
}
return mysqlDS;
}
}
Instantiating a new DataSource for checking user login details
public boolean checkUser(String username, String password){
boolean isValid = false;
DataSource ds = DataSourceFactory.getMySQLDataSource();
Connection con = null;
ResultSet rs = null;
PreparedStatement ps = null;
try{
con = ds.getConnection();
sql = "SELECT username, password FROM morleys_user WHERE username=? AND password=? AND isActive=1 LIMIT ";
ps = con.prepareStatement(sql);
ps.setString(1, username);
ps.setString(2, password);
rs = ps.executeQuery();
if(rs.next()){
return true;
}
}catch(SQLException e){
e.printStackTrace();
}finally{
try {
if(rs != null) rs.close();
if(ps != null) ps.close();
if(con != null) con.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
return isValid;
}
Given that you've never heard of a connection pool before I'm assuming that you not are not very effectively managing database resources.
The most basic way to access the database is to obtain a connection, execute some statements & close the connection.
In the code you provided I don't see you obtaining or closing a connection, so I assume that you create a single connection when you start your application and keep the connection open "forever". After a certain amount of time your MySql server decides to kill the connection as it's been open for too long.
When you create and close a connection each time you need one, you normally won't encounter any connection timeouts, but you might experience a lot overhead from creating a connection each time your application needs one.
This is where a connection pool comes in; a connection pool manages a number of database connections and your application borrows one each time it needs one. By properly configuring your connection pool the pool will normally transparently take care of broken connections (you might for example configure the pool to renew a connection once it's x minutes/hours old).
You also need to pay attention to resource management; e.g. close a statement as soon as you no longer need it.
The following code demonstrates how your "check user" method can be improved:
public boolean checkUser(String username, String password) throws SQLException {
//acquire a java.sql.DataSource; the DataSource is typically a connection pool that's set-up in the application of obtained via jndi
DataSource dataSource = acquireDataSource();
//java 7 try-with-resources statement is used to make sure that resources are properly closed
//obtain a connection from the pool. Upon closing the connection we return it to the pool
try (Connection connection = dataSource.getConnection()) {
//release resources associated with the PreparedStatement as soon as we no longer need it.
try(PreparedStatement ps = connection.prepareStatement("SELECT username, password FROM morleys_user WHERE username=? AND password=? AND isActive=1 LIMIT 1");){
ps.setString(1, username);
ps.setString(2, password);
ResultSet resultSet = ps.executeQuery();
return resultSet.next();
}
}
}
Common connections pools are Apache Commons-DBCP and C3P0.
As managing sql resources can be quite repetitive and cumbersome you might want to consider using a template: for example Spring's JdbcTemplate
Example C3p0 configuration:
public ComboPooledDataSource dataSource(String driver, String url, String username,String password) throws PropertyVetoException {
ComboPooledDataSource dataSource = new ComboPooledDataSource();
dataSource.setDriverClass(driver);
dataSource.setJdbcUrl(url);
dataSource.setUser(username);
dataSource.setPassword(password);
dataSource.setAcquireIncrement(1);
dataSource.setMaxPoolSize(100);
dataSource.setMinPoolSize(1);
dataSource.setInitialPoolSize(1);
dataSource.setMaxIdleTime(300);
dataSource.setMaxConnectionAge(36000);
dataSource.setAcquireRetryAttempts(5);
dataSource.setAcquireRetryDelay(2000);
dataSource.setBreakAfterAcquireFailure(false);
dataSource.setCheckoutTimeout(30000);
dataSource.setPreferredTestQuery("SELECT 1");
dataSource.setIdleConnectionTestPeriod(60);
return dataSource;
}//in order to do a "clean" shutdown you should call datasource.close() when shutting down your web app.
MySQL times out the connection after some period of time. The standard way to deal with this is to use a properly configured connection pool (with a configured DataSource) instead of using DriverManager directly.
The connection pool will check for and discard "stale" connections.
Recently I was looking at the frequent hangs of the tomcat server and came across some exceptions thrown frequently in some part of the code.
When I examined the code, this what it looked like
public static String doSomething() {
String returnVakue = "default value";
try {
ResultSet rs = getMyResultSet();
rs.first();
returnValue = rs.getString("my_field"); // Note that Exception happens at times when the ResultSet is empty
} catch (Exception e) {
throw new Exception(e);
} finally {
return returnValue;
}
}
While I am aware that it is perfectly OK to have both throws exception and return, wondering if this can cause any kind of leaks in tomcat. And Is there any potential risk with performance. ? Hoowever my caller function stops the execution at this point. Any views on this? Does it affect the GC?
EDIT : NOTE : I Know how to correct this code. Please share your views whether this can potentially cause tomcat hanging.
First check if returned ResultSet is empty.
while( rs.next() ) {
// ResultSet processing here
// Result set is not empty
}
In my opinion throwing exception is your decision, but in finally you should be doing clean up e.g. closing Connections.
Open Connections if not closed will cause tomcat to hang because new requests coming to server will be waiting for connections to become available.
Any object in Java which is referenced is not garabage collected, in your case if Connections are not closing then these objects will not be garbage collected.
Cheers !!
If a query takes a long time, not a question of JDBC. The database is responsible. Of course, if JDBC is used properly. In another hand, if you use simple JDBC, it is best that you add a layer DAO in your application.
public class UserDAO extends DAO {
private static final String FIND_BY_PK_SQL =
"SELECT mail, password " +
" FROM user WHERE mail = ?";
public User findByPk(final String mail) throws DAOException {
Connection conn = null;
PreparedStatement ps = null;
ResultSet rs = null;
try {
conn = getConnection();
ps = conn.prepareStatement(FIND_BY_PK_SQL);
ps.setString(1, mail);
rs = ps.executeQuery();
if (rs.next()) {
return fill(rs);
}
return null;
} catch (final SQLException e) {
throw new DAOException(e);
} finally {
DbUtils.closeQuietly(conn, ps, rs);
}
}
private User fill(final ResultSet rs) throws SQLException {
final User user = new User();
user.setMail(rs.getString("mail"));
user.setPassword(rs.getString("password"));
return user;
}
}