I have a problem with a really slow connection between my Java code and a MySQL Database. I don't know where the bottle neck is.
My program is more or less a chatbot. The user types something in, my program splits the sentence into words and sends it word per word to the database. If it finds something there, the user gets an output.
The database is on an external Server, but I also tried to connect to a pc next to me. Both is slow.
I tried the connection once at another place then where I normally work and there it was fast, most of the time.
My SQL Code:
SELECT info.INFORMATION FROM INFORMATION info, INFO_SCHLUESSEL sch
WHERE LCASE(sch.SCHLUESSELWORT) LIKE '" + input + "%' AND info.ID_INFO = sch.ID_INFO
Order BY info.PRIORITAET DESC LIMIT 1;
(just remembered, if it helps to understand the sql code:
schluessel = key
Schluesselwort = key word
prioritaet = priority)
My Java Database Code is more or less standard stuff:
String driver = "com.mysql.jdbc.Driver";
String dbase = "jdbc:mysql://bla";
String dbuser = "bla";
String dbpw = "bla";
Class.forName(driver);
Connection con = DriverManager.getConnection(dbase, dbuser, dbpw);
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next())
{
ergebnis = rs.getString("info.INFORMATION");
}
rs.close();
stmt.close();
con.close();
edit:
I have tried this DBCP for a while now, and I can't seem to get it to work. It seems to be as slow as the old connection. This is the example provided by the website that I use:
GenericObjectPool connectionPool = new GenericObjectPool(null);
ConnectionFactory connectionFactory = new DriverManagerConnectionFactory("jdbc:mysql://bla", "bla", "bla");
PoolableConnectionFactory poolableConnectionFactory = new PoolableConnectionFactory(connectionFactory,connectionPool,null,null,false,true);
PoolingDriver driver = new PoolingDriver();
driver.registerPool("example",connectionPool);
Connection conn = DriverManager.getConnection("jdbc:apache:commons:dbcp:example");
I suspect that it's the connection setup that is causing the problem. It would be worth timing how long this takes:
Connection con = DriverManager.getConnection(dbase, dbuser, dbpw);
and if so, check out Apache Commons DBCP, which allows you to pool database connections.
Well I think this warrants a discussion on the design.There are a few things which you can do in order to improve the performance. Since you are not persisting anything here, its better to preload all the data in memory in some custom java object, a map, list or whatever and then do an in-memory lookup for the word and get the results. Another approach could be to use a batch statement so that you dont go ahead and create and release connections for each word. Oh and if using batch statements make sure you set the batch size to an appropriate number, preferably a prime number
Related
i created a kafka producer on java that works fine, now im trying to store the data produced in an mySQL database but i don't know how. i tried this code but it doesn't work
try {
String MyUrl = "jdbc:mysql://130.2.2.2/pfa";
Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection(MyUrl, "root", "");
Statement stm = con.createStatement();
ResultSet rs = stm.executeQuery("select * from tmp");
while(rs.next()) {
producer.send(new ProducerRecord<String, String>("test",rs.getString(1)));
con.close();
}
}catch(Exception e) {System.out.println(e);}
any help is much appreciated about sending data to the database
I'm not sure I understand what you're expecting to happen from that code.
You're reading from the database, never inserting into it
You're closing the database connection after every loop iteration, meaning you'll get an exception that the database is already closed on the second loop, and therefore the resultset might terminate as well (I'm a little fuzzy on if those are lazy objects or not)
Might I suggest that you follow the Deep Dive on the JDBC connector? https://dev.to/rmoff/kafka-connect-jdbc-sink-tips-tricks-video-walkthrough-2egf
i copy pasted it literally becoz the writer said it worked
This will never be a good way to learn anything. You can read the code and type it all back out, explaining it to yourself as you go... But never blindly copy something that you assume solves your problem
I'm having a weird problem with an Grails application accessing data. Going deeper I've isolated the problem to a plain java8 small application using PreparedStatement.executeQuery vs Statement.executeQuery.
Consider the following snippet of code:
// executes in milliseconds
directSql = "select top(10) * from vdocuments where codcli = 'CCCC' and serial = 'SSSS' ORDER BY otherField DESC;";
stmt = con.createStatement();
rs = stmt.executeQuery(directSql);
// More than 10 minutes
sqlPrepared = "select top(10) * from vdocuments where codCli = ? and serial = ? ORDER BY otherField DESC;";
PreparedStatement pStatement = con.prepareStatement( sqlPrepared );
pStatement.setString(1, "CCCC");
pStatement.setString(2, "SSSS");
rsPrepared = pStatement.executeQuery();
Same query.
Data comes from a view on SqlServer (2008, I think, have no access right now) from a table with more than 15 Million records. There are indexes for all needed fields and the same query (the first one) executed from console runs also quite fast.
If I execute the slow PreparedStatement query without the ORDER clause it also runs fast.
It looks clear to me that for any cause the database it's not using indexes and make a full scan when using preparedStatement, but maybe I'm wrong so I'm open to any idea.
I thought maybe the driver (sqlserver official latest and jtds has been tested) was holding the data waiting for any kind of EOF from connection but I've checked with tcpdump on my side and no data is received.
I can't find why this is happening so any idea will be welcomed.
Thank you in advanced!
I've finally found a solution, at least in for my case. I got it here http://mehmoodbluffs.blogspot.com.es/2015/03/hibernate-queries-are-slow-sql-servers.html . Telling (driver? sqlServer?) not to send parameters as Unicode have resolved the problem.
Current connection string it's now:
String connectionUrl = "jdbc:sqlserver://server:port;databaseName=myDataBase;sendStringParametersAsUnicode=false";
And now both direct queries and preparedStatements runs at millisecond speed.
Thank you #DanGuzman for your suggestions!
I have 26 CSV files that I want to grab from the internet on a nightly basis and upload them into a Postgresql table. I have this working using Java, PreparedStatement, and Batch. Despite this, performance is painfully slow. To grab the 6000 or so entries and put them into Postgresql, it's taking 30 minutes. This is my first time doing something like this, so I don't exactly have a reference point as to whether this is fast or slow.
To get the file, I am using this code.
URL grabberUrl = new URL(csvUrl);
URLConnection grabberConn = grabberUrl.openConnection();
BufferedReader grabberReader = new BufferedReader(new InputStreamReader(grabberConn.getInputStream()));
I am then using PreparedStatement to and taking values from the input stream and setting them
con = DriverManager.getConnection(url, user, password);
pst = con.prepareStatement("insert into blah(name, year) values(?, ?)");
pst.setString(1, name);
pst.setString(2, year);
I am then batching up the inserts. I've tried values from 100 to 1000 with no meaningful change to performance.
pst.addBatch();
if (count == 100) {
count = 0;
pst.executeBatch();
}
Has anyone got any suggestions as to what I can do to make things faster?
If you can access the files from the PostgreSQL server try using the copy statement. See link
http://www.postgresql.org/docs/9.3/static/sql-copy.html
Also, if you know the data quality you can temporarily remove any table constraints and drop any index's. You can add the constraints and the index's after loading the data.
Try the following:
PGConnection con = (PGConnection) DriverManager.getConnection(...);
CopyManager copyManager = con.getCopyAPI();
copyManager.copyIn("copy mytable from stdin with (format csv)", grabberReader);
If mytable is heavily indexed, then drop the indexes, load, and recreate the indexes.
We are trying to fetch data from Oracle DB using a PreparedStatement. It keeps fetching zero records while the same runs and fetches data when run from PL/SQL developer.
We found the root cause while trying to debug. While debugging the code fetched the two records properly.
We did a temporary fix by placing this piece of code.
ResultSet rs = ps.executeQuery();
while(!rs.hasNext()){
ps.executeQuery();}
This works. But this is not the best solution since it results in an unwanted DB hit.It clearly looks like a time issue. We also explicitly committed earlier transactions since they can affect the result of this query.
What could be causing this. What's the best way to solve this?
The method is quite big: I'll just post some parts here:
private static boolean loadCommission(Member member){
Connection conn = getConnection("schema1"); //obtained through connection pool
//insertion into table
conn.close();
Conn conn2 = getConnection("schema2"); //obtained through connection pool
PreparedStatement ps = conn2.prepareStatement(sql);
//this sql combines data from schema1
// and 2 with DB links
ResultSet rs = ps.executeQuery();
//business logic
conn2.close();
return true;
}
Thanks
We tried a few more things yesterday. We replaced the second connection code with direct jdbc connection like so
Connection conn = DriverManager.getConnection(URL, USER, PASS);
This too works. Now we are not sure if the delay is in getting connection from pool or in completing previous transaction like we thought earlier.
If your query selects from a materialized view, then there may be some elapsed time before it will yield results (as materialized views do not necessarily refresh instantly after a commit, depending upon how they've been created).
If this is the case, then you can resolve the problem by either selecting directly from the base table (or equivalent non-materialized views), or forcing the materialized view to refresh.
I'm trying to determine the best way to ping a database via JDBC. By 'best' I mean fast and low overhead. For example, I've considered executing this:
"SELECT 1 FROM DUAL"
but I believe the DUAL table is Oracle-specific, and I need something more generic.
Note that Connection has an isClosed() method, but the javadoc states that this cannot be used to test the validity of the connection.
With JDBC 4 you can use isValid(int) (JavaDoc) from the Connection Interface. This basically does the trial statement for you.
Some driver implement this by sending the correct dummy SQL to the database and some directly uses low level operations which reduces the parsing overhead.
However beware of the timeout, some drivers (DB/400 and Oracle Thin) do spawn a new time thread for each invocation, which is not really acceptable for most Pool validation scenarios). And Oracle also does not seem to use a prepared statement, so it’s kind of relying on the implicit cache.
Yes, that would be Oracle-only, but there is no generic way to do this in JDBC.
Most connection pool implementations have a configuration parameter where you can specify the SQL that will be used for ping, thus pushing the responsiblity to figure out how to do it to the user.
That seems like the best approach unless someone comes up with a little helper tool for this (of course, it precludes using potentially even faster non-SQL-based methods like Oracle's internal ping function)
MySQL has a nice mechanism, documented in this SO answer. From the answer:
"/* ping */ SELECT 1"
This will actually cause the driver send a ping to the server and return a fake, light-weight, result set.
Having said that, #eckes answer is the best (using JDBC 4's Connection.isValid(int)).
I'm not aware of a generic solution, either. For IBM's UDB on iSeries (and perhaps other DB2 systems) it would be
select 1 from SYSIBM.SYSDUMMY1;
You could try to get the db name from the connection meta data and execute a matching sql staement. E.g.
Connection con = null;
Statement st = null;
ResultSet rs = null;
try {
con = dataSource.getConnection();
String dbProductName = con.getMetaData().getDatabaseProductName();
Statement st = con.createStatement();
if ( "PostgreSQL".equalsIgnoreCase(dbProductName) ) {
rs = st.executeQuery("select version();");
} else if ( "Oracle".equalsIgnoreCase(dbProductName) ) {
rs = st.executeQuery("select 1 from dual");
} else {
...
}
} catch ( Exception ex ) {
System.out.prinln("DB not reachable");
} finally {
// close statement, connection etc.
...
}
I may be out to lunch on this one, but could you simply execute some non-sense query, such as:
SELECT * FROM donkey_giraffe_87
I don't know very much about JDBC's error handling, but perhaps you could check to see if the database is at least telling you that the table does not exist. If JDBC's error codes are vendor-specific, the Spring Framework has some utilities for mapping these codes to more meaningful exceptions.