I have problem with java and MySQL. My code:
Connection connection;
// ...
for (String query : updateAndInsertQuery) {
Statement stm = connection.createStatement();
stm.execute(query);
stm.close();
}
Statement stm2 = connection.createStatement();
System.out.println("Before query");
System.out.flush();
ResultSet Result = stm2.executeQuery(selectQuery);
System.out.println("After query");
System.out.flush();
int vfrom, vto;
while (Result.next()) {
// ...
}
When I run program i see in MySQL queries and run
show processlist;
selectQuery is visible on list wth status Sending data or Writing to net. On console print: Before query. Next
show processlist;
returns empty list, but application don't print After query. Do you have similar problem?
-- edit
I resolve my problem.
I think:
wen MySQL returns data and query isn't visible on processlist in MySQL
I should immediately get on console message: After query
but console was empty, java process works (cpu usage was 90-100%) so I think it was my mistake, but after 1h application throws Exception
Increase memory limit resolve my problem.
So I have next question why application throw exception after hour? Garbage collection try dealocate unised objects?
executing queries manually usually leads into many different problems - all of which are platform-specific and DB-specific. I think your best answer will be : "switch to ORM".
This framework has proven to be exceptionally good, wrappig all your SQL-data into Entities and transactions (if required) will resolve most of your problems at the same time - you will only need to annotate your entities and relationships correctly. Database-queries can be executed via JPA-"criteria"s which are platform-independent AND allow you to avoid a lot of problems as well as making your code READABLE.
Tutorial : http://www.vogella.com/tutorials/JavaPersistenceAPI/article.html
SO-question : https://stackoverflow.com/questions/743065/what-is-the-fastest-way-to-learn-jpa
With JPA, you wont need to care about statements or queries anymore (well, at least most of the time) and your mentioned problem will disappear - PLUS : it only takes 30-60min to implement.
Additional tip : use Maven & Eclipselink (JPA2 implementation) - thats a very powerful, portable combination
Related
PreparedStatement.executeQuery() is taking ~20x longer to execute than if it were run directly via the shell. I've logged with timers to determine that this method is the culprit.
The query and some DB info (ignoring the Java issue for the moment):
mysql> SELECT username from users where user_id = 1; // lightning fast
Running that same query 1,000 times via mysqlslap is also lightning fast.
mysqlslap --create-schema=mydb --user=root -p --query="select username from phpbb_users where user_id = 1" --number-of-queries=1000 --concurrency=1
Benchmark
Average number of seconds to run all queries: 0.051 seconds
Minimum number of seconds to run all queries: 0.051 seconds
Maximum number of seconds to run all queries: 0.051 seconds
Number of clients running queries: 1
Average number of queries per client: 1000
The Problem: Performing the same query in JDBC slows things significantly. In a for loop calling the below queryUsername() 1,000 times (this is called in the Main method, which isn't shown here) takes around 872ms. That's ~17x slower! I've tracked down the heavy usage by placing timers in various spots (omitted some for brevity). The primary suspect is stmt.executeQuery() which took 776ms of the 872ms runtime.
public static String queryUsername() {
String username = "";
// DBCore.getConnection() returns HikariDataSource.getConnection() implementation exactly as per https://www.baeldung.com/hikaricp
try (Connection connection = DBCore.getConnection();
PreparedStatement stmt = connection.prepareStatement("SELECT username from phpbb_users where user_id = ?");) {
stmt.setInt(1, 1); // just looking for user_id 1 for now
// Google timer used to measure how long executeQuery() is taking
// Another Timer is used outside of this method call to see how long
// total execution takes.
// Approximately 1 second in for loop calling this method 1000 times
Stopwatch s = Stopwatch.createStarted();
try (ResultSet rs = stmt.executeQuery();) {
s.stop(); // stopping the timer after executeQuery() has been called
timeElapsed += s.elapsed(TimeUnit.MICROSECONDS);
while (rs.next())
{
username = rs.getString("username"); // the query returns 1 record
}
}
} catch (SQLException e) {
e.printStackTrace();
}
return username;
}
Additional context and things tried:
SHOW OPEN TABLES has several tables open, but all have In_use=0 and Name_locked=0.
SHOW FULL PROCESSLIST looks healthy.
user_id is an indexed primary key
The Server is an Upcloud $5/month 1-Core, 1GB RAM running Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-66-generic x86_64). Mysql Ver 8.0.23-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))
JDBC Driver is mysql-connector-java_8.0.23.jar, which was obtained from mysql-connector-java_8.0.23-1ubuntu20.04_all via https://dev.mysql.com/downloads/connector/j/
Don't reconnect each time. Open the connection at the start; reuse it until the web page (or program) is finished.
Chances are that you are comparing different realities.
When running mysqlslap you are most likely using Unix Domain Sockets in the communication between the tool and MySQL server. Try changing that to TCP and you should observe an immediate performance drop. Connector/J, on the other hand, creates TCP based connections by default (Unix Domain Sockets can be used but only by using a third party library).
Also, in mysqlslap you are running a simple query directly, which are handled by a COM_QUERY protocol command. In the Java sample you are preparing the query first and then executing it. Depending on how Connector/J is configured this may result in a single COM_QUERY protocol command or a pair of commands, namely, COM_STMT_PREPARE and COM_STMT_EXECUTE. Connector/J is also affected by how its statement caches are configured (and/or the CP ones). However, you are only measuring the executeQuery part so, theoretically, Connector/J could be being favored here.
Finally, unless you actually come up with a use case where you guarantee that both executions are effectively doing the same work under the same circumstances, you can compare results and point out differences, but you can't take any conclusions from it. For example, it's not that hard to introduce caches and make those simple iterations even completely skip communicating to the server... that would make things extremely fast.
move borrowing connection and Stopwatch related code out of method. then measure as:
Stopwatch s = Stopwatch.createStarted();
try (Connection con = ....) {
for (int i=0; i < 1000; i++) {
queryUsername( con );
}
}
s.stop();
print s.elapsed(TimeUnit.MICROSECONDS);
I am required to execute a stored procedure in a SQL server to fetch some data, and since I will later save the data into a Mongo and this one is with ReactiveMongoTemplate and so on, I introduced Spring R2DBC.
implementation("org.springframework.data:spring-data-r2dbc:1.0.0.RELEASE")
implementation("io.r2dbc:r2dbc-mssql:0.8.1.RELEASE")
I see that I can do SELECT and INSERT and so on with R2DBC, but is it possible to EXEC prod_name? I tried it and it hangs forever and then the test terminates, without success but neither failure. The last line of log is:
io.r2dbc.mssql.QUERY - Executing query: EXEC "SCHEMA"."MY_PROCEDURE"
The code is like:
public Flux<Coupon> selectWithProcedure() {
return databaseClient
.execute("EXEC \"SCHEMA\".\"MY_PROCEDURE\" ")
.as(Coupon.class)
.fetch().all()
.doOnNext(coupon -> {
coupon.setCouponStatusRefFromId(coupon.getCouponStatusRefId());
});
}
And it seems that no data is retrieved.
If I test some other methods with simple queries like SELECT... it works. But the problem is, DBAs do not allow my app to read table data, instead, they create a procedure for me. If this query is not possible, I must go with traditional JPA way and going reactive at Mongo side has lost its sense.
Well. I just saw this:
https://github.com/r2dbc/r2dbc-mssql, version 0.8.1:
Next steps:
Execution of stored procedures
Add support for TVP and UDTs
And:
https://r2dbc.io/2019/05/13/r2dbc-0-8-milestone-8-released
We already have a few tickets lined up for the next milestone, and we know that they will require further SPI modifications:
Support for Auto-Commit
Connection Validation
Support for Stored Procedures
My mysql-query in Java always stops (i.e. freezes and does not continue) at a certain position, which namely is 543,858; even though the table contains approx. 2,000,000 entries. I've checked this by logging the current result-fetching.
It is reproducible and happens every time at the very same position.
"SELECT abc from bcd WHERE DATEDIFF(CURDATE(), timestamp) <= '"+days+"'");
Addition: It definitely is a Java error, I've just tried out this statement in Navicat (50s running time).
The query seems to freeze after the log tells me that it's now adding the result of position 543,858.
try {
...
ResultSet res = new ResultSet();
PreparedStatement stmt = new PreparedStatement(); // prepare statmenet etc.
stmt.setFetchSize(Integer.MIN_VALUE);
res = stmt.executeQuery();
...
System.out.println(res.getStatement());
...
while (res.next())
treeSet.add(res.getString("userid"));
} catch (Exception e) {
e.printStackTrace();
}
Edit: We were able to figure out the problem. This method is fine and the returned result (500,000 instead of 2,000,000) is right as well (looked up in the wrong db to verify the amount); the problem was, that the next method-call that used the result of the one posted above takes literally forever, but had no logging-implemented. So I've been fooled by missing console-logs.
Thanks anyways!
I think you might be running out of memory after processing half a million records. Try assigning more memory using command line options -Xmx etc. See here for more info about command line options.
In mysql to use streaming ResultSets you have to specify more parameters, not only fetchSize.
Try:
stmt = conn.createStatement('select ...', java.sql.ResultSet.TYPE_FORWARD_ONLY, java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
and see if that works.
It's documented in the ResultSet section.
Strange that it doesn't throw exception, but this is the only suspect I have. Maybe it starts garbage collection/flushes memory to disk and it takes so much time it doesn't get to throw it.
I would try to add to your query " LIMIT 543857" and then " LIMIT 543857" and see what happens.
If the above does not help, use the limit directive combined with order by.
I suspect that there is invalid entry in your table and the way to find it is binary search.
I'm having some trouble with EclipseLink. My program has to interact with a database (representing a building). I've written a little input-testmode where I can manually insert stuff through the console.
My problem: a normal getByID-operation works just fine if I try to retrieve an entity I previously inserted through EclipseLink itself (by commit()), but throws a NoResultException when trying to select a row manually inserted via SQL-script (building -> lots of rooms -> script).
This (oversemplified) works fine:
main() {
MyRoom r = new MyRoom();
r.setID("floor1-roomnr4");
em.commit(r); //entity manager
DAO.getRoomByID("floor1-roomnr4"); // works
}
and the combination of generation-script + simply getRoomByID() throws an exception.
If I try it in SQL Developer I get the result I want for the exact select statement which just threw a NoResultException. I also only get this problem in the input-mode, otherwise selecting the generated rows works also fine.
Does EclipseLink have some cache-mechanism I'm unaware of which is causing some problem?
Are you sure EclipseLink and SQL Developer are connected to the same Database? Please verify the connection information for both. Is the generation-script committing the changes with the "commit" command?
If EclipseLink works similarly to Hibernate then yes there is a cache. The "first level cache" guaranties that you get the exact same instance within one transaction which makes sense. If you know EclipseLink/transactions then try to evict all loaded instances or commit the transaction and then try your DAO again. This would force EclipseLink to fetch the data from the database again
See Answer to similar question
I've a strange situation with a little application in Java using a JDBC-OBDC. I'm inspecting a Database using the DatabaseMetaData Class. When I execute the program, everything works without anyproblem. But when I want to debug to see the values inside the Resulset containing the DatabaseMetaData a java.sql.SQLException is thrown only if I put a breakpoint within the while. Here's my code:
DatabaseMetaData patrol = con.getMetaData();
ResultSet answer = patrol.getTables(null, null, null, null);
while(answer.next()) {
if (answer.wasNull() == false) {
tableNamesAsOne = tableNamesAsOne + answer.getString("TABLE_NAME") + " ";
}
}
answer.close();
Why I cannot put my breakpoint in this section of code??
This is the printStackTrace.
Exception in thread "main" java.sql.SQLException: No data found
at sun.jdbc.odbc.JdbcOdbc.standardError(Unknown Source)
at sun.jdbc.odbc.JdbcOdbc.SQLGetDataString(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcResultSet.getDataString(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcResultSet.getString(Unknown Source)
at sun.jdbc.odbc.JdbcOdbcResultSet.getString(Unknown Source)
at Snooper.inspect(Snooper.java:56)
at Snooper.<init>(Snooper.java:26)
at Snooper.createAndShowGUI(Snooper.java:112)
at Snooper.main(Snooper.java:125)
Line Snooper.java:56 in my code refers to
tableNamesAsOne = tableNamesAsOne + answer.getString("TABLE_NAME") + " ";
Thanks.
I have installed SQL Server to reproduce your problem and verify it.
Short explanation
You must read the values ONLY ONCE and in the ORDER they appear in the SELECT. JdbcOdbc sucks. While debugging, you're reading them multiple times.
Long explanation
What you are doing is inspecting a stateful object in the debugger, which leads to dynamic results.
In this case it's the sun.jdbc.odbc.JdbcOdbcResultSet and executing the expression resultSet.getString(...) multiple times. The first time, it will work (in case your breakpoint suspends the Thread before the resultSet is asked). Then, the second time you (or your debugger) inspects the value of the expression again, the getString() method is called again and this method changes the internal state of the ResultSet object.
Although the method's name suggests it's a simple getter, it's not. It does more than that. It may actually retrieve the data from the database, change its internal position counters etc. You cannot execute the getter methods multiple times.
The ODBC Driver is a very bad thing and of low quality. Expect strange behavior and other dragons. You can get some debug information by enabling JdbcOdbc Tracing. That is done by setting a LogWriter on the DriverManager, before the JdbcOdbc-Bridge is activated:
java.sql.DriverManager.setLogWriter(new PrintWriter(System.out));
Then, you will get verbose debugging output of the JdbcOdbc-Driver like the following. It may help you to debug the problem you have. When debugging, just ensure to store the data retrieved from the ResultSet objects in local objects, so you can inspect them multiple times in the debugger.
DriverManager.getConnection("jdbc:odbc:testdbodbc")
JdbcOdbcDriver class loaded
registerDriver: driver[className=sun.jdbc.odbc.JdbcOdbcDriver,sun.jdbc.odbc.JdbcOdbcDriver#7b479feb]
DriverManager.initialize: jdbc.drivers = null
JDBC DriverManager initialized
trying driver[className=sun.jdbc.odbc.JdbcOdbcDriver,sun.jdbc.odbc.JdbcOdbcDriver#7b479feb]
*Driver.connect (jdbc:odbc:testdbodbc)
JDBC to ODBC Bridge: Checking security
No SecurityManager present, assuming trusted application/applet
JDBC to ODBC Bridge 2.0001
Current Date/Time: Wed Jan 26 00:31:27 CET 2011
Loading JdbcOdbc library
Allocating Environment handle (SQLAllocEnv)
hEnv=115724512
Allocating Connection handle (SQLAllocConnect)
hDbc=116219184
Connecting (SQLDriverConnect), hDbc=116219184, szConnStrIn=DSN=testdbodbc
RETCODE = 1
WARNING - Generating SQLWarning...
SQLWarning: reason([Microsoft][ODBC SQL Server Driver][SQL Server]Changed database context to 'master'.) SQLState(01000) vendor code(5701)
SQLWarning: reason([Microsoft][ODBC SQL Server Driver][SQL Server]Changed language setting to us_english.) SQLState(01000) vendor code(5703)
*Connection.getMetaData
*DatabaseMetaData.getDriverName
Get connection info string (SQLGetInfo), hDbc=116219184, fInfoType=6, len=300
SQLSRV32.DLL
*DatabaseMetaData.getDriverVersion
Get connection info string (SQLGetInfo), hDbc=116219184, fInfoType=7, len=300
06.01.7600
*DatabaseMetaData.getDriverName
Get connection info string (SQLGetInfo), hDbc=116219184, fInfoType=6, len=300
SQLSRV32.DLL
Driver name: JDBC-ODBC Bridge (SQLSRV32.DLL)
*DatabaseMetaData.getDriverVersion
P.S. And this was the reproduced exception, including line numbers of the Sun code for JDK 1.6.0_22. As you can see in the first line, this is what is printed out on the console when I inspected the getString() method.
Get string data (SQLGetData), hStmt=108067024, column=3, maxLen=257
RETCODE = 100
ERROR - No data found
java.sql.SQLException: No data found
at sun.jdbc.odbc.JdbcOdbc.standardError(JdbcOdbc.java:7138)
at sun.jdbc.odbc.JdbcOdbc.SQLGetDataString(JdbcOdbc.java:3907)
at sun.jdbc.odbc.JdbcOdbcResultSet.getDataString(JdbcOdbcResultSet.java:5698)
at sun.jdbc.odbc.JdbcOdbcResultSet.getString(JdbcOdbcResultSet.java:354)
at sun.jdbc.odbc.JdbcOdbcResultSet.getString(JdbcOdbcResultSet.java:411)
at sandbox.DatabaseMetadataTest.testDBMetadata(DatabaseMetadataTest.java:27)
Yeah, the debugger runs in a different thread than the metadata obtained by con.getMetaData();... so, you know, it's a different transaction with a different metadata.
Well, ok, that would be my 1st guess. I have not looked into Obdc driver source code to confirm.
Edit:
thanks for mhaller excellent remark a made a 2nd look/guess: you call wasNull() prematurely, it has meaning after some get operation of the ResultSet. Copy from javadoc
* Note that you must first call one of the getter methods
* on a column to try to read its value and then call
* the method <code>wasNull</code> to see if the value read was
* SQL <code>NULL</code>.
phew, me sucks tonight!