I have a view defined in SQL server 2008 that joins 4 tables together. Executing this view in SQL Server Management Studio takes roughly 3 seconds to run and returns about 45,000 records. My application is written in Java using hibernate to simply do a "from MyViewObject" query in HQL. When this is run, the execution time is consistently around 45 seconds. I have also tried simply using JDBC to run this query and received the same level of performance, so I've assumed it has nothing to do with hibernate.
My question: What can I do to diagnose this problem? There is obviously something different between how Management Studio is running the query vs how my application is running the query but I have not been able to come up with much.
The only thing I've come up with as a potentially viable explanation is an issue with the jtds library that contains the driver for SQL Server in Java.
Any guidance here would be greatly appreciated.
UPDATE
I went back to trying pure JDBC and tried adding the selectMethod and responseBuffering attributes to my connection string but didn't get any improvements. I also took my JDBC code from my application and ran it from a test program containing nothing but my JDBC code and it ran in the expected 3 seconds. So to me this seems environmental for the application.
My application is a Google Web Toolkit(GWT) based app, and the JDBC code is being run in my primary RPC Servlet. Essentially, the RPC method receives the call and immediately executes the JDBC code. Nothing in this setup gives me much indication of why the performance is terrible though. I am going to try the JDBC 3.0 driver and see if that works any better, but it doesn't feel like that will fix the issue to me quite yet.
My goal for the moment is to get my query working live with JDBC and then switch it back over to Hibernate so I can keep the testing simple enough. Thanks for the help so far!
UPDATE 2
I'm finally starting to zero in on the source of the problem, though still no idea what the actual issue is. I opened up the view in SQL Server and copied the SQL statement (rather large) exactly into my code and executed it using JDBC instead of pulling the data from the view and most of the performance issues are gone. It seems that some combination of GWT, SQL Server Views and JDBC is not working properly here. I don't see keeping a very large hand-written query in my code as a long term solution, but it does offer a bit more insight.
<property name="hibernate.show_sql">true</property>
setting this will show you the SQL query generated by hibernate. Analyze the query and make sure you are not missing a relationship.
reply for Update 1 and 2:
Like you mentioned, ran the query on your sql query and it seems like it is fast. So another thing to remember about hibernate is that it creates the object that is returned by your query (of course this depends if you initialize lazy obj. Dont remember what it is called). How many objects does your query return? also you can do a simple bench on where the issue is.
For example, before running the query, sysout the current time and then sysout the current time after. do these for all the places that you suspect is slowing your application down.
To analyze the problem you should look up you manual for tools that display the query or execution plan. Maybe you're missing an index on a join column.
Related
I am attempting to use GDS 1.8.2 as part of a system running an embedded Neo4j 4.4.3 server. The embedded server has been an operational component for several years, and several versions of Neo4j, so that component on its own has been time-tested and functions great. This is the first attempt to add support for graph algorithms into that component.
My first test is simply to submit a CQL query:
CALL gds.graph.create("someNamedGraph", ["SomeNodeLabel"], ["SomeRelationshipType"])
In the course of getting this to work, I found I had to register the org.neo4j.gds.catalog.GraphCreateProc class in the GlobalProcedures registry of the graph database. This seems to have been successful because, while I was initially encountering a CQL exception saying the gds.graph.create procedure is unknown, now it appears to execute without exception. However, I am now seeing that the transaction doesn't produce the named graph (verified by checking the graph database using out-of-the-box Neo4j Community Edition server mode). It only runs for perhaps 0.1 seconds (vs several seconds when done through the Neo4j Community Edition server mode where it works just fine).
What I now see is that the Query Execution Type (as indicated in the Result object coming back from the execution) is marked as READ_ONLY. There are no exceptions, notifications etc. I have verified that a subsequent write transaction in the same test code, which creates a simple node (as a test), succeeds in writing a node and the Result object provides all the verifying information for that transaction.
Can anyone suggest why the gds.graph.create procedure would seem to execute with no exceptions yet somehow is getting marked as a READ_ONLY transaction? Is this even the reason why the named graph isn't getting created?
Thank you for suggestions or tips! I'm happy to provide more details if anyone has exploratory questions that might help unearth the root cause for this.
Providing an answer to my own question as this was resolved with assist from Mats Rydberg. The issue was that the call alone does not execute the operation. The result has to be iterated.
So a more fitting way to do this in embedded mode would be:
CALL gds.graph.create("someNamedGraph", ["someNodeLabel"], ["someRelationshipType"]) YIELD graphName
And, on the server-side:
try(Transaction tx = graphDb.beginTx())
{
Result resultSet = tx.execute(cql);
while(resultSet.hasNext())
resultSet.next();
}
which indeed produces the named graph.
The root problem, to me, is that my original query (with no YIELD clause) works correctly in the built-in browser app for neo4j community edition server-mode, suggesting there are things happening behind the scenes, such that its obfuscated as to how that ends up working. Regardless, the solution to the problem is now understood and hopefully, in the future, there will be more documentation about making GDS work in embedded mode specifically.
I'm having an interesting issue with Spring JDBC. I have a simple SELECT query which just selects a single row from the database by its identifier. When I run this query through PgAdmin raw on the database it returns in approximately 500ms. When I run it through my Java stack it takes a full 20 seconds, and it does not get faster on subsequent runs.
This is the last layer of my code that hangs:
return jdbcTemplate.queryForObject(QUERY_SINGLE_COMPONENT, queryArguments, new ComponentRowMapper());
(my jdbcTemplate is a NamedParameterJdbcTemplate). I have run VisualVM on my code during this execution:
the PostgreSQL driver readMore is taking essentially all of the time. I'm using the PostgreSQL driver 42.0.0 but I tried updating to the latest version which did not help.
Other IDs in this same query return just fine, other queries I'm making through this same codebase work just fine. Its been driving me crazy for days and would appreciate any insight!
EDIT: I've managed to isolate the issue to our schema search path. We use a test schema to stage database changes. When test is in the schema search path it is slow, when test is removed it is fast. In both cases it is still slower than when I run it in PgAdmin by an order of magnitude.
I have few questions:
1) I am newbie to performance testing as a starting assignment I have to investigate the slow queries in MariaDb version : 10.0.17-MariaDB MariaDB Server .
I tried with these settings in the /etc/my.cnf.d/server.cnf
[mysqld]
long_query_time=1
log-slow-queries=/var/log/mysql/log-slow-queries.log
And after doing that I could no start the database. I get a simple
starting MySQL.... [FAILED] message.
I came across Slow query log overview for mariadb which made a little sense :(
Can any one provide me a tutorial of how it should be done.
2) In my application we already use Hibernate for data layer, Does it even make sense to find out the slow query log in the above mentioned way ?
3) How can i achieve the same thing in the mongodb.? like to list out the most frequently used queries, Slow queries ?
Any help would be appreciated.
Converting comment to answer:
When mysql won't start, as first thing you should check mysql error log (probably /var/log/(mysql/)mysqld.log) for exact error.
In your case "log-slow-queries" is starting option name (and deprecated too), you should use slow_query_log with boolean value and slow_query_log_file for filename.
slow_query_log=1 means ENABLE logging
long_query_time=1 means IF ENABLED log queries longer than 1 second
then there is
- log_queries_not_using_indexes=0/1 which, if enabled, will log even queries faster than 1s if they are not using indexes to locate rows
All these and other can be found with descriptions in MySQL manual
For MongoDB there seems to be profiler which is described in answers to this question How to find queries not using indexes or slow in mongodb
I need to log the SQL execution times in my Java EE application (Any further statistics would be an optional bonus).
Things are setup in a more-less standard way: Datasource on Application server serving pooled JDBC connections.
Application uses for DB access mix of:
Hibernate and
Spring JDBCTemplate
It runs on:
Glassfish OSE and
Oracle DBS
I know about: Anything better than P6Spy? however the question/answers are outdated, from my point of view.
What I've found so far:
I could go for the pure Hibernate approach (hibernate show query execution time)
but it's not feasible due to mixed DB access in my case
I could use some of the custom JDBC drivers
p6spy - however project seems couple years dead (last commit 3 years ago: http://sourceforge.net/p/p6spy/code/23/tree/trunk/)
log4jdbc - however no release for more than 1 year, and source activity seems to be cca 6 months not touched (http://code.google.com/p/log4jdbc/source/list)
another 2: log4jdbc-log4j2 and log4jdbc-remix - which seem alive, but I'm not sure about stability and broad usage
Recommendations based on experiences are very welcome.
Please note, I'm interested in kind of answers like: We're using XYZ and this is our experience, rather than I googled just now and feel like...
If you want dead accurate SQL execution time then best option is sql trace. But you want to put it in your Java EE application, so obviously want a somewhat accurate execution time.
Following are the things i would suggest [I have implemented them in my code]:
If you just want it for loggin purpose then have appropriate Log4j debug messages which wil l print time along with log entry.
I implemented a BatchLog table for my application which use to record start and end time of the operation. So in your case it will be start and end time of your query. Now if it is just a single query then probably triggers might help here or else you can update log table just before and after running query. Or even better will be a stored procedure which can take care of whole thing and give more accurate data.
Measuring time and logging it seems like a job for AOP. If you are using EJB, a simple interceptor should solve your problem(for example http://www.javacodegeeks.com/2013/07/java-ee-ejb-interceptors-tutorial-and-example.html). If its Spring(judging from JDBCTemplate), try Aspectj.
OK, thanks a lot for your answers.
Finally I decided to go for the P6Spy with patches specific to our scenario.
Moreover, as I believe that the gap P6spy is filling still exists these days, I decided to participate in it's development (https://github.com/p6spy/p6spy). Feel free to report your issues or feature requests to: https://github.com/p6spy/p6spy/issues to make it fit your needs.
I am currently responsible for migrating data for our application, for upgrading to new version.I am trying to migrate from HSQL to HSQL, later we will move on to other combinations.
So I have a stand alone utility to do this. I am using MockServletContext to initialize my services(this migration is to be done without starting the servers).
The problem is that all the tables are migrated except for 2-3 tables, the number depending on size of the data migrated. On extensive debugging I found nothing wrong. Meaning that all the data is getting migrated on debugging via eclipse, but on normal running it fails to complete for the last 3 tables.
Any clue where to look at?
In normal run I have put loggers to see if we are reading all the data from the source database and indeed the logs prove we do.
The only place where I am unable to put logs is when it calls a method in driver.
In the last step we give a call to PreparedStatement object's executeBatch()/executeUpdate() methods(Tried with both but exactly same result).
I am completeley clueless what to do and where to look for. Any suggestions?
Thanks
In normal run I have put loggers to see if we are reading all the data from the source database and indeed the logs prove we do. The only place where I am unable to put logs is when it calls a method in driver.
If you suspect something wrong there, try wrapping your driver in log4jdbc. It will show the SQL issued to DB. Good luck!