For a security system which works as a big brother (like a watched mandatory access control), we have to intercept and handle all select statements hibernate is generating. We store the user, the timestamp and the sql select in a database to enable some profiling with other tools. The informations allow to determine what a user tried to look at. For the select statements the prepared properties are valuable. We need the complete SQL statement including all parameters.
Is there any listener or interceptor where we can join and handle all these things? The biggest outstanding problem so far is collecting of the statement parameters.
Thank you
The actual parameter values happen to be made available (atleast to my knowledge), when the logging level of the org.hibernate package is set to DEBUG, and with the hibernate.show_sql property set. Use a JDBCAppender, if you want the logger output in the database itself.
Alternatively, you could take a look at the log4jdbc project, which claims the following:
In the logged output, for prepared
statements, the bind arguments are
automatically inserted into the SQL
output. This greatly Improves
readability and debugging for many
cases.
If that is not suitable, you could investigate whether P6Spy can be used in your situation. On WebLogic Server, the equivalent functionality is achieved via the WebLogic JDBC Spy, which comes out of the box with the WebLogic JDBC drivers for certain databases. Both of these write to System.out and not to a database (unless I'm mistaken), so it might not be that useful.
You can use Interceptor.prepareSQL() (3.1+) to intercept the prepared statements.
I don't think you can get the actual parameters without going down in the abstraction layer. A possible solution would be to use a JDBC proxy driver (see P6Spy).
Hope that helps.
Related
I am trying to understand "changing database without changing code". Currently working with micro services using springboot, java, thymeleaf and cloud foundry.
I have a spring boot application and attached a database as a service using cloud foundry.
My problem is I am seeing that the purpose of micro service is allowing the ease to change services without changing code.
Here is where I got stuck
In java I have a sql script, "select * from ORDER where Status = 'ACCEPTED';"
Images source
My database would be attached as a service on cloud foundry using CUPS
"jdbc:oracle:thin:username/password//host:port/servicename"
So let say I want to change this database to CUSTOMER table(take it as a different database). This will throw an error because CUSTOMER table will not have "select * from ORDER where Status = 'ACCEPTED';"
I've changed database, but wouldn't I still have to go back to my code and change the sql script?
My Attempt to resolve this issue
So instead of hard coding my sql script in java "select * from ORDER where Status = 'ACCEPTED';"
I created a system environment variable and set it as sqlScript with value of select * from ORDER where Status = 'ACCEPTED'
Then in java I called the env variable String sqlScript= System.getenv("sqlScript");
So now instead of going back into java to change sql script, user can change it through environment variables.
this is a very dirty method to go around my issue, what would be a better alternative?
I know my logic of understanding is really wrong. Please guide me to the right path.
I think the phrase 'changing database without changing code' doesn't mean that if you add/remove fields in DB you do not have to modify your codebase - it just doesn't make any sense.
What it really means is that you should use good database abstractions, so in case you need to change your database vendor from, let's say, MYSQL to OracleDB your Java code should stay the same. The only thing that may differ is some configurations.
A good example of it is ORM like Hibernate. You write your java code once, no matter what is the SQL Database that you are using underneath. To switch databases the only thing that you need to change is a dialect configuration property (In reality it's not that easy to do, but probably easier than if we were coupled to a one specific DB).
Hibernate gives you a good abstraction over SQL databases. Nowadays we have a new trend - having the abstraction over different DB families like SQL and NoSQL. So in the ideal world, your codebase should stay unchanged even if you want to change MySQL to MongoDB or even Neo4j. Spring Data probably is the most popular framework that tries to solve this problem. Another framework that I found recently is Kundera but I haven't used it so far.
So answering your question - you do not need to keep your SQL queries as system variables. All you need to do is to use proper abstractions in your language of choice.
In my opinion, it would be better to use something like Flyway or Liquibase, which are integrated really well in Spring Boot. You can find more information here.
I prefer Liquibase, since it uses a higher level format to describe your database migrations, allowing you to switch databases quite easily. This way, you can also use different databases per environment, for example:
HSQLDB during local development
MySQL in DEV and TEST
Oracle in Production
It's also possible to export your current database schema from an existing database to have an initial version in Flyway or Liquibase, this will give you a good baseline for your scripts.
I am migrating an application from WAS7 to Liberty.
The existing code usesWSCallHelper.clearStatementCache(connection) in some scenarios(example : package not found in case of procedure calls).
I found WSCallHelper doesn't exist in liberty server.
Could you please help me with an alternative solution for this in liberty.
It can be either specific liberty or a general approach which will support all servers.
It would be helpful to know more about the scenario where you need to call WSCallHelper.clearStatementCache(connection), but based on what you have described I'll assume that it is only called on an error path.
In Liberty, there is no API to programmatically clear the statement cache. However, cached statements will only be matched if a number of properties are the same such as: SQL string, RS holdability, schema, isolation level, and several others.
Why you don't need clearStatementCache:
The example scenario you described for calling clearStatementCache is when the package is not found in the DB, but the package should be reflected in either the SQL string or Schema. So, assuming that your application does not retry a failed SQL string, you should not need to invoke clearStatementCache at all.
If you really really want to clear the statement cache:
As defined by the JDBC spec, Statements are child objects of a Connection. So a creative way of clearing the statement cache would be to get rid of the connection with the bad statements. Invoking connection.close() may not accomplish this because connections can be pooled by the application server, but invoking connection.abort() will get rid of the underlying connection and therefore clear your statement cache.
We should also point out that if you want a more targeted approach and have a specific statement that you want removed from the cache (or to never go into the cache in the first place), you can use JDBC spec API java.sql.Statement.setPoolable(false)
. Invoke this prior to closing the statement handle. The application server will not cache statements that are marked as poolable=false.
One of the requirements on my current project is to be able to log the SQL statement generated by either Hibernate or Ibatis, and save them to a specific table (on SQLServer), so an administrator can come back and see what queries were run and who ran them, or even reuse the statements on demand. While I don't agree with this approach, I would like to know if there actually exists a library that can achieve this. I am using Spring framework for my web application.
There are 3rd party SQL Server tools that can capture T-SQL statements, and store them for later manipulation and analysis, such as Idera SQL Compliance Manager and ApexSQL Comply
Disclaimer: I work as a Product Support Engineer at ApexSQL
If you want to catch all queries of a middleware like Hibernate or Ibatis, the simplest way is to use SQL Server profiler or to create a trace with SQLTrace stored procedures (sp_trace_create, sp_trace_setevent).
With SQL Profiler you can save traces directly to a table, with SQLTrace stored procedure who produces a trc file, you will have to insert them in your table with sql statements.
they are lot of examples on the web, for example here
SQL Profiler have an option to generate the SQLTrace SQL script once you have defined your trace (File/export)
The SQL Profiler is in the developper edition but not in the Express edition.
SQLTrace stored procedures are in all editions (IIRC).
You can generate the trace script on your developper edition and run it on your express edition.
You can also create a stored proc that create and start the trace on server startup.
You can do it on the server, with database audit if you have the enterprise edition of SQL Server. Or you can use Extended Events. You just need to filter by the application name, which should mention the ORM. To help you further, please provide the version and edition of SQL Server.
I know a way though it's not easy to implement. Make a custom DataSource - a wrapper over a real DataSource like Apache BasicDataSource. It will intercept getConnection calls and wrap Connection returned by the target DataSource. This wrapped Connections will intercept createStatement / createPreparedStatement calls and wrap Statements / PreparedStatements returned. The Statement wrapper can now intercept SQL executions and log them.
Yet another way to do it would be to enable SQL statement logging in the Hibernate configuration settings, and then "hoover" the statements out of the log files.
But I think that a database-server-side solution is probably the best, because it can also captures queries and updates performed using the database's interactive query tool.
I am adding SLF4J and Logback to my application and I am unsure if I should log the SQL statements that I generate in the repository layer (using Spring JDBC). The level of these statements would of course be set to DEBUG since it could generate a lot of log statements.
Is it common to log SQL statements generated by the application?
Yes, It is common.
All ORMs, including openjpa and hibernate do it. All mappers like MyBatis have some logging mechanism to hook into any of several logging implementations.
Even in immemorial times. The drivers used to do it when a java.sql.DriverManager#setLogStream was invoked
;)
I'd write this kind of information to database, keeping statistics so I can query and summarize them.
It's certainly possible to do it with the log, but it's not as easy to summarize as it is from the database. No SQL for log files without extraordinary efforts.
Actually if you have a huge application that is considered a financial asset in your point of view, than of course you should log your application since the logging will be considered a security major you can refer to when ever you want. Plus the logging is useful for debugging.
But you have to consider which level of logging you want to choose since it will have a huge load on your database if you want log all your SQL statements.
A simple question: what is the more efficient way to access a db in Java/JDBC?
I'm a web developper and I'd like to write some reusable and scalable code. What is interesting for me is the use of tools like the ResultSupport: is it too expansive in terms of resource usage?
What can you suggest?
Not just JDBC specific, just general SQL stuff
If you have to rerun a query multiple times, use PreparedStatement. Use stored procedure if it is available. This is obviously not portable so YMMV.
Always close your ResultSet or Statement if you are not using it. Closing a Statement will auto close all ResultSet associated with the Statement. Still it is a good habit to close the ResultSet.
Try to restrict what can be queried eg. select * from orders where order_date between XXX and yyy. In MySQL the query may either be a full table scan or 'range' depending on how much data is returned. So deciding a how 'flexible' you want your queries to be
If you are using MySQL, use explain to optimize your query. If you are using JPA, then you don't get to see the SQL generated. (This is not strictly JDBC) You might want to enable logging on the ORM manager to display the SQL statement used. Then use explain to optimize that. You may want to use #NamedNativeQuery if the ORM generates a really convoluted query
If your JDBC driver supports batch update then use that. Batch updates is supported in PreparedStatement and ResultSet.
You can also control the commit. Good idea to turn it off if you are performing lots of updates. The call commit() yourself.
Best answer to this simple question is: "it depends".
There are many API's you can use for database access. Nearly all of them will use the JDBC API as their means to communicate with the database. So in theory, nothing can beat raw low level JDBC (just as machine code is in theory always faster than higherlevel programming languages).
But as you also like to write reusable code, I suggest you look into JPA. It's the Java standard for object persistence to relational databases. It’s performance is quite good and it’s very portable.
As JPA is just a specification, you can choose you’re own implementation: Hibernate, OpenJPA or any compliant Java EE server.
It is very important to use always a connection pool DataSource such as c3p0.
There is a project that maps java objects to mysql databases. Azirt.
Use connection pooling (either the one from your container or a standalone connection pool like c3p0 or DBCP) and something like DBUtils or Spring's JdbcTemplate.
I think the easiest and most common way is to use Spring and their JDBCTemplate.
The best approach likely depends on the stack you are using to create your web app. If you're starting afresh then Spring is a good way to go.