I am adding SLF4J and Logback to my application and I am unsure if I should log the SQL statements that I generate in the repository layer (using Spring JDBC). The level of these statements would of course be set to DEBUG since it could generate a lot of log statements.
Is it common to log SQL statements generated by the application?
Yes, It is common.
All ORMs, including openjpa and hibernate do it. All mappers like MyBatis have some logging mechanism to hook into any of several logging implementations.
Even in immemorial times. The drivers used to do it when a java.sql.DriverManager#setLogStream was invoked
;)
I'd write this kind of information to database, keeping statistics so I can query and summarize them.
It's certainly possible to do it with the log, but it's not as easy to summarize as it is from the database. No SQL for log files without extraordinary efforts.
Actually if you have a huge application that is considered a financial asset in your point of view, than of course you should log your application since the logging will be considered a security major you can refer to when ever you want. Plus the logging is useful for debugging.
But you have to consider which level of logging you want to choose since it will have a huge load on your database if you want log all your SQL statements.
Related
I am developing a middleware application using Java, Spring, Hibernate, slf4j, log4j, Oracle db stack. Currently I log output to text file. I want to store logs in database for troubleshooting purpose. I tried using log4j db appender to directly log into db but I found the performance to be too slow. So now instead I let log4j append to a file and in a separate thread I read the log file line by line and insert into database. This method is not too slow and also it does not affect the performance of main application.
My question is, does anyone else have a better idea or is there a better way to do it ? I dont want to use any tools like loggy or splunk, because for my purpose those tools are an overkill. I want to know of any homegrown techniques I can use.
I know you have said you don't want to use external tools but I think that is a mistake. The effort you are putting in to create a bespoke solution for your logging has already be made by others and will provide a better more efficient and robust solution than you can.
For starters, loading the log files into the database simply to make them more searchable is a bad idea. You then have the performance overhead of running the database and loading all the files in, plus having to write and test all your code to do this.
I would recommend looking at the logstash tool.
If you are determined to go with your solution of loading then to a database then you need to provide some information on what type of database you intend to use.
At my workplace, we wrote a custom log4j appender that writes log messages to the database (uses a dedicated thread asynchronously, so no performance hit).
I prefer it a lot over writing to log files - a database-based log is much more easy to query and analyze.
Is there an open source solution that does this (for log4j specifically, or any other java loggers)?
Some things that our appender has, and I would like to see in an alternative:
Logs exceptions (duh!)
Database writes are from a separate thread/pool
Our appender supports the following columns, and I would like to see all of them in whatever solution we find.
LogId
Time
message
stacktrace
process id
thread id
machine name
component
Level (debug/info/warn/...)
ThreadName
There is also a DBAppender class provided by log4j (log4j requires a specific set of tables to log using this appender).
http://logging.apache.org/log4j/companions/receivers/apidocs/org/apache/log4j/db/DBAppender.html
There is an updated non-Apache jdbc logger available here you may also want to try:
http://www.dankomannhaupt.de/projects/index.html
Just curious, wouldn't it severely affect the performance of an application hosting such appender? Logging directly into relational database is quite costly even when you do it asynchronously.
You don't need a custom appender for LOG4J to write to databases. You can use JDBCAppender bundled with Apache's distribution.
According to APACHE's documentation, this API could be replaced in the future. But for now, we use it and it works very well.
For a security system which works as a big brother (like a watched mandatory access control), we have to intercept and handle all select statements hibernate is generating. We store the user, the timestamp and the sql select in a database to enable some profiling with other tools. The informations allow to determine what a user tried to look at. For the select statements the prepared properties are valuable. We need the complete SQL statement including all parameters.
Is there any listener or interceptor where we can join and handle all these things? The biggest outstanding problem so far is collecting of the statement parameters.
Thank you
The actual parameter values happen to be made available (atleast to my knowledge), when the logging level of the org.hibernate package is set to DEBUG, and with the hibernate.show_sql property set. Use a JDBCAppender, if you want the logger output in the database itself.
Alternatively, you could take a look at the log4jdbc project, which claims the following:
In the logged output, for prepared
statements, the bind arguments are
automatically inserted into the SQL
output. This greatly Improves
readability and debugging for many
cases.
If that is not suitable, you could investigate whether P6Spy can be used in your situation. On WebLogic Server, the equivalent functionality is achieved via the WebLogic JDBC Spy, which comes out of the box with the WebLogic JDBC drivers for certain databases. Both of these write to System.out and not to a database (unless I'm mistaken), so it might not be that useful.
You can use Interceptor.prepareSQL() (3.1+) to intercept the prepared statements.
I don't think you can get the actual parameters without going down in the abstraction layer. A possible solution would be to use a JDBC proxy driver (see P6Spy).
Hope that helps.
I need to set up a logging system for my java web application that not only logs the usual stuff (error message, error level, etc) but can also log additional information as well such as session ID. Sure I suppose I could put the session ID in the error message, but the problem is that I will end up logging lots and lots of data for lots of different users and I need to end up having a system where I can look at the log and sort the log based on session ID.
I've been looking at log4j coupled with chainsaw, and I think I could extend log4j to add additional attributes which is great, but then how do I view those custom attributes in chainsaw?
Surely i'm not the first one to have had this problem, is there something else I could use besides log4j coupled with chainsaw?
I don't know chainsaw, but logging additional cross-cutting information such as session ids, user names, requesting ip, ... is usually done through the nested diagnostic context.
Disclaimer: I'm one of Chainsaw's committers...
Chainsaw will display the NDC value in its own column, and will display any MDC entries as their own column as well.
Lots of new features coming in the upcoming release (soon) which make it easy to filter, colorize, search and sort..or, pull Chainsaw + the log4j companions down via svn and build with maven...
One really handy feature: the ability to add comments to individual events, save off the events from inside Chainsaw and email the resulting file to others, who can see your comments in the table.
Here's a screen video: http://people.apache.org/~sdeboy/chainsawdemo.avi
Are you logging or auditing your users?
Auditing involves reviewing user actions as a part of normal operations and belongs in a database.
Logging is more for break/fix.
Best thing to do would be to insert that data into a database. That way you can have indexes on session id and quickly retrieve and sort all the information either using straight SQL, or creating a light weight webapp for viewing the data from the database given a session id or other criteria to search on.
If you want a quick solution that you can feed existing log files into - try out splunk
HTH
My Apache Tomcat Server is getting periodic updates from a Java based client Application, At the moment the scenario is just one client and talking to the server.
I want to log the messages from the client onto the server with time-stamp what kind of framework will help me in achieving this?
EDIT: The OP goal was actually pretty unclear and I'm modifying my answer after some clarifications.
Well, I'm not sure, but maybe a logging framework will suit your needs. If so, have a look at:
Log4J: The most famous logging framework, widely used.
Java Logging aka java.util.logging: didn't succeed to replace Log4J.
Logback: "Logback is intended as a successor to the popular log4j project. It was designed, in addition to many individual contributors, by Ceki Gülcü, the founder of log4j".
SL4J: A "Simple Logging Facade for Java serves as a simple facade or abstraction for various logging frameworks, e.g. java.util.logging, log4j and logback, allowing the end user to plug in the desired logging framework at deployment time".
And pick one of them (I'd use Log4J or Logback).
To save your messages for later processing from the webapp (e.g. generating a web page with some graphs/charts), the best approach is to use a database. Just read/write them from/to a simple table with a timestamp column.
If you are not really familiar with Java, JDBC, persistence, connection pooling, datasource, etc, I'd suggest to use the Spring framework as it will hide most of the complexity. For the database part, have a look at the Chapter 11. Data access using JDBC from the Spring documentation. Pay a special attention to the JdbcTemplate or the SimpleJdbcTemplate, they should allow you to get the job done.
Create a special JSP page for accepting log entries, and invoke it with
http://..... foo.jsp?l=the%20stuff%to%log (i.e. URL encoded)
You then just need to pick out the "l" parameter and do with it what you need to do. An initial implementation could be invoking the log(String s) method in the servlet context.