I need to set up a logging system for my java web application that not only logs the usual stuff (error message, error level, etc) but can also log additional information as well such as session ID. Sure I suppose I could put the session ID in the error message, but the problem is that I will end up logging lots and lots of data for lots of different users and I need to end up having a system where I can look at the log and sort the log based on session ID.
I've been looking at log4j coupled with chainsaw, and I think I could extend log4j to add additional attributes which is great, but then how do I view those custom attributes in chainsaw?
Surely i'm not the first one to have had this problem, is there something else I could use besides log4j coupled with chainsaw?
I don't know chainsaw, but logging additional cross-cutting information such as session ids, user names, requesting ip, ... is usually done through the nested diagnostic context.
Disclaimer: I'm one of Chainsaw's committers...
Chainsaw will display the NDC value in its own column, and will display any MDC entries as their own column as well.
Lots of new features coming in the upcoming release (soon) which make it easy to filter, colorize, search and sort..or, pull Chainsaw + the log4j companions down via svn and build with maven...
One really handy feature: the ability to add comments to individual events, save off the events from inside Chainsaw and email the resulting file to others, who can see your comments in the table.
Here's a screen video: http://people.apache.org/~sdeboy/chainsawdemo.avi
Are you logging or auditing your users?
Auditing involves reviewing user actions as a part of normal operations and belongs in a database.
Logging is more for break/fix.
Best thing to do would be to insert that data into a database. That way you can have indexes on session id and quickly retrieve and sort all the information either using straight SQL, or creating a light weight webapp for viewing the data from the database given a session id or other criteria to search on.
If you want a quick solution that you can feed existing log files into - try out splunk
HTH
Related
I have an ELK mounted and my app send logs use Logback. The problem is that I loose the events order when there are many logs with the same timestamp (many fast events in the same millisecond).
I would like to add a sequential number to keep the log events order when there is many logs at the same time. Is that possible? How?
Thank you and sorry for my English.
EDIT:
Sorry, I give more information about the scenario: I'm using a Springboot application which sends the logs to an ELK (Elastic-Logstash-Kibana) stack.
I need to add a field with the sequence number so then I'll be able to order the logs in Kibana using that field. Currently Kibana is ordering the logs by timestamp field but sometimes there is too many logs at the same time. The logs at the same time are unordered.
You can use custom log pattern. Please follow below link.
https://reflectoring.io/logging-format-logback/
I want to build a more advanced logging mechanism for my java web applications, similar to App engine logs.
My needs are:
Stream logs to a database (for ex. sql, bigquery or something else)
Automatically log important data (like app context, request url, request id, browser user agent, user id, etc.)
For point 1, I ca use a "buffering" implementation, where logs are put into different lists, and periodically a cron (thread) gathers all the logs in memory and write's them to database (which can also be on another server)
For point 2, the only way I found of doing this is to inject needed objects into my classes (subsystems), like ServletContext, HttpServletReqest, current user, etc, all modeled into a custom class (let's say AppLogContext), which then can be used by the logging mechanism.
The problem here is that I don't know if this is a good practice. For example, that means that many classes will have to contain this object which has access to servlet context and http request objects and I'm thinking this may create architectural problems (when building modules, layers etc) or even security issues.
App Engine will automatically log this kind of information (and much more, like latencies, cpu usage etc, but this more complicated), and it can be found in the project's Console logs (also it can duplicate logs to big query tables) and I need something for Jetty or other java web app servers.
So, is there another way of doing this, other patterns, different approaches? (couldn't find 3rd party libraries for any of these points)
Thank you.
You don't really need to invent a bicycle.
There is a common practice that you can follow:
Just log using standard logger to a file
(if you need to see logs in request context) Logback, Log4J and SLF4J supports Mapped Diagnostic Context (MDC), that's what you can use to put current request into every log line (just initialize context in a filter, put request id for example, or generate a random uuid). You can aggregate log entries by this id later
Then use ELK:
Logstash got gather logs into
ElasticSearch for storing logs
to analyze using Kibana
What is the best way to keep a log of user changes in my web application (java/tomcat/struts/mysql)? I give out accounts and each account has multiple users. I want the account administrators to be able to see who did what at any given time. And I'd like to be able to access ALL of it. First, I need a way to know which fields have been changed, then I need to log the changes for each account in a place where they can see them. Obviously, I don't want to slow the app down. I read an answer on this site suggesting keeping a db log - querying the database for changes after each query is sent. Wasn't sure how to do that.
This depends on the nature of your web application. Let's assume your web application is a e-commerce system and it allows the user to add new product, or updating an existing product. When a user perform a specific action like adding a new product, the basic goal is to capture his user name, action and time stamp. Same for updating a product, you might want to keep track what values he updated, what was the old value and when did he change that.
To achieve this, firstly you need to
Create an audit table
Obviously you want to keep track the last modified person, timestamp, created by and etc.
Create a logging mechanism whenever some changes/actions performed.
There are few ways to do this, you can either do it via application or leave everything to database trigger. I would suggest to use triggers to detect any Create/Update/Delete event in the database, and ask the trigger to capture the details and write to the Audit table. I think this is the cleanest and less maintenance way. However, if you want to log using application, you have to make code changes, create new methods to capture the details to the Audit table in your action classes.
More information on MYSQL Trigger here
I was looking on a similar "Method" to log the transactions and other stuffs in my web app. Just while browsing Google, i found this link:
https://www.owasp.org/index.php/Logging_Cheat_Sheet telling about two possible ways to log: Either on database or on filesystem at some log files...
When using the file system, it is preferable to use a separate
partition than those used by the operating system, other application
files and user generated content For file-based logs, apply strict
permissions concerning which users can access the directories, and the
permissions of files within the directories In web applications, the
logs should not be exposed in web-accessible locations, and if done
so, should have restricted access and be configured with a plain text
MIME type (not HTML) When using a database, it is preferable to
utilize a separate database account that is only used for writing log
data and which has very restrictive database , table, function and
command permissions Use standard formats over secure protocols to
record and send event data, or log files, to other systems e.g. Common
Log File System (CLFS), Common Event Format (CEF) over syslog,
possibly Common Event Expression (CEE) in future; standard formats
facilitate integration with centralised logging services
They've beautifully explained the possible ways we can log, what should be logged, what to be avoided too.
Hope it's useful to you.
Using java and made a small test server to log incoming user connections.
Adding the MongoDb as Databas and the log4j mongo appender log4j.properties
text file configured to save the Log class to Mongo.
Everything is nice.
I can now search among all Log Collections and trace a particular user based on the "message"
I immediately thought that it could be cool to log each individual user event to respective UserData Collection Document.
Users are also stored in the mongo as a UserData class Collection.
Why i think about this is because its hard to trace user log activity if you have multiple threads all writing to the same Log Collection.
When a user loggin i wanted to attach/append the user to a new Logger and let this logger live as long as user is on the server.
Is this what the MongoDbPatternLayoutAppender is all about?
The documentations and tutorials about this is sparse maybe someone can explain?
Am i on the right track here?
What's best to do?
This answer is not related to MongoDB in particular...
If you're willing to try LogBack you could use a SiftingAppender that does exactly what you're looking for!
For logging best practices (slf4j, log4j, LogBack, etc) you can take a look at this document here.
what kind of logging frame work or API to use for swing applications which is used by multiple users in Unix.
Is it possible to log all verbose/exception in one file per day or event one user one file per day? Since the user can open the same application with multiple instance.
I also have another solution is to save the exceptions into database. But if I miss the excetpions, those will not be saved in DB.
anybody has better solutions? Thank you very much!
You might like this article and discussion. The author mentions java.util.logging, which is discussed more extensively in this Java Logging Overview. In the context you describe, FileHandler should be able to sort out multiple instances per user without requiring a database.
If you are distributing your software across a network then you have less chances of knowing each and every event user does. Not sure If log4j or any other framework helps to track every user actions in your situation. Unless if you have something running on your app server.
Well..If I were you I will do it this way.
For exceptional conditions:
Come up good solid exceptional framework(something like assigning a unique Id for each exception).
In case of exception condition catch it and write the full stack trace to database table with the same unique id.
Come up some kind of search tool (web application) which helps you see what went wrong during user actions.
For Normal tracking I probably save user actions into table, but it hurts performance unless you come up with good framework. Not sure If I answered your questions. Please let me know if you have something to say.
-padur
Saving to database seems a good idea, something like when user logs in to your swing app. Create a file in user temp directory write all his actions/exceptions etc etc into the file and when he log out read the file and save it into database.Wells there are several ways to track user actions, this is one among them.