Using java and made a small test server to log incoming user connections.
Adding the MongoDb as Databas and the log4j mongo appender log4j.properties
text file configured to save the Log class to Mongo.
Everything is nice.
I can now search among all Log Collections and trace a particular user based on the "message"
I immediately thought that it could be cool to log each individual user event to respective UserData Collection Document.
Users are also stored in the mongo as a UserData class Collection.
Why i think about this is because its hard to trace user log activity if you have multiple threads all writing to the same Log Collection.
When a user loggin i wanted to attach/append the user to a new Logger and let this logger live as long as user is on the server.
Is this what the MongoDbPatternLayoutAppender is all about?
The documentations and tutorials about this is sparse maybe someone can explain?
Am i on the right track here?
What's best to do?
This answer is not related to MongoDB in particular...
If you're willing to try LogBack you could use a SiftingAppender that does exactly what you're looking for!
For logging best practices (slf4j, log4j, LogBack, etc) you can take a look at this document here.
Related
I have an ELK mounted and my app send logs use Logback. The problem is that I loose the events order when there are many logs with the same timestamp (many fast events in the same millisecond).
I would like to add a sequential number to keep the log events order when there is many logs at the same time. Is that possible? How?
Thank you and sorry for my English.
EDIT:
Sorry, I give more information about the scenario: I'm using a Springboot application which sends the logs to an ELK (Elastic-Logstash-Kibana) stack.
I need to add a field with the sequence number so then I'll be able to order the logs in Kibana using that field. Currently Kibana is ordering the logs by timestamp field but sometimes there is too many logs at the same time. The logs at the same time are unordered.
You can use custom log pattern. Please follow below link.
https://reflectoring.io/logging-format-logback/
I am developing a middleware application using Java, Spring, Hibernate, slf4j, log4j, Oracle db stack. Currently I log output to text file. I want to store logs in database for troubleshooting purpose. I tried using log4j db appender to directly log into db but I found the performance to be too slow. So now instead I let log4j append to a file and in a separate thread I read the log file line by line and insert into database. This method is not too slow and also it does not affect the performance of main application.
My question is, does anyone else have a better idea or is there a better way to do it ? I dont want to use any tools like loggy or splunk, because for my purpose those tools are an overkill. I want to know of any homegrown techniques I can use.
I know you have said you don't want to use external tools but I think that is a mistake. The effort you are putting in to create a bespoke solution for your logging has already be made by others and will provide a better more efficient and robust solution than you can.
For starters, loading the log files into the database simply to make them more searchable is a bad idea. You then have the performance overhead of running the database and loading all the files in, plus having to write and test all your code to do this.
I would recommend looking at the logstash tool.
If you are determined to go with your solution of loading then to a database then you need to provide some information on what type of database you intend to use.
At my workplace, we wrote a custom log4j appender that writes log messages to the database (uses a dedicated thread asynchronously, so no performance hit).
I prefer it a lot over writing to log files - a database-based log is much more easy to query and analyze.
Is there an open source solution that does this (for log4j specifically, or any other java loggers)?
Some things that our appender has, and I would like to see in an alternative:
Logs exceptions (duh!)
Database writes are from a separate thread/pool
Our appender supports the following columns, and I would like to see all of them in whatever solution we find.
LogId
Time
message
stacktrace
process id
thread id
machine name
component
Level (debug/info/warn/...)
ThreadName
There is also a DBAppender class provided by log4j (log4j requires a specific set of tables to log using this appender).
http://logging.apache.org/log4j/companions/receivers/apidocs/org/apache/log4j/db/DBAppender.html
There is an updated non-Apache jdbc logger available here you may also want to try:
http://www.dankomannhaupt.de/projects/index.html
Just curious, wouldn't it severely affect the performance of an application hosting such appender? Logging directly into relational database is quite costly even when you do it asynchronously.
You don't need a custom appender for LOG4J to write to databases. You can use JDBCAppender bundled with Apache's distribution.
According to APACHE's documentation, this API could be replaced in the future. But for now, we use it and it works very well.
I need to set up a logging system for my java web application that not only logs the usual stuff (error message, error level, etc) but can also log additional information as well such as session ID. Sure I suppose I could put the session ID in the error message, but the problem is that I will end up logging lots and lots of data for lots of different users and I need to end up having a system where I can look at the log and sort the log based on session ID.
I've been looking at log4j coupled with chainsaw, and I think I could extend log4j to add additional attributes which is great, but then how do I view those custom attributes in chainsaw?
Surely i'm not the first one to have had this problem, is there something else I could use besides log4j coupled with chainsaw?
I don't know chainsaw, but logging additional cross-cutting information such as session ids, user names, requesting ip, ... is usually done through the nested diagnostic context.
Disclaimer: I'm one of Chainsaw's committers...
Chainsaw will display the NDC value in its own column, and will display any MDC entries as their own column as well.
Lots of new features coming in the upcoming release (soon) which make it easy to filter, colorize, search and sort..or, pull Chainsaw + the log4j companions down via svn and build with maven...
One really handy feature: the ability to add comments to individual events, save off the events from inside Chainsaw and email the resulting file to others, who can see your comments in the table.
Here's a screen video: http://people.apache.org/~sdeboy/chainsawdemo.avi
Are you logging or auditing your users?
Auditing involves reviewing user actions as a part of normal operations and belongs in a database.
Logging is more for break/fix.
Best thing to do would be to insert that data into a database. That way you can have indexes on session id and quickly retrieve and sort all the information either using straight SQL, or creating a light weight webapp for viewing the data from the database given a session id or other criteria to search on.
If you want a quick solution that you can feed existing log files into - try out splunk
HTH
what kind of logging frame work or API to use for swing applications which is used by multiple users in Unix.
Is it possible to log all verbose/exception in one file per day or event one user one file per day? Since the user can open the same application with multiple instance.
I also have another solution is to save the exceptions into database. But if I miss the excetpions, those will not be saved in DB.
anybody has better solutions? Thank you very much!
You might like this article and discussion. The author mentions java.util.logging, which is discussed more extensively in this Java Logging Overview. In the context you describe, FileHandler should be able to sort out multiple instances per user without requiring a database.
If you are distributing your software across a network then you have less chances of knowing each and every event user does. Not sure If log4j or any other framework helps to track every user actions in your situation. Unless if you have something running on your app server.
Well..If I were you I will do it this way.
For exceptional conditions:
Come up good solid exceptional framework(something like assigning a unique Id for each exception).
In case of exception condition catch it and write the full stack trace to database table with the same unique id.
Come up some kind of search tool (web application) which helps you see what went wrong during user actions.
For Normal tracking I probably save user actions into table, but it hurts performance unless you come up with good framework. Not sure If I answered your questions. Please let me know if you have something to say.
-padur
Saving to database seems a good idea, something like when user logs in to your swing app. Create a file in user temp directory write all his actions/exceptions etc etc into the file and when he log out read the file and save it into database.Wells there are several ways to track user actions, this is one among them.