What is the best way to keep a log of user changes in my web application (java/tomcat/struts/mysql)? I give out accounts and each account has multiple users. I want the account administrators to be able to see who did what at any given time. And I'd like to be able to access ALL of it. First, I need a way to know which fields have been changed, then I need to log the changes for each account in a place where they can see them. Obviously, I don't want to slow the app down. I read an answer on this site suggesting keeping a db log - querying the database for changes after each query is sent. Wasn't sure how to do that.
This depends on the nature of your web application. Let's assume your web application is a e-commerce system and it allows the user to add new product, or updating an existing product. When a user perform a specific action like adding a new product, the basic goal is to capture his user name, action and time stamp. Same for updating a product, you might want to keep track what values he updated, what was the old value and when did he change that.
To achieve this, firstly you need to
Create an audit table
Obviously you want to keep track the last modified person, timestamp, created by and etc.
Create a logging mechanism whenever some changes/actions performed.
There are few ways to do this, you can either do it via application or leave everything to database trigger. I would suggest to use triggers to detect any Create/Update/Delete event in the database, and ask the trigger to capture the details and write to the Audit table. I think this is the cleanest and less maintenance way. However, if you want to log using application, you have to make code changes, create new methods to capture the details to the Audit table in your action classes.
More information on MYSQL Trigger here
I was looking on a similar "Method" to log the transactions and other stuffs in my web app. Just while browsing Google, i found this link:
https://www.owasp.org/index.php/Logging_Cheat_Sheet telling about two possible ways to log: Either on database or on filesystem at some log files...
When using the file system, it is preferable to use a separate
partition than those used by the operating system, other application
files and user generated content For file-based logs, apply strict
permissions concerning which users can access the directories, and the
permissions of files within the directories In web applications, the
logs should not be exposed in web-accessible locations, and if done
so, should have restricted access and be configured with a plain text
MIME type (not HTML) When using a database, it is preferable to
utilize a separate database account that is only used for writing log
data and which has very restrictive database , table, function and
command permissions Use standard formats over secure protocols to
record and send event data, or log files, to other systems e.g. Common
Log File System (CLFS), Common Event Format (CEF) over syslog,
possibly Common Event Expression (CEE) in future; standard formats
facilitate integration with centralised logging services
They've beautifully explained the possible ways we can log, what should be logged, what to be avoided too.
Hope it's useful to you.
Related
I want to build a more advanced logging mechanism for my java web applications, similar to App engine logs.
My needs are:
Stream logs to a database (for ex. sql, bigquery or something else)
Automatically log important data (like app context, request url, request id, browser user agent, user id, etc.)
For point 1, I ca use a "buffering" implementation, where logs are put into different lists, and periodically a cron (thread) gathers all the logs in memory and write's them to database (which can also be on another server)
For point 2, the only way I found of doing this is to inject needed objects into my classes (subsystems), like ServletContext, HttpServletReqest, current user, etc, all modeled into a custom class (let's say AppLogContext), which then can be used by the logging mechanism.
The problem here is that I don't know if this is a good practice. For example, that means that many classes will have to contain this object which has access to servlet context and http request objects and I'm thinking this may create architectural problems (when building modules, layers etc) or even security issues.
App Engine will automatically log this kind of information (and much more, like latencies, cpu usage etc, but this more complicated), and it can be found in the project's Console logs (also it can duplicate logs to big query tables) and I need something for Jetty or other java web app servers.
So, is there another way of doing this, other patterns, different approaches? (couldn't find 3rd party libraries for any of these points)
Thank you.
You don't really need to invent a bicycle.
There is a common practice that you can follow:
Just log using standard logger to a file
(if you need to see logs in request context) Logback, Log4J and SLF4J supports Mapped Diagnostic Context (MDC), that's what you can use to put current request into every log line (just initialize context in a filter, put request id for example, or generate a random uuid). You can aggregate log entries by this id later
Then use ELK:
Logstash got gather logs into
ElasticSearch for storing logs
to analyze using Kibana
We would like to start using salesforce for managing sales contacts, but there is also some business functions regarding contacts that we would like to retain in our current system.
As far as I can see, that means that we're going to need a two-way sync? Ie, when anything changes on salesforce, we need to update it on our system and vice versa.
I'm suggesting some kind of messaging product that can sit in the middle and retry failed messages, because I have a feeling that without that, things are going to get very messy? Eg, when one or other service is down.
The manager on the project would like to keep it simple and feels that using messages rather then realtime point-to-point calls is overkill, but I feel like without it we're going to be in for a world of pain.
Does anyone have any experience with trying to do two-way syncs (actually even one-way suffers from the same risks I think?)
Many thanks for your insights..
I can't speak for your system, but on the side Salesforce API, take a look at the getUpdated() and getDeleted() calls, which are designed for data replication. The SOAP API doc has a section that goes into detail about how to use them effectively.
We use Jitterbit to achieve two way sync between Salesforce and billing system. The Salesforce has a last-modified field and so does our biling system (you system should have this, if not, add a timestamp field to the table in its SQL storage). The only important thing is to chose one of the keys as primary (either SF_ID or other system's key) and create that key field in another system as it will be used for conflict resolution. The process is simple and multistep, load all modified SF data into flat file, load all modified secondary system data into another flat file, look for conflicts by comparing two files over a common key field, notify admin on conflicts, if any, and propagate all non-conflicting changes to another system. We run this process every 10 minutes and we store the last timestamp on both systems between cycle runs so that we only take records that were modified between two cycles.
In case two users edit at the same time, you will either encounter a confict and resolve it manually or you will get the "last-saved-wins" outcome.
You also have to cater for new provisions, on SF side use upsert instead of update (using external or SF key depending on which you chose above), on your other side it depends on the system.
Are there any possible ways by which a malicious user on a web application can manipulate the input that is sent by the front-end of web application (not talking about the FORM DATA, of course) but the requests that are sent like for e.g., when I allow him to edit his profile or his content, he may manipulate the IDs (userId or the contentId) so that he may maliciously do evil with other users content? These inputs are fixed on a webpage & are not editable but still can the users manipulate them?
Is it possible that users may do harm in this manner? How can I safeguard my application against this? Besides, verifying user's identity and his contents/properties on the application prior to allowing each of his actions.
Yes of course. Anything that comes from the client can be modified and cannot be trusted at all.
You need to do server-side checks if the user is editing his own profile or something he's allowed to edit.
For things like editing the profile you could simply use the userid stored in his session though (assuming it's secure, i.e. stored server-side or in cryptographically signed cookies). Only let data go through the client if it's necessary - if the data is already available on the server, you don't even have to give the user the feeling that he might be able to tamper with it. Even though it could be used as a honey-pot - but that's not really the purpose of most webapps...
Yes, it is possible and it is a real danger.
There are two things you can do:
Implement an access control / permission system which controls which data records a user can access or modify.
Store information that is none of their business in a session object on the server.
(By the way, these are not exclusive options, ideally you should do both.)
Both solutions still leave you prone to session hijacking though, which is a different, more global problem.
Is there a way to detect database events, e.g. insert, update and delete, comparable to file access monitors like JNotify (can detect read, create, modify of files and directories)?
Looking for something like database event listeners because I don't want to do polling.
Thanks!
In general no. AFAIK, no such facility exists in JDBC or in the SQL standards.
It might be possible for certain databases / configurations using database specific functionality. For example, if the database can be configured to run arbitrary Java code in a trigger, you might be able to get it to send an event into a pubsub system that will deliver it to your application code.
But I think it would be better to modify your application code-base to generate the events itself.
It depends on the database you are using, as some allow you to run code that would allow you to make a call to a server, through a trigger, but, then you take a performance hit on these modifications, so you would want to use a webservice that doesn't send any information back to, to limit the performance hit.
It also depends on if you are using a server, as then you could use AspectJ to monitor any of the update queries.
I need to set up a logging system for my java web application that not only logs the usual stuff (error message, error level, etc) but can also log additional information as well such as session ID. Sure I suppose I could put the session ID in the error message, but the problem is that I will end up logging lots and lots of data for lots of different users and I need to end up having a system where I can look at the log and sort the log based on session ID.
I've been looking at log4j coupled with chainsaw, and I think I could extend log4j to add additional attributes which is great, but then how do I view those custom attributes in chainsaw?
Surely i'm not the first one to have had this problem, is there something else I could use besides log4j coupled with chainsaw?
I don't know chainsaw, but logging additional cross-cutting information such as session ids, user names, requesting ip, ... is usually done through the nested diagnostic context.
Disclaimer: I'm one of Chainsaw's committers...
Chainsaw will display the NDC value in its own column, and will display any MDC entries as their own column as well.
Lots of new features coming in the upcoming release (soon) which make it easy to filter, colorize, search and sort..or, pull Chainsaw + the log4j companions down via svn and build with maven...
One really handy feature: the ability to add comments to individual events, save off the events from inside Chainsaw and email the resulting file to others, who can see your comments in the table.
Here's a screen video: http://people.apache.org/~sdeboy/chainsawdemo.avi
Are you logging or auditing your users?
Auditing involves reviewing user actions as a part of normal operations and belongs in a database.
Logging is more for break/fix.
Best thing to do would be to insert that data into a database. That way you can have indexes on session id and quickly retrieve and sort all the information either using straight SQL, or creating a light weight webapp for viewing the data from the database given a session id or other criteria to search on.
If you want a quick solution that you can feed existing log files into - try out splunk
HTH