I have a layered application in Java which has a multi thread data access layer which is invoked from different points. A single call to this layer is likely to spawn several threads to parallelize requests to the DB.
What I'm looking for is a logging tool that would allow me to define "activities" that are composed by various threads. Therefore, the same method in the data access layer should log different outputs depending on its caller. The ability to group different outputs to summarize the total cost of an operation is also important.
Although the application is in Java, language is not a restriction; what I need are the design guidelines so to eventually implement it. We are currently using log4j, but can't get this behaviour from it.
You should also have a look at the nested diagnostic context feature of log4j. Pushing different contexts to the logger for different callers might do the trick for you.
You should be able to pass a logger around, so you create a logger based on some "common" for the task data - i.e. username, etc. Then, pass this logger as parameter to all methods you need. That way, you'll be able to set different filters and/or rules in your log4j config file. Or to scrape the output file based on the logger name.
EDIT: Also check MDC and NDC classes in log4j. You can add there context data.
In log4j you can log the thread name with the "%t" pattern. See log4j Pattern Layout.
In one of my (web) applications, i use a ThreadLocal logger that captures logging information into a StringBuilder. The logger object is initialized in the HttpServlet#service method, if a trace parameter is set (if it is not set, there is a very fast null-logger). The resulting output is either dumped as a HTML comment into the requesting page, or written to a log file in one segment.
In Java5 (and later) you can call
StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();
Inspect the stack trace to whatever depth you want and log accordingly.
In Java 1.4 you can get the same info with
StackTraceElement[] stackTrace = new Exception().getStackTrace();
You want to associate logger objects with threads I think. A ThreadLocal variable holding a log4j logger instance for each thread might help:
http://java.sun.com/javase/6/docs/api/java/lang/ThreadLocal.html
You will need to pass some structure to the data access layer that identifies the current "activity". You might already have an "Activity"-class that makes sense, you might use a Logger-instance as Sunny suggested or you might use a third structure to keep track of the activity-context.
In any case, since your "activity" is processed across multiple threads you cannot use thread-local-storage for keeping track of the current "activity", like most of the other current answers suggest. You will need to pass it around explicitly.
I would suggest making a small facade on top of log4j that expands the interface with methods like
void debug(Activity activity, String message);
and passing the activity-context into this from the data access layer.
You will need to make some modification to the data access layer to allow you to pass the current activity to it, but how best to do that depends strongly on the current interface.
If you use the Workspace-pattern, you might just need to add a setActivity() method on the Workspace-class, but other interface-pattern might require you to add an Activity parameter to all methods.
If you for some reason is unable or unwilling to change the data access layer, you might of course store the activity-context in thread-local-storage before invoking the data access layer and retrieve it just before spawning the sub-threads or enqueing the jobs in the data access layer. That is a workable solution, but is it a bit dangerous to pass information around in that way.
You can use MDC or NDC for your scenario, NDC works on principle of stack while MDC works on Map, here is official documentation for both
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/MDC.html
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/NDC.html
Related
I'm looking to log the request and response to a web service. I'm using slf4j with underlying log4j2 implementation. My logger statement looks like the below.
LOGGER.info("{}",new CustomObject(request,response,param1,param2));
I have implemented the toString methods in all the necessary objects and in the CustomObject class to log all attributes of that object.
I see that the toString method of the CustomObject is called before it passes the log message to the Asynch logger.
Is there anyway that the serialization / toString method call of the custom object be deferred to when the actual logging takes place?
This is by design: if an Object is logged it's possible that it is mutated before the background thread can log it, and you would end up with a log entry different from what you intended.
In your example your application doesn't hold a reference to the CustomObject, so it cannot change, but Log4j2 can't know that, so it takes the conservative approach.
There is a system property to switch this off, but using it means all objects logged must be effectively immutable or you will find your logs are lying to you... (I'm also not 100% sure that the system property still works since Log4j2 became garbage free in version 2.6.)
Update (2017-12-09): the system property still works in post 2.6 versions. (But I would not recommend rendering the message in the background thread unless you are very confident that the application doesn’t modify the objects that were logged.)
When we add logs in to the java class (using log4j), Is it fine to add thread id with that log messages? is it a bad practice? My idea was to add this thread id; Once we examine a log file of a multithreaded application, it is difficult to find out the correct flow using logs. (As an example, say authentication flow). Is there any better approach for this rather than logging thread id?
Log4j already supports the thread name using t placeholder in its pattern layout. So this is a supported feature that you should use if you find it useful. This way you don't need to pass the thread name manually. However it doesn't make use of the thread ID. So you should give meaningful names to your threads. This should be preferred as it is more indicative to what is going on in your application than just plain thread ids.
If it is the thread id, please refer this answer
However if you only need the thread name, you can use the t pattern configuration, please refer here.
If you are using Java Logger API - LogRecord has method getThreadID() and can be configured to log.
For log4j there are no ThreadId methods available and No harm in logging it.
In cases If your class inherits from Thread, you can use methods getName and setName to name each thread. Otherwise you could just add a name field to MyTask, and initialize it in your constructor And use a more sensible Thread Name instead of ID.
Logback has a special appender called SiftingAppender which provides a very nice solution to the type of problems you describe. A SiftingAppender can be used to separate (or sift) logging according to any runtime attribute, including thread id
If your concern is about several JVMs writing to the same FileAppender, then i'd suggest two things:
using SLF4J as a logging facade
using logback as logging implementation, in prudent mode
In prudent mode, FileAppender will safely write to the specified
file, even in the presence of other FileAppender instances running in
different JVMs, potentially running on different hosts.
Refer this: http://logback.qos.ch/manual/appenders.html#SiftingAppender
I have a scenario wherein I invoke a third party REST URL from inside a FOR loop for different values of the field JOBNAME. I wish to log the logging statements for each JOBNAME inside a separate log file i.e. the request sent to the REST URL, all the intermediate business logic logging and the response received would need to be part of different log file for each JOBNAME.
The catch here is that the processing involves no threads. Is the MDC approach for log4j feasible for non-threaded scenario's as well. If not, how can I achieve this?
Having a single thread you can still put a variable into MDC context and let the calls propagate and log whatever they do. Just make sure that you do MDC.put(..) before interesting staff gets called and remove it when you are done so it doesn't linger. For example, you can do MDC.put(..) within the FOR loop and each iteration will be with different context even though it's the same thread. Not sure about elsewhere but in Java it works fine.
After my research and discussion here I decided I need to set the same name for threads on different JVMs which belong to the same control flow in the distributed system. Threads are created e.g. by RMI. Is it possible to set name when thread is created in that way?
There's no automatic means to transfer this info from client-to-server.
It sounds like you want/need some sort of Context object set up on the client (per-thread?) and passed as a method argument to your RMI servers. That Context object could contain not just the thread-name, but perhaps also other info like the calling process pid etc.
You'd then have to use that Context object to set thread-names etc. accordingly via Thread.setName() once it's been passed across the wire. Going forwards, you could set up context-specific info in your logging framework using this (e.g. using Log4j nested diagnostic contexts)
The use of aspects to automate this further is left as a further exercise for the reader :-)
The Thread class has a static method to setName(String). If you can have your threads, wherever they come from, running that method, you should be good to go. These guys had similar issues with Tomcat related threads.
I have a system which contains multiple applications connected together using JMS and Spring Integration. Messages get sent along a chain of applications.
[App A] -> [App B] -> [App C]
We set a global id in the message header so we can trace each message lifecycle through the system.
I would like to be able to prepend any log message in the system with the message global id.
Has anyone else done this? Is there any way to associate this variable to the Thread so I can access it in future methods? I'd rather not pass the variable around in methods of the system.
I think ThreadLocal may be what you want here, though some may find this approach an abuse of ThreadLocal's purpose, or good design. Something like:
public class MyIDManager {
public static final ThreadLocal<Long> myID = new ThreadLocal<Long>();
}
...
// set ID at some point
MyIDManager.myID.set(theNewID);
...
// read it later
long currentID = MyIDManager.get();
The magic here is that myID's value is actually specific to a Thread, and will be different when accessed from different threads.
You can then do what you like with the ID, including logging it.
Thread t = Thread.currentThread();
t.setName("Your ID Here");
Another answer, which I believe the first post is alluding to, is to simply ask your logging framework to include the thread name in the log statement. For example, log4j lets you add the thread name with 't' in its PatternLayout: http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html I've seen this in other frameworks too.
BTW, this approach is part of a wider enterprise logging pattern. I attempted to document this pattern here. The summary is:
TL;DR
At the earliest inception of a ‘process’ create an ID which is a
combination of an ‘origin id’ and ‘unique id’. This non-business
related global inception ID (GIID), should be used for every log
output to provide machine readability and tool use. When each new
business level or operational id is obtained or generated, it is
logged to provide an ‘association’ with this GIID. This is similar to
various existing practices so it is presented as a Software Design
Pattern.