I'm using logback as my logging framework and have a couple of jobs that run the same main function with different parameters and would like to create a log file for each job and name the log file with the job's name.
For example, if I had jobs a,b,c that all run MyClass.main() but with different parameters, then I'd like to see a-{date}.log, b-{date}.log, c-{date}.log.
I can achieve the {date} part by specifying a <fileNamePattern>myjob-%d{yyyy-MM-dd}.log</fileNamePattern> in my logback.xml, but I'm not sure how to (or if it is even possible) create the prefix of the file names dynamically (to be the job's name).
Is there a way to dynamically name logfiles in logback? Is there another logging framework that makes this possible?
As a follow up question, am I just taking a bad approach for having multiple jobs that call the same main function with different parameters and wanting a log file named after each job? If so is there a standard/best practice solution for this case?
EDIT: The reason why I want to name each log file after the name of the job is that each job naturally defines a "unit of work" and it is easier for me to find the appropriate log file in case one of the job fails. I could simply use a rolling log file for jobs a,b,c but I found it harder for me to look through the logs and pinpoint where each job started and ended.
I would use you own logging.
public static PrintWriter getLogerFor(String prefix) {
SimpleDatFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
String filename= prefix + sdf.format(new Date());
return new PrintWriter(filename, true); // auto flush.
}
You can write a simple LRU cache e.g. with LinkedHashMap to reuse the PrintWriters.
Is there a way to dynamically name logfiles in logback? Is there another logging framework that makes this possible?
I don't believe this is possible using the out of the box appenders (File, RollingFile etc) configured by a standard logback.xml file. To do what you want, you would need to dynamically create appenders on the fly and assign loggers to different appenders. Or you would need to invent a new appender that was smart enough to write to multiple files at the same time, based on the logger name.
am I just taking a bad approach for having multiple jobs that call the same main function with different parameters and wanting a log file named after each job?
The authors of logback address this issue and slightly discourage it in the section on Mapped Diagnostic Context
A possible but slightly discouraged approach to differentiate the logging output of one client from another consists of instantiating a new and separate logger for each client. This technique promotes the proliferation of loggers and may increase their management overhead. ... A lighter technique consists of uniquely stamping each log request servicing a given client.
Then they go on to discuss mapped diagnostic contexts as a solution to this problem. They give an example of a NumberCruncherServer which is crunching numbers, for various clients in various threads simultaneously. By setting the mapped diagnostic context and an appropriate logging pattern it becomes easy to determine which log events originated from which client. Then you could simply use a grep tool to separate logging events of interest into a separate file for detailed analysis.
Yes you can.
First you have to familiarize your self with these 2 concepts: Logger and Appender. Generally speaking, your code obtains a Logger, and invoke logging method such as debug(), warn(), info() etc. Logger has Appender attached to it, and Appender presents the logging information to the user according to the configuration set to it.
Once you're familiar, what you need to do is to dynamically create a FileAppender with a different file name for each different job type, and attach it to your Logger.
I suggest you spend some time with logback manual if none of above make sense.
You can make use of the logback discriminators, as discriminators' keys can be used in the <FileNamePattern> tag. I can think of two options:
Option One:
You can use the Mapped Diagnostic Context discriminator to implement your logging separation, you'll need to set a distinct value from each job using MDC.put();
Once you've done that your appender on logback configuration would look something like:
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="ch.qos.logback.classic.sift.MDCBasedDiscriminator">
<key>jobName</key> <!-- the key you used with MDC.put() -->
<defaultValue>none</defaultValue>
</discriminator>
<sift>
<appender name="jobsLogs-${jobName}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>${jobName}.%d{dd-MM-yyyy}.log.zip</FileNamePattern>
.
.
.
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>...</Pattern>
</layout>
</appender>
</sift>
</appender>
Option Two:
Implement your own discriminator -implementing ch.qos.logback.core.sift.Discriminator-, to discriminate based on the thread name. Would look something like this:
public class ThreadNameDiscriminator implements Discriminator<ILoggingEvent> {
private final String THREAD_NAME_KEY = "threadName";
#Override
public String getDiscriminatingValue(ILoggingEvent event) {
return Thread.currentThread().getName();
}
#Override
public String getKey() {
return THREAD_NAME_KEY;
}
// implementation for more methods
.
.
.
}
The logging appender would look like option one with the discriminator class being ThreadNameDiscriminator and the key being threadName. In this option there is no need to set a value to the MDC from your jobs, hence, no modification on them is required.
Related
I have developed a custom rolling file appender and it works fine. the only problem is that it's getting re-initialized each time I restart the server or change the log4j2.xml file(any part of the file) and suddenly all the previous logs would get wiped off. I haven't observed such behavior with the shipped default appenders so I wonder what can I do to retain my configurations.
<CustomAppender name="CustomAppender"
fileName="${log.file.directory}/file.log"
filePattern="${log.file.directory}/file.log.%d{yyyy-MM-dd}-%i.gz"
immediateflush="true"
append="true">
<CustomLayout/>
<Policies>
<SizeBasedTriggeringPolicy size="3 KB"/>
<TimeBasedTriggeringPolicy interval="1"/>
</Policies>
</CustomAppender>
P.S. I have tried to make it a singleton but aside from that it didn't work I really don't want to keep it away from being reconfigured, I just want to retain my previously generated logs.
update
Apparently everytime the server shutdown or log4j2.xml file is being altered the manager rebuilds the appender from scratch, though built-in appenders retain their state even after restart or reconfiguration. They do this by overriding "stop" method from AbstractOutputStreamAppender. I did the same for my appender but it still doesn't behave as I expect.
#Override
public boolean stop(long timeout, TimeUnit timeUnit) {
setStopping();
final boolean stopped = super.stop(timeout, timeUnit, false);
setStopped();
return stopped;
}
And this is the manager I used in appender builder:
final RollingFileManager manager = RollingFileManager.getFileManager(fileName, filePattern, append,
isBufferedIo, policy, strategy, advertiseUri, layout, bufferSize, isImmediateFlush(),
createOnDemand, filePermissions, fileOwner, fileGroup, getConfiguration());
if (manager == null) {
return null;
}
manager.initialize();
When a reconfiguration takes place the following happens:
The new configuration file is read and converted to a Node tree.
The Plugins associated with the Nodes are accessed and their corresponding classes are created.
If the configuration was created successfully it is started.
The Loggers are updated to reference the new configuration.
The old configuration is stopped.
As you might imagine, since there are two configuration running at the same time this can lead to problems. For example, if both components try to use the same port then the second configuration will likely fail. To avoid these problems Appenders in Log4j 2 use Managers. The Manager will typically have a "name" that includes any items that, if they have the same value in the new configuration, indicate that the Manager should be reused. This allows things like OutputStreams to remain open during the reconfiguration. However, the side effect of this is that if a parameter is changed on an appender and is not part of the "name" then the change in value may be ignored since the Manager was not modified.
So when creating a new Appender, care must be given to the name passed to the Manager and the Appender should try to update the relevant values in the Manager when they are not part of the name. Also, the Manager must account for it being started and stopped multiple times. AbstractManager will actually handle this for you.
I have a Springboot micro-service. For logging I'm using Elastic common scheme, implemented using ecs-logging-java.
I want to set the trace.ID and a transaction.ID but I'm not sure how?
Bonus question, I'm I right in thinking trace.ID should be the ID to following the request through multiple system. transaction.ID is just for within the service?
Configure your logging patter as below
<pattern> %d{yyyy-MM-dd HH:mm:ss.SSS} %thread [%X{trace-id}] [%-5level] %class{0} - %msg%n </pattern>
Put trace Id in MDC. (MDC belongs to particular thread context)
`MDC.put("trace-id", "traceid1");`
So whenever your log will print a message, it will print trace id.
Follow below artical.http://logback.qos.ch/manual/mdc.html
Step 1: Add trace id in the thread context.
This can be done using MDC (manages contextual information on a per-thread basis).
Add the below line at the start of any method, from where you want to trace logs.
MDC.put("TRACE_ID", UUID.randomUUID().toString());
Step 2: Add trace id in log format
Logs in java do not add trace id by default, so to make this possible we can add the trace id we previously added in the thread context to the log.
This can be added to the application.properties I have added [%X{TRACE_ID}] in the default log console pattern.
logging.pattern.console=%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} [%X{TRACE_ID}] %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}
I thought I had documented this but the closest I could come is in Log4j-Audit's RequestContext.. I guess I need to add a new entry to my blog. The short answer to this is that you use Log4j 2's ThreadContextMap. First, when a user logs in create a session map that contains the data you want to capture in each request, such as the user's ip address and loginId. Then create servlet Filter or Spring Interceptor to add that data as well as a unique request id to Log4j 2's Thread Context Map.
All Leg Events will include the data in the ThreadContext. The ECSLayout automatically includes all the fields in the ThreadContextMap.
Lastly, you need to propagate the RequestContext to downstream services. You do that by creating a Spring Interceptor that gets wired into the RestTemplate which converts the RequestContext fields into HTTP headers. The downstream service then has a Filter or Spring Interceptor that converts the headers back into RequestContext attributes. Log4j Audit (referenced above) has examples and implementations of all these components.
I should add that the method described above does not implement tracing as described by the WSC Trace Context spec so it is also not compatible with Elasticsearch's distributed tracing support. It is worth noting however, that if one were to include Elasticsearch's distributed tracing support along with New Relic's distributed tracing support they would step on each other.
Let's say that I have a REST API with endpoint /user/{user_id}/foo.
Now when it is called I would like that all logs that come from handling this request contain information about {user_id}. Is it possible to achieve that without passing {user_id} to every method?
I'm using SLF4j for logging, my application is based on Spring Boot.
You could also use MDC for this, see here. It's essentially a map, you just put your contextual information in it (e.g. user id) and then you can use it in your log layout. Be aware that this only works with certain underlying frameworks like logback, where a sample layout pattern would look like this:
<Pattern>%X{user_id} %m%n</Pattern>
Check the logback manual for more details on this.
You can use Logback's Mapped Diagnotic Context to propagate the {user_id} to every log message.
There are two parts to this:
Push your {user_id} into MDC e.g. MDC.put("user_id", "Pawel");
Include the MDC entry in your log statements. You do this by specifying it in your logging pattern. So, if you store the user id in a MDC entry named "user_id" the you would set logging.pattern.level=user_id:%X{user_id} %5p to include the value of that entry in every log event.
More details in the docs
How do I change a remote application's logback log level through http (rest)? to
#Controller
public class ChangeLog2Controller {
#PostMapping("/api/testlog1")
public ResponseModel testLogeBack(#RequestBody LogLevelModel levelModel) {
...
}
}
Something like this ...
#PostMapping("/api/testlog1")
public ResponseModel testLogeBack(#RequestBody LogLevelModel levelModel) {
// change the log level for the root logger
Logger rootLogger = (Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
rootLogger.setLevel(...);
// change the log level for a specific logger
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
Logger specificLogger = loggerContext.getLogger("com.some.specific.package");
specificLogger.setLevel(...);
}
... where the parameter passed into setLevel() is something like Level.INFO which would, I presume, be derived from your LogLevelModel.
Note: if you are looking for some way to dynamically change logging configuration and a JMX client (rather than a HTTP client) would suffice then Logback already provides a JMX Configurator which yuou can engage by simply adding the following to your logback.xml: <jmxConfigurator />. This exposes a JMX MBean which you can use to view and set log levels.
Edit 1: based on the comments below it seems like your requirement might be to invoke WebApplicationA and somehow enable WebApplicationA to change the log level for loggers inside WebApplicationB? If so, then either
WebApplicationA has to use the JMXConfigurator MBean exposed by WebApplicationB. There are examples of Java JMX clients here and here. However, in order to expose the JMXConfigurator you must have some control over WebApplicationB's logback.xml and, if so, then perhaps you also have some control over WebApplicationB's implementation in which case it might be easier to just expose a simple setLogLevel REST endpoint in WebApplicationB and let WebApplicationA invoke that rather than playing around with a Java JMX client.
Or
WebApplicationB has to expose a changeLogLevel REST API (with an implementation like the one I provided above) which WebApplicationA can invoke.
The only other alternative is that WebApplicationA somehow changes the logback.xml used by WebApplicationB and WebApplicationB's logback.xml starts looks like this ...
<configuration scan="true" scanPeriod="30 seconds" >
...
</configuration>
... so that any changes made to this file on WebApplicationB's classpath are picked up within (for example) 30 seconds. But, since WebApplicationB's logback.xml is likely to be embedded in a WAR (or similar) this approach seems very awkward and very likely undesireable.
We maintain an application wide system variable [debug=true|false], I want to disable all CONSOLE appenders in log4j upon startup when the system variable debug is false.
Is the best way just a programatic scan of the appenders? Or is there a more elegant approach that I don't know about perhaps?
Any primer on scanning the appenders would be welcome also if that's the right approach.
I would write a special filter for the console appender. Along the line of
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.TTCCLayout">
<param name="ConversionPattern" value="%d...m%n"/>
</layout>
<filter class="OnDebugOnly"/>
</appender>
With the Filter defined as follows
import org.apache.log4j.spi.Filter;
import org.apache.log4j.spi.LoggingEvent;
public class OnDebugOnly extends Filter {
static boolean debug;
#Override
public int decide(LoggingEvent event) {
return ( debug ? Filter.NEUTRAL : Filter.DENY ) ;
}
}
Of course this needs adjustments. Like where debug is defined and how it is accessed.
The Neutral is just in case someone adds another filter...
Plus the layout is just mine, use your preferred layout here.
Caveat. I did not test it ;-)
Have you looked into setting up a .properties file for log4j? You can configure log4j to send logging messages to a file, turn off sending log messages to the console for info, warn, etc. You can even configure it on a per class basis if needed. If you go to this site and navigate to the Configuration section, it should tell you what you need to know.
http://logging.apache.org/log4j/1.2/manual.html
Hopefully this answers your question.