In my Java application, I use SLF4J + Logback for logging. I use DEBUG level logging on development and ERROR level logging on the production environment. But there are some messages that I want to log in any case, regardless of the log level (similar to System.out.println("Some Message") but using logger).
Is there any practical way to achieve this? I can use error or lower levels, but what I want to do is to give some information, so logger.error("Some message"); is semantically wrong, it is not an error.
I can define another logger in my logback.xml for my class/classes, but it is not convenient.
What is the right way to achieve this?
This is where the wonderful "Marker" feature of SLF4J/Logback comes in very handy. Say you want to log to the console all warnings/errors, as well as anything that's a special "Status" message.
You can have a class do logging like this:
public class MyClass {
private static final Marker status = MarkerFactory.getMarker("STATUS");
private static final Logger logger = LoggerFactory.getLogger(MyClass.class);
public void doSomething() {
logger.info(status, "Beginning");
logger.info("Regular info log");
logger.warn("Warning");
logger.info(status, "Done");
}
}
Then, in logback.xml, you filter to show all warnings and messages marked "STATUS":
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
<marker>STATUS</marker>
</evaluator>
<OnMatch>ACCEPT</OnMatch>
<OnMismatch>NEUTRAL</OnMismatch>
</filter>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>WARN</level>
</filter>
<encoder>
<pattern>${pattern}</pattern>
</encoder>
</appender>
Basically, you have the logger's level be DEBUG so that everything goes to the appender, and then in the appender do further filtering to get the exact lines you're looking for. If you are trying to have more different logging levels for different classes it can get a bit trickier, but Logback gives a lot of flexibility with filters (and you can even make your own filters if needed) which lets one handle pretty much anything needed.
Related
According to the manual:
Logback collects its internal status data in a StatusManager object,
accessible via the LoggerContext.
Given a StatusManager you can access all the status data associated
with a logback context. To keep memory usage at reasonable levels, the
default StatusManager implementation stores the status messages in two
separate parts: the header part and the tail part. The header part stores
the first H status messages whereas the tail part stores the last T messages. At present time H=T=150, although these values may change in future releases.
The purpose seems to be to allow incoming log messages to be listened-to, also viewed via a servlet.
In our case we use the logstash-logback-encoder and emit our Logback messages to the console so fluentd / Filebeat etc. can ship to a remote Logstash - and so our microservices have no need to inspect or listen to them.
So the question is: can / should we override the default BasicStatusManager (with a NOOP implementation) to suppress the internal storage of these messages, or does anyone know if the Logstash encoder depends upon, or hooks into this status mechanism itself?
Our logback.xmls are pretty simple:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="lsAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<logger name="a.b.c" level="debug"/>
<root level="debug">
<appender-ref ref="lsAppender"/>
</root>
</configuration>
My second reason for asking is that we recently found ourselves getting OOMEs from microservices with a 120 MB heap. Sure, those individual log messages are too big, but that's only a problem if they're being held-onto by Logback itself rather than sent off and forgotten about. It would be one less scalability concern.
Screenshot of YourKit output:
I noticed this again recently. The problem was actually nothing related to log size, or logging throughput, and there is no need to control the size of StatusManager in any way.
The actual cause was that the Logstash Logback Encoder (version 6.4) was failing "silently" due to unencodable Markers:
import net.logstash.logback.marker.Markers;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.context.request.WebRequest;
import java.util.HashMap;
#ControllerAdvice
public class MyExceptionHandler {
private static final Logger LOG = LoggerFactory.getLogger(MyExceptionHandler.class);
public void logException(Throwable e, WebRequest request) {
var fields = new HashMap<>();
fields.put("Request", request);
LOG.error(Markers.appendEntries(fields), "Error occurred", e);
}
}
The WebRequest is an instance of org.springframework.web.context.request.ServletWebRequest which, when serialized to JSON by the encoder using jackson-databind, throws a StackOverflowError, which is caught and ignored by said encoder. The original log request is not flushed from the internal Logback cache.
Given enough errors of this type, we eventually ran out of memory.
How do I change a remote application's logback log level through http (rest)? to
#Controller
public class ChangeLog2Controller {
#PostMapping("/api/testlog1")
public ResponseModel testLogeBack(#RequestBody LogLevelModel levelModel) {
...
}
}
Something like this ...
#PostMapping("/api/testlog1")
public ResponseModel testLogeBack(#RequestBody LogLevelModel levelModel) {
// change the log level for the root logger
Logger rootLogger = (Logger) LoggerFactory.getLogger(Logger.ROOT_LOGGER_NAME);
rootLogger.setLevel(...);
// change the log level for a specific logger
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
Logger specificLogger = loggerContext.getLogger("com.some.specific.package");
specificLogger.setLevel(...);
}
... where the parameter passed into setLevel() is something like Level.INFO which would, I presume, be derived from your LogLevelModel.
Note: if you are looking for some way to dynamically change logging configuration and a JMX client (rather than a HTTP client) would suffice then Logback already provides a JMX Configurator which yuou can engage by simply adding the following to your logback.xml: <jmxConfigurator />. This exposes a JMX MBean which you can use to view and set log levels.
Edit 1: based on the comments below it seems like your requirement might be to invoke WebApplicationA and somehow enable WebApplicationA to change the log level for loggers inside WebApplicationB? If so, then either
WebApplicationA has to use the JMXConfigurator MBean exposed by WebApplicationB. There are examples of Java JMX clients here and here. However, in order to expose the JMXConfigurator you must have some control over WebApplicationB's logback.xml and, if so, then perhaps you also have some control over WebApplicationB's implementation in which case it might be easier to just expose a simple setLogLevel REST endpoint in WebApplicationB and let WebApplicationA invoke that rather than playing around with a Java JMX client.
Or
WebApplicationB has to expose a changeLogLevel REST API (with an implementation like the one I provided above) which WebApplicationA can invoke.
The only other alternative is that WebApplicationA somehow changes the logback.xml used by WebApplicationB and WebApplicationB's logback.xml starts looks like this ...
<configuration scan="true" scanPeriod="30 seconds" >
...
</configuration>
... so that any changes made to this file on WebApplicationB's classpath are picked up within (for example) 30 seconds. But, since WebApplicationB's logback.xml is likely to be embedded in a WAR (or similar) this approach seems very awkward and very likely undesireable.
I'm using logback as my logging framework and have a couple of jobs that run the same main function with different parameters and would like to create a log file for each job and name the log file with the job's name.
For example, if I had jobs a,b,c that all run MyClass.main() but with different parameters, then I'd like to see a-{date}.log, b-{date}.log, c-{date}.log.
I can achieve the {date} part by specifying a <fileNamePattern>myjob-%d{yyyy-MM-dd}.log</fileNamePattern> in my logback.xml, but I'm not sure how to (or if it is even possible) create the prefix of the file names dynamically (to be the job's name).
Is there a way to dynamically name logfiles in logback? Is there another logging framework that makes this possible?
As a follow up question, am I just taking a bad approach for having multiple jobs that call the same main function with different parameters and wanting a log file named after each job? If so is there a standard/best practice solution for this case?
EDIT: The reason why I want to name each log file after the name of the job is that each job naturally defines a "unit of work" and it is easier for me to find the appropriate log file in case one of the job fails. I could simply use a rolling log file for jobs a,b,c but I found it harder for me to look through the logs and pinpoint where each job started and ended.
I would use you own logging.
public static PrintWriter getLogerFor(String prefix) {
SimpleDatFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
String filename= prefix + sdf.format(new Date());
return new PrintWriter(filename, true); // auto flush.
}
You can write a simple LRU cache e.g. with LinkedHashMap to reuse the PrintWriters.
Is there a way to dynamically name logfiles in logback? Is there another logging framework that makes this possible?
I don't believe this is possible using the out of the box appenders (File, RollingFile etc) configured by a standard logback.xml file. To do what you want, you would need to dynamically create appenders on the fly and assign loggers to different appenders. Or you would need to invent a new appender that was smart enough to write to multiple files at the same time, based on the logger name.
am I just taking a bad approach for having multiple jobs that call the same main function with different parameters and wanting a log file named after each job?
The authors of logback address this issue and slightly discourage it in the section on Mapped Diagnostic Context
A possible but slightly discouraged approach to differentiate the logging output of one client from another consists of instantiating a new and separate logger for each client. This technique promotes the proliferation of loggers and may increase their management overhead. ... A lighter technique consists of uniquely stamping each log request servicing a given client.
Then they go on to discuss mapped diagnostic contexts as a solution to this problem. They give an example of a NumberCruncherServer which is crunching numbers, for various clients in various threads simultaneously. By setting the mapped diagnostic context and an appropriate logging pattern it becomes easy to determine which log events originated from which client. Then you could simply use a grep tool to separate logging events of interest into a separate file for detailed analysis.
Yes you can.
First you have to familiarize your self with these 2 concepts: Logger and Appender. Generally speaking, your code obtains a Logger, and invoke logging method such as debug(), warn(), info() etc. Logger has Appender attached to it, and Appender presents the logging information to the user according to the configuration set to it.
Once you're familiar, what you need to do is to dynamically create a FileAppender with a different file name for each different job type, and attach it to your Logger.
I suggest you spend some time with logback manual if none of above make sense.
You can make use of the logback discriminators, as discriminators' keys can be used in the <FileNamePattern> tag. I can think of two options:
Option One:
You can use the Mapped Diagnostic Context discriminator to implement your logging separation, you'll need to set a distinct value from each job using MDC.put();
Once you've done that your appender on logback configuration would look something like:
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="ch.qos.logback.classic.sift.MDCBasedDiscriminator">
<key>jobName</key> <!-- the key you used with MDC.put() -->
<defaultValue>none</defaultValue>
</discriminator>
<sift>
<appender name="jobsLogs-${jobName}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>${jobName}.%d{dd-MM-yyyy}.log.zip</FileNamePattern>
.
.
.
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>...</Pattern>
</layout>
</appender>
</sift>
</appender>
Option Two:
Implement your own discriminator -implementing ch.qos.logback.core.sift.Discriminator-, to discriminate based on the thread name. Would look something like this:
public class ThreadNameDiscriminator implements Discriminator<ILoggingEvent> {
private final String THREAD_NAME_KEY = "threadName";
#Override
public String getDiscriminatingValue(ILoggingEvent event) {
return Thread.currentThread().getName();
}
#Override
public String getKey() {
return THREAD_NAME_KEY;
}
// implementation for more methods
.
.
.
}
The logging appender would look like option one with the discriminator class being ThreadNameDiscriminator and the key being threadName. In this option there is no need to set a value to the MDC from your jobs, hence, no modification on them is required.
We maintain an application wide system variable [debug=true|false], I want to disable all CONSOLE appenders in log4j upon startup when the system variable debug is false.
Is the best way just a programatic scan of the appenders? Or is there a more elegant approach that I don't know about perhaps?
Any primer on scanning the appenders would be welcome also if that's the right approach.
I would write a special filter for the console appender. Along the line of
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.TTCCLayout">
<param name="ConversionPattern" value="%d...m%n"/>
</layout>
<filter class="OnDebugOnly"/>
</appender>
With the Filter defined as follows
import org.apache.log4j.spi.Filter;
import org.apache.log4j.spi.LoggingEvent;
public class OnDebugOnly extends Filter {
static boolean debug;
#Override
public int decide(LoggingEvent event) {
return ( debug ? Filter.NEUTRAL : Filter.DENY ) ;
}
}
Of course this needs adjustments. Like where debug is defined and how it is accessed.
The Neutral is just in case someone adds another filter...
Plus the layout is just mine, use your preferred layout here.
Caveat. I did not test it ;-)
Have you looked into setting up a .properties file for log4j? You can configure log4j to send logging messages to a file, turn off sending log messages to the console for info, warn, etc. You can even configure it on a per class basis if needed. If you go to this site and navigate to the Configuration section, it should tell you what you need to know.
http://logging.apache.org/log4j/1.2/manual.html
Hopefully this answers your question.
I have 2 logging files.
I have defined two appenders for the 2 files.
In a class, if i need to output few logs to one file, and few to the other, do i need to have 2 logger instances.
Is there a cleaner way of achieving this requirement?
Or is there some log4j configuration that will help me?
You can create two named loggers like the following:
log4j.logger.system=debug, sys
log4j.appender.sys=org.apache.log4j.RollingFileAppender
log4j.appender.sys.file=/logs/system.log
log4j.appender.sys.maxFileSize=1MB
log4j.appender.sys.maxBackupIndex=25
log4j.appender.sys.layout=org.apache.log4j.PatternLayout
log4j.appender.sys.layout.conversionPattern=%d{MMM dd HH:mm:ss} %-5p (%F:%L) - %m%n
which you can locate and use from code:
Logger.getLogger("system").debug("...");
If this log filtering is based on severity, you can define a different log level for each appender (DEBUG, INFO, WARNING, etc.).
If you want a different filtering, you can create a personnal log filter, and apply this filter on your appender. Look at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/spi/Filter.html