According to the manual:
Logback collects its internal status data in a StatusManager object,
accessible via the LoggerContext.
Given a StatusManager you can access all the status data associated
with a logback context. To keep memory usage at reasonable levels, the
default StatusManager implementation stores the status messages in two
separate parts: the header part and the tail part. The header part stores
the first H status messages whereas the tail part stores the last T messages. At present time H=T=150, although these values may change in future releases.
The purpose seems to be to allow incoming log messages to be listened-to, also viewed via a servlet.
In our case we use the logstash-logback-encoder and emit our Logback messages to the console so fluentd / Filebeat etc. can ship to a remote Logstash - and so our microservices have no need to inspect or listen to them.
So the question is: can / should we override the default BasicStatusManager (with a NOOP implementation) to suppress the internal storage of these messages, or does anyone know if the Logstash encoder depends upon, or hooks into this status mechanism itself?
Our logback.xmls are pretty simple:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="lsAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<logger name="a.b.c" level="debug"/>
<root level="debug">
<appender-ref ref="lsAppender"/>
</root>
</configuration>
My second reason for asking is that we recently found ourselves getting OOMEs from microservices with a 120 MB heap. Sure, those individual log messages are too big, but that's only a problem if they're being held-onto by Logback itself rather than sent off and forgotten about. It would be one less scalability concern.
Screenshot of YourKit output:
I noticed this again recently. The problem was actually nothing related to log size, or logging throughput, and there is no need to control the size of StatusManager in any way.
The actual cause was that the Logstash Logback Encoder (version 6.4) was failing "silently" due to unencodable Markers:
import net.logstash.logback.marker.Markers;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.context.request.WebRequest;
import java.util.HashMap;
#ControllerAdvice
public class MyExceptionHandler {
private static final Logger LOG = LoggerFactory.getLogger(MyExceptionHandler.class);
public void logException(Throwable e, WebRequest request) {
var fields = new HashMap<>();
fields.put("Request", request);
LOG.error(Markers.appendEntries(fields), "Error occurred", e);
}
}
The WebRequest is an instance of org.springframework.web.context.request.ServletWebRequest which, when serialized to JSON by the encoder using jackson-databind, throws a StackOverflowError, which is caught and ignored by said encoder. The original log request is not flushed from the internal Logback cache.
Given enough errors of this type, we eventually ran out of memory.
Related
I have developed a custom rolling file appender and it works fine. the only problem is that it's getting re-initialized each time I restart the server or change the log4j2.xml file(any part of the file) and suddenly all the previous logs would get wiped off. I haven't observed such behavior with the shipped default appenders so I wonder what can I do to retain my configurations.
<CustomAppender name="CustomAppender"
fileName="${log.file.directory}/file.log"
filePattern="${log.file.directory}/file.log.%d{yyyy-MM-dd}-%i.gz"
immediateflush="true"
append="true">
<CustomLayout/>
<Policies>
<SizeBasedTriggeringPolicy size="3 KB"/>
<TimeBasedTriggeringPolicy interval="1"/>
</Policies>
</CustomAppender>
P.S. I have tried to make it a singleton but aside from that it didn't work I really don't want to keep it away from being reconfigured, I just want to retain my previously generated logs.
update
Apparently everytime the server shutdown or log4j2.xml file is being altered the manager rebuilds the appender from scratch, though built-in appenders retain their state even after restart or reconfiguration. They do this by overriding "stop" method from AbstractOutputStreamAppender. I did the same for my appender but it still doesn't behave as I expect.
#Override
public boolean stop(long timeout, TimeUnit timeUnit) {
setStopping();
final boolean stopped = super.stop(timeout, timeUnit, false);
setStopped();
return stopped;
}
And this is the manager I used in appender builder:
final RollingFileManager manager = RollingFileManager.getFileManager(fileName, filePattern, append,
isBufferedIo, policy, strategy, advertiseUri, layout, bufferSize, isImmediateFlush(),
createOnDemand, filePermissions, fileOwner, fileGroup, getConfiguration());
if (manager == null) {
return null;
}
manager.initialize();
When a reconfiguration takes place the following happens:
The new configuration file is read and converted to a Node tree.
The Plugins associated with the Nodes are accessed and their corresponding classes are created.
If the configuration was created successfully it is started.
The Loggers are updated to reference the new configuration.
The old configuration is stopped.
As you might imagine, since there are two configuration running at the same time this can lead to problems. For example, if both components try to use the same port then the second configuration will likely fail. To avoid these problems Appenders in Log4j 2 use Managers. The Manager will typically have a "name" that includes any items that, if they have the same value in the new configuration, indicate that the Manager should be reused. This allows things like OutputStreams to remain open during the reconfiguration. However, the side effect of this is that if a parameter is changed on an appender and is not part of the "name" then the change in value may be ignored since the Manager was not modified.
So when creating a new Appender, care must be given to the name passed to the Manager and the Appender should try to update the relevant values in the Manager when they are not part of the name. Also, the Manager must account for it being started and stopped multiple times. AbstractManager will actually handle this for you.
I have a Springboot micro-service. For logging I'm using Elastic common scheme, implemented using ecs-logging-java.
I want to set the trace.ID and a transaction.ID but I'm not sure how?
Bonus question, I'm I right in thinking trace.ID should be the ID to following the request through multiple system. transaction.ID is just for within the service?
Configure your logging patter as below
<pattern> %d{yyyy-MM-dd HH:mm:ss.SSS} %thread [%X{trace-id}] [%-5level] %class{0} - %msg%n </pattern>
Put trace Id in MDC. (MDC belongs to particular thread context)
`MDC.put("trace-id", "traceid1");`
So whenever your log will print a message, it will print trace id.
Follow below artical.http://logback.qos.ch/manual/mdc.html
Step 1: Add trace id in the thread context.
This can be done using MDC (manages contextual information on a per-thread basis).
Add the below line at the start of any method, from where you want to trace logs.
MDC.put("TRACE_ID", UUID.randomUUID().toString());
Step 2: Add trace id in log format
Logs in java do not add trace id by default, so to make this possible we can add the trace id we previously added in the thread context to the log.
This can be added to the application.properties I have added [%X{TRACE_ID}] in the default log console pattern.
logging.pattern.console=%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} [%X{TRACE_ID}] %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}
I thought I had documented this but the closest I could come is in Log4j-Audit's RequestContext.. I guess I need to add a new entry to my blog. The short answer to this is that you use Log4j 2's ThreadContextMap. First, when a user logs in create a session map that contains the data you want to capture in each request, such as the user's ip address and loginId. Then create servlet Filter or Spring Interceptor to add that data as well as a unique request id to Log4j 2's Thread Context Map.
All Leg Events will include the data in the ThreadContext. The ECSLayout automatically includes all the fields in the ThreadContextMap.
Lastly, you need to propagate the RequestContext to downstream services. You do that by creating a Spring Interceptor that gets wired into the RestTemplate which converts the RequestContext fields into HTTP headers. The downstream service then has a Filter or Spring Interceptor that converts the headers back into RequestContext attributes. Log4j Audit (referenced above) has examples and implementations of all these components.
I should add that the method described above does not implement tracing as described by the WSC Trace Context spec so it is also not compatible with Elasticsearch's distributed tracing support. It is worth noting however, that if one were to include Elasticsearch's distributed tracing support along with New Relic's distributed tracing support they would step on each other.
I have a class that extends LayoutBase. In the doLayout method, I add a key to my logging called MSG and the value is set to ILoggingEvent event.getMessage().
I'm seeing values added to my logging but they're not consistent; some logging messages and some exception stack traces.
Can anyone tell me where ILoggingEvent event.getMessage() gets its value from?
Please refer to Logback architecture
It mentions that each log will go through series of step
Get Filter Chain Decision
If it exists, the TurboFilter chain is invoked. Turbo filters can set a context-wide threshold, or filter out certain events based on information such as Marker, Level, Logger, message, or the Throwable that are associated with each logging request.
Apply the selection rule
At this step, logback compares the effective level of the logger with the level of the request. If the logging request is disabled according to this test, then logback will drop the request without further processing.
Create a LoggingEvent object
If the request survived the previous filters, logback will create a LoggingEvent object containing all the relevant parameters of the request
Invoking appenders
After the creation of a LoggingEvent object, logback will invoke the doAppend() methods of all the applicable appenders, that is, the appenders inherited from the logger context.
Formatting the output
It is the responsibility of the invoked appender to format the logging event. However, some (but not all) appenders delegate the task of formatting the logging event to a layout.
Sending out the LoggingEvent
After the logging event is fully formatted it is sent to its destination by each appender.
I'm seeing values added to my logging but they're not consistent; some logging messages and some exception stack traces.
Now, coming to your query, it seems that you are receiving events which have exception information (Throwable) during the formatting step.
You can create a CustomFilter to filter out such events. All you have to do is extends Filter<ILoggingEvent>
public class DenyExceptionFilter extends Filter<ILoggingEvent> {
#Override
public FilterReply decide(ILoggingEvent iLoggingEvent) {
final IThrowableProxy throwableProxy = iLoggingEvent.getThrowableProxy();
if (throwableProxy != null && throwableProxy instanceof ThrowableProxy)
return FilterReply.DENY;
return FilterReply.ACCEPT;
}
}
This can be much more powerful, can filter specific types of exception. You can take this as your homework :P
Then you can add this Custom Filter to your appender as
<appender name="APPENDER_NAME" class="ch.qos.logback.classic.AsyncAppender">
<filter class="com.stackoverflow.DenyExceptionFilter" />
</appender>
Of course, add your layout as well after the filter.
I'm using logback as my logging framework and have a couple of jobs that run the same main function with different parameters and would like to create a log file for each job and name the log file with the job's name.
For example, if I had jobs a,b,c that all run MyClass.main() but with different parameters, then I'd like to see a-{date}.log, b-{date}.log, c-{date}.log.
I can achieve the {date} part by specifying a <fileNamePattern>myjob-%d{yyyy-MM-dd}.log</fileNamePattern> in my logback.xml, but I'm not sure how to (or if it is even possible) create the prefix of the file names dynamically (to be the job's name).
Is there a way to dynamically name logfiles in logback? Is there another logging framework that makes this possible?
As a follow up question, am I just taking a bad approach for having multiple jobs that call the same main function with different parameters and wanting a log file named after each job? If so is there a standard/best practice solution for this case?
EDIT: The reason why I want to name each log file after the name of the job is that each job naturally defines a "unit of work" and it is easier for me to find the appropriate log file in case one of the job fails. I could simply use a rolling log file for jobs a,b,c but I found it harder for me to look through the logs and pinpoint where each job started and ended.
I would use you own logging.
public static PrintWriter getLogerFor(String prefix) {
SimpleDatFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
String filename= prefix + sdf.format(new Date());
return new PrintWriter(filename, true); // auto flush.
}
You can write a simple LRU cache e.g. with LinkedHashMap to reuse the PrintWriters.
Is there a way to dynamically name logfiles in logback? Is there another logging framework that makes this possible?
I don't believe this is possible using the out of the box appenders (File, RollingFile etc) configured by a standard logback.xml file. To do what you want, you would need to dynamically create appenders on the fly and assign loggers to different appenders. Or you would need to invent a new appender that was smart enough to write to multiple files at the same time, based on the logger name.
am I just taking a bad approach for having multiple jobs that call the same main function with different parameters and wanting a log file named after each job?
The authors of logback address this issue and slightly discourage it in the section on Mapped Diagnostic Context
A possible but slightly discouraged approach to differentiate the logging output of one client from another consists of instantiating a new and separate logger for each client. This technique promotes the proliferation of loggers and may increase their management overhead. ... A lighter technique consists of uniquely stamping each log request servicing a given client.
Then they go on to discuss mapped diagnostic contexts as a solution to this problem. They give an example of a NumberCruncherServer which is crunching numbers, for various clients in various threads simultaneously. By setting the mapped diagnostic context and an appropriate logging pattern it becomes easy to determine which log events originated from which client. Then you could simply use a grep tool to separate logging events of interest into a separate file for detailed analysis.
Yes you can.
First you have to familiarize your self with these 2 concepts: Logger and Appender. Generally speaking, your code obtains a Logger, and invoke logging method such as debug(), warn(), info() etc. Logger has Appender attached to it, and Appender presents the logging information to the user according to the configuration set to it.
Once you're familiar, what you need to do is to dynamically create a FileAppender with a different file name for each different job type, and attach it to your Logger.
I suggest you spend some time with logback manual if none of above make sense.
You can make use of the logback discriminators, as discriminators' keys can be used in the <FileNamePattern> tag. I can think of two options:
Option One:
You can use the Mapped Diagnostic Context discriminator to implement your logging separation, you'll need to set a distinct value from each job using MDC.put();
Once you've done that your appender on logback configuration would look something like:
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="ch.qos.logback.classic.sift.MDCBasedDiscriminator">
<key>jobName</key> <!-- the key you used with MDC.put() -->
<defaultValue>none</defaultValue>
</discriminator>
<sift>
<appender name="jobsLogs-${jobName}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>${jobName}.%d{dd-MM-yyyy}.log.zip</FileNamePattern>
.
.
.
</rollingPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>...</Pattern>
</layout>
</appender>
</sift>
</appender>
Option Two:
Implement your own discriminator -implementing ch.qos.logback.core.sift.Discriminator-, to discriminate based on the thread name. Would look something like this:
public class ThreadNameDiscriminator implements Discriminator<ILoggingEvent> {
private final String THREAD_NAME_KEY = "threadName";
#Override
public String getDiscriminatingValue(ILoggingEvent event) {
return Thread.currentThread().getName();
}
#Override
public String getKey() {
return THREAD_NAME_KEY;
}
// implementation for more methods
.
.
.
}
The logging appender would look like option one with the discriminator class being ThreadNameDiscriminator and the key being threadName. In this option there is no need to set a value to the MDC from your jobs, hence, no modification on them is required.
We maintain an application wide system variable [debug=true|false], I want to disable all CONSOLE appenders in log4j upon startup when the system variable debug is false.
Is the best way just a programatic scan of the appenders? Or is there a more elegant approach that I don't know about perhaps?
Any primer on scanning the appenders would be welcome also if that's the right approach.
I would write a special filter for the console appender. Along the line of
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.TTCCLayout">
<param name="ConversionPattern" value="%d...m%n"/>
</layout>
<filter class="OnDebugOnly"/>
</appender>
With the Filter defined as follows
import org.apache.log4j.spi.Filter;
import org.apache.log4j.spi.LoggingEvent;
public class OnDebugOnly extends Filter {
static boolean debug;
#Override
public int decide(LoggingEvent event) {
return ( debug ? Filter.NEUTRAL : Filter.DENY ) ;
}
}
Of course this needs adjustments. Like where debug is defined and how it is accessed.
The Neutral is just in case someone adds another filter...
Plus the layout is just mine, use your preferred layout here.
Caveat. I did not test it ;-)
Have you looked into setting up a .properties file for log4j? You can configure log4j to send logging messages to a file, turn off sending log messages to the console for info, warn, etc. You can even configure it on a per class basis if needed. If you go to this site and navigate to the Configuration section, it should tell you what you need to know.
http://logging.apache.org/log4j/1.2/manual.html
Hopefully this answers your question.