Context: Java web application running in Tomcat, Log4j 2 version 2.5. Logging set up to go to one Sentry instance using the Raven connector and one ELK (ElasticSearch + Logstash + Kibana) instance.
Configuration looks like this (anonymized):
<configuration>
<appenders>
<Raven name="Sentry">
<dsn>https://foo#bar.baz/1</dsn>
</Raven>
<Socket name="Logstash" host="1.2.3.4" port="1234">
<SerializedLayout />
</Socket>
</appenders>
<loggers>
<root level="info">
<appender-ref ref="Sentry" level="warn" />
<appender-ref ref="Logstash" />
</root>
</loggers>
</configuration>
Problem is that if Sentry or ELK starts acting up (long response times, unreachable, ...), the application threads get blocked trying to log and the application effectively dies.
Is there a preferred solution to this problem? Some kind of circuit breaker would be ideal here: after detecting a problem with the remote logging destination Log4j 2 would disable the appender for some time.
We use another appender wrapped in an <Async/> appender, but as far as I understand that is a good solution to make the logging... well, asynchronous, but not to fix the problem of unresponsive log destination.
In our system we have a special component called logs-forwarder that collects all the logs by log4j and other systems and forwards them to logstash / splunk etc. what might have you. So that's one valid option.
Related
How can I use Logback to capture System.out messages in a Java program?
For example, I would like to use this code:
System.out.println("test: console out to file instead");
... and capture it's output to a file.
Can this be done using the logback.xml config file?
There's a little jar that does this for you: http://projects.lidalia.org.uk/sysout-over-slf4j/index.html
Please do read the FAQ: http://projects.lidalia.org.uk/sysout-over-slf4j/faq.html - If you use this module and configure logback to output to stdout, you'll get a stream of StackOverFlowErrors :D
If you want to do this yourself, you need to replace System.out and System.err with a custom PrintWriter (?I cannot fully remember the class name). This custom PrintWriter needs to forward the calls to logback in a semi intelligent manner, as print writters can also print character by character rather than by line.
The sysout-over-slf4j solutions answers the question 100%.
Just take note: If you setup the Java Util Logger -> SLF4j bridge (jul-to-slf4j), all the System.out.println messages will also be in you logback log file, along with all the messages printed to the Java util logger.
Here is how to set it up:
https://stackoverflow.com/a/43242620/2359435
Yes you can do it with some configuration similar to..
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<target>System.out</target>
<filter class="com.foo.StdOutFilter" />
...
</appender>
<logger name="mylogger" level="debug">
<appender-ref ref="stdout" />
</logger>
Let's say my application consists of several components which are logged separately:
myapp.log
myapp-foo.log
myapp-bar.log
My classes all define a "standard" java logger:
private static final Logger logger = LogManager.getLogger(ThisClass.class);
Logging for components "foo" and "bar" is set up via package-based configuration (each of which logs to it's own logfile) :
<Logger name="com.mycompany.myapp.foo" level="info">
<AppenderRef ref="foo"/>
</Logger>
<Logger name="com.mycompany.myapp.bar" level="info">
<AppenderRef ref="bar"/>
</Logger>
This works fine, up to the point where we want to see log messages from a framework we use, such as spring, since spring log statements end up in myapp.log - not myapp-foo.log or myapp-bar.log.
It's convenient to have one log4j.xml or log4j2.xml file, but can we use package-based log configuration in this way - and still have spring log statements logged to the "correct" log file, myapp-foo.log of myapp-bar.log?
Or is there a different solution?
Edit: we are using spring 4.0.3.RELEASE and log4j2
I have migrated from log4j 1.2.17 to log4j 2.2. and I'm getting the following issues:
My AsyncLogger is working like a sync one. This is my config for this logger:
<RollingFile name="ACCESS_LOG"
fileName="${sys:log.dir}vproxy_access.${date:yyyy-MM-dd}"
filePattern="${sys:log.dir}vproxy_access.${date:yyyy-MM-dd}"
append="true"
bufferedIO="true"
bufferSize="8192"
immediateFlush="false">
<PatternLayout>
<Pattern>%m%d{yyyy-MM-dd HH:mm:ss}%n
</PatternLayout>
<Policies/>
</RollingFile>
<AsyncLogger name="LOGGER_ACCESS" level="info" includeLocation="true" additivity="false">
<AppenderRef ref="ACCESS_LOG"/>
</AsyncLogger>
The log creation is correct but everything I log to this file is not buffered and the lines are committed immediately. I've been compared my config with others I've seen in other posts and I can't see what's wrong.
When I shutdown my application, now I'm getting the following error:
Exception in thread "pool-1-thread-1" java.lang.NoClassDefFoundError: org/apache/logging/log4j/message/ParameterizedMessage
at org.apache.logging.log4j.message.ParameterizedMessageFactory.newMessage(ParameterizedMessageFactory.java:47)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:737)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:708)
at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:314)
at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry.run(DefaultShutdownCallbackRegistry.java:77)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.message.ParameterizedMessage
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
... 6 more
When this error appears I'm not logging anything. The last line I log is shown correctly in its corresponding log.
When logging to a file with a sync logger, strange characters occasionally appear. See below "^[[?1;2c^[[?1;2c" (this didn't happen with 1.2.17).
2015-04-02 13:58:51 Starting messaging service ...
^[[?1;2c^[[?1;2c2015-04-02 13:59:06 Messaging service successfully started.
2015-04-02 13:59:06 Starting balancer test port ...
2015-04-02 13:59:06 Balancer test port successfully started.
Thanks in advance.
Joan.
This is the expected behaviour. The background thread is simply keeping up with the log events your application is putting in the queue. Note that with log4j2, async logging will flush the buffer if the queue is empty, so results are immediately visible on disk. (See the docs for FileAppender immediateFlush.)
You may need to add the log4j-web module to your deployment. This is necessary to clean up log4j resources in a web application, and will also disable the shutdown hook. See https://logging.apache.org/log4j/2.x/manual/webapp.html
This one is trickier... Can you provide your full log4j2 config? Any chance there are 2 processes writing to the same file?
I use JMX client to change logger level in log4j programatically. It works fine, but how can specify logger level for non-registered (that aren't specified in log4j2.xml) loggers?
For instance I have the following loggers block:
<Loggers>
<Root level="WARN">
<AppenderRef ref="async"/>
</Root>
<Logger name="com.example" level="INFO" />
<Logger name="com.example.java" level="INFO" />
</Loggers>
I have "com.example" and "com.example.java" loggers available for using via JMX. But what if I need specify logger level for "com.example.java.Runner" or for "com.example.groovy"?
At the moment (version 2.1) this is not possible. JMX only instruments Loggers that are in the configuration.
Without JMX, you can do something like mentioned here:
Programmatically change log level in Log4j2
But this also only works for loggers that are named in the configuration...
Have you tried remotely editing the Log4j configuration as outlined in the Client GUI section here?
Clicking the "Reconfigure with XML below" button will send the configuration text to the remote application where it will be used to reconfigure Log4j on the fly. This will not overwrite any configuration file. Reconfiguring with text from the editor happens in memory only and the text is not permanently stored anywhere.
How do I filter log messages from external third party frameworks? I am using Hibernate and Spring framework and I would like to suppress the logs so that only my log4j logs appears.
In my log4j.properties file I set the root logger logging level to ERROR. Then for packages I specifically want to log, like my application code, I set the logging level to INFO or DEBUG.
log4j.rootLogger=ERROR, stdout
log4j.logger.com.initech.tps=DEBUG
log4j.logger.org.hibernate.SQL=INFO
I see co-workers who set root logging low and then end up listing everything they don't want to see, that just seems backward to me. I would rather list what I want to log than all the things I don't want to log.
BTW turning logging off entirely for a third-party component seems like a bad idea to me. For instance, Spring is relatively noisy and uses WARN for things I really don't need to know about, but if it logs an ERROR entry for something I want to see it.
You can do it by changing logger level in log4j.properties/log4j.xml file.
You need to set logger's <level value="off"/> if you want to filter logs from package but keep logger configuration for later use.
You could also set it to highest level to log only in case of error or fatal issue.
Following entries should be added to log4j.xml to turn off logging from hibernate and springframework packages:
<logger name="org.springframework">
<level value="off"/>
</logger>
<logger name="org.hibernate">
<level value="off"/>
</logger>
In log4j.properties you can define individual levels on a per logger basis:
log4j.logger.<name>=FATAL
In log4j.xml the syntax is
<logger name="<name>">
<level value="fatal"/>
</logger>
<name> is often the full qualified classname. You might want to use WARN or ERROR instead of FATAL
Just don't add those packages in your log4j.properties. For instance, you must have this for Spring in your properties file. Take it out if you have some entries like below (anything that starts with org.springframework). Same needs to be done for hibernate.
#Logs the SQL from Spring
log4j.logger.org.springframework.jdbc.core.JdbcTemplate=ERROR
#Logs the SQL parameters from Spring
log4j.logger.org.springframework.jdbc.core.StatementCreatorUtils=ERROR
Also as it was mentioned you should also set the following. I forgot to mention this.
log4j.rootLogger=FATAL
or
log4j.rootLogger=ERROR