Develop Reference

Apache Flink - Can not create Hourly/Daily Log File with Log4j

I can not create daily and also hourly log file (especially for taskexecutor log) with log4j
Here is the my log4j.properties
# This affects logging for both user code and Flink
log4j.rootLogger=INFO, file
# Uncomment this if you want to _only_ change Flink's logging
#log4j.logger.org.apache.flink=INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.File=${log.file}
log4j.appender.file.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
${log.file} points to :
2020-02-13 17:40:51,105 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog.file=/.../flink-1.9.1/log/flink-tarantula-taskexecutor-0-...log
In the meantime there is a logback.xml configuration also(which one flink uses ? )
logback.xml file :
<configuration>
<appender name="file" class="ch.qos.logback.core.FileAppender">
<file>${log.file}</file>
<append>false</append>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern>
</encoder>
</appender>
<!-- This affects logging for both user code and Flink -->
<root level="INFO">
<appender-ref ref="file"/>
</root>
<!-- Uncomment this if you want to only change Flink's logging -->
<!--<logger name="org.apache.flink" level="INFO">-->
<!--<appender-ref ref="file"/>-->
<!--</logger>-->
<!-- The following lines keep the log level of common libraries/connectors on
log level INFO. The root logger does not override this. You have to manually
change the log levels here. -->
<logger name="akka" level="INFO">
<appender-ref ref="file"/>
</logger>
<logger name="org.apache.kafka" level="INFO">
<appender-ref ref="file"/>
</logger>
<logger name="org.apache.hadoop" level="INFO">
<appender-ref ref="file"/>
</logger>
<logger name="org.apache.zookeeper" level="INFO">
<appender-ref ref="file"/>
</logger>
<!-- Suppress the irrelevant (wrong) warnings from the Netty channel handler -->
<logger name="org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline" level="ERROR">
<appender-ref ref="file"/>
</logger>
</configuration>
here is the flink/lib folder
apache-log4j-extras-1.2.17.jar flink-dist_2.11-1.9.1.jar flink-table_2.11-1.9.1.jar flink-table-blink_2.11-1.9.1.jar log4j-1.2.17.jar slf4j-log4j12-1.7.15.jar
Solution Finally
After all tried, I found the solution, I could not create daily log file because of this statement was blocking creating daily file(s).
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, filelog4j.rootLogger=INFO, file
Example of creating logs at midday and midnight of each day.
log4j.category.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, nettyFileAppender
log4j.rootLogger=INFO, file
# Uncomment this if you want to _only_ change Flink's logging
#log4j.logger.org.apache.flink=INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.DatePattern='.'yyyy-MM-dd-a
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
log4j.appender.nettyFileAppender=org.apache.log4j.FileAppender
log4j.appender.nettyFileAppender.file=/path/to/nettyLog/nettyChannelIrrelevant.log
log4j.appender.nettyFileAppender.append=false
log4j.appender.nettyFileAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.nettyFileAppender.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

You don't specify how you provide your log4j.properties file. But a common issue is that someone has this file in their jar, and it gets ignored, because Flink will use the conf/log4j.properties file located on the cluster.
And assuming your jar doesn't include anything other than classes from the slf4j-api jar, then Flink will pick up the slf4j-log4j12.jar in flink/lib (since that's on the classpath), and thus use log4j (not logback), so the logback.xml configuration file is ignored.

How to select Logback appender based on property file or environment variable

I have configured logback xml for a spring boot project.
I want to configure another appender based on the property configured. We want to create an appender either for JSON logs or for text log, this will be decided either by property file or by environment variable.
So I am thinking about the best approach to do this.
Using filters to print logs to 1 of the file (either to JSON or to Txt). But this will create both of the appenders. I want to create only 1 appender.
Use "If else" blocks in logback XML file. To put if else around appenders, loggers seems untidy and error prone. So will try to avoid as much as possible.
So now exploring options where I can add appender at runtime.
So I want to know if it is possible to add appender at runtime. And will it be added before spring boots up or it could be done anytime in the project.
What could be the best approach to include this scenario.

As you're already using Spring, I suggest using Spring Profiles, lot cleaner than trying to do the same programmatically. This approach is also outlined in Spring Boot docs.
You can set an active profile from either property file:
spring.profiles.active=jsonlogs
or from environment value:
spring_profiles_active=jsonlogs
of from startup parameter:
-Dspring.profiles.active=jsonlogs
Then have separate configurations per profile:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout-classic" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{dd-MM-yyyy HH:mm:ss.SSS} %magenta([%thread]) %highlight(%-5level) %logger{36}.%M - %msg%n</pattern>
</encoder>
</appender>
<appender name="stdout-json" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
<timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
</layout>
</encoder>
</appender>
<!-- begin profile-specific stuff -->
<springProfile name="jsonlogs">
<root level="info">
<appender-ref ref="stdout-json" />
</root>
</springProfile>
<springProfile name="classiclogs">
<root level="info">
<appender-ref ref="stdout-classic" />
</root>
</springProfile>
</configuration>

As the previous answer states, you can set different appenders based on Spring Profiles.
However, if you do not want to rely on that feature, you can use environments variables as described in the Logback manual. I.e.:
<appender name="json" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
<appendLineSeparator>true</appendLineSeparator>
</layout>
</encoder>
</appender>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>
%cyan(%d{HH:mm:ss.SSS}) %gray([%thread]) %highlight(%-5level) %magenta(%logger{36}) - %msg%n
</pattern>
</encoder>
</appender>
<root level="info">
<!--
! Use the content of the LOGBACK_APPENDER environment variable falling back
! to 'json' if it is not defined
-->
<appender-ref ref="${LOGBACK_APPENDER:-json}"/>
</root>

Logback Not Sending Email (for Level less than ERROR)

I am using slf4j and logback for logging in my java web application. I need the info logs from a specific class (MyClass in the example below) to be sent in an email. I configured an email appender in logback. The rest of it can go wherever the root logger is set to. But the email doesn't go out with my current set up. See below...
Set Up:
Here's the relevant information about jar versions and other setup for this:
jars in the classpath:
activation-1.1.jar
jcl-over-slf4j-1.7.25.jar
logback-classic-1.2.3.jar
logback-core-1.2.3.jar
logback-ext-spring-0.1.4.jar
logstash-logback-encoder-4.11.jar
mail-1.4.jar
slf4j-api-1.7.25.jar
logback.xml from classpath:
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d [%thread] %-5level %logger{5}:%L - %msg%n</pattern>
</encoder>
</appender>
<appender name="email" class="ch.qos.logback.classic.net.SMTPAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>info</level>
</filter>
<smtpHost>smtp.server</smtpHost>
<to>code4kix#email.com</to>
<from>do-not-reply#email.com</from>
<subject>code4kix - ${HOSTNAME}: %logger{20} - %m</subject>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d [%thread] %-5level %logger{5}:%L - %msg%n</pattern>
</layout>
<!-- <STARTTLS>true</STARTTLS> -->
<!-- <cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker"> -->
<!-- <bufferSize>1</bufferSize> -->
<!-- </cyclicBufferTracker> -->
<!-- <asynchronousSending>false</asynchronousSending> -->
</appender>
<root level="error">
<appender-ref ref="console" />
</root>
<logger name="mypackage.MyClass" level="info" additivity="true">
<appender-ref ref="email"/>
</logger>
The Issue:
The email seems to go out fine if I have logger.error statements in MyClass.java, but if they have just logger.info, the email doesn't go out... despite configuring the threshold to info!
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MyClass
{
private static Logger logger = LoggerFactory.getLogger(MyClass.class);
public void myMethod()
{
logger.error("using this sends the email out");
logger.info("using this doesn't");
}
}
I do get this in the console log, but the email never goes out. What could possibly be wrong?
SystemOut O 13:18:00,012 |-INFO in ch.qos.logback.classic.net.SMTPAppender[email] - SMTPAppender [email] is tracking [1] buffers

From my recent reading of the logback it sounds as though only under ERROR will an email be sent out. Chapter 4: Appenders

Thanks to Elijah for pointing me in the right direction. As mentioned in the documentation in SMTP Appender:
by default, the email transmission is triggered by a logging event of
level ERROR
There are multiple ways to solve this. The simplest way for me was to implement my own EventEvaluator like so, and use it in the smtp appender config (shown further below). Alternatively, you can explore using Markers to solve this issue.
public class OnInfoEvaluator extends EventEvaluatorBase<ILoggingEvent> {
#Override
public boolean evaluate(ILoggingEvent loggingEvent) throws NullPointerException, EvaluationException {
if(loggingEvent.getLevel().toInt() >= Level.INFO_INT)
{
return true;
}
else
{
return false;
}
}
}
In the config:
<appender name="email" class="ch.qos.logback.classic.net.SMTPAppender">
<evaluator class="mypackage.OnInfoEvaluator" />
...
</appender>

Application with log4j using log file of another application with logback on weblogic server

In my weblogic server I have two independent application. App1 is old web application with web services using log4j:
# Console logger
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout.ConversionPattern=%-4r %-5p %c %x - %m%n
log4j.appender.CONSOLE.Threshold=ERROR
# APP logger
log4j.appender.APP=org.apache.log4j.DailyRollingFileAppender
log4j.appender.APP.DatePattern=.yyyy-MM-dd-HH
log4j.appender.APP.File=${logfile.app}
log4j.appender.APP.encoding=UTF-8
log4j.appender.APP.layout=org.apache.log4j.PatternLayout
log4j.appender.APP.layout.ConversionPattern=%d{DATE} %-5p [%t] %-15c : %m%n
log4j.rootLogger=ERROR, CONSOLE
log4j.logger.cz.isvs=INFO, APP
log4j.logger.org.springframework=WARN, APP
${logfile.app} is replaced during build to target/log/app1.log
When there is just this application deployed everything is working OK.
Second app2 is spring boot with also web services application using logback:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProfile name="default">
<property name="LOGS_HOME" value="/app/app_logs/app2" />
<appender name="appfile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOGS_HOME}.log</file>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
</Pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOGS_HOME}.log.%d{yyyy-MM-dd-HH}</fileNamePattern>
</rollingPolicy>
</appender>
<root level="error">
<appender-ref ref="appfile" />
</root>
<logger name="cz.isvs" additivity="false">
<level value="debug" />
<appender-ref ref="appfile" />
</logger>
</springProfile>
</configuration>
When I deployed second application to my weblogic server then both application is logging into app2.log. I am really confused why and how can this happened

Recently I was facing a similar issue with spring boot deployed in weblogic 12c. After investigation found that spring boot is using spring boot logger starter which has some dependency jar i.e. sl4fj, logback, jcl-over-sl4j, jul-to-slf4j, log4j-over-slf4j.jar etc.
From slf4j manual found this:
The implementation of JCL over SLF4J, i.e jcl-over-slf4j.jar, will
allow your project to migrate to SLF4J piecemeal, without breaking
compatibility with existing software using JCL. Similarly,
log4j-over-slf4j.jar and jul-to-slf4j modules will allow you to
redirect log4j and respectively java.util.logging calls to SLF4J.
The bridging jars are responsible to redirect log4j logs, java util logs, commons logging logs to logback via slf4j. After excluding the bridging jar dependencies got the expected result.

Disable highly verbose logging

I am getting this highly verbose logging and I want to get rid of it.
1:39:20.187 [main] INFO c.v.c.c.ConfigManager - End XML Read
11:39:20.187 [main] INFO c.v.c.c.ConfigManager - The content of ConfigCfg_pdMetering.xml is ConfigCfg [xxx=yyy]
11:39:37.335 [Thread-1] DEBUG o.a.h.c.p.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
11:39:37.335 [Thread-1] DEBUG o.a.h.i.c.DefaultHttpClient - Attempt 1 to execute request
The logging comes from a jar I have imported in the project. I modified the logback.xml and log4j.properties in the jar using "jar uf"
logback.xml now looks like:
<?xml version="1.0"?>
<configuration debug="false">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are by default assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder -->
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
</pattern>
</encoder>
</appender>
<logger name="com.hccl" level="ERROR"/>
<logger name="org.apache.http.wire" level="ERROR"/>
<root level="debug">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
and log4j.properties:
log4j.rootCategory=INFO, stdout
log4j.logger.org.springframework.ws.client.MessageTracing.sent=ERROR
log4j.logger.org.springframework.ws.client.MessageTracing.received=ERROR
log4j.logger.org.springframework.ws.server.MessageTracing=ERROR
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%p [%c{3}] %m%n
log4j.logger.httpclient.wire=ERROR
I imported this modified jar and imported it to my project. This has not affected the logging.
Also I am getting the following in the console:
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
I do not know why I am seeing this warning.

Muting Apache http
The packages you want to mute are c.v.c.c. and o.a.h. Those are abbreviations. Also, you specified <logger name="org.apache.http.wire" level="ERROR"/> which doesn't match the above (take the first letters of each package name).
You might have more luck with
<logger name="org.apache.http" level="ERROR"/>
to shut down all org apache http logging (which is probably o.a.h.*).
Muting c.v.c.c
To mute the first two lines beginning with c.v.c.c.ConfigManager, you need to find out the classes. You can do this either by replacing %logger{36} with just %logger to have the full package and class name printed. Or perhaps it's your own class. In this case you can just putt a corresponding logger entry into your file by yourself.
Warning for slf4j-Binding
If slf4j outputs warnings about "actual bindings", you have probably more than one slf4j binding in your classpath. As I read from your question, you try to use both logback and log4j, which won't work - slf4j decides to use logback only. Remove log4j and the log4j.properties. If they get included by maven dependencies, specify exclude tags for those dependencies.