Develop Reference

How to include mdc key when logging exception using Logstash encoder

I have logback-spring.xml file where I have json console appender:
I include mdc key requestId:
however, this requestId is lost when there is a runtime exception being logged.
is there a way I can have the requestId printed in the logs when there is an exception?
<configuration>
<appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<fieldNames>
<timestamp>log_datetime</timestamp>
<thread>thread</thread>
<logger>[ignore]</logger>
<version>[ignore]</version>
<levelValue>[ignore]</levelValue>
<callerClass>class_name</callerClass>
<callerMethod>method_name</callerMethod>
<callerFile>[ignore]</callerFile>
<callerLine>line_num</callerLine>
</fieldNames>
<includeMdcKeyName>requestId</includeMdcKeyName>
<timestampPattern>yyyy-MM-dd HH:mm:ss.SSS</timestampPattern>
<includeCallerData>true</includeCallerData>
<throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
<maxDepthPerThrowable>30</maxDepthPerThrowable>
<maxLength>2048</maxLength>
<shortenedClassNameLength>20</shortenedClassNameLength>
<exclude>sun\.reflect\..*\.invoke.*</exclude>
<exclude>net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
<rootCauseFirst>true</rootCauseFirst>
<inlineHash>true</inlineHash>
</throwableConverter>
</encoder>
</appender>
<springProfile name="!local">
<root level="INFO">
<appender-ref ref="jsonConsoleAppender"/>
</root>
</springProfile>
</configuration>

Logging using Logback on Spark StandAlone

We are using Spark StandAlone 2.3.2 and logback-core/logback-classic with 1.2.3
Have very simple Logback configuration file which allows us to log the data to a specific directory and on local I can pass the vm parameters from editor
-Dlogback.configurationFile="C:\path\logback-local.xml"
and it works and logs properly
On Spark StandAlone I am trying to pass the arguments using external link
spark-submit
--master spark://127.0.0.1:7077
--driver-java-options "-Dlog4j.configuration=file:/path/logback.xml"
--conf "spark.executor.extraJavaOptions=-Dlogback.configurationFile=file:/path/logback.xml"
Here is the config file (bit ansibilized), have verified the actual paths and they exist, any idea what could be the issue on the cluster. I have verified the Environment variables on Spark UI and they reflect the same for drvier and executor options.
Any potential issues with Logback and Spark StandAlone together?
There is nothing specific to configuration file here, it just filters the data for json logging vs file for better visualization on log server
<configuration>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>{{ app_log_file_path }}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!--daily-->
<fileNamePattern>{{ app_log_dir }}/{{ app_name }}.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<maxHistory>90</maxHistory>
<totalSizeCap>10GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%d [%thread] %-5level %logger{36} %X{user} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE_JSON" class="ch.qos.logback.core.rolling.RollingFileAppender">
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator>
<expression>
return message.contains("timeStamp") &&
message.contains("logLevel") &&
message.contains("sourceLocation") &&
message.contains("exception");
</expression>
</evaluator>
<OnMismatch>DENY</OnMismatch>
<OnMatch>NEUTRAL</OnMatch>
</filter>
<file>{{ app_json_log_file_path }}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!--daily-->
<fileNamePattern>{{ app_log_dir }}/{{ app_name }}_json.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<maxHistory>90</maxHistory>
<totalSizeCap>10GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%msg%n</pattern>
</encoder>
</appender>
<logger name="com.baml.ctrltech.greensheet.logging.GSJsonLogging" level="info" additivity="false">
<appender-ref ref="FILE_JSON" />
</logger>
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="FILE_JSON"/>
</root>
</configuration>

We couldn't get Logback to work with Spark, as Spark uses Log4J internally, we had to switch to the same

I fixed adding one dependency for logback and excluding the transitive dependency from Spark in sbt:
val sparkV = "3.3.1"
val slf4jLogbackV = "2.1.4"
val slf4jLogback = "com.networknt" % "slf4j-logback" % slf4jLogbackV
val sparkSql = ("org.apache.spark" %% "spark-sql" % sparkV)
.cross(CrossVersion.for3Use2_13)
.exclude("org.apache.logging.log4j", "log4j-slf4j-impl")

Ways to configure logs other than log4j xml, properties file, and source code (mainly java)?

I'm looking at the source code of some application. It is using Spring framework, Apache Tiles, JSP, Log4j, java, javascript, jquery, jqplot, Jsch, and etc.
I know where the logs are created. (a/b/logs) However, when I look at source code, I don't know how logs are created under the folder name 'logs'. I looked at log4j.xml, web.xml , property files. I found the code for how the path 'a/b' is created, but not logs. Also that folder has 4 types of logs. And they are name in a pattern like access.20181227001 , errors.20182111. I want to know where I have to look to find how the logs are created in this manner.
Log4J.xml
<!-- Appenders -->
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d %5p [%c] %m%n" />
</layout>
</appender>
<appender name="console-infolog" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d %5p %m%n" />
</layout>
</appender>
<!-- Application Loggers -->
<logger name="com.dsmentoring.chakan" additivity="false">
<level value="debug" />
<appender-ref ref="console"/>
</logger>
<!-- 3rdparty Loggers -->
<logger name="org.springframework.core">
<level value="error"/>
</logger>
<!-- Bean logger -->
<logger name="org.springframework.beans">
<level value="error"/>
</logger>
<!-- Context logger -->
<logger name="org.springframework.context">
<level value="error"/>
</logger>
<!-- Web logger -->
<logger name="org.springframework.web">
<level value="error"/>
</logger>
<logger name="org.springframework.ldap" additivity="true">
<level value="error"/>
</logger>
<!-- LDAP logger -->
<logger name="com.unboundid.ldap" additivity="true">
<level value="error"/>
</logger>
<!-- Root Logger -->
<root>
<priority value="off" />
<appender-ref ref="console" />
</root>
To sum up :
1) Is there a way to configure where logs are created and how they are created (4 types of logs) Other than log4j.xml, xml files and property files? I looked at all the java, jsp, js code but can't seem to find the configuration for logs. So I want to know if there are other ways to do that or where I should look for those configuration.
2) The 'logs' folder is possibly default for log4j?
ldap.properties
#LDAP Connection Info
ldap.host=192.168.0.17
ldap.port=22389
ldap.userName=cn=directory manager
ldap.password= 9074B18A0DE2D50C068D37B60BE5DFDE
ldap.baseDN=o=sso30root
ldap.defaultLoadSize=1000
ldap.start=start-ds
ldap.stop=stop-ds
ldap.workdir=/home/KB_openDJ // logs are created under this path
// /home/KB_openDJ/logs
In other java class, they use this.
#Value("${ldap.workdir}")
private String WORK_DIR;
// I ommited many lines in between
try{
diskUsage = sigar.getFileSystemUsage(WORK_DIR);
diskIOInfo.setDiskRead((int)(diskUsage.getDiskReadBytes()));
diskIOInfo.setDiskWrite((int)(diskUsage.getDiskWriteBytes()));
}catch(SigarException sigarEx){
log.debug("Disk Usage info load Error : " + sigarEx.getMessage());
}
I used the 'Search' feature in Eclipse many times. ( logs, WORK_DIR, and 4 types of log name, and many others. I am unable to locate the code about logging configuration. :(
My log4j version :
1.2.15

Ok, it seems that there's a solution which might help you. It requires some debugging of static initalization block of org.apache.log4j.LogManager class. This class is responsible for loading logger configuration file. Here's a documention link which thoroughly describes initalization process: link.
Here's an excerpt from a LogManager source file:
String configurationOptionStr = OptionConverter.getSystemProperty(DEFAULT_CONFIGURATION_KEY, null);
String configuratorClassName = OptionConverter.getSystemProperty(CONFIGURATOR_CLASS_KEY, null);
What I'm trying to show you here, is that your logger configuration file might be specified as a JVM option supplied to your application-server. That's why you are not able to determine actual file being used.
Even if this approach fails, I'd recommend you to investigate the appenders list retrieved at run-time. Here's a stackoverflow thread which describes how to do so: link.

Logback Dynamic Files using Two Properties

my problem is : My application maintains three buildings, and each building has a different process.
So, using logback, I want to create a log which has a specification :
each building will have a specific folder, and inside that specific folder of each building, there will be many log files, with each log file indicates a process.
My logback.xml right now is :
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
</pattern>
</encoder>
</appender>
<appender name="logAppender" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator>
<key>processName</key>
<defaultValue>unknown</defaultValue>
</discriminator>
<sift>
<appender name="FILE-${processName}"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/${distributor}/${processName}.log</file>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d [%thread] %level %mdc %logger{35} - %msg%n</pattern>
</layout>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>logs/${distributor}/${processName}.%i.log</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
<!-- <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>5KB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> -->
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
</appender>
</sift>
</appender>
<logger name="processLog" level="debug" additivity="false">
<appender-ref ref="logAppender" />
<appender-ref ref="stdout" />
</logger>
<root level="debug">
<appender-ref ref="stdout" />
<appender-ref ref="logAppender" />
</root>
</configuration>
And my java servlet code is :
public class DistributorServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
private static Logger processLog = LoggerFactory.getLogger("processLog");
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String office = req.getParameter("office");
MDC.put("distributor", office);
String process = req.getParameter("process");
MDC.put("process", process);
processLog.debug("Processing");
}
}
However, a log file is not generated.
Can anyone help me?
Thank you very much

1. Make the below change
private static Logger processLog = LoggerFactory.getLogger("processLog");
to
private static Logger processLog = LoggerFactory.getLogger(DistributorServlet .class);
2. Add additional discriminator for distributor
From the logback.xml it appears that only one discriminator has been added. Did you try adding another one for distributor
3. Do not forget
To add MDC.remove("keyName"); after its usage.
4. In case if you observe issue with multiple MDC keys
I faced an issue with the MDC.put in trying to add multiple keys into it. (I wondered why no putAll has been defined)
Although the underlying implementation is that of a HashMap<Key k, Value v> that should allow adding multiple keys, I was only able to see that the last key you put into MDC would be applied to logback.xml
While for the other keys I observed _IS_UNDEFINED value that gets appended instead.
Of course then again you can refer to the other various links and although this may not be a best idea, here is what I have done in my Java code,
System.setProperty("distributor", req.getParameter("office"));
System.setProperty("process", req.getParameter("process"));
Modify the logback.xml by removing discriminator. Well, alternatively you can remove one of the above properties and have the MDC.put for that property.
However please do refer to the links System.setProperty is safe in java?
Alternatively I suggest https://stackoverflow.com/a/32615172/3838328

How to hide the output of a single Logback logger?

I use logback in the following case
package ninja.template;
public class TemplateEngineManagerImpl implements TemplateEngineManager {
private final Logger logger = LoggerFactory.getLogger(TemplateEngineManagerImpl.class);
...
logger.info("Registered response template engines");
...
and I want to hide all the INFO output of this TemplateEngineManagerImpl class (or all the output - the class only logs at INFO level) but this class only.
Unfortunately, the following configuration doesn't work as I can still see "Registered response template engines" in my console output.
<configuration>
<appender name="STDOUT_TERSE" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern> | %msg%n</pattern>
</encoder>
</appender>
<logger name="ninja.template.TemplateEngineManagerImpl" level="OFF"/>
<root level="info">
<appender-ref ref="STDOUT_TERSE" />
</root>
</configuration>
note I tried also the following without results
<logger name="ninja.template.TemplateEngineManagerImpl" level="WARN"/>
and
<logger name="ninja.template" level="WARN"/>
and
<logger name="TemplateEngineManagerImpl" level="WARN"/>

The provided logback configuration works
<logger name="ninja.bodyparser.BodyParserEngineManagerImpl" level="WARN" additive="false"/>
The error I had that the string was logged in two different places.
So by displaying the full logger name with the following configuration:
<appender name="STDOUT_TERSE" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern> | %logger | %msg%n</pattern>
</encoder>
I was able to copy and paste the string that should be used as the logger name!