Develop Reference

How to destroy log4j logger instances

I have a code that run concurrently, and for each run it need log something in file. Each execution - new story with unique file.
So, I can't just get the logger for classname, add file appender, write logs, close and remove appender, because when concurrect code run - logger will contains both appenders, and logs will be written to both files.
So, I can create a new logger instance for each execution Logger.getLogger(classname + counter), but how to mark it as garbage after work is done?
P.S. Moreover... I need somethimes print to console from all this loggers.
Maybe I do something wrong, maybe log4j not created for this pattern and I have to implement it. But log4j - priority choose for me, because it already widely used in this big application.
Thank you in advance, Andrei!

You can't destroy / clean up Appender instances. But you can improve the solution you have with Log4j2 toolbox.
Routing things to dynamic files in Log4j is possible, but most of the time a Marker in a file is sufficient, or at least a starting point for more complex routing.
Please do read up on its documentation. Especially the last paragraph is important to your case:
Some important rules about Markers must be considered when using them.
Markers must be unique. They are permanently registered by name so care should be taken to insure that Markers used in your application are distinct from those in the application's dependencies, unless that is what is desired.
Parent Markers can be added or removed dynamically. However, this is fairly expensive to do. Instead, it is recommended that the parents be identified when obtaining the Marker the first time as shown in the examples above. Specifically, the set method replaces all the markers in a single operation while add and remove act on only a single Marker at a time.
Evaluating Markers with multiple ancestors is much more expensive than Markers with no parents. For example, in one set of tests to evaluate whether a Marker matched its grandparent took 3 times longer than evaluating the Marker itself. Even then though, evaluating Markers is inexpensive compared to resolving the callers class name or line number.
Then you can use the marker in the configuration in places where a pattern is expected like this: $${marker:}. I haven't used that in a filename though and doubt that it works, but you can create routing based on the Marker.
I used this simple test script, please note the use of the Marker that is created for each line from the Scanner. In your case it would be created by config or from Servlet input or similar.
package toTest;
import java.util.Scanner;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.Marker;
import org.apache.logging.log4j.MarkerManager;
public class TestMe {
private final Logger myOneAndOnlyLogger = LogManager.getLogger("MyCentralName");
public static void main(String[] args) {
new TestMe().doMyThing();
}
private void doMyThing() {
Scanner input = new Scanner(System.in);
String line = "";
while(!line.equals("QUIT")) {
System.out.println("Line: ");
line = input.nextLine();
Marker forThisRound = MarkerManager.getMarker(line);
myOneAndOnlyLogger.log(Level.ERROR, forThisRound, "1");
myOneAndOnlyLogger.log(Level.ERROR, forThisRound, "2");
System.out.println("Line done.");
}
}
}
and this log4j2.properties (a rolling file example I had at hand with marker in the pattern):
status = error
name = MarkerExample
#Make sure to change log file path as per your need
property.filename = /tmp/java/marker.log
filters = threshold
filter.threshold.type = ThresholdFilter
filter.threshold.level = debug
appenders = rolling
appender.rolling.type = RollingFile
appender.rolling.name = RollingFile
appender.rolling.fileName = ${filename}
appender.rolling.filePattern = /tmp/java/debug-backup-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} $${marker:} %-5p %c{1}:%L - %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=10MBONE
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 20
loggers = rolling
#Make sure to change the package structure as per your application
logger.rolling.name = MyCentralName
logger.rolling.level = debug
logger.rolling.additivity = false
logger.rolling.appenderRef.rolling.ref = RollingFile

Deeplearning4j Disable Logging

I have a deeplearning for java project which is producing huge amounts of logger output on STDO. I want to disable that but I cant seem to figure out how to do it.
I have a log4j.properties file in my src/main/resources folder which looks like this:
log4j.rootLogger=ERROR, Console
log4j.logger.play=WARN
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{ABSOLUTE} %-5p ~ %m%n
log4j.appender.org.springframework=WARN
log4j.appender.org.nd4j=WARN
log4j.appender.org.canova=WARN
log4j.appender.org.datavec=WARN
log4j.appender.org.deeplearning4j=WARN
log4j.appender.opennlp.uima=OFF
log4j.appender.org.apache.uima=OFF
log4j.appender.org.cleartk=OFF
log4j.logger.org.springframework=WARN
log4j.logger.org.nd4j=WARN
log4j.logger.org.canova=WARN
log4j.logger.org.datavec=WARN
log4j.logger.org.deeplearning4j=WARN
log4j.logger.opennlp.uima.util=OFF
log4j.logger.org.apache.uima=OFF
log4j.logger.org.cleartk=OFF
log4j.logger.org.deeplearning4j.optimize.solvers.BaseOptimizer=OFF
slf4j.logger.org.deeplearning4j.optimize.solvers.BaseOptimizer=OFF
The specific output that is far too much is:
21:26:34.860 [main] DEBUG o.d.optimize.solvers.BaseOptimizer - Hit termination condition on iteration 0: score=1.2894165074915344E19, oldScore=1.2894191699433697E19, condition=org.deeplearning4j.optimize.terminations.EpsTermination#55f111f3
which happens multiple times a second while training.

The output of the log entry that you have provided look very much as the SLF4J output with Logback format (not LOG4J output).
Also dependencies of deeplearning4j-core advice SLF4J is used for logging.
Hence your log4j.properties have no effect on deeplearning4j logging. Try to add logback.xml configuration to the resources as well and switch to WARN or ERROR level for root logger, see https://logback.qos.ch/manual/configuration.html

There are some properties in the framework DL4J (1.0.0-beta7) that activate/deactivate the logs. I found some of them:
import org.nd4j.common.config.ND4JSystemProperties;
System.setProperty(ND4JSystemProperties.LOG_INITIALIZATION, "false");
System.setProperty(ND4JSystemProperties.ND4J_IGNORE_AVX, "true");
System.setProperty(ND4JSystemProperties.VERSION_CHECK_PROPERTY, "false");
Notice that this is an unconventional solution. On the other hand, there are some messages impossible to avoid:
MultiLayerNetwork.init()
In this method you can find a OneTimeLogger without validations:
OneTimeLogger.info(log, "Starting MultiLayerNetwork with WorkspaceModes set to [training: {}; inference: {}], cacheMode set to [{}]",
layerWiseConfigurations.getTrainingWorkspaceMode(),
layerWiseConfigurations.getInferenceWorkspaceMode(),
layerWiseConfigurations.getCacheMode());
If you find a better way to disable log messages inside DL4J please share it. There are some other ways outside the DL4J library.

Filters using ConfigurationBuilder

Having an issue getting filters to work with Log4j 2 (2.4 and 2.6). Previously I had custom filters using logback and XML configuration like . Decided it would be reasonable to use the ConfigurationBuilder instead.
Just to give a simple example:
ConfigurationBuilder< BuiltConfiguration > builder =
ConfigurationBuilderFactory.newConfigurationBuilder();
builder.setConfigurationName("TestLogger");
AppenderComponentBuilder appenderBuilder = builder.newAppender("file", "FILE")
.addAttribute("fileName", "C:\\client_test.log").addAttribute("append", false).add(builder.newFilter("LevelRangeFilter", Filter.Result.ACCEPT, Filter.Result.NEUTRAL).
addAttribute("minLevel", Level.INFO).addAttribute("maxLevel", Level.INFO));
appenderBuilder.add(builder.newLayout("PatternLayout").addAttribute("pattern", "%msg%n%throwable"));
builder.add(appenderBuilder);
builder.add(builder.newLogger("TestLogger", Level.INFO)
.add(builder.newAppenderRef("file"))
.addAttribute("additivity", false));
LoggerContext logctx = Configurator.initialize(builder.build());
logFile = logctx.getLogger( "TestLogger" );
Excuse my ignorance, I am fairly new to the Java config for log4j!
If I initialize the context (just the current class I'm in):
LoggerContext logctx = Configurator.initialize(builder.build());
logFile = logctx.getLogger( "TestLogger" );
and try:
logFile.info("Info level logging");
logFile.error("Error level logging");
I see both of these outputs. I thought with the LevelRangeFilter with a range of INFO->INFO I would only see the info log levels ?
Realistically what I want is to have different appenders based on the log level (as I have with the logback xml configuration).
I've found it difficult to follow the documentation - trying to determine the various plugins and attributes to be used with the various component builders and just the general concepts.
The LevelRangeFilter seemed to fit the bill - where I could add different appenders as newAppenderRef to the build and write them out on to different files based on a tight range.
Any thoughts ?
Thanks!
c

Solved it. Used this for reference: https://logging.apache.org/log4j/2.x/log4j-core/apidocs/index.html
I had my mismatch set to NEUTRAL instead of DENY. Putting deny in cut the bad ranges out of the logs. So I was able to split the logs with two appenders using this filter as so:
AppenderComponentBuilder appenderBuilderInfo = builder.newAppender("fileInfo", "FILE")
.addAttribute("fileName", "C:\\client_info.log").addAttribute("append", false).add(builder.newFilter("LevelRangeFilter", Filter.Result.ACCEPT, Filter.Result.DENY)
.addAttribute("minLevel", Level.INFO).addAttribute("maxLevel", Level.INFO));
AppenderComponentBuilder appenderBuilderError = builder.newAppender("fileError", "FILE")
.addAttribute("fileName", "C:\\client_error.log").addAttribute("append", false).add(builder.newFilter("LevelRangeFilter", Filter.Result.ACCEPT, Filter.Result.DENY)
.addAttribute("minLevel", Level.ERROR).addAttribute("maxLevel", Level.ERROR));
appenderBuilderInfo.add(builder.newLayout("PatternLayout").addAttribute("pattern", "%msg%n%throwable"));
appenderBuilderError.add(builder.newLayout("PatternLayout").addAttribute("pattern", "%msg%n%throwable"));
builder.add(appenderBuilderInfo);
builder.add(appenderBuilderError);
builder.add(builder.newLogger("TestLogger", Level.INFO)
.add(builder.newAppenderRef("fileInfo"))
.addAttribute("additivity", false)
.add(builder.newAppenderRef("fileError"))
.addAttribute("additivity", false));

Dropwizard custom logger does not dump logs into the file

I need to set maxfile size for my application but currently i am using Dropwizard core version 0.8.4 and its file appender does not support this feature.
So I approached like below by writing a custom appender as updating to latest dropwizard(which support my need) version not an option right now.
private void initLogging(Configuration configuration) throws JoranException {
final File logDir = new File("/tmp/enforcer");
final File logFile = new File(logDir, "wallet.log");
final LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
RollingFileAppender<ILoggingEvent> rollingFileAppender = new RollingFileAppender<ILoggingEvent>();
rollingFileAppender.setFile(logFile.getAbsolutePath());
rollingFileAppender.setName("com.documents4j.logger.server.file");
rollingFileAppender.setContext(loggerContext);
FixedWindowRollingPolicy fixedWindowRollingPolicy = new FixedWindowRollingPolicy();
fixedWindowRollingPolicy.setFileNamePattern(logFile.getAbsolutePath() +"%i.gz");
fixedWindowRollingPolicy.setMaxIndex(7);
fixedWindowRollingPolicy.setContext(loggerContext);
fixedWindowRollingPolicy.setParent(rollingFileAppender);
fixedWindowRollingPolicy.start();
SizeBasedTriggeringPolicy<ILoggingEvent> sizeBasedTriggeringPolicy = new SizeBasedTriggeringPolicy<ILoggingEvent>();
sizeBasedTriggeringPolicy.setMaxFileSize("2KB");
sizeBasedTriggeringPolicy.setContext(loggerContext);
sizeBasedTriggeringPolicy.start();
rollingFileAppender.setRollingPolicy(fixedWindowRollingPolicy);
rollingFileAppender.setTriggeringPolicy(sizeBasedTriggeringPolicy);
rollingFileAppender.start();
System.out.println("Logging: The log is written to " + logFile);
final ch.qos.logback.classic.Logger log = loggerContext.getLogger(Logger.ROOT_LOGGER_NAME);
log.setLevel(Level.DEBUG);
log.addAppender(rollingFileAppender);
}
#Override
public void run(Configuration configuration, Environment environment) throws Exception
{
initLogging(configuration);
}
My yaml file config is
logging:
level: INFO
org.springframework.retry.support.RetryTemplate: DEBUG
appenders:
- type: file
currentLogFilename: /tmp/enforcer.log
threshold: ALL
archive: true
archivedLogFilenamePattern: /tmp/enforcer-%d.log
archivedFileCount: 5
timeZone: UTC
logFormat: '%-5level [%date{yyyy-MM-dd HH:mm:ss,SSS}] [%X{realdocRequestId}] %logger{15}: %m%n'
Now when i run my application i noticed, even though custom log file (/tmp/enforcer/wallet.log) is created in the particular directory but actual log is not dumped i.e. wallet.log file size is 0 kb where as the log file configured in yaml is created and size is certain kb and goes on increasing as log event is generated.
I am not able figure out what is wrong im doing, help will be appreciated.

your logger doesn't log anything because you never set an encoder to it. Set a breakpoint in:
OutputStreamAppender#start() and you will see that there is no encoder.
It will work once you add:
LayoutWrappingEncoder<ILoggingEvent> layoutEncoder = new LayoutWrappingEncoder<>();
DropwizardLayout formatter = new DropwizardLayout(loggerContext, TimeZone.getDefault());
formatter.setPattern("[%level] [%h] %d{yyyy-MM-dd'T'HH:mm:ss.SSS'Z', UTC} [%thread] [%logger] %msg%n");
layoutEncoder.setLayout(formatter);
formatter.start();
rollingFileAppender.setEncoder(layoutEncoder);
Of course you can define whichever format you like (and formatter).
But keep in mind that this is a fairly hacky example of what you try to achieve. You now have 2 points in code where you configure logging (DW and your own). You have yaml configured logging as well as your own. I recommend investing a bit of work and adding a proper AppenderFactory that you can use.
Hope that helps,
Artur

log4j - Why doesn't my appender show message?

At my work, I have inherited of the maintenance of a satandalone application.
The following code configures Log4J but no messages can be seen on the console.
LogManager.resetConfiguration();
PatternLayout layout = new PatternLayout();
layout.setConversionPattern("RECORD-BACKEND / (%-5p) %m (%F:%L)%n");
ConsoleAppender stderr = new ConsoleAppender();
stderr.setTarget(ConsoleAppender.SYSTEM_ERR);
stderr.setLayout(layout);
stderr.addFilter(new CurrentThreadLogFilter());
stderr.setThreshold(Level.INFO);
stderr.activateOptions();
Logger loggerRECORD = getLoggerRECORD();
loggerRECORD.setAdditivity(false);
loggerRECORD.addAppender(stderr);
Logger root = Logger.getRootLogger();
root.setLevel(Level.WARN);
root.addAppender(stderr);
// some lines forward ...
loggerRECORD.info("Process starting...");
What am I missing ?

Unfortunately you have not sent the code that actually prints the message. However I can assume that you try to do something like this:
logger.info("my message");
In this case your message will not be printed because it is filtered out by root logger defined to print warnings.
Loggers in Log4J are stored as hierarchical structure. The root logger is the entry point to this tree. Each logger filters log messages according to configured level. Therefore if upper level logger (root in your case) filters log message it even does not arrive to lower level logger and definitely does not arrive to appender.
The solution for you is to define root logger to allow ALL messages.
And the last note: do you have any special reasons to configure logger programmatically? Log4J can be configured using properties or (better) xml file. Take a look on this document.
And yet another note. Log4J has been deprecated by its creator. If you are starting now go forward to Logback as a logger and SLF4J as a light weight log interface.

Thanks to AlexR, here how I solved my problem:
LogManager.resetConfiguration();
PatternLayout layout = new PatternLayout();
layout.setConversionPattern("RECORD-BACKEND / (%-5p) %m (%F:%L)%n");
ConsoleAppender stderr = new ConsoleAppender();
stderr.setTarget(ConsoleAppender.SYSTEM_ERR);
stderr.setLayout(layout);
stderr.addFilter(new CurrentThreadLogFilter());
stderr.setThreshold(Level.INFO);
stderr.activateOptions();
Logger loggerRECORD = getLoggerRECORD();
loggerRECORD.setLevel( /* get Log Level from env. */ );
loggerRECORD.setAdditivity(false);
loggerRECORD.addAppender(stderr);
Logger root = Logger.getRootLogger();
root.setLevel(Level.WARN);
root.addAppender(stderr);
// some lines forward ...
loggerRECORD.info("Process starting...");
Since additivity is set to false for loggerRECORD, the appender sucessfully print any message from INFO to ERROR.