If you run a hadoop flume node, as default it generates logs under /var/log/flume using log4j. The files will look like
/var/log/flume/flume-$FLUME_IDENT_STRING-$command-$HOSTNAME.log
According to the flume user guide here, the only way to change the flume log configuration is via flume-daemon.sh which runs flume node using the Flume Environment Variables like:
export FLUME_LOGFILE=flume-$FLUME_IDENT_STRING-$command-$HOSTNAME.log
export FLUME_ROOT_LOGGER="INFO,DRFA"
export ZOOKEEPER_ROOT_LOGGER="INFO,zookeeper"
export WATCHDOG_ROOT_LOGGER="INFO,watchdog"
The questions are:
if I want to change the log level from INFO to DEBUG, this is the only place to do it?
Is there a configuration somewhere I can do this?
what about I want to set some packages' log level to DEBUG while others stay INFO?
Check if the log4j.properties or log* related files exist to set the variables -- which will also allow you to check and have some components of the logging part do excessive/DEBUG while other do INFO.
Noticed that under /etc/flume/conf.empty, there is a log4j.properties. Copied it to /etc/flume/conf, restart the flume node service, the log4j.properties file starts taking effect.
The overriding order is like flume-env.sh->flume-daemon.sh->log4j.properties.
eg. if you set up your flume_root_logger to DEBUG in flue-daemon.sh, it will override whatever setting you have for root_logger in your log4j.properties.
Related
the log file is generated when I run the code within IDE (Intellij IDEA).
as soon as I create runnable jar of the code and then try to run the jar then the logs are not generating.
I have made sure the log4j2.xml file is a part of classpath.
is there anything extra I have to do while creating jar in the Intellij IDEA?
Taken from the FAQ: How do I debug my configuration?
First, make sure you have the right jar files on your classpath. You need at least log4j-api and log4j-core.
Next, check the name of your configuration file. By default, log4j2 will look for a configuration file named log4j2.xml on the classpath. Note the “2” in the file name! (See the configuration manual page for more details.)
From log4j-2.9 onward
From log4j-2.9 onward, log4j2 will print all internal logging to the console if system property log4j2.debug is either defined empty or its value equals to true (ignoring case).
Prior to log4j-2.9
Prior to log4j-2.9, there are two places where internal logging can be controlled:
If the configuration file is found correctly, log4j2 internal status logging can be controlled by setting in the configuration file. This will display detailed log4j2-internal log statements on the console about what happens during the configuration process. This may be useful to trouble-shoot configuration issues. By default the status logger level is WARN, so you only see notifications when there is a problem.
If the configuration file is not found correctly, you can still enable log4j2 internal status logging by setting system property -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=TRACE.
I am using this doc to change my Tomcat loggers to log4j:
https://tomcat.apache.org/tomcat-7.0-doc/logging.html
my log4j.properties file is as shown in the document, except I changed this line to confirm it's working:
log4j.appender.CATALINA.File = ${catalina.base}/logs/test_catalina
I still see a file logs/catalina.out and not logs/test_catalina, which makes me think either log4j is not being used or my properties file is not read.
I have tomcat-juli-adapters.jar, log4j-1.2.17.jar and log4j.properties in lib
and overwrote bin/tomcat-juli.jar
running set -x catalina.sh start shows:
eval "/a/java64/jdk1.8.0/bin/java" "-Dnop" -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dlog4j.debug=true -Djava.endorsed.
dirs="/opt/apache-tomcat-7.0.62/endorsed" -classpath "/opt/apache-tomcat-7.0.62/bin/bootstrap.jar:/opt/apache-tomcat-7.0.62/bin/tomcat-juli.jar" -Dcatalina.base="/opt/apache-tomcat-7.0.62" -Dcatalina.home="/opt/apache-tom
cat-7.0.62" -Djava.io.tmpdir="/opt/apache-tomcat-7.0.62/temp" org.apache.catalina.startup.Bootstrap start &
I also have no logging.properties file on disk:
ls conf/*.properties
conf/catalina.properties
I would really appreciate advice on how to debug this -- I'm not sure if a jar is not being found properly, if my log4j.properties has a problem, or Tomcat just plain can't tell I'd like log4j.
Using Tomcat version Server number: 7.0.62.0
I would really appreciate advice on how to debug this
The switch of sending Tomcat log messages from java.util.logging to log4j 1.x is performed by Apache Commons Logging library. (Its compiled package-renamed version is in those tomcat-juli.jar and tomcat-juli-adapters.jar)
There is a debug setting available in Apache Commons Logging. After package renaming the setting name becomes org.apache.juli.logging.diagnostics.dest.
I do not see any evident error in those steps that you performed. Dummy issues may be: are the files readable?
Notes:
It is possible to put tomcat-juli-adapters.jar into $CATALINA_HOME/bin instead of /lib, but you need to update CLASSPATH variable (in your bin/setenv.sh) so that it is included in the value of -classpath argument to java. Repacking the two jars into a single jar file is also an option.
As mentioned by tomcat-7.0-doc/logging.html, deleting conf/logging.properties makes java.util.logging to fallback to default configuration. You may wish to explicitly configure that file.
Source code for tomcat-juli.jar and tomcat-juli-adapters.jar can be found at Maven Central. Those libraries have groupId=org.apache.tomcat.extras
I have a few Maven dependencies in my Java project that clutter the console output with redundant log info. I want to disable such logging.
Setting the additivity property to false might help. But could not use it properly.
I am looking for a log4j.xml config that will only print log output (warn, error, ...) from my project and not from any dependencies.
Redirect all the third party lib logs in a target appender, use another appender for your app
log4j.rootLogger=debug,thirdPartyLibAppender
log4j.logger.com.yourapp=debug, yourAppAppender
log4j.additivity.com.yourapp=false
# define where do you want third party lib logs to land : in a file
log4j.appender.thirdPartyLibAppender=org.apache.log4j.FileAppender
log4j.appender.thirdPartyLibAppender.append=true
log4j.appender.thirdPartyLibAppender.file=/tmp/app.log
log4j.appender.thirdPartyLibAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.thirdPartyLibAppender.layout.ConversionPattern=[%p] %c:%m%n
# define where do you want your app logs to land : stdout
log4j.appender.yourAppAppender=org.apache.log4j.ConsoleAppender
log4j.appender.yourAppAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.yourAppAppender.layout.ConversionPattern=[%p] %c:%m%n
Setting additivity to false will prevent that your app logs end in the thirdPartyLibAppender
In those 2 lines, don't forget to replace com.yourapp by the top level package name
log4j.logger.com.yourapp=debug, yourAppAppender
log4j.additivity.com.yourapp=false
It looks like the log4j2.xml was overriding all other configs. As of now, I have switched that dependency off. Maybe log4j2 > log4j hence the issue. Also, XML gets a higher priority over properties as I have seen somewhere.
Is it possible to have conditions in log4j.properties. I have a situation where I want to have logging level set to Info on production environment and DEBUG on local. Is it possible to read environment variables in log4j.properties.
No, you have to have 2 different log4j.properties file
Configuring logging is something that should happen as part of the deployment, not as part of the build, i.e. you should NOT create multiple builds for different log configurations, the risk of introducing also other differences in artifacts is to big.
Create ONE build containing a default configuration, possibly the one you want to use in production.
Implement a way to find and use an alternative configuration without changing your artifact. Most of the time this is achieved by adding an additional directory to the classpath of your application and store a log4j configuration there. You can use the default initialization of log4j by using a configuration format that has higher precedence then the one contained in the artifact. This also allows you to reconfigure logging without new deployment, which can be very helpful when troubleshooting.
Alternatively you can provide the location of the configuration file to use via a environment variable at startup: -Dlog4j.configuration=log4j-prod.xml (borrowed from Keerthi Ramanathan's answer)
You can prepare different builds and decide which log4j.propeties you want to include on build time, for example using maven params, profiles or any other way.
There is no way to declare condition in log4j.properties
No.
But just to outline some other options
a) I would encourage you to have a look at logback which provides a simple facade over log4j and you can then change your config at runtime. The relevant documentation can be found here.
b) If you have a build process in place (ant/maven) you can do the replacement as part of the build process. If you use maven you can set up a profile to build and the in the build-cycle apply filtering
c) Load the log4j files from a conf directory for each environment. The idea for that is that the files once set for an environment are changed minimally over time. You maintain both in your repository and as part of your deployment process ensure that additional/deleted files/props get added/removed.
What i would suggest as said in comment, have a separate version of log4j properties file for every environment and follow the naming convention for easy maintainance. say, for dev environment, it would be log4j-dev.xml and for production, log4j-prod.xml. Now, you can configure the appropriate file to pick up during runtime using
-Dlog4j.configuration=log4j-prod.xml
during server startup. so, that appropriate conffiguration file will be taken by log4j.
You can use programmatic configuration when using log4j, which gives you more control over what options to use in what environment. You can have your own configuration files and use your own logic to convert them into a log4j configuration. The downside is that you need to do init() somewhere in your application. This answer provides good reference.
I used a this approach when I had similar question. A default log level if nothing is explicitly specified, and option to override.
So, I added a log4j.properties file in application resources.
log4j.rootLogger=ALL, stdout
...
log4j.appender.stdout.Threshold=INFO
...
And then added more log config properties (log4j-n.properties, for n in {d, i, w, e}) defining log levels at debug, info, warning and error. Now, during startup I would supply the config file explicitly if I wanted to override the default.
java ... -Dlog4j.configuration=file:///<path>/log4j-n.properties ...
This would override any config I had in the default log4j.properties.
Later I went with this approach. I removed all the extra config files. In the log4j.properties file in resources, I used a JVM arg placeholder:
log4j.appender.stdout.Threshold=${app.log.level}
And supplied that as JVM argument.
java ... -Dapp.log.level=<LOG-LEVEL> ...
Voila!
I've done my best to setup Eclipse and my Java application to use a log4j.properties file. However, it does not seem to be using the properties file and I'm not sure why.
Libraries: slf4j-api-1.6.1, slf4j-jdk14-1.6.1
Within the application the logging works fine. I am able to print info, warnings, and errors into the Eclipse console.
What I would like to be able to do is change the log level to debug and print all logging messages to both the console and a log file.
I have created a log4j.properties file that looks like this:
log4j.rootLogger=DEBUG,console,file
log4j.rootCategory=DEBUG, R, O
# Stdout
log4j.appender.O=org.apache.log4j.ConsoleAppender
# File
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=log4j.log
# Control the maximum log file size
log4j.appender.R.MaxFileSize=100KB
# Archive log files (one backup file here)
log4j.appender.R.MaxBackupIndex=5
log4j.appender.file.File=checkLog.log
log4j.appender.file.threshold=DEBUG
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.O.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=[%d{ISO8601}]%5p%6.6r[%t]%x - %C.%M(%F:%L) - %m%n
log4j.appender.O.layout.ConversionPattern=[%d{ISO8601}]%5p%6.6r[%t]%x - %C.%M(%F:%L) - %m%n
My directory structure looks like this:
My Project
--src/
----MYProject/
------*.java
--bin/
----MYProject/
------*.class
--log4j/
----log4j.properties
In Eclipse I this this:
Run Configurations -> Classpath (tab) ->, right clicked on User Entries -> Added "log4j" as a new folder, and saved.
Then in my code I call the logger like this (sample code to demonstrate my approach so it may have syntax errors):
package MYProject;
import org.slf4j.LoggerFactory;
public class MyClass{
final org.slf4j.Logger test_logger = LoggerFactory.getLogger(MyClass.class);
public MyClass(){}
public someMethod(){
test_logger.debug("Some Debug");
test_logger.info("Some Info");
test_logger.warn("Some Warning");
test_logger.error("An Error");
}
}
I then call someMethod and it prints INFO, WARN, ERROR to the Eclipse console. It won't print DEBUG and won't print to a file.
I'd appreciate any suggestions on what I may be doing wrong.
There may be another log4j.properties or log4j.xml file in the classpath ahead of your log4j.properties. Open the run configuration for your project, and add -Dlog4j.debug=true as a VM Argument for your project. This will instruct log4j to print a lot of additional information on the console, including the config file that it is using.
If you are using the logging façade slf4j, then you need to specify exactly one logging backend by including the corresponding jar file for that backend. In your case, you have installed slf4j-jdk14-x.x.x.jar on your classpath, which is just a generic logger backend.
In order to use the log4j backend, you need to remove slf4j-jdk14-x.x.x.jar and replace it with slf4j-log4j12-x.x.x.jar. If you don't remove it, slf4j must choose only one backend jar, and probably not the one you want.
Of course you will also need the actual log4j-x.x.x.jar file on your classpath too.
Once these jars are properly in place, then the VM parameter of -Dlog4j.debug will actually work and be useful in debugging where your logging configs are coming from.
You need to tell your code to use the properties file. Before any logging is done please put
PropertyConfigurator.configure("log4j/log4j.properties");
You might have an old version in your target directory.
Clean the project and try again.
Apart from that you should make sure that you refresh the eclipse project if you didn't add the log4j.properties via eclipse.
delete jre(JRE System Library) of project in path project properties/library tab and set jre again!