Develop Reference

How to write logs of Hadoop 2.2 (or higher)?

How can I write logs to stderr, stdout, and syslog in Hadoop 2.2 or higher? I tried to use log.info, log.error, System.out.println, and System.err.println but I only got the following from the log director:
stderr : Total file length is 222 bytes. log4j:WARN No appenders could
be found for logger (org.apache.hadoop.ipc.Server). log4j:WARN Please
initialize the log4j system properly. log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
stdout : Total file length is 0 bytes.
syslog : Total file length is 34828 bytes.
(I searched and can't find my contents.)

It's a Java problem rather an Hadoop one, you did not correctly configured the Log4J.
log4j:WARN No appenders could be found for logger
on cloudera you can find the log4j.properties into the config dir of each module.

How to change log level of userlogs in Hadoop 2?

I've been trying to change the log level on userlogs i.e the files that appear under /var/log/hadoop-yarn/userlogs/application_<id>/container_<id> on CDH 5.2.1. However, no matter what I try, only INFO level logs will appear. I want to enable TRACE level logs for debugging.
Things I have tried so far:
Setting all loggers to TRACE level in /etc/hadoop/conf/log4j.properties.
Setting mapreduce.map.log.level and mapreduce.reduce.log.level in mapred-site.xml.
Setting mapreduce.map.log.level and mapreduce.reduce.log.level in the job configuration before submitting it.
Including a log4j.properties in my job jar file that sets the root Log4j logger to TRACE.
Modifying yarn-env.sh to specify YARN_ROOT_LOGGER=TRACE,console
None of these worked -- they didn't break anything, but they didn't have any effect on the log outputs under the userlogs directory. Modifying yarn-env.sh did cause the ResourceManager and NodeManager logs to enter trace level. Unfortunately these are not useful for my purpose.
I get the following error appearing in /var/log/hadoop-yarn/userlogs/application_<id>/container_<id>/stderr that may be relevant.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/hadoop-yarn/nm-local-dir/usercache/tomcat/appcache/application_1419961570089_0001/filecache/10/job.jar/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don't understand why the log4j "no configuration" message would happen, given that there is a log4j.properties file at the root of the job jar file that specifies a root logger:
log4j.rootLogger=TRACE, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] %m%n
My code does not knowingly use SLF4J for logging, it purely uses Log4j.

The actual answer was to set yarn.app.mapreduce.am.log.level to the level you need, but, crucially, it needs to be set in the Hadoop job configuration at submission time. It cannot be set on the cluster globally. The cluster global will always default to INFO, as it is hardcoded.
Using container-log4j.properties alone will not work as YARN will override the log level value on the command line. See the method addLog4jSystemProperties of org.apache.hadoop.mapreduce.v2.util.MRApps and cross reference with org.apache.hadoop.mapreduce.MRJobConfig.
container-log4j.properties will indeed be honoured, but it can't override the level set by this property.

On CDH5.8, I am overwriting the application log level at job submission time. To modify the log level of the mappers and reducers, I had to specify them explicitly using a command like this:
hadoop jar my-mapreduce-job.jar -libjars ${LIBJARS} -Dyarn.app.mapreduce.am.log.level=DEBUG,console -Dmapreduce.map.log.level=DEBUG,console -Dmapreduce.reduce.log.level=DEBUG,console
The logs are written to the applications's syslog which can be browsed using the Yarn ResourceManager Web UI.

You should try to edit (or create if it does'nt exist) : /etc/hadoop/conf/container-log4j.properties.
You can see it with a simple ps aux : the command line includes -Dlog4j.configuration=container-log4j.properties.

Need to Remove Log File Warning in Selenium

I am using the log4j.properties in My Selenium Package.
Every Time I run the Module the below 3 Lines are always been added in the Console and the Applications.log file.
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
For Reference, below is the log4j.properties Code:
#Application Logs
log4j.logger.devpinoyLogger = DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.maxFileSize= 5000KB
log4j.appender.dest1.maxBackupIndex=3
log4j.appender.dest1.layout=org.apache.log4j.PatternLayout
log4j.appender.dest1.layout.ConversionPattern=%d{dd/MM/YYYY HH:mm:ss} %m%n
log4j.appender.dest1.File=D:\\Automation\\src\\Logs\\Application.log
#do not append the old file .Create a new log File everytime
log4j.appender.dest1.Append=false
Please let me know what needs to be change in order to remove the 3 lines of Warnings from the console output and Application logs.

Please include the following snippet to your code:
System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.Jdk14Logger");
Hope this helps!

This will resolve your errors and the message appears in the console:
System.setProperty("org.apache.commons.logging.Log",
"org.apache.commons.logging.impl.Jdk14Logger");

Log4j JPA Appender

I am trying to save some Log Events with log4j's JPA Appender , I've followed the tutorial
here (JPAAppender).
But when I am testing the logger , I get some log4j errors/warnings about the log4j.properties :
log4j:WARN Continuable parsing error 2 and column 31
log4j:WARN Document root element "Configuration", must match DOCTYPE root "null".
log4j:WARN Continuable parsing error 2 and column 31
log4j:WARN Document is invalid: no grammar found.
log4j:WARN Please initialize the log4j system properly.
Supposing that the problem is at log4j.properties file , any help??
Thanks.

Solved with JDBC appender , i used the command :
log4j.appender.JDBC.sql=INSERT INTO ....
I just pass the requested parameters to the insert query ,nevertheless i have persistence configured.
Thanks to log4j.MDC
MDC tutorial

UnableToCompleteException with no log

Trying to run my GWT app, I get
EntryPoint initialization exception
Exception while loading module ch.swisstph.mortqual.mqui.client.MqInput. See Development Mode for details.
com.google.gwt.core.ext.UnableToCompleteException: (see previous log entries)
at com.google.gwt.dev.shell.ModuleSpace.rebindAndCreate(ModuleSpace.java:513)
at com.google.gwt.dev.shell.ModuleSpace.onLoad(ModuleSpace.java:385)
at com.google.gwt.dev.shell.OophmSessionHandler.loadModule(OophmSessionHandler.java:200)
at com.google.gwt.dev.shell.BrowserChannelServer.processConnection(BrowserChannelServer.java:526)
at com.google.gwt.dev.shell.BrowserChannelServer.run(BrowserChannelServer.java:364)
at java.lang.Thread.run(Thread.java:722)
in the web browser, and just
/usr/lib/jvm/java-6-openjdk/bin/java -Xmx256m -Didea.launcher.port=7537 -Didea.launcher.bin.path=/home/dhardy/code/download/idea-IU-129.239/bin -Dfile.encoding=UTF-8 -classpath /home/install/gwt/gwt-dev.jar:/home/dhardy/p/mortqual/mqui/src:/usr/lib/jvm/java-6-openjdk/jre/lib/compilefontconfig.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/jsse.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/rhino.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/jce.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/management-agent.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/rt.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/javazic.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/resources.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/charsets.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/java-atk-wrapper.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/pulse-java.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-6-openjdk/jre/lib/ext/dnsns.jar:/home/dhardy/p/mortqual/mqui/out/test/mqui:/home/dhardy/p/mortqual/mqui/out/production/mqui:/home/install/appengine-java-sdk/lib/shared/jsp-api.jar:/home/install/appengine-java-sdk/lib/shared/appengine-local-runtime-shared.jar:/home/install/appengine-java-sdk/lib/shared/el-api.jar:/home/install/appengine-java-sdk/lib/shared/servlet-api.jar:/home/install/appengine-java-sdk/lib/user/appengine-api-1.0-sdk-1.7.5.jar:/home/install/gwt/gwt-user.jar:/home/dhardy/p/mortqual/anacod/target/test-classes:/home/dhardy/p/mortqual/anacod/target/classes:/home/dhardy/.m2/repository/net/sourceforge/jexcelapi/jxl/2.6.12/jxl-2.6.12.jar:/home/dhardy/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar:/home/dhardy/.m2/repository/com/beust/jcommander/1.30/jcommander-1.30.jar:/home/dhardy/.m2/repository/org/apache/poi/poi/3.9-20130311/poi-3.9-20130311.jar:/home/dhardy/.m2/repository/org/apache/poi/poi-ooxml/3.9/poi-ooxml-3.9.jar:/home/dhardy/.m2/repository/org/apache/poi/poi-ooxml-schemas/3.9/poi-ooxml-schemas-3.9.jar:/home/dhardy/.m2/repository/org/apache/xmlbeans/xmlbeans/2.3.0/xmlbeans-2.3.0.jar:/home/dhardy/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/dhardy/.m2/repository/dom4j/dom4j/1.6.1/dom4j-1.6.1.jar:/home/dhardy/.m2/repository/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar:/home/install/gwt/validation-api-1.0.0.GA-sources.jar:/home/install/gwt/validation-api-1.0.0.GA.jar:/home/dhardy/code/download/idea-IU-129.239/lib/idea_rt.jar com.intellij.rt.execution.application.AppMain com.google.gwt.dev.DevMode -war /home/dhardy/.IntelliJIdea12/system/gwt/mqui.5589a251/mqui.33ff3210/run/www -remoteUI 7901:IntelliJIdea -startupUrl mqInput.html ch.swisstph.mortqual.mqui.mqui
log4j:WARN No appenders could be found for logger (org.apache.jasper.compiler.JspRuntimeContext).
log4j:WARN Please initialize the log4j system properly.
Dev Mode initialized. Startup URL:
http://127.0.0.1:8888/mqInput.html?gwt.codesvr=127.0.0.1:9997
on the command line. There are no logs I can find (I tried configuring log4j via a .properties file, which removed its warnings but still didn't give me any logs).
So how do I solve this?
The two most likely causes are renaming of my start-up page and pushing some code out to a library.

try putting -logLevel SPAM in your command line arguments, this will print detailed logs, also you can put explicit GWT.log("message") in your entry point code, that will tell your how far it is going (normal logging doesn't work directly with gwt).
This page describes how to debug GWT in general, how ever my suggestion would be run debug this directly in eclipse with gwt plugin, GWT support in eclipse is amazing.