Develop Reference

Tha jar is not executing

I have submitted the following command
$java -cp spark-kafka-fraud-detection-assembly-1.0.jar com.jcalc.feed.CCStreamingFeed 50
Its seems like started executing.I can see the following things in my console
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Its remain in same state for long time .Nothing got executed expect the above three lines . Can any one help me .Thanks!

How to write logs of Hadoop 2.2 (or higher)?

How can I write logs to stderr, stdout, and syslog in Hadoop 2.2 or higher? I tried to use log.info, log.error, System.out.println, and System.err.println but I only got the following from the log director:
stderr : Total file length is 222 bytes. log4j:WARN No appenders could
be found for logger (org.apache.hadoop.ipc.Server). log4j:WARN Please
initialize the log4j system properly. log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
stdout : Total file length is 0 bytes.
syslog : Total file length is 34828 bytes.
(I searched and can't find my contents.)

It's a Java problem rather an Hadoop one, you did not correctly configured the Log4J.
log4j:WARN No appenders could be found for logger
on cloudera you can find the log4j.properties into the config dir of each module.

How to change log level of userlogs in Hadoop 2?

I've been trying to change the log level on userlogs i.e the files that appear under /var/log/hadoop-yarn/userlogs/application_<id>/container_<id> on CDH 5.2.1. However, no matter what I try, only INFO level logs will appear. I want to enable TRACE level logs for debugging.
Things I have tried so far:
Setting all loggers to TRACE level in /etc/hadoop/conf/log4j.properties.
Setting mapreduce.map.log.level and mapreduce.reduce.log.level in mapred-site.xml.
Setting mapreduce.map.log.level and mapreduce.reduce.log.level in the job configuration before submitting it.
Including a log4j.properties in my job jar file that sets the root Log4j logger to TRACE.
Modifying yarn-env.sh to specify YARN_ROOT_LOGGER=TRACE,console
None of these worked -- they didn't break anything, but they didn't have any effect on the log outputs under the userlogs directory. Modifying yarn-env.sh did cause the ResourceManager and NodeManager logs to enter trace level. Unfortunately these are not useful for my purpose.
I get the following error appearing in /var/log/hadoop-yarn/userlogs/application_<id>/container_<id>/stderr that may be relevant.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/hadoop-yarn/nm-local-dir/usercache/tomcat/appcache/application_1419961570089_0001/filecache/10/job.jar/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don't understand why the log4j "no configuration" message would happen, given that there is a log4j.properties file at the root of the job jar file that specifies a root logger:
log4j.rootLogger=TRACE, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] %m%n
My code does not knowingly use SLF4J for logging, it purely uses Log4j.

The actual answer was to set yarn.app.mapreduce.am.log.level to the level you need, but, crucially, it needs to be set in the Hadoop job configuration at submission time. It cannot be set on the cluster globally. The cluster global will always default to INFO, as it is hardcoded.
Using container-log4j.properties alone will not work as YARN will override the log level value on the command line. See the method addLog4jSystemProperties of org.apache.hadoop.mapreduce.v2.util.MRApps and cross reference with org.apache.hadoop.mapreduce.MRJobConfig.
container-log4j.properties will indeed be honoured, but it can't override the level set by this property.

On CDH5.8, I am overwriting the application log level at job submission time. To modify the log level of the mappers and reducers, I had to specify them explicitly using a command like this:
hadoop jar my-mapreduce-job.jar -libjars ${LIBJARS} -Dyarn.app.mapreduce.am.log.level=DEBUG,console -Dmapreduce.map.log.level=DEBUG,console -Dmapreduce.reduce.log.level=DEBUG,console
The logs are written to the applications's syslog which can be browsed using the Yarn ResourceManager Web UI.

You should try to edit (or create if it does'nt exist) : /etc/hadoop/conf/container-log4j.properties.
You can see it with a simple ps aux : the command line includes -Dlog4j.configuration=container-log4j.properties.

Need to Remove Log File Warning in Selenium

I am using the log4j.properties in My Selenium Package.
Every Time I run the Module the below 3 Lines are always been added in the Console and the Applications.log file.
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
For Reference, below is the log4j.properties Code:
#Application Logs
log4j.logger.devpinoyLogger = DEBUG,dest1
log4j.appender.dest1=org.apache.log4j.RollingFileAppender
log4j.appender.dest1.maxFileSize= 5000KB
log4j.appender.dest1.maxBackupIndex=3
log4j.appender.dest1.layout=org.apache.log4j.PatternLayout
log4j.appender.dest1.layout.ConversionPattern=%d{dd/MM/YYYY HH:mm:ss} %m%n
log4j.appender.dest1.File=D:\\Automation\\src\\Logs\\Application.log
#do not append the old file .Create a new log File everytime
log4j.appender.dest1.Append=false
Please let me know what needs to be change in order to remove the 3 lines of Warnings from the console output and Application logs.

Please include the following snippet to your code:
System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.Jdk14Logger");
Hope this helps!

This will resolve your errors and the message appears in the console:
System.setProperty("org.apache.commons.logging.Log",
"org.apache.commons.logging.impl.Jdk14Logger");

log4j : Logs aren't being updated due to an NameNotFoundException

I am using Apache commons logging with log4j for logging Mechanism .
I observered that , the logs aren't being updated or refreshed due to the below exception .
The below Exception is being printed inside catalina.log
Please let me know how this Exception is linked with Logs being Updated
javax.naming.NameNotFoundException: Name AKADbPool is not bound in this Context
at org.apache.naming.NamingContext.lookup(NamingContext.java:770)
at org.apache.naming.NamingContext.lookup(NamingContext.java:153)
at org.apache.naming.SelectorContext.lookup(SelectorContext.java:152)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at com.scivantage.middleware.util.J2EEUtil.connectToDataSource(J2EEUtil.java:48)
Why aren't my Application Logs aren't being updated due to above Exception ??
This is my Log4j.properties file
log4j.rootCategory=Info, A1
# A1 is a DailyRollingFileAppender
log4j.appender.A1=org.apache.log4j.DailyRollingFileAppender
log4j.appender.A1.file=D:\\Greetings\\Ravk.log
log4j.appender.A1.datePattern='.'yyyy-MM-dd
log4j.appender.A1.append=true
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-22d{dd/MMM/yyyy HH:mm:ss} - %m%n

I don't think it is because of that exception that you're not seeing logs. It is just a coincidence that you're seeing the exception and the logs aren't working!
See if you have multiple log4j.properties, log4j.xmls loaded in the classpath. Do not forget to look inside the classes folder and JAR files.

Add
-Dlog4j.debug=true
to your runtime-configuration, to see how log4j is configuring itself during startup..