Develop Reference

How can I straighten out my Tomcat logging?

Am working on a Tomcat configuration I did not do and would like any help to fix the logging. For reasons unknown, there are three -D parameters related to logging passed into the startup. The process looks like:
./bin/java -Djava.util.logging.config.file=CATALINA_BASE_DIR_HERE/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Dlog4j.configuration=file:CATALINA_BASE_DIR_HERE/tomcat/conf/log4j.properties ...
The relevant section of the logging.properties file looks as shown below. I believe this is standard.
handlers = 1catalina.org.apache.juli.FileHandler, 2localhost.org.apache.juli.FileHandler, 3manager.org.apache.juli.FileHandler, 4host-manager.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
...
2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.
For completeness, here is relevant portion of the log4j.properties file:
log4j.rootLogger=INFO, logfile
log4j.appender.logfile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logfile.DatePattern='.'yyyy-MM-dd
log4j.appender.logfile.File=MY_DIR/my_log.txt
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern = [%d{ABSOLUTE}] [%t] %-5p [%c] - %m%n
log4j.appender.logfile.Append=true
# per first answer given below, added:
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost] = INFO, logfile
Prior to today, there were no real issues - all the log data went to the log4j file as desired. The war file deployed in this Tomcat uses Spring Data JPA and Hibernate. To debug, showSql was enabled. All the queries came out, but to the localhost.DATE.log file. Can anyone tell how to fix so that ALL the statements go to the log4j designated file?

If you set showSql to true, hibernate will print SQL statement to SystemOut. You should add log4j.logger.org.hibernate.SQL=DEBUG to log4j config, so hibernate can also log the SQL statement. (The reason can be found in this answer)
By default, Tomcat use java.util.logging API for all internal logging. So the output goto localhost.DATE.log as you mentioned.
You can change the configuration, please refer the section Using Log4j
(for Tomcat 6.x~8.x)

rosjava set log level to WARN

I wrote a ROS node using rosjava.
When it starts up, there are a few logging outputs like:
Loading node class: MyApp.RosWrapper
Jun 18, 2015 3:12:39 PM org.ros.internal.node.client.Registrar <init>
INFO: MasterXmlRpcEndpoint URI: http://localhost:11311
Jun 18, 2015 3:12:39 PM org.ros.internal.node.client.Registrar onPublisherAdded
INFO: Registering publisher: Publisher<PublisherDefinition<PublisherIdentifier<NodeIdentifier</mynode, http://127.0.0.1:39009/>, TopicIdentifier</rosout>>, Topic<TopicIdentifier</rosout>, TopicDescription<rosgraph_msgs/Log, acffd30cd6b6de30f120938c17c593fb>>>>
I've found out that rosjava uses the org.apache.commons.logging.Log approach (see https://github.com/rosjava/rosjava_core/blob/indigo/rosjava/src/main/java/org/ros/internal/node/client/Registrar.java#L54)
To configure this, you should normally put a file called commons-logging.properties in your Classpath. I tried to add the folder containing this file to the classpath, but nothing changed.
UPDATE:
I found out that by default rosjava uses the Jdk14Logger class.
How can I decrease the log-level to e.g. WARN? Where do I have to put the corresponding config files?

Disclaimer: As stated in the comment, this does not work for rosjava and therefore does not answer the question. I do not delete the post, however, to keep records that this has already been tried.
Permanent setting
To permanently modify the log-level, create a file and add something like the following:
# Set the default ros output to warning and higher
log4j.logger.ros=WARN
# Override my package to output everything
log4j.logger.ros.my_package_name=DEBUG
For ROS to use this file, you have to define the environment ROSCONSOLE_CONFIG_FILE and set it to the path of your file. This can be done by adding the following line to ~/.bashrc:
export ROSCONSOLE_CONFIG_FILE=/path/to/config_file
See also the ROS wiki about rosconsole configuration.
Temporary setting
If you only want to temporarily change the log-level for a currently running node, you can use the tool rqt_logger_level. This will start a GUI where you can change the log-level for each running node. The settings are not stored, however, so it will be lost after restarting the node.

To manage the log levels for any ROSJava project, create a file like this one:
log-config.properties
# The following creates two handlers
handlers=java.util.logging.ConsoleHandler, java.util.logging.FileHandler
# Set the default logging level for the root logger
.level=ALL
# log level for the "com.example" package
org.ros.logging.level=ALL
# Set the default logging level
java.util.logging.ConsoleHandler.level=ALL
java.util.logging.FileHandler.level=ALL
# Set the default formatter
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
# Specify the location and name of the log file
java.util.logging.FileHandler.pattern=/home/robot/test.log
and later, load the configuration in the execution:
java -Djava.util.logging.config.file=/home/robot/log-config.properties -jar rosjava-helloworld-0.1.0-SNAPSHOT-all.jar
Juan Antonio

How to change log level of userlogs in Hadoop 2?

I've been trying to change the log level on userlogs i.e the files that appear under /var/log/hadoop-yarn/userlogs/application_<id>/container_<id> on CDH 5.2.1. However, no matter what I try, only INFO level logs will appear. I want to enable TRACE level logs for debugging.
Things I have tried so far:
Setting all loggers to TRACE level in /etc/hadoop/conf/log4j.properties.
Setting mapreduce.map.log.level and mapreduce.reduce.log.level in mapred-site.xml.
Setting mapreduce.map.log.level and mapreduce.reduce.log.level in the job configuration before submitting it.
Including a log4j.properties in my job jar file that sets the root Log4j logger to TRACE.
Modifying yarn-env.sh to specify YARN_ROOT_LOGGER=TRACE,console
None of these worked -- they didn't break anything, but they didn't have any effect on the log outputs under the userlogs directory. Modifying yarn-env.sh did cause the ResourceManager and NodeManager logs to enter trace level. Unfortunately these are not useful for my purpose.
I get the following error appearing in /var/log/hadoop-yarn/userlogs/application_<id>/container_<id>/stderr that may be relevant.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/hadoop-yarn/nm-local-dir/usercache/tomcat/appcache/application_1419961570089_0001/filecache/10/job.jar/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
I don't understand why the log4j "no configuration" message would happen, given that there is a log4j.properties file at the root of the job jar file that specifies a root logger:
log4j.rootLogger=TRACE, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] %m%n
My code does not knowingly use SLF4J for logging, it purely uses Log4j.

The actual answer was to set yarn.app.mapreduce.am.log.level to the level you need, but, crucially, it needs to be set in the Hadoop job configuration at submission time. It cannot be set on the cluster globally. The cluster global will always default to INFO, as it is hardcoded.
Using container-log4j.properties alone will not work as YARN will override the log level value on the command line. See the method addLog4jSystemProperties of org.apache.hadoop.mapreduce.v2.util.MRApps and cross reference with org.apache.hadoop.mapreduce.MRJobConfig.
container-log4j.properties will indeed be honoured, but it can't override the level set by this property.

On CDH5.8, I am overwriting the application log level at job submission time. To modify the log level of the mappers and reducers, I had to specify them explicitly using a command like this:
hadoop jar my-mapreduce-job.jar -libjars ${LIBJARS} -Dyarn.app.mapreduce.am.log.level=DEBUG,console -Dmapreduce.map.log.level=DEBUG,console -Dmapreduce.reduce.log.level=DEBUG,console
The logs are written to the applications's syslog which can be browsed using the Yarn ResourceManager Web UI.

You should try to edit (or create if it does'nt exist) : /etc/hadoop/conf/container-log4j.properties.
You can see it with a simple ps aux : the command line includes -Dlog4j.configuration=container-log4j.properties.

Logger logs are not showing up in Console and Tomcat logs

Well I have my Tomcat logging configured and I know the properties are being read, because it now genereates the log files according to the name prefix that I specified. Well the problems is that the log files contain all the info that Tomcat itself logs, however it doesn't contain anything from what I try to output in my servlet classes with log.info(...) etc
my logging.properties:
handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
org.apache.juli.FileHandler.level = FINEST
org.apache.juli.FileHandler.directory = ${catalina.base}/logs
org.apache.juli.FileHandler.prefix = GDIA_
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
As you can see it's all set as visible from FINEST, however I don't see any output when i do:
My imports:
import java.util.logging.Level;
import java.util.logging.Logger;
My log variable:
private final static Logger log = Logger.getLogger(GenTreeUploader.class.getName());
My testing the output:
log.severe("GREAT!!!!!!!!!!");
I'm running Tomcat 7 in Eclipse. What's interesting is that before I added configuration to tomcat run configuration in eclipse The log output would work fine to console, but whenever I added this:
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=D:\Dropbox\EclipseWorkspace\.metadata\.plugins\org.eclipse.wst.server.core\tmp7\conf\logging.properties"
I couldn't see any output in console or in the log file. Of course prior to adding this my logging.properties were not to be found...
So any ideas why when I add this configuration I can't see any log output from the application classes when I do "log.severe("GREAT!!!!!!!!!!");"...?

How to log stdout output in Tomcat?

Is there a way to log all stdout output to the catalina.log file in Tomcat? (i.e. everything that gets printed to System.out.println())
The console window that opens when you run TOMCAT/bin/startup.bat displays output from stdout, but it's not saved to TOMCAT/logs/catalina.<date>.log.
My specific problem is that I have a console appender defined in log4j to output to the console. These log messages appear correctly in the Tomcat console window, but they are not written to catalina.log. I'm running Tomcat 5.5 on Windows. Thanks.
EDIT:
Here is my log4j.properties file. It is located at TOMCAT/webapps/app/WEB-INF/classes/log4j.properties:
log4j.rootCategory=DEBUG, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d{ABSOLUTE} %-5p %c{1}]: %m%n

I've come across similar questions before, and haven't found a way to do this by logging System.out in Windows unless you are running Tomcat as a Windows service. This seems to work by default in Unix since startup.sh points to catalina.sh which logs stdout to the catalina.out file like below
org.apache.catalina.startup.Bootstrap "$#" start >> "$CATALINA_BASE"/logs/catalina.out 2>&1 &
In log4j, ConsoleAppender by itself does not append to a File, only to System.out
However, I've modified your log4j properties to add a FileAppender and this config works, but of course this logs into a separate log file.
New config
# Set root logger level to DEBUG.
log4j.rootLogger=DEBUG, console, myFile
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d{ABSOLUTE} %-5p %c{1}]: %m%n
# myFile writes to file
log4j.appender.myFile=org.apache.log4j.RollingFileAppender
log4j.appender.myFile.File=logs/tomcatlog4j.log
log4j.appender.myFile.MaxFileSize=100KB
log4j.appender.myFile.layout=org.apache.log4j.PatternLayout
log4j.appender.myFile.layout.ConversionPattern==[%d{ABSOLUTE} %-5p %c{1}]: %m%n
Output
=[15:24:03,819 INFO A1]: In my.jsp
=[15:24:03,975 INFO A1]: Out of my.jsp
=[15:24:04,880 INFO A1]: In my.jsp
=[15:24:04,880 INFO A1]: Out of my.jsp
also see
How to log exceptions from a specific package deployed in tomcat
log select events into a separate file
https://serverfault.com/questions/201178/tomcat-5-5-how-to-redirect-the-logging-output-to-one-file-per-web-application

Did you checked, whether the log4j.properties file can be found from your application?
Maybe you can check, by setting a hardcoded file path like
-Dlog4j.configuration=file:///C:\Dev\log4j.properties
If the logs are written after these change, the relativ path to the log4j file is wrong.

If I look at the default logging config of tomcat 5.5 in logging.properties:
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler
That looks to me as if stdout of web applications might be logged to files only for level INFO and above, regading that http://tomcat.apache.org/tomcat-5.5-doc/logging.html states that in tomcat JULI logging configuration loggers do not use parent's handlers when they are assigned their own handlers. Also the file should be prefixed localhost and not catalina. But then I do not understand how the output comes to your output window :/