Develop Reference

How to control Spark logging in IntelliJ

I'm usually running stuff from JUnit but I've also tried running from main and it makes no difference.
I have read nearly two dozen SO questions, blog posts, and articles and tried almost everything to get Spark to stop logging so much.
Things I've tried:
log4j.properties in resources folder (in src and test)
Using spark-submit to add a log4j.properties which failed with "error: missing application resources"
Logger.getLogger("com").setLevel(Level.WARN);
Logger.getLogger("org").setLevel(Level.WARN);
Logger.getLogger("akka").setLevel(Level.WARN);Logger.getRootLogger().setLevel(Level.WARN);spark.sparkContext().setLogLevel("WARN");
In another project I got the logging to be quiet with:
Logger.getLogger("org").setLevel(Level.WARN);
Logger.getLogger("akka").setLevel(Level.WARN);
but it is not working here.
How I'm creating my SparkSession:
SparkSession spark = SparkSession
.builder()
.appName("RS-LDA")
.master("local")
.getOrCreate();
Let me know if you'd like to see more of my code.
Thanks

I'm using IntelliJ and Spark and, this work for me:
Logger.getRootLogger.setLevel(Level.ERROR)
You could change Log Spark configurations too.
$ cd SPARK_HOME/conf
$ gedit log4j.properties.template
# find this lines in the file
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
and change to ERROR
log4j.rootCategory=ERROR, console
In this file you have other options tho change too
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN
# Settings to quiet third party logs that are too verbose
.....
And finally rename the log4j.properties.template file
$ mv log4j.properties.template log4j.properties
You can follow this link for further configuration:
Logging in Spark with Log4j
or this one too:
Logging in Spark with Log4j. How to customize the driver and executors for YARN cluster mode.

It might be an old question, but I just ran by the same problem.
To fix it what I did was:
Adding private static Logger log = LoggerFactory.getLogger(Spark.class);
as a field for the class.
spark.sparkContext().setLogLevel("WARN"); after creating the spark session
Step 2 will work only after step 1.

How can I straighten out my Tomcat logging?

Am working on a Tomcat configuration I did not do and would like any help to fix the logging. For reasons unknown, there are three -D parameters related to logging passed into the startup. The process looks like:
./bin/java -Djava.util.logging.config.file=CATALINA_BASE_DIR_HERE/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Dlog4j.configuration=file:CATALINA_BASE_DIR_HERE/tomcat/conf/log4j.properties ...
The relevant section of the logging.properties file looks as shown below. I believe this is standard.
handlers = 1catalina.org.apache.juli.FileHandler, 2localhost.org.apache.juli.FileHandler, 3manager.org.apache.juli.FileHandler, 4host-manager.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
...
2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.
For completeness, here is relevant portion of the log4j.properties file:
log4j.rootLogger=INFO, logfile
log4j.appender.logfile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logfile.DatePattern='.'yyyy-MM-dd
log4j.appender.logfile.File=MY_DIR/my_log.txt
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern = [%d{ABSOLUTE}] [%t] %-5p [%c] - %m%n
log4j.appender.logfile.Append=true
# per first answer given below, added:
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost] = INFO, logfile
Prior to today, there were no real issues - all the log data went to the log4j file as desired. The war file deployed in this Tomcat uses Spring Data JPA and Hibernate. To debug, showSql was enabled. All the queries came out, but to the localhost.DATE.log file. Can anyone tell how to fix so that ALL the statements go to the log4j designated file?

If you set showSql to true, hibernate will print SQL statement to SystemOut. You should add log4j.logger.org.hibernate.SQL=DEBUG to log4j config, so hibernate can also log the SQL statement. (The reason can be found in this answer)
By default, Tomcat use java.util.logging API for all internal logging. So the output goto localhost.DATE.log as you mentioned.
You can change the configuration, please refer the section Using Log4j
(for Tomcat 6.x~8.x)

Log4J log does not create back up files

I have configured my application log in the generic JVM arguments of my Websphere application server as :
-Dlog4j.configuration=file:D:/app_logs/log4j/log4j.properties
The log4j.properties file looks like:
log4j.appender.app.layout=org.apache.log4j.PatternLayout
log4j.appender.app.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
log4j.appender.app=org.apache.log4j.RollingFileAppender
log4j.appender.app.File=D:/app_logs/log4j/logs/app.log
log4j.appender.app.MaxFileSize=8000KB
log4j.appender.app.MaxBackupIndex=10
#root logger option
log4j.rootLogger=debug,app
# restrictions
log4j.logger.net.sf.hibernate=info
#log4j.logger.net.sf.hibernate.type=debug
#log4j.logger.net.sf.hibernate.ps.PreparedStatementCache=debug
log4j.logger.com.abc.wm.eu.app.interop=off
log4j.logger.com.abc.wm.eu.app.webservice.servlet.ADOServlet=off
#
log4j.logger.org.apache.axis=error
log4j.logger.org.apache.commons.httpclient=error
log4j.logger.httpclient.wire.header=error
log4j.logger.httpclient.wire.content=error
log4j.logger.org.htmlparser=error
Problem:
The log rolls over. But the back up logs are not created. The same configuration works fine for another application running on the same server.
I have no idea what is wrong here.
All suggestions and solutions are welcome.
Thanks

log4j.appender.FILE.Append=true
This might do the trick

How to log stdout output in Tomcat?

Is there a way to log all stdout output to the catalina.log file in Tomcat? (i.e. everything that gets printed to System.out.println())
The console window that opens when you run TOMCAT/bin/startup.bat displays output from stdout, but it's not saved to TOMCAT/logs/catalina.<date>.log.
My specific problem is that I have a console appender defined in log4j to output to the console. These log messages appear correctly in the Tomcat console window, but they are not written to catalina.log. I'm running Tomcat 5.5 on Windows. Thanks.
EDIT:
Here is my log4j.properties file. It is located at TOMCAT/webapps/app/WEB-INF/classes/log4j.properties:
log4j.rootCategory=DEBUG, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d{ABSOLUTE} %-5p %c{1}]: %m%n

I've come across similar questions before, and haven't found a way to do this by logging System.out in Windows unless you are running Tomcat as a Windows service. This seems to work by default in Unix since startup.sh points to catalina.sh which logs stdout to the catalina.out file like below
org.apache.catalina.startup.Bootstrap "$#" start >> "$CATALINA_BASE"/logs/catalina.out 2>&1 &
In log4j, ConsoleAppender by itself does not append to a File, only to System.out
However, I've modified your log4j properties to add a FileAppender and this config works, but of course this logs into a separate log file.
New config
# Set root logger level to DEBUG.
log4j.rootLogger=DEBUG, console, myFile
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=[%d{ABSOLUTE} %-5p %c{1}]: %m%n
# myFile writes to file
log4j.appender.myFile=org.apache.log4j.RollingFileAppender
log4j.appender.myFile.File=logs/tomcatlog4j.log
log4j.appender.myFile.MaxFileSize=100KB
log4j.appender.myFile.layout=org.apache.log4j.PatternLayout
log4j.appender.myFile.layout.ConversionPattern==[%d{ABSOLUTE} %-5p %c{1}]: %m%n
Output
=[15:24:03,819 INFO A1]: In my.jsp
=[15:24:03,975 INFO A1]: Out of my.jsp
=[15:24:04,880 INFO A1]: In my.jsp
=[15:24:04,880 INFO A1]: Out of my.jsp
also see
How to log exceptions from a specific package deployed in tomcat
log select events into a separate file
https://serverfault.com/questions/201178/tomcat-5-5-how-to-redirect-the-logging-output-to-one-file-per-web-application

Did you checked, whether the log4j.properties file can be found from your application?
Maybe you can check, by setting a hardcoded file path like
-Dlog4j.configuration=file:///C:\Dev\log4j.properties
If the logs are written after these change, the relativ path to the log4j file is wrong.

If I look at the default logging config of tomcat 5.5 in logging.properties:
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler
That looks to me as if stdout of web applications might be logged to files only for level INFO and above, regading that http://tomcat.apache.org/tomcat-5.5-doc/logging.html states that in tomcat JULI logging configuration loggers do not use parent's handlers when they are assigned their own handlers. Also the file should be prefixed localhost and not catalina. But then I do not understand how the output comes to your output window :/

log4j logging to catalina.out in unix instead of log file

In my web application I am inititalizing logging in my web application by using the PropertyConfigurator.configure(filepath) function in a servlet which is loaded on startup.
String log4jfile = getInitParameter("log4j-init-file");
if (log4jfile != null) {
String propfile = getServletContext().getRealPath(log4jfile);
PropertyConfigurator.configure(propfile);
I have placed my log4j.properties file in WEB_INF/classes directory and am specifying the file as
log4j.appender.rollingFile.File=${catalina.home}/logs/myapp.log
My root logger was initally configured as:
log4j.rootLogger=DEBUG, rollingFile,console
(I believe the console attribute also logs the statements into catalina.out)
In windows the logging seems to occor normally with the statements appearing in the log file.
In unix my logging statements are being redirected to the catalina.out instead of my actual log file(Which contains only logs from the initalization servlet). All subsequent logs show up in catalina.out.
This leads me to believe that my log4j is not getting set properly but I am unable to figure out the cause. Could anyone help me figure out where the problem might be.
Thanks in advance,
Fell

It may be a permissions problem - check the owners and permissions of the ${catalina.home}/logs directory.

It will depend on the server you are using.
Is it Tomcat? Is it the same version on Windows and on Linux?

Check whether you have set CATALINA_HOME environmental variable in your machine.
You can check it in linux by typing 'echo $CATALINA_HOME'.