I have an Spring-Boot application on Linux that writes to service logs using Logback. Logback is setup to write to a file (account-service.log) and rotate the file at 10.5MB and finally delete the file on the 8th rotate. This is the default configuration.
The issue I'm having is the first log file created is written to past the 10.5MB size limit (it does rotate at the 10.5MB mark). So it's increasing in size until it becomes the 8th file and then it tries to get deleted. At this point the file is 84MB (10.5MB * 8) size. You can see here the file increasing in size
The main problem with that is that the OS tries to delete the file but since the application, because of Logback, still keeps it open then the filesystem doesn't show the deleted file (like with command find or du) and the system still keeps the space allocated on the disk. Also the file is still written to so it's taking up more and more disk space. I ran sudo lsof | grep deleted to find out that this file isn't being fully deleted.
Also interestingly the group that that first file belongs to is different than the rest. The first file has group root while the rest have the correct group name of account-service. This might be why the application can't close the file but I'm not sure.
The Logback dependency comes from my Spring-Boot pom:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.1.RELEASE</version>
</parent>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Dalston.SR4</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
The subsequent log files do get deleted ok and don't get written to past the 10.5 MB limit.
Has anyone seen this issue or found a way to fix it?
So I was able to solve most of my problems but one issue still remains.
The main issue was that I was using the built in logback xml for logging (base.xml). There it was configured for both console and file logging to be enabled.
<root level="INFO">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>
So when I run it locally I get my logs written to my terminal and service log file which is good but the issue occurs on deployed instances. What happened was that both "CONSOLE" and "FILE" loggers where writing to the file which is why even after the first file rotated, it was still being written too. The "CONSOLE" logger was writing to the first file even though it rotated because from its point of view that file is the terminal. This also helped fix a double logging issue we were seeing.
My fix was to add my own logback xml (logback-spring.xml) to override the base.xml one.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<springProfile name="default">
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</springProfile>
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
</root>
This stops console logging on deployed instances and thus keeps the first file from logging past the 10.5MB limit. This fix fixed the disk space issue we experienced too.
The only issue I was not able to solve yet it why the file is kept open by the application and thus not allowing it to be fully deleted from the disk space.
As you have mentioned the file as root user access permissions which do not look correct. Can you check the application by doing ps -efw | grep applicationName to see what user permissions it has as your application should have application user permissions only.
Related
This app works fine in IntelliJ and creates the log files as per configurations, however when launched the app from command line its not using logback-spring.xml file and instead goes on to create *tmp/spring.log file which seems to coming from spring *logback/base.xml.
I have spent couple of days to troubleshoot this issue but nothing seems to work so far and other questions do not address the underlying issue, your help is appreciated.
I am launching the app as -
java -jar abc.jar -Dspring.profiles.active=test
I can see that logback-spring.xml is present inside abc.jar as
BOOT-INF/classes/ logback-spring.xml
Here you can find how to configure logback with spring-boot howto.logging.logback
With src/main/resources/logback-spring.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<!-- include spring boot file-appender -->
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" /> <!-- add file-appender -->
</root>
<logger name="com.example" level="DEBUG"/>
</configuration>
And providing a logging file-name in src/main/resources/application.properties
logging.file.name=application.log
You should see a file application.log in the same directory as you launched your application with
java -jar your-app.jar --spring.profiles.active=test
I have a Spring boot application deployed to GCP App Engine with logback as a logging framework. The problem which I am facing is that I can not find the logger messages which I put into my code. I found a similar post: Google App Engine - Can not find my logging messages. But it is pretty much older and the solutions are not working anymore. On this post, it is mentioned to set the logger level manually. On a similar line, I used logging.properties file to set the level as INFO. But still, it was of no use.
I thought the issue might be with the file permission on the App Engine. So, I specified the log file location at /tmp having full read-write permissions. It did not work. If there are any exception scenarios then stack traces are visible on Stackdriver Logging. Please suggest.
Here is how I managed to make the logs appear in Stackdriver under java.log.
I created a file myApp/src/main/resources/logback.xml and added this configuration:
<configuration>
<appender name="STDOUT" class="com.google.cloud.logging.logback.LoggingAppender">
</appender>
<root level="info">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
I also added the dependency in the file myApp/pom.xml :
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging-logback</artifactId>
<version>0.80.0-alpha</version>
</dependency>
<!-- other dependencies -->
</dependencies>
Then in my main.java I did logger.info("my info log") and logger.error("my error log") and the logs appeared in Stackdriver after a few seconds of accessing my web app.
I took the sample code from the GoogleCloudPlatform/getting-started-java repository to deploy my app.
I've been banging my head over this one for a few days and can't get it figured out. Log4j2 is backwards compatible if you add the log4j1 compatibility library.
My web-inf\lib has:
slf-api
log4j-1.2-api (backwards compat. library)
log4j-api (log4j2)
log4j-core (log4j2)
log4j-web (auto-wiring for web applications)
My web.xml has:
<!-- log4j2 auto-wiring -->
<context-param>
<param-name>log4jConfiguration</param-name>
<param-value>file:///${catalina.base}/conf/log4j2.xml</param-value>
</context-param>
My [tomcat]/conf/log4j2.xml is:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug">
<Appenders>
<!-- Appends to CONSOLE -->
<Console name="consoleAppender" target="SYSTEM_OUT">
<ThresholdFilter level="DEBUG" onMatch="ACCEPT" onMismatch="DENY" />
<PatternLayout pattern="%5p (%F:%L) - %m%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="com.mycompany.componentA" level="WARN" />
<Logger name="com.mycompany.componentA.QueryLogger" level="DEBUG" />
<Logger name="com.mycompany.mycode" level="DEBUG" />
<Root level="WARN">
<AppenderRef ref="consoleAppender" />
</Root>
</Loggers>
</Configuration>
I have upgraded code under my control (com.mycompany.mycode) to log4j2 APIs and they work/log flawlessly. Code that is not under my control but was written against log4j1 (com.mycompany.componentA) just simply fails to log at all. No errors, no debugs, nothing.
Something interesting though... when I start the application I get a log4j1 warning about incorrect configurations when the application starts. This also stumps me because there are no log4j1 libraries (except the compatibility API) in the WAR. Warning is:
log4j:WARN No appenders could be found for logger (com.mycompany.componentB)
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
EDIT: I (finally) figured out what's going on. One of my dependencies did a horribly bad thing and BUNDLED the log4j1 classes into it's final jar. So there are no log4j1 jars on the classpath, but there are log4j1 CLASSES on the classpath.
The only way I was able to get this working was to:
Create BOTH a log4j1 and log4j2 XML configuration files (even though the log4j2 configuration contained all the loggers I wanted)
Let Log4j2 auto-wire itself via the 'log4j-web' artifact + 'log4jConfiguration' web.xml parameter
Manually wire log4j1 by calling the deprecated Log4jConfigurer.initLogging(...) API on server startup
This is probably horribly incorrect, but as indicated above, it was the only way I got it working after weeks of fooling around.
My understanding of the lo4j1 bridge is that wiring up log4j2 and including the bridge is all that should be required (e.g. no need to manually wire log4j1). In practice, that does not seem to be occurring.
That error message means you still have the log4j-1.x jar in your application. Look for it in your WEB-INF/lib and remove it and then it should work.
If not in WEB-INF/lib, then perhaps in your web container (Tomcat?) shared lib folder? Ralph is right that this error message is generated by Log4j-1.2, so it is on the classpath somewhere... Try printing the value of System property java.class.path if necessary.
Update: another way to find the location of the Log4j1 jar is by printing the value of org.apache.log4j.AppenderSkeleton.class.getResource("/org/apache/log4j/AppenderSkeleton.class") from your application.
(I originally suggested Category but this also exists in the Log4j 1 bridge, so AppenderSkeleton is better.)
I am using spring boot for my application and I am using default spring boot logging.
In my application.properties, I have added file path for logging.file,
logging.file= ${logger_path}
and my pom.xml contains
<logger_path>/tmp/app.log</logger_path>
When I start the application, it prints the logging messages to the file at /tmp/app.log, but the problem is it also prints the log messages on the console. I really don't understand why it is printing on console (though it is printing them to the specified file) when I have specified a log file.
Is there any configuration to prevent spring boot from printing the log messages to console?
Spring boot comes with build-in logback logger, which is configured to print to the console by default.
You need to overwrite the logback configuration (providing your own logback.xml on the classpath).
This is described here - - section 66.1
How to disable logback logging read here -
as you can see you have to provide the value OFF...something like:
<configuration>
<include resource="base.xml" />
.
.
<property name="root.level.console" value="OFF" />
</configuration>
Note: Keep in mind that the general idea here is that the logback configuration coming from spring-boot is minimal. The intention is that you provide your own logback configuration and completely overwrite the existing one - e.g. providing your own logback configuration with only log file-appender configured - this should be your general approach.
Include file-appender.xml and not console-appender with following configuration in logback-spring.xml:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
</root>
</configuration>
You also need to add logging.file to your application.properties
This is in compliance to what is mentioned in spring boot documentation - http://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html
I tried removing console configuration from logback.xml. But, It was still logging in console. So What I did is, I just removed the appender being added in the logging configuration by springboot. Thereby, we can stop springboot logging in console and separate log file. Add the below lines at the end of your application specific appenders are added. Your custom appender should not match any of these appender names. It worked for me.
// get instance of your log4j instance
Logger logger = LogManager.getRootLogger();
logger.removeAppender("CONSOLE"); // stops console logging
logger.removeAppender("LOGFILE"); // stops file logging
A similar discussion can be found here.
My logback.xml looks like that:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
</root>
</configuration>
Just in case you still experience an issue: I have read many discussions about that topic and it was not working at all for me. Unfortunately I have made a really stupid mistake when I have created the file logback.xml for the very first time: A space was added at the beginning of the filename. Therefore the file was never used. So just have another look on the filename, add the file in "src/main/resources" and remove any "logging.config" entries from your property files.
Tested with latest 1.3.1 release, with newer releases base.xml has been renamed to defaults.xml
Answer here
I am facing this trouble for a long time now without accessing the debug logs of the managed Threads in my Spring boot application when run on Tomcat. All the logs appear when run on the Eclipse/STS.
In Tomcat logs, I can only see the main Tread Logs.
I am connection to a database through JDBC and this is happening in a separate thread. I tried to follow the log configuration documentation but none of them helps to get the debug logs of these threads. So I do not actually see the exact problem of what is causing the connection to fail.
Here is what I tried so far:
I tried with the following logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- <appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<Target>System.out</Target> <encoder> <pattern>%d{yyyy-MM-dd HH:mm:ss} %-5p
%c{1}:%L - %m%n</pattern> </encoder> </appender> <logger name="com.biscoind"
additivity="false" level="TRACE"> <appender-ref ref="stdout" /> </logger>
<root level="debug"> <appender-ref ref="stdout" /> </root> -->
<include resource="org/springframework/boot/logging/logback/base.xml" />
<logger name="org.springframework.web" level="DEBUG" />
</configuration>
When that did not resolve the issue I removed this file and see if by default, if it logs all the treads. But it did not.
So, I added the following configurations to the application.properties
logging.level.org.springframework.web:TRACE
logging.level.org.hibernate:ERROR
Then It seemed to me that this is only logging out the above namespaces, I again added
debug=true
logging.level.org.springframework.web:DEBUG
logging.level.org.hibernate:DEBUG
Tried and it did not work.
I added my namespaces also and tried as follwing,
debug=true
logging.level.com.mydomain:DEBUG
logging.level.org.springframework.web:DEBUG
logging.level.org.hibernate:DEBUG
That did not work also, I am now confused on the what should I do with the config relative to logging to make the logs to appear for the tread executions.
Irrespective of the treads, because of the property spring.jpa.show-sql=true it logs the queries that are made.
It was not a problem with the threads at all. The application was working correctly in the development environment. The problem was in the deployment environment.
It turned out to be a Java version miss-match with the jar files and the JVM version. The jars were build using Java 8 and it was running on Java 7 JVM.
When the JMV was changed to Java-8. It worked fine. So Next time I will be more careful with the version mismatch.