I am trying to eliminate a leading hyphen from our console and file logs in SpringBoot 1.3.5.RELEASE with default logback config.
Logging pattern looks like:
logging:
pattern:
console: '%d{yyyy-MM-dd HH:mm:ss.SSS} %clr([${spring.application.name}]){red} %clr(%5p) %clr(${PID:- }){magenta} %clr(---){faint} %X{req.requestId} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%rEx}'
The file log pattern is similar, without the color coding. Both output every line after the first with a leading hyphen, which is making our syslog - logstash - grok filtering trickier. Example output:
2016-06-21 11:52:00.576 [my-app] INFO etc.. (application started)
-2016-06-21 11:52:00.583 [my-app] DEBUG etc..
-2016-06-21 11:52:00.583 [my-app] INFO etc..
I can't see anything in the docs mentioning this behaviour. Welcome any advice on how to eliminate it, if possible!
Update
Thanks to Edgar's answer below, it turns out this was caused by the following at the end of our logging pattern:
${LOG_EXCEPTION_CONVERSION_WORD:-%rEx}
I replaced it with:
${LOG_EXCEPTION_CONVERSION_WORD:%rEx}
et voila, the hyphen at the start of the following line disappears. See http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html#boot-features-custom-log-configuration
The problem is in this part of logging.pattern.console:
${LOG_EXCEPTION_CONVERSION_WORD:-%rEx}
:- is Logback's default value delimiter and is what you should use in logback.xml. However, you're configuring things in application.properties where it's Spring Framework's default value delimiter (:) which should be used.
As you've used :-, you're saying use the value of LOG_EXCEPTION_CONVERSION_WORD and, if it's not set, use -%rEx instead.
The correct syntax is:
${LOG_EXCEPTION_CONVERSION_WORD:%rEx}
It's hard to diagnose without your providing the full logging format. I'm seeing something similar in our code, and it seems to be related to including this in the format:
${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}
If you're using it, then the hyphen may be the one before the %w. I'm not sure what the intended purpose of this is. If I find it, I'll add it to my answer.
Related
This is my sample java log I tried to parse using Logstash
[#|2022-04-06T07:02:47.885+0800|INFO|sun-appserver2.1|javax.enterprise.system.stream.out|_ThreadID=245;_ThreadName=sun-bpel-engine-thread-6;Process Instance Id=192.168.1.1:2001:0db8:85a3:0000:0000:8a2e:0370:7334;Service Assembly Name=CommComposite;BPEL Process Name=testname;|
Register BPEL ID : 192.168.1.1:2001:0db8:85a3:0000:0000:8a2e:0370:7334|#]
I tried to use this filter to parse it
%{TIMESTAMP_ISO8601:time} %{LOGLEVEL:logLevel} %{GREEDYDATA:logMessage}
It seems this filter always left last line thus creating invalid log line. I suspect due to the [#| and |#] opening and closing tag.
Could anyone help me how to parse this kind of log so I can parse it properly?
Here is the grok pattern for the sample data provided by you:
%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:loglevel}\|(?<message>(.|\r|\n)*)
Output:
I have Filebeat pulling logs from stdout. I want to ensure my logs are outputted as JSON, so they can be properly parsed.
Thus far, here's what I've found:
org.jboss.logmanager.formatters doesn't have a JSON formatter
There's an "extension" module that provides a JsonFormatter class.
I can use it in my logging.properties by doing something like this:
handler.CONSOLE=org.jboss.logmanager.handlers.ConsoleHandler
handler.CONSOLE.properties=autoFlush,target
handler.CONSOLE.autoFlush=true
handler.CONSOLE.formatter=JSON-FORMATTER
handler.CONSOLE.target=SYSTEM_OUT
formatter.JSON-FORMATTER=org.jboss.logmanager.ext.formatters.JSONFormatter
I need to know:
Am I missing anything with this configuration?
How can I customise the JSON output (i.e. add or remove fields)?
There is a json-formatter in WildFly 14. I would not suggest editing the logging.properties. The following CLI commands are an example of configuring a json-formatter.
/subsystem=logging/json-formatter=json:add(exception-output-type=formatted, pretty-print=false, meta-data={label=value})
/subsystem=logging/console-handler=CONSOLE:write-attribute(name=named-formatter, value=json)
Note the meta-data attribute is just a key/value pair separated by commas.
How can I customise the JSON output (i.e. add or remove fields)?
You can really only add metadata or change field names. You can't remove fields though.
Can log4j be configured to automatically indent messages with an amount proportional to the message log level? I need to get an output like this:
[2013-09-13 09:38:24,638] INFO - Processing graph nodes...
[2013-09-13 09:38:24,640] DEBUG - Processed node 1...
[2013-09-13 09:38:24,646] DEBUG - Processed node 2...
[2013-09-13 09:38:24,649] DEBUG - Processed node 3...
[2013-09-13 09:38:25,948] INFO - Processed 3 node(s)
I don't think you can do that by configuration. But you should be able to do it by implementing an custom Layout class that understands how to implement your indentation scheme. Then add that to your appender via the configuration file and all of your log messages will be formatted as you want them to be.
Not sure if log4j provides any such setting. But you can always write a wrapper method which takes the string message and the Log level as input and within it use a switch statement to do different intendation for different levels.
Below link is old but it is working. It extends Layout class but may be helpful to extend PatternLayout class on similar lines.
https://github.com/zepheira/tracer
I have a log4j logger that I instantiate like this:
logger = Logger.getLogger("EQUIP(" + id + ")");
Doing so, when I call logger.info("message"), I should get an output like this (with some date formatting):
13/11/12 15:08:27 INFO: EQUIP(1): message
But I'm only getting:
13/11/12 15:08:27 INFO: message
I'm also printing logger.getName() to the console for debugging and it gives me back the correct "EQUIP(1)" name. This behaviour is happening in some cases in my program, where I have several loggers like this, but mostly in this specific class. I want to know if I'm doing something wrong, if this name should be only the class/package name, or if it can be anything (it works well in 80+% of my loggers). I need to print the ID of each equipment because I have several of them working simultaneous, and tracking them without this would be next to impossible.
How should I fix this, preferably without resourcing to changing all my log calls to include this prefix?
The output format depends on the pattern you've configured for the appender. If the pattern string includes %c then you'll get the logger name included, if it doesn't then you won't.
An alternative approach might be to use the mapped diagnostic context, which is designed to disambiguate between log output from different threads writing to the same logger.
I am using Fortify SCA to find the security issues in my application (as a university homework). I have encountered some 'Log Forging' issues which I am not able to get rid off.
Basically, I log some values that come as user input from a web interface:
logger.warn("current id not valid - " + bean.getRecordId()));
and Fortify reports this as a log forging issue, because the getRecordId() returns an user input.
I have followed this article, and I am replacing the 'new line' with space, but the issue is still reported
logger.warn("current id not valid - " + Util.replaceNewLine(bean.getRecordId()));
Can anyone suggest a way to fix this issue?
I know this was already answered, but I thought an example would be nice :)
<?xml version="1.0" encoding="UTF-8"?>
<RulePack xmlns="xmlns://www.fortifysoftware.com/schema/rules">
<RulePackID>D82118B1-BBAE-4047-9066-5FC821E16456</RulePackID>
<SKU>SKU-Validated-Log-Forging</SKU>
<Name><![CDATA[Validated-Log-Forging]]></Name>
<Version>1.0</Version>
<Description><![CDATA[Validated-Log-Forging]]></Description>
<Rules version="3.14">
<RuleDefinitions>
<DataflowCleanseRule formatVersion="3.14" language="java">
<RuleID>DDAB5D73-8CF6-45E0-888C-EEEFBEFF2CD5</RuleID>
<TaintFlags>+VALIDATED_LOG_FORGING</TaintFlags>
<FunctionIdentifier>
<NamespaceName>
<Pattern/>
</NamespaceName>
<ClassName>
<Pattern>Util</Pattern>
</ClassName>
<FunctionName>
<Pattern>replaceNewLine</Pattern>
</FunctionName>
<ApplyTo implements="true" overrides="true" extends="true"/>
</FunctionIdentifier>
<OutArguments>return</OutArguments>
</DataflowCleanseRule>
</RuleDefinitions>
</Rules>
</RulePack>
Alina, I'm actually the author of the article you used to solve your log injection issue. Hope it was helpful.
Vitaly is correct with regards to Fortify. You'll need to build what Fortify calls a "custom rule".
It will likely be a dataflow cleanse rule. A basic example can be found here: http://www.cigital.com/newsletter/2009-11-tips.php. If you own Fortify, there should be a custom rule writing guide in your product documentation.
I don't know what the taint flag you'll use is, but it would look something like "-LOG_FORGING". You would essentially write a rule to remove the log forging "taint" whenever data is passed through your utility method. Fortify will them assume that any data passed through there is now safe to be written to a log, and will not cause log forging.
You need to mark your replaceNewLine as sanitiser in Fortify (if I remember correctly) and it will stop reporting the issue.
You can actually create a new rule from a particular method.
Navigate to the function on the right side of audit workbench after you've done a scan.
Find your sanitizing method and right click on it.
You can generate a rule from it. What you want is a general DataflowCleanseRule.
I just did this based on the xml someone posted above. You can save the rule as a .xml file.
When updating your scan you can pass the -rule argument and point at the .xml file.