I have a basic facility for allowing users to remotely apply changes to the logging files in my application. Some logs are configured using java.util.logging properties files, and some are configured using log4j/log4cplus-style properties files. I'd like to do some basic validation of the properties that users try to apply. Namely, I want to assure the following:
Every logging.properties file must always contain at least a root logger/logging level
The logger/level must be set to a valid value. That is, they should not be able to set .level = GIBBERISH or anything like that.
I'll probably allow them to set MaxFileSize and MaxBackupIndex (log4j), and .limit and .count properties (java.util.logging), too.
What's the best way to accomplish this? I can obviously just loop over the keys and values in a Properties object and look for their values in a hard-coded Map or some other data structure that tells what valid properties are, but I'm trying to come up with a solution that's a little more elegant than that.
The problem with running any set of partial syntax checks against the properties files is that they'll always be inadequate by definition unless you capture every partial variation acceptable by the logging system, in which case you'll have recreated a portion of the logging system. No matter what properties you choose to validate theres bound to be additional ways to submit broken files.
Rather than testing for individual properties, why not create an additional (temporary, for the scope of the check only) logger object based on the input file and detect if it throws an error?
The "elegant" solution would be to write a rule-based engine for checking sets of name-value pairs. But IMO that is totally over the top for this use-case ... unless the checks are far more complex than I imagine.
I'd say that the simple (inelegant) solution is best in this case.
Related
I'm looking for a way to prevent some sensitive data from being logged.
Ideally i would like to prevent / capture things like
String sensitive = "";
log.info ("This should be prevented or caught by something : {} ", sensitive);
this post is a bit of a longshot, I'm willing to investigate on any lead.
annotation, new types, Sonar Rules, logger hacking etc...
thx for your brainstorming :)
guillaume
Create custom type for it.
Make sure that toString doesn't return actual content.
I imagine there are multiple ways to do this, but one way is to use the Logback configuration file, to specify a message provider for the "arguments" and "message". In those providers, you define a "writeTo" method that looks for particular patterns in the output, and masks them.
This is the path to a solution, but I obviously don't provide many details here. I'm not aware of any "standard" solutions for this.
Another possibility would avail itself if your architecture has services running in transient containers, and the log output is sent to a centralized log aggregator, like Splunk. If you were ok with the initial logs written in the container having sensitive data, you could have the log aggregator look for patterns to mask for.
I would recommend two options, can you split your PII data into a separate log and then log that data securely?
If not, consider something like Cribl Logstream. Point your log shipper at it and let it strip away any PII you are concerned about. LogStream makes it very very easy to remove/mask/encrypt sensitive data. It has all sorts of other features as well.
At my last job we used LogStream as the router to make decisions about the data based on the content. PII data was detected and one copy was pushed to a secure PII certified logging platform and another copy was pushed to the operational logging platform but the PII data was masked so a wider audience could use the logging with no risk. It was a very useful workflow that solved a log of problems.
I am facing this security flaw in my project at multiple places. I don't have any white-list to do a check at every occurrence of this flaw. I want to use ESAPI call to perform a basic blacklist check on the file name. I have read that we can use SafeFile object of ESAPI but cannot figure out how and where.
Below are a few options I came up with, Please let me know which one will work out?
ESAPI.validator().getValidInput() or ESAPI.validator().getValidFileName()
Blacklists are a no-win scenario. This can only protect you against known threats. Any code scanning tool you use here will continue to report the vulnerability... because a blacklist is a vulnerability. See this note from OWASP:
This strategy, also known as "negative" or "blacklist" validation is a
weak alternative to positive validation. Essentially, if you don't
expect to see characters such as %3f or JavaScript or similar, reject
strings containing them. This is a dangerous strategy, because the set
of possible bad data is potentially infinite. Adopting this strategy
means that you will have to maintain the list of "known bad"
characters and patterns forever, and you will by definition have
incomplete protection.
Also, character encoding and OS makes this a problem too. Let's say we accept an upload of a *.docx file. Here's the different corner-cases to consider, and this would be for every application in your portfolio.
Is the accepting application running on a linux platform or an NT platform? (File separators are \ in Windows and / in linux.)
a. spaces are also treated differently in file/directory paths across systems.
Does the application already account for URL-encoding?
Is the file being sent stored in a database or on the system itself?
Is the file you're receiving executable or not? For example, if I rename netcat.exe to foo.docx does your application actually check to see if the file being uploaded contains the magic numbers for an exe file?
I can go on. But I won't. I could write an encyclopedia.
If this is across multiple applications against your company's portfolio it is your ethical duty to state this clearly, and then your company needs to come up with an app/by/app whitelist.
As far as ESAPI is concerned, you would use Validator.getValidInput() with a regex that was an OR of all the files you wanted to reject, ie. in validation.properties you'd do something like: Validator.blackListsAreABadIdea=regex1|regex2|regex3|regex4
Note that the parsing penalty for blacklists is higher too... every input string will have to be run against EVERY regex in your blacklist, which as OWASP points out, can be infinite.
So again, the correct solution is to have every application team in your portfolio construct a whitelist for their application. If this is really impossible (and I doubt that) then you need to make sure that you've stated the risks cited here clearly to management and you refuse to proceed with the blacklist approach until you have written documentation that the company chooses to accept the risk. This will protect you from legal liability when the blacklist fails and you're taken to court.
[EDIT]
The method you're looking for was called HTTPUtilites.safeFileUpload() listed here as acceptance criteria but this was most likely never implemented due to the difficulties I posted above. Blacklists are extremely custom to the application. The best you'll get is a method HTTPUtilities.getFileUploads() which uses a list defined in ESAPI.properties under the key HttpUtilities.ApprovedUploadExtensions
However, the default version needs to be customized as I doubt you want your users uploading .class files and dll to your system.
Also note: This solution is a whitelist and NOT a blacklist.
The following code snippet works to get past the issue CWE ID 73, if the directory path is static and just the filename is externally controlled :
//'DIRECTORY_PATH' is the directory of the file
//'filename' variable holds the name of the file
//'myFile' variable holds reference to the file object
File dir = new File(DIRECTORY_PATH);
FileFilter fileFilter = new WildcardFileFilter(filename);
File[] files = dir.listFiles(fileFilter);
File myFile = null ;
if(files.length == 1 )
myFile = files[0];
For many projects I have worked on, programming teams work with the style of placholding every piece of static text in an xhtml file into a properties file. For example:
xhtml=
...
<h1>${messages.resourceBundle['key.to.static.text.placeholder']}</h1>
...
messages.properties=
...
key.to.static.text.placeholder=This will be the heading for this page only
...
Would anybody be able to explain what the advantage in this is?
So far, I can only see the following disadvantages:
making changes to any xhtml file requires you to hunt for the correct .properties file, and then the individual property to make the change to
if others have re-used properties, then deleting them becomes tricky as you have to be certain no other page is referencing the property, therefore after several change request rounds, properties files become large with redundant properties
if there are 1000 xhtmls, there will be 1000 .properties files to load, which is more cycles on the cpu to load and inject static pieces of text
if your using WebFlow and have flows that pass into other flows, properties have to be duplicated, meaning that sometimes you must place the same property in many different properties files to render correctly
hard to read code; if you know you want to work on the text 'This will be the heading for this page' only, you'll need to work out where that is on the xhtml from the property files first - you can't simply look at the xhtml and see clearly how the content will be laid out once rendered.
The only advantages I can see are text reuse and possibly html escaping.
Apologies if its coding 101, but I've had a hunt around Google and can't find the reasoning to the pattern.
Many Thanks
This is a common practice for internationalizing content.
You create one property file per language (or locale) and use a dynamic way off resolving which one to load depending on the context. (e.g. Language HTTP header the browser sends).
It is arguably more flexible than providing 1 jsp file per language, and can still deal with complex cases where plurals or stylistic differences might change the way you write localized text.
This is a standard JDK feature, lookup resource bundles.
You do not have to build 1 file per jsp (maybe your framework works this way?), although doing so can help the person writing the translation.
I would like to create rolling files that contains statistical data about my service.
for example, logging each request that contained parameter X with a certain result set.
I have to write these files to comply with other systems statistical data:
Roll the file every half an hour
Each file have to have column headers
I have to follow a strict file name convention such as tracking.display.1314116577.done
My service is written in Java.
Since I need to roll files, using loggers seems like a good direction so I have tried an approach where I would log the data using logback logger (my logger of choise), but the conventional rolling file appender cannot role the file every half an hour (or am i wrong?), cannot add column headers and has a strict naming convention of its own.
I have tried to write my own RollingPolicy, but can't find enough resources or examples of how it's done.
Can anyone show/refer me how to accomplish this?
If not, would you recommend a different approach?
Thank you!
Yes, you can do it with a logback appender.
Take a look at TimeBasedRollingPolicy of RollingFileAppender you can easily roll the file each half hour.
To write the header you can extend RollingFileAppender and add the header based on your needs.
A very dumb answer: getting logging right costs much time. So it is easier to implement your own logger from scratch. It seems you have very rigid requirements. For instance that ".done" - you probably first write it as ".part" and then rename it when done.
I'm a newbie when it comes to properties, and I read that XML is the preferred way to store these. I noticed however, that writing a regular .properties file in the style of
foo=bar
fu=baz
also works. This would mean a lot less typing (and maybe easier to read and more efficient as well). So what are the benefits of using an XML file?
In XML you can store more complex (e.g. hierarchical) data than in a properties file. So it depends on your usecase. If you just want to store a small number of direct properties a properties file is easier to handle (though the Java properties class can read XML based properties, too).
It would make sense to keep your configuration interface as generic as possible anyway, so you have no problem to switch to another representation ( e.g. by using Apache Commons Configuration ) if you need to.
The biggest benefit to using an XML file is that XML declares its encoding, while .properties does not.
If you are translating these properties files to N languages, it is possible that these files could come back in N different encodings. And if you're not careful, you or someone else could irreversibly corrupt the character encodings.
If you have a lot of repeating data, it can be simpler to process
<connections>
<connection>this</connection>
<connection>that</connection>
<connection>the other</connection>
</connections>
than it is to process
connection1=this
connection2=that
connection3=the other
especially if you are expecting to have to store a lot of data, or it must be stored in a definite hierarchy
If you are just storing a few scalar values though, I'd go for the simple Properties approach every time
If you have both hierarchical data & duplicate namespaces, then use XML.
1) To emulate just a hierarchical structure in a properties file, simply use dot notation:
a.b=The Joker
a.b.c=Batgirl
a.b=Batman
a.b=Superman
a.b.c=Supergirl
So, complex (hierarchical) data representation is *not a reason to use xml.
2) For just repeating data, we can use a 3rd party library like ini4j to peg explicitly in java a count identifier on an implicit quantifier in the properties file itself.
a.b=The Joker
a.b=Batgirl
a.b=Batman
is translated to (in the background)
a.b1=The Joker
a.b2=Batgirl
a.b3=Batman
However, numerating same name properties still doesn't maintain the specific parent-child relationships. ie. how do we represent whether Batgirl is with The Joker or Batman?
So, xml is required when both features are needed. We can now decide if the 1st xml entry is what we want or the 2nd.
[a]
[b]Joker[/b]
[b]
[c]Batgirl[/c]
[/b]
[a]
--or--
[a]
[b]Batman[/b]
[b]
[c]Batgirl[/c]
[/b]
[/a]
Further detail in ....
http://ilupper.blogspot.com/2010/05/xml-vs-properties.html
XML is handy for complex data structures and or relationships. It does a decent job for having a "common language" between systems.
However, xml comes at a cost. Its is heavy to consume. You've got to load a parser, ensure the file is in the correct format, find the information etc...
Whereas properties files is pretty light weight and easy to read. Works for simple key/value pairs.
It depends on the data you're encoding. With XML, you can define a more complex representation of the configuration data in your application. Take something like the struts framework as an example. Within the framework you have a number of Action classes that can contain 1...n number of forward branches. With an XML configuration file, you can define it like:
<action class="MyActionClass">
<forward name="prev" targetAction="..."/>
<forward name="next" targetAction="..."/>
<forward name="help" targetAction="..."/>
</action>
This kind of association is difficult to accomplish using just the key-value pair representation of the properties file. Most likely, you would need to come up with a delimiting character and then include all of the forward actions on a single property separated by this delimiting character. It's quite a bit of work for a hackish solution.
Yet, as you pointed out, the XML syntax can become a burden if you just want to state something very simple, like set feature blah to true.
The disadvantages of XML:
It is hard to read - the tags make it look busier than it really is
The hierarchies and tags make it hard to edit and more prone to human errors
It is not possible to "append" to an XML property file to introduce a new property or provide an overriding value for an existing property so that the last one wins. The ability to append a property can be very powerful - we can implement a property management logic around this so that certain properties are "hot" and we don't need to restart the instance when these change
The Java property file solves the above problems. Consistent naming conventions and dot notation can help in solving the issue of hierarchy.