We're currently running into an interesting problem regarding the sanitization of error logs being printed into our server logs. We have proper global error handling set up and have custom error messages that are sent back as responses from our OSGi java servlets.
We use dockerized containers as server instances that are autoscaled, so we're thinking about setting up a log aggregator and storing our exceptions within a DB in the cloud, that way we can also track metrics about our exceptions and pinpoint how we could improve our development process to reduce certain types of errors, etc.
I did a bit of research about how that should be done and I found this. The OWASP Logging sheet cheat. It mentions that passwords should never be logged among a few other things. That brings us to my question:
How do I go about properly sanitizing my logs without using some janky text processing or manually covering up all the potential cases?
Example stacktrace:
pkg.exceptions.CustomException: some registration error
ERROR: duplicate key value violates unique constraint "x_username_org_id_key"
Detail: Key (username, org_id)=(SOME EMAIL, 1) already exists.
Query: with A as (some query) insert into someTable (..values...) Parameters: [X, X, X, X, X, SOME_EMAIL, THE_PASSWORD]
at somepkg.etc
This is a pretty common error with registration systems that happens due to username collisions. Sure there's ways that this specific case can be avoided by ensuring the username isn't taken before the insertion isn't attempted and handling that case separately, but that's just a single case among many others.
After looking around to find a solution there doesn't seem to be an obvious way to solve the problem and I'm wondering if everyone out there has simply implemented their own version of a log sanitizer? We could simply purge the stacktrace if some troublesome strings are present, but that's not the best solution. Any suggestions?
If you only store and pass around password hashes you won't need to sanitize the logs for passwords. In cases where a password must be preserved temporarily in code use char[]s rather than Strings. This is a more secure approach in general and is considered a best practice. The standard library APIs all use character arrays for passwords.
Related
I'm looking for a way to prevent some sensitive data from being logged.
Ideally i would like to prevent / capture things like
String sensitive = "";
log.info ("This should be prevented or caught by something : {} ", sensitive);
this post is a bit of a longshot, I'm willing to investigate on any lead.
annotation, new types, Sonar Rules, logger hacking etc...
thx for your brainstorming :)
guillaume
Create custom type for it.
Make sure that toString doesn't return actual content.
I imagine there are multiple ways to do this, but one way is to use the Logback configuration file, to specify a message provider for the "arguments" and "message". In those providers, you define a "writeTo" method that looks for particular patterns in the output, and masks them.
This is the path to a solution, but I obviously don't provide many details here. I'm not aware of any "standard" solutions for this.
Another possibility would avail itself if your architecture has services running in transient containers, and the log output is sent to a centralized log aggregator, like Splunk. If you were ok with the initial logs written in the container having sensitive data, you could have the log aggregator look for patterns to mask for.
I would recommend two options, can you split your PII data into a separate log and then log that data securely?
If not, consider something like Cribl Logstream. Point your log shipper at it and let it strip away any PII you are concerned about. LogStream makes it very very easy to remove/mask/encrypt sensitive data. It has all sorts of other features as well.
At my last job we used LogStream as the router to make decisions about the data based on the content. PII data was detected and one copy was pushed to a secure PII certified logging platform and another copy was pushed to the operational logging platform but the PII data was masked so a wider audience could use the logging with no risk. It was a very useful workflow that solved a log of problems.
How can one of our many users get
java.lang.NoSuchMethodError:
at com.mycelium.wallet.activity.settings.SettingsPreference.getLanguage (SettingsPreference.kt:73)
at com.mycelium.wallet.WalletApplication.onCreate (WalletApplication.java:109)
at android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1127)
on this line of Kotlin code:
#JvmStatic
fun getLanguage(): String? = sharedPreferences.getString(Constants.LANGUAGE_SETTING, Locale.getDefault().language)
There are three function calls on this line: android.content.SharedPreferences::getString(String,String), java.util.Locale::getDefault() and java.util.Locale::getLanguage() all of which are available since API 1.
The only affected user (Samsung Galaxy A5(2017) (a5y17lte), 2816MB RAM, Android 8.0) tried to start the app 180 times with this insta-crash.
The conversion to Kotlin might have issues still?
try { ... } catch (NoSuchMethodError e) { ... } might be a suitable workaround. But they already might have given up; if you don't have any email address or alike, you won't be able to notify them. You'd could return a static string in case of a NoSuchMethodError. Besides, if one has written down the seed phrase, the wallet is on the block-chain, the device only has the keys. I'd file that as an individual destiny - and that device probably could still be rooted, to have the keys extracted. It's difficult to help them without having a support request, so that one could notify them of a new version, which not relies on whatever unknown method. Maybe they still use it and would receive an auto-update and try again, but only maybe - but there's no guarantee, that this is the only one unknown method on this device.
It's definitely not a Kotlin issue, but rather a storage defect; google "eMMC corruption".
And if the user has not written down their seed phrase, it's their very own fault.
This all is an assumption, but the probability isn't that low.
As you know (I am sure) a NoSuchMethodError is caused by a mismatch between the versions of classes at compile time versus at runtime.
And, I agree with you that the three methods called by that line of code should be present at runtime.
I was a suspicious that there isn't a message string for the NoSuchMethodError to say which method was missing, but there are other examples for the Android platform where the message is missing. So I am (tentatively) calling this not significant.
So we have to look for other explanations. Here are some:
The line number in the stacktrace could be inaccurate. People sometimes report this kind of thing; e.g. Crashlytics is reporting wrong line numbers
This particular user could be running a different (older?) version of your app where the code at that line is different to code you are looking at.
The user has "rooted" his device and messed around with its standard libraries. Alternatively, the user's device has been hacked and the hacker has interfered with the standard libraries (rather crudely in this case).
The user has been messing with the bytecodes for your app and has accidentally got it trying to call a non-existent method. Alternatively, the user is running a (crudely) trojaned version of your app where the bad guy has done the same thing.
The fact that your app involves Bitcoins means that there could be strong incentive for someone to be doing nefarious things ... so the last two alternatives should not be discounted.
The conversion to Kotlin might have issues still?
I don't see why that would affect only one user.
The fact the user tried 180 times is why I care. This is a Bitcoin wallet, so ... they might have money in that wallet and I hate to not fix issues if I can.
(Or conversely, this might be a bad guy trying to get bitcoins out of a wallet via a stolen device. The fact that the user is being so persistent ... and hasn't contacted you for help ... is suspicious in itself.)
But the point is that if you don't have any way to contact this user, fixing the issue in general is unlikely to help them directly. And right now you don't have enough information to know what the problem really is.
Firstly I'll explain my code in a nutshell
A config file is present which has a list of ID's. In a for loop these ID's are read one at a time and a list of JSON Structures are created. If everything goes well without any exception (e.g. entire data not being present) they are pushed into a database.
Coming to my question... For each ID, there are a bunch of business rules which are being executed. I've coded in such a way that even if any of the expected data is missing or if the business rules fails at any point of time I'm not inserting the data into the DB. The processing for that ID stops there, error message is written to a log file and then proceeding with the next ID. Can this be defined as a Fail Fast Design Patter is my question.
As answered at CodeRanch: http://www.coderanch.com/t/663709/java/java/Fail-Fast-Design-Pattern
As is often the case the answer is "it depends". It depends on what you consider a failure.
Let's say that an empty config file is an error condition that you want to know about. A "fail fast" approach might be to throw an exception immediately after detecting the file is empty, thus halting the operation at the point of failure. The alternative might be to continue with the processing and end up writing no data into the database. At this point you have a bug in that there's no data in the database. Why is that? Did the database insert code fail? Is there a problem with the database itself? Did the file processing fail? Is the data corrupt? You're now on a bug hunt to track down the error and you have lots of possibilities.
For processing each of the ID's using some business rule. If the requirement is that all ID's are valid then to abort processing upon finding an invalid ID would be the "fail fast" approach. However, in your example you appear to be simply ignoring invalid data and only writing valid data to the database. This is not failing in any way at all, rather you are making a decision on how to handle invalid data where you simply ignore them. So no, you are not failing fast in this design.
The purpose of "fail fast" is to abort the program at the exact point of failure citing the exact cause of failure in order to avoid entering into a running but invalid state.
This question already has answers here:
Handling passwords used for auth in source code
(7 answers)
Closed 7 years ago.
I'm writing a Java class that connects to a server and reads messages in a given queue.
I would like to protect the username and password, which, right now, appear as plain text in the source code.
What I'm wondering, is, what is a good way to do this? If I encrypt the username and password in a text file, won't I need to store the key, in plain text, in any source code that accesses this file? And then anyone else who decides to use my class will be able to gain access to these fields.
There is no prompt where someone can enter the key, either, as this class will autonomously be used by the system.
EDIT: this will become a java lib file. But those can easily be decompiled and thus are basically the original class files anyway, right? And the people this is being protected from are fellow developers of other systems who will gain access to this lib file.
My End Goal: is to have the username and password strings not appear as plain text anywhere, and for them to be as difficult as possible to crack.
It is not possible to do this. Even if you encrypt the login/password and store it somewhere (may it be your class or an external file) you'd still need to save the encryption key somewhere in plain text. This is actually just marginally better than saving username/password in plain text, in fact I would avoid doing so as it creates a false sense of security.
So I'd suggest that your class takes username/password as a parameter and that the system which is using your class will have to care about protecting the credentials. It could do so by asking an end user to enter the credentials or store them into an external file which is only readable to the operating system user that your process is running as.
Edit: You might also think about using mechanisms such as OAuth which use tokens instead of passwords. Tokens have a limited life time and are tied to a certain purpose so they pose a good alternative to access credentials. So your end users could get an access token with their,say, Windows credentials, which is then used inside your class to call the protected service.
This is a classic authentication issue, except that here, Eve can wear Bob's skin like a suit. Is that stretching the metaphor? I'm not sure.
The short answer is that there is no true answer, because what you want is something that basically violates information theory, in that anything transmittable is copyable and thus anything accessible can be viewed as no-longer-unique. Even if you had a magic box, they could just yank out the magic box with some serious JVM hacking.
The long answer is that there are a few solutions that are almost pretty okay, by making it really quite darn hard. I suggest you read the article linked, acquaint yourself with the ideas behind SRP, the vulnerabilities the spec entails, and try to figure out how to get the right to use and implement it. The problem is still there though. It's that you want a system that ensures Bob can never become a flesh-chariot, or fall to the dark side.
Fundamentally, you're breaking the tenth law. I agree with Kork, there's no solution that really does what you want, because you're trying to solve a social problem with a technical feat, one that is quite nearly provably impossible.
There are a few ways of handling this problem. The challenge as you've noted is associating an account with this automated process. So, here are some of the possibilities (from least secure to more secure):
Encrypt the username and password with a calculated key.
The calculated key is based on something both the client and the server can infer (like machine name and IP address)
Associate an authentication token with the client (OAuth style).
The token is negotiated by a one time user interaction to set up the client
The negotiated token is used for all future requests
The negotiated token is only valid for that client on that machine using that user account (server uses socket info to determine the match)
Use multiple forms of authentication
OAuth style token
Calculated token based on time + secondary id (requires clients and servers to be synched to the same time server)
It is important to note that your security measures should be more restrictive than it is worth to crack. In short, if all the potential bad guy is only going to be able to get your food preferences of the day you might not need to be as vigilent as protecting something more high profile like a bank account. User names and paswords are not the only means of authentication.
It's not clear which code has to know the user name & password. Are these credentials just for the queue being read? If so, only the server code would need to know them. In that case, you could store them in a server file whose permissions allow only the server code to read them. The file permissions would then be enforced by the server operating system, which presuambly is much better at security than most programmers will ever be.
I know this question is long since abandoned, but I want to point out that of course you can do this by requiring typed credentials at runtime but only storing a hash of the password. Of course, it needs to be a really good hash. Use a standard one, don't make up your own. The whole point of a hash is that even if you plain text the hashed result, no one else will be able to come up with a string that yields that hash, even if they know how the hash is computed.
Of course users can try a brute force attack, and since they know the result they want they can run it fast, so you need to use a highly secure password.
I'm developing a web application, and facing some security problems.
In my app users can send messages and see other's (a bulletin board like app). I'm validating all the form fields that users can send to my app.
There are some very easy fields, like "nick name", that can be 6-10 alpabetical characters, or message sending time, which is sended to the users as a string, and then (when users ask for messages, that are "younger" or "older" than a date) I parse this with SimpleDateFormat (I'm developing in java, but my question is not related to only java).
The big problem is the message field. I can't restrict it to only alphabetical characters (upper or lowercase), because I have to deal with some often use characters like ",',/,{,} etc... (users would not be satisfied if the system didn't allow them to use these stuff)
According to this http://ha.ckers.org/xss.html, there are a lot of ways people can "hack" my site. But I'm wondering, is there any way I can do to prevent that? Not all, because there is no 100% protection, but I'd like a solution that can protect my site.
I'm using servlets on the server side, and jQuery, on the client side. My app is "full" AJAX, so users open 1 JSP, then all the data is downloaded and rendered by jQuery using JSON. (yeah, I know it's not "users-without-javascript" friendly, but it's 2010, right? :-) )
I know front end validation is not enough. I'd like to use 3 layer validation:
- 1. front end, javascript validate the data, then send to the server
- 2. server side, the same validation, if there is anything, that shouldn't be there (because of client side javascript), I BAN the user
- 3. if there is anything that I wasn't able to catch earlier, the rendering process handle and render appropriately
Is there any "out of the box" solution, especially for java? Or other solution that I can use?
To minimize XSS attacks important thing is to encode any field data before putting it back on the page. Like change > to > and so on. This would never allow any malicious code to execute when being added to the page.
I think you are doing lot of right things by white listing the data you expect for different fields. Beyond that for fields which can allow other characters which can be problematic encoding would fix the issue for you.
Further since you are using Ajax it gives you some protection as people cannot override values in URL parameters etc.
Look at the AntiSamy library. It allows you to define rulesets for your application, then run your user input through AntiSamy to clean it per your rules.
The easiest way is to do a simple replacement for the following
< with <
> with >
' with \'
That will solve most database vulnerability.