We're using Spring/Hibernate on a Websphere Application Server for AIX. On my Windows machine, the problem doesn't occur--only when running off AIX. When a user logs in with an account number, if they prefix the '0' to their login ID, the application rejects the login. In the DB2 table, the column is of numeric type, and there shouldn't be a problem converting '090....' to '90...'
Anyone else experience a problem like this? Both machines have Java v1.5.
To be more specific, the flow is FormView -> LoginValidator -> LoginController
In LoginValidator, the value of login is null with the prefixed 0. Without the 0, the value is what it should be (But again, this is only on the AIX environment--on 2 Windows environments it's fine). Here's the snippet of code where the object equals null..
public class LoginValidator implements Validator {
public boolean supports(Class clazz) {
return Login.class.equals(clazz);
}
#SuppressWarnings("all")
public void validate(Object obj, Errors errors) {
System.out.println("Inside LoginValidator");
Login login = (Login) obj;
//null value
System.out.println("Before conversion in Validator, store id = "
+ login.getStoreId());
}
}
I've also written this short Java program for constructing a Long from a String, and using the java binary that is packaged with WebSphere
public class String2Long {
public static void main(String[] args){
String a = "09012179";
String b = "9012179";
Long _a = new Long(a);
Long _b = new Long(b);
System.out.println(a + " => " + _a); //09012179 => 9012179
System.out.println(b + " => " + _b); //9012179 => 9012179
System.out.println("_a.equals(_b) " + _a.equals(_b)); //_a.equals(_b) true
}
}
SOLUTION
Well there's an awful lot of things going on there. You really need to try to isolate the problem - work out what's being sent to the database, what's being seen by Java etc.
Try to pin it down in a short but complete program which just shows the problem - then you'll be in a much stronger position to file a bug or fix your code.
SOLUTION
A co-worker did some research on Spring updates, and apparently this error was correct in v. 2.5.3:
CustomNumberEditor treats number with leading zeros as decimal (removed unwanted octal support while preserving hex)
We were using Spring 2.0.5. We simply replaced the jars with Spring 2.5.4, and it worked as it should have!
Thanks to everyone for your help/assistance. We will make use of Unit tests in the future, but this just turned out to be a Spring bug.
Trace through the program following the path of the String all the way to database and make unit tests for every single method on that path. And don't just take the shortest possible route here, make multiple unit tests with different inputs and expected outputs to really see what went possibly wrong. Assuming you don't find any errors, run the same unit tests on the other computer and you should be able to pinpoint the bug. From the top of my head I'd assume it may have something to do with case sensitivity but there really is no way to be sure.
Next time, use TDD.
I don't know much about Java, but this might happen the string is interpreted as octal string because of the leading "0".
You can probably work around this using Long.parseLong(a, 10).
Related
I am working on fixing Veracode issues in my application. Veracode has highlighted the flaw "External Control of File Name or Path (CWE ID 73) " in below code.
Thread.currentThread().getContextClassLoader().getResourceAsStream(lookupName)
How do I validate the parameter? If I need to use below ESAPI validation, then what is the exact parameter I should be passing in getValidFileName() method. Currently I am passing the parameters as below.
ESAPI.validator().getValidFileName(lookupName, lookupName,
ESAPI.securityConfiguration().getAllowedFileExtensions(), false);
Correct me whether I am following the right approach for fixing this issue.
There are several suggestions at: https://community.veracode.com/s/article/how-do-i-fix-cwe-73-external-control-of-file-name-or-path-in-java
You can use hardcoded values, if these files are stored in the server side.
(i.e.: in a HashMap).
Another solution is to use a custom validator (from veracode page) :
// GOOD Code
String extension = request.getParameter("extension");
File f = new File(buildValidAvatarPath(extension))
#FilePathCleanser
public String buildValidAvatarPath(extension) {
String[] allowedExtensions = new String[]{"jpg","gif","png"};
String extension = "png"; // Default extension
for (String allowedExtension: allowedExtensions) {
if (allowedExtension.equals(request.getParameter("extension"))) {
extension = request.getParameter("extension");
}
}
// See "Note on authorization"
User user = getCurrentUser();
if (!userMayAccessFile(user, path)) {
throw new AuthorizationException("User may not access this file", user);
}
File(configPath + "avatar." + extension)
return path;
}
Okay, so the problem is that you are allowing user-control of that file path. Imagine its on a UNIX box and they enter:
../../../../../../../etc/shadow
Whatever user privileges are granted to the user running that java Thread is possible to expose to the user in question. I don't know what processing is going on in your application, but the danger is that you need to prevent user control of that lookup variable.
The call you're making is consistent with the single test in ValidatorTest.java, which is definitely a deficiency in code coverage on our behalf.
Now, there's an excellent chance that even if you use this call that Veracode might still flag it: the default file list in ESAPI.properties will need to be either truncated for your use case, or you'll have to create your own Validator rule for legal file extensions for your specific use case.
Which brings up the next bit: There's a lot of mischief that can happen in regards to file uploads.
In short, to be actually secure about file uploads will require more than what ESAPI currently offers, which is unfortunately, only an extension check. In your particular case, make sure you try some directory traversal attacks. And use that OWASP link to help analyze your application.
Given that the OP wants to clear the issue in Veracode, you would want to chain a couple calls:
ESAPI.validator().getValidDirectoryPath() and ESAPI.Validator.getValidFileName()
But be sure you've properly truncated the extension list in HttpUtilities.ApprovedUploadExtensions in validator.properties as the default list is too permissive, at least until we release 2.1.0.2.
I have to stress however that even with this particular combination there is absolutely nothing ESAPI does to prevent a user from renaming "netcat.exe" to "puppies.xlsx" and bypassing your validation check, that's why the rant on the first part of this answer.
ESAPI's file validation is NOT secure, it's quite simply better than nothing at all.
Doing this correctly requires more work than just using 1-2 calls to ESAPI.
DISCLAIMER: as of this writing I am the project co-lead for ESAPI.
You can change file name by sanitizing it as below code snippet:
private static String sanitizeFileName(String name) {
return name
.chars()
.mapToObj(i -> (char) i)
.map(c -> Character.isWhitespace(c) ? '_' : c)
.filter(c -> Character.isLetterOrDigit(c) || c == '-' || c == '_' || c == ':')
.map(String::valueOf)
.collect(Collectors.joining());
}
So, back again
I have a JHipster generated project which uses an elasticsearch java client embedded in spring boot.
I have recently done some major changes to the datasets since we've been migrating a whole new bunch of data from different repositories
When deploying the application it all works fine, all SearchRepositories are loaded with no problem and all search capabilities roll smooth
The issues come when running from the test environment. There have been no changes what so ever to the application-test.yml file nor to the elasticsearch java config file.
We have some code which updates the indices and I've run it several times, it seems to update the clusters indices just fine, but where I'm suffering is in the target folder, it just won't create the new indices
There are 12 indices that I cannot get in to the target folder when running in test mode, however, only 5 of them fail in their ResourceIntTest because of the error mentioned in the title
I don't want to fill this post with hundreds of irrelevant lines of code, so suffice for now to include the workaround that helps test not to fail:
When in the initTest of the 5 failing test cases, if I write the following line (obviously changing the class name in each different case):
surveyDataQualitySearchRepository.save(surveyDataQualityRepository.findAll());
Then the index will create itself and the testcase will not fail, however this shouldn't be necessary to do manually, it should be created when the resetIndex method in the IndexReinitializer class is called upon deployment
resetIndex:
#PostConstruct
public void resetIndex() {
long t = currentTimeMillis();
elasticsearchTemplate.deleteIndex("_all");
t = currentTimeMillis() - t;
logger.debug("ElasticSearch indexes reset in {} ms", t);
}
Commenting this piece of code also allows all indices to be loaded, but it should not be commented as this serves as an updater for the indices, plus it works fine in an old version of the application which is still pointing to the old dataset
All help will be very welcome, I've been on this almost a full day now trying to understand where the error is coming from, I'm also more than happy to upload any pieces of code that may be relevant to anyone willing to help here.
EDIT To add code for the indices rebuild as requested via comments
#Test
public void synchronizeData() throws Exception{
resetIndex();
activePharmaIngredientSearchRepository.save(activePharmaIngredientRepository.findAll());
countrySearchRepository.save(countryRepository.findAll());
dosageUnitSearchRepository.save(dosageUnitRepository.findAll());
drugCategorySearchRepository.save(drugCategoryRepository.findAll());
drugQualityCategorySearchRepository.save(drugQualityCategoryRepository.findAll());
formulationSearchRepository.save(formulationRepository.findAll());
innDrugSearchRepository.save(innDrugRepository.findAll());
locationSearchRepository.save(locationRepository.findAll());
manufacturerSearchRepository.save(manufacturerRepository.findAll());
outletTypeSearchRepository.save(outletTypeRepository.findAll());
publicationSearchRepository.save(publicationRepository.findAll());
publicationTypeSearchRepository.save(publicationTypeRepository.findAll());
qualityReferenceSearchRepository.save(qualityReferenceRepository.findAll());
reportQualityAssessmentAssaySearchRepository.save(reportQualityAssessmentAssayRepository.findAll());
//rqaaQualitySearchRepository.save(rqaaQualityRepository.findAll());
rqaaTechniqueSearchRepository.save(rqaaTechniqueRepository.findAll());
samplingTypeSearchRepository.save(samplingTypeRepository.findAll());
//surveyDataQualitySearchRepository.save(surveyDataQualityRepository.findAll());
surveyDataSearchRepository.save(surveyDataRepository.findAll());
techniqueSearchRepository.save(techniqueRepository.findAll());
tradeDrugApiSearchRepository.save(tradeDrugApiRepository.findAll());
tradeDrugSearchRepository.save(tradeDrugRepository.findAll());
publicationDrugTypesSearchRepository.save(publicationDrugTypesRepository.findAll());
wrongApiSearchRepository.save(wrongApiRepository.findAll());
}
private void resetIndex() {
long t = currentTimeMillis();
elasticsearchTemplate.deleteIndex("_all");
t = currentTimeMillis() - t;
logger.debug("ElasticSearch indexes reset in {} ms", t);
}
Please try to update to the latest version of spring-data-elasticsearch
I'll put the question upfront:
Is there a logger available in Java that does encryption(preferably 128-bit AES or better)?
I've done a lot of searching for this over the last couple of days. There's a few common themes to what I've found:
Dissecting information between log4j and log4j2 is giving me headaches(but mostly unrelated to the task at hand)
Most threads are dated, including the ones here on SO. This one is probably the best I've found on SO, and one of the newer answers links to a roll-your-own version.
The most common answer is "roll-your-own", but these answers are also a few years old at this point.
A lot of people question why I or anyone would do this in Java anyway, since it's simple enough to analyze Java code even without the source.
For the last point, it's pretty much a moot point for my project. We also use a code obfuscator and could employ other obfuscation techniques. The point of using encryption is simply to raise the bar of figuring out our logs above "trivially easy", even if it's only raised to "mildly time-consuming". A slightly relevant aside - the kind of logging we're going to encrypt is intended merely for alpha/beta, and will likely only include debug, warn, and error levels of logging(so the number of messages to encrypt should be fairly low).
The best I've found for Log4j2 is in their documentation:
KeyProviders
Some components within Log4j may provide the ability to perform data encryption. These components require a secret key to perform the encryption. Applications may provide the key by creating a class that implements the SecretKeyProvider interface.
But I haven't really found anything other than wispy statements along the lines of 'plug-ins are able of doing encryption'. I haven't found a plug-in that actually has that capability.
I have also just started trying to find other loggers for Java to see if they have one implemented, but nothing is really jumping out for searches like 'java logging encryption'.
Basically log encryption is not best practise there are limited situations where you can need this functionality. As mainly people which have access to logs have also access to JVM, and in JVM all the logs are at least generated as Strings so even if you encrypt them in the log file or console the real values will be available in JVM String Pool, so if anyone will every need to hack your logs it will be as easy as have a look in string pool.
But anyway if you need a way to encrypt the logs, and as there is no generic way for this, the best way in my opinion is to go with Aspect J. This will have minimum impact on you sources, you will write code as you have done before, but the logs will be encrypted. Following is a simple application code which will encrypt all the logs from all the compiled sources using Aspctj, and Slf4j as logging facade and Log4j2 as logging implementation.
The simple class which logs the "Hello World"
public class Main {
private static final transient Logger LOG = LoggerFactory
.getLogger(Main.class);
public static void main(String[] args) {
LOG.info("Hello World");
LOG.info("Hello {0}", "World 2");
}
}
Aspect which encrypts (in this case just edits the text)
#Aspect
public class LogEncryptAspect {
#Around("call(* org.slf4j.Logger.info(..))")
public Object encryptLog (ProceedingJoinPoint thisJoinPoint) throws Throwable{
Object[] arguments = thisJoinPoint.getArgs();
if(arguments[0] instanceof String){
String encryptedLog = encryptLogMessage ((String) arguments[0], arguments.length > 1 ? Arrays.copyOfRange(arguments, 1, arguments.length) : null);
arguments[0] = encryptedLog;
}
return thisJoinPoint.proceed(arguments);
}
// TODO change this to apply some kind of encryption
public final String encryptLogMessage (String message, Object... args){
if(args != null){
return MessageFormat.format(message, args) + " encrypted";
}
return message + " encrypted";
}
}
The output is :
[main] INFO xxx.Main - Hello World encrypted
[main] INFO xxx.Main - Hello World 2 encrypted
Part of a program I am working on requires looking up preprocessor macros by name, and then getting their values. I opted to use the CDT Indexer API. In order to make sure I am on the right track, I wrote a test method that does nothing but create a simple C file and confirm that it can find certain symbols in the index. However, I failed to get that test to run properly. Attempting to use IIndex.findBindings(char[], IndexFilter, IProgressMonitor) returns empty arrays for symbols that I know exist in the AST because they are part of the example file in the test method.
I can't post the exact test method because I use some custom classes and it would be overkill to post all of them, so I will just post the important code. First, my example file:
final String exampleCode =
"#define HEAVY 20\n" +
"#define TEST 5\n" +
"void function() { }\n" +
"int main() { return 0; }\n";
IFile exampleFile = testProject.getFile("findCodeFromIndex.c");
exampleFile.create(new ByteArrayInputStream(exampleCode.getBytes("UTF-8") ), true, null);
I have a custom class that automatically gets the IASTTranslationUnit from that file. The translation unit is fine (I can see the nodes making up everything except the macros). I get the index from that AST, and the code I use to look up in the index is
try {
index.acquireReadLock();
returnBinding = index.findBindings(name.toCharArray(), IndexFilter.ALL, null);
... catch stuff...
} finally {
index.releaseReadLock();
}
Where 'name' is going to be either "HEAVY", "TEST", or "function". None of them are found, despite existing in the example test c file.
I am guessing that the issue is the index is not rebuilt, which causes findBindings to return an empty array even if I know the given variable name exists in the AST.
My current attempt to start up the indexer looks like this:
final ICProject cProject = CoreModel.getDefault().getCModel().getCProject(testProject.getName());
CCorePlugin.getIndexManager().reindex(cProject);
CCorePlugin.getIndexManager().joinIndexer(IIndexManager.FOREVER, new NullProgressMonitor() );
Question Breakdown:
1) Is my method for searching the index sound?
2) If the issue is the index needing to be rebuilt, how should I properly force the index to be up to date for my test methods? Otherwise, what exactly is the reason I am not resolving the bindings for macros/functions I know exist?
I solved my own issue so I will post it here. I was correct in my comment that the lack of the project being a proper C project hindered the Indexer from working properly, however I also discovered I had to use a different method in the indexer to get the macros I needed.
Setting up the test enviornment:
Here is the code I have that creates a basic C project. The only purpose it serves is to allow the indexer to work for test methods. Still, it is large:
public static IProject createBareCProject(String name) throws Exception {
IProject bareProjectHandle = ResourcesPlugin.getWorkspace().getRoot().getProject(name);
IProjectDescription description =
bareProjectHandle.getWorkspace().newProjectDescription("TestProject");
description.setLocationURI(bareProjectHandle.getLocationURI() );
IProject bareProject =
CCorePlugin.getDefault().createCDTProject(description, bareProjectHandle, new NullProgressMonitor() );
IManagedBuildInfo buildInfo = ManagedBuildManager.createBuildInfo(bareProject);
IManagedProject projectManaged =
ManagedBuildManager
.createManagedProject(bareProject,
ManagedBuildManager.getExtensionProjectType("cdt.managedbuild.target.gnu.mingw.exe") );
List<IConfiguration> configs = getValidConfigsForPlatform();
IConfiguration config =
projectManaged.createConfiguration(
configs.get(0),
ManagedBuildManager.calculateChildId(configs.get(0).getId(), null));
ICProjectDescription cDescription =
CoreModel.getDefault().getProjectDescriptionManager().createProjectDescription(bareProject, false);
ICConfigurationDescription cConfigDescription =
cDescription.createConfiguration(ManagedBuildManager.CFG_DATA_PROVIDER_ID, config.getConfigurationData() );
cDescription.setActiveConfiguration(cConfigDescription);
cConfigDescription.setSourceEntries(null);
IFolder srcFolder = bareProject.getFolder("src");
srcFolder.create(true, true, null);
ICSourceEntry srcFolderEntry = new CSourceEntry(srcFolder, null, ICSettingEntry.RESOLVED);
cConfigDescription.setSourceEntries(new ICSourceEntry[] { srcFolderEntry });
buildInfo.setManagedProject(projectManaged);
cDescription.setCdtProjectCreated();
IIndexManager indexMgr = CCorePlugin.getIndexManager();
ICProject cProject = CoreModel.getDefault().getCModel().getCProject(bareProject.getName() );
indexMgr.setIndexerId(cProject, IPDOMManager.ID_FAST_INDEXER);
CoreModel.getDefault().setProjectDescription(bareProject, cDescription);
ManagedBuildManager.setDefaultConfiguration(bareProject, config );
ManagedBuildManager.setSelectedConfiguration(bareProject, config );
ManagedBuildManager.setNewProjectVersion(bareProject);
ManagedBuildManager.saveBuildInfo(bareProject, true);
return bareProject;
}
As I discovered when debugging, it is indeed important to set proper configurations and descriptions as the indexer was postponed so long as the project didn't have those features set. To get the configurations for the platform as a starting point for an initial configuration:
public static List<IConfiguration> getValidConfigsForPlatform() {
List<IConfiguration> configurations =
new ArrayList<IConfiguration>();
for (IConfiguration cfg : ManagedBuildManager.getExtensionConfigurations() ) {
IToolChain currentToolChain =
cfg.getToolChain();
if ( (currentToolChain != null ) &&
(ManagedBuildManager.isPlatformOk(currentToolChain) ) &&
(currentToolChain.isSupported() ) ) {
configurations.add(cfg);
}
}
return configurations;
}
This basically answers the second part of the question, and thus I can create a c project for the purposes of testing code using the index. The testing code still needs to do some work.
Testing Code
I create files in the the "src" folder in the project (created in the above code), and I either have to name them .c, or if I want to name them .h have them included by some .c file (otherwise the indexer won't see them). Finally, I can populate the files with some test code. To answer number 1,
I need to block on both auto refresh jobs in Eclipse and then the index:
public static void forceIndexUpdate(IProject project) throws Exception {
ICProject cProject = CoreModel.getDefault().create(project);
Job.getJobManager().join(ResourcesPlugin.FAMILY_AUTO_REFRESH, null);
CCorePlugin.getIndexManager().reindex(cProject);
CCorePlugin.getIndexManager().joinIndexer(IIndexManager.FOREVER, new NullProgressMonitor() );
assertTrue(CCorePlugin.getIndexManager().isIndexerIdle() );
assertFalse(CCorePlugin.getIndexManager().isIndexerSetupPostponed(cProject));
}
After I change the files in the project. This makes sure Eclipse is refreshed, and then makes sure the indexer completes without being postponed. Finally, I can run tests depending on the indexer.
And the last point, I was wrong about using IBinding. The correct way in which I was able to get the macros was using the method IIndex.findMacros(char[] name, IndexFilter filter, IProgressMonitor monitor)
I hope this helps at least someone out there. I would also appreciate it if there was some feedback regarding the validity of this solution, as this is simply the first solution I managed to create that worked. Just to confirm, I am not testing the indexer itself, but rather code I wrote that uses the indexer and I want to test it under as realistic conditions as I can given how critical it is.
Our web app needs to be made PCI compliant, i.e. it must not store any credit card numbers. The app is a frontend to a mainframe system which handles the CC numbers internally and - as we have just found out - occasionally still spits out a full CC number on one of its response screens. By default, the whole content of these responses are logged at debug level, and also the content parsed from these can be logged in lots of different places. So I can't hunt down the source of such data leaks. I must make sure that CC numbers are masked in our log files.
The regex part is not an issue, I will reuse the regex we already use in several other places. However I just can't find any good source on how to alter a part of a log message with Log4J. Filters seem to be much more limited, only able to decide whether to log a particular event or not, but can't alter the content of the message. I also found the ESAPI security wrapper API for Log4J which at first sight promises to do what I want. However, apparently I would need to replace all the loggers in the code with the ESAPI logger class - a pain in the butt. I would prefer a more transparent solution.
Any idea how to mask out credit card numbers from Log4J output?
Update: Based on #pgras's original idea, here is a working solution:
public class CardNumberFilteringLayout extends PatternLayout {
private static final String MASK = "$1++++++++++++";
private static final Pattern PATTERN = Pattern.compile("([0-9]{4})([0-9]{9,15})");
#Override
public String format(LoggingEvent event) {
if (event.getMessage() instanceof String) {
String message = event.getRenderedMessage();
Matcher matcher = PATTERN.matcher(message);
if (matcher.find()) {
String maskedMessage = matcher.replaceAll(MASK);
#SuppressWarnings({ "ThrowableResultOfMethodCallIgnored" })
Throwable throwable = event.getThrowableInformation() != null ?
event.getThrowableInformation().getThrowable() : null;
LoggingEvent maskedEvent = new LoggingEvent(event.fqnOfCategoryClass,
Logger.getLogger(event.getLoggerName()), event.timeStamp,
event.getLevel(), maskedMessage, throwable);
return super.format(maskedEvent);
}
}
return super.format(event);
}
}
Notes:
I mask with + rather than *, because I want to tell apart cases when the CID was masked by this logger, from cases when it was done by the backend server, or whoever else
I use a simplistic regex because I am not worried about false positives
The code is unit tested so I am fairly convinced it works properly. Of course, if you spot any possibility to improve it, please let me know :-)
You could write your own layout and configure it for all appenders...
Layout has a format method which makes a String from a loggingEvent that contains the logging message...
A better implementation of credit card number masking is at http://adamcaudill.com/2011/10/20/masking-credit-cards-for-pci/ .
You want to log the issuer and the checksum, but not the PAN (Primary Account Number).