Develop Reference

md5_file() PHP different that Java MD5 for txt files

I'm trying to develop a file updater for some files in a folder, to Sync an FTP server with a local folder, using Java on the client and PHP on the server side.
On the server side, I'm calculating the md5_file($filename) for the file and returning every of them on a JSON.
On Java, I'm checking first if the file exists in the local folder. If the file exists, then I check for the MD5 checksum to see if the file is exactly the same as the online one.
The MD5 is not matching when checking .txt or .lua files. It's ok when checking other file types, as .dds texture files.
The MD5 I'm using on Java is this:
private String md5(File f) throws FileNotFoundException, NoSuchAlgorithmException {
MessageDigest digest = MessageDigest.getInstance("MD5");
InputStream is = new FileInputStream(f);
byte[] buffer = new byte[8192];
int read = 0;
try {
while( (read = is.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
return output;
}
catch(IOException e) {
throw new RuntimeException("Unable to process file for MD5", e);
}
finally {
try {
is.close();
}
catch(IOException e) {
throw new RuntimeException("Unable to close input stream for MD5 calculation", e);
}
}
}
As an example, for a description.lua file, with the following contents:
livery = {
{"KC-130_fusel", 0 ,"KC-130_map_fus",false};
{"KC-130_wing", 0 ,"KC-130_map_wingS",false};
{"KC-130_wing_2", 0 ,"KC-130_map_wings_2",false};
{"KC-130_notes", 0 ,"KC-130_notes_empty",true};
{"KC-130_FPod", 0 ,"kc-130_map_drg",false};
}
name = "Spain ALA 31 TK.10-06"
countries = {"SPN"} -- and any others you want to add
PHP md5_file($filename) = d0c32f9e38cc6e1bb8b54a6aca4a0190
JAVA md5(File) = 08bf57441b904c69e9ce3ca02a9257c7
I've been trying to find a relation between those two codes to see what's making the difference, but have not find any. I have checked like 10 md5 scripts for Java and all of them give the same result.
Is there any way I can fix this?
EDIT: Solution given on first comment: Change the Transfer type on the FTP Client to Binary to avoid changing txt files to ASCII encoding, changing their length and md5.

Unable to load and store keystore from p12 inside J2EE container

I was trying to load the keystore from a p12 file , the behaviour is highly inconsistent where in the keystore.aliases().nextElement() gives proper alias once and displaying CN in other cases. In the later case I am not able to store the keystore (using keystore.store) and the output stream is empty .
Below is the code snippet. Let me know if I overlooked anything.
// the main code where i am facing issue
private byte[] generateKeyStoreData(String appName, Map<String, String> credentials)
throws ApplicationException {
try {
if (!credentials.containsKey(KEYSTORE)) {
throw new NullPointerException("No keystore provided");
}
if (!credentials.containsKey(KEYSTORE_PASSWORD)) {
throw new NullPointerException("No keystore password provided");
}
String keystoreStr = credentials.get(KEYSTORE);
char[] keystorePass = credentials.get(KEYSTORE_PASSWORD).toCharArray();
// I have printed the base64 string here and tried loading inside a standalone code
and it is working. The method is below
InputStream keystoreIs = base64stringToInputStream(keystoreStr);
KeyStore keyStore = KeyStore.getInstance("PKCS12");
keyStore.load(keystoreIs, keystorePass);
// I printed the keyStore.aliases().nextElement() which returns correct alias "omss"
// but returns CN in cases where it fails.
ByteArrayOutputStream keyStoreOut = new ByteArrayOutputStream();
keyStore.store(keyStoreOut, keystorePass);
// I printed the keystoreOut.toByteArray() and it is empty in failing cases
return keyStoreOut.toByteArray();
} catch (Exception e) {
// exception
}
}
// the conversion code from base64string to bytearrayinputstream
private InputStream base64stringToInputStream(String str) {
byte[] ba = DatatypeConverter.parseBase64Binary(str);
return new ByteArrayInputStream(ba);
}
//--------------------------------------------------------------------
// Below is api which calls the generateKeystore
//-------------------------------------------------------------------
// We get the inputstream from the uploaded p12 file and the below api is called
public void createKeystore(InputStream certFile,
char[] password) {
Util.nullCheck(certFile,
"Certificate File cannot be null or empty");
Util.nullCheck(password, "Password Cannot be null");
try {
// the method is below
byte[] raw = toByteArray(certFile);
// converting to base64 string
String base64encodedString = DatatypeConverter
.printBase64Binary(raw);
//....... we create a map of keystore string and password
// and the call is made to generateKeystore method above
}
catch(Exception e){
}
// the inputstream is converted to bytearray inputstream
private static byte[] toByteArray(InputStream is) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int reads = is.read();
while (reads != -1) {
baos.write(reads);
reads = is.read();
}
return baos.toByteArray();
}

Looks like the keystore.load() is not using "SunJSSE" as the default keystore provider in my j2ee environment rather it was using oraclepki provider. Now that i am loading keystore.load(Is,"SunJSSE") it is able to load properly.

Different digital signature for the same message when Java webservice deployed on different operating systems that run the same application server

I am programming a webservice that requires a digital signature for the sent and received XML messages, which is programmed in JAVA using bouncycastle library for signing the sent messages and verifying the received ones. The Webservice is deployed on Weblogic application server. I noticed that the same exact message produces different signature when the webservice is deployed on a Windows server than when deployed on a Linux server.
I have tried deploying the same exact code on different machines that has Weblogic application server and the ones that are running on Windows produce the same signature while the ones running on Linux produce another one. The entity that is going to consume the webservice successfully verified the signature from the messages that are produced on the windows server but weren't able to do the same for the ones produced on the Linux server (they use Windows as an operating system on their servers).
Both the private key and the certificate used in signing and verifying the digital signatures are read from files using FileInputStream. To make sure the problem is not with the certificates I tried reading the certificates from a hard coded String inside the code and the same problem happens. I also wrote a simple webservice that verifies the signature that I produce, the ones produced on Windows are successfully verified on the verifier deployed on windows while the ones produced on Linux aren't and vice-versa.
Taking into notice that the XML messages are reformatted before the signing process (all white spaces are removed and then the XML gets reformatted), is there any suggestions or ideas about the reason for this as I am running out of ideas? Does it have any thing to do with the character encoding or the way line feed and carriage return are handled in different operating systems?
Thanks.
Edit: Here is a simplified code for the signing process that follows the same logic that I used in my original code, the imports are not added and no exception handling in this code and most of the functions has been in-lined to make it simpler, also the class "Utils" that is used in the XML formatting process is not provided for the sake of simplicity but there is a description written beside each call for a function from that class, I will provide that class and the original code if needed:
public class Initializer {
private static Signer signer = null; // a reference to the class that will do the signing process
private static XPathExpression mssgBody;
private static XPathExpression signatureNodePath;
static {
try {
Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
KeyStore keystore;
keystore = KeyStore.getInstance("PKCS12", "BC");
keystore.load(new FileInputStream("/home/path/path/MyKeyPair.p12"), "123".toCharArray()); //loading My keypair to a keystore (only the private key will be used)
Key key = keystore.getKey("CertificateAlias", "Password".toCharArray());
java.security.cert.Certificate cert;
cert = keystore.getCertificate("CertificateAlias");
PrivateKey privateKey = (PrivateKey)key;
signer = new Signer(privateKey, "UTF-16LE", "SHA256withRSA"); //Creating a new Signer object and providing the private key and the message encoding and the signature algorithm
mssgBody = Utils.compileExpression(null, "//MsgBody", null); // set the path for the message body that will be signed
signatureNodePath = Utils.compileExpression(null, "//Signature", null); // set the path for the signature node where the signature will be put
} catch (KeyStoreException e) {
} catch (NoSuchProviderException e) {
} catch (IOException e) {
} catch (NoSuchAlgorithmException e) {
} catch (CertificateException e) {
} catch (UnrecoverableKeyException e) {
} catch (XPathExpressionException e) {
}
}
public static String sign(Node request) throws XPathExpressionException, Exception { //the function that will be called by the web service
String mssgBodyString = Utils.convertNode2String(Utils.extractNodeList(request, mssgBody).item(0)); // Extract the message body and convert it into a String
mssgBodyString = Utils.getFromattedXML(mssgBodyString); //format the string containing XML data, each new opening or closing tag will be on a new line and each new nested tag will be indented by two spaces
String signature = signer.sign(mssgBodyString); // Return signature
return signature;
}
}
public class Signer{
private KeyStore signKeyStore;
private PrivateKey privateKey;
/** Final Instances **/
private String MSG_ENCODER;
private String SIGNATURE_ALG;
public Signer(PrivateKey prk, String msgEncoder, String signatureAlg) {
privateKey = prk;
MSG_ENCODER = msgEncoder; //"UTF-16LE"
SIGNATURE_ALG = signatureAlg; //"SHA256withRSA"
}
/******************* Signing process ************************/
public String sign(final String msg) throws Exception { // the function that will be called form the initializer class
return sign(msg, SIGNATURE_ALG, MSG_ENCODER);
}
/**
* #param msgStr: String for which digital signature will be calculated
* #param signatureAlg: Name of requested Signature Algorithm
* #param msgEncoder: Encoder used encoding the input message to bytes array
* #param sigEncoder: Type of byte encoder (Hex, Base64, ...)
*
* #return strSignature: String Formatted Signature for passed string data, based on passed encoder
* #throws Exception
*/
public String sign(final String msgStr, final String signatureAlg, final String msgEncoder) throws Exception {
byte[] signature;
String strSignature = "";
signature = this.sign(msgStr.getBytes(msgEncoder));
// encode the result signature based on the specified encoder
strSignature = encode(signature);
return strSignature;
}
public byte[] sign(final byte[] data) throws Exception {
Signature signer;
byte[] signature = null;
try {
signer = Signature.getInstance(SIGNATURE_ALG, new BouncyCastleProvider());
signer.initSign(privateKey); // calling initSign from java.security Package
signer.update(data, 0, data.length);
signature = signer.sign();
} catch (Exception e) {
e.printStackTrace();
throw new Exception("Message Signing failed; ", e);
}
return signature;
}
/*** encoding / decoding****/
/**
* encodes byte array into String using Base64
* #param bytes
* #return encoded String
*/
private String encode(byte[] bytes) {
return new String(Base64.encode(bytes));
}
/**
* Decode String using Base64
* #param str
* #return
*/
private byte[] decode(String str) {
return Base64.decode(str);
}
}

As it turns out, the problem was that how the new line is handled differently between Windows and Linux; in Windows the new line adds carriage return and line feed while in Linux it adds only Line feed.
The code that I'm working on requires the XML to be formatted with new line and two indentation spaces between the parent tag and its child tags, I was using the following code inside my "convertNode2String" method in the Utils Class:
if (indent){
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2");
}
The previous code adds 0x0D0A at each new line in Windows while in Linux it adds 0x0A at each new line. That makes it obvious why the signature produced is different in Windows than in Linux.
Since the entity that will consume the webservice uses Windows as their operating system of choice, the solution to this problem is to change the new line in the Linux environment from LF only to CRLF when the method is called, we can do this by using java.lang.System.setProperty(String key, String value) method for the "line.separator" property then return it back to its original status at the end of the method.
To make it not specific to a certain OS or if the default is not LF only we should save the current "line.separator" value using the "getProperty" method.
String defaultSeparator = System.getProperty("line.separator");
String WindowsSeparator = "\r\n";
System.setProperty("line.separator", WindowsSeparator);
And after the formatting of the XML is done:
System.setProperty("line.separator", defaultSeparator);
Hope someone finds this useful.

org.jasypt.exceptions.EncryptionOperationNotPossibleException in Tomcat

I'm using the Jasypt encryption library to encrypt/decrypt some text. This code is embedded in a WAR file and deployed to a server.
When running locally, and in unit tests, the encrypt/decrypt cycle works perfectly. I use Jetty to develop the application. The code works perfectly in that server. For some reason, deploying to Tomcat breaks it with the following exception:
FYI, I have the strong encryption libraries installed in both my local and server environments and I'm using the latest 1.6 version (patch level 25).
org.jasypt.exceptions.EncryptionOperationNotPossibleException
The exception has no message.
The code is fully symmetric. I pasted it here for examination. Here are the relevant bits:
I found one old Nabble post where a user had a very similar problem. Code worked everywhere except inside Tomcat. No solution was given.
Any insights would be most appreciated.
**Update: ** Running in Tomcat on my local system, it appears to work. So there's something about my server. On the server, I'm using a 64-bit JVM on Windows Server 2008. I'm using a 32-bit JVM locally (due to my system being a bit older). I wonder if this has something to do with the issue.
public void initializeService() {
binaryEncryptor = new BasicBinaryEncryptor();
binaryEncryptor.setPassword(keyBase64);
}
#Override
public <T extends Serializable> String simpleEncrypt(T objectToEncrypt) throws EncryptionException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(objectToEncrypt);
byte[] bytes = binaryEncryptor.encrypt(bos.toByteArray());
return new String(Base64.encodeBase64(bytes));
} catch (IOException e) {
LOGGER.error("failed to encrypt String: " + e.getMessage());
throw new EncryptionException(e.getMessage(), e);
} catch (Exception e) {
LOGGER.error("failed to encrypt String: " + e.getMessage());
throw new EncryptionException(e.getMessage(), e);
}
};
#SuppressWarnings("unchecked")
#Override
public <T> T simpleDecrypt(String objectToDecrypt) throws EncryptionException {
try {
byte[] bytes = Base64.decodeBase64(objectToDecrypt);
byte[] decryptedBytes = binaryEncryptor.decrypt(bytes);
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(decryptedBytes));
T object = (T)ois.readObject();
return object;
} catch (IOException e) {
LOGGER.error("failed to decrypt String: '" + objectToDecrypt + "', mesage = " + e.getMessage());
throw new EncryptionException(e.getMessage(), e);
} catch (Exception e) {
LOGGER.error("failed to decrypt String: '" + objectToDecrypt + "', mesage = " + e.getMessage());
throw new EncryptionException(e.getMessage(), e);
}
}

Here is a link to the docs: http://www.jasypt.org/faq.html#i-keep-on-receiving-encryption-operation-not-possible
Is encryption and decryption config identical
Check to make sure table columns are large enough
Base64 encoding and urlencoding can conflict, so it has to be done just right.

#biniam_Ethiopia
I would have commented your answer but I have not enough reputation, so I write my own answer:
I had a very similiar problem, but in my case it was because of changing the encryption algorithm (PBEWithMD5AndTripleDES), while entries in the db were saved with a different one before (PBEWithMD5AndDES).
So I got a EncryptionOperationNotPossibleException too, which is without information because of #Nathan Feger's comment above.
I hope this could help somebody someday too ;)

I faced similar problem.
For me, it was because it was trying to decrypt a password which could not have been decrypted using the decrypting mechanism.
Hence, I encrypted the password and stored it in database before the decrypt method tries to decrypt it.
I hope it helps someone.

How to Pack/Encrypt/Unpack/Decrypt a bunch of files in Java?

I'm essentially trying to do the following on a Java/JSP-driven web site:
User supplies a password
Password is used to build a strongly-encrypted archive file (zip, or anything else) containing a text file as well as a number of binary files that are stored on the server. It's essentially a backup of the user's files and settings.
Later, the user can upload the file, provide the original password, and the site will decrypt and unpack the archive, save the extracted binary files to the appropriate folder on the server, and then read the text file so the site can restore the user's old settings and metadata about the binary files.
It's the building/encrypting the archive and then extracting its contents that I'm trying to figure out how to do. I really don't care about the archive format, other than that it is very secure.
The ideal solution to my problem will be very easy to implement, and will require only tried-and-tested libraries with free and nonrestrictive licenses (e.g. apache, berkeley, lgpl).
I'm aware of the TrueZIP and WinZipAES libraries; the former seems like massive overkill and I can't tell how stable the latter is... Are there other solutions out there that would work well?

If you know how to create a zip file using the java.util.zip package, you can create a PBE Cipher and pass that to a CipherOutputStream or a CipherInputStream (depending on if you're reading or writing).
The following should get you started:
public class ZipTest {
public static void main(String [] args) throws Exception {
String password = "password";
write(password);
read(password);
}
private static void write(String password) throws Exception {
OutputStream target = new FileOutputStream("out.zip");
target = new CipherOutputStream(target, createCipher(Cipher.ENCRYPT_MODE, password));
ZipOutputStream output = new ZipOutputStream(target);
ZipEntry e = new ZipEntry("filename");
output.putNextEntry(e);
output.write("helloWorld".getBytes());
output.closeEntry();
e = new ZipEntry("filename1");
output.putNextEntry(e);
output.write("helloWorld1".getBytes());
output.closeEntry();
output.finish();
output.flush();
}
private static Cipher createCipher(int mode, String password) throws Exception {
String alg = "PBEWithSHA1AndDESede"; //BouncyCastle has better algorithms
PBEKeySpec keySpec = new PBEKeySpec(password.toCharArray());
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(alg);
SecretKey secretKey = keyFactory.generateSecret(keySpec);
Cipher cipher = Cipher.getInstance("PBEWithSHA1AndDESede");
cipher.init(mode, secretKey, new PBEParameterSpec("saltsalt".getBytes(), 2000));
return cipher;
}
private static void read(String password) throws Exception {
InputStream target = new FileInputStream("out.zip");
target = new CipherInputStream(target, createCipher(Cipher.DECRYPT_MODE, password));
ZipInputStream input = new ZipInputStream(target);
ZipEntry entry = input.getNextEntry();
while (entry != null) {
System.out.println("Entry: "+entry.getName());
System.out.println("Contents: "+toString(input));
input.closeEntry();
entry = input.getNextEntry();
}
}
private static String toString(InputStream input) throws Exception {
byte [] data = new byte[1024];
StringBuilder result = new StringBuilder();
int bytesRead = input.read(data);
while (bytesRead != -1) {
result.append(new String(data, 0, bytesRead));
bytesRead = input.read(data);
}
return result.toString();
}
}

The answer is already given (use a cipher as Kevin pointed out), so I am only doing a suggestion about an important matter which seems to be missing in your topicstart: ensure that you're using HTTPS instead of HTTP. Otherwise one with a network sniffer would be able to get the user-supplied password from the packets. How to do it depends on the appserver in question. Best is to refer its documentation. If it is for example Apache Tomcat, then you can find everything in the Tomcat SSL HOW-TO.
Hope this helps.

Though it may not be specific to your query I wonder if truecrypt could be of use. Your webserver could create an encrypted container into which the zip file would be copied. The encrypted container could then be downloaded. Potentially a little messy however the encryption should be strong and the downloaded image could be mounted on a variety of operating systems.

There are surely a few suggestions here on how to solve your problem, but I'm missing a very big BUT in the responses. You cannot fulfill both "password based" and "strong encryption" for any reasonable definition of "strong encryption".