Develop Reference

Java Digest Hash and PHP Hash are different

I'm trying to authenticate a webhook from starling bank on a PHP 7.0.22 (Apache/2.4.6 (Red Hat Enterprise Linux)) server.
I've been told by support that the following java code is being used to generate the digest
private String calculateSignature(String sharedSecret, String requestJson) {
try {
String contentToDigest = sharedSecret + requestJson;
MessageDigest messageDigest = MessageDigest.getInstance("SHA-512");
byte[] digest = messageDigest.digest(contentToDigest.getBytes());
return Base64.getEncoder().encodeToString(digest);
} catch (NoSuchAlgorithmException e) {
throw new RuntimeException("Error calculating digest for payload [" + requestJson + "]", e);
}
}
The sharedSecret I already have and the requestJson I take from the webhook POST using:
$requestJson=file_get_contents('php://input') ;
my php code to generate the hash is as follows:
$concatenated_string=$sharedSecret . json_encode($requestJson) ;
$generated_hash=base64_encode(hash('sha512', $concatenated_string ));
This doesn't give the same hash. Whilst hacking to try and find an answer, I've also tried the following :
$concatenated_string=$sharedSecret . $requestJson ;
and different hash types and options:
$generated_hash=base64_encode(hash('sha512', $concatenated_string, true ))
$generated_hash=base64_encode(openssl_digest($concatenated_string, 'sha512')) ;

base64_encode and hash are effectively doing the same thing in this case:
https://stackoverflow.com/a/11195855/3323777
You should specify third argument as TRUE at your php code to match the java version:
raw_output - Setting to TRUE will return as raw output data, otherwise the return value is binhex encoded.
http://php.net/manual/ru/function.openssl-digest.php
I've ran your both snippets on java and php and found not difference when encoding a string "test". I advise you to output the json payloads to two files on both environments and use diff to compare them.

Java URL encoding with url encode parameters

How to encode in Java this String
http://demo.pl/sample?id=tests%trg=https%3A%2F%2Fwww.google.com%sample.html%3Fwmc%3DAFF48+_LS.%23%7NUMBER_ID%7D_%23%7NUMBER_ID%7D..
java.net.URLEncoder encode this String like this:
http%3A%2F%2Fdemo.pl%2Fsample%3Fid%3Dtests%25trg%3Dhttps%253A%252F%252Fwww.google.com%25sample.html%253Fwmc%253DAFF48%2B_LS.%2523%257NUMBER_ID%257D_%2523%257NUMBER_ID%257D..
I expect this result:
http%3A%2F%2Fdemo.pl%2Fsample%3Fid%3Dtests%25trg%3Dhttps%3A%2F%2Fwww.google.com%sample.html%3Fwmc%3DAFF48+_LS.%23%7NUMBER_ID%7D_%23%7NUMBER_ID%7D..

I think following code can help you:
String s = "http://demo.pl/sample?id=tests%trg=https%3A%2F%2Fwww.google.com%sample.html%3Fwmc%3DAFF48+_LS.%23%7NUMBER_ID%7D_%23%7NUMBER_ID%7D";
int i = s.indexOf("%");
String result1 = URLEncoder.encode(s.substring(0, i)) + "%25" + s.substring(i + 1);
System.out.println(result1); // print http%3A%2F%2Fdemo.pl%2Fsample%3Fid%3Dtests%25trg=https%3A%2F%2Fwww.google.com%sample.html%3Fwmc%3DAFF48+_LS.%23%7NUMBER_ID%7D_%23%7NUMBER_ID%7D
I do not want to encode encoded part of String. Needs a universal
algorithm. String is not always encoded fragmentarily
I think universal algorithm is impossible in that case, what you can do that find encoded part manually and not encoded it again (see code above).

Is Java 8 java.util.Base64 a drop-in replacement for sun.misc.BASE64?

Question
Are the Java 8 java.util.Base64 MIME Encoder and Decoder a drop-in replacement for the unsupported, internal Java API sun.misc.BASE64Encoder and sun.misc.BASE64Decoder?
EDIT (Clarification): By drop-in replacement
I mean that I can switch legacy code using sun.misc.BASE64Encoder and sun.misc.BASE64Decoder to Java 8 MIME Base64 Encoder/Decoder for any existing other client code transparently.
What I think so far and why
Based on my investigation and quick tests (see code below) it should be a drop-in replacement because
sun.misc.BASE64Encoder based on its JavaDoc is a BASE64 Character encoder as specified in RFC1521. This RFC is part of the MIME specification...
java.util.Base64 based on its JavaDoc Uses the "The Base64 Alphabet" as specified in Table 1 of RFC 2045 for encoding and decoding operation... under MIME
Assuming no significant changes in the RFC 1521 and 2045 (I could not find any) and based on my quick test using the Java 8 Base64 MIME Encoder/Decoder should be fine.
What I am looking for
an authoritative source confirming or disproving the "drop-in replacement" point OR
a counterexample which shows a case where java.util.Base64 has different behaviour than the sun.misc.BASE64Encoder OpenJDK Java 8 implementation (8u40-b25) (BASE64Decoder) OR
whatever you think answers above question definitely
For reference
My test code
public class Base64EncodingDecodingRoundTripTest {
public static void main(String[] args) throws IOException {
String test1 = " ~!##$%^& *()_+=`| }{[]\\;: \"?><,./ ";
String test2 = test1 + test1;
encodeDecode(test1);
encodeDecode(test2);
}
static void encodeDecode(final String testInputString) throws IOException {
sun.misc.BASE64Encoder unsupportedEncoder = new sun.misc.BASE64Encoder();
sun.misc.BASE64Decoder unsupportedDecoder = new sun.misc.BASE64Decoder();
Base64.Encoder mimeEncoder = java.util.Base64.getMimeEncoder();
Base64.Decoder mimeDecoder = java.util.Base64.getMimeDecoder();
String sunEncoded = unsupportedEncoder.encode(testInputString.getBytes());
System.out.println("sun.misc encoded: " + sunEncoded);
String mimeEncoded = mimeEncoder.encodeToString(testInputString.getBytes());
System.out.println("Java 8 Base64 MIME encoded: " + mimeEncoded);
byte[] mimeDecoded = mimeDecoder.decode(sunEncoded);
String mimeDecodedString = new String(mimeDecoded, Charset.forName("UTF-8"));
byte[] sunDecoded = unsupportedDecoder.decodeBuffer(mimeEncoded); // throws IOException
String sunDecodedString = new String(sunDecoded, Charset.forName("UTF-8"));
System.out.println(String.format("sun.misc decoded: %s | Java 8 Base64 decoded: %s", sunDecodedString, mimeDecodedString));
System.out.println("Decoded results are both equal: " + Objects.equals(sunDecodedString, mimeDecodedString));
System.out.println("Mime decoded result is equal to test input string: " + Objects.equals(testInputString, mimeDecodedString));
System.out.println("\n");
}
}

Here's a small test program that illustrates a difference in the encoded strings:
byte[] bytes = new byte[57];
String enc1 = new sun.misc.BASE64Encoder().encode(bytes);
String enc2 = new String(java.util.Base64.getMimeEncoder().encode(bytes),
StandardCharsets.UTF_8);
System.out.println("enc1 = <" + enc1 + ">");
System.out.println("enc2 = <" + enc2 + ">");
System.out.println(enc1.equals(enc2));
Its output is:
enc1 = <AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
>
enc2 = <AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA>
false
Note that the encoded output of sun.misc.BASE64Encoder has a newline at the end. It doesn't always append a newline, but it happens to do so if the encoded string has exactly 76 characters on its last line. (The author of java.util.Base64 considered this to be a small bug in the sun.misc.BASE64Encoder implementation – see the review thread).
This might seem like a triviality, but if you had a program that relied on this specific behavior, switching encoders might result in malformed output. Therefore, I conclude that java.util.Base64 is not a drop-in replacement for sun.misc.BASE64Encoder.
Of course, the intent of java.util.Base64 is that it's a functionally equivalent, RFC-conformant, high-performance, fully supported and specified replacement that's intended to support migration of code away from sun.misc.BASE64Encoder. You need to be aware of some edge cases like this when migrating, though.

I had same issue, when i moved from sun to java.util.base64, but then org.apache.commons.codec.binary.Base64 solved my problem

There are no changes to the base64 specification between rfc1521 and rfc2045.
All base64 implementations could be considered to be drop-in replacements of one another, the only differences between base64 implementations are:
the alphabet used.
the API's provided (e.g. some might take only act on a full input buffer, while others might be finite state machines allowing you to continue to push chunks of input through them until you are done).
The MIME base64 alphabet has remained constant between RFC versions (it has to or older software would break) and is: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/
As Wikipedia notes, only the last 2 characters may change between base64 implementations.
As an example of a base64 implementation that does change the last 2 characters, the IMAP MUTF-7 specification uses the following base64 alphabet: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+,
The reason for the change is that the / character is often used as a path delimiter and since the MUTF-7 encoding is used to flatten non-ASCII directory paths into ASCII, the / character needed to be avoided in encoded segments.

Assuming both encoders are bug free, then the RFC requires distinct encodings for every 0 byte, 1 byte, 2 byte and 3 bytes sequence. Longer sequences are broken down into as many 3 byte sequences as needed followed by a final sequence. Hence if the two implementations handle all 16,843,009 (1+256+65536+16777216) possible sequences correctly, then the two implementations are also identical.
These tests only take a few minutes to run. By slightly changing your test code, I have done that and my Java 8 installation passed all the test. Hence the public implementation can be used to safely replace the sun.misc implementation.
Here is my test code:
import java.util.Base64;
import java.util.Arrays;
import java.io.IOException;
public class Base64EncodingDecodingRoundTripTest {
public static void main(String[] args) throws IOException {
System.out.println("Testing zero byte encoding");
encodeDecode(new byte[0]);
System.out.println("Testing single byte encodings");
byte[] test = new byte[1];
for(int i=0;i<256;i++) {
test[0] = (byte) i;
encodeDecode(test);
}
System.out.println("Testing double byte encodings");
test = new byte[2];
for(int i=0;i<65536;i++) {
test[0] = (byte) i;
test[1] = (byte) (i >>> 8);
encodeDecode(test);
}
System.out.println("Testing triple byte encodings");
test = new byte[3];
for(int i=0;i<16777216;i++) {
test[0] = (byte) i;
test[1] = (byte) (i >>> 8);
test[2] = (byte) (i >>> 16);
encodeDecode(test);
}
System.out.println("All tests passed");
}
static void encodeDecode(final byte[] testInput) throws IOException {
sun.misc.BASE64Encoder unsupportedEncoder = new sun.misc.BASE64Encoder();
sun.misc.BASE64Decoder unsupportedDecoder = new sun.misc.BASE64Decoder();
Base64.Encoder mimeEncoder = java.util.Base64.getMimeEncoder();
Base64.Decoder mimeDecoder = java.util.Base64.getMimeDecoder();
String sunEncoded = unsupportedEncoder.encode(testInput);
String mimeEncoded = mimeEncoder.encodeToString(testInput);
// check encodings equal
if( ! sunEncoded.equals(mimeEncoded) ) {
throw new IOException("Input "+Arrays.toString(testInput)+" produced different encodings (sun=\""+sunEncoded+"\", mime=\""+mimeEncoded+"\")");
}
// Check cross decodes are equal. Note encoded forms are identical
byte[] mimeDecoded = mimeDecoder.decode(sunEncoded);
byte[] sunDecoded = unsupportedDecoder.decodeBuffer(mimeEncoded); // throws IOException
if(! Arrays.equals(mimeDecoded,sunDecoded) ) {
throw new IOException("Input "+Arrays.toString(testInput)+" was encoded as \""+sunEncoded+"\", but decoded as sun="+Arrays.toString(sunDecoded)+" and mime="+Arrays.toString(mimeDecoded));
}
}
}

Stuart Marks' answer is almost correct. The getMimeEncoder in his example above should be configured like this to emulate sun.misc:
String enc2 = new String(java.util.Base64.getMimeEncoder(76, new byte[]{0xa}).encode(bytes),
StandardCharsets.UTF_8);
At this point, it will be a drop-in as requested in the original post.

JAVA: failing to get encrypted data in string using xor

I was trying to print encrypted text using string perhaps i was wrong somewhere. I am doing simple xor on a plain text. Coming encrypted text/string i am putting in a C program and doing same xor again to get plain text again.
But in between, I am not able to get proper string of encrypted text to pass in C
String xorencrypt(byte[] passwd,int pass_len){
char[] st = new char[pass_len];
byte[] crypted = new byte[pass_len];
for(int i = 0; i<pass_len;i++){
crypted[i] = (byte) (passwd[i]^(i+1));
st[i] = (char)crypted[i];
System.out.println((char)passwd[i]+" "+passwd[i] +"= " + (char)crypted[i]+" "+crypted[i]);/* characters are printed fine but problem is when i am convering it in to string */
}
return st.toString();
}
I don't know if any kind of encoding also needed because if i did so how I will decode and decrypt from C program.
example if suppose passwd = bond007
then java program should return akkb78>
further C program will decrypt akkb78> to bond007 again.

Use
return new String(crypted);
in that case you don't need st[] array at all.
By the way, the encoded value for bond007 is cmm`560 and not what you posted.
EDIT
While solution above would most likely work in most java environments, to be safe about encoding,
as suggested by Alex, provide encoding parameter to String constructor.
For example if you want your string to carry 8-bit bytes :
return new String(crypted, "ISO-8859-1");
You would need the same parameter when getting bytes from your string :
byte[] bytes = myString.getBytes("ISO-8859-1")
Alternatively, use solution provided by Alex :
return new String(st);
But, convert bytes to chars properly :
st[i] = (char) (crypted[i] & 0xff);
Otherwise, all negative bytes, crypted[i] < 0 will not be converted to char properly and you get surprising results.

Change this line:
return st.toString();
with this
return new String(st);

java decoding base64 String

I realise this is probably more of a general java question, but since it's running in Notes\ Domino environment, thought I'd check that community first.
Summary:
I don't seem to be able to decode the string: dABlAHMAdAA= using lotus.domino.axis.encoding.Base64 or sun.misc.BASE64Decoder
I know the original text is: test
I confirmed by decoding at http://www5.rptea.com/base64/ it appears it is UTF-16.
As simple test, using either of below:
String s_base64 = "dABlAHMAdAA=";
byte[] byte_base64 = null;
String s_decoded = "";
byte_base64 = new sun.misc.BASE64Decoder().decodeBuffer(s_base64);
s_decoded = new String(byte_base64, "UTF-16");
System.out.println("Test1: " + s_decoded);
byte_base64 = lotus.domino.axis.encoding.Base64.decode(s_base64);
s_decoded = new String(byte_base64, "UTF-16");
System.out.println("Test2: " + s_decoded);
System.out.println("========= FINISH.");
I get the output:
Test1: ????
Test2: ????
If I create String as UTF-8
s_decoded = new String(byte_base64, "UTF-8");
it outputs:
t
no error is thrown, but it doesn't complete the code, doesn't get to the "FINISH".
Detail
I'm accessing an asmx web service, in the SOAP response, some nodes contain base64 encoded data. At this point in time, there is no way to get the service changed, so I am having to XPath and decode myself. Encoded data is either text or html. If I pass the encoded data thru http://www5.rptea.com/base64/ and select UTF-16, it decodes correctly, so I must be doing something incorrectly.
As side note, I encoded "test":
s_base64 = lotus.domino.axis.encoding.Base64.encode(s_text.getBytes());
System.out.println("test1 encodes to: " + s_base64);
s_base64 = new sun.misc.BASE64Encoder().encode(s_text.getBytes());
System.out.println("test2 encodes to: " + s_base64);
they both encode to:
dGVzdA==
...which if you then feed into 2 decoders above, as expected, decodes correctly.
If I go to site above, and encode "test" as UTF-16, I get: dABlAHMAdAA= so that confirms that data is in UTF-16.
It's like the data is genuine base64 data, but the decoder doesn't recognise it as such. I'm slightly stumped at the moment.
Any pointers or comments would be gratefully received.

The string has been encoded in UTF-16LE (little-endian), where the least significant byte is stored first. Java defaults to big-endian. You need to use:
s_decoded = new String(byte_base64, "UTF-16LE");

i have used your sample "dABlAHMAdAA=" on my base64 decode online tool and it seems like you are missing the Apache base64 jar files
Click the link below.
http://www.hosting4free.info/Base64Decode/Base64-Decode.jsp
The code behind the website is
import org.apache.commons.codec.binary.Base64;
public class base64decode
{
public static void main(String[] args) throws UnsupportedEncodingException
{
byte[] decoded = Base64.decodeBase64("YWJjZGVmZw==".getBytes());
System.out.println(new String(decoded) + "\n");
}
}