Develop Reference

Convert UCS-2 HEX в UTF16 BE java

I have a string in which the SMS text is encrypted in USC2 format received from a GSM modem
I'm trying to convert it to UTF16 but it doesn't work. Please tell me what am I doing wrong
public class USC {
public static void main(String[] args) {
String hex = "0412044B0020043F043E043B044C043704430435044204350441044C002004420430044004380444043D044B043C0020043F043B0430043D043E043C0020002204110438002B002200200441002000300033002E00310032002E0032003000320031002E002004230442043E0447043D04380442044C002004430441043B043E04320438044F";
byte[] v = hex.getBytes(StandardCharsets.UTF_16BE);
String str = new String(v);
System.out.println(str);
}
}
On the online decoder through the service https://dencode.com/ works fine

Try the following:
BigInteger bi = new BigInteger(hex, 16);
byte[] a = bi.toByteArray();
System.out.println(new String(a, Charset.forName("UTF-16")));

String hex = "0412044B0020043F043E043B044C043704430435044204350441044C002004420430044004380444043D044B043C0020043F043B0430043D043E043C0020002204110438002B002200200441002000300033002E00310032002E0032003000320031002E002004230442043E0447043D04380442044C002004430441043B043E04320438044F";
int n = hex.length/4;
char[] chars = new char[n];
for (int i = 0; i < n; ++i) {
chars[i] = Integer.parseInt(hex.substring(4*i, 4*i+4), 15) & 0xFFFF);
}
String str = new String(chars);
System.out.println(str);
4 hex chars form one UCS-2 big endian char. Same size as java char (2 bytes).
UTF-16 is superior to UCS-2 which forms a fixed-size subset. So from UCS-2 to UTF-16 needs no special treatment, only whether the 2 bytes of a char are big endian or little endian.

With JDK17 or above you could also make use of HexFormat class:
String str = new String(HexFormat.of().parseHex(hex), StandardCharsets.UTF_16BE);

Convert hex to INT 32 and STRING Little Endian

First question:
Hex: F1620000
After convert hex to INT 32 i expect 253229, but i get -245235712.
I've tried these methods:
Integer.parseUnsignedInt(value, 16));
(int)Long.parseLong(value, 16));
new BigInteger(value, 16).intValue());
How i can get correct value?
Second question:
Hex: 9785908D9B9885828020912E208D2E
After covert hex to STRING i can get this value:
\u0097\u0085\u0090\u008d\u009b\u0098\u0085\u0082\u0080 \u0091. \u008d.
How can I display this value correctly in json? (usning JSONObject).
StringBuilder result = new StringBuilder();
for (int i = 0; i < value.length(); i += 2) {
String str = value.substring(i, i + 2);
result.append((char)Integer.parseInt(str, 16));
}

All your attempts are sufficient for parsing a hexadecimal string in an unsigned interpretation, but did not account for the “little endian” representation. You have to reverse the bytecode manually:
String value = "F1620000";
int i = Integer.reverseBytes(Integer.parseUnsignedInt(value, 16));
System.out.println(i);
25329
For your second task, the missing information was how to interpret the bytes to get to the character content. After searching a bit, the Codepage 866 seems to be the most plausible encoding:
String value = "9785908D9B9885828020912E208D2E";
byte[] bytes = new BigInteger(value, 16).toByteArray();
String result = new String(bytes, bytes[0]==0? 1: 0, value.length()/2, "CP866");
ЧЕРНЫШЕВА С. Н.

Encoding byte array to Hex string

I have come across a legacy piece of code encoding byte array to hex string which is in production and have never caused an issue.
This piece of code is used as:
We encrypt a user password. The encryptor returns a byte[].
We convert the byte[] to Hex String using this encoder code and then use that String representation in our properties file and so on.
However, yesterday we have hit a password, whose encrypted byte[] version is getting encoded incorrectly.
import java.math.BigInteger;
import java.util.HashMap;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Hex;
public class ByteArrayToHexEncoder {
public static void main(String[] args) throws DecoderException {
String hexString = "a0d21588c0a2c2fc68dc859197fc78cd"; // correct hex representation
// equivalent byte array: this is the byte array returned by the encryptor
byte[] byteArray = Hex.decodeHex(hexString.toCharArray());
// legacy encoder
System.out.println("Legacy code encodes as: " + encodeHexBytesWithPadding(byteArray));
// commons-codec encoder
System.out.println("Commons codec encode as: " + new String(Hex.encodeHex(byteArray)));
}
private static final String PADDING_ZEROS =
"0000000000000000000000000000000000000000000000000000000000000";
private static final HashMap<Integer, Character> MAP_OF_HEX = new HashMap<>();
static {
MAP_OF_HEX.put(0, '0');
MAP_OF_HEX.put(1, '1');
MAP_OF_HEX.put(2, '2');
MAP_OF_HEX.put(3, '3');
MAP_OF_HEX.put(4, '4');
MAP_OF_HEX.put(5, '5');
MAP_OF_HEX.put(6, '6');
MAP_OF_HEX.put(7, '7');
MAP_OF_HEX.put(8, '8');
MAP_OF_HEX.put(9, '9');
MAP_OF_HEX.put(10, 'a');
MAP_OF_HEX.put(11, 'b');
MAP_OF_HEX.put(12, 'c');
MAP_OF_HEX.put(13, 'd');
MAP_OF_HEX.put(14, 'e');
MAP_OF_HEX.put(15, 'f');
}
public static String encodeHexBytesWithPadding(byte[] inputByteArray) {
String encodedValue = encodeHexBytes(inputByteArray);
int expectedSize = inputByteArray.length * 2;
if (encodedValue.length() < expectedSize) {
int zerosToPad = expectedSize - encodedValue.length();
encodedValue = PADDING_ZEROS.substring(0, zerosToPad) + encodedValue;
}
return encodedValue;
}
public static String encodeHexBytes(byte[] inputByteArray) {
String encodedValue;
if (inputByteArray[0] < 0) {
// Something is wrong here! Don't know what!
byte oldValue = inputByteArray[0];
inputByteArray[0] = (byte) (oldValue & 0x0F);
int nibble = (oldValue >> 4) & 0x0F;
encodedValue = new BigInteger(inputByteArray).toString(16);
inputByteArray[0] = oldValue;
encodedValue = MAP_OF_HEX.get(nibble) + encodedValue;
} else {
encodedValue = new BigInteger(inputByteArray).toString(16);
}
return encodedValue;
}
}
The legacy code outputs the encoded value as: 0ad21588c0a2c2fc68dc859197fc78cd while the correct expected value should be: a0d21588c0a2c2fc68dc859197fc78cd.
I am trying to understand what's wrong with the encoder and need some help understanding.

BigInteger(byte[]) constructor is there to handle two's complement representation of a number where the most significant bit also denotes the sign. The Hex common-codec simply translates each byte into a hex representation, there is no special meaning to the most significant bit.
Your legacy code in the if (inputByteArray[0] < 0) branch attempts to modify the first byte in the byte[] input probably to work around the representation of negative numbers in the two-complement's form e.g. -1 being represented as ff. Unfortunately this is implemented incorrectly in your legacy code:
String input = "a000000001";
byte[] bytes = Hex.decodeHex(input.toCharArray());
System.out.println(encodeHexBytesWithPadding(bytes));
System.out.println(Hex.encodeHexString(bytes));
will print
00000000a1
a000000001
showing that the legacy code values are completely wrong.
There is not much to salvage here IMO, instead use Hex.encodeHexString() instead or check other options discussed in this question.

SHA-1 hashing on Java and C#

I'm trying to validate the content of an XML node with SHA-1 , basically, we generate an SHA-1 hash with the content of that node and both sides (client C# and server Java) should have exactly the same hash.
The problem is , I have checked with a diff tool the content of both texts and there is not any difference. But I'm getting a different hash than the client.
C# hash : 60-53-58-69-29-EB-53-BD-85-31-79-28-A0-F9-42-B6-DE-1B-A6-0A
Java hash: E79D7E6F2A6F5D776447714D896D4C3A0CBC793
The way the client (C#) is generating the hash is this:
try
{
Byte[] stream = null;
using (System.Security.Cryptography.SHA1CryptoServiceProvider shaProvider = new System.Security.Cryptography.SHA1CryptoServiceProvider())
{
stream = shaProvider.ComputeHash(System.Text.Encoding.UTF8.GetBytes(text));
if (stream == null)
{
hash = "Error";
}
else
{
hash = System.BitConverter.ToString(stream);
}
}
}
catch (Exception error)
{
hash = string.Format("Error SHA-1: {0}", error);
}
return hash;
and this is how the server (Java) is generating the hash:
byte[] key = content.getBytes();
MessageDigest md = MessageDigest.getInstance("SHA1");
byte[] hash = md.digest(key);
String result = "";
for (byte b : hash) {
result += Integer.toHexString(b & 255);
}
return result.toUpperCase();
can someone help me ? .. thanks :)
UPDATE:
In order to check what's going on I have checked other ways to get a SHA1 hash in C# and I found this:
/// <summary>
/// Compute hash for string encoded as UTF8
/// </summary>
/// <param name="s">String to be hashed</param>
/// <returns>40-character hex string</returns>
public static string SHA1HashStringForUTF8String(string s)
{
byte[] bytes = Encoding.UTF8.GetBytes(s);
using (var sha1 = SHA1.Create())
{
byte[] hashBytes = sha1.ComputeHash(bytes);
return System.BitConverter.ToString(hashBytes).Replace("-",string.Empty);
}
}
This code gives this output:
E79D07E6F2A6F5D776447714D896D4C3A0CBC793
AND !! I just noticed that Python is giving the same output (sorry, I should double checked this)
So this is the deal
Using this provider: System.Security.Cryptography.SHA1CryptoServiceProvider shaProvider = new System.Security.Cryptography.SHA1CryptoServiceProvider()
Is giving a completly different output on three different machines ..
Using the above method in C# gives the same result as python does, also, for some reason Java is giving a sightly different output:
E79D7E6F2A6F5D776447714D896D4C3A0CBC793
Ideas?, is java the problem? the byte to hex method on java is the problem? there is another alternative?

Try using this as your hashing in C#:
static string Hash(string input)
{
using (SHA1Managed sha1 = new SHA1Managed())
{
var hash = sha1.ComputeHash(Encoding.UTF8.GetBytes(input));
var sb = new StringBuilder(hash.Length * 2);
foreach (byte b in hash)
{
// can be "x2" if you want lowercase
sb.Append(b.ToString("x2"));
}
return sb.ToString();
}
}
Hash("test"); //a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
And then use this as your Java hashing:
private static String convertToHex(byte[] data) {
StringBuilder buf = new StringBuilder();
for (byte b : data) {
int halfbyte = (b >>> 4) & 0x0F;
int two_halfs = 0;
do {
buf.append((0 <= halfbyte) && (halfbyte <= 9) ? (char) ('0' + halfbyte) : (char) ('a' + (halfbyte - 10)));
halfbyte = b & 0x0F;
} while (two_halfs++ < 1);
}
return buf.toString();
}
public static String SHA1(String text) throws NoSuchAlgorithmException, UnsupportedEncodingException {
MessageDigest md = MessageDigest.getInstance("SHA-1");
byte[] textBytes = text.getBytes("iso-8859-1");
md.update(textBytes, 0, textBytes.length);
byte[] sha1hash = md.digest();
return convertToHex(sha1hash);
}
SHA1("test"); //a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
Note you need the following imports:
import java.io.UnsupportedEncodingException; import
java.security.MessageDigest; import
java.security.NoSuchAlgorithmException;
Throws declarations are option, adjust to best fit your code!

Your problem is that you're not hashing the same bytes in both API.
If you choose to modify java's version, it should look like this:
byte[] key = content.getBytes("UTF8");
[...]
If you choose to modify c#' version, it should look like this:
stream = shaProvider.ComputeHash(System.Text.Encoding.UTF16.GetBytes(text));
[...]
Either way, both api should get the key's bytes through the same encoding.

Format sha-1 hash like .net does

I have a question about how java prints the HexString and how to print it in the same format that C# using the BitConverter does.
We (both) are generating this Hex as result of hashing a string with SHA-1 :
a94a8fe5ccb19ba61c4c873d391e987982fbbd3 (hash for "test" string)
The problem is when we test the hashing in C# , C# prints with uppercase and separated with "-" , like A9-4A-8F .. and so on.
How can I print the same format with java?
The code for hashing and printing is this:
String password = "test";
byte[] key = password.getBytes();
MessageDigest md = MessageDigest.getInstance("SHA-1",new BouncyCastleProvider());
byte[] hash = md.digest(key);
String result = "";
for (byte b : hash) {
result += Integer.toHexString(b & 255);
}
return result;

String result = "";
for (byte b : hash) {
result += Integer.toHexString(b & 255).toUpperCase() + "-";
}
return result.subString(0, result.length()-1);