Develop Reference

Convert UCS-2 HEX в UTF16 BE java

I have a string in which the SMS text is encrypted in USC2 format received from a GSM modem
I'm trying to convert it to UTF16 but it doesn't work. Please tell me what am I doing wrong
public class USC {
public static void main(String[] args) {
String hex = "0412044B0020043F043E043B044C043704430435044204350441044C002004420430044004380444043D044B043C0020043F043B0430043D043E043C0020002204110438002B002200200441002000300033002E00310032002E0032003000320031002E002004230442043E0447043D04380442044C002004430441043B043E04320438044F";
byte[] v = hex.getBytes(StandardCharsets.UTF_16BE);
String str = new String(v);
System.out.println(str);
}
}
On the online decoder through the service https://dencode.com/ works fine

Try the following:
BigInteger bi = new BigInteger(hex, 16);
byte[] a = bi.toByteArray();
System.out.println(new String(a, Charset.forName("UTF-16")));

String hex = "0412044B0020043F043E043B044C043704430435044204350441044C002004420430044004380444043D044B043C0020043F043B0430043D043E043C0020002204110438002B002200200441002000300033002E00310032002E0032003000320031002E002004230442043E0447043D04380442044C002004430441043B043E04320438044F";
int n = hex.length/4;
char[] chars = new char[n];
for (int i = 0; i < n; ++i) {
chars[i] = Integer.parseInt(hex.substring(4*i, 4*i+4), 15) & 0xFFFF);
}
String str = new String(chars);
System.out.println(str);
4 hex chars form one UCS-2 big endian char. Same size as java char (2 bytes).
UTF-16 is superior to UCS-2 which forms a fixed-size subset. So from UCS-2 to UTF-16 needs no special treatment, only whether the 2 bytes of a char are big endian or little endian.

With JDK17 or above you could also make use of HexFormat class:
String str = new String(HexFormat.of().parseHex(hex), StandardCharsets.UTF_16BE);

Converting hexadecimal to little endian

What is the formula to convert the value of a textfield from hex to little endian?
Example input: 5A109061
Example output: 1636831322

Get the value from the EditText as a String.
Parse the string value as hex, using Integer.parseInt(...) and radix 16.
Flip the byte order of the int, either using ByteBuffer (simpler) or using bit shifts (faster).
For example:
String hex = "5A109061"; // mEditText.getText().toString()
// Parse hex to int
int value = Integer.parseInt(hex, 16);
// Flip byte order using ByteBuffer
ByteBuffer buffer = ByteBuffer.allocate(4);
buffer.order(ByteOrder.BIG_ENDIAN);
buffer.asIntBuffer().put(value);
buffer.order(ByteOrder.LITTLE_ENDIAN);
int flipped = buffer.asIntBuffer().get();
System.out.println("hex: 0x" + hex);
System.out.println("flipped: " + flipped);
Output:
hex: 0x5A109061
flipped: 1636831322

Use ByteBuffer
ByteBuffer byteBuffer = ByteBuffer.allocate(8)order(ByteOrder.LITTLE_ENDIAN).putLong(5A109061)
byte[] result = byteBuffer.array();

You can also use this extension for kotlin.
Example :
val str = "6a3b7043"
val hex2Float = str.hex2Float
fun String.hex2Float(): Float{
val i = toLong(16)
val data = java.lang.Float.intBitsToFloat(i.toInt()) // Big endian
val buffer = ByteBuffer.allocate(4)
buffer.asFloatBuffer().put(data)
buffer.order(ByteOrder.LITTLE_ENDIAN)
val lData = buffer.asFloatBuffer().get() // Little endian
return lData
}

How to convert byte[] to String in Java?

There are some SO quetions but no helped me. I would like to convert byte[] from org.apache.commons.codec.digest.HmacUtils to String. This code produces some weird output:
final String value = "value";
final String key = "key";
byte[] bytes = HmacUtils.hmacSha1(key, value);
String s = new String(bytes);
What am I doing wrong?

Try to use:
String st = HmacUtils.hmacSha1Hex(key, value);

First, the result of hmacSha1 would produce a digest, not not a clear String. Besides, you may have to specify an encoding format, for example
String s = new String(bytes, "US-ASCII");
or
String s = new String(bytes, "UTF-8");

For a more general solution, if you don't have HmacUtils available:
// Prepare a buffer for the string
StringBuilder builder = new StringBuilder(bytes.length*2);
// Iterate through all bytes in the array
for(byte b : bytes) {
// Convert them into a hex string
builder.append(String.format("%02x",b));
// builder.append(String.format("%02x",b).toUpperCase()); // for upper case characters
}
// Done
String s = builder.toString();
To explain your problem:
You are using a hash function. So a hash is usually an array of bytes which should look quite random.
If you use new String(bytes) you try to create a string from these bytes. But Java will try to convert the bytes to characters.
For example: The byte 65 (hex 0x41) becomes the letter 'A'. 66 (hex 0x42) the letter 'B' and so on. Some numbers can't be converted into readable characters. Thats why you see strange characters like '�'.
So new String(new byte[]{0x41, 0x42, 0x43}) will become 'ABC'.
You want something else: You want each byte converted into a 2 digit hex String (and append these strings).
Greetings!

You may need to have an encoding format. Check out this link here.
UTF-8 byte[] to String

Java String HEX to String ASCII with accentuation [duplicate]

This question already has answers here:
Converting UTF-8 to ISO-8859-1 in Java - how to keep it as single byte
(8 answers)
Closed 9 years ago.
I have the String String hex = "6174656ec3a7c3a36f"; and i wanna get the String output = "atenção" but in my test i only get String output = "aten????o";
what i m doing wrong?
String hex = "6174656ec3a7c3a36f";
StringBuilder output = new StringBuilder();
for (int i = 0; i < hex.length(); i+=2) {
String str = hex.substring(i, i+2);
output.append((char)Integer.parseInt(str, 16));
}
System.out.println(output); //here is the output "aten????o"

Consider
String hex = "6174656ec3a7c3a36f"; // AAA
ByteBuffer buff = ByteBuffer.allocate(hex.length()/2);
for (int i = 0; i < hex.length(); i+=2) {
buff.put((byte)Integer.parseInt(hex.substring(i, i+2), 16));
}
buff.rewind();
Charset cs = Charset.forName("UTF-8"); // BBB
CharBuffer cb = cs.decode(buff); // BBB
System.out.println(cb.toString()); // CCC
Which prints: atenção
Basically, your hex string represents the hexidecimal encoding of the bytes that represent the characters in the string atenção when encoded in UTF-8.
To decode:
You first have to go from your hex string to bytes (AAA)
Then go from bytes to chars (BBB) -- this is dependent on the encoding, in your case UTF-8.
The go from chars to a string (CCC)

Your hex string appears to denote a UTF-8 string, rather than ISO-8859-1.
The reason I can say this is that if it was ISO-8859-1, you'd have two hex digits per character. Your hex string has 18 characters, but your expected output is only 7 characters. Hence, the hex string must be a variable width encoding, and not a single byte per character like ISO-8859-1.
The following program produces the output: atenção
String hex = "6174656ec3a7c3a36f";
ByteArrayOutputStream baos = new ByteArrayOutputStream();
for (int i = 0; i < hex.length(); i += 2) {
String str = hex.substring(i, i + 2);
int byteVal = Integer.parseInt(str, 16);
baos.write(byteVal);
}
String s = new String(baos.toByteArray(), Charset.forName("UTF-8"));
If you change UTF-8 to ISO-8859-1, you'll see: atenção.

The Java Strings are Unicode: each character is encoded on 16 bits. Your String is - I suppose - a "C" string. You have to know the name of the character encoder and use CharsetDecoder.
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
public class Char8859_1Decoder {
public static void main( String[] args ) throws CharacterCodingException {
String hex = "6174656ec3a7c3a36f";
int len = hex.length();
byte[] cStr = new byte[len/2];
for( int i = 0; i < len; i+=2 ) {
cStr[i/2] = (byte)Integer.parseInt( hex.substring( i, i+2 ), 16 );
}
CharsetDecoder decoder = Charset.forName( "UTF-8" ).newDecoder();
CharBuffer cb = decoder.decode( ByteBuffer.wrap( cStr ));
System.out.println( cb.toString());
}
}

The ç and ã are 16-bit characters, so they are not represented by a byte as you assume in your decode routine, but rather by a full word.
I would, instead of converting each byte to a char, convert the bytes to java Bytes, and then use a string routine to decode the array of Bytes to a string, allowing java the dull task of determining the decoding routine.
Of course, java may guess wrong, so you might have to know ahead of time what the encoding is, as per the answer given by #Aubin or #Martin Ellis

Convert "php unicode" to character

How can I convert so called "php unicode"(link to php unicode) to normal character via Java? Example \xEF\xBC\xA1 -> A. Are there any embedded methods in jdk or should I use regex for this conversion?

You first need to get the bytes out of the string into a byte-array without changing them and then decode the byte-array as a UTF-8 string.
The simplest way to get the string into a byte array is to encode it using ISO-8859-1 which map every character with a unicode value less than 256 to a byte with the same value (or the equivalent negative)
String phpUnicode = "\u00EF\u00BC\u00A1"
byte[] bytes = phpUnicode.getBytes("ISO-8859-1"); // maps to bytes with the same ordinal value
String javaString = new String(bytes, "UTF-8");
System.out.println(javaString);
Edit
The above converts the UTF-8 to the Unicode character. If you then want to convert it to a reasonable ASCII equivalent, there's no standard way of doing that: but see this question
Edit
I assumed that you had a string containing characters that had the same ordinal value as the UTF-8 sequence but you indicate that your string literally contains the escape sequence, as in:
String phpUnicode = "\\xEF\\xBC\\xA1";
The JDK doesn't have any built-in methods to convert Strings like this so you'll need to use your own regex. Since we ultimately want to convert a utf-8 byte-sequence into a String, we need to set up a byte-array, using maybe:
Pattern oneChar = Pattern.compile("\\\\x([0-9A-F]{2})|(.)", Pattern.CASE_INSENSITIVE | Pattern.DOTALL);
Matcher matcher = oneChar.matcher(phpUnicode);
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
while (matcher.find()) {
int ch;
if (matcher.group(1) == null) {
ch = matcher.group(2).charAt(0);
}
else {
ch = Integer.parseInt(matcher.group(1), 16);
}
bytes.write((int) ch);
}
String javaString = new String(bytes.toByteArray(), "UTF-8");
System.out.println(javaString);
This will generate a UTF-8 stream by converting \xAB sequences. This UTF-8 stream is then converted to a Java string. It's important to note that any character that is not part of an escape sequence will be converted to a byte equivalent to to the low-order 8 bites of the unicode character. This works fine for ascii but can cause transcoding problems for non-ascii characters.
#McDowell:
The sequence:
String phpUnicode = "\u00EF\u00BC\u00A1"
byte[] bytes = phpUnicode.getBytes("ISO-8859-1");
creates a byte array containing as many bytes as the original string has characters and for each character with a unicode value below 256, the same numeric value is stored in the byte-array.
The character FULLWIDTH LATIN CAPITAL LETTER A (U+FF41) is not present in the original String so the fact that it is not in ISO-8859-1 is irrelevant.
I know that transcoding bugs can occur when you convert characters to bytes that's why I said that ISO-8859-1 would only "map every character with a unicode value less than 256 to a byte with the same value"

The character in question is U+FF21 (FULLWIDTH LATIN CAPITAL LETTER A). The PHP form (\xEF\xBC\xA1) is a UTF-8 encoded octet sequence.
In order to decode this sequence to a Java String (which is always UTF-16), you would use the following code:
// \xEF\xBC\xA1
byte[] utf8 = { (byte) 0xEF, (byte) 0xBC, (byte) 0xA1 };
String utf16 = new String(utf8, Charset.forName("UTF-8"));
// print the char as hex
for(char ch : utf16.toCharArray()) {
System.out.format("%02x%n", (int) ch);
}
If you want to decode the data from a string literal you could use code of this form:
public static void main(String[] args) {
String utf16 = transformString("This is \\xEF\\xBC\\xA1 string");
for (char ch : utf16.toCharArray()) {
System.out.format("%s %02x%n", ch, (int) ch);
}
}
private static final Pattern SEQ
= Pattern.compile("(\\\\x\\p{Alnum}\\p{Alnum})+");
private static String transformString(String encoded) {
StringBuilder decoded = new StringBuilder();
Matcher matcher = SEQ.matcher(encoded);
int last = 0;
while (matcher.find()) {
decoded.append(encoded.substring(last, matcher.start()));
byte[] utf8 = toByteArray(encoded.substring(matcher.start(), matcher.end()));
decoded.append(new String(utf8, Charset.forName("UTF-8")));
last = matcher.end();
}
return decoded.append(encoded.substring(last, encoded.length())).toString();
}
private static byte[] toByteArray(String hexSequence) {
byte[] utf8 = new byte[hexSequence.length() / 4];
for (int i = 0; i < utf8.length; i++) {
int offset = i * 4;
String hex = hexSequence.substring(offset + 2, offset + 4);
utf8[i] = (byte) Integer.parseInt(hex, 16);
}
return utf8;
}