Develop Reference

Illegal base64 character "a" using java.util.Base64 from within Scala

Suppose I have the following Base64 encoded String from a github API call to a file:
LyoKICogQ29weXJpZ2h0IDIwMTkgY29tLmdpdGh1Yi50aGVvcnlkdWRlcwog
KgogKiBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNp
b24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKICogeW91IG1heSBub3QgdXNlIHRo
aXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNl
LgogKiBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQK
ICoKICogICAgIGh0dHA6Ly93d3cuYXBhY2hlLm9yZy9saWNlbnNlcy9MSUNF
TlNFLTIuMAogKgogKiBVbmxlc3MgcmVxdWlyZWQgYnkgYXBwbGljYWJsZSBs
YXcgb3IgYWdyZWVkIHRvIGluIHdyaXRpbmcsIHNvZnR3YXJlCiAqIGRpc3Ry
aWJ1dGVkIHVuZGVyIHRoZSBMaWNlbnNlIGlzIGRpc3RyaWJ1dGVkIG9uIGFu
ICJBUyBJUyIgQkFTSVMsCiAqIFdJVEhPVVQgV0FSUkFOVElFUyBPUiBDT05E
SVRJT05TIE9GIEFOWSBLSU5ELCBlaXRoZXIgZXhwcmVzcyBvciBpbXBsaWVk
LgogKiBTZWUgdGhlIExpY2Vuc2UgZm9yIHRoZSBzcGVjaWZpYyBsYW5ndWFn
ZSBnb3Zlcm5pbmcgcGVybWlzc2lvbnMgYW5kCiAqIGxpbWl0YXRpb25zIHVu
ZGVyIHRoZSBMaWNlbnNlLgogKi8KCnBhY2thZ2UgY29tLmdpdGh1Yi50aGVv
cnlkdWRlcy5tb2RlbAoKaW1wb3J0IGNvbS5naXRodWIudGhlb3J5ZHVkZXMu
dXRpbC5LaXZ5UHJldHR5UHJpbnRlcgppbXBvcnQgb3JnLmJpdGJ1Y2tldC5p
bmt5dG9uaWsua2lhbWEuPT0+CmltcG9ydCBvcmcuYml0YnVja2V0Lmlua3l0
b25pay5raWFtYS5yZXdyaXRpbmcuUmV3cml0ZXIuXwppbXBvcnQgb3JnLmJp
dGJ1Y2tldC5pbmt5dG9uaWsua2lhbWEucmV3cml0aW5nLlN0cmF0ZWd5Cgov
KioKICogQmFzZSBUeXBlIGZvciBhbGwgbm9kZXMgb2YgYSBLaXZ5LUFTVAog
Ki8KdHJhaXQgQVNUTm9kZSBleHRlbmRzIEZvbGRhYmxlQVNUIHsgc2VsZiA9
PgogIC8qKgogICAqIFRyYXZlcnNlcyB0aGUgQVNUTm9kZSBhbmQgYXBwbGll
cyBTdHJhdGVneSBgc2Agb250byBgc2VsZmAgYW5kIGFsbCBjaGlsZHJlbiBv
ZiBzZWxmLgogICAqCiAgICogYHNgIGlzIGhlcmVieSBhcHBsaWVkIGJvdHRv
bSB1cCBpbiBsZWZ0IHRvIHJpZ2h0IG9yZGVyLgogICAqCiAgICogQHNlZSBb
W2h0dHBzOi8vYml0YnVja2V0Lm9yZy9pbmt5dG9uaWsva2lhbWEvc3JjLzAz
MjYzMGZhMjFkZGFkNWNmMzNjYmQ2ZWY5YzJmMDI3ODY2MWE2NzUvd2lraS9S
ZXdyaXRpbmcubWRdXQogICAqIEBwYXJhbSBzIHN0cmF0ZWd5IHRoYXQgaXMg
YXBwbGllZCB0byBgc2VsZmAgYW5kIGFsbCBjaGlsZHJlbi4KICAgKiBAcmV0
dXJuIGEgcmV3cml0dGVuIEFTVE5vZGUgYWNjb3JkaW5nIHRvIHRoZSBzdHJh
dGVneSBgc2AKICAgKi8KICBwcml2YXRlW3RoZW9yeWR1ZGVzXSBkZWYgdHJh
dmVyc2VBbmRBcHBseShzOlN0cmF0ZWd5KTpBU1ROb2RlCgogIC8qKgogICAq
IFJld3JpdGUgdGhlIEFTVE5vZGUgYHNlbGZgIGJ5IHRoZSBzcGVjaWZpY2F0
aW9uIG9mIGEgcGFydGlhbCBmdW5jdGlvbiBgZnBgLgogICAqCiAgICogSWYg
d2Ugd2FudCB0byBjaGFuZ2UgYSBzcGVjaWZpYyBbW21vZGVsLlB5dGhvbl1d
LW5vZGUgaW4gdGhlIEFTVCBmb3IgZXhhbXBsZSB3ZSBjb3VsZAogICAqIGFw
cGx5IHRoZSBmb2xsb3dpbmcgcmV3cml0ZSBzdHJhdGVneToKICAgKnt7ewog
ICAqICAgYXN0LnJld3JpdGUoewogICAqICAgIGNhc2UgUHl0aG9uKCJbMSwy
LDNdIikgPT4gUHl0aG9uKCJbMSwyLDMsNF0iKQogICAqICAgfSkKICAgKn19
fQogICAqCiAgICogUGxlYXNlIG5vdGUsIHRoYXQgQVNUTm9kZXMgY2FuIG5v
dCBiZSByZXdyaXR0ZW4gYXJiaXRyYXJpbHkuIFNpbmNlIGVhY2ggQVNUTm9k
ZSBpbXBsaWVzCiAgICogYSBzcGVjaWZpYyBwYXJhbWV0ZXIgbGlzdC4gQW4g
QVNUIGhhcyB0byBzdGF5IHN0cnVjdHVyZS1jb25zaXN0ZW50IGFmdGVyIGFw
cGx5aW5nIHJld3JpdGluZyBydWxlcy4KICAgKiBBIHJld3JpdGluZyBydWxl
IGFzOgogICAqIHt7ewogICAqICAgewogICAqICAgIGNhc2UgUHl0aG9uKHMp
ID0+IFRvcExldmVsKE5pbCkKICAgKiAgIH0KICAgKiB9fX0KICAgKiBpcyBu
b3QgdmFsaWQgYXMgYSBbW21vZGVsLlRvcExldmVsXV0tbm9kZSBjYW4gbm90
IG9jY3VyIGF0IHBvc2l0aW9ucyB3aGVyZSBhIFtbbW9kZWwuUHl0aG9uXV0t
bm9kZSBjYW4uCiAgICoKICAgKiBAc2VlIFtbaHR0cHM6Ly9iaXRidWNrZXQu
b3JnL2lua3l0b25pay9raWFtYS9zcmMvMDMyNjMwZmEyMWRkYWQ1Y2YzM2Ni
ZDZlZjljMmYwMjc4NjYxYTY3NS93aWtpL1Jld3JpdGluZy5tZF1dCiAgICog
QHBhcmFtIGZwIFBhcnRpYWwgZnVuY3Rpb24gdGhhdCBkZWZpbmVzIGhvdyB0
aGUgYXN0IHNob3VsZCBiZSByZXdyaXR0ZW4uCiAgICogQHJldHVybiBBIHJl
d3JpdHRlbiBBU1QgYWNjb3JkaW5nIHRvIHRoZSBzcGVjaWZpY2F0aW9uIGlu
IGBmcGAgb3IgdGhlIHNhbWUgYXN0IGlmIGBmcGAgY291bGQgbm90IGJlIGFw
cGxpZWQuCiAgICovCiAgZGVmIHJld3JpdGUoZnA6QVNUTm9kZSA9PT4gQVNU
Tm9kZSk6IEFTVE5vZGUgPSBzZWxmLnRyYXZlcnNlQW5kQXBwbHkocnVsZShm
cCkpCgogIC8qKgogICAqIFRyYW5zZm9ybXMgYHNlbGZgIGludG8gYSB3ZWxs
IGZvcm1hdHRlZCBraXZ5IHByb2dyYW0gdGhhdCBjYW4gYmUgd3JpdHRlbgog
ICAqIGludG8gYSBmaWxlLgogICAqCiAgICogVGhlIGZvbGxvd2luZyBBU1RO
b2RlIGZvciBleGFtcGxlOgogICAqIHt7ewogICAqICAgVG9wTGV2ZWwoCiAg
ICogICAgTGlzdCgKICAgKiAgICAgIFJvb3QoCiAgICogICAgICAgIFdpZGdl
dCgKICAgKiAgICAgICAgICBQbG90LAogICAqICAgICAgICAgIExpc3QoCiAg
ICogICAgICAgICAgICBXaWRnZXQoCiAgICogICAgICAgICAgICAgIExpbmVH
cmFwaCwKICAgKiAgICAgICAgICAgICAgTGlzdCgKICAgKiAgICAgICAgICAg
ICAgICBQcm9wZXJ0eShiYWNrZ3JvdW5kX25vcm1hbCxMaXN0KCcnKSksCiAg
ICogICAgICAgICAgICAgICAgUHJvcGVydHkoYmFja2dyb3VuZF9jb2xvcixM
aXN0KFswLDAsMCwxXSkpCiAgICogICApKSkpKSkpCiAgICogfX19CiAgICoK
ICAgKiBpcyBwcmludGVkOgogICAqIHt7ewogICAqIFBsb3Q6CiAgICogIExp
bmVHcmFwaDoKICAgKiAgICBiYWNrZ3JvdW5kX25vcm1hbDogJycKICAgKiAg
ICBiYWNrZ3JvdW5kX2NvbG9yOiBbMCwwLDAsMV0KICAgKiB9fX0KICAgKgog
ICAqIEByZXR1cm4gQSBmb3JtYXR0ZWQgQVNUTm9kZSB0aGF0IGNhbiBiZSBp
bnRlcnByZXRlZCBhcyBhIEtpdnkgZmlsZS4KICAgKi8KICBkZWYgcHJldHR5
OlN0cmluZyA9IEtpdnlQcmV0dHlQcmludGVyLmZvcm1hdChzZWxmKS5sYXlv
dXQKfQ==
As far as I see, this encoding is correct and only contains the standard alphabet of characters for a Base64 encoding. If I decode this encoding here, I get a correct translation. However, I tried various approaches to decode it programmatically and did not find a solution yet.
Let contentEncoded be the string containing the encoded file. I tried the following:
java.util.Base64.getDecoder.decode(contentEncoded)
java.util.Base64.getDecoder.decode(contentEncoded.getBytes)
java.util.Base64.getDecoder.decode(contentEncoded.getBytes(StandardCharsets.UTF_8))
java.util.Base64.getUrlDecoder.decode(contentEncoded))
java.util.Base64.getUrlDecoder.decode(contentEncoded.getBytes(StandardCharsets.UTF_8))
java.util.Base64.getMimeDecoder.decode(contentEncoded.replaceAll("\\n", "").replaceAll("\\r", ""))
However, all of them resulted in an error message: java.lang.IllegalArgumentException: Illegal base64 character a.
My question is: Am I not seeing something obvious? Are there some hidden control characters? Has anybody had similar issues and was able to fix them?

Just remove line breaks and it should work.
contentEncoded.replace("\n", "")

The following snippet decodes the encoding correctly:
val decodedWithMime = java.util.Base64.getMimeDecoder.decode(contentEncoded)
val convertedByteArray = decodedWithMime.map(_.toChar).mkString
as pointed out by comments, the error message Illegal Base64 character a corresponds to the hex value for the newline character \n. Using the Mime Decoder it is possible to decode the string without removing the newline characters beforehand.

Encode to UTF-8. Encode character eg. ö to ö

I want to encode a string in Android to UTF-8. For example this string:
Grüne Ähren beißen Flöhe
to
Grüne Ãhren beiÃen Flöhe
But no matter what I do I encode ü to ü or ü to %C3%BC (online often called 'raw URL encode').
Found solutions to convert to byte[] or URI.toASCIIString(). But non of them work for me.
UPDATE
I am participating in the eBay partner network and try to concat a searchword to my partner url.
The people of eBay must use a wrong character set, as UTF-8 URL encoded string don't work.
A searchword with UTF-8 URL encoding
(Grüne Ähren beißen Flöhe
to
Gr%C3%BCne%20%C3%84hren%20bei%C3%9Fen%20Fl%C3%B6he)
comes out to this result in the eBay searchbox:
If I encode my searchword with ISO_8859_1 it works (Grüne Ãhren beiÃen Flöhe):
Thank you very much community

What you essentially want is to convert a String to it's byte representation according to UTF-8 and interpret these bytes using a different Charset, such as ISO-8859-1.
This is usually the cause of many problems. You want to intentionally do what most developers do incorrectly (or they simply ignore the problems of charsets).
Since you just need this to work, use this piece of code:
byte[] bytes = "Grüne Ähren beißen Flöhe".getBytes("UTF-8");
String result = new String(bytes, "ISO-8859-1");
see it at work here.

Base64 encoding btoa

I am encoding some text on my frontend part using btoa function:
const encodedText = btoa(searchText);
This seems to work totally fine and decoding goes like this on backend part:
byte[] decodedBytes = Base64.getDecoder().decode(searchedText);
String decodedString = new String(decodedBytes, Charset.defaultCharset());
Which also works fine. However, this seems to fail when using ü letter. My program encodes it as A==, and as far as I know, it should be w7w=
I am not sure what I did wrong.

You could use
const encodedText = btoa(unescape(encodeURIComponent(searchText)));
instead to encode unicode characters first.
See Unicode strings and The "Unicode Problem" for further reading.
console.log(btoa('ü'));
console.log(btoa(unescape(encodeURIComponent('ü'))));

Charset bug in AES Decryption on Android system

Good evening!
In my android app the smartphones load a AES encrypted String from my server and store it in a variable. After that process the variable and a key are pass to a method which decrypt the string. My mistake is that german umlauts (ä, ü, ö) aren't correct decoded. All umlauts displayed as question marks with black background...
My Code:
public static String decrypt(String input, String key) {
byte[] output = null;
String newString = "";
try {
SecretKeySpec skey = new SecretKeySpec(key.getBytes(), "AES");
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, skey);
output = cipher.doFinal(Base64.decode(input, Base64.DEFAULT));
newString = new String(output);
} catch(Exception e) {}
return newString;
}
The code works perfectly - only umlauts displayed not correctly, an example is that (should be "ö-ä-ü"):
How can I set the encoding of the decrypted String? In my iOS app I use ASCII to encoding the decoded downloaded String. That works perfectly! Android and iOS get the String from the same Server on the same way - so I think the problem is the local Code above.
I hope you can help me with my problem... Thanks!

There is no text but encoded text.
It seems like you are guessing at the character set and encoding—That's no way to communicate.
To recover the text, you need to reverse the original process applied to it with the parameters associated with each step.
For explanation, assume that the server is taking text from a Java String and sending it to you securely.
String uses the Unicode character set (specifically, Unicode's UTF-16 encoding).
Get the bytes for the String, using some specific encoding, say ISO8859-1. (UTF-8 could be better because it is also an encoding for the Unicode character set, whereas ISO8859-1 has a lot fewer characters.) As #Andy points out, exceptions are your friends here.
Encrypt the bytes with a specific key. The key is a sequence of bytes, so, if you are generating this from a string, you have to use a specific encoding.
Encode the encrypted bytes with Base64, producing a Java String (again, UTF-16) with a subset of characters so reduced that it can be re-encoded in just about any character encoding and placed in just about any context such as SMTP, XML, or HTML without being misinterpreted or making it invalid.
Transmit the string using a specific encoding. An HTTP header and/or HTML charset value is usually used to communicate which encoding.
To receive the text, you have to get:
the bytes,
the encoding from step 5,
the key from step 3,
the encoding from step 3 and
the encoding from step 2.
Then you can reverse all of the steps. Per your comments, you discovered you weren't using the encoding from step 2. You also need to use the encoding from step 3.

How do you convert binary data to Strings and back in Java?

I have binary data in a file that I can read into a byte array and process with no problem. Now I need to send parts of the data over a network connection as elements in an XML document. My problem is that when I convert the data from an array of bytes to a String and back to an array of bytes, the data is getting corrupted. I've tested this on one machine to isolate the problem to the String conversion, so I now know that it isn't getting corrupted by the XML parser or the network transport.
What I've got right now is
byte[] buffer = ...; // read from file
// a few lines that prove I can process the data successfully
String element = new String(buffer);
byte[] newBuffer = element.getBytes();
// a few lines that try to process newBuffer and fail because it is not the same data anymore
Does anyone know how to convert binary to String and back without data loss?
Answered: Thanks Sam. I feel like an idiot. I had this answered yesterday because my SAX parser was complaining. For some reason when I ran into this seemingly separate issue, it didn't occur to me that it was a new symptom of the same problem.
EDIT: Just for the sake of completeness, I used the Base64 class from the Apache Commons Codec package to solve this problem.

String(byte[]) treats the data as the default character encoding. So, how bytes get converted from 8-bit values to 16-bit Java Unicode chars will vary not only between operating systems, but can even vary between different users using different codepages on the same machine! This constructor is only good for decoding one of your own text files. Do not try to convert arbitrary bytes to chars in Java!
Encoding as base64 is a good solution. This is how files are sent over SMTP (e-mail). The (free) Apache Commons Codec project will do the job.
byte[] bytes = loadFile(file);
//all chars in encoded are guaranteed to be 7-bit ASCII
byte[] encoded = Base64.encodeBase64(bytes);
String printMe = new String(encoded, "US-ASCII");
System.out.println(printMe);
byte[] decoded = Base64.decodeBase64(encoded);
Alternatively, you can use the Java 6 DatatypeConverter:
import java.io.*;
import java.nio.channels.*;
import javax.xml.bind.DatatypeConverter;
public class EncodeDecode {
public static void main(String[] args) throws Exception {
File file = new File("/bin/ls");
byte[] bytes = loadFile(file, new ByteArrayOutputStream()).toByteArray();
String encoded = DatatypeConverter.printBase64Binary(bytes);
System.out.println(encoded);
byte[] decoded = DatatypeConverter.parseBase64Binary(encoded);
// check
for (int i = 0; i < bytes.length; i++) {
assert bytes[i] == decoded[i];
}
}
private static <T extends OutputStream> T loadFile(File file, T out)
throws IOException {
FileChannel in = new FileInputStream(file).getChannel();
try {
assert in.size() == in.transferTo(0, in.size(), Channels.newChannel(out));
return out;
} finally {
in.close();
}
}
}

If you encode it in base64, this will turn any data into ascii safe text, but base64 encoded data is larger than the orignal data

See this question, How do you embed binary data in XML?
Instead of converting the byte[] into String then pushing into XML somewhere, convert the byte[] to a String via BASE64 encoding (some XML libraries have a type to do this for you). The BASE64 decode once you get the String back from XML.
Use http://commons.apache.org/codec/
You data may be getting messed up due to all sorts of weird character set restrictions and the presence of non-priting characters. Stick w/ BASE64.

How are you building your XML document? If you use java's built in XML classes then the string encoding should be handled for you.
Take a look at the javax.xml and org.xml packages. That's what we use for generating XML docs, and it handles all the string encoding and decoding quite nicely.
---EDIT:
Hmm, I think I misunderstood the problem. You're not trying to encode a regular string, but some set of arbitrary binary data? In that case the Base64 encoding suggested in an earlier comment is probably the way to go. I believe that's a fairly standard way of encoding binary data in XML.