Develop Reference

Datadog string encoding when character is not from English alphabet

I have a service and send Datadog events from it using com.github.arnabk.java-dogstatsd-client. In order to send json string I use JsonObject where I put all properties which I need then convert it to string using toString() method on JsonObject and send string as a message body. Everything works perfect unless I have a character in a string which is not from english alphabet. Example: µ. In this case instead of having correct json {"Smth":"µ"} in Datadog I'm getting incorrect string without closing curly brace {"Smth":"µ". Has anybody experienced the same and knows how to deal with this?

Illegal Character in XML are not being replaced

SOLUTION So this was not an xml issue at all. My xml escapes were done properly, however there was an encoding issue. So i would like to share my solution with everyone, i hope you find this useful.
public static String entityEncode(String text) throws UnsupportedEncodingException {
String result = text;
if (result == null) {
return result;
}
byte ptext[] = result.getBytes("ISO-8859-1");
String value = new String(ptext, "UTF-8");
String temp = XMLStringUtil.escapeControlChrs(value);
return temp;
}
EXPLANATION The xml function above is for XML 1.0. We take our given text, convert it into a byte since String does not have an associated encoding. After which we create a new string off of the byte in "UTF-8". That is also why java was just telling me that character reference error with &#, it couldn't recognize the character at fault. Now that I did the encoding and assigned it to UTF-8, there are no issues and the xml escape proceeds properly!
EDIT: How do i print out all illegal xml characters in the provided string? According to StringEscapeUtils.escapeXml parameters? The problem i have is that i don't want to escape everything, because it doesn't properly decode after. So right now, i just need to find out what my invalid characters in the text are. The oens that are causing issues and need to be encoded.
I have the following error message:
ERROR: 'Character reference "&#'
ERROR: 'com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Character reference "&#'
It does not specifically tell me what the character is which is a problem.
I do my original XML parse to convert to an xml document and then after that. I sanitize further to remove illegal characters
String xml10pattern = "[^"
+ "\u0009\r\n"
+ "\u0020-\uD7FF"
+ "\uE000-\uFFFD"
+ "\ud800\udc00-\udbff\udfff"
+ "]";
However, it's not removing them so i'm not sure how to go about this. Currently i have:
String temp = entityEncode(temp);
String legal = temp.replaceAll(xml10pattern , "");
item.setResponseBody(legal);
Entity encode just uses a standard xml parse class to escape characters XMLStringUtil.escapeControlChrs which is based off of StringEscapeUtils.escapeXml and just has additional escapes, nothing removed. But something is being missed.

Concatenating a string variable to a url properly

This is a follow up question of Removing linebreak from php json output.I couldn't find out what was making that problem but i somehow got rid of value <br ...JSONException.
The Issue
when i use
String url = "http://192.168.32.1/Aafois/notice.php?isBatch=2010&section1='IT'";
I get what i want i.e parsing the JSON to my android app.However when i use
String URL="http://192.168.32.1/Aafois/notice.php?isBatch="+isbatch+"&section1="+"'"+section1+"'";
I get Value of java.lang.string can't be converted to JSONArray...JSONException.So obviously there must be some problem there in this previous line.isbatch is an integer variable and secton1 is a string variable which is URL encoded to utf-8.
P.S
I need single quote ' before and after the variable section1 as the url goes like
http://192.168.32.1/Aafois/notice.php?isBatch=2010&section1='IT'.

I think of your variables is a JSONArray. To concatenate everything, it tries to turn your String into a JSONArray, which is not possible as your string is not JSON but part of a URL.

I checked for the variables isBatch and section1.They were returning null.Modified what was needed.Worked as expected.No more JSONException.

I solved the issue with this
String batch= "C";
String section = "2";
String URL = "http://192.168.32.1/Aafois/notice.php?isBatch=";
URL= URL.concat(batch+"& section ="+section);

conwert object which contains strings with utf-8 to string with proper coding

I'm processing MMS and got it text part as :
mmsBodyPart.getContent();
it's simpy Object. Now i need to convert it to String using utf-8. I have tried:
String contentText = (String) mmsBodyPart.getContent();
but it doesn't works with specyfics characters and some strange chars appear.
Also i tried :
String content = new String(contentText.getBytes("UTF-8"), "UTF-8"));
not a mystery that also failed.
How that can be done ?
EDIT: Problem was caused by bad encoding in file. Nothing wrong was in code, ya didn't thought about it in first place...

Strings haven't an Encoding in Java. If you need one, you should use byte[] with Encoding to get a String

What is the most efficient way to format UTF-8 strings in java?

I am doing the following:
String url = String.format(WEBSERVICE_WITH_CITYSTATE, cityName, stateName);
String urlUtf8 = new String(url.getBytes(), "UTF8");
Log.d(TAG, "URL: [" + urlUtf8 + "]");
Reader reader = WebService.queryApi(url);
The output that I am looking for is essentially to get the city name with blanks (e.g., "Overland Park") to be formatted as Overland%20Park.
Is it this the best way?

Assuming you are actually wanting to encode your string for use in a URL (ie, "Overland Park" can also be formatted as "Overland+Park") you want URLEncoder.encode(url, "UTF-8"). Other unsafe characters will be converted to the %xx format you are asking for.

The simple answer is to use URLEncoder.encode(...) as stated by #Recurse. However, if part or all of the URL has already been encoded, then this can lead to double encoding. For example:
http://foo.com/pages/Hello%20There
or
http://foo.com/query?keyword=what%3f
Another concern with URLEncoder.encode(...) is that it doesn't understand that certain characters should be escaped in some contexts and not others. So for example, a '?' in a query parameter should be escaped, but the '?' that marks the start of the "query part" should not be escaped.
I think that safer way to add missing escapes would be the following:
String safeURI = new URI(url).toASCIIString();
However, I haven't tested this ...