I am trying to transfer form data from an Android application to a NodeJs server.
My client code is the following (the strings that can contain UTF-8 characters are the values of params):
final HttpPost post = new HttpPost(url);
final MultipartEntityBuilder mpb = MultipartEntityBuilder.create()
.setCharset(Charset.forName("UTF-8")) // tried with or without this line
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE); // tried with or without this line
for (final Entry<String, String> e : params.entrySet()) {
mpb.addTextBody(e.getKey(), e.getValue());
}
post.setEntity(mpb.build());
final HttpClient httpClient = new DefaultHttpClient();
final HttpResponse response = httpClient.execute(request);
And my server code is the following:
app.post('/accesspoint', function(req, res) {
var body = req.body;
var form = new formidable.IncomingForm();
form.encoding = 'utf-8';
form.parse(req, function(err, fields, files) {
console.log(fields);
...
When my input java params has a value containing an UTF-8 character, the log I get server side prints the corresponding value without this character, so it is kind of swallowed at some point. For instance if my input string is "ê", then my server log will print a "" value.
I use a multipart form as I read that it was the best way to send data that can contain non-ASCII characters. Formidable is also apparently the best node package to handle form that can contain UTF-8 characters.
My client side uses Apache HttpClient 4.3.3.
What am I doing wrong?
Ok so I tested with a simple query and the key value ("foo","[ê]") looked at the headers and I saw that my query was still using ISO-8859-1
Content-Disposition: form-data; name="foo"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
[]
in contradiction to my builder parameter.
I found the solution via https://stackoverflow.com/a/21020435/592392 and changed my code to:
for (final Entry<String, String> e : params.entrySet()) {
mpb.addTextBody(e.getKey(), e.getValue(), ContentType.create("text/plain", Charset.forName("UTF-8")));
}
And now the server gets the UTF8 chars :)
Anyway, the form builder is quite misleading in my opinion.
Related
I setting up a new server in AWS EBS with linux and configured the server to be UTF-8 because I use some text in Hebrew.
The code works on Windows but when I migrated to Linux this stopped working.
The code send HTTP post to another server and get JSON object from it and start to parse, All the Hebrew text is something like this: קבוצ×
I tried to add to AWS software configuration : JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
I tried to add JAVA_OPTIONS="-Dfile.encoding=UTF-8" to tomcat8.conf file. (aslo tried with JAVA_OPTS).
I tried to add catalina options also with encoding.
HttpPost request = new HttpPost(URL);
// Create JSON and set the API Token
JSONObject SendJson = new JSONObject();
SendJson.put(field, key);
StringEntity params =new StringEntity(SendJson.toString());
request.setEntity(params);
// set Header Type
request.addHeader("content-type", "application/json;charset=UTF-8");
request.addHeader("Accept-Encoding", "UTF-8");
// Execute and wait for response
HttpResponse httpResponse = httpClient.execute(request);
HttpEntity resEntity = httpResponse.getEntity();
// cast InputStream to String for JSON conversion
BufferedReader reader = new BufferedReader(new InputStreamReader( resEntity.getContent()));
//String strInputStream = EntityUtils.toString(resEntity,"UTF-8");
String strInputStream = reader.readLine();
//debug
System.out.println("David Encoding problem");
System.out.println(Charset.defaultCharset().name());
System.out.println(strInputStream);
JSONObject answerObj = new JSONObject(strInputStream);
Charset.defaultCharset().name() - output UTF-8
I get JSON with no Hebrew text :/
part of the JSON for example:
{"item_group_id":1,"item_group_name":"קבוצה ×","picture_link":""},
Thanks,
David
I fixed it.
I changed the locale to he_IL and he_IL.UTF8, then restarted the server and it worked.
Thanks!
Is it please possible to escape UTF-8 characters when using JSON.toString() method in jetty-util-ajax package?
I understand that the package might be an internal library, but until now it works well for me in a servlet which among other tasks sends push
notifications to mobile phones via FCM (Firebase Cloud Messaging) and ADM (Amazon Device Messaging).
However my problem with is that ADM does not accept any
UTF-8 chars (in my case Cyrillic) and reproducibly fails with the rather
misleading error message (Amazon talks about XML in all its error messages while their API expects JSON data being POSTed):
<SerializationException>
<Message>Could not parse XML</Message>
</SerializationException>
java.lang.IllegalStateException:
unknown char '<'(60) in |||<SerializationException>| <Message>Could
not parse XML</Message>|</SerializationException>||
So is there maybe some possibility in Jetty 9.4.8.v20171121 to encode the chars?
Here is my Java code:
// this string is POSTed to ADM server
public String toAdmBody() {
Map<String, Object> root = new HashMap<>();
Map<String, String> data = new HashMap<>();
root.put("data", data);
data.put("body", mBody);
// ADM does not accept integers for some reason
data.put("gid", String.valueOf(mGid));
// HOW TO ENCODE UTF-8 CHARS TO \uXXXX HERE?
return JSON.toString(root);
}
private void postMessage(String registrationId, int uid, String jsonStr) {
mHttpClient.POST(String.format("https://api.amazon.com/messaging/registrations/%1$s/messages", registrationId))
.header(HttpHeader.ACCEPT, "application/json; charset=utf-8")
.header(HttpHeader.CONTENT_TYPE, "application/json; charset=utf-8")
.header(HttpHeader.AUTHORIZATION, "Bearer " + mAccessToken)
.header("X-Amzn-Type-Version", "com.amazon.device.messaging.ADMMessage#1.0")
.header("X-Amzn-Accept-Type", "com.amazon.device.messaging.ADMSendResult#1.0")
// add registrationID and notification body - for retrying after fetching token
.attribute("registrationId", registrationId)
.attribute("body", jsonStr)
.content(new StringContentProvider(jsonStr))
.send(mMessageListener);
}
When looking at the Jetty source code JSON.java there is some decoding happening (i.e. from \uXXXX to UTF-8 chars):
case 'u':
char uc = (char)((TypeUtil.convertHexDigit((byte)source.next()) << 12)
+ (TypeUtil.convertHexDigit((byte)source.next()) << 8)
+ (TypeUtil.convertHexDigit((byte)source.next()) << 4)
+ (TypeUtil.convertHexDigit((byte)source.next())));
scratch[i++] = uc;
break;
But how to do the reverse thing?
The ContentProvider's are the source of the Content-Type, not your manually set header.
Change your ...
.content(new StringContentProvider(jsonStr))
to ...
.content(new StringContentProvider(jsonStr, "application/json", StandardCharsets.UTF_8))
as the default for StringContentProvider is text/plain (not JSON)
How do I pass json as a query parameter on Rest Post web service in java
For example:
https://testvpa.net/WebService/ViALoggingRestWS/ViALoggingService.svc/StartCall?parameter={"machineName":"KK-IVR01","appName":"KKApp","startTime":"2018-02-06T21:38:32","portID":"01","ani":"9189280000","dnis":"8559281111","ctiCallID":"01"}
I am trying something like this:
....
try{
JSONObject obj = new JSONObject();
obj.put("machineName",machineName);
obj.put("appName", appName);
obj.put("startTime", formattedCurrentDate);
obj.put("portID",portID);
obj.put("ani",ani);
obj.put("dnis", dnis);
obj.put("ctiCallID", ctiCallID);
String strobj = obj.toString();
String uri = wsUri+"/StartCall?";
HttpClient client = new HttpClient();
client.getParams().setConnectionManagerTimeout(1300);
client.getParams().setSoTimeout(13000);
PostMethod method = new PostMethod(uri);
method.addRequestHeader("Content-Type", "application/x-www-form-urlencoded");
method.setQueryString("parameter="+strobj );
int statusCode = client.executeMethod(method);
byte[] responseBody = method.getResponseBody();
output = new String(responseBody);
}
....
But I am getting an "Invalid URI" at runtime. It doesn't seem to like the query parameter being a json string. I read somewhere about encoding the json string ... Do I somehow need to encode the json string?
If you are using POST request, you should pass the json object in the body of the request and not in the query params.
You can check this question for more details: Which characters make a URL invalid?
Generally speaking, the accepted characters in a URI are: [A-Z][a-z][0-9]-._~
The following characters are also allowed, but have special meaning in some parts of the URI: :/?#[]#!$&'()*+,;=
Every other character is not allowed and must be percent-encoded. The second set of characters should also be percent-encoded to avoid any parsing problems.
To percent encode a character you take its hex value (e.g. for the space character the hex value is 20) and prefix it with the % character. So John Doe becomes John%20Doe.
The method below is hit or miss if it parses the request properly...
Here is what the request looks like:
POST /addEvent/ HTTP/1.1
Host: localhost:1234
Content-Type: multipart/form-data; boundary=Boundary+0xAbCdEfGbOuNdArY
Accept-Encoding: gzip, deflate
Content-Length: 201
Accept-Language: en;q=1, fr;q=0.9, de;q=0.8, ja;q=0.7, nl;q=0.6, it;q=0.5
Accept: application/json
Connection: keep-alive
User-Agent: XXXXXXXXXXXXXXXXXX
--Boundary+0xAbCdEfGbOuNdArY
Content-Disposition: form-data; name="userInfo"
{ "user_id" : 1, "value" : "Water", "typeCode" : "Searched" }
Here is how we are extracting it now...
//Key where the request begins
String keyString = "\"userInfo\"";
//Get the index of the key
int end = bufferedJson.lastIndexOf("\"userInfo\"");
//Create substring at beginning of the json
String json = bufferedJson.substring(end+keyString.length(), bufferedJson.length());
//Convert json to feed item
Gson gson = new Gson();
Event eventItem = gson.fromJson(json, Event.class);
I get this error pretty often:
Expected BEGIN_OBJECT but was STRING at line 1 column 1
How can we parse this efficiently?
Use Apache HTTP Client 4 to read Http response body in a convenient way. If you need to marshall your json further to a java object then make use of jackson. Here is the sample code:
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
/**
* This example demonstrates the use of the {#link ResponseHandler} to simplify
* the process of processing the HTTP response and releasing associated resources.
*/
public class ClientWithResponseHandler {
public final static void main(String[] args) throws Exception {
HttpClient httpclient = new DefaultHttpClient();
try {
HttpGet httpget = new HttpGet("http://www.google.com/");
System.out.println("executing request " + httpget.getURI());
// Create a response handler
ResponseHandler<String> responseHandler = new BasicResponseHandler();
// Body contains your json stirng
String responseBody = httpclient.execute(httpget, responseHandler);
System.out.println("----------------------------------------");
System.out.println(responseBody);
System.out.println("----------------------------------------");
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}
To parse this better:
First, it seems like you're taking a "raw" HTTP POST request, and then reading it line by line using BufferedReader (your comment suggests this), this way you'll lose the new line chars; if you are going to do so, add a new line ("\n") every time you read a line to your final String, this way it doesn't lose the new lines and facilitates the things for the next step.
Now, with this final String, you can use this:
String json = null;
Pattern pattern = Pattern.compile("\n\n");
Matcher matcher = pattern.matcher(myString); // myString is the String you built from your header
if(matcher.find() && matcher.find()) {
json = myString.substring(matcher.start() + 2);
} else {
// Handle error: json string wasn't found
}
CAVEATS: this works if:
POST Request will always be multipart/form-data
There are not other parameters in the request
you STOP reading the request as soon as you find your json data
you included "\n" every time you read a line as I said in the first step
Personally I wouldn't read the raw HTTP header, I'd rather use Apache commons FileUpload or the like, but if your are going to do it this way, I think this is the less terrible solution.
You can use
gson().fromJson(request.getReader(), Event.class);
or
String json = request.getReader().readLine();
gson().fromJson(json, Event.class);
I'm trying to submit a form to a certain website from Java (over HTTP), but when reading the response I don't see what I expected.
What exactly I do: first of all, I open the website in a browser, fill in the form by hand and submit it. In Chrome I can see the data that goes over the wire, i.e.:
Request URL:http://wizzair.com/en-GB/Select
Request Method:POST
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:1061
Content-Type:application/x-www-form-urlencoded
Cookie:WRUID=0; ASP.NET_SessionId=3e3ahach1d34oyhtoqfshxhe; Culture=en-GB; __utma=17431487.361991764.1292186668.1354138010.1354651562.81; __utmb=17431487.9.9.1354652614319; __utmc=17431487; __utmz=17431487.1319145359.34.18.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=wizz
Host:wizzair.com
Origin:http://wizzair.com
Referer:http://wizzair.com/en-GB/Select
User-Agent:Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11
Form Dataview URL encoded
__EVENTTARGET:HeaderControlGroupRibbonSelectView_AvailabilitySearchInputRibbonSelectView_ButtonSubmit
__VIEWSTATE:/wEPDwUBMGRkNSMYF94e4mXCiiJGEJbRixyidoa2QXSambTT2mm6cLs=
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$OriginStation:EIN
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DestinationStation:OTP
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DepartureDate:02/02/2013
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ReturnDate:05/02/2013
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountADT:1
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountCHD:0
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountINFANT:0
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$BaggageCount:0
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ButtonSubmit:Search
So I try to simulate the same request from a Java program, i.e.:
public void doSubmit(String url, Map<String, String> data) throws Exception {
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestProperty("Cookie", "WRUID=0; ASP.NET_SessionId=3e3ahach1d34oyhtoqfshxhe; Culture=en-GB; __utma=17431487.361991764.1292186668.1354138010.1354651562.81; __utmb=17431487.9.9.1354652614319; __utmc=17431487; __utmz=17431487.1319145359.34.18.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=wizz");
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator();
String content = "";
for(int i=0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if(i!=0) {
content += "&";
}
content += key + "=" + URLEncoder.encode(data.get(key), "UTF-8");
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = "";
while((line=in.readLine())!=null) {
System.out.println(line);
}
in.close();
}
.... which I call with the following parameters, as seen in the HTTP form data above:
String url = "http://wizzair.com/en-GB/Select";
Map<String, String> data = new TreeMap<String, String>();
data.put("__EVENTTARGET", "HeaderControlGroupRibbonSelectView_AvailabilitySearchInputRibbonSelectView_ButtonSubmit");
data.put("__VIEWSTATE", "/wEPDwUBMGRkNSMYF94e4mXCiiJGEJbRixyidoa2QXSambTT2mm6cLs=\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$OriginStation:EIN\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DestinationStation:OTP\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DepartureDate:02/02/2013\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ReturnDate:05/02/2013\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountADT:1\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountCHD:0\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountINFANT:0\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$BaggageCount:0\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ButtonSubmit:Search"
However, the response I get back is simply a generic web page from that web site, not the answer I expected. What am I doing wrong ?
Thanks a lot,
Greetings,
Sorin
I think the way you populate the post data is not correct. You should have ten or so key/value pairs, and not only two. The second item you see in Chrome is not one big string. Things like 'HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$OriginStation' is a key in its own right. This also applies to the other very very ugly named keys who all start with 'HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$'.
Further, you need to encode the 'key' of the each post data item as well and not only the value of the key (because of the $-characters inside). Encode both sides with separate calls, to avoid encoding the assignment '='.
You also need to remove the newline at the end of your original second, but now split key/value, because it is just not there. This also applies to the third, fourth, etc. item.
Be carefull with interpreting what you see in Chrome :-P
Another thing: the website you are referring to is strongly session based: it keeps track of the current state of your interaction using the cookie value 'ASP.NET_SessionId'. This value is only short lifed. In general you should first make a call to this website without this cookie value, and the website will provide it for you (redirecting you to a country specific localized catch-all page). Subsequently you can use its value in your (second) request to harvest data. If you supply an invalid session id, you will be redirected to the same default page again, again and again.