Develop Reference

UTF-8 Encoding issue with HTTP Post object on AWS Elastic Beanstalk

I setting up a new server in AWS EBS with linux and configured the server to be UTF-8 because I use some text in Hebrew.
The code works on Windows but when I migrated to Linux this stopped working.
The code send HTTP post to another server and get JSON object from it and start to parse, All the Hebrew text is something like this: קבוצ×
I tried to add to AWS software configuration : JAVA_TOOL_OPTIONS -Dfile.encoding=UTF8
I tried to add JAVA_OPTIONS="-Dfile.encoding=UTF-8" to tomcat8.conf file. (aslo tried with JAVA_OPTS).
I tried to add catalina options also with encoding.
HttpPost request = new HttpPost(URL);
// Create JSON and set the API Token
JSONObject SendJson = new JSONObject();
SendJson.put(field, key);
StringEntity params =new StringEntity(SendJson.toString());
request.setEntity(params);
// set Header Type
request.addHeader("content-type", "application/json;charset=UTF-8");
request.addHeader("Accept-Encoding", "UTF-8");
// Execute and wait for response
HttpResponse httpResponse = httpClient.execute(request);
HttpEntity resEntity = httpResponse.getEntity();
// cast InputStream to String for JSON conversion
BufferedReader reader = new BufferedReader(new InputStreamReader( resEntity.getContent()));
//String strInputStream = EntityUtils.toString(resEntity,"UTF-8");
String strInputStream = reader.readLine();
//debug
System.out.println("David Encoding problem");
System.out.println(Charset.defaultCharset().name());
System.out.println(strInputStream);
JSONObject answerObj = new JSONObject(strInputStream);
Charset.defaultCharset().name() - output UTF-8
I get JSON with no Hebrew text :/
part of the JSON for example:
{"item_group_id":1,"item_group_name":"קבוצה ×","picture_link":""},
Thanks,
David

I fixed it.
I changed the locale to he_IL and he_IL.UTF8, then restarted the server and it worked.
Thanks!

HTTP/1.1 400 Bad Request in java HTTP post

I am trying to post xml data to API using HTTP post method with credentials but a getting HTTP/1.1 400 Bad Request error .. Can anyone pl help me out ....
Here is my sample code:
BufferedReader br = new BufferedReader(new FileReader(new File("Data.xml")));
StringBuilder sb = new StringBuilder();
while((line=br.readLine())!= null){
sb.append(line.trim());
}
System.out.println("xml: "+sb);
params=sb.toString();
HttpPost request = new HttpPost("*****************url***************");
String urlaparam=URLEncoder.encode("importFormatCode:1&data:"+params,"UTF-8");
String userCredentials = "****:******";
byte[] auth = Base64.encodeBase64(userCredentials.getBytes());
StringEntity entity=new StringEntity(urlaparam);
request.addHeader("Content-type","application/x-www-form-urlencoded");
request.addHeader("Accept", "application/xml");
request.addHeader("Accept-Language", "en-US,en;q=0.5");
request.addHeader("Authorization", "Basic " + new String(auth));
request.setEntity(entity);
HttpResponse response = httpClient.execute(request);
System.out.println(response.getStatusLine());
System.out.println(request);
}
catch(Exception e)
{
}

First of all, your form parameters are not encoded correctly. You are using colon (:) to separate keys from their values, but instead, the equal sign (=) must be used:
Wrong: "importFormatCode:1&data:" + params
Correct: "importFormatCode=1&data=" + params
(See also W3C.org - Forms in HTML Documents - application/x-www-form-urlencoded)
Apart from that, you must not URL-encode the entire string but only the keys and the values. Otherwise you'll also encode the separator characters = and &!
The easiest way is to use the existing utility class org.apache.http.client.utils.URLEncodedUtils (assuming that you're using Apache HTTP Components):
String xmlData = // your xml data from somewhere
List<NameValuePair> params = Arrays.asList(
new BasicNameValuePair("importFormatCode", "1"),
new BasicNameValuePair("data", xmlData)
);
String body = URLEncodedUtils.format(params, encoding); // use encoding of request
StringEntity entity = new StringEntity(body);
// rest of your code

how to insert cypher statement in http post request

I have just started with neo4J and wanted to try the transactional cypher endpoint. I have my neo4J server running on localhost:7474/ and have inserted the movie data.
As stated in the documentation, I have to do a post request and include some parameters. Unfortunately I don't know how I have to include my query in the POST request. As far as I have understood it, I have to pass a JSON String.
private static String sendPost() throws Exception {
String url = "http://localhost:7474/db/data/transaction";
String statement ="[ { \"statement\" : \"MATCH (n:Person) RETURN n.name, n.born\"} ]";
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url);
List<NameValuePair> urlParameters = new ArrayList<NameValuePair>();
urlParameters.add(new BasicNameValuePair("Accept", "application/json; charset=UTF-8"));
urlParameters.add(new BasicNameValuePair("Content-Type", "application/json"));
urlParameters.add(new BasicNameValuePair("statements", statement));
post.setEntity(new UrlEncodedFormEntity(urlParameters));
HttpResponse response = client.execute(post);
StringBuilder builder = new StringBuilder();
builder.append("\nSending 'POST' request to URL : " + url+"<br>");
builder.append("Post parameters : " + post.getEntity()+"<br>");
builder.append("Response Code : " + response.getStatusLine().getStatusCode()+"<br>");
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
result.append("<p>");
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line+"\n");
}
result.append("</p>");
return builder.toString();
}
When I execute the code, I get the following output:
Sending 'POST' request to URL : http://localhost:7474/db/data/transaction
Post parameters : org.apache.http.client.entity.UrlEncodedFormEntity#76adb5f6
Response Code : 415
Can anyone help me on how I have to include my query in the POST request?

http://docs.neo4j.org/chunked/stable/rest-api-transactional.html
Looking at that, you can see the body of your POST request isn't what the server is expecting, i.e. you should be sending an entire JSON document, and not a k/v pair w/ "statements" as a key and your JSON Cypher query as the value. Remember you're sending JSON here, and not a URLEncoded body.
Also, it looks like you're setting the "Accept" and "Content-Type" k/v pairs as part of the POST body when they should, in fact, be part of the headers.
Also also, consider using the Cypher endpoint: http://docs.neo4j.org/chunked/stable/rest-api-cypher.html
HTH

Http POST in Java (with file upload)

What I want to do is submit a web form from a java application. The form I need to fill out is located here: http://cando-dna-origami.org/
When the form is submitted, the server sends a confirmation email to the email address given, which for now I'm just checking by hand. I've tried filling out the form manually, and the emails get sent fine. (It should also be noted that when the form is filled out incorrectly, the page just refreshes and doesn't give any feedback).
I've never done anything with http before, but I looked around for a while, and came up with the following code, which is supposed to send a POST request to the server:
String data = "name=M+V&affiliation=Company&email="
+ URLEncoder.encode("m.v#gmail.com", "UTF-8")
+ "&axialRise=0.34&helixDiameter=2.25&axialStiffness=1100&bendingStiffness=230" +
"&torsionalStiffness=460&nickStiffness=0.01&resolution=course&jsonUpload="
+ URLEncoder.encode("C:/Users/Marjie/Downloads/twisted_DNA_bundles/monotwist.L1.v1.json",
"UTF-8") + "&type=square";
URL page = new URL("http://cando-dna-origami.org/");
HttpURLConnection con = (HttpURLConnection) page.openConnection();
con.setDoOutput(true);
con.setRequestMethod("POST");
con.connect();
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream());
out.write(data);
out.flush();
System.out.println(con.getResponseCode());
System.out.println(con.getResponseMessage());
out.close();
con.disconnect();
However, when it runs it doesn't appear to do anything - that is, I don't get any emails, although the program does print "200 OK" to System.out, which seems to indicate that something got received from the server, although I'm not sure what it means exactly. I think the problem might be in the file uploading, since I wasn't sure whether that data type required a different format.
Is this a correct way to send a POST request using Java? Do I need to do something different for the file uploading? Thanks!
After reading Adam's post, I used Apache HttpClient and wrote the following code:
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("type", "square"));
//... add more parameters
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(params, HTTP.UTF_8);
HttpPost post = new HttpPost("http://cando-dna-origami.org/");
post.setEntity(entity);
HttpResponse response = new DefaultHttpClient().execute(post);
post = new HttpPost("http://cando-dna-origami.org/");
post.setEntity(new FileEntity(new File("C:/Users/Marjie/Downloads/twisted_DNA_bundles/monotwist.L1.v1.json"), "text/plain; charset=\"UTF-8\""));
HttpResponse responseTwo = new DefaultHttpClient().execute(post);
However, it still doesn't seem to be working; again, I wasn't sure how the uploaded file fit into the form, so I tried just sending two separate POST requests, one with the form and one with the other data. I am still looking for a way to combine these into one request; does anybody know something about this?

You would probably be better off using something like Apache HttpClient, with which you can build up a POST request programatically.
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://.../whatever");
List <NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("param1", "value1"));
params.add(new BasicNameValuePair("param2", "value2"));
...
httpost.setEntity(new UrlEncodedFormEntity(params, HTTP.UTF_8));
HttpResponse response = httpclient.execute(httppost);
If you need to upload a file along with your form, you will need to use a MultipartEntity instead:
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("someParam", "someValue");
reqEntity.addPart("someFile", new FileBody("/some/file"));
....
httpost.setEntity(reqEntity);
There are some sample programs over on their site. The "Form based logon" and "Multipart encoded request entity" are good examples to start from.
It may also be worthwhile testing out your connections and taking a look at the underlying network data to see what is happening. Something like Firebug will let you see exactly what is happening in your browser, and you can turn up the HttpClient logging to see all of the data exchanged in your program. Alternatively, you can use something like Wireshark or Fiddler to watch your network traffic in real-time. This may give you a better idea of exactly what your browser is doing, versus what your program is doing.

As most of the suggested Java HTTP POST request code out there is not operational, I decided to give you my fully operational code that I'm sure you'll find helpful to create any Java-based POST request in the future.
This POST request is of multipart type to allow sending/uploading a file to the server.
Multipart request consist of a main header and a separator string called boundary to tell each part from the other (this separator will come in the stream with "--" (two dashes) string before it, and each part has its own small header to tell its type and some more meta-data.
My task was to create a PDF file using some online services but all the multipart POST examples just didn't do the trick...
I needed to pack an HTML document along with its pics, JS and CSS files in a ZIP/TAR file, upload it to an online html2pdf conversion service and get the result as a PDF document back to me as the response (stream) from the service.
The current service I've checked the using following code is: Htmlpdfapi.com but I'm sure that with minor adjustments you'll be able to use it with any other service.
The method call (for that service) looks something like:
[class instance name].sendPOSTRequest("http://htmlpdfapi.com/api/v1/pdf", "Token 6hr4-AmqZDrFVjAcJGykjYyXfwG1wER4", "/home/user/project/srv/files/example.zip", "result.pdf");
Here is my code that was checked and 100% works:
public void sendPOSTRequest(String url, String authData, String attachmentFilePath, String outputFilePathName)
{
String charset = "UTF-8";
File binaryFile = new File(attachmentFilePath);
String boundary = "------------------------" + Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
int responseCode = 0;
try
{
//Set POST general headers along with the boundary string (the seperator string of each part)
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
connection.addRequestProperty("User-Agent", "CheckpaySrv/1.0.0");
connection.addRequestProperty("Accept", "*/*");
connection.addRequestProperty("Authentication", authData);
OutputStream output = connection.getOutputStream();
PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
// Send binary file - part
// Part header
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"file\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
writer.append("Content-Type: application/octet-stream").append(CRLF);// + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
writer.append(CRLF).flush();
// File data
Files.copy(binaryFile.toPath(), output);
output.flush();
// End of multipart/form-data.
writer.append(CRLF).append("--" + boundary + "--").flush();
responseCode = ((HttpURLConnection) connection).getResponseCode();
if(responseCode !=200) //We operate only on HTTP code 200
return;
InputStream Instream = ((HttpURLConnection) connection).getInputStream();
// Write PDF file
BufferedInputStream BISin = new BufferedInputStream(Instream);
FileOutputStream FOSfile = new FileOutputStream(outputFilePathName);
BufferedOutputStream out = new BufferedOutputStream(FOSfile);
int i;
while ((i = BISin.read()) != -1) {
out.write(i);
}
// Cleanup
out.flush();
out.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}

I am currently writing a small web server, and I tested your request client. My server is receiving the following request:
User-Agent: Java/1.6.0_20
Host: localhost:1700
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-type: application/x-www-form-urlencoded
Content-Length: 287
name=M+V&affiliation=Company&email=m.v%40gmail.com&axialRise=0.34&helixDiameter=2.25&axialStiffness=1100&bendingStiffness=230&torsionalStiffness=460&nickStiffness=0.01&resolution=course&jsonUpload=C%3A%2FUsers%2FMarjie%2FDownloads%2Ftwisted_DNA_bundles%2Fmonotwist.L1.v1.json&type=square
You should check the format of the POST data you are sending, most probably it is not processed by the server as you would expect.

You should definitively use apaches HTTPClient for that job! It makes life much easier. Here is an example how to upload a file with apaches HttpClient.
byte[] data = outStream.toByteArray()
HttpClient client = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://localhost:8080/YourResource");
ByteArrayBody byteArrayBody = new ByteArrayBody(data, "application/json", "some.json");
MultipartEntity multipartEntity = new MultipartEntity();
multipartEntity.addPart("upload", byteArrayBody);
httpPost.setEntity( multipartEntity );
HttpResponse response = client.execute(httpPost);
Reader reader = new InputStreamReader(response.getEntity().getContent());
Let me know if you have further questions.

Here's an example I got working that uses apache httpclient. Also, don't forget to add these dependencies:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.4.1</version>
</dependency>
The code:
HttpClient httpclient = HttpClientBuilder.create().build();
HttpPost httppost = new HttpPost(DataSources.TORRENT_UPLOAD_URL);
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.addPart("a_field_name", new FileBody(torrentFile));
HttpEntity entity = builder.build();
httppost.setEntity(entity);
HttpResponse response = httpclient.execute(httppost);

Unreadable Characters in Apache HttpClient

I'm trying to login to a webpage, but even before that, I'm loading the page using HttpGet, and this is one the lines that's being returned,
ÓA;
That's all I could put, won't let me paste any other characters. But they are all like that, like I'm somehow getting the wrong encoding? Here is the code I am using to GET
HttpGet httpget = new HttpGet(url);
if(headers == null) {
headers = getDefaultHeaders();
}
for(String s : headers.keySet()) {
httpget.addHeader(s, headers.get(s));
}
HttpResponse response = getClient().execute(httpget);
HttpEntity entity = response.getEntity();
System.out.println("Status Line: " + response.getStatusLine());
if (entity != null) {
InputStream input = entity.getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String ln = "";
while((ln = reader.readLine()) != null) {
System.out.println("During Get - " + ln);
}
}
What am I doing wrong?
Thanks for any help.
If you need any more information like headers, just ask.

The following line is possibly the cause of your problems:
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
You are creating a reader using the default characterset of your platform, and completely ignoring any character set that may be specified in the HTTP response headers.
If you are getting the same problem when reading the content the correct way, then it is possible that the server is at fault for not setting the response header correctly.

DO the entity reading like this:
String content = org.apache.http.util.EntityUtils.toString( entity );
System.out.println(content);
This is going to read it all for you so you can check what's being really returned.

Make sure that you didn't accidentally go to port 443 with a simple HTTP connection. Because in that case you will get back the SSL handshake instead of an HTTP response.