What I want to do is submit a web form from a java application. The form I need to fill out is located here: http://cando-dna-origami.org/
When the form is submitted, the server sends a confirmation email to the email address given, which for now I'm just checking by hand. I've tried filling out the form manually, and the emails get sent fine. (It should also be noted that when the form is filled out incorrectly, the page just refreshes and doesn't give any feedback).
I've never done anything with http before, but I looked around for a while, and came up with the following code, which is supposed to send a POST request to the server:
String data = "name=M+V&affiliation=Company&email="
+ URLEncoder.encode("m.v#gmail.com", "UTF-8")
+ "&axialRise=0.34&helixDiameter=2.25&axialStiffness=1100&bendingStiffness=230" +
"&torsionalStiffness=460&nickStiffness=0.01&resolution=course&jsonUpload="
+ URLEncoder.encode("C:/Users/Marjie/Downloads/twisted_DNA_bundles/monotwist.L1.v1.json",
"UTF-8") + "&type=square";
URL page = new URL("http://cando-dna-origami.org/");
HttpURLConnection con = (HttpURLConnection) page.openConnection();
con.setDoOutput(true);
con.setRequestMethod("POST");
con.connect();
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream());
out.write(data);
out.flush();
System.out.println(con.getResponseCode());
System.out.println(con.getResponseMessage());
out.close();
con.disconnect();
However, when it runs it doesn't appear to do anything - that is, I don't get any emails, although the program does print "200 OK" to System.out, which seems to indicate that something got received from the server, although I'm not sure what it means exactly. I think the problem might be in the file uploading, since I wasn't sure whether that data type required a different format.
Is this a correct way to send a POST request using Java? Do I need to do something different for the file uploading? Thanks!
After reading Adam's post, I used Apache HttpClient and wrote the following code:
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("type", "square"));
//... add more parameters
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(params, HTTP.UTF_8);
HttpPost post = new HttpPost("http://cando-dna-origami.org/");
post.setEntity(entity);
HttpResponse response = new DefaultHttpClient().execute(post);
post = new HttpPost("http://cando-dna-origami.org/");
post.setEntity(new FileEntity(new File("C:/Users/Marjie/Downloads/twisted_DNA_bundles/monotwist.L1.v1.json"), "text/plain; charset=\"UTF-8\""));
HttpResponse responseTwo = new DefaultHttpClient().execute(post);
However, it still doesn't seem to be working; again, I wasn't sure how the uploaded file fit into the form, so I tried just sending two separate POST requests, one with the form and one with the other data. I am still looking for a way to combine these into one request; does anybody know something about this?
You would probably be better off using something like Apache HttpClient, with which you can build up a POST request programatically.
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://.../whatever");
List <NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("param1", "value1"));
params.add(new BasicNameValuePair("param2", "value2"));
...
httpost.setEntity(new UrlEncodedFormEntity(params, HTTP.UTF_8));
HttpResponse response = httpclient.execute(httppost);
If you need to upload a file along with your form, you will need to use a MultipartEntity instead:
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("someParam", "someValue");
reqEntity.addPart("someFile", new FileBody("/some/file"));
....
httpost.setEntity(reqEntity);
There are some sample programs over on their site. The "Form based logon" and "Multipart encoded request entity" are good examples to start from.
It may also be worthwhile testing out your connections and taking a look at the underlying network data to see what is happening. Something like Firebug will let you see exactly what is happening in your browser, and you can turn up the HttpClient logging to see all of the data exchanged in your program. Alternatively, you can use something like Wireshark or Fiddler to watch your network traffic in real-time. This may give you a better idea of exactly what your browser is doing, versus what your program is doing.
As most of the suggested Java HTTP POST request code out there is not operational, I decided to give you my fully operational code that I'm sure you'll find helpful to create any Java-based POST request in the future.
This POST request is of multipart type to allow sending/uploading a file to the server.
Multipart request consist of a main header and a separator string called boundary to tell each part from the other (this separator will come in the stream with "--" (two dashes) string before it, and each part has its own small header to tell its type and some more meta-data.
My task was to create a PDF file using some online services but all the multipart POST examples just didn't do the trick...
I needed to pack an HTML document along with its pics, JS and CSS files in a ZIP/TAR file, upload it to an online html2pdf conversion service and get the result as a PDF document back to me as the response (stream) from the service.
The current service I've checked the using following code is: Htmlpdfapi.com but I'm sure that with minor adjustments you'll be able to use it with any other service.
The method call (for that service) looks something like:
[class instance name].sendPOSTRequest("http://htmlpdfapi.com/api/v1/pdf", "Token 6hr4-AmqZDrFVjAcJGykjYyXfwG1wER4", "/home/user/project/srv/files/example.zip", "result.pdf");
Here is my code that was checked and 100% works:
public void sendPOSTRequest(String url, String authData, String attachmentFilePath, String outputFilePathName)
{
String charset = "UTF-8";
File binaryFile = new File(attachmentFilePath);
String boundary = "------------------------" + Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
int responseCode = 0;
try
{
//Set POST general headers along with the boundary string (the seperator string of each part)
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
connection.addRequestProperty("User-Agent", "CheckpaySrv/1.0.0");
connection.addRequestProperty("Accept", "*/*");
connection.addRequestProperty("Authentication", authData);
OutputStream output = connection.getOutputStream();
PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
// Send binary file - part
// Part header
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"file\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
writer.append("Content-Type: application/octet-stream").append(CRLF);// + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
writer.append(CRLF).flush();
// File data
Files.copy(binaryFile.toPath(), output);
output.flush();
// End of multipart/form-data.
writer.append(CRLF).append("--" + boundary + "--").flush();
responseCode = ((HttpURLConnection) connection).getResponseCode();
if(responseCode !=200) //We operate only on HTTP code 200
return;
InputStream Instream = ((HttpURLConnection) connection).getInputStream();
// Write PDF file
BufferedInputStream BISin = new BufferedInputStream(Instream);
FileOutputStream FOSfile = new FileOutputStream(outputFilePathName);
BufferedOutputStream out = new BufferedOutputStream(FOSfile);
int i;
while ((i = BISin.read()) != -1) {
out.write(i);
}
// Cleanup
out.flush();
out.close();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I am currently writing a small web server, and I tested your request client. My server is receiving the following request:
User-Agent: Java/1.6.0_20
Host: localhost:1700
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-type: application/x-www-form-urlencoded
Content-Length: 287
name=M+V&affiliation=Company&email=m.v%40gmail.com&axialRise=0.34&helixDiameter=2.25&axialStiffness=1100&bendingStiffness=230&torsionalStiffness=460&nickStiffness=0.01&resolution=course&jsonUpload=C%3A%2FUsers%2FMarjie%2FDownloads%2Ftwisted_DNA_bundles%2Fmonotwist.L1.v1.json&type=square
You should check the format of the POST data you are sending, most probably it is not processed by the server as you would expect.
You should definitively use apaches HTTPClient for that job! It makes life much easier. Here is an example how to upload a file with apaches HttpClient.
byte[] data = outStream.toByteArray()
HttpClient client = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://localhost:8080/YourResource");
ByteArrayBody byteArrayBody = new ByteArrayBody(data, "application/json", "some.json");
MultipartEntity multipartEntity = new MultipartEntity();
multipartEntity.addPart("upload", byteArrayBody);
httpPost.setEntity( multipartEntity );
HttpResponse response = client.execute(httpPost);
Reader reader = new InputStreamReader(response.getEntity().getContent());
Let me know if you have further questions.
Here's an example I got working that uses apache httpclient. Also, don't forget to add these dependencies:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.4.1</version>
</dependency>
The code:
HttpClient httpclient = HttpClientBuilder.create().build();
HttpPost httppost = new HttpPost(DataSources.TORRENT_UPLOAD_URL);
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.addPart("a_field_name", new FileBody(torrentFile));
HttpEntity entity = builder.build();
httppost.setEntity(entity);
HttpResponse response = httpclient.execute(httppost);
Related
NOTICE UPDATE!!
The problem got solved and i added my own answer in the thread
In short, I have attempted to add the parameter "scan_id" value but since it is a POST i can't add the value directly in the url path.
using the code i already have, how would i go about modifying or adding so that the url is correct, that is, so that it accepts my POST?.
somehow i have been unable to find any examples that have helped me in figuring out how i would go about doing this..
I know how to do a POST with a payload, a GET with params. but a post with Params is very confusing to me.
Appreciate any help. (i'd like to continue using HttpUrlConnection unless an other example is provided that also tells me how to send the request and not only configuring the path.
I've tried adding it to the payload.
I've tried UriBuilder but found it confusing and in contrast with the rest of my code, so wanted to ask for help with HttpUrlConnection.
URL url = new URL("http://localhost/scans/{scan_id}/launch");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("tmp_value_dont_mind_this", "432432");
con.setRequestProperty("X-Cookie", "token=" + "43432");
con.setRequestProperty("X-ApiKeys", "accessKey="+"43234;" + " secretKey="+"43234;");
con.setDoInput(true);
con.setDoOutput(true); //NOT NEEDED FOR GETS
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
//First example of writing (works when writing a payload)
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
//second attemp at writing, doens't work (wanted to replace {scan_id} in the url)
DataOutputStream writer = new DataOutputStream(con.getOutputStream());
writer.writeChars("scan_id=42324"); //tried writing directly
//writer.write(payload);
writer.close();
Exception:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 400 for URL: http://localhost/scans/launch
I'd like one of the three response codes because then i know the Url is correct:
200 Returned if the scan was successfully launched.
403 Returned if the scan is disabled.
404 Returned if the scan does not exist.
I've tried several urls
localhost/scans/launch,
localhost/scans//launch,
localhost/scans/?/launch,
localhost/scans/{scan_id}/launch,
So with the help of a friend and everyone here i solved my problem.
The below code is all the code in an entire class explained bit by bit. at the bottom you have the full class with all its syntax etc, that takes parameters and returns a string.
in a HTTP request there are certain sections.
Such sections include in my case, Request headers, parameters in the Url and a Payload.
depending on the API certain variables required by the API need to go into their respective category.
My ORIGINAL URL looked like this: "http://host:port/scans/{scan_id}/export?{history_id}"
I CHANGED to: "https://host:port/scans/" + scan_Id + "/export?history_id=" + ID;
and the API i am calling required an argument in the payload called "format" with a value.
String payload = "{\"format\" : \"csv\"}";
So with my new URL i opened a connection and set the request headers i needed to set.
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
The setDoOutput should be commented out when making a GET request.
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Cookie", "token=" + token);
con.setRequestProperty("X-ApiKeys", "accessKey="+"23243;" +"secretKey="+"45543;");
Here i write to the payload.
//WRITING THE PAYLOAD to the http call
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
After i've written the payload i read whatever response i get back (this depends on the call, when i do a file download (GET Request) i don't have a response to read as i've already read the response through another piece of code).
I hope this helps anyone who might encounter this thread.
public String requestScan(int scan_Id, String token, String ID) throws MalformedInputException, ProtocolException, IOException {
try {
String endpoint = "https://host:port/scans/" + scan_Id + "/export?history_id=" ID;
URL url = new URL(endpoint);
String payload= "{\"format\" : \"csv\"}";
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Cookie", "token=" + token);
con.setRequestProperty("X-ApiKeys", "accessKey="+"324324;" +
"secretKey="+"43242;");
//WRITING THE PAYLOAD to the http call
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
//READING RESPONSE
BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuffer jsonString = new StringBuffer();
String line;
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
con.disconnect();
return jsonString.toString();
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
}
As discussed here the solution would be to change the content type to application/x-www-form-urlencoded, but since you are already using application/json; charset=UTF-8 (which I am assuming is a requirement of your project) you have no choise to redesign the whole thing. I suggest you one of the following:
Add another GET service;
Add another POST service with content type application/x-www-form-urlencoded;
Replace this service with one of the above.
Do not specify the content type at all so the client will accept anything. (Don't know if possible in java)
If there are another solutions I'm not aware of, I don't know how much they would be compliant to HTTP protocol.
(More info)
Hope I helped!
Why you are not using like this. Since you need to do a POST with HttpURLConnection, you need to write the parameters to the connection after you have opened the connection.
String urlParameters = "scan_id=42324";
byte[] postData = urlParameters.getBytes(StandardCharsets.UTF_8);
DataOutputStream dataOutputStream = new DataOutputStream(conn.getOutputStream());
dataOutputStream.write(postData);
Or if you have launch in the end, just change the above code to the following,
String urlParameters = "42324/launch";
byte[] postData = urlParameters.getBytes(StandardCharsets.UTF_8);
DataOutputStream dataOutputStream = new DataOutputStream(conn.getOutputStream());
dataOutputStream.write(postData);
URL url = new URL("http://localhost/scans/{scan_id}/launch");
That line looks odd to me; it seems you are trying to use a URL where you are intending the behavior of a URI Template.
The exact syntax will depend on which template implementation you choose; an implementation using the Spring libraries might look like:
import org.springframework.web.util.UriTemplate;
import java.net.url;
// Warning - UNTESTED code ahead
UriTemplate template = new UriTemplate("http://localhost/scans/{scan_id}/launch");
Map<String,String> uriVariables = Collections.singletonMap("scan_id", "42324");
URI uri = template.expand(uriVariables);
URL url = uri.toURL();
I have written a REST client for this endpoint:
textmap.com/ethnicity_api/api
However, while passing it a name string like jennífer garcía in the POST params, and setting encoding to UTF-8, the response that I get is not the same string. How to get the same name in the response object?
Below is how I set the request and the response thatI get:
httpClient = HttpClients.createDefault();
httpPost = new HttpPost(baseurl);
StringEntity input = new StringEntity(inputJSON, StandardCharsets.UTF_8);
input.setContentType("application/json");
//input.setContentType("application/json; charset=UTF-8");
httpPost.setEntity(input);
response = httpClient.execute(httpPost);
if (response.getStatusLine().getStatusCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : " + response.getStatusLine().getStatusCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader((response.getEntity().getContent())));
output = org.apache.commons.io.IOUtils.toString(br);
System.out.println(output);
Value of the name in output is : jenn�fer garc�a
This is a completely different charset from what I had sent in the request. How can I get the same charset as I had sent in request?
Secondly, I want the same code to work in both Java-6 and Java-7. The above code is using Java-7 only. How can I make the code work for both these versions?
I think the BufferedReader is breaking the UTF8 encoding, so this is actually pretty unrelated to HTTP. On a side note, the br may not be needed at all.
I'm trying to get my Android multipart image upload to work with the default blueimp PHP UploadHandler, which I'm using for the web version of my app.
It didn't work out of the box, so I experimented with the header of the requests.
I played around with it for quite a while, but since nothing worked I ended up firing up Wireshark and comparing the packets of the blueimp demo and the ones that are sent by the Android app.After some further tweaking, they look exactly the same (well the important parts):
Blueimp demo: http://i.imgur.com/T9styyR.png
Android MultipartEntity: http://i.imgur.com/OuDuyJ0.png
This is what my customized generate_response function returns:
Array
(
[0] => stdClass Object
(
[name] => 1387215600-9207
[size] => 97894
[type] => multipart/form-data; boundary=-----Don'tmindmeI'mjustaboundary
[error] => Filetype not allowed
)
)
It seems to load the whole packet instead of just it's multipart part.
Is there any way to fix that?
Here's how I create the request (not quite but these are the relevant lines):
public static String BOUNDARY = "-----Don'tmindmeI'mjustaboundary";
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE, BOUNDARY, null);
File file = new File(params[2]); //params[2] contains the absolute path + filename
String filename = file.getName();
entity.addPart(filename, new FileBody(file, getMimeType(params[2]))); //getMimeType returns the MIME Type of the image e.g. image/png
HttpPost request = new HttpPost(params[0]); //params[0] contains the URL
request.setHeader("Content-Type", "multipart/form-data; boundary="+BOUNDARY);
request.setEntity(entity);
HttpResponse response = client.execute(request, context);
I've been working on this for hours and it's really driving me nuts.
What am I doing wrong?
Edit:
Nevermind I got it to work
I forgot to set the name of the form to files[], which is what the UploadHandler is looking for ...
So now it looks like this:
...
public static String BOUNDARY = "-----Don'tmindmeI'mjustaboundary";
...
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE, BOUNDARY, null);
File file = new File(params[2]);
String filename = file.getName();
FileBody body = new FileBody(file, getMimeType(params[2]));
FormBodyPart part = new FormBodyPart("files[]", body);
part.addField("Content-Type", getMimeType(params[2]));
part.addField("Content-Disposition", "form-data; filename=\""+filename+"\"");
entity.addPart(part);
HttpPost request = new HttpPost(params[0]);
request.setHeader("Content-Type", "multipart/form-data; boundary="+BOUNDARY);
request.addHeader("Accept", "application/json, text/javascript, */*; q=0.01");
request.addHeader("Accept-Language", "en-US,en;q=0.8,de;q=0.6,es;q=0.4");
request.setEntity(entity);
HttpResponse response = client.execute(request, context);
...
I am writing my own implementation of the Google Drive client for the Android and I am using the docs list api. Recently I've encountered a following problem:
At first I was using the HttpURLConnection to upload the file but it seems like it writes the data to the socket after a call to getResponseCose(), not when I am writing to the connection's OutputStream, which is a must for me.
Then I've switched to the Apache HttpClient but I'm still getting a 400 response, not sure why. Maybe you will be able to help me. Here is the code used to upload a file.
String putUrl = conn.getHeaderField("Location");//from the previous request
final HttpClient client = new DefaultHttpClient();
final HttpPut put = new HttpPut(putUrl);
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
put.addHeader("Content-Type", mime==null?"file":mime);
//put.addHeader("Content-Length", String.valueOf(length));
put.addHeader("Content-Range", "bytes 0-"+(length-1)+"/"+length);
put.addHeader("GData-Version", "3.0");
put.addHeader("Authorization", getAuthorizationProperty());
entity.addPart("content", new InputStreamBody(in, name));
put.setEntity(entity);
HttpResponse resp = client.execute(put);
int response = resp.getStatusLine().getStatusCode();
if(response == HttpStatus.SC_CREATED){
lastCreated = parseSingleXMLEntry(resp.getEntity().getContent());
}
Exactly the same headers worked for HttpURLConnection. Maybe the entity is wrong?
Ok, the solution is quite simple, hope it will be useful for someone.
I had to delete all lines which added headers to the request. After that I've added the mime type to the InputStreamBody constructor and overriden the getContentLength() method to provide stream length. Finally it looks like this:
MultipartEntity entity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
entity.addPart("content", new InputStreamBody(in, ,mime, name){
#Override
public long getContentLength() {
return length;
}
});
put.setEntity(entity);
HttpResponse resp = client.execute(put);
And that's all.
I'm trying out the twitter streaming api. I could succesfully filter tweets by using curl, as stated here:
curl -d #tracking http://stream.twitter.com/1/statuses/filter.json -u <user>:<pass>
where tracking is a plain file with the content:
track=Berlin
Now I tried to do the same thing in JavaSE, using Apache's HTTPComponents:
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(<user>, <pass>);
DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpPost httpPost = new HttpPost("http://stream.twitter.com/1/statuses/filter.json");
HttpParams params = new BasicHttpParams();
params = params.setParameter("track", "Berlin");
httpPost.setParams(params);
try {
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity entity = httpResponse.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
String t;
BufferedReader br = new BufferedReader(new InputStreamReader(instream));
while(true) {
t = br.readLine();
if(t != null) {
linkedQueue.offer(t);
}
}
}
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
}
finally{
httpClient.getConnectionManager().shutdown();
}
When I run that, I get:
No filter parameters found. Expect at least one parameter: follow track
as a single entry in my linkedQueue. Seems the api wants the parameter in a different form, but cannot find any hint in the documentation. Can somebody share some experiences with the api or see any other problem with the code? Thanks!
EDIT
Putting the filter parameter into the params was a bad idea. As it's post data, it needs to be defined as an Entity before the request is being made:
StringEntity postEntity = new StringEntity("track=Berlin", "UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
That's what I was doing wrong. Thanks Brian!
I suspect you need to post the data as the contents of your HTTP post. The man page for curl -d says:
(HTTP) Sends the specified data in a
POST request to the HTTP server, in
the same way that a browser does when
a user has filled in an HTML form and
presses the submit button. This will
cause curl to pass the data to the
server using the content-type
application/x-www-form-urlencoded.
so I believe you have to set that content type and put the contents of the tracking file in the body of your post.