I'm trying to submit a form to a certain website from Java (over HTTP), but when reading the response I don't see what I expected.
What exactly I do: first of all, I open the website in a browser, fill in the form by hand and submit it. In Chrome I can see the data that goes over the wire, i.e.:
Request URL:http://wizzair.com/en-GB/Select
Request Method:POST
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:1061
Content-Type:application/x-www-form-urlencoded
Cookie:WRUID=0; ASP.NET_SessionId=3e3ahach1d34oyhtoqfshxhe; Culture=en-GB; __utma=17431487.361991764.1292186668.1354138010.1354651562.81; __utmb=17431487.9.9.1354652614319; __utmc=17431487; __utmz=17431487.1319145359.34.18.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=wizz
Host:wizzair.com
Origin:http://wizzair.com
Referer:http://wizzair.com/en-GB/Select
User-Agent:Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11
Form Dataview URL encoded
__EVENTTARGET:HeaderControlGroupRibbonSelectView_AvailabilitySearchInputRibbonSelectView_ButtonSubmit
__VIEWSTATE:/wEPDwUBMGRkNSMYF94e4mXCiiJGEJbRixyidoa2QXSambTT2mm6cLs=
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$OriginStation:EIN
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DestinationStation:OTP
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DepartureDate:02/02/2013
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ReturnDate:05/02/2013
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountADT:1
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountCHD:0
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountINFANT:0
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$BaggageCount:0
HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ButtonSubmit:Search
So I try to simulate the same request from a Java program, i.e.:
public void doSubmit(String url, Map<String, String> data) throws Exception {
URL siteUrl = new URL(url);
HttpURLConnection conn = (HttpURLConnection) siteUrl.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestProperty("Cookie", "WRUID=0; ASP.NET_SessionId=3e3ahach1d34oyhtoqfshxhe; Culture=en-GB; __utma=17431487.361991764.1292186668.1354138010.1354651562.81; __utmb=17431487.9.9.1354652614319; __utmc=17431487; __utmz=17431487.1319145359.34.18.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=wizz");
DataOutputStream out = new DataOutputStream(conn.getOutputStream());
Set keys = data.keySet();
Iterator keyIter = keys.iterator();
String content = "";
for(int i=0; keyIter.hasNext(); i++) {
Object key = keyIter.next();
if(i!=0) {
content += "&";
}
content += key + "=" + URLEncoder.encode(data.get(key), "UTF-8");
}
System.out.println(content);
out.writeBytes(content);
out.flush();
out.close();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line = "";
while((line=in.readLine())!=null) {
System.out.println(line);
}
in.close();
}
.... which I call with the following parameters, as seen in the HTTP form data above:
String url = "http://wizzair.com/en-GB/Select";
Map<String, String> data = new TreeMap<String, String>();
data.put("__EVENTTARGET", "HeaderControlGroupRibbonSelectView_AvailabilitySearchInputRibbonSelectView_ButtonSubmit");
data.put("__VIEWSTATE", "/wEPDwUBMGRkNSMYF94e4mXCiiJGEJbRixyidoa2QXSambTT2mm6cLs=\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$OriginStation:EIN\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DestinationStation:OTP\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$DepartureDate:02/02/2013\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ReturnDate:05/02/2013\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountADT:1\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountCHD:0\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$PaxCountINFANT:0\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$BaggageCount:0\n"+
"HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$ButtonSubmit:Search"
However, the response I get back is simply a generic web page from that web site, not the answer I expected. What am I doing wrong ?
Thanks a lot,
Greetings,
Sorin
I think the way you populate the post data is not correct. You should have ten or so key/value pairs, and not only two. The second item you see in Chrome is not one big string. Things like 'HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$OriginStation' is a key in its own right. This also applies to the other very very ugly named keys who all start with 'HeaderControlGroupRibbonSelectView$AvailabilitySearchInputRibbonSelectView$'.
Further, you need to encode the 'key' of the each post data item as well and not only the value of the key (because of the $-characters inside). Encode both sides with separate calls, to avoid encoding the assignment '='.
You also need to remove the newline at the end of your original second, but now split key/value, because it is just not there. This also applies to the third, fourth, etc. item.
Be carefull with interpreting what you see in Chrome :-P
Another thing: the website you are referring to is strongly session based: it keeps track of the current state of your interaction using the cookie value 'ASP.NET_SessionId'. This value is only short lifed. In general you should first make a call to this website without this cookie value, and the website will provide it for you (redirecting you to a country specific localized catch-all page). Subsequently you can use its value in your (second) request to harvest data. If you supply an invalid session id, you will be redirected to the same default page again, again and again.
Related
NOTICE UPDATE!!
The problem got solved and i added my own answer in the thread
In short, I have attempted to add the parameter "scan_id" value but since it is a POST i can't add the value directly in the url path.
using the code i already have, how would i go about modifying or adding so that the url is correct, that is, so that it accepts my POST?.
somehow i have been unable to find any examples that have helped me in figuring out how i would go about doing this..
I know how to do a POST with a payload, a GET with params. but a post with Params is very confusing to me.
Appreciate any help. (i'd like to continue using HttpUrlConnection unless an other example is provided that also tells me how to send the request and not only configuring the path.
I've tried adding it to the payload.
I've tried UriBuilder but found it confusing and in contrast with the rest of my code, so wanted to ask for help with HttpUrlConnection.
URL url = new URL("http://localhost/scans/{scan_id}/launch");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("tmp_value_dont_mind_this", "432432");
con.setRequestProperty("X-Cookie", "token=" + "43432");
con.setRequestProperty("X-ApiKeys", "accessKey="+"43234;" + " secretKey="+"43234;");
con.setDoInput(true);
con.setDoOutput(true); //NOT NEEDED FOR GETS
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
//First example of writing (works when writing a payload)
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
//second attemp at writing, doens't work (wanted to replace {scan_id} in the url)
DataOutputStream writer = new DataOutputStream(con.getOutputStream());
writer.writeChars("scan_id=42324"); //tried writing directly
//writer.write(payload);
writer.close();
Exception:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 400 for URL: http://localhost/scans/launch
I'd like one of the three response codes because then i know the Url is correct:
200 Returned if the scan was successfully launched.
403 Returned if the scan is disabled.
404 Returned if the scan does not exist.
I've tried several urls
localhost/scans/launch,
localhost/scans//launch,
localhost/scans/?/launch,
localhost/scans/{scan_id}/launch,
So with the help of a friend and everyone here i solved my problem.
The below code is all the code in an entire class explained bit by bit. at the bottom you have the full class with all its syntax etc, that takes parameters and returns a string.
in a HTTP request there are certain sections.
Such sections include in my case, Request headers, parameters in the Url and a Payload.
depending on the API certain variables required by the API need to go into their respective category.
My ORIGINAL URL looked like this: "http://host:port/scans/{scan_id}/export?{history_id}"
I CHANGED to: "https://host:port/scans/" + scan_Id + "/export?history_id=" + ID;
and the API i am calling required an argument in the payload called "format" with a value.
String payload = "{\"format\" : \"csv\"}";
So with my new URL i opened a connection and set the request headers i needed to set.
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
The setDoOutput should be commented out when making a GET request.
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Cookie", "token=" + token);
con.setRequestProperty("X-ApiKeys", "accessKey="+"23243;" +"secretKey="+"45543;");
Here i write to the payload.
//WRITING THE PAYLOAD to the http call
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
After i've written the payload i read whatever response i get back (this depends on the call, when i do a file download (GET Request) i don't have a response to read as i've already read the response through another piece of code).
I hope this helps anyone who might encounter this thread.
public String requestScan(int scan_Id, String token, String ID) throws MalformedInputException, ProtocolException, IOException {
try {
String endpoint = "https://host:port/scans/" + scan_Id + "/export?history_id=" ID;
URL url = new URL(endpoint);
String payload= "{\"format\" : \"csv\"}";
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("POST");
con.setRequestProperty("Accept", "application/json");
con.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
con.setRequestProperty("X-Cookie", "token=" + token);
con.setRequestProperty("X-ApiKeys", "accessKey="+"324324;" +
"secretKey="+"43242;");
//WRITING THE PAYLOAD to the http call
OutputStreamWriter writer = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
//READING RESPONSE
BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuffer jsonString = new StringBuffer();
String line;
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
con.disconnect();
return jsonString.toString();
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
}
As discussed here the solution would be to change the content type to application/x-www-form-urlencoded, but since you are already using application/json; charset=UTF-8 (which I am assuming is a requirement of your project) you have no choise to redesign the whole thing. I suggest you one of the following:
Add another GET service;
Add another POST service with content type application/x-www-form-urlencoded;
Replace this service with one of the above.
Do not specify the content type at all so the client will accept anything. (Don't know if possible in java)
If there are another solutions I'm not aware of, I don't know how much they would be compliant to HTTP protocol.
(More info)
Hope I helped!
Why you are not using like this. Since you need to do a POST with HttpURLConnection, you need to write the parameters to the connection after you have opened the connection.
String urlParameters = "scan_id=42324";
byte[] postData = urlParameters.getBytes(StandardCharsets.UTF_8);
DataOutputStream dataOutputStream = new DataOutputStream(conn.getOutputStream());
dataOutputStream.write(postData);
Or if you have launch in the end, just change the above code to the following,
String urlParameters = "42324/launch";
byte[] postData = urlParameters.getBytes(StandardCharsets.UTF_8);
DataOutputStream dataOutputStream = new DataOutputStream(conn.getOutputStream());
dataOutputStream.write(postData);
URL url = new URL("http://localhost/scans/{scan_id}/launch");
That line looks odd to me; it seems you are trying to use a URL where you are intending the behavior of a URI Template.
The exact syntax will depend on which template implementation you choose; an implementation using the Spring libraries might look like:
import org.springframework.web.util.UriTemplate;
import java.net.url;
// Warning - UNTESTED code ahead
UriTemplate template = new UriTemplate("http://localhost/scans/{scan_id}/launch");
Map<String,String> uriVariables = Collections.singletonMap("scan_id", "42324");
URI uri = template.expand(uriVariables);
URL url = uri.toURL();
I'm trying to do a "POST" method in Java. I create my output with the OrientDB method like this:
"http://xxxxxxxxxxx:2480/command/mydb/sql/CREATE VERTEX V SET name = ' datoAletarorio'"
I need to use the write and flush methods to send the command.
My DB is empty with this method.
Where is my error? Here is my code:
//...
PrintWriter out = null;
//...
conexion = (HttpURLConnection) url.openConnection();
conexion.setDoOutput(true);
conexion.setRequestMethod("POST");
out = new PrintWriter(conexion.getOutputStream());
conexion.connect();
//...
String cumuloDatos1 = "http://xxxxxxxxxxx:2480/command/mydb/sql/CREATE VERTEX V SET name = ' datoAletarorio'"
out.write(cumuloDatos1);
out.flush();
//..
conexion.disconnect();
Thank in advance.
The docs says:
The command-text can appear in either the URL or the content of the
POST transmission. Where the command-text is included in the URL, it
must be encoded as per normal URL encoding.
So you probably have to encode the URL before sending the request:
String cumuloDatos1 =
"http://xxxxxxxxxxx:2480/command/mydb/sql/" +
"CREATE%20VERTEX%20V%20SET%20name%20%3D%20%27%20datoAletarorio%27"
Anyway, you should see messages in the logs for a 400 or similiar in the server, if the request isn't valid.
I'm trying to login web site using Java and I succeeded. Below is the code I used.
String query = "myquery";
URL url = new URL(loginUrl);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-length", String.valueOf(query.length()));
con.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
con.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0;Windows98;DigExt)");
con.setDoOutput(true);
con.setDoInput(true);
DataOutputStream output = new DataOutputStream(con.getOutputStream());
output.writeBytes(query);
output.close();
DataInputStream input = new DataInputStream( con.getInputStream() );
for( int c = input.read(); c != -1; c = input.read() ) {
System.out.print( (char)c );
// this page returns JavaScript code
}
After this, I want to access another web page in same domain, so I tried below code.
URL url = new URL(anotherUrl);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
... similar to above code ...
But this page asks me to login again. I think connection has been disconnected in the process of changing URL. (Onlt login page uses HTTPS protocol and other pages use HTTP protocol)
How can I fix this?
Someone please help
Keep in mind that HTTP is completely stateless. The idea of "logging in" to a site translates to (usually) setting cookies from an HTTP perspective. Those cookies are simply HTTP headers and they are sent with each subsequent request by your browser. So for you to maintain the logged in state its up to you get the cookies from the response headers and send them along with future requests.
Here is how:
Retrieving cookies from a response:
Open a java.net.URLConnection to the server:
URL myUrl = new URL("http://www.hccp.org/cookieTest.jsp");
URLConnection urlConn = myUrl.openConnection();
urlConn.connect();
Loop through response headers looking for cookies:
Since a server may set multiple cookies in a single request, we will need to loop through the response headers, looking for all headers named "Set-Cookie".
String headerName=null;
for (int i=1; (headerName = uc.getHeaderFieldKey(i))!=null; i++) {
if (headerName.equals("Set-Cookie")) {
String cookie = urlConn.getHeaderField(i);
...
Extract cookie name and value from cookie string:
The string returned by the getHeaderField(int index) method is a series of name=value separated by semi-colons (;). The first name/value pairing is actual data string we are interested in (i.e. "sessionId=0949eeee22222rtg" or "userId=igbrown"), the subsequent name/value pairings are meta-information that we would use to manage the storage of the cookie (when it expires, etc.).
cookie = cookie.substring(0, cookie.indexOf(";"));
String cookieName = cookie.substring(0, cookie.indexOf("="));
String cookieValue = cookie.substring(cookie.indexOf("=") + 1, cookie.length());
This is basically it. We now have the cookie name (cookieName) and the cookie value (cookieValue).
Setting a cookie value in a request:
Values must be set prior to calling the connect method:
URL myUrl = new URL("http://www.hccp.org/cookieTest.jsp");
URLConnection urlConn = myUrl.openConnection();
Create a cookie string:
String myCookie = "userId=igbrown";
Add the cookie to a request:
Using the
setRequestProperty(String name, String value);
method, we will add a property named "Cookie", passing the cookie string created in the previous step as the property value.
urlConn.setRequestProperty("Cookie", myCookie);
Send the cookie to the server:
To send the cookie, simply call connect() on the URLConnection for which we have added the cookie property:
urlConn.connect()
I am trying to transfer form data from an Android application to a NodeJs server.
My client code is the following (the strings that can contain UTF-8 characters are the values of params):
final HttpPost post = new HttpPost(url);
final MultipartEntityBuilder mpb = MultipartEntityBuilder.create()
.setCharset(Charset.forName("UTF-8")) // tried with or without this line
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE); // tried with or without this line
for (final Entry<String, String> e : params.entrySet()) {
mpb.addTextBody(e.getKey(), e.getValue());
}
post.setEntity(mpb.build());
final HttpClient httpClient = new DefaultHttpClient();
final HttpResponse response = httpClient.execute(request);
And my server code is the following:
app.post('/accesspoint', function(req, res) {
var body = req.body;
var form = new formidable.IncomingForm();
form.encoding = 'utf-8';
form.parse(req, function(err, fields, files) {
console.log(fields);
...
When my input java params has a value containing an UTF-8 character, the log I get server side prints the corresponding value without this character, so it is kind of swallowed at some point. For instance if my input string is "ê", then my server log will print a "" value.
I use a multipart form as I read that it was the best way to send data that can contain non-ASCII characters. Formidable is also apparently the best node package to handle form that can contain UTF-8 characters.
My client side uses Apache HttpClient 4.3.3.
What am I doing wrong?
Ok so I tested with a simple query and the key value ("foo","[ê]") looked at the headers and I saw that my query was still using ISO-8859-1
Content-Disposition: form-data; name="foo"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
[]
in contradiction to my builder parameter.
I found the solution via https://stackoverflow.com/a/21020435/592392 and changed my code to:
for (final Entry<String, String> e : params.entrySet()) {
mpb.addTextBody(e.getKey(), e.getValue(), ContentType.create("text/plain", Charset.forName("UTF-8")));
}
And now the server gets the UTF8 chars :)
Anyway, the form builder is quite misleading in my opinion.
i am using
sendRedirect("http://api.mVaayoo.com/mvaayooapi/MessageCompose?user=someuser#gmail.com:123456&senderID=TEST SMS&receipientno=0987654321&dcs=0&msgtxt="+ message + "&state=4")
to call mvaayoo api for sendind sms .But the parameters are displayed in address bar to the client .
Is there a way to hide query string? I dont want to purchase SSL certificate.
The problem is not how you redirect, rather the problem is in the provider of the redirect URL you are trying to use:
http://api.mVaayoo.com/mvaayooapi/MessageCompose
No sensitive information should be used as a GET/query param.
Is there a way to hide query string?
Pass additional data as redirect attributes instead of passing it as query parameters.
To carry data across a redirect use RedirectAttributes#addFlashAttribute(key, value).
What Java doc says:
A RedirectAttributes model is empty when the method is called and is never used unless the method returns a redirect view name or a RedirectView.
After the redirect, flash attributes are automatically added to the model of the controller that serves the target URL.
Read more...
Take a look
List<String> pathParam = null;
if(null!=request.getPathInfo() && !request.getPathInfo().trim().isEmpty()){
String paths[] = this.getPathInfo().replaceAll("\\/$|^\\/", "").split("/");
pathParam = new ArrayList<>(Arrays.asList(paths));
}
// assume your servlet url is "/login/*"
// called url in browser /login/param1/param2/param3
String param1 = pathParam.get(0);
String param2 = pathParam.get(1);
String param3= pathParam.get(2);
You need to use URLConnection to achieve this:
HTTP GET:
URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
Target URL's doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().
HTTP POST:
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
try (OutputStream output = connection.getOutputStream()) {
output.write(query.getBytes(charset));
}
InputStream response = connection.getInputStream();
Target URL's doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().