I want to execute some post but this post required PHPSESSID and one parameter.
If i take this parameters from browser and execute in my code all work correct.
Unfortunately when i use my code to get page content and cookie for "PHPSESSID" and parameter and use this to execute post, it doesn't work.
Example
PHPSESSID and param value we can obtain when we go in browser to 'http://www.kreskowkazone.pl/odcinki-online_rick-i-morty-2013_7'
PHPSESSID we can get from cookie and value for param "o" is a long number in icon/link play for "openload.io" (code look like this 181774:bb19d7426f8eda85ba82265a01eda9c7 but it change).
Whan we use this param in the code all works fine we see iframe html
Code to execute post
public void sendPostTest(String param, String phpSESSID) throws IOException {
URL url = new URL("http://www.kreskowkazone.pl/odcinki_emb");
Map<String, Object> params = new LinkedHashMap<>();
params.put("o", param);
StringBuilder postData = new StringBuilder();
for (Map.Entry<String, Object> p : params.entrySet()) {
if (postData.length() != 0) postData.append('&');
postData.append(URLEncoder.encode(p.getKey(), "UTF-8"));
postData.append('=');
postData.append(URLEncoder.encode(String.valueOf(p.getValue()), "UTF-8"));
}
byte[] postDataBytes = postData.toString().getBytes("UTF-8");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Host", "www.kreskowkazone.pl");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0");
conn.setRequestProperty("Accept", "text/html, */*; q=0.01");
conn.setRequestProperty("Accept-Language", "pl,en-US;q=0.7,en;q=0.3");
conn.setRequestProperty("Accept-Encoding", "gzip, deflate");
conn.setRequestProperty("Referer", "http://www.kreskowkazone.pl/odcinki-online_rick-i-morty-2013_24");
conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
conn.setRequestProperty("X-Requested-With", "XMLHttpRequest");
conn.setRequestProperty("Content-Length", String.valueOf(postDataBytes.length));
conn.setRequestProperty("Cookie", phpSESSID);
conn.setRequestProperty("DNT", "1");
conn.setRequestProperty("Connection", "keep-alive");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.getOutputStream().write(postDataBytes);
Reader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
for (int c; (c = in.read()) >= 0; ) {
System.out.print((char) c);
}
}
Unfortunately when i take this param by my code i don't get any error and i don't see iframe.
To get page content and PHPSESSID I use this code.
public class CustomCookieManager {
public static void main(String[] args) {
try {
getPageContent("http://www.kreskowkazone.pl/odcinki-online_rick-i-morty-2013_7");
} catch (IOException e) {
e.printStackTrace();
}
}
public static void getPageContent(String pageUrl) throws IOException {
CookieManager ckman = new java.net.CookieManager();
ckman.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(ckman);
URL url = new URL(pageUrl);
URLConnection connection = url.openConnection();
connection.getContent();
String html = getHtml(connection);
CookieStore ckStore = ckman.getCookieStore();
List<HttpCookie> cks = ckStore.getCookies();
for (HttpCookie ck : cks) {
System.out.println("PHPSESSID=" + ck.getValue());
System.out.println(html);
}
}
public static String getHtml(URLConnection conn) {
StringBuilder sb = new StringBuilder();
try {
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = br.readLine()) != null) {
sb.append(inputLine);
}
br.close();
} catch (IOException e) {
e.printStackTrace();
}
return sb.toString();
}
}
I know PHPSESSID is related with param value because when i take PHPSESSID from one browser and parameter value from other browser
post doesn't work (no error no iframe).
My question is, is it posible that my CookieManager return cookie and PHPSESSID not for page which i'm reading in CustomCookieManager?
I tried the same example from HttpUnit with no result.
Try only using the same PHPSESSID as the one in your browser for every request, i was supposed to comment this but i wasn't allowed to.
Dr. Wells
I could but it will be one-off solution because PHPSESSID or number which is sending as parameter expired after some time.
After about half hour PHPSESSID and parameter are invalid (i don't see iframe html).
I can't use PHPSESSID from browser and parameter value returned by my java code because it's doesn't work.
It looks like generating parameter value depending on PHPSESSID.
I start thinking that my CookieManager is not releated with page which content i take in CustomCookieManager.
Is it possible that CookieManager joust return some PHPSESSID which is not related with page i downloaded.
I start thinking that page have some javascript which change something so i use HttpUnit but with no result.
Maybe it's impossible maybe it's some kind of security for the page but this CookieManager bites me.
Related
I'm trying to get an access token from https://www.reddit.com/api/v1/access_token. To do so I need to send a CLIENT_ID and a CLIENT_SECRET to the above URL. I did so using Postman:
As highlighted on the screenshot, I've sent a grant_type as a GET parameter with value client_credentials and an Authorization parameter with value Basic heregoestheencodedkeyandid. The reuest type was set as POST. It worked correctly - I got an access token in a JSON response.
However, when I try to do the same thing by means of Java, I receieve a Server returned HTTP response code: 411 error:
public class RedditExample {
private static String loginLink = "https://www.reddit.com/api/v1/access_token";
public static void main(String[] args) {
RedditExample redditExample = new RedditExample ();
redditExample.login();
}
public boolean login() {
try {
URL loginURL = new URL(loginLink + "?grant_type=client_credentials");
HttpURLConnection connection = (HttpURLConnection) loginURL.openConnection();
setupPOSTConnection(connection);
InputStream input = connection.getInputStream();
String inputString = new Scanner(input, "UTF-8").useDelimiter("\\Z").next();
System.out.println(inputString);
}
catch (Exception e) {
System.out.println(e.toString());
}
return true;
}
private static void setupPOSTConnection(HttpURLConnection connection) throws Exception {
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Authorization", "Basic heregoestheencodedkeyandid");
connection.connect();
}
}
I'm not sure what I'm doing different here, compared to Postman, so any help would be appreciated.
EDIT: Here is what I tried adding:
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestProperty("Content-Length", "0");
connection.setRequestProperty("Content-Length", "10");
String userAgent = "test /u/someuser";
connection.setRequestProperty("User-Agent", userAgent);
Unfortunately, neither worked - the error is still the same.
Setting content-length explicitly is not taken by HttpUrlConnection. So just provide request body with no content.
StringBuilder postDataBuilder = new StringBuilder();
byte[] postData = postDataBuilder.toString().getBytes("UTF-8");
OutputStream out = conn.getOutputStream();
out.write(postData);
out.close();
So the method will be like this:-
private static void setupPOSTConnection(HttpURLConnection connection) throws Exception {
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Authorization", "Basic heregoestheencodedkeyandid");
StringBuilder postDataBuilder = new StringBuilder();
byte[] postData = postDataBuilder.toString().getBytes("UTF-8");
OutputStream out = conn.getOutputStream();
out.write(postData);
out.close();
connection.connect();
}
also i found another way of simply adding one line:-
private static void setupPOSTConnection(HttpURLConnection connection) throws Exception {
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Authorization", "Basic heregoestheencodedkeyandid");
conn.setFixedLengthStreamingMode(0);
connection.connect();
}
HTTP Status Code 411 (Length Required) is sent by the server as a response when it refuses to accept a message without a content-length header, for whatever reason.
A server simply may or may not accept a content without a Content-Length header, and it seems this API is picky. See this post detailing a working POST connection using Java.
I have SugarCRM trail account. I can able to get Authenticate and get the AccessToken by the following url.
https://xxxxxxx.trial.sugarcrm.eu/rest/v10/oauth2/token
Method : POST
POST Data : postData: { "grant_type":"password", "client_id":"sugar", "client_secret":"", "username":"admin", "password":"Admin123", "platform":"base" }
Code I used to get the AccessToken
public static String getAccessToken() throws JSONException {
HttpURLConnection connection = null;
JSONObject requestBody = new JSONObject();
requestBody.put("grant_type", "password");
requestBody.put("client_id", CLIENT_ID);
requestBody.put("client_secret", CLIENT_SECRET);
requestBody.put("username", USERNAME);
requestBody.put("password", PASSWORD);
requestBody.put("platform", "base");
try {
URL url = new URL(HOST_URL + AUTH_URL);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setUseCaches(false);
connection.setDoOutput(true);
connection.connect();
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(connection.getOutputStream()));
out.write(requestBody.toString());
out.close();
int responseCode = connection.getResponseCode();
BufferedReader in = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
JSONObject jObject = new JSONObject(response.toString());
if(!jObject.has("access_token")){
return null;
}
String accessToken = jObject.getString("access_token");
return accessToken;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
Now I have retrive Leads from CRM using rest API I can not able to find the appropriate method and Url to do the thing.
I can see the list rest of API's from /help but I cant understand what should be my module name and what I have to :record and how do I pass my access token for authentication.
Can anyone please help me?
The module name is simply the module you which to fetch records from, so in your case you'll want to do a GET request to rest/v10/Leads for a list of Leads. If you want to fetch a specific Lead you replace :record with the id of a Lead - for example: GET rest/v10/Leads/LEAD-ID-HERE
SugarCRM's documentation has a lot of relevant information that might not be included in /help plus working examples.
http://support.sugarcrm.com/Documentation/Sugar_Developer/Sugar_Developer_Guide_7.8/Integration/Web_Services/v10/Endpoints/module_GET/
http://support.sugarcrm.com/Documentation/Sugar_Developer/Sugar_Developer_Guide_7.8/Integration/Web_Services/v10/Examples/PHP/How_to_Fetch_Related_Records/
You need to include your retrieved token into an OAuth-Token header for subsequent requests, and then just use the module name as the endpoint i.e. in your case: "rest/v10/Leads" and call the GET method to retrieve them. Try something akin to this:
String token = getAccessToken();
HttpURLConnection connection = null;
try {
URL url = new URL(HOST_URL + "/rest/v10/Leads");
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Type", "application/json");
connection.setRequestProperty("OAuth-Token", token);
connection.setUseCaches(false);
connection.setDoOutput(true);
connection.connect();
int responseCode = connection.getResponseCode();
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
JSONObject jObject = new JSONObject(response.toString());
System.out.println(jObject);
} catch (Exception e) {
e.printStackTrace();
}
In the case you want to filter it down to specific id's to cut down on the amount of returned data, you can specify it after the module name i.e. "rest/v10/Leads/{Id}"
I have a java method which should get Set-Cookie property for following login into webpage. But the conn.getHeaderFields().get("Set-Cookie") does not return anything. Any advice?
private String GetPageContent(String url) throws Exception {
URL obj = new URL(url);
conn = (HttpsURLConnection) obj.openConnection();
// default is GET
conn.setRequestMethod("GET");
conn.setUseCaches(false);
// act like a browser
conn.setRequestProperty("User-Agent", USER_AGENT);
conn.setRequestProperty("Accept",
"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
conn.setRequestProperty("Accept-Language", "s-CZ,cs;q=0.8,en;q=0.6");
if (cookies != null) {
for (String cookie : this.cookies) {
conn.addRequestProperty("Cookie", cookie.split(";", 1)[0]);
}
}
int responseCode = conn.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader in =
new BufferedReader(new InputStreamReader(conn.getInputStream(),"UTF-8"));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// Get the response cookies
System.out.println(conn.getHeaderFields().get("Set-Cookie")); //print for testing
setCookies(conn.getHeaderFields().get("Set-Cookie"));
return response.toString();
Whole program: http://pastebin.com/3nB682L7
Anyone?..:-)
Recent Java versions have "fixed" URLConnection to hide cookies that are marked HttpOnly, and I don't think there's a setting to disable that. I would recommend using HttpClient from Apache HttpComponents.
I need to post data to server using HttpURLConnection. Data contains Thai character as well. Server which accept post request accept encoding UTF-8 and TIS-620 both. When I directly post data from rest client it works fine, but when I send same request from java code it is not working properly, I mean when I send UTF-8 format data it gives parse exception and when I use TIS-620 instead thai text in server I am getting some speial character square etc.( I do not any have handle on server which accept data )
I am setting same header property for HttpURLConnection which I set for rest client in browser.Please let me know what could be going wrong here
As per my requirement I have to write this code in servlet , and servlet is called from browser AJAX call. In JQUERY AJAX call while sending data I am setting
beforeSend: function(xhr) {
xhr.setRequestHeader( "Content-type", "application/json; charset=UTF-8" );
},
I changed UTF-8 to TIS-620 All places but no luck
Some finding :
when I set
connection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
and Print connection.getContentType() it is only printing application/json
If I send only English text it works fine. I am giving servlet code below which accept post request and POST it to server
protected void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException {
System.err.println("************** POST CALLED **************");
//PRINT SERVLET REQUEST PROPERTY
System.err.println("CharacterEncoding : "+ request.getCharacterEncoding());
System.err.println("ContentType : "+request.getContentType());
//SEND GET REQUEST AND FETCH XCSRF TOKEN
String dummyServiceUrl = "GET_LOT_SRV/get_lot";
String xcsrfToken = null;
HttpURLConnection connection = null;
String requestURL = httpPrefix + hostName + semiColon + portNumber + forwardSlash + dummyServiceUrl;
List<String> cookies = null;
try {
URL gatewayServiceUrl = new URL(requestURL);
connection = (HttpURLConnection) gatewayServiceUrl.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Authorization", this.getBasicAuth());
connection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
connection.setRequestProperty("x-csrf-token", "fetch");
connection.connect();
if (HttpURLConnection.HTTP_OK == connection.getResponseCode()) {
//ON SUCCESS GET XCSRF TOKEN AND IN SAME SESSION POST DATA
requestURL = httpPrefix + hostName + semiColon + portNumber + forwardSlash + request.getQueryString();
gatewayServiceUrl = new URL(requestURL);
connection = (HttpURLConnection) gatewayServiceUrl.openConnection();
//SET CONNECTION PROPERTY
connection.setRequestMethod("POST");
xcsrfToken = connection.getHeaderField("x-csrf-token");
cookies = connection.getHeaderFields().get("set-cookie");
// SET COOKIES
for (String cookie : cookies) {
String tmp = cookie.split(";", 2)[0];
connection.addRequestProperty("Cookie", tmp);
}
//SET HEADERS
connection.setRequestProperty("Authorization", this.getBasicAuth());
connection.setRequestProperty("x-csrf-token", xcsrfToken);
connection.setRequestProperty("Content-Type", "application/json; charset=utf-8");
connection.setRequestProperty("Accept", "application/json");
connection.setRequestProperty("DataServiceVersion", "2.0");
connection.setRequestProperty("X-Requested-With", "XMLHttpRequest");
connection.setRequestProperty("Accept-Encoding", "gzip, deflate");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setUseCaches(false);
//SET USERS INPUT DATA TO OUTPUT STREAM
String payload = this.getDataFromStreamPost(request.getInputStream());
DataOutputStream dataOutputStream = new DataOutputStream(connection.getOutputStream());
dataOutputStream.write(payload.getBytes());
dataOutputStream.flush();
dataOutputStream.close();
//POST DATA AND CHECK RESPONSE
connection.connect();
response.setStatus(HttpURLConnection.HTTP_CREATED);
response.setContentType("application/json; charset=TIS-620");
response.getWriter().println(this.getDataFromStream(connection.getInputStream()));
} else {
System.err.println("XCSRF GET FAILURE "+connection.getResponseCode());
response.setStatus(connection.getResponseCode());
response.setContentType("application/json; charset=TIS-620");
response.getWriter().println(this.getDataFromStream(connection.getInputStream()));
}
} catch (Exception e) {
System.err.println("EXCEPTION OCCURED IN POST : "+e.getMessage());
response.setStatus(connection.getResponseCode());
response.setContentType("application/json; charset=TIS-620");
response.getWriter().println(this.getDataFromStream(connection.getErrorStream()));
}
}
private String getBasicAuth() {
String userpass = userName + ":" + password;
return "Basic "
+ javax.xml.bind.DatatypeConverter.printBase64Binary(userpass
.getBytes());
}
private String getDataFromStream(InputStream stream) throws IOException {
StringBuffer dataBuffer = new StringBuffer();
BufferedReader inStream = new BufferedReader(new InputStreamReader(
stream));
String data = "";
while ((data = inStream.readLine()) != null) {
dataBuffer.append(data);
}
inStream.close();
return dataBuffer.toString();
}
private String getDataFromStreamPost(InputStream stream) throws IOException {
StringBuffer dataBuffer = new StringBuffer();
BufferedReader inStream = new BufferedReader(new InputStreamReader(
stream,"UTF-8"));
String data = "";
while ((data = inStream.readLine()) != null) {
dataBuffer.append(data);
}
inStream.close();
return dataBuffer.toString();
}
I keep keep getting the above error when I run the code below. All signs point to a problem with COOKIES from what I've read. If I am correct,how would I go about Implementing the CookieManager to fix this issue? Or how would I fix the issue if it is not an issue with COOKIES?
public class Client {
public Client(){
}
String executePost(String targetURL, String urlParameters){
URL url;
HttpURLConnection connection = null;
try{
//Create connection
url = new URL(targetURL);
connection = (HttpURLConnection)url.openConnection();
connection.setChunkedStreamingMode(0);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Length", "" +
Integer.toString(urlParameters.getBytes().length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
//send Request
DataOutputStream dataout = new DataOutputStream(connection.getOutputStream());
dataout.writeBytes(urlParameters);
dataout.flush();
dataout.close();
//get response
InputStream is = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = br.readLine()) != null){
response.append(line);
response.append('\n');
}
System.out.println(response.toString());
br.close();
return response.toString();
}catch(Exception e){
System.out.println("Unable to full create connection");
e.printStackTrace();
return null;
}finally {
if(connection != null) {
connection.disconnect();
}
}
}
}
I removed : connection.setChunkedStreamingMode(0); and the code worked as it should
String executePost(String targetURL, String urlParameters){
URL url;
HttpURLConnection connection = null;
try{
//Create connection
url = new URL(targetURL);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("Content-Language", "en-US");
connection.setRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
//send Request
DataOutputStream dataout = new DataOutputStream(connection.getOutputStream());
dataout.writeBytes(urlParameters);
dataout.flush();
dataout.close();
//get response
InputStream is = connection.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = br.readLine()) != null){
response.append(line);
response.append('\n');
}
System.out.println(response.toString());
br.close();
return response.toString();
}catch(Exception e){
System.out.println("Unable to full create connection");
e.printStackTrace();
return null;
}finally {
if(connection != null) {
connection.disconnect();
}
}
}
A 403 response means "forbidden".
All signs point to a problem with COOKIES from what I've read. If I am correct,how would I go about Implementing the CookieManager to fix this issue? Or how would I fix the issue if it is not an issue with COOKIES?
It is not as simple as that.
It may be that you need to supply valid credentials (in the form of cookies, basic auth headers, or something else). However, you expect the server to respond with a 401 in that case, or a 302 to redirect your browser the a login page.
It may also be that you've supplied credentials, and they are not sufficient for the request you are trying to perform.
Your best bet is to figure out exactly what is happening when you try to login and use the service from your web browser. Then try to replicate that. Alternatively, read the site documentation or ask the site admins what to do.
If it is your site / server that you are trying to access, then you need to figure out how security is implemented. Perhaps you've misconfigured the server, or neglected to set up / enable login.
It is unlikely (IMO) that you will solve this problem by just setting up a Cookie Manager.
If you see the API documentation of setChunkedStreamingMode, it has been mentioned there that not all servers support this mode. Are you sure that the server you are making a connection to supports this ?