Develop Reference

Java HttpClient Request return 403 status while postman returns the intended response

Tried to request with postman which gives desired response.
The java code used for making the same request is as below fails with a 403 status.
String url = "https://steamcommunity.com/inventory/76561198865293952/440/2?l=english&count=5000";
String cookiesString = "sessionid=" + generateSessionId() + ";steamCountry=IN%7Ce744269b3c4e531facb33ecaff29eb44";
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest
.newBuilder()
.GET()
.header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36")
.header("Accept", "*/*")
.header("Cookie", cookiesString)
.uri(URI.create(url))
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode()); //prints 403
System.out.println(response.body()); // prints �+�� O��%
/**
* Generates a Steam Session Id.
* #return sessionId.
*/
public static String generateSessionId() {
return new BigInteger(96, new Random()).toString(16);
}
Postman request does not need any headers(Not even User-Agent, only Host is kept, No cookies are needed) at all to get the desired response. Even navigating to link in browser shows the json response.
PostMan request ScreenShot
Postman Request Image with cookies
Thank you. Have a great day.

I am pretty sure that the 2 cookies you have in postman and not in the java code are responsible for having the 403 response.
Postman is a google chrome plugin so when connecting w/ chrome, maybe you stored in the cache the cookies.
You also need to add them to the java code.

Firstly Thanks for all your answers and replies.
So I found that the inventory that I was trying to access was set visible to friends only, so response was 403. So it does work for inventories that are set to public.
I am sorry, I should have check it first.
What I still dont understand is how did the postman desktop client get the intended response? I understand that chrome was able to get the inventory because I have active steam login with cookies that would have been send but postman request did not have any cookies other than sessionid and country. If someone could explain how this happens it would really help me.

Cannot get 'location' header in response using HttpClient

The location header is there, I can see it in browser:
I'm using org.apache.httpcomponents.httpclient to send http request with cookie:
```
URI uri = new URIBuilder().setScheme("https").setHost("api.weibo.com").setPath("/oauth2/authorize").setParameter("client_id","3099336849").setParameter("redirect_uri","http://45.78.24.83/authorize").setParameter("response_type", "code").build();
HttpGet req1 = new HttpGet(uri);
RequestConfig config = RequestConfig.custom().setRedirectsEnabled(false).build();
req1.setConfig(config);
req1.setHeader("Connection", "keep-alive");
req1.setHeader("Cookie", cookie);
req1.setHeader("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36");
response = httpclient.execute(req1);
```
I googled a lot and tried enable/disable auto redirect,but it doesn't seem to work for me.So can somebody tell me how to get the location header in response just like the browser did?

You cannot see 'location' header, because HttpClient followed that redirect immediately - even before giving you that response.
Try disabling redirect while setting up your HttpClient:
HttpClient instance = HttpClientBuilder.create().disableRedirectHandling().build();
Check this URL and you'll see the Location Header:
URI uri = new URIBuilder().setScheme("https").setHost("www.googlemail.com").setPath("/").build();

I found out my real question...I didn't pass the auth process in my code,so I keep getting oauth2 page.After I set all the headers in my request just like the browser did and finally I get the right response.

Server returned HTTP response code 503 for URL

I'm able to access website kissmanga.com yet I can't access it via program. I fixed error 403 that I was getting before that but now I get error 503.
URL url = new URL("http://kissmanga.com/");
System.setProperty("http.agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.29 Safari/537.36");
BufferedReader bf = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
while((str = bf.readLine()) != null){
System.out.println(str);
}
Error that I get:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 503 for URL: http://kissmanga.com/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at KissManga.main(KissManga.java:10)
Okay this code works with one small annoying problem. I don't get full html but just 2/3 of it.
HtmlUnitDriver driver = new HtmlUnitDriver();
driver.get("http://kissmanga.com/");
Thread.sleep(5000);
System.out.println(driver.getPageSource());
driver.quit();

You won't get any data this way, because site checks for Javascript enabled.
You should try tools which can emulate browser behaviour. For example, that's how you can get page source with the help of Selenium Htmlunit Driver:
HtmlUnitDriver drv = new HtmlUnitDriver(BrowserVersion.FIREFOX_38);
drv.setJavascriptEnabled(true);
drv.get("http://kissmanga.com/");
drv.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
System.out.println(drv.getPageSource());

Error 503 means that server is reachable, but returned an error status code
503 is for "Service Unavailable"
Maybe a problem happened temporarily on server or server rejected your request for some reason

It's because the site appears to use Cloudflare.
You can tell when you visit the site and get 'please wait while we check your browser'
503 = HTTP 503 Service Unavailable
This is Cloudflare telling you to hang on while it makes sure you aren't a DDOS.
You will need to code your parser to review the body and either wait out the redirect, or visit it manually yourself.

Program can not login into website

This is my code:
URL url = new URL("http://superchillin.com/login2.php");
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setUseCaches(false);
urlConnection.setRequestMethod("POST");
String data = "email="+URLEncoder.encode(name, "UTF-8")+"&password="+URLEncoder.encode(pass, "UTF-8");
urlConnection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
urlConnection.setRequestProperty("Accept-Encoding", "gzip,deflate");
urlConnection.setRequestProperty("Accept-Language", "en-US,en;q=0.8,lt;q=0.6");
urlConnection.setRequestProperty("Cache-Control", "max-age=0");
urlConnection.setRequestProperty("Connection", "keep-alive");
urlConnection.setRequestProperty("Content-Length", Integer.toString(data.getBytes().length));
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.addRequestProperty("Cookie", "place=1");
urlConnection.addRequestProperty("Cookie", "lvca_unique_user=1");
urlConnection.setRequestProperty("Host", "superchillin.com");
urlConnection.setRequestProperty("Origin", "http://superchillin.com");
urlConnection.setRequestProperty("Referer", "http://superchillin.com/login.php");
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setInstanceFollowRedirects(true);
DataOutputStream wr = new DataOutputStream(urlConnection.getOutputStream());
wr.writeBytes(data);
wr.flush();
wr.close();
After that code I only read the response. It redirects me to "login.php" and is trying to set cookie "place=1"...
Connecting via browser works great. The reason for so many headers is I thought they may be the problem so I copied all headers from which I see when using a browser.
The response code is 200.
I also noticed that if password or email is incorrect, there's a message saying that in HTML which i retrieve.
When I use a browser I get redirected to index.php and cookie "auth" is set. So that's what I'm expecting from my program aswell. Curently I get redirected back to "login.php".

There is no universal answer to this question, I'm afraid. What you're asking is "why does the remote server not return an auth cookie when I send this exact request?" And that depends entirely on what the server's documentation says about those requests, whether it has any bugs in its implementation, etc.
If you don't have access to the server's own source and logs, then you'll likely have to get by with experimentation. Use something like Firebug or Chrome's Developer Tools to capture the exact requests sent by the browser with the login works successfully. Since these text strings are the only thing the remote server sees, if you replicate them exactly with your Java program you will(/should) get exactly the same responses.
If you think you're sending the same requests from Java and find that you're still not getting the expected responses, there must be some difference. Try recording the network traffic with something like Wireshark in order to see exactly what your app is sending - and then address the differences.
And if you get to the point where e.g. a redirect isn't being followed, and you're not sure how to do that with a URLConnection - then that's a good concrete question to ask.

login to http website java

I am trying to login to a http website for the first time and I am having a hard time understanding the proper format for sending arguments. I have looked at other examples and they don't seem to work for me so I thought I would see if someone can explain this to me. At this point my code seems to do absolutely nothing but here it is...
HttpURLConnection url= (HttpURLConnection)new URL("http://www.myameego.com/index2.php?do=login").openConnection();
url.setDoOutput(true);
url.setRequestMethod("POST");
OutputStreamWriter writer = new OutputStreamWriter(url.getOutputStream());
writer.write("X-Mapping-fjhppofk=6A991610BA398B3A39F4B491D5382BB4;
PHPSESSID=kbo25e08t3qvu08l1shkq8kk94; userName=coled; pass=ed45d626b07112a8a501d9672f3b92796a6754b8d8d9cb4c617fec9774889220; clientID=129; X-Mapping-fjhppofk=DCE62FE972E1EF2F12D0060EC74C3681; PHPSESSID=ukeo21oldb5pqsntu7kl8j3b96");
writer.flush();
I downloaded an http sniffer thinking that I could read what the browser was sending. that is how I got the write() line, it is the cookie that was sent by explorer. I also viewed the source code for the login screen and found a block of code near the bottom that looks like its responsible for login.
http://www.myameego.com/index2.php?do=login
Can someone tell me how I would go about hooking into this interface I don't understand how this works. if it helps this is the full packet from my manual login through the browser. I got it from my http sniffer.
Host Name: www.myameego.com
Method: POST
Path: /index2.php?do=login
User Agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; NP06)
Response Code: 302
Response String: found
Content Type: text/html; charset=UTF-8
Referer: http://www.myameego.com/index.php?do=login
Transfer Encoding: chunked
Server: Apache
Content Length: 17817
Connection: Keep-Alive
Cache Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Location: /Ameego/index.php
Cookie: X-Mapping-fjhppofk=6A991610BA398B3A39F4B491D5382BB4; PHPSESSID=kbo25e08t3qvu08l1shkq8kk94; userName=coled; pass=ed45d626b07112a8a501d9672f3b92796a6754b8d8d9cb4c617fec9774889220; clientID=129; X-Mapping-fjhppofk=DCE62FE972E1EF2F12D0060EC74C3681; PHPSESSID=ukeo21oldb5pqsntu7kl8j3b96
URL: http://www.myameego.com/index2.php?do=login
How can I make a packet like the one above? any guidance would be greatly appreciated.
I Looked into that link you posted and the http sniffer shows that the POST request is being called but the cookie line doesn't match up with that of the manual browser request.
HttpURLConnection httpConnection = (HttpURLConnection)new URL("http://www.myameego.com/index2.php?do=login").openConnection();
httpConnection.setDoOutput(true);
httpConnection.setRequestMethod("POST");
httpConnection.setRequestProperty("Accept-Charset","UTF-8");
httpConnection.setRequestProperty("User-Agent","Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; NP06)");
httpConnection.setRequestProperty("Content-Type","application/x-www-form-urlencoded;charset=UTF-8");
String info = String.format("user=%s&coled=%s",URLEncoder.encode("user","UTF-8"),URLEncoder.encode("coled","UTF-8"));
info += String.format("pass=%s&MYPASS=%s",URLEncoder.encode("pass","UTF-8"),URLEncoder.encode("MYPASS","UTF-8"));
info += String.format("clientID=%s&129=%s",URLEncoder.encode("clientID","UTF-8"),URLEncoder.encode("129","UTF-8"));
info += String.format("login=%s&Sign In=%s",URLEncoder.encode("login","UTF-8"),URLEncoder.encode("Sign In","UTF-8"));
httpConnection.setRequestProperty("Cookie",info);
OutputStream output = httpConnection.getOutputStream();
output.write(info.getBytes("UTF-8"));
int x;
while((x = httpConnection.getInputStream().read()) != -1)System.out.print((char)x);
my Cookie:
user=user&coled=coledpass=pass&MYPASS=MYPASSclientID=clientID&129=129login=login&Sign In=Sign+In
browsers cookie:
X-Mapping-fjhppofk=6A991610BA398B3A39F4B491D5382BB4; PHPSESSID=112tg9i4afau5i382hui705553
anyone know what I may be missing here?

With Jsoup this should be simple like this:
Connection.Response response = Jsoup.connect("http://www.myameego.com/index2.php?do=login")
.method(Connection.Method.GET)
.execute();
Document page = Jsoup.connect("http://www.myameego.com/index2.php?do=login")
.data("user", "login")
.data("pass", "password")
.data("clientID", "123456")
.cookies(response.cookies())
.post();
Gathered with Google Chrome Developer Tools