so far, I have been able to use the HttpURLConnection class in java to make an app that can GET the form of my php website, put in the proper login details (username, password) and POST them back. I have double checked this with the response codes and am getting 200 for both GET and POST.
I'm having an issue now accessing the page that a successful login should redirect to. It is to my understanding that after a POST or GET, the connection is terminated once the response code is requested. My attempts to get the response cookies while logging in produce a "null" cookie.
The PHP site I am accessing does not seem to have any response cookies after a login when using "inspect element" in Chrome. Regardless of this, I have tried accessing the cookies all sorts of ways with no such luck. The request cookie header is there when I go the the website.
Am I missing something and the cookies are actually there? Or is it possible that the site does not use cookies to maintain a session? If that's the case, how would I access the page I want after logging in on my Android app?
Response Headers
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:23030
Content-Type:text/html; charset=utf-8
Date:Mon, 10 Aug 2015 23:03:26 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Keep-Alive:timeout=15, max=100
Pragma:no-cache
Server:Apache/2.2.22 (Debian)
Vary:Accept-Encoding
X-Powered-By:PHP/5.4.4-14+deb7u11
Request Header
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:52
Content-Type:application/x-www-form-urlencoded
Cookie:__utma=83554121.1278939357.1435860313.1435944069.1438202297.3; __utmc=83554121; __utmz=83554121.1438202297.3.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _ga=GA1.2.1278939357.1435860313; PHPSESSID=4q03j4ihb7trnm1pvvofc9f3f5
Host:WEBSITE
Origin:WEBSITE
Referer:WEBSITE
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36
Just an update on what I have figured out so far.
I logged in to my website on my browser on my PC. I took this PHPSESSID and used it in my app and I can access the page that the login page redirects to just fine.
My new question is: Why can't I assign myself a random PHPSESSID and go back to it? In other words, if I give myself PHPSESSID=123456 for example and use that when posting my login details, why does using that specific PHPSESSID still bring me back to the login page instead of the redirected one?
I'm currently trying to read the request header "set-cookie" but am having trouble doing so. In addition, even putting in the wrong user and/or password gives a response code of 200. Suggestions on how to check if logging in was successful?
I'm trying to retrieve links from this page: http://www.seas.harvard.edu/academics/areas
There is a link named "Computer Science" in the middle of the page. Its underlying link is given as "/academics/areas/computer-science". I'm able to convert it to an absolute URL with the Java built-in URL class, obtaining "http://www.seas.harvard.edu/academics/areas/computer-science".
When I click the link in Chrome browser, however, the absolute URL changes to "http://www.seas.harvard.edu/computer-science".
So my question is two-fold:
How does the URL redirect work in this page?
Is there any library or method in Java that would help me obtain the URL after redirect?
I need to obtain the URL after redirect because I want to read the source code of the page but the URL before redirect doesn't work for me. I'm using the JSoup library to read from the URL so I suspect it might be a javascript-based redirect.
From curl --dump-header [file] [URL] the file looked like:
HTTP/1.1 301 Moved Permanently
Age: 0
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
Content-Type: text/html
Date: Tue, 13 Aug 2013 13:00:12 GMT
ETag: "1376398812"
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Tue, 13 Aug 2013 13:00:12 GMT
Location: http://www.seas.harvard.edu/computer-science
Server: nginx
Vary: Accept-Encoding
Via: 1.1 varnish
X-AH-Environment: prod
X-Cache: MISS
X-Drupal-Cache: MISS
X-Redirect-ID: 44
X-Varnish: 2704315535
transfer-encoding: chunked
Connection: keep-alive
As you can see this is a 301 permanent redirect served from the server.
To obtain the data:
You can use HttpURLConnection to connect, but before connecting, call myConn.setInstanceFollowRedirects(true). Redirects are followed and you can get your output stream and read it.
To obtain the URL itself:
You can use HttpURLConnection to connect, but before connecting, call myConn.setInstanceFollowRedirects(false) to not follow redirects. This will save the actual URL in the right place.
The trick here is that for some odd reason, HttpURLConnection doesn't allow to retrieve a header by name unless you parse it as a date.
So, you will need to iterate an integer, calling getHeaderFieldKey after making the connection and checking if it equal to Location and if it is, getting getHeaderField with the same integer to get the location. Annoying, I know. But a location isn't a date and this is a JRE oversight.
I used Fiddler to investigate and the site return for link http://www.seas.harvard.edu/academics/areas/computer-science HTTP 301 response code, that performs the redirect.
I you want to get real URL. You should perform real request to harvard.edu web server and parse response. (Redirect URL is located in Location key in HTTP Header).
Sorry about your second question. I don't have skill in Java.
This SO question may help (httpclient-4-how-to-capture-last-redirect-url)
There is probably e.g. a .htaccess and mod_rewrite redirect. Using Firefox's Console I could see the requests. As you can see below the server is sending back a 301 Moved Permanently message. This tells the browser to redirect to the address returned in the Location header of the response.
The way you obtain the changed URL depends on the way you load the page:
If you use ready libraries & code to load the page to e.g. a DOM object, the you could use that ready HTTP system to load the response, this will probably result to it automatically redirecting -> you will get the URL from the URL of the loaded page. If it does not do that, then you must check for status code 301 or 302 and when those are received then the changed URL is in the Location header of the response.
If you have your own code written to load the response via TCP sockets, then you must just load the response as normal, but again check for the 301 and 302 status codes and do as described in the previous section.
I can only attempt to address Q1 since I'm not a Java programmer. The source code says they're using Drupal, so I speculate that they're using Drupal's global redirect module (SO discussion about Drupal redirect module here). Looking at the module's documentation might shed some light on how to obtain the correct url with Java.
There's also numerous ways within javascript to have url requests automatically redirect to some base page (e.g., CS homepage), while physically navigating the site allows the user to advance to new pages. This is standard practice in many single page web apps. If this is the case, then #hexafraction 's suggestion might be able to help you retrieve the desired url, though I'm unfamiliar with the Java methods (s)he is suggesting.
You can get the Redirect URL from the below code setting followRedirects to false.
You will get the source code of the redirected page if you set it to true and that's the default behavior of Jsoup
Connection con = Jsoup.connect("http://www.seas.harvard.edu/academics/areas/computer-science")
.userAgent("Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36")
.followRedirects(false);
System.out.println("Redirected Url : " + con.execute().header("Location")); //null if followRedirect is true
Document doc = con.get();
System.out.println(doc.html());
System.out.println("=================================================");
I have java application running under tomcat which is fronted by apache webserver.
In my code I set cookie domain as
.example.com
but still my cookies shows up under www.example.com instead of under example.com in the client browser. What is so strange google analytics cookies shows up under example.com but my own code cannot store cookies under example.com?
Apache server is setup such that requests for example.com shows up as www.example.com in the client browser address bar if that is related to the issue ? I do need this otherwise different session id are generated for example.com and www.example.com which is bad for my applicaton.
Apache server is setup such that
requests for example.com shows up as
www.example.com in the client browser
address bar if that is related to the
issue ?
I am not 100% sure, but this looks like the root of the problem. How does Apache make the client browser to display www.example.com instead of example.com? Most probably, by redirecting each request for example.com to www.example.com. When the browser processes redirection, it sends a request for www.example.com and from that point on thinks that it is working with www.example.com.
Now, what happens when there is a Set-Cookie in the response header? It will obviously treat it as coming from www.example.com. There is no way a browser would allow such cookie to set its domain to .example.com because it would be a security problem. Imagine that mysite.somefreehosting.com sets a cookie for the domain .somefreehosting.com. Then someothersite.somefreehosting.com would receive this cookie which may lead to a lot of trouble. The standard specifies that such cookie should be rejected, but I wouldn't be surprised if some browsers are smart enough to handle such cases and to treat .example.com as www.example.com.
To be sure, I recommend that you check what exactly your site sends to the browser by sending a request with something like lwp-request script. You'll see what redirections are happening and what headers are actually set in the response, like this:
alqualos#ubuntu:~$ lwp-request -sSed http://google.com/
GET http://google.com/ --> 301 Moved Permanently
GET http://www.google.com/ --> 302 Found
GET http://www.google.co.il/ --> 200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Sat, 18 Dec 2010 18:54:57 GMT
Server: gws
Content-Type: text/html; charset=windows-1255
Content-Type: text/html; charset=windows-1255
Expires: -1
Client-Date: Sat, 18 Dec 2010 18:54:57 GMT
Client-Peer: 173.194.37.104:80
Client-Response-Num: 1
Set-Cookie: PREF=ID=368e9cfd56643257:FF=0:TM=1292698497:LM=1292698497:S=s-Jur84NgaNH5Mzx;
expires=Mon, 17-Dec-2012 18:54:57 GMT; path=/; domain=.google.co.il
Set-Cookie: NID=42=bZ6goDV_b2MiWlTMONwiijaON5U_TBGB2_yNheonEwA1GVLU77EhyfUhk9Wvj70xTFrpvGy4s_aBp1UZtvRRnsnYjacjz_UVx0_iSr9R3nYXMyRtwkS5qV98_Egb16pZ;
expires=Sun, 19-Jun-2011 18:54:57 GMT; path=/; domain=.google.co.il; HttpOnly
Title: Google
X-XSS-Protection: 1; mode=block
Can someone please help me with a HTTP caching question?
I thought that if I set the "Expires" and "Cache-Control:max-age" request headers, then a browser wouldn't make a GET request until after the max-age had passed?
My server is seeing lots of conditional GET requests, and it replies with a 304. However, I'd like to eliminate these conditional GETs if possible and have the browser only ask for a resource once it thinks it has expired.
Here's the details I get from the developer tools in Chrome:
Request URL:http://localhost:8080/img/branding.gif
Request Method:GET
Status Code:304 Not Modified
Request Headers:
Cache-Control:max-age=0
If-Modified-Since:Thu, 22 Jul 2010 10:54:34 GMT
Referer:http://localhost:8080/
User-Agent:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4
Response Headers:
Cache-Control:max-age=2246400, s-maxage=0
Connection:Keep-Alive
Content-Location:/img/branding.gif
Content-Type:image/gif
Date:Fri, 23 Jul 2010 15:06:10 GMT
Expires:Fri, 07 Jan 2011 15:06:10 GMT
Keep-Alive:timeout=15, max=100
Last-Modified:Thu, 22 Jul 2010 10:54:34 GMT
Pragma:cache
Server:Oracle Application Server Containers for J2EE 10g (9.0.4.0.0)
I set the cache headers in a filter in my Java code. The code is running on Oracle's OC4J.
Also, I want to run this over HTTPS. Will using the same headers work for HTTPS too?
Thanks in advance.
Remove "Last-Modified" header. Some browsers always send conditional GET if that header exists.
Could it be the s-maxage=0? Can you lose that, or set it the same as the maxage?
The conditional gets could be coming from proxies, or from browsers that have somehow decided to act like proxies. Seems unlikely, i admit.
I recently built a simple web-app deployed over Tomcat. The app uses pretty standard session based security where a user who has logged in is given a session.
Sessions work fine in Firefox and Chrome, but require the use of jsessionid in the URL for IE (tested 7 & 8), set to medium privacy. In IE 8, I tried to override cookie handling, setting "Allow all 3rd party cookies" and "Allow all session cookies"- no dice. However, when I run Tomcat on my local machine, IE accepts the cookie, and sessions work just fine.
And now, for the HTTP headers.
From Chrome, a logged in user gets a session
GET http://devl:8080/testing/ HTTP/1.1
Host: devl:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
P3P: CP="NON CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT STA"
Set-Cookie: JSESSIONID=9280023BCE2046F32B13C89130CBC397; Path=/testing
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Content-Length: 2450
Date: Fri, 26 Mar 2010 14:14:40 GMT
GET http://devl:8080/testing/logout HTTP/1.1
Host: devl:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1036 Safari/532.5
Referer: http://devl:8080/testing/
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=9280023BCE2046F32B13C89130CBC397
...
From IE 8, with standard medium level security and privacy-
GET http://devl:8080/testing/ HTTP/1.1
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, */*
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDC; Tablet PC 2.0)
UA-CPU: AMD64
Accept-Encoding: gzip, deflate
Host: devl:8080
Connection: Keep-Alive
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
P3P: CP="NON CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT STA"
Set-Cookie: JSESSIONID=192999F922D6E9C868314452726764BA; Path=/testing
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Content-Length: 2450
Date: Fri, 26 Mar 2010 14:32:34 GMT
GET http://devl:8080/testing/logout HTTP/1.1
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, */*
Referer: http://devl:8080/testing/;jsessionid=6371A83EFE39A46997544F9146AA5CEA
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDC; Tablet PC 2.0)
UA-CPU: AMD64
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Host: devl:8080
...
I thought it might be P3P, but on adding a compact policy, nothing changes. This is the standard Tomcat session, so I'm really surprised I haven't been able to find other people with the same problem so far. Anyone have any ideas?
EDIT 4/3/2010 -
Sorry if I didn't make this clear- I've tried from multiple other instances of IE - co-workers down the hall, etc.
EDIT 4/3/2010 -
I've also tried turning on prompting for all cookies, but I don't get a prompt. Setting the domain in the "Set-Cookie" header using Fiddler didn't make a difference, either.
I ran into this exact problem, dug around for a while, and found this:
http://forums.iis.net/p/1147938/1879164.aspx
which says that domain names that have underscores in them cause problems with Windows Server, tomcat and IE
not sure if this fixes your problem (and at this point, you probably don't care) but maybe the next person who comes along can gain some value from it.
Problem: IE8 refused to accept cookies on a site I had built, but Firefox and IE7 worked just fine and had done so for ages - this was stable code.
Solution (for me): My server is in a different time zone to the client machine. The STUPID, IDIOTIC IE8 tries to be clever and refuses to accept cookies (stored in the local client machine) with a 20 minute life. My PHP code was straight from the text book, thus:
setcookie($name,$value,time()+1200);
But it works fine if I change it to, for example -
setcookie($name,$value,time()+120000);
This still leaves me with the problem of making the cookie die after 20 minutes, but at least my users can now use my website with IE8. I pass on this information in case it may help someone else.
Have you checked that the server time is correct?
I have had similar problems recently with IE not accepting cookies properly. After a lot of head scratching it turned out to be because the time difference between the server and client machines was so big that IE refused to accept the cookie. This was in Apache however.
Try using the standard HTTP port (80). I've read about issues with port numbers in URLs regarding privacy/security in IE more than once but can't seem to find relevant links at this time.
I agree with Lexicore - the cookie protocol from the web server looks right, so there's something with IE. It would be easier to figure out how to address the issue if we understood better why IE is rejecting the cookie. Alternatively, ask a friend to hit the site for you in IE to help confirm its a server issue not a browser instance issue.
Here is some things to check to help debug with IE and cookies - unfortunately, there's a mess of options to check. Sorry if some of these items seem basic - I just don't wnat to make any assumptions. I'm following along in IE 8.0 for this.
First, browse to the target site (http://devl:8080/testing/) in IE. Then:
Confirm what zone IE classifies 'http://devl:8080/testing/'. (This could explain why its works with Tomcat on your local machine.) The zone is displayed in the bottom bar of the browser and it most likely says "Internet". If it instead says "Local intranet", "Trusted Site", or "Restricted Site", this may be part of the problem and you should update your question or figure out why it isn't classified as Internet.
Double-click on the zone indicator in the bottom bar (presumably "Internet") to open the Security dialog. Is the Security Level for Internet set to Medium-high? If it isn't, this could be part of the problem and you should probably reset it back to match your users.
Select the "Internet" zone and then click the "Custom level ..." button to open the Security Settings dialog. Confirm the "Userdata persistence" option is set to "Enable". The "Userdata persistence" option is in the bottom 1/4 of the list of options in the "Miscenllaneous" section (near the bottom of the section just above the next section "Scripting").
Click OK on each dialog to close both of them.
On the menubar (enable it if it is not enabled), click "Tools" > "Internet Options". Select the "Privacy" tab. I know you mentioned you tried some things here, but those changes may not affect your site if your site is not in the Internet zone or if your site in the "Per Site Privacy Actions" exception list, so its best to just confirm.
Is the privacy setting in the Privacy tab set to Medium? If not, you may want to reset to default.
Click the "Sites" button to open the Per Site Privacy Actions dialog. Is your dev1 site listed? If so, remove it. Click OK to dismiss the dialog. Alternatively, you could force your dev1 site to always Allow cookies.
Click the "Advanced" button. Is "Override automatic cookie handling checked? If so, you might want to uncheck it to match your users. Alternatively, try checking it and checking "Always allow session cookies."
Click OK on each dialog to close both of them.
Confirm the browser is still at target site ('http://devl:8080/testing/'). Click "View" > "Webpage Privacy Policy..." to view the Privacy Report dialog. Does the list include "http://dev1:8080/testing/"? Does the Cookie column indicate "Accepted" for "http://dev1:8080/testing/"?
Select "http://dev1:8080/testing/" from the list. Click Summary to see the Privacy Policy. If set one for the your site, you should see it here. Otherwise, you should get a message that a privacy policy was not found. Look at the bottom of the dialog to see how the site is set to use cookies (compare, always allow, or never allow).
Hope this helps or gives you some ideas to pursue.
Ref:
http://blogs.msdn.com/ieinternals/archive/2009/08/20/WinINET-IE-Cookie-Internals-FAQ.aspx
http://www.practicalmachinist.com/vb/general/how-manage-cookies-internet-explorer-181641/
http://support.microsoft.com/kb/283185
This forum concerning P3P seems relevant.
Also have you considered setting your domain and expiration date for the session cookie?
This has clearly nothing to do with Tomcat, since the cookie is being set - just not accepted by the IE. This must be security issue in IE then.
Maybe this MS article would help to tune it.
What security zone is the dev1 site part of? IE handles cookies and lots of other security differently depending on which zone (and how the zone is configured).
Try setting the dev1 site to explicitly be part of the Trusted Sites for example and see what happens.
Zones:
Internet
Local Intranet
Trusted Sites
Restricted Sites
Also, does the cookie have to be restricted to the /testing path? Try setting it for / and see if that makes a difference.
I would try using the fully qualified hostname of the server. MSIE treats hostname without domains as being in the "Local intranet" and handles security differently.
Specifically, instead of:
http://devl:8080/testing/
Try using something like:
http://devl.mydomain.com:8080/testing/
It seems from what you're saying that you've only seen this issue in IE and only using computers in your office. Is there any sort of "security suite" installed by IT on all office computers, and if so, can you temporarily disable it? Oftentimes, these types of applications hook into IE and muck with its HTTP stack. If you do have software like that installed, do you have a "clean" installation or non-company computer you can test with?
The time on our servers were off by 14 minutes (and in the correct time zone EST).
Once we set the time on the server to the correct time cookies starting working again.
Ed