Develop Reference

Java HTTP GET timeouts but works fine in browser

NOTE: This is not a duplicate question. I'm aware of concrete problems that looked identical to mine and were solved by adding some data to the header requests. This is not the case, I've tried all the solutions and none works. Tried: this question and this one, nothing seems to work.
I'm trying to read contents of a website using Java. The url is URL to fetch. There's no authentication involved, and no forms are filled before. I can open that url in a cookie-free session and it still works with browser. I can even fetch it with Selenium, but HttpClient keeps refusing to load the URL.
The problem has nothing to do with certificates, right now I'm using a working "allow-all" certificate manager, so that's not the issue.
I've inspected my browser sent headers, nothing special:
Host: www.hipercor.es
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
As I said, I've already tried configuring the user agent to "fake" being Firefox.
Just to give some background, I'm building a enhanced version of crawler4j, my idea is to build a web scraper, and I found this issue testing random shops I knew are currently being crawled in bussiness environment by other scrapers.
Note that HeadRequest also fail.
The errors are either
java.net.SocketException: Connection reset
or
java.net.SocketTimeoutException: Read timed out
Please, note that browser loads it perfectly, as well as using Selenium Drivers to load the page (although it is slow as hell in that case)

Android to PHP session without cookies

so far, I have been able to use the HttpURLConnection class in java to make an app that can GET the form of my php website, put in the proper login details (username, password) and POST them back. I have double checked this with the response codes and am getting 200 for both GET and POST.
I'm having an issue now accessing the page that a successful login should redirect to. It is to my understanding that after a POST or GET, the connection is terminated once the response code is requested. My attempts to get the response cookies while logging in produce a "null" cookie.
The PHP site I am accessing does not seem to have any response cookies after a login when using "inspect element" in Chrome. Regardless of this, I have tried accessing the cookies all sorts of ways with no such luck. The request cookie header is there when I go the the website.
Am I missing something and the cookies are actually there? Or is it possible that the site does not use cookies to maintain a session? If that's the case, how would I access the page I want after logging in on my Android app?
Response Headers
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:23030
Content-Type:text/html; charset=utf-8
Date:Mon, 10 Aug 2015 23:03:26 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Keep-Alive:timeout=15, max=100
Pragma:no-cache
Server:Apache/2.2.22 (Debian)
Vary:Accept-Encoding
X-Powered-By:PHP/5.4.4-14+deb7u11
Request Header
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:52
Content-Type:application/x-www-form-urlencoded
Cookie:__utma=83554121.1278939357.1435860313.1435944069.1438202297.3; __utmc=83554121; __utmz=83554121.1438202297.3.3.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _ga=GA1.2.1278939357.1435860313; PHPSESSID=4q03j4ihb7trnm1pvvofc9f3f5
Host:WEBSITE
Origin:WEBSITE
Referer:WEBSITE
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36

Just an update on what I have figured out so far.
I logged in to my website on my browser on my PC. I took this PHPSESSID and used it in my app and I can access the page that the login page redirects to just fine.
My new question is: Why can't I assign myself a random PHPSESSID and go back to it? In other words, if I give myself PHPSESSID=123456 for example and use that when posting my login details, why does using that specific PHPSESSID still bring me back to the login page instead of the redirected one?
I'm currently trying to read the request header "set-cookie" but am having trouble doing so. In addition, even putting in the wrong user and/or password gives a response code of 200. Suggestions on how to check if logging in was successful?

Basic http authentication from browser to server via socket

I am trying to write a simple java server that receives HTTP GET requests from web browser and sends back some data. All the communication is done via sockets.
I am able to process the requests and now I am trying to implement a simple authentication with BASIC AUTH so some requests will be handled only if correct credentials are provided in the request header. For sake of simplicity, I am using only http protocol (not https). I am not sure how to access the header and read the credentials on my server, though:
The server runs on localhost, port 9000 and this is the sample URL that I am trying to process:
http://user:password#localhost:9000/files/text?tid=file3
I open the socket and read everything:
InputStream input = clientSocket.getInputStream();
// Reading line by line with a BufferedReader
java.io.BufferedReader in = new java.io.BufferedReader(new String line;
while (!(line = in.readLine()).equals("")) {
System.out.println(line);
}
This is what I get:
GET /files/text?tid=file3 HTTP/1.1
Host: localhost:9000
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/41.0.2272.76 Chrome/41.0.2272.76 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
There is no trace of the auth credentials I put in the URL so I am not sure how to parse the request. Could you tell me what am I missing here?
I know that this example is very simple and there are more clever ways to implement this but I am curious how to access these credentials in this simple model case.

Use a client like curl:
$ curl -v -u user:password "http://localhost:9000/files/text?tid=file3"
Since HTTP is stateless, sending the Authentication header is enough. That's what curl does. It is not necessary to wait for the server to return a 401.

How to get Cookies in the websocket connection?

GET / HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:1255
Origin: http://testt:8080
Sec-WebSocket-Protocol: json
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: sGGDklOmMNFmY2AniKkkGw==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
this is what I've get during the handshake of websocket, but how could I get from here the cookie id?, I've thinked to pass the cookie after the handshake is done, but will that be correct?
where are the cookies hidden?
its connecting to my java socket, what class I can use in this case to get the cookie? or it would be better if I get it from the string above...

The string above does not contain any cookie. The browser sends cookies using a "Cookie" HTTP header, which is not in that request.
If you want the browser to send the webpage cookie, you have to put the WebSocket server and the web server in the same domain, otherwise the cookie won't be send.
It is technically possible to return cookies in the negotiation HTTP response (in a "Set-Cookie" HTTP header), and it will be resend in the next connection.

How to identify Web request from PDA/Desktop/Server?

I would like to know if is possible to identify (with JAVA) the kind of computer used to make a request, for example: Server, desktop, PDA (tablet,cellphone,etc)?
Thank you!

Depends on what are you using to accept requests. For http requests, informations are in User agent section of request header.

Yes it is to a degree. You have to get the User-Agent string from the HTTP request. How to do that will depend on your Java and framework implementation but that's the direction you should take. You will have to examin the string for browser versions, mobile, etc...
Here is the request from my Mac:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/536.30.1 (KHTML, like Gecko) Version/6.0.5 Safari/536.30.1
And here from my Windows server:
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
And here from my iPhone:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_3 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10B329 Safari/8536.25