I am trying to write a simple java server that receives HTTP GET requests from web browser and sends back some data. All the communication is done via sockets.
I am able to process the requests and now I am trying to implement a simple authentication with BASIC AUTH so some requests will be handled only if correct credentials are provided in the request header. For sake of simplicity, I am using only http protocol (not https). I am not sure how to access the header and read the credentials on my server, though:
The server runs on localhost, port 9000 and this is the sample URL that I am trying to process:
http://user:password#localhost:9000/files/text?tid=file3
I open the socket and read everything:
InputStream input = clientSocket.getInputStream();
// Reading line by line with a BufferedReader
java.io.BufferedReader in = new java.io.BufferedReader(new String line;
while (!(line = in.readLine()).equals("")) {
System.out.println(line);
}
This is what I get:
GET /files/text?tid=file3 HTTP/1.1
Host: localhost:9000
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/41.0.2272.76 Chrome/41.0.2272.76 Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
There is no trace of the auth credentials I put in the URL so I am not sure how to parse the request. Could you tell me what am I missing here?
I know that this example is very simple and there are more clever ways to implement this but I am curious how to access these credentials in this simple model case.
Use a client like curl:
$ curl -v -u user:password "http://localhost:9000/files/text?tid=file3"
Since HTTP is stateless, sending the Authentication header is enough. That's what curl does. It is not necessary to wait for the server to return a 401.
Related
NOTE: This is not a duplicate question. I'm aware of concrete problems that looked identical to mine and were solved by adding some data to the header requests. This is not the case, I've tried all the solutions and none works. Tried: this question and this one, nothing seems to work.
I'm trying to read contents of a website using Java. The url is URL to fetch. There's no authentication involved, and no forms are filled before. I can open that url in a cookie-free session and it still works with browser. I can even fetch it with Selenium, but HttpClient keeps refusing to load the URL.
The problem has nothing to do with certificates, right now I'm using a working "allow-all" certificate manager, so that's not the issue.
I've inspected my browser sent headers, nothing special:
Host: www.hipercor.es
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
As I said, I've already tried configuring the user agent to "fake" being Firefox.
Just to give some background, I'm building a enhanced version of crawler4j, my idea is to build a web scraper, and I found this issue testing random shops I knew are currently being crawled in bussiness environment by other scrapers.
Note that HeadRequest also fail.
The errors are either
java.net.SocketException: Connection reset
or
java.net.SocketTimeoutException: Read timed out
Please, note that browser loads it perfectly, as well as using Selenium Drivers to load the page (although it is slow as hell in that case)
This problem seems to have been asked multiple times but for this particular case I haven't found a proper solution.
The application is a Spring Boot 1.5.x that uses a embedded Tomcat server and runs on Openshift. The latter has a Router with an HTTPS route that terminates the TLS tunnel and forwards the traffic over HTTP to the application pod. In addition it inserts the X-Forwarded- headers (including the X-Forwarded-Proto header) so that the application redirects are composed using the https protocol.
I have configured server.use-forward-headers: true in the Spring Boot Application and tested it:
1) OK -> https://ocproute/myapp/ redirects 302 to my home page keeping the https protocol (the Tomcat RemoteIpValve took care of it).
2) FAIL -> https://ocproute/myapp (note there's no trailing slash) redirects 302 to http://ocproute/myapp/ As you can see it changed the protocol to http since the RemoteIpValve hasn't been invoked yet).
The logs show that Tomcat's Http11InputBuffer receives the request and at some point redirects it without taking into account the X-Forwarded-Proto header.
How can this be fixed?
2019-06-03T17:31:59.230 ( - - - - - ) o.a.t.u.n.NioEndpoint DEBUG - Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper#1511311d:org.apache.tomcat.util.net.NioChannel#5209052:java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1:8080 remote=/0:0:0:0:0:0:0:1:64871]], Read direct from socket: [595]
2019-06-03T17:31:59.230 ( - - - - - ) o.a.c.h.Http11InputBuffer DEBUG - Received [GET /myapp HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
X-Forwarded-Proto: https
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,es;q=0.7,cy;q=0.6
Cookie: JSESSIONID=4693A1F63CD3E18058F98E129D11CE57
]
...
you can set this prop in your springboot config:
server.tomcat.use-relative-redirects=true
reference: https://github.com/spring-projects/spring-boot/issues/22908
To make this work I had to disable the context root redirection customizing the Tomcat Context:
#Configuration
class TomcatConfiguration : EmbeddedServletContainerCustomizer {
override fun customize(container: ConfigurableEmbeddedServletContainer) {
val factory = container as TomcatEmbeddedServletContainerFactory
factory.tomcatContextCustomizers = listOf(CustomCustomizer())
}
class CustomCustomizer : TomcatContextCustomizer {
override fun customize(context: Context) {
context.mapperContextRootRedirectEnabled = false
context.addServletContainerInitializer(WsSci(), null)
}
}
}
GET / HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:1255
Origin: http://testt:8080
Sec-WebSocket-Protocol: json
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: sGGDklOmMNFmY2AniKkkGw==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
this is what I've get during the handshake of websocket, but how could I get from here the cookie id?, I've thinked to pass the cookie after the handshake is done, but will that be correct?
where are the cookies hidden?
its connecting to my java socket, what class I can use in this case to get the cookie? or it would be better if I get it from the string above...
The string above does not contain any cookie. The browser sends cookies using a "Cookie" HTTP header, which is not in that request.
If you want the browser to send the webpage cookie, you have to put the WebSocket server and the web server in the same domain, otherwise the cookie won't be send.
It is technically possible to return cookies in the negotiation HTTP response (in a "Set-Cookie" HTTP header), and it will be resend in the next connection.
I'm developing a WebSocket server using Netty 4.0.21Final.
Before using Netty, I was sending data via socket directly and, now, I'm still doing the compression of the data by my self.
But, when looking closely at the HTTP Headers of my test client on Chrome browser, I saw this:
Request URL:ws://127.0.0.1:8089/echo
Request Method:GET
Status Code:101 Switching Protocols
Request Headers CAUTION: Provisional headers are shown.
Cache-Control:no-cache
Connection:Upgrade
Host:127.0.0.1:8089
Origin:null
Pragma:no-cache
Sec-WebSocket-Extensions:permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
Sec-WebSocket-Key:U0CPp11Bhqxp2lffj4tebw==
Sec-WebSocket-Version:13
Upgrade:websocket
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Response Headers
Connection:Upgrade
Sec-WebSocket-Accept:0vvWjhf27ScZauqx+jSfm/Xsuho=
Upgrade:websocket
The permessage-deflate on the Extensions section means that my messages are been compressed?
So, to try to answer this, I used a software called Wireshark and, when looking at the messages I could see that It was not compressed at all, they were all in plain text.
So, what do I must do to Netty really compress the messages for me?
Compression support was just added yesterday. So it is not included in any release yet:
https://github.com/netty/netty/commit/282d6e73b82ec943a739201f7be1985c45ef032b
In below code sample am getting the http method get is used as reQuest now want to know what are all the http methods my tomcat server support.Please help to find this.
GET / HTTP/1.0
Connection: Keep-Alive
User-Agent: Mozilla/4.6 (X11; I; Linux 2.2.6-15apmac ppc)
Host: zink.demon.co.uk:1126
Accept: image/gif, */*
There is no way to know this unless you ask the server which methods it supports.
Usually this is done with the OPTIONS HTTP method, but not all web servers support it. Also, it applies to a specific URL, not the whole server.