Develop Reference

Java, Get response via socket

I know that I had to search on Google, Youtube and Stackoverflow before I ask my question on Stackoverflow .
But I swear I watched many videos on Youtube, And I searched for many ways on google and Stackoverflow ! to solve my problem. ,
I can't get the full response,
And now i'm stuck with this,

Facebook checks for browser:
Add agent string to request:
String content1 = "GET /zuck HTTP/1.1\r\nHost: www.facebook.com\r\nuser-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36\r\n\r\n";

Java HTTP GET timeouts but works fine in browser

NOTE: This is not a duplicate question. I'm aware of concrete problems that looked identical to mine and were solved by adding some data to the header requests. This is not the case, I've tried all the solutions and none works. Tried: this question and this one, nothing seems to work.
I'm trying to read contents of a website using Java. The url is URL to fetch. There's no authentication involved, and no forms are filled before. I can open that url in a cookie-free session and it still works with browser. I can even fetch it with Selenium, but HttpClient keeps refusing to load the URL.
The problem has nothing to do with certificates, right now I'm using a working "allow-all" certificate manager, so that's not the issue.
I've inspected my browser sent headers, nothing special:
Host: www.hipercor.es
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
As I said, I've already tried configuring the user agent to "fake" being Firefox.
Just to give some background, I'm building a enhanced version of crawler4j, my idea is to build a web scraper, and I found this issue testing random shops I knew are currently being crawled in bussiness environment by other scrapers.
Note that HeadRequest also fail.
The errors are either
java.net.SocketException: Connection reset
or
java.net.SocketTimeoutException: Read timed out
Please, note that browser loads it perfectly, as well as using Selenium Drivers to load the page (although it is slow as hell in that case)

error 503 with gwt project

I'm trying to run a GWT project into DevMode, After running google compilation and lanching the devMode i'm getting a url, so when running into mozila, i'm getting the 503 error.
my console is displaying the following :
[ERROR] 503 - GET /travelApp?gwt.codesvr=127.0.0.1:49822 (127.0.0.1) 1296 bytes
Request headers
Host: localhost:8888
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Response headers
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=ISO-8859-1
Content-Length: 1296
i don't understand the problem, can you help me with that ?

What you see is the response from the webserver. The Error-Code is 503 - Service Unavailable (https://en.wikipedia.org/wiki/List_of_HTTP_status_codes).
Most likey, your application failed to start.
Do you have any logs? There might be any usefull stacktraces.
Do you use the build-in Jetty? I found out, that it's sometimes hard to get log entries from it. Maybe you can try to deploy the application to tomcat to get a more helpful errorlog.

Spring boot http response compression doesn't work for some User-Agents

I'm trying to enable http response compression on Spring boot web application.
It works for some user-agents, and for some reason doesn't for others (specific cases below).
My basic question is:
Why http response compression (gzip) in Spring Boot works only for some User-Agent headers and where it is configured.
Spring boot reference doesn't say anything about it.
I prepared simple web application with enabled compression: sample spring-boot-compression app
There are integration tests that verify that gzip encoding works for some cases only.
I configured spring boot with:
server:
tomcat:
compression: on
compressable-mime-types: text/html,text/css,application/javascript,application/json,application/font-sfnt,application/font-woff,application/font-woff2
When I try to do some requests with curl:
$ curl -i -H "Accept-Encoding: gzip,deflate" http://localhost:8080
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Vary: Accept-Encoding
I see that Content-Encoding: gzip header is set.
When I setUser-Agent to AppleWebKit (or some other browsers like IE) it does not compress:
$ curl -i -H "Accept-Encoding: gzip,deflate" -H "User-Agent: AppleWebKit" http://localhost:8080
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Vary: Accept-Encoding
I repeated my tests with some other browsers and User-Agent header modification and received some strange results.
Here some of working (response is compressed) User-Agent headers:
- Mozilla/5.0
- Mozilla/5.0 (Windows NT 6.1; WOW64) Chrome/46.0.2490.80 Safari/537.36
- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Firefox/34.0
- SomeUnknownBrowser
Some of not working User-Agent headers:
- AppleWebKit
- Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
- Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
- Gecko/20100101
I also tried to use compression with GzipFilter and it behaves exactly the same. Also tried embedded Jetty instead of Tomcat - same result.
Maybe I'm just missing something.

You're probably using some sort of antivirus (maybe ESET). Try turning off HTTP protection.
With Eset you can try something like: Advanced -> Internet and email -> Web Access Protection - turn off.

This worked for me with Spring 1.4
server.compression.enabled: true
server.compression.mime-types: application/json,application/xml,text/html,text/xml,text/plain,text/css,application/javascript

How to get Cookies in the websocket connection?

GET / HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: localhost:1255
Origin: http://testt:8080
Sec-WebSocket-Protocol: json
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: sGGDklOmMNFmY2AniKkkGw==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36
this is what I've get during the handshake of websocket, but how could I get from here the cookie id?, I've thinked to pass the cookie after the handshake is done, but will that be correct?
where are the cookies hidden?
its connecting to my java socket, what class I can use in this case to get the cookie? or it would be better if I get it from the string above...

The string above does not contain any cookie. The browser sends cookies using a "Cookie" HTTP header, which is not in that request.
If you want the browser to send the webpage cookie, you have to put the WebSocket server and the web server in the same domain, otherwise the cookie won't be send.
It is technically possible to return cookies in the negotiation HTTP response (in a "Set-Cookie" HTTP header), and it will be resend in the next connection.