I am trying to build a REST web server where GET requests are non-blocking, even if it needs to make a slightly time consuming call.
#RestController
public class Endpoint {
static int callCount = 0;
#RequestMapping (value = "/endpoints", method = RequestMethod.GET)
public DeferredResult<Integer> someGetMethod() {
System.out.println("Mita: GET Called. Count#: " + callCount++);
DeferredResult<Integer> deferredResult = new DeferredResult<>();
new Thread( () -> {
deferredResult.setResult(getSomething());
}).start();
System.out.println("Mita: Thread started");
return deferredResult;
}
private int getSomething() {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
return 100;
}
}
To test, I sent two GET requests from Restlet, to the same endpoint, in intervals of around 2 seconds. Yet, I see the following in DEBUG level logs of my spring application.
2017-08-26 01:16:38.231 DEBUG 1252 --- [nio-8080-exec-1] o.a.coyote.http11.Http11InputBuffer : Received [GET /endpoints/ HTTP/1.1
Host: localhost:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Accept: */*
DNT: 1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8,es;q=0.6,zh-CN;q=0.4,zh;q=0.2
]
...
...
2017-08-26 01:16:43.399 DEBUG 1252 --- [nio-8080-exec-2] o.a.coyote.http11.Http11InputBuffer : Received [GET /endpoints/ HTTP/1.1
Host: localhost:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Accept: */*
DNT: 1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8,es;q=0.6,zh-CN;q=0.4,zh;q=0.2
]
Notice, how the two requests came right after 5 seconds, the exact interval I have called sleep for (even though I had sent the request ~2 seconds after the first one). So, it seems like tomcat is sequentializing the incoming requests. How can I make tomcat not to sequentialize the requests? Or is it that I am missing something very obvious.
Using DeferredResult (or Servlet asynch mode that it uses) wont prevent the client (in this case Restlet) not to wait 5 seconds before it gets response from the first request.
To the client calling your Spring endpoint will look just as if the Spring endpoint was not using asynch mode at all. This means that Restlet or Tomcat can potentially wait till first request is finished before serving second request to your endpoint.
Related
My goal
Im trying to scrap raw video stream data (.ts files) from twitch.tv using Selenium 4.
All live streams are fed in chunks of video,
I can access them manually by:
opening a chrome tab with a running twitch.tv livestream
open DevTools (F12)
go to Network tab > XHR
The stream of .ts (transport stream) files being fetched are my desired files.
I can just doubleclick on them and chrome downloads this small video chunk file.
I want to reproduce this using Selenium 4 but I have no experience with Web Programming (POST, Flow etc). My current programm
is able to scrap image files. But once the response received is of .ts file (XHR/Fetch) it returns.
DevToolsException: {"id":11,"error":{"code":-32000,"message":"No data found for resource with given identifier"},"sessionId":"79BA2C212FABA878DB3524D7D0F49BDC"}
I have tried
Calling Network.getResponseBody when the Network.loadingFinished event has fired but this also doesn't work. There is never the same requestID on either event.
Remarks: Im aware there is a Twitch API.
public static void main(String[] args) {
InitializeSeleniumDrivers();
driver.get("https://www.twitch.tv/thebausffs");
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.clearBrowserCache());
devTools.send(Network.setCacheDisabled(true));
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.of(100000000)));
devTools.addListener(Network.responseReceived(), responseReceived -> {
RequestId requestId = responseReceived.getRequestId();
try {
Command<Network.GetResponseBodyResponse> getBody = Network.getResponseBody(requestId);
Network.GetResponseBodyResponse response = devTools.send(getBody);
} catch (DevToolsException e) {
e.printStackTrace();
}
});
}
Headers Example
GENERAL
Request URL: https://video-edge-c55dd0.ams02.abs.hls.ttvnw.net/v1/segment/CrEFZRTkEBMVDg5w4Ygn2pwqXKLGK5NAUAQ7ZWHeCORCjjFxfh9McgTBm_DTCvfP1MrZIg1jb2-oo2769tLAjFKjUd4AQaKtV3LeTEpPJyB_7ZAgolK-dSlLAqnC1xaI7z6iJCC4W1fb5RkkJmLk2D5nYEpyA17gSqe1eoB5zYsrDnal6Sm__B5LhxzOwTPOKI66jxXeIThm8tpaFGabccyd8AcT7RIfqCRv9Jas-IMQCqnBLLpIjk5rC-n4USQzLI6R4xGeTyTwMgX3BQ7EcxB-X62kUvsJm2O7Q2iJEI-ongDyyFRCapzo8iBtGgN2ruxvp8SeCKHO8j9NbS4jymG276ZigtnDXEQbxa6f5i9dHEcf9g1ump4RZtd48eOv6bPsGCDhFfULRd8adcM369ew90NrzyYbImQZnhFcnyqvfYIlCg-FFyjqJHVz37MZGc7TLbSh1YqmrkAClamXb8fFPGCXpsIrY-IDmKgTxh8tEmjbdacBWsKxxwJAOv-H6MUZB67MP1KMeT94YMjGXBcIjJo4JKeFCKoITCLJI4jjzqNmFa_efdlaJ89mUodxQRHJARV3qwdp04TSvZALBbOua6m-0T-01lOEYlr6w408mr5araj7c7gjpvrj_83jb0wqJG7ala1DBUg0U0Vx2rQxzumokyz66MxfMJy3ZSY92L-JdS47RjcOpilnpTI9bI8RPRyY4grds2SHDudWxgp-jJWgHdtbbFpuDCZENwOuU_-Agsf0lA_g59KnXnAuz59yovCO2C_O8ptkyoImgZ47qBPBIn-DDD-rzJloGD-GTQn4zGlmAFcg6GunjeW3PbHjKjMz8vA_K8NOF7ofO94YOtj_1khbCFGfH2_dF8zDwMSieR5Mvg7upQdzwgl_GAmf7OIAbHXwA1DqamnbAeWundcaDEM8dWDJF-pfTicm0CABKglldS13ZXN0LTIwtwQ.ts
Request Method: GET
Status Code: 200 OK
Remote Address: 185.42.204.31:443
Referrer Policy: strict-origin-when-cross-origin
RESPONSE HEADER
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, private
Content-Length: 1589164
Content-Type: application/octet-stream
Date: Sun, 14 Aug 2022 16:56:31 GMT
REQUEST HEADER
Provisional headers are shown
Learn more
Referer
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36
I want to know how can okHTTP response from Web server (which returns json data) be cached?
I want my app to download all the data needed for RecycleView and cache it once the user runs the app first time - and avoid re-downloading and parsing all the same data from Web server, if data has not changed.
I tried to get response headers, but this is what I get:
Request URL: https://somedomain.com/wp-json/?categories=3&per_page=100&status=publish
Request Method: GET
Status Code: 200 OK
Remote Address: 176.28.12.139:443
Referrer Policy: no-referrer-when-downgrade
Access-Control-Allow-Headers: Authorization, Content-Type
Access-Control-Expose-Headers: X-WP-Total, X-WP-TotalPages
Allow: GET
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json; charset=UTF-8
Date: Fri, 23 Mar 2018 15:41:23 GMT
Link: <https://somedomain.com/wp-json/>; rel="https://api.w.org/"
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: PleskLin
X-Powered-By: PHP/7.1.15
X-Robots-Tag: noindex
X-WP-Total: 43
X-WP-TotalPages: 1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: hr,en-US;q=0.9,en;q=0.8,de;q=0.7,nb;q=0.6
Connection: keep-alive
Host: somedomain.com
Referer: https://somedomain.com/somedir/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
categories: 3
per_page: 100
status: publish
I have read that I could find ETAG or Last-Modified from the Server response to check for any changes, but as you can see there is no such thing.
Do you have any idea what to do if app only needs to download data at the first run - and after that only if the data has changed?
You need cache interceptor like this:
public class CacheInterceptor implements Interceptor {
#Override
public Response intercept(Chain chain) throws IOException {
Response response = chain.proceed(chain.request());
CacheControl cacheControl = new CacheControl.Builder()
.maxAge(15, TimeUnit.MINUTES) // 15 minutes cache
.build();
return response.newBuilder()
.removeHeader("Pragma")
.removeHeader("Cache-Control")
.header("Cache-Control", cacheControl.toString())
.build();
}
}
Add this interceptor with Cache to your OkHttpClient like this:
File httpCacheDirectory = new File(applicationContext.getCacheDir(), "http-cache");
int cacheSize = 10 * 1024 * 1024; // 10 MiB
Cache cache = new Cache(httpCacheDirectory, cacheSize);
OkHttpClient okHttpClient = new OkHttpClient.Builder()
.addNetworkInterceptor(new CacheInterceptor())
.cache(cache)
.build();
Just add a cache to your OkHttp client.
First, you'll need to create the cache:
int cacheSize = 10 * 1024 * 1024;
Cache cache = new Cache(cacheDirectory, cacheSize);
Note that with this code you're providing a cache with a size limit of 10MB. After this, you'll create your client with the following code:
OkHttpClient client = new OkHttpClient.Builder()
.cache(cache)
.build();
The client will honor cache related headers. You can relate to the official documentation for further details:
https://github.com/square/okhttp/wiki/Recipes
I want to use HttpOnly cookies and I set it in Java as follows:
...
Cookie accessTokenCookie = new Cookie("token", userToken);
accessTokenCookie.setHttpOnly(true);
accessTokenCookie.setSecure(true);
accessTokenCookie.setPath("/");
response.addCookie(accessTokenCookie);
Cookie refreshTokenCookie = new Cookie("refreshToken", refreshToken);
refreshTokenCookie.setHttpOnly(true);
refreshTokenCookie.setSecure(true);
refreshTokenCookie.setPath("/");
response.addCookie(refreshTokenCookie);
...
I got the client side the response with the cookies, but when I send the next request I do not have the cookies on the request. Maybe I miss something, but as I understood, these HttpOnly cookies has to be sent by the browser back on every request (JavaScript does not have access to those cookies) coming to the defined path.
I have the following Request Headers:
Accept:application/json, text/plain, */*
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,fr;q=0.2,de;q=0.2
Authorization:Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Connection:keep-alive
Content-Length:35
content-type:text/plain
Host:localhost:8080
Origin:http://localhost:4200
Referer:http://localhost:4200/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
X-Requested-With:XMLHttpRequest
and the following response headers:
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://localhost:4200
Access-Control-Expose-Headers:Access-Control-Allow-Origin, Content-Type, Date, Link, Server, X-Application-Context, X-Total-Count
Cache-Control:no-cache, no-store, max-age=0, must-revalidate
Content-Length:482
Content-Type:application/json;charset=ISO-8859-1
Date:Fri, 03 Feb 2017 13:11:29 GMT
Expires:0
Pragma:no-cache
Set-Cookie:token=eyJhbGciO;Max-Age=10000;path=/;Secure;HttpOnly
Set-Cookie:refreshToken=eyJhb8w;Max-Age=10000;path=/;Secure;HttpOnly
Vary:Origin
Also in the client side I use withCredentials: true in Angular2 and X-Requested-With:XMLHttpRequest as request header.
And it is Cross Domain.
Yes you are correct having the cookie your browser should send the cookie automatically while it is not expired and the httpOnly flag means it cannot be accessed or manipulated via JavaScript.
However
You need to ensure that the cookie you are sending is not cross domain, if you require it cross domain you will need to handle it differently.
I am receiving this POST request from a client:
HTTP method: POST
Host: 127.0.0.1:52400
Connection: keep-alive
Content-Length: 18
Pragma: no-cache
Cache-Control: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: null
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip,deflate
Accept-Language: da-DK,da;q=0.8,en-US;q=0.6,en;q=0.4,es;q=0.2
fname=foof&pw=bar
I have a small and very simple Java Webserver running, getting this request from InputStream.
From the BufferedReader I set data to a String, containing the request, like this:
for (String line; (line = in.readLine()) != null; ) {
if (line.isEmpty()) break;
header += line + "\n";
}
When I print header to the console, I get this:
POST / HTTP/1.1
Host: 127.0.0.1:52400
Connection: keep-alive
Content-Length: 18
Pragma: no-cache
Cache-Control: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: null
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip,deflate
Accept-Language: da-DK,da;q=0.8,en-US;q=0.6,en;q=0.4,es;q=0.2
The POST parameters are left out
I guess the problem occurs due to the blank line in the post-request.
How can I make sure the BufferedReader does read the request to the end, and not stopping at the blankline, all though stopping when the BufferedReader hits the end of the request.
Please ignore the lack of security in this example - I simply need to get the POST request into plain string representation for now.
Any help on this i appreciated, thanks!
Jesper.
The problem is because you have a break in your for loop. When you reach the blank line, it hits the break and exits the loop, not adding any of the lines after that. Instead, you should use this:
for (String line; (line = in.readLine()) != null; ) {
if (line.isEmpty()) continue;
header += line + "\n";
}
By using continue instead of break, the loop will simply proceed to the next iteration, and the rest of the lines can be added.
More information can be found here
Trying to read a webpage using HttpClient. But some of the html is hidden by some js magic, try hitting view source on this page http://uc.worldoftanks.eu/uc/accounts/#wot&at_search=a
Any idea how to get HttpClient to return the "full" html page?
HttpClient does not process javascript, which means there is no content that can be hidden when reading the http content from the server.
It's probably the other way round, the javascript that runs on the page likely creates new html elements and appends them to the DOM... which is not something you can handle using HttpClient, HttpClient is a communication client designed purely to read data accross a HTTP connection.
When that page loads, a request is being sent to
http://uc.worldoftanks.eu/uc/accounts/?type=table&offset=0&limit=25&order_by=name&search=a&echo=1&id=accounts_index
Try hitting that address up with your HttpClient to see the table data. Play with the offset, limit and order_by values to change pagination and sorting.
Manually browsing to said URL yields a redirect, though, so there appears to be some of the Request headers that you need to include in your HttpClient. The full headers of the request my browser issues, that does yield a JSON response with the table data, is as follows:
GET /uc/accounts/?type=table&offset=0&limit=25&order_by=name&search=&echo=1&id=accounts_index HTTP/1.1
Host: uc.worldoftanks.eu
Connection: keep-alive
Referer: http://uc.worldoftanks.eu/uc/accounts/?type=table&offset=0&limit=25&order_by=name&search=a&echo=1&id=accounts_index
X-Requested-With: XMLHttpRequest
X-CSRFToken: 5e33bf57602f76de9285e9b14bcfe7fe
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ar;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: csw_popup=true; __utma=21812543.1316104722.1312873581.1312873581.1312873581.1; __utmb=21812543.2.10.1312873581; __utmc=21812543; __utmz=21812543.1312873581.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); csrftoken=5e33bf57602f76de9285e9b14bcfe7fe
They might be looking for X-Requested-With or Accept or Referrer, for instance.