As I said in https://stackoverflow.com/q/35060446/487313 I'm working on a php proxy.Actually I have a servlet with the annotation:
#MultipartConfig
public class MyServlet extends HttpServlet {
Further in the doPost I have:
Collection<Part> lesParts = request.getParts();
And strangely I always get an empty Collection when coming through my proxy and the right thing when comming directly from my form
That is to say:
<form name="mon-formulaire1" id="proxyform" action="newPrxi.php?proxy_url=http://localhost:8888/testServlet/myservlet.html" method="post" enctype="multipart/form-data" > doesn't work
and
<form name="mon-formulaire1" id="proxyform" action="http://localhost:8888/testServlet/myservlet.html" method="post" enctype="multipart/form-data" > is ok.
Actually I'm working on my localhost with php on Vstudio(port 30645) and the servlet on Tomcat 7 under Eclipse (port 8888).
I used request.getInputStream(); and iterating through the headers with Enumeration headerNames = request.getHeaderNames(); AND I can't see no difference between the case which is ok (direct from form) and the one which is ng (through the proxy). All this code is normally commented to avoid to have an empty request after reading the stream....
My form got two input text and two file input elements, it doesn't change anything when I uplpoad a file or not (the file is well saved when the part list is not empty).
My headers are the followings:
Key: content-type, Value: multipart/form-data; boundary=----WebKitFormBoundaryHEJ3XuT0gdC8RG4X
Key: connection, Value: keep-alive
Key: accept-language, Value: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Key: host, Value: localhost:8888
Key: accept, Value: text/plain, */*; q=0.01
Key: content-length, Value: 105639
Key: origin, Value: http://localhost:30645
Key: user-agent, Value: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36
Key: accept-encoding, Value: gzip, deflate
Key: referer, Value: http://localhost:30645/index.html?XDEBUG_SESSION_START=76F0F204
ANy help would be more than welcome.... :)
If the servlet cannot return the parts of a multipart request when running behind a proxy, the most likely explanation is that the proxy added or removed stuff which makes the request incorrect or unparsable.
As detected in the comment conversation this was actually the case.
Related
My goal
Im trying to scrap raw video stream data (.ts files) from twitch.tv using Selenium 4.
All live streams are fed in chunks of video,
I can access them manually by:
opening a chrome tab with a running twitch.tv livestream
open DevTools (F12)
go to Network tab > XHR
The stream of .ts (transport stream) files being fetched are my desired files.
I can just doubleclick on them and chrome downloads this small video chunk file.
I want to reproduce this using Selenium 4 but I have no experience with Web Programming (POST, Flow etc). My current programm
is able to scrap image files. But once the response received is of .ts file (XHR/Fetch) it returns.
DevToolsException: {"id":11,"error":{"code":-32000,"message":"No data found for resource with given identifier"},"sessionId":"79BA2C212FABA878DB3524D7D0F49BDC"}
I have tried
Calling Network.getResponseBody when the Network.loadingFinished event has fired but this also doesn't work. There is never the same requestID on either event.
Remarks: Im aware there is a Twitch API.
public static void main(String[] args) {
InitializeSeleniumDrivers();
driver.get("https://www.twitch.tv/thebausffs");
DevTools devTools = ((ChromeDriver) driver).getDevTools();
devTools.createSession();
devTools.send(Network.clearBrowserCache());
devTools.send(Network.setCacheDisabled(true));
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.of(100000000)));
devTools.addListener(Network.responseReceived(), responseReceived -> {
RequestId requestId = responseReceived.getRequestId();
try {
Command<Network.GetResponseBodyResponse> getBody = Network.getResponseBody(requestId);
Network.GetResponseBodyResponse response = devTools.send(getBody);
} catch (DevToolsException e) {
e.printStackTrace();
}
});
}
Headers Example
GENERAL
Request URL: https://video-edge-c55dd0.ams02.abs.hls.ttvnw.net/v1/segment/CrEFZRTkEBMVDg5w4Ygn2pwqXKLGK5NAUAQ7ZWHeCORCjjFxfh9McgTBm_DTCvfP1MrZIg1jb2-oo2769tLAjFKjUd4AQaKtV3LeTEpPJyB_7ZAgolK-dSlLAqnC1xaI7z6iJCC4W1fb5RkkJmLk2D5nYEpyA17gSqe1eoB5zYsrDnal6Sm__B5LhxzOwTPOKI66jxXeIThm8tpaFGabccyd8AcT7RIfqCRv9Jas-IMQCqnBLLpIjk5rC-n4USQzLI6R4xGeTyTwMgX3BQ7EcxB-X62kUvsJm2O7Q2iJEI-ongDyyFRCapzo8iBtGgN2ruxvp8SeCKHO8j9NbS4jymG276ZigtnDXEQbxa6f5i9dHEcf9g1ump4RZtd48eOv6bPsGCDhFfULRd8adcM369ew90NrzyYbImQZnhFcnyqvfYIlCg-FFyjqJHVz37MZGc7TLbSh1YqmrkAClamXb8fFPGCXpsIrY-IDmKgTxh8tEmjbdacBWsKxxwJAOv-H6MUZB67MP1KMeT94YMjGXBcIjJo4JKeFCKoITCLJI4jjzqNmFa_efdlaJ89mUodxQRHJARV3qwdp04TSvZALBbOua6m-0T-01lOEYlr6w408mr5araj7c7gjpvrj_83jb0wqJG7ala1DBUg0U0Vx2rQxzumokyz66MxfMJy3ZSY92L-JdS47RjcOpilnpTI9bI8RPRyY4grds2SHDudWxgp-jJWgHdtbbFpuDCZENwOuU_-Agsf0lA_g59KnXnAuz59yovCO2C_O8ptkyoImgZ47qBPBIn-DDD-rzJloGD-GTQn4zGlmAFcg6GunjeW3PbHjKjMz8vA_K8NOF7ofO94YOtj_1khbCFGfH2_dF8zDwMSieR5Mvg7upQdzwgl_GAmf7OIAbHXwA1DqamnbAeWundcaDEM8dWDJF-pfTicm0CABKglldS13ZXN0LTIwtwQ.ts
Request Method: GET
Status Code: 200 OK
Remote Address: 185.42.204.31:443
Referrer Policy: strict-origin-when-cross-origin
RESPONSE HEADER
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, private
Content-Length: 1589164
Content-Type: application/octet-stream
Date: Sun, 14 Aug 2022 16:56:31 GMT
REQUEST HEADER
Provisional headers are shown
Learn more
Referer
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36
I am trying to implement the "Trouble Signing In" link endpoint on my back end. I followed the steps enumerated here, and I am successfully getting a POST request via my Gitkit Widget JavaScript, but there seems to be some missing information in the request.
As soon as I call
OobResponse oobResponse = Utils.getGitkitClient().getOobResponse(request);
the following stack trace is thrown. I'll only include the first few lines for the sake of brevity.
com.google.identitytoolkit.GitkitServerException: org.json.JSONException: JSONObject["oobCode"] not found.
at com.google.identitytoolkit.GitkitClient.getOobResponse(GitkitClient.java:411)
at com.google.identitytoolkit.GitkitClient.getOobResponse(GitkitClient.java:372)
at myProject.myController.sendEmail(myController.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
...
Just so you can rule this out as a possible cause of the issue, here are the contents of the utility method I use to get my GitkitClient instance:
public static GitkitClient getGitkitClient() throws FileNotFoundException {
GitkitClient gitkitClient = GitkitClient
.newBuilder()
.setGoogleClientId(Constants.CLIENT_ID)
.setServiceAccountEmail(Constants.SERVICE_ACCOUNT_EMAIL)
.setKeyStream(new FileInputStream(Constants.SERVICE_ACCOUNT_PRIVATE_KEY_FILE_PATH))
.setWidgetUrl(Constants.WIDGET_URL)
.setCookieName(Constants.COOKIE_NAME)
.build();
return gitkitClient;
}
Here is my sendEmail endpoint method signature (I'm using Spring Boot):
#RequestMapping(value = "/sendEmail", method = RequestMethod.POST)
#ResponseBody
public void sendEmail(HttpServletRequest request, HttpServletResponse response) {
And finally, here are the contents of the HttpServletRequest that is passed into my sendEmail method:
Request Method
POST
Request Headers
Header name: host
Header value: example.com:8080
Header name: connection
Header value: keep-alive
Header name: content-length
Header value: 1100
Header name: origin
Header value: http://example.com:8080
Header name: user-agent
Header value: Mozilla/5.0 (Linux; Android 5.0.2; HTC6500LVW Build/LRX22G; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/46.0.2490.76 Mobile Safari/537.36
Header name: content-type
Header value: application/x-www-form-urlencoded;charset=UTF-8
Header name: accept
Header value: */*
Header name: referer
Header value: http://example.com:8080/myProject/callback?mode=recoverPassword&identifier=test#test.com
Header name: accept-encoding
Header value: gzip, deflate
Header name: accept-language
Header value: en-US
Header name: x-requested-with
Header value: [MY APP PACKAGE NAME OMITTED]
Request Parameters
Parameter name: action
Parameter value: resetPassword
Parameter name: email
Parameter value: test#test.com
Parameter name: challenge
Parameter value:
Parameter name: response
Parameter value: 03AHJ_Vuued2d7eKM-hD[... CONTENTS OMITTED ...]G1mzdsuc8
Notice that the request parameter "challenge" is the empty string, and that there is no mention of "oobCode," as the stack trace so aptly points out.
Am I missing something here? Do I need to take an extra step in my widget endpoing? Do I need to add something more to my HTML file that houses my JavaScript widget? Any help would be greatly appreciated!
After talking to someone working on identity toolkit in Google, I was told that there was a misconfiguration on their end in the Identity Toolkit project. After they made some changes I can now successfully call the aforementioned getOobResponse(HttpServletRequest req) method without getting a stack trace!
I'm trying to connect this website : https://ent.enteduc.fr/CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=1 with the following code :
Connection.Response response = Jsoup.connect("https://ent.enteduc.fr/CookieAuth.dll?GetLogon?curl=Z2F&flags=0&forcedownlevel=0&formdir=1&username=XXX&password=XXX&trusted=4&SubmitCreds.x=36&SubmitCreds.y=7&SubmitCreds=Ouvrir+une+session")
.method(Connection.Method.GET)
.execute();
Document Doc = Jsoup.connect("https://ent.enteduc.fr/CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=1")
.data("username","myusername")
.data("password","mypassword")
.data("curl","Z2F")
.data("flags","0")
.data("forcedownlevel","0")
.data("formdir","1")
.data("trusted","4")
.data("SubmitCreds.x","40") //Seems to send the coordinates of the cursor
.data("SubmitCreds.y","12") //Seems to send the coordinates of the cursor
.data("SubmitCreds","Ouvrir une session")
.cookies(response.cookies())
.post();
Log.e("Body", Doc.body().toString());
But The displayed "Body" is still the authentication page (No error in the Logcat)
What's wrong ?
Here are the details of the connection, get with the Chromes's Console
Remote Address:85.90.60.205:443
Request URL:https://ent.enteduc.fr/CookieAuth.dll?Logon
Request Method:POST
Status Code:302 Moved Temporarily
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:165
Content-Type:application/x-www-form-urlencoded
Cookie:ISAWPLB{FE9B5C07-18E7-4D86-BC7C-2F0AFE4F36BF}={8A3F320B-C8EB-40F9-A11E-D036A91F953F}; __utma=136247269.742318163.1408441429.1408445338.1408450626.3; __utmb=136247269.5.10.1408450626; __utmc=136247269; __utmz=136247269.1408441429.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); WSS_KeepSessionAuthenticated=; logondata=acc=0&lgn=*********
Host:ent.enteduc.fr
Origin:https://ent.enteduc.fr
Referer:https://ent.enteduc.fr/CookieAuth.dll?GetLogon?curl=Z2F&reason=0&formdir=1
User-Agent:Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36
Query String Parametersview sourceview URL encoded
Logon:
Form Dataview sourceview URL encoded
curl:Z2F
flags:0
forcedownlevel:0
formdir:1
username:myusername
password:mypass
trusted:4
SubmitCreds.x:53
SubmitCreds.y:12
SubmitCreds:Ouvrir une session
Response Headersview source
Connection:close
Content-Length:0
Location:https://ent.enteduc.fr/
Set-Cookie:cadata6A45CD714D774496A399F96AC521E21E....
There is nothing wrong with your code. It works. I tried the default user name and password you supplied. This is what the site does...
You login successfully and it sends a HTTP 302 to the path / and also gives you a cookie that identifies you.
HTTP/1.1 302 Moved Temporarily
Location: https://ent.enteduc.fr/
Set-Cookie: XXX
The browser requests for / and the server responds with another HTTP 302
HTTP/1.1 302 Found
Connection: Keep-Alive
Location: /etabs/0680001F/Pages/Accueil.aspx
Requesting for /etabs/0680001F/Pages/Accueil.aspx results in a 200 OK with HTML content written in french. Excusez moi ! Je ne parle pas francais.
Change your code to follow the redirects and set the cookies on each step and you should be fine.
[EDIT]
When you're done please remove the authentication info you supplied on this post.
I have a jquery script that sends data to my spring controller using PUT type. But controller never gets hit. If i change PUT to POST everything works as expected, but i need to use exactly PUT. Could you please review my code and tell me what i am doing wrong?
jQuery
var values = $("form").serialize();
$.ajax({
type: "PUT",
url: "/user/" + elogin.val(),
async: false,
data: values,
success: function(resp) {\\doStuff}
})
Controller
#Controller
#RequestMapping(value = "/user")
public class RestController {
#RequestMapping(value = "/{userLogin}", method = RequestMethod.PUT)
#ResponseBody
public boolean updateUser(#PathVariable String userLogin,
#RequestParam("epassword") String password,
...)
throws ParseException {
if (...) {
return false;
}
\\doStuff
return true;
}
}
FireBug error message
400 Bad Request
Response Headers
Connection close
Content-Length 1072
Content-Type text/html;charset=utf-8
Date Tue, 03 Sep 2013 10:21:28 GMT
Server Apache-Coyote/1.1
Request Headers
Accept */*
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection keep-alive
Content-Length 168
Content-Type application/x-www-form-urlencoded; charset=UTF-8
Cookie JSESSIONID=5A510B1FB82DA6F3DD9E9FA8D67A8295
Host localhost:8085
Referer http://localhost:8085/welcome
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0
X-Requested-With XMLHttpRequest
Error
HTTP Status 400 - Required String parameter 'epassword' is not present
type Status report
message Required String parameter 'epassword' is not present
description The request sent by the client was syntactically incorrect.
Solved by adding to web.xml
<filter>
<filter-name>HttpPutFormContentFilter</filter-name>
<filter-class>org.springframework.web.filter.HttpPutFormContentFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>HttpPutFormContentFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Hi I dont know am correct or not but I think we cannot use PUT for jquery ajax
type (default: 'GET')
Type: String
The type of request to make ("POST" or "GET"), default is "GET". Note: Other HTTP request methods, such as PUT and DELETE, can also be used here, but they are not supported by all browsers.
check this
http://api.jquery.com/jQuery.ajax/
Trying to read a webpage using HttpClient. But some of the html is hidden by some js magic, try hitting view source on this page http://uc.worldoftanks.eu/uc/accounts/#wot&at_search=a
Any idea how to get HttpClient to return the "full" html page?
HttpClient does not process javascript, which means there is no content that can be hidden when reading the http content from the server.
It's probably the other way round, the javascript that runs on the page likely creates new html elements and appends them to the DOM... which is not something you can handle using HttpClient, HttpClient is a communication client designed purely to read data accross a HTTP connection.
When that page loads, a request is being sent to
http://uc.worldoftanks.eu/uc/accounts/?type=table&offset=0&limit=25&order_by=name&search=a&echo=1&id=accounts_index
Try hitting that address up with your HttpClient to see the table data. Play with the offset, limit and order_by values to change pagination and sorting.
Manually browsing to said URL yields a redirect, though, so there appears to be some of the Request headers that you need to include in your HttpClient. The full headers of the request my browser issues, that does yield a JSON response with the table data, is as follows:
GET /uc/accounts/?type=table&offset=0&limit=25&order_by=name&search=&echo=1&id=accounts_index HTTP/1.1
Host: uc.worldoftanks.eu
Connection: keep-alive
Referer: http://uc.worldoftanks.eu/uc/accounts/?type=table&offset=0&limit=25&order_by=name&search=a&echo=1&id=accounts_index
X-Requested-With: XMLHttpRequest
X-CSRFToken: 5e33bf57602f76de9285e9b14bcfe7fe
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.107 Safari/535.1
Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ar;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: csw_popup=true; __utma=21812543.1316104722.1312873581.1312873581.1312873581.1; __utmb=21812543.2.10.1312873581; __utmc=21812543; __utmz=21812543.1312873581.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); csrftoken=5e33bf57602f76de9285e9b14bcfe7fe
They might be looking for X-Requested-With or Accept or Referrer, for instance.