I am trying to use littleproxy-mitm to inspect traffic. I have access to the headers and can easily read them. However, I cant find the body of a response consitently. To see if I can get the body I am using this testing my app by visiting https://www.google.com/humans.txt, but the wanted body is no where to be found. But when I visit other sites like google, facebook and twitter I seem to get gibberish(encoded body gzip most prob) and sometimes html.
Here is the filter:
#Override
public HttpObject serverToProxyResponse(HttpObject httpObject) {
if(httpObject instanceof FullHttpResponse){
System.out.println("FullHttpResponse ----------------------------------------");
FullHttpResponse response = (FullHttpResponse) httpObject;
CompositeByteBuf contentBuf = (CompositeByteBuf) response.content();
String contentStr = contentBuf.toString(CharsetUtil.UTF_8);
System.out.println(contentStr);
}
return httpObject;
}
Any idea why I am unable to get body from https://www.google.com/humans.txt ?
To answer my own question.
This code snippet works and will print the whole response. But the reason I was not getting the body response is either the header "Modified-since.." or the "Cache-control: public".
Related
I am trying to build a Rest API automation based on Java, RestAssured, and Cucumber. I ' trying to hit an endpoint via POST. The problem is when I am converting the response as string and when I print the response, it is printing the XML file contents but not the response. I also see the status code as 200. I'm not sure what is going wrong in here. Below is the sample snippet from my codebase.
I am trying to hit a WebService (SOAP WSDL).
// required imports
public class Service_Steps {
Response xmlResponse;
#When("I create a POST request for RCP for endpoint using XML")
public void i_create_a_post_request_for_endpoint_using_xml() {
// xml Request body
String path = "pah_to_xml_file";
File file = new File(path);
xmlResponse = RestAssured.given().when().relaxedHTTPSValidation().accept(ContentType.XML).header(<headers>)
.body(file)
.post(url);
String xmlResponseAsString = xmlResponse.then().extract().body().asString();
}
Not sure why I am seeing this ambiguity. Sometimes it is printing the response, and sometimes it is printing the XML file (request body) contents.
After checking with developers I came to know that the SOAP EndPoint is sending out the responses in two different ways, randomly!
Try this one:
xmlResponse = RestAssured.given().log().all().when().relaxedHTTPSValidation().accept(ContentType.XML).header(<headers>)
.body(file)
.post(url)
.then()
.log().all()
.extract()
.response();
This will print out all the request & response stuff
I am giving request to OData POST in Json format and it's returning the same request to me.
POST URL= http://localhost:8085/MagicXpiOData/Odata_get.OData_1/Student_details
Body:
{"Division": "Nashik"}
Content-Type = application/json
Response:
{
"#odata.context": "$metadata#Student_details",
"Division": "Nashik"
}
Is this a correct response?
I think it should return a status code for success or not.
What you show above is the response body, which usually contains the OData.Context. Beside that you should find the HTTP Status Code in the header, which might be something like
HTTP/1.1 200 OK
HTTP/1.1 201 Created
and when you create a new entry also a location header.
Some examples that show typical requests and responses can be found in this Basic Tutorial
So generally your response looks ok. Maybe you can add the response header in your question, then we can see if everything is there what should be there.
When you send your request with a tool like Postman or Fiddler, you might by default only see the response body. To see the headers you need to switch to Headers, or Raw in Fiddler to see the full response (header and body).
I followed the simple example shown GitHub:LittleProxy and have added the following in clientToProxyRequest(HttpObject httpObject) Method.
public HttpResponse clientToProxyRequest(HttpObject httpObject)
{
if(httpObject instanceof DefaultHttpRequest)
{
DefaultHttpRequest httpRequest = (DefaultHttpRequest)httpObject;
logger.info(httpRequest.getUri());
logger.info(httpRequest.toString());
// How to access the POST Body data?
HttpPostRequestDecoder d = new HttpPostRequestDecoder(httpRequest);
d.getBodyHttpDatas(); //NotEnoughDataDecoderException
}
return null;
}
The logger report this, IMO only these two header are relevant here. It's a POST request and there is content ...
POST http://www.... HTTP/1.1
Content-Length: 522
Looking into Netty API documentation the HttpPostRequestDecoder seems to be promising, but I get a NotEnoughDataDecoderException. In Netty JavaDoc this is written, but I do not know how to offer data?
This getMethod returns a List of all HttpDatas from body.
If chunked, all chunks must have been offered using offer() getMethod. If not, NotEnoughDataDecoderException will be raised.
In fact I'm also unsure if this is the right approach to get the POST data in the proxy.
try to add this in your HttpFiltersSourceAdapter to aviod NotEnoughDataDecoderException:
#Override
public int getMaximumRequestBufferSizeInBytes() {
return 1048576;
}
1048576 here is the maximum length of the aggregated content. See POSTing data to netty with Apache HttpClient.
This will enables decompression and aggregation of content, see the source code in org.littleshoot.proxy.impl.ClientToProxyConnection:
// Enable aggregation for filtering if necessary
int numberOfBytesToBuffer = proxyServer.getFiltersSource()
.getMaximumRequestBufferSizeInBytes();
if (numberOfBytesToBuffer > 0) {
aggregateContentForFiltering(pipeline, numberOfBytesToBuffer);
}
I am working on RESTEasy services to generate API for my application.
I tested with the below code to produce a simple string response,
#GET
#Path("/api")
public Response getUsers(#QueryParam("from") String from,) throws ProtocolException,
MalformedURLException, IOException {
return Response.status(200)
.entity("*************Hi Welcome*********************")
.build();
}
It is working fine with the following url
http://localhost:8080/myApp/f/api?from=any_string_here
But, this response available only while the query parameter value does not exceed 6246 characters.
If the query parameter value more than 6246 chars, there is no response available. Also, the browser network console shows the status code 400.
http://localhost:8080/myApp/f/api?from=more_than_6246_chars
I read that longer url needs to be send using POST, so I tried also with #POST method too for this, but browser network console shows the status code 405 and the following appears in eclipse console.
Apr 07, 2016 12:52:25 PM org.apache.tomcat.util.http.Cookies processCookieHeader
INFO: Cookies: Invalid cookie. Value not a token or quoted value
Note: further occurrences of Cookie errors will be logged at DEBUG level.
Is this longer URL is restricted by browser or RESTEasy application.
What would be the solution for this? Do I need to send more chars to my rest api parameter.
Webservers may reject requests if the URL exceeds a certain size.
Using a POST request alone does not help, you also need to decrease the URL size by putting URL parameters into the POST body.
You can try sending the parameters in request headers. I am using Jersey framework and angular JS in the front end. Sometimes I need to send a long JSON string for my application. I am sending it in the request headers and so far, I haven't got any issue like this.
My Rest Service class looks like below :
#Path("getStatus/agentName")
public class getStatus(){
#GET
public Response getStatus(#HeaderParam("header_name") String header_value){
String response = "Success" + header_value;
return Response.ok(response, MediaType.TEXT_PLAIN).build();
}
}
You can send your parameters in custom headers.
I think this should solve your problem.
We're creating small project in which we have flask as a FrontEnd and restlet based WebService.
We try to send login data as a JSON from flask to restlet:
def login():
error = None
if request.method == 'POST':
payload = {'username' : request.form['username'], 'password' : request.form['password']}
headers = {'Content-Type': 'application/json'}
req = requests.post(WEBSERVICE_IP + '/login', data=json.dumps(payload), headers=headers)
(...)
Flask based website is shouting about:
ValueError: No JSON object could be decoded
We have no idea how to reconcile communication between flask and restlet.
Edit (22-04 10:08pm GMT):
I found out that responce was:
<html>
(...)
Unsupported Media Type
The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method
(...)
</html>
Edit (22-04 11:26pm GMT):
I'm still not sure why, but I supposed that it may be something with JSON format. After correcting my code so it will send correct JSON (said JSONLint), I still get the same messages. Anyone know how to create JSONObject in Python? WebService have method:
#Post("json")
public JSONObject verifyAccount(JSONObject dane){
Edit (23-04 7:26pm GMT):
Ok. So we're almost sure that it's problem with invisible header. Can anyone confirm that header creation in python code here is correct?
Edit (24-04 5:40pm GMT):
Problem is still actual. As some other suggested, I changed requests back to the urllib2. This helped with the first thing - "Value problem". Now browser is having
urllib2.HTTPError
HTTPError: HTTP Error 415: Unsupported Media Type
POST Request now:
#app.route('/login', methods=['GET', 'POST'])
def login():
error = None
if request.method == 'POST':
payload = {"Login": request.form['username'],
"Haslo": request.form['haslo']}
data = json.dumps(payload)
clen = len(data)
req = urllib2.Request(WEBSERVICE_IP + '/login', data,
{'Content-Type': 'application/json', 'Content-Length': clen})
f = urllib2.urlopen(req)
response = f.read()
f.close()
Edit (24-04 6:20pm GMT):
Wireshark captured POST request and it looks ok.
If data is a dictionary, Requests will serialize it. You want to pass a string instead:
import json
req = requests.post(WEBSERVICE_IP + '/login', data=json.dumps(payload), ...
Ok. Solution was easier than I would think.
Problem was on the WebService side. It was resolved by changing JSONObject to JsonRepresentation:
#Post("json")
public JSONObject verifyAccount(JsonRepresentation data){
In addition to Blender's point (which is a much more likely culprit), it's worth mentioning that content-type ought to be set to application/json rather than json.
To add on Hazards response, I had to edit #Post("json") to #Post("application/json"):
#Post("application/json")
public JSONObject verifyAccount(JsonRepresentation data){