I am trying to use littleproxy-mitm to inspect traffic. I have access to the headers and can easily read them. However, I cant find the body of a response consitently. To see if I can get the body I am using this testing my app by visiting https://www.google.com/humans.txt, but the wanted body is no where to be found. But when I visit other sites like google, facebook and twitter I seem to get gibberish(encoded body gzip most prob) and sometimes html.
Here is the filter:
#Override
public HttpObject serverToProxyResponse(HttpObject httpObject) {
if(httpObject instanceof FullHttpResponse){
System.out.println("FullHttpResponse ----------------------------------------");
FullHttpResponse response = (FullHttpResponse) httpObject;
CompositeByteBuf contentBuf = (CompositeByteBuf) response.content();
String contentStr = contentBuf.toString(CharsetUtil.UTF_8);
System.out.println(contentStr);
}
return httpObject;
}
Any idea why I am unable to get body from https://www.google.com/humans.txt ?
To answer my own question.
This code snippet works and will print the whole response. But the reason I was not getting the body response is either the header "Modified-since.." or the "Cache-control: public".
I'm migrating from the Apache HTTP legacy client to OkHttp and I'm having some problems finding equivalences between both. A couple of days ago I asked about credentials in this same subject and now I'm stuck again:
In the old implementation I have this:
TimeStampRequestGenerator reqGen = new TimeStampRequestGenerator();
reqGen.setCertReq(true);
MessageDigest digest = MessageDigest.getInstance("SHA256");
digest.update(myData);
TimeStampRequest request = reqGen.generate(TSPAlgorithms.SHA256, digest.digest(), BigInteger.valueOf(100));
byte[] enc_req = request.getEncoded();
myHttpPost.setEntity(new ByteArrayEntity(enc_req));
The most relevant line is the last one (as the others just build the request and, lucky enough, I won't need to change them), which adds the entity to the HttpPost.
Checking this answer it seems the entity of a request is
the majority of an HTTP request or response, consisting of some of the headers and the body, if present. It seems to be the entire request or response without the request or status line
But this definition confuses me as I can't find the equivalence to something with "headers and the body" in OkHttp. What I've tried:
MediaType textPlain = MediaType.parse("text/plain; charset=utf-8");
RequestBody requestBody = RequestBody.create(textPlain, request.getEncoded().toString());
Request myNewRequest = (new Request.Builder()).url(urlString).post(requestBody).build();
But it didn't work (I'm getting a 500 from the server). Does anyone know the correct equivalence?
I finally found the answer: I can use the TimeStampRequest encoded as I did before, without any modification. The change is, as I thought, only for the setEntity.
This is the request using OkHttp:
MediaType textPlain = MediaType.parse("binary");
RequestBody requestBody = RequestBody.create(textPlain, request.getEncoded());
Request myNewRequest = (new Request.Builder()).url(urlString).post(requestBody).build;
As you can see the only change from the previous code I tried is that I use binary as the MediaType, which make sense as we are sending a byte array (previously used ByteArrayEntity from the Apache client).
Hope it helps somebody.
everyone!
I've been building and app that consumes an API from the company where I work. When I perform a request (GET) with URL connection (through HttpURlConnection) with header (encoding, authorization and etc.) AND a body (as a JSON, using Content-Type = "application/json") the response code is 200 and the connection works pretty good, but and I use the same header in another request BUT WITHOUT a content, I get a Bad Request.
I already tried setting the doOutout = false, but It doesn't work.
I think that everything is OK with the URL and the header because I performed the request by Postman and I got the response.
Here is the way I'm creating the connection and connecting:
val uri = Uri.parse("Base URL").buildUpon().appendPath(path).build() val url = URL(uri.toString()) val connection = url.openConnection() as HttpURLConnection connection.apply {
addRequestProperty("Charset", Charsets.UTF_8.displayName())
addRequestProperty("Authorization", "Basic $token")
addRequestProperty("X-Serial", 5036.toString())
requestMethod = "GET"
connectTimeout = 0
readTimeout = 0
doOutput = false
if (responseCode != HTTP_OK) throws Exception("Erro")
}
Bad Request suggest that it's an issue with the way your request is formed.
It is likely a problem with one of the request headers that you are setting. Have a look at the accepted request headers here.
Charset should potentially be named Accept-Charset, and I haven't seen any examples of X-Serial being used.
Here is an example of setting up a GET request using HttpUrlConnection. You could try removing all headers apart from Authorization and setting the request method to GET, like in this example.
I found out what was the problem: Me!
The $token value was generate with the flag DEFAULT. After I realized that, I putted the flag NO_WRAP and my request worked.
My bad! But thanks for all support!
In my Ruby on Rails application I tried to upload an image through the POSTMAN REST client in Base64 format. When I POST the image I am getting a 406 Not Acceptable Response. When I checked my database, the image was there and was successfully saved.
What is the reason for this error, is there anything I need to specify in my header?
My request:
URL --- http://localhost:3000/exercises.json
Header:
Content-Type - application/json
Raw data:
{
"exercise": {
"subbodypart_ids": [
"1",
"2"
],
"name": "Exercise14"
},
"image_file_name": "Pressurebar Above.jpg",
"image":"******base64 Format*******"
}
Your operation did not fail.
Your backend service is saying that the response type it is returning is not provided in the Accept HTTP header in your Client request.
Ref: http://en.wikipedia.org/wiki/List_of_HTTP_header_fields
Find out the response (content type) returned by Service.
Provide this (content type) in your request Accept header.
http://en.wikipedia.org/wiki/HTTP_status_code -> 406
406 Not Acceptable
The resource identified by the request is only capable of generating response entities which have content characteristics not
acceptable according to the accept headers sent in the request.
406 happens when the server cannot respond with the accept-header specified in the request.
In your case it seems application/json for the response may not be acceptable to the server.
You mentioned you're using Ruby on Rails as a backend. You didn't post the code for the relevant method, but my guess is that it looks something like this:
def create
post = Post.create params[:post]
respond_to do |format|
format.json { render :json => post }
end
end
Change it to:
def create
post = Post.create params[:post])
render :json => post
end
And it will solve your problem. It worked for me :)
"Sometimes" this can mean that the server had an internal error, and wanted to respond with an error message (ex: 500 with JSON payload) but since the request headers didn't say it accepted JSON, it returns a 406 instead. Go figure. (in this case: spring boot webapp).
In which case, your operation did fail. But the failure message was obscured by another.
You can also receive a 406 response when invalid cookies are stored or referenced in the browser - for example, when running a Rails server in Dev mode locally.
If you happened to run two different projects on the same port, the browser might reference a cookie from a different localhost session.
This has happened to me...tripped me up for a minute. Looking in browser > Developer Mode > Network showed it.
const request = require('request');
const headers = {
'Accept': '*/*',
'User-Agent': 'request',
};
const options = {
url: "https://example.com/users/6",
headers: headers
};
request.get(options, (error, response, body) => {
console.log(response.body);
});
Changing header to Accept: */* resolved my issue and make sure you don't have any other Accept Header
In my case, I added:
Content-Type: application/x-www-form-urlencoded
solved my problem completely.
If you are using 'request.js' you might use the following:
var options = {
url: 'localhost',
method: 'GET',
headers:{
Accept: '*/*'
}
}
request(options, function (error, response, body) {
...
})
In my case for a API in .NET-Core, the api is set to work with XML (by default is set to response with JSON), so I add this annotation in my Controller :
[Produces("application/xml")]
public class MyController : ControllerBase {...}
Thank you for putting me on the path !
It could also be due to a firewall blocking the request. In my case the request payload contained string properties - "like %abc%" and ampersand symbol "&" - which caused the firewall to think it is a security risk (eg. a sql injection attack) and it blocked the request. Note here the request does not actually go to the server but is returned at the firewall level itself.
In my case, there were no application server logs generated so I knew that the request did not actually reach the server and was blocked before that. The logs that helped me were Web application firewall (WAF) logs.
In my java client application, I am accessing a endpoint URL and could able to get response back, but it is in HTML code!.
Method : Post
resource.accept(MediaType.APPLICATION_JSON_TYPE);
WebResource resource = Client.create().resource(
communicatorVO.getTargetURL());
String **response** = resource.queryParams(communicatorVO.getFormData()).type(MediaType.APPLICATION_JSON_TYPE).post(String.class, gson.toJson(communicatorVO.getRequestObject()));
The response object always contains HTML code! How to get the actual data?
If I try using chrome restful client, am getting below response.
{
"access_token" : "YOUR_NEW_ACCESS_TOKEN",
"token_type" : "bearer",
"expires_in" : 10800,
"refresh_token" : "YOUR_REFRESH_TOKEN",
"scope" : "write read offline_access"
}
This issue has been resolved.
I added type & accept in single line and it started returning expected json response. Now I can parse the json into any java object.
Code :
response = resource.type(MediaType.APPLICATION_FORM_URLENCODED_TYPE).accept(MediaType.APPLICATION_JSON_TYPE).post(ClientResponse.class, communicatorVO.getFormData());;