I am using Restlet in java as a simple client to make calls to a RESTful service provided by tastypie (python django)
I am monitoring the payload through tcpmon and I am seeing the following payload get returned however i can't get at the data in the payload.
Client calls
ClientResource resource = new ClientResource("http://localhost/someplace/");
String rep = resource.get(String.class);
Payload
HTTP/1.0 200 OK
Date: Thu, 12 Jul 2012 20:22:12 GMT
Server: WSGIServer/0.1 Python/2.7.1
Vary: Cookie
Content-Type: application/json; charset=utf-8
Set-Cookie: sessionid=63c5ea23113073e489cb8920819f37d; expires=Thu, 26-Jul-2012 20:22:12 GMT; httponly; Max-Age=1209600; Path=/
{"jsonData": "someData"}
When i debug into restlet i notice that the InboundWay.createEntity(..) is setting the data to EmptyRepresentation due to the lack of length, chunkEncoding, or connection headers. Why is this? Why can't I just stream the data?
Does any one know why this is happening? Is there a better client? So far tried jersey and Resteasy with limited success. (jeresy has a bug where it is trying to mark and reset a autoCloseSteam, and resteasy was just a pain to use as a client) I was hoping not to have to write an HTTPClient.
Did a bit more digging it looks like RESTLET is doing something quite silly in my oppinion.
It is looking a the response headers and trying to find if it knows the size, or if its chunkedEncoding, or if the Connection:closed is specified. Otherwise it does NOT try to parse the payload ... Does anyone know why this is? I didn't realize that those headers are required in any way. Why can't we use the ClosingInputStream when the connection close is not specified...
Restlet Code
public Representation createInboundEntity(Series<Parameter> headers) {
Representation result = null;
long contentLength = HeaderUtils.getContentLength(headers);
boolean chunkedEncoding = HeaderUtils.isChunkedEncoding(headers);
// In some cases there is an entity without a content-length header
boolean connectionClosed = HeaderUtils.isConnectionClose(headers);
// Create the representation
if ((contentLength != Representation.UNKNOWN_SIZE && contentLength != 0)
|| chunkedEncoding || connectionClosed) {
InputStream inboundEntityStream = getInboundEntityStream(
contentLength, chunkedEncoding);
ReadableByteChannel inboundEntityChannel = getInboundEntityChannel(
contentLength, chunkedEncoding);
...
Related
I am new to Flutter and I am trying to call my ASP.NET server web API.
From the logs on my server, everything goes fine but Android Studio throws an exception: "invalid header field name".
Here is the code in dart:
import 'package:http/http.dart' as http;
...
_getService() async {
String result;
try {
var url = 'http://192.168.1.14:34263/api/Mobile/test/1';
Future<http.Response> response = http.get( url );
result = response.toString();
} catch(exception){
result = exception.toString();
debugPrint(result);
}
...
}
Here is the response header (obtained via Chrome):
Access-Control-Allow-Headers:accept, authorization, Content-Type
Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, PATCH, DELETE
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: WWW-Authenticate
Cache-Control: no-cache
Content-Encoding: deflate
Content-Length:79
Content-Type: application/xml; charset=utf-8
Date: Thu, 08 Mar 2018 01:01:25 GMT
Expires:-1
Pragma:no-cache
Server:MyTestServer
X-Content-Type-Options:NOSNIFF
X-Permitted-Cross-Domain-Policies:master-only
X-SourceFiles:=?UTF-8?BDpcTXlJbmNyZWRpYmxlRHJlc3NpbmdcTXlJbmNyZWRpYmxlRHJlc3NpbmdcTXlJbmNyZWRpYmxlRHJlc3NpbmdcYXBpXE1vYmlsZVxjb3Vjb3VcMQ==?=
X-XSS-Protection:1;mode=block
Here is the answer which is returned:
<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">test</string>
Can anyone tell me what am I doing wrong?
Many thanks
Ok, I finally found out, by debugging the code.
In fact, my server added a series of field names in the response's header (via the Web.config) and the last character of one of these field names was a space.
As a result, the http_parser.dart threw an exception since spaces are no authorized characters in header field name.
Nothing was detected by Chrome (or any browser) nor by Postman.
I had similar problem and after some heavy debugging
I removed these headers from nginx:
#add_header X−Content−Type−Options nosniff;
#add_header X−Frame−Options SAMEORIGIN;
#add_header X−XSS−Protection 1;
and it works fine. So most likely it's backend - header related issue
I have a problem with Google HTTP Client Library for Java (1.22.0).
This is my code
String url = "http://gazetapraca.pl/ogl/2502758";
GenericUrl genericUrl = new GenericUrl(url);
ApacheHttpTransport apacheHttpTransport = new ApacheHttpTransport();
HttpRequest httpRequest = apacheHttpTransport.createRequestFactory().buildGetRequest(genericUrl);
httpRequest.setFollowRedirects(true);
HttpResponse httpResponse = httpRequest.execute();
and httpRequest.execute() throws
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
Below is follow from Wireshark
GET /ogl/2502758 HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Set-Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597; path=/; HttpOnly
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
GET /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie HTTP/1.1
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.22.0 (gzip)
Host: gazetapraca.pl
Connection: Keep-Alive
Cookie: JSESSIONID_JOBS=2f1TffY6JYcb6zvBSrQ72fds7rfdsSnHM3sefw6D31Lfr434bnkDmdLQJLvLFZ6zkYBF!-12116034235597
HTTP/1.1 301 Moved Permanently
Date: Sat, 26 Nov 2016 22:15:52 GMT
Server: Apache
Location: /ogl/2502758/pakowacz+-+mile+widziane+panie
Content-Length: 0
Content-Language: pl
P3P: CP="NOI DSP COR NID PSAo OUR IND"
Vary: User-Agent
Keep-Alive: timeout=2, max=99
Connection: Keep-Alive
and repeat a few times.
Maybe the problem is with url, because location is /ogl/2502758/pakowacz+-+mile+widziane+panie and next request method get is /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie.
In other software and library everything is working (google chrome browser, postman - addon to chrome, JSOUP - java library).
Does anyone have an idea how to solve the problem?
This is not your library's fault.
To understand why this problem is occurring, we must first understand the "error" message associated with your problem:
com.google.api.client.http.HttpResponseException: 301 Moved Permanently
So, what does this mean? Well, the last part of the error message, the description says "301 Moved Permanently". What that is referring to is an HTTP Status Code. An HTTP Status Code indicates what the outcome of a specific request is. In this case, the status code was 301, which according to RFC protocol means:
The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs.
So, this means that the URL that you are using is no longer valid, and that you have to use the new URL given to you by the Location response header. Now, it seems that the library that you're using is smart enough to detect this, and initializes a new request to the new URL. That's great and all, but your library that you are using, is incorrectly escaping the url provided by the Location header, and using that for the new request (turning /ogl/2502758/pakowacz+-+mile+widziane+panie into /ogl/2502758/pakowacz%20-%20mile%20widziane%20panie), and the server receiving this request recognizes that those to paths are not the same (even though they should be. So, the server sends another 301 response, telling the client (the library in this case) to use the un-escaped URL instead of the escaped one, even though they should be the same.
Now, why is your library doing this? It turns out that, according to RFC spec, the '+' character is reserved for URIs. That means that that character, along with other characters are only intended for use in URIs for their intended purpose. Therefore, it is not standard to include the '+' character in URIs, unless it is used for a very specific purpose, which it looks like is not the case.
So, this all means that you cannot blame the library for this error, you can only blame the people who developed this site.
The reason that this works in your browser and other places is because those clients do not seem to be escaping the requested URL for you before sending it to the server.
I got a lot of 405 errors because I haven't support HEAD requests yet. Now while I try to fix this issue, the following question come to my head:
a) Is there an easier way to support HEAD for GET-URLs/resources?
http://en.wikipedia.org/wiki/List_of_HTTP_header_fields
#RequestMapping(value = "/**/p/{productCode}", method = RequestMethod.GET)
public String productDetail(#PathVariable("productCode"), final Model model, final HttpServletRequest request) {
// some stuff...
return ControllerConstants.GET_PRODUCT_DETAIL;
}
or do I have to put a mapping for every Method?
#RequestMapping(value = "/**/p/{productCode}", method = RequestMethod.HEAD)
public String productDetailHead() {
final HttpHeaders headers = new HttpHeaders();
return new ResponseEntity(null, headers, HttpStatus.OK);
}
b) What HTTP header attributes for HEAD should be supported? (rule of thumb?)
My actually reponse with:
curl -I http://localhost:9001
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: _seaT.tenantID_=""; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/
Set-Cookie: JSESSIONID=D07B464BBA02DC4148F00C5A08421B51; Path=/
Content-Length: 0
Date: Thu, 05 Dec 2013 13:41:42 GMT
Additional Info:
Webshop, STS 3.1, JDK 6, JSR 2.5
Thanks.
Answering you (b) question. As a rule of thumb all header attributes/directives that are passed during a GET method should also be passed during a HEAD method call, and that is what the HTTP standard says.
The HEAD method is identical to GET except that the server MUST NOT
return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification.
same question is here : Spring 3.0 HEAD Requests
shortly refer this : http://axelfontaine.com/blog/http-head.html
This is my first question. I love stackOverflow but I've always been a reader, not a writer. Now here goes nothing:
So I've seen a LOT of different questions like this, believe me, I looked... but none of them seemed to answer my question exactly. I'm designing a test harness to test an API, and it involves sending HTTP requests using HttpClient (in Java). For some of the requests, such as POST requests, a JSON string or an XML string must be sent with the request. (I'm only asking about JSON here, if anyone had the answer of how to do the XML as well, I'd love that, but I'll save it for another question.)
Anyways, I have the following code so far which seems like it SHOULD do what I want... but I'm getting 400 error. I'm going to assume that I've properly created an instance of an HttpClient and an HttpPost, applied appropriate headers, etc... the pertinent part is below:
JSONObject JSONpayload = new JSONObject();
JSONpayload.put("quantity", 1);
JSONpayload.put("sku", "21-53429");
String JSONstring = JSONpayload.toString();
System.out.println("JSON PAYLOAD BEING SENT: " + JSONstring);
request.setEntity(new StringEntity(JSONstring));
response = client.execute(request);
System.out.println("SERVER RESPONSE STRING: " + response.toString());
And I receive the following output:
JSON PAYLOAD BEING SENT: {"quantity":1,"sku":"21-53429"}
SERVER RESPONSE STRING: HTTP/1.1 401 Unauthorized
[Date: Thu, 27 Jun 2013 19:57:29 GMT,
Server: Mule Core/3.3.1,
Set-Cookie: sid=h8jumUyMxMztmB1AHtbvmUzzc9WchbiR9dQahD6Q; Version=1;
Domain=192.168.235.9;
Path=/,
http.status: 401,
X-MULE_SESSION: rO0ABXNyACNvcmcubXVsZS5zZXNzaW9uLkRlZmF1bHRNdWxlU2Vzc2lvbi7rdtEW7GGKAwAEWgAFdmFsaWRMAA1mbG93Q29uc3RydWN0dAAmTG9yZy9tdWxlL2FwaS9jb25zdHJ1Y3QvRmxvd0NvbnN0cnVjdDtMAAJpZHQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAD3NlY3VyaXR5Q29udGV4dHQAJ0xvcmcvbXVsZS9hcGkvc2VjdXJpdHkvU2VjdXJpdHlDb250ZXh0O3hwAXB0ACRjYzQwMmMwNy1kZjYzLTExZTItOGY1Yi1kOTk2MTM2MzZkZjJwc3IAJWphdmEudXRpbC5Db2xsZWN0aW9ucyRTeW5jaHJvbml6ZWRNYXAbc/kJS0s5ewMAAkwAAW10AA9MamF2YS91dGlsL01hcDtMAAVtdXRleHQAEkxqYXZhL2xhbmcvT2JqZWN0O3hwc3IAJG9yZy5tdWxlLnV0aWwuQ2FzZUluc2Vuc2l0aXZlSGFzaE1hcJ3R2e9nRc4AAwAAeHB3DD9AAAAAAAAQAAAAB3QABWR1bW15dAAYOU9WX0FBQUJHQmdBQUFFX0lTQVFBNGMydAAJUm91dGVkVVJMdABhaHR0cDovLzE5Mi4xNjguMjM1Ljk6MzA4MC9nc2kvcmVzdC9XRlMvVE1TTkEtVE1TVVMtU2l0ZS9yZXN0L2NhcnRzLzlPVl9BQUFCR0JnQUFBRV9JU0FRQTRjMi9pdGVtc3QAC0NvbnRleHRQYXRodAATZ3NpLWV3cy1zZXJ2aWNlL3N2Y3QAC291dHB1dC10eXBldAAQYXBwbGljYXRpb24vanNvbnQAClJlcXVlc3RVUkx0AEsvZ3NpLWV3cy1zZXJ2aWNlL3N2Yy92MS4wL3N0b3Jlcy9UTVNVUy9jYXJ0cy85T1ZfQUFBQkdCZ0FBQUVfSVNBUUE0YzIvaXRlbXN0AAdTdG9yZUlEdAAMc3RvcmVzL1RNU1VTdAAKaW5wdXQtdHlwZXEAfgATeHEAfgAJeHg=,
X-MULE_ENCODING: UTF-8,
Content-Type: text/plain,
Transfer-Encoding: chunked]
Now, I know the basic things and I'm pretty sure I've taken care of them... like setting the headers Accept application/json, Content-Type application/json, etc..
Any advice would be greatly appreciated. I'm new to using HttpClient and I'm a little lost haha. Thanks guys!
The problem is that this server requires authentication and you're not providing it (or using invalid credentials). I can tell that because the response code was 401 (Unauthorized).
401: Similar to 403 Forbidden, but specifically for use when authentication
is required and has failed or has not yet been provided [...]
HTTPClient's home page has a lot of information regarding authentication. You can check how to do that here also check some code examples here.
In my application I need to parse a website and save some data from ir to the database. I am using HttpClient to get the page content. My code looks like this:
HttpClient client = new DefaultHttpClient();
System.out.println(doc.getUrl());
HttpGet contentGet= new HttpGet(siteUrl + personUrl);
HttpResponse response = client.execute(contentGet);
String html = convertStreamToString(response.getEntity().getContent());
/*
parse the page
*/
/***********************************************************************/
public static String convertStreamToString(InputStream is) throws Exception {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
return sb.toString();
}
I am doing this in a loop - I try to get content of some pages (their structure is the same). Sometimes it works fine, but unfortunately, my response in many cases is a sequence of similar trash liek this:
�=�v7���9�Hdz$�d7/�$�st��؎I��X^�$A6t_D���!gr�����C^��k#��MQ�2�d�8�]
I
I don't know where is the problem, please help me.
I have displayed headers of all responses that I got. For correct ones, there are:
Server : nginx/1.0.13
Date : Sat, 23 Mar 2013 21:50:31 GMT
Content-Type : text/html; charset=utf-8
Transfer-Encoding : chunked
Connection : close
Vary : Accept-Encoding
Expires : Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control : no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma : no-cache
Set-Cookie : pfSC=1; path=/; domain=.profeo.pl
Set-Cookie : pfSCvp=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.profeo.pl
For incorrect ones:
Server : nginx/1.2.4
Date : Sat, 23 Mar 2013 21:50:33 GMT
Content-Type : text/html
Transfer-Encoding : chunked
Connection : close
Set-Cookie : pfSCvp=3cff2422fd8f9b6e57e858d3883f4eaf; path=/; domain=.profeo.pl
Content-Encoding : gzip
Any other suggestions? My guess is that this gzip encoding is a problem here, but what can I do about it?
This probably has to do with some websites using a different character encoding in their response than your JVM default. To convert from a raw byte stream, like those provided by InputStreams, to a character stream (or a String), you have to choose a character encoding. HTTP responses can use different encodings, but they'll typically tell you what encoding they're using. You could do this manually by finding the "Content-Encoding" header of the HttpResponse, but your library provides a utility for doing this, since it's a common need. It's found in the EntityUtils class, and you can use it like so:
String html = EntityUtils.toString(response.getEntity());
You'll have to add
import org.apache.http.util.EntityUtils;
to the top of your file for that to work.
If that doesn't help, another possibility is that some of the URLs you're retrieving are binary, not textual, in which case the things you're trying to do don't make sense. If that's the case, you can possibly try to distinguish between the textual responses and the binary responses by checking Content-Type header, like so:
boolean isTextual = response.getFirstHeader("Content-Type").getValue().startsWith("text");
NEW MATERIAL:
After looking at the HTTP headers you added to your question, my best guess is that this is being caused by gzip compression of the responses. You can find more info on how to deal with that in this question, but the short version is that you should try using ContentEncodingHttpClient instead of DefaultHttpClient.
Another edit: ContentEncodingHttpClient is now deprecated, and you're supposed to use DecompressingHttpClient instead.
You need a httpclient which don't use compression.
I use this HttpClientBuilder.create().disableContentCompression().build() httpclient