I am currently implementing an HTTP server in Java but faced one problem when it comes to transfer encoding.
While
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Encoding: gzip
Transfer-Encoding: chunked
works properly, using gzip and chunked as transfer encoding only like this:
Transfer-Encoding: gzip, chunked
results in the browser not displaying the response correctly.
While Chrome tries downloading the resource as a .gz file, Firefox tries to display it which results in this:
The strange thing about this is that the message body generated by the server is exactly the same as when using gzip as Content-Encoding instead, because RFC7230 allows to apply multiple transfer encodings if the last one applied is chunked.
For example,
Transfer-Encoding: gzip, chunked
indicates that the payload body has been compressed using the gzip
coding and then chunked using the chunked coding while forming the
message body.
This is the original response from the server:
HTTP/1.1 200 OK
Date: Tue, 09 Jul 2019 17:52:41 GMT
Server: jPuzzle
Content-Type: text/plain
Transfer-Encoding: gzip, chunked
1c
òHÍÉÉW(Ï/ÊIQ ÿÿ
a
0
As one can guess, the body is gziped and chunked after that.
I would appreciate any help because I can't see where the specs have been violated.
You shoul use content-encoding header for end to end compression.
Transfer-Encoding is a hop-by-hop header, that is applied to a message between two nodes, not to a resource itself. Each segment of a multi-node connection can use different Transfer-Encoding values. If you want to compress data over the whole connection, use the end-to-end Content-Encoding header instead.
Also, send Accept-Encoding: gzip request header to tell the server what the client expects.
Related
I am trying to download a file from an application through two frameworks. one with struts 1(older framework) and the other with Spring MVC(migrated from old).In Spring migrated application it is showing file could not be downloaded in IE 11 when a compressed(gzipped) response is sent to the client. It is showing "File could not be downloaded". It works fine in chrome as well as in older struts framework. Also if the response is not compressed, file gets downloaded successfully on IE as well on Spring MVC. I cannot really identify the cause here. Requesting some guidance and help to identify this problem.
Request headers is
<code>
Request URL: //edited
Request Method: POST
Status Code: 200 / OK
- Request Headers
Accept: text/html, application/xhtml+xml, image/jxr, */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US
Authorization: Basic YW3Rt2aW46dG4V3zdD5EyMz6Q=
Cache-Control: no-cache
Connection: Keep-Alive
Content-Length: 3521
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID=urVBPpjD3QrP6KhkqCK4r8KSAuvKFSVPdp-UXyz-FYSz4W0cQmV9sh!4524586920
Host: localhost:7001
Referer: //edited
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
</code>
Below is the response headers
<code>
Response Headers
Cache-Control: private, no-cache, no-store
Content-Disposition: attachment; filename="Closed DSP01 CRD0037_2019-12-26_133924.csv"
Content-Encoding: gzip
Content-Type: text/csv; charset=UTF-8
Date: Thu, 26 Dec 2019 08:09:23 GMT
Expires: 0
Pragma: no-cache
Transfer-Encoding: chunked
</code>
particular code where compression is done
<code>
if (canUseGzip) {
response.setHeader("Content-Encoding", "gzip");
GZIPOutputStream out = new GZIPOutputStream(response.getOutputStream());
pw = new OutputStreamWriter(out, "UTF-8");
} else {
pw = response.getWriter();
}
</code>
I tried different possibilities and when I tried to explicitly set content-length header ,say,
response.setHeader("Content-Length", String.valueOf(1024));
the file got downloaded successfully in IE. When I googled, I found transfer-encoding and content-length are mutually exclusive and former is already on the response. I don't know why adding content-length worked here and it was required only on the Spring migrated code. File gets downloaded successfully on older struts framework without mentioning the content-length header.
Is there anything specific I am missing here? Is there any other permanent solution as well?
Also how to correctly set the content length? I randomly tried with 1024 and it may fail for some other data.
Any help is appreciated
What is special with API Gateway, it is not required to include Access-Control-Allow-Headers in the response header.
This is AWS API Gateway Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 152
Connection: keep-alive
Date: Tue, 11 Oct 2016 02:39:40 GMT
Access-Control-Allow-Origin: *
x-amzn-RequestId: f3838f6a-8f5b-11e6-b13a-XXXXXXX
X-Cache: Miss from cloudfront
Via: 1.1 XXXXXXXXXXX.cloudfront.net (CloudFront)
X-Amz-Cf-Id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==
This is My Own Rest Server Response Header:
HTTP/1.1 200
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type
Content-Type: application/json
Content-Length: 335
Date: Tue, 11 Oct 2016 02:34:31 GMT
The Problem with My Own Rest Server is that I need include Access-Control-Allow-Headers in the response otherwise I will encounter Request header field content-type is not allowed by Access-Control-Allow-Headers in preflight response.
With AWS API Gateway, I am not encounter that error even the Access-Control-Allow-Headers is not in response header.
According to the documentation here, Content-Type should be allowed by default.
"Apart from the headers set automatically by the user agent (e.g. Connection, User-Agent, etc.), the only headers which are allowed to be manually set are:
Accept
Accept-Language
Content-Language
Content-Type
The only allowed values for the Content-Type header are:
application/x-www-form-urlencoded
multipart/form-data
text/plain"
Hope that helps, Ritisha.
I am trying to send a gziped multipart POST to a Tomcat server from a Java application using Jersey. When the multipart request is not compressed, it works perfectly fine. Other types of compressed POSTS work fine, such as sending a single entity XML. I (believe) posting compressed data isn't an HTTP standard, but it does seem Tomcat supports it to some degree.
a working uncompressed multipart post:
POST /myApp/rest/data HTTP/1.1
Content-Type: multipart/mixed; boundary=Boundary_1_23237284_1331130438482
Cookie: JSESSIONID=XXXXXXXXXXXXXXXXXXXXXXXXX;Version=1;Path=/myApp/
MIME-Version: 1.0
User-Agent: Java/1.6.0_26
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Transfer-Encoding: chunked
d3
--Boundary_1_23237284_1331130438482
Content-Type: application/octet-stream
Content-Disposition: form-data; filename="uploadFile.war"; modification-date="Wed, 29 Feb 2012 18:01:38 GMT"; size=25343899; name="file"
{binary data here}
--Boundary_1_25179713_1331128929019
Content-Type: application/xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><myXMLEntity>stuff</myXMLEntity>
--Boundary_1_25179713_1331128929019--
When I compress it using the Jersey GZIPContentEncodingFilter() the following headers are sent, and I get back an HTTP 400 with a description of "incorrect syntax"
POST /myApp/rest/data HTTP/1.1
Content-Type: multipart/mixed
Cookie: JSESSIONID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX;Version=1;Path=/myApp/
Accept-Encoding: gzip
Content-Encoding: gzip
User-Agent: Java/1.6.0_26
Host: localhost:8080
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Transfer-Encoding: chunked
{binary data here}
Is what I'm trying to do possible? Should the Content-Type actually read multipart/x-gzip? I notice that when it gets compressed, the boundary text gets left off of the Content-Type header - is this also a problem?
I ran into this same issue (or something very similar) and tracked it down to the Content-Type header missing the boundary parameter when using GZIPContentEncodingFilter. I was able to work around it by using MultiPartMediaTypes.createFormData() when setting the type of the entity I was POSTing from the Jersey client. Doing so makes sure the boundary parameter is set earlier than Jersey would automatically set it, which seems to be too late when using the GZIPContentEncodingFilter for compressing the request entity. There is an equivalent method for multipart/mixed.
I don't have an IDE handy but something similar to this:
// client is a Jersey Client object
client.resource(uri).entity(multipartFormData, MultiPartMediaTypes.createFormData()).post(ClientResponse.class);
All that said, this will still only work if your server is able to handle GZIP compressed requests.
IMO you can't do this that way, because the server and the client need to agree on how to communicate (e.g. zip compression). HTTP is designed as request/response and server can return what the client can support.
The client sends request to the saying, "Hey server, I need this resource and I support gzip, so you can return gzip if you can". :)
Imagine a situation that your client sends to the server a few megabytes in gzip, but the server doesn't support this.
I make very simple HTTP server in Java. The response sent to the browser is
HTTP 1.1 200 OK
Server: OneFile 1.0
Content-Type: text/html; charset=utf-8
Content-Length: 202
Transfer-Encoding: chunked
<HTML><HEAD><TITLE>My website</TITLE></HEAD>
<BODY><H1>Document </H1>
</BODY></HTML>
mozilla firefox displays it as text/plain although it should be text/html Why?
I suspect the Setup info is ignored...is it any difference for browser if I make connection on port 8080?
Thanks for any help
The browser will honor your headers. Unfortunately, your response is malformed for several reasons:
the response should start HTTP/1.1, not HTTP 1.1
you specify Transfer-Encoding: chunked, but your response does not follow the chunked format.
It appears that Firefox, quite sensibly, refuses to interpret such malformed response and just shows it unchanged.
How do I handle properly a client Connection: close request field? As of now if I get this particular field I close the socket and wait for a following request from the client than reply again and start serving the data.
I don't know why my client/server communication is not working as the Apache Server I tested with.
Thanks for any clarifications...
Client/Server comunication:
CLIENT:
HEAD /stream.mpeg HTTP/1.0
Host: 127.0.0.1
User-Agent: SuperPlayer
Connection: Close
SERVER:
HTTP/1.0 200 OK
Date: Wed, 1 Jun 2011 20:05:13 GMT
Server: HTTP Server
Last-Modified: Mon, 06 Aug 2009 01:02:23 GMT
Accept-Ranges: bytes
Connection: Close
Content-Type: audio/mpeg
CLIENT:
HEAD /stream.mpeg HTTP/1.0
Host: 127.0.0.1
User-Agent: SuperPlayer
Connection: Close
SERVER:
HTTP/1.0 200 OK
Date: Wed, 1 Jun 2011 20:05:13 GMT
Server: HTTP Server
Last-Modified: Mon, 06 Aug 2009 01:02:23 GMT
Accept-Ranges: bytes
Connection: Close
Content-Type: audio/mpeg
231489172304981723409817234981234acvass123412323
21312hjdfaoi8w34yorhadl4hi8rali45mhalo3i,wmotw
345fqw354aoicu43yocq2i3hr
Client/ApacheServer Comunication:
CLIENT:
GET /test.mp3 HTTP/1.0
Host: 192.168.1.120
User-Agent: SuperPlayer
Connection: Close
SERVER:
HTTP/1.1 200 OK
Date: Wed, 01 Jun 2011 19:15:11 GMT
Server: Apache/2.2.16 (Win32)
Last-Modified: Thu, 29 Apr 2010 21:06:34 GMT
ETag: "14000000047049-4f75c8-4856680636a80"
Accept-Ranges: bytes
Content-Length: 5207496
Connection: close
Content-Type: audio/mpeg
...d.....<).0.. ..........<.#.. ( .h.$.J...1...i....A. ......c....a.9..!g.N...A. ........ ....>......|.......8....a......|..|N.............'>........?...C.....#..TJt.n .e...r.iL..#..IH...pR|.
Yes closing the socket is the right action to take. If the client is using this header properly, they are closing the socket on their end once they receive your response.
What I'm noticing here is that your server is not returning a Content-Length header. Even though the client is issuing a HEAD request, based on the W3C proposal (sec. 9.4):
The metainformation contained in the HTTP
headers in response to a HEAD request
SHOULD be identical to the information
sent in response to a GET request.
This method can be used for obtaining
metainformation about the entity
implied by the request without
transferring the entity-body itself.
This method is often used for testing
hypertext links for validity,
accessibility, and recent
modification.
The response to a HEAD request MAY be
cacheable in the sense that the
information contained in the response
MAY be used to update a previously
cached entity from that resource. If
the new field values indicate that the
cached entity differs from the current
entity (as would be indicated by a
change in Content-Length, Content-MD5,
ETag or Last-Modified), then the cache
MUST treat the cache entry as stale.
The key here is to make sure you're telling the client the size of the response without actually sending the data.
The Connection: close header just means that the client is expecting you to close the connection after sending the response. That also absolves you of having to send a Content-Length: header.
May I ask why are you using http 1.0 in the request?
There were no persistent connections in http 1.0, so the server is supposed to terminate the TCP connection after the response, whether you send Connection: close or not.
If you are using HTTP 1.0, there is no persistent connections as alexrs pointed, instead, Connection: keep-alive is being used with HTTP 1.0. On HTTP 1.1, you do not need that because HTTP connections are persistent by default on HTTP 1.1.
8.1.2 Overall Operation
A significant difference between HTTP/1.1 and earlier versions of HTTP
is that persistent connections are the default behavior of any HTTP
connection. That is, unless otherwise indicated, the client SHOULD
assume that the server will maintain a persistent connection, even
after error responses from the server.
Persistent connections provide a mechanism by which a client and a
server can signal the close of a TCP connection. This signaling takes
place using the Connection header field (section 14.10). Once a close
has been signaled, the client MUST NOT send any more requests on that
connection.
You can take a look at to the HTTP 1.1 RFC;
RFC for HTTP 1.1