Hi I need to evaluate if a server supports gzip compression.
Therefore I want to set gzip as Accept-Encoding in my request and evaluate if the response of the server containts "Content-Encoding: gzip".
However I am not quite clear on how to do this with HTTPURLConnection. Especially when to query my Connection object about the response. I currently do:
HttpURLConnection con = (HttpURLConnection) feedurl.openConnection();
con.setRequestProperty("Accept-Encoding", "gzip");
System.out.println("Current Accept-Encoding: "+con.getRequestProperty("Accept-Encoding"));
//check the response for the content-size
float feedsize = con.getContentLength()/1024f;
//the server uses transfer-encoding=chunked and does not specify
//a content length
if(con.getContentLength()==-1)
{
//count the content size manually
CountingInputStream counter = new CountingInputStream(con.getInputStream());
while(counter.read()!=-1)
{}
feedsize=counter.getCount()/1024f;
}
con.disconnect();
Have a look at this OReilly article. It illustrates how to generate the request, and how to interrogate the response and then create the appropriate stream (normal or gzipped) dependent on what's being returned.
Related
I'm trying to use an existing API that calls for a json object to be "submitted" using POST, but the page also requires a few parameters in request, for this example we will use name, and email. I'm super new to REST so I'm probably making an ignorant mistake somewhere here.
Here is the code I have so far in my servlet:
String path = "http://www.test.com/submit";
URL url = new URL(path);
conn = (HttpURLConnection)url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
Gson gson = new Gson();
//julySchedule is the object I want to submit with this request alongside the parameters.
String input = gson.toJson(julySchedule);
OutputStream os = conn.getOutputStream();
os.write(input.getBytes());
os.flush();
if (conn.getResponseCode() != 200) {
throw new RuntimeException("HTTP POST Request Failed with Error code : "
+ conn.getResponseCode());
}
I've tried to put the parameters in the URL, like:
/submit?name=Name&email=test#gmail.com
But that didn't work, because POST requests wont accept parameters like that.
Then I tried to add it to the Output stream like:
String params = "name=Name&email=test#gmail.com"
OutputStream os = conn.getOutputStream();
os.write(path.toBytes());
os.write(input.getBytes());
os.flush();
But that didn't work either. Am I making a really dumb mistake somewhere?
It is perfectly fine for a POST request to contain query string(Though not best practice).
Since you use application/json as Content-Type, I assume your payload has to be a JSON object. You cannot mix your params in POST body. You have 2 options:
Kep email&name in QueryString, and json in POST payload: What you did should work.
Use POST payload only: Set Content-Type: application/x-www-form-urlencoded as follows: name=NAME&email=EMAIL&jsonPayLoad={Your Json Object}. Server should extract jsonPayLoad from POST payload.
Make sure to escape using encodeURIComponent before writing to POST output stream.
I am using HttpsURLConnection to call a server and return the response returned from the HttpsURLConnection from my servlet. I am copying the response from HttpssURLConnection to HttpServletresponse using streams, copying bytes from the httpconnection response input stream to the response's output stream, checking the end by seeing if read returns < 0.
Following is the code for copying the response. The variable response is of type HttpServletResponse and the variable httpCon is of type HttpsURLConnection.
InputStream responseStream = httpCon.getInputStream();
if (responseStream != null)
{
OutputStream os = response.getOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = responseStream.read(buffer)) >= 0)
{
os.write(buffer, 0, len);
}
os.flush();
os.close();
}
On the client side, I am using python requests library to read the response.
What I am seeing that if I use the curl to test my servlet, I am getting the proper response json, response = u'{"key":"value"}'.
If i read it from the requests python, it is putting some extra characters in the response , the response looks like the following
response = u'b0\r\n{"key":"value"}\r\n0\r\n\r\n'
Both the strings are unicode. But the second one has extra characters.
Same resonse if I try from curl/Postman restclient, I am able to get it properly. But from python requests, it is not working. I tried another livetest library in python, with that also, it is not working and the response has same characters. I also tried to change the accept-encoding header but it did not have any effect.
Because of this, I am not able to parse the json.
I don't want to change the client to parse this kind of string.
Can I change something on the server so that it will work correctly?
Did the response contain the below header "Transfer-Encoding: chunked"?
The response should be in Chunked transfer encoding
https://en.wikipedia.org/wiki/Chunked_transfer_encoding.
In this case, you get \r\n0\r\n\r\n at the end of the response is as expected since it is terminating symbol of this encoding. I guest curl/Postman just help us to handle Chunked transfer encoding, so you can't find these chunked symbols.
I want to read the second line of the text at this URL: "http://vuln2014.picoctf.com:51818/" (this is a capture-the-flag competition but only asking for flags or direction to flags breaks the competition rules). I am attempting to open an input stream from the URL but I get an Invalid HTTP Response exception. Any help is appreciated, and I recognize that my error is likely quite foolish.
Code:
URL url = new URL("http://vuln2014.picoctf.com:51818");
URLConnection con = url.openConnection();
InputStream is = con.getInputStream()
The error occurs at the third line.
java.io.IOException: Invalid Http response at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1342) at name.main(name.java:41)
curl happily gets the text from the page, and it is perfectly accessible from a web browser.
When you do this:
URL url = new URL("http://vuln2014.picoctf.com:51818");
URLConnection con = url.openConnection();
You are entering into a contract that says that this URL uses the http protocol. When you call openConnection it expects to get http responses because you used http:// in the URL as the protocol. The Java Documentation says:
If for the URL's protocol (such as HTTP or JAR), there exists a public, specialized URLConnection subclass belonging to one of the following packages or one of their subpackages: java.lang, java.io, java.util, java.net, the connection returned will be of that subclass. For example, for HTTP an HttpURLConnection will be returned, and for JAR a JarURLConnection will be returned.
The server you are connecting to just returns a couple lines of data. I retrieved them with the command nc vuln2014.picoctf.com 51818. There is no http response code like HTTP/1.1 200 OK:
Welcome to the Daedalus Corp Spies RSA Key Generation Service. The public modulus you should use to send your updates is below. Remember to use exponent 65537.
b4ab920c4772c5247e7d89ec7570af7295f92e3b584fc1a1a5624d19ca07cd72ab4ab9c8ec58a63c09f382aa319fa5a714a46ffafcb6529026bbc058fc49fb1c29ae9f414db4aa609a5cab6ff5c7b4c4cfc7c18844f048e3899934999510b2fe25fcf8c572514dd2e14c6e19c4668d9ad82fe647cf9e700dcf6dc23496be30bb
In this case I would use java.net.Socket to establish a connection and then read the lines. This is a simplistic approach that assumes there are 2 lines of data:
Socket theSocket;
try {
theSocket = new Socket("vuln2014.picoctf.com", 51818);
BufferedReader inFile = new BufferedReader(new InputStreamReader(theSocket.getInputStream()));
String strGreet = inFile.readLine();
String strData = inFile.readLine();
} catch (IOException e) {
e.printStackTrace();
}
As for why curl and browsers may render it properly? They are likely more lenient about the data they read and will just dump what is read from the port even if it doesn't conform to the specified protocol (like http)
I am trying to make a Http POST request using apache HTTP client. I am trying to copy contents of an HTTP POST request (received at my application) to another HTTP POST request (initiated from my application to another URL). Code is shown below:
httpPost = new HttpPost(inputURL);
// copy headers
for (Enumeration<String> e = request.getHeaderNames(); e.hasMoreElements();) {
String headerName = e.nextElement().toString();
httpPost.setHeader(headerName, request.getHeader(headerName));
}
BufferedInputStream clientToProxyBuf = new BufferedInputStream(request.getInputStream());
BasicHttpEntity basicHttpEntity = new BasicHttpEntity();
basicHttpEntity.setContent(clientToProxyBuf);
basicHttpEntity.setContentLength(clientToProxyBuf.available());
httpPost.setEntity(basicHttpEntity);
HttpResponse responseFromWeb = httpclient.execute(httpPost);
Basically, I am trying to implement a proxy application which will get a url as parameter, froward the request to the URL and then serve pages etc in custom look and feel.
Here request is HttpServletRequest. I am facing problem in setting content length. Through debugging I found out that clientToProxyBuf.available() is not giving me correct length of input stream and I am getting Http error 400 IE and Error 354 (net::ERR_CONTENT_LENGTH_MISMATCH): The server unexpectedly closed the connection in chrome.
Am I doing it wrong? Is there any other way to achieve it?
The available() function doesn't provide the actual length of the content of the stream, rather
Returns the number of bytes that can be read from this input stream without blocking. (From javadoc)
I would suggest you to first read the whole content from the stream, and then set that to the content, rather than passing the stream object. That way, you will also have the actual length of the content.
It was rather simple and very obvious. I just needed to get content length from header as:
basicHttpEntity.setContentLength(Integer.parseInt(request.getHeader("Content-Length")));
I am writing a Restlet application on GAE similar as described here:
First Application
I am sending back a JSON represntation of an entity, and this works. But I am so far unsuccessful in sending the response compressed.
I tried to add to request an accept-encoding header with "gzip". but that didn't help. Here is how i tested it:
URL url = new URL(address);
URLConnection urlConn = url.openConnection();
urlConn.setRequestProperty("Accept-Encoding", "gzip");
InputStream openStream = urlConn.getInputStream();
Any ideas would be very much appreciated!
I believe you also need to specify the User-Agent header to force the compression. From the docs:
https://developers.google.com/appengine/docs/python/runtime#Responses
If the client sends HTTP headers with the request indicating that the
client can accept compressed (gzipped) content, App Engine compresses
the response data automatically and attaches the appropriate response
headers. It uses both the Accept-Encoding and User-Agent request
headers to determine if the client can reliably receive compressed
responses. Custom clients can force content to be compressed by
specifying both Accept-Encoding and User-Agent headers with a value of
"gzip".