I working on HTTP Traffic Data set which is composed of complete POST and GET request Like given below. I have written code in java that has separated each of these request and saved it as string element in array list. Now i am confused how to parse these raw HTTP request in java is there any method better than manual parsing?
GET http://localhost:8080/tienda1/imagenes/3.gif/ HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.8 (like Gecko)
Pragma: no-cache
Cache-control: no-cache
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: x-gzip, x-deflate, gzip, deflate
Accept-Charset: utf-8, utf-8;q=0.5, *;q=0.5
Accept-Language: en
Host: localhost:8080
Cookie: JSESSIONID=FB018FFB06011CFABD60D8E8AD58CA21
Connection: close
Here is a General Http Request Parser For all Method types (GET, POST, etc.) for your convinience:
package util.dpi.capture;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.StringReader;
import java.util.Hashtable;
/**
* Class for HTTP request parsing as defined by RFC 2612:
*
* Request = Request-Line ; Section 5.1 (( general-header ; Section 4.5 |
* request-header ; Section 5.3 | entity-header ) CRLF) ; Section 7.1 CRLF [
* message-body ] ; Section 4.3
*
* #author izelaya
*
*/
public class HttpRequestParser {
private String _requestLine;
private Hashtable<String, String> _requestHeaders;
private StringBuffer _messagetBody;
public HttpRequestParser() {
_requestHeaders = new Hashtable<String, String>();
_messagetBody = new StringBuffer();
}
/**
* Parse and HTTP request.
*
* #param request
* String holding http request.
* #throws IOException
* If an I/O error occurs reading the input stream.
* #throws HttpFormatException
* If HTTP Request is malformed
*/
public void parseRequest(String request) throws IOException, HttpFormatException {
BufferedReader reader = new BufferedReader(new StringReader(request));
setRequestLine(reader.readLine()); // Request-Line ; Section 5.1
String header = reader.readLine();
while (header.length() > 0) {
appendHeaderParameter(header);
header = reader.readLine();
}
String bodyLine = reader.readLine();
while (bodyLine != null) {
appendMessageBody(bodyLine);
bodyLine = reader.readLine();
}
}
/**
*
* 5.1 Request-Line The Request-Line begins with a method token, followed by
* the Request-URI and the protocol version, and ending with CRLF. The
* elements are separated by SP characters. No CR or LF is allowed except in
* the final CRLF sequence.
*
* #return String with Request-Line
*/
public String getRequestLine() {
return _requestLine;
}
private void setRequestLine(String requestLine) throws HttpFormatException {
if (requestLine == null || requestLine.length() == 0) {
throw new HttpFormatException("Invalid Request-Line: " + requestLine);
}
_requestLine = requestLine;
}
private void appendHeaderParameter(String header) throws HttpFormatException {
int idx = header.indexOf(":");
if (idx == -1) {
throw new HttpFormatException("Invalid Header Parameter: " + header);
}
_requestHeaders.put(header.substring(0, idx), header.substring(idx + 1, header.length()));
}
/**
* The message-body (if any) of an HTTP message is used to carry the
* entity-body associated with the request or response. The message-body
* differs from the entity-body only when a transfer-coding has been
* applied, as indicated by the Transfer-Encoding header field (section
* 14.41).
* #return String with message-body
*/
public String getMessageBody() {
return _messagetBody.toString();
}
private void appendMessageBody(String bodyLine) {
_messagetBody.append(bodyLine).append("\r\n");
}
/**
* For list of available headers refer to sections: 4.5, 5.3, 7.1 of RFC 2616
* #param headerName Name of header
* #return String with the value of the header or null if not found.
*/
public String getHeaderParam(String headerName){
return _requestHeaders.get(headerName);
}
}
I [am] working on [an] HTTP Traffic Data set which is composed of complete POST and GET request[s]
So you want to parse a file or list that contains multiple HTTP requests. What data do you want to extract? Anyway here is a Java HTTP parsing class, which can read the method, version and URI used in the request-line, and that reads all headers into a Hashtable.
You can use that one or write one yourself if you feel like reinventing the wheel. Take a look at the RFC to see what a request looks like in order to parse it correctly:
Request = Request-Line ; Section 5.1
*(( general-header ; Section 4.5
| request-header ; Section 5.3
| entity-header ) CRLF) ; Section 7.1
CRLF
[ message-body ] ; Section 4.3
If you just want to send the raw request as it is, it's very easy, just send the actual String using a TCP socket!
Something like this:
Socket socket = new Socket(host, port);
BufferedWriter out = new BufferedWriter(
new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
for (String line : getContents(request)) {
System.out.println(line);
out.write(line + "\r\n");
}
out.write("\r\n");
out.flush();
See this blog post by JoeJag for the full code.
UPDATE
I started a project, RawHTTP to provide HTTP parsers for request, responses, headers etc... it turned out so good that it makes it quite easy to write HTTP servers and clients on top of it. Check it out if you're looking for something lowl level.
Related
I want send a rest service call from Java using "GET" request.But i am getting the following error.I am able to use it in the postman but i am unable to send java application.
Exception in thread "main" java.lang.IllegalArgumentException: Key length not 128/192/256 bits.
at org.bouncycastle.crypto.engines.AESFastEngine.generateWorkingKey(Unknown Source)
at org.bouncycastle.crypto.engines.AESFastEngine.init(Unknown Source)
at org.bouncycastle.crypto.modes.CBCBlockCipher.init(Unknown Source)
at org.bouncycastle.crypto.macs.CMac.init(Unknown Source)
at com.rest.OAuth1.generateCmac(OAuth1.java:262)
at com.rest.OAuth1.generateSignature(OAuth1.java:180)
at com.rest.OAuth1.main(OAuth1.java:61)
This the my sample code
package com.rest;
// Java Libraries
import java.io.*;
import java.net.URL;
import java.net.URLEncoder;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.TreeMap;
import javax.net.ssl.HttpsURLConnection;
// Apache Commons Libraries used for the Nonce & Base64
import org.apache.commons.lang3.RandomStringUtils;
import org.apache.commons.codec.binary.Base64;
// Bouncy Castle Libraries used for CMAC encryption
import org.bouncycastle.crypto.engines.AESFastEngine;
import org.bouncycastle.crypto.macs.CMac;
import org.bouncycastle.crypto.params.KeyParameter;
/**
* Very basic sample code that demonstrates how to make an OAuth 1.0 System-to-System
* request to the LearningStudio API
*/
public class OAuth1 {
public static void main(final String[] args) throws Exception
{
// Setup the variables necessary to create the OAuth 1.0 signature and make the request
String httpMethod = "GET";
String URI = "example.com/one/oauth1/userManagement/v5/users";
//String appID = "{applicationId}";
String consumerKey = "1234567-1234-4186-1234-1234567891011!mailid#example.com";
String secret = "12345678-1234-1234-1234-12345678";
String body = "{var:val}";
String signatureMethod = "HMAC-SHA1";
byte[] requestBody = null;
HttpsURLConnection request = null;
BufferedReader in = null;
URL url = new URL(String.format("https://api.example.com%s", URI));
// Set the Nonce and Timestamp parameters
String nonce = getNonce();
String timestamp = getTimestamp();
// Set the request body if making a POST or PUT request
if ("POST".equals(httpMethod) || "PUT".equals(httpMethod))
{
requestBody = body.getBytes("UTF-8");
}
// Create the OAuth parameter name/value pair
Map<String, String> oauthParams = new LinkedHashMap<String, String>();
oauthParams.put("oauth_consumer_key", consumerKey);
//oauthParams.put("application_id", appID);
oauthParams.put("oauth_signature_method", signatureMethod);
oauthParams.put("oauth_timestamp", timestamp);
oauthParams.put("oauth_nonce", nonce);
// Get the OAuth 1.0 Signature
String signature = generateSignature(httpMethod, url, oauthParams, requestBody, secret);
System.out.println(String.format("OAuth 1.0 Signature = %s", signature));
// Add the oauth_signature parameter to the set of OAuth Parameters
oauthParams.put("oauth_signature", signature);
// Generate a string of comma delimited: keyName="URL-encoded(value)" pairs
StringBuilder sb = new StringBuilder();
String delimiter = "";
for (String keyName : oauthParams.keySet()) {
sb.append(delimiter);
String value = oauthParams.get((String) keyName);
sb.append(keyName).append("=\"").append(URLEncoder.encode(value, "UTF-8")).append("\"");
delimiter=",";
}
String urlString = url.toString();
// omit the queryString from the url
int startOfQueryString = urlString.indexOf('?');
if(startOfQueryString != -1) {
urlString = urlString.substring(0,startOfQueryString);
}
// Build the X-Authorization request header
String xauth = String.format("OAuth realm=\"%s\",%s", urlString, sb.toString());
System.out.println(String.format("X-Authorization request header = %s", xauth));
try
{
// Setup the Request
request = (HttpsURLConnection)url.openConnection();
request.setRequestMethod(httpMethod);
request.addRequestProperty("X-Authorization", xauth);
// Set the request body if making a POST or PUT request
if ("POST".equals(httpMethod) || "PUT".equals(httpMethod))
{
request.setRequestProperty("Content-Length", "" + requestBody.length);
request.setDoOutput(true);
OutputStream postStream = request.getOutputStream();
postStream.write(requestBody, 0, requestBody.length);
postStream.close();
}
// Send Request & Get Response
InputStreamReader reader = new InputStreamReader(request.getInputStream());
in = new BufferedReader(reader);
// Get the response stream
String response = in.readLine();
System.out.println(String.format("Successful Response: \r\n%s", response));
} catch (IOException e )
{
// This exception will be raised if the serve didn't return 200 - OK
System.out.print(e.getMessage());
} finally
{
if (in != null) in.close();
if (request != null) request.disconnect();
}
}
/**
* Generates a random nonce
*
* #return A unique identifier for the request
*/
private static String getNonce()
{
return RandomStringUtils.randomAlphanumeric(32);
}
/**
* Generates an integer representing the number of seconds since the unix epoch using the
* date/time the request is issued
*
* #return A timestamp for the request
*/
private static String getTimestamp()
{
return Long.toString((System.currentTimeMillis() / 1000));
}
/**
* Generates an OAuth 1.0 signature
*
* #param httpMethod The HTTP method of the request
* #param URL The request URL
* #param oauthParams The associative set of signable oAuth parameters
* #param requestBody The serialized POST/PUT message body
* #param secret Alphanumeric string used to validate the identity of the education partner (Private Key)
*
* #return A string containing the Base64-encoded signature digest
*
* #throws UnsupportedEncodingException
*/
private static String generateSignature(
String httpMethod,
URL url,
Map<String, String> oauthParams,
byte[] requestBody,
String secret
) throws UnsupportedEncodingException
{
// Ensure the HTTP Method is upper-cased
httpMethod = httpMethod.toUpperCase();
// Construct the URL-encoded OAuth parameter portion of the signature base string
String encodedParams = normalizeParams(httpMethod, url, oauthParams, requestBody);
// URL-encode the relative URL
String encodedUri = URLEncoder.encode(url.getPath(), "UTF-8");
// Build the signature base string to be signed with the Consumer Secret
String baseString = String.format("%s&%s&%s", httpMethod, encodedUri, encodedParams);
return generateCmac(secret, baseString);
}
/**
* Normalizes all OAuth signable parameters and url query parameters according to OAuth 1.0
*
* #param httpMethod The upper-cased HTTP method
* #param URL The request URL
* #param oauthParams The associative set of signable oAuth parameters
* #param requstBody The serialized POST/PUT message body
*
* #return A string containing normalized and encoded oAuth parameters
*
* #throws UnsupportedEncodingException
*/
private static String normalizeParams(
String httpMethod,
URL url,
Map<String, String> oauthParams,
byte[] requestBody
) throws UnsupportedEncodingException
{
// Sort the parameters in lexicographical order, 1st by Key then by Value
Map<String, String> kvpParams = new TreeMap<String, String>(String.CASE_INSENSITIVE_ORDER);
kvpParams.putAll(oauthParams);
// Place any query string parameters into a key value pair using equals ("=") to mark
// the key/value relationship and join each parameter with an ampersand ("&")
if (url.getQuery() != null)
{
for(String keyValue : url.getQuery().split("&"))
{
String[] p = keyValue.split("=");
kvpParams.put(p[0],p[1]);
}
}
// Include the body parameter if dealing with a POST or PUT request
if ("POST".equals(httpMethod) || "PUT".equals(httpMethod))
{
String body = Base64.encodeBase64String(requestBody).replaceAll("\r\n", "");
// url encode the body 2 times now before combining other params
body = URLEncoder.encode(body, "UTF-8");
body = URLEncoder.encode(body, "UTF-8");
kvpParams.put("body", body);
}
// separate the key and values with a "="
// separate the kvp with a "&"
StringBuilder combinedParams = new StringBuilder();
String delimiter="";
for(String key : kvpParams.keySet()) {
combinedParams.append(delimiter);
combinedParams.append(key);
combinedParams.append("=");
combinedParams.append(kvpParams.get(key));
delimiter="&";
}
// url encode the entire string again before returning
return URLEncoder.encode(combinedParams.toString(), "UTF-8");
}
/**
* Generates a Base64-encoded CMAC-AES digest
*
* #param key The secret key used to sign the data
* #param msg The data to be signed
*
* #return A CMAC-AES hash
*
* #throws UnsupportedEncodingException
*/
private static String generateCmac(String key, String msg)
throws UnsupportedEncodingException
{
byte[] keyBytes = key.getBytes("UTF-8");
byte[] data = msg.getBytes("UTF-8");
CMac macProvider = new CMac(new AESFastEngine());
macProvider.init(new KeyParameter(keyBytes));
macProvider.reset();
macProvider.update(data, 0, data.length);
byte[] output = new byte[macProvider.getMacSize()];
macProvider.doFinal(output, 0);
// Convert the CMAC to a Base64 string and remove the new line the Base64 library adds
String cmac = Base64.encodeBase64String(output).replaceAll("\r\n", "");
return cmac;
}
}
Is there any thing else which i am missing.
Also if i need to do POST request i need to add json data in the body tag directly?
EDIT: Is this really your key? maybe you changed the secret key to another size for not posting the original key here. If so check if keyBytes.length really gives you 16,24 or 32
I really digged deep now... I can't find any error in any of your code.
Your key is 256 bits long:
byte[] keyBytes = "12345678-1234-1234-1234-12345678".getBytes("UTF-8");
int bits = keyBytes.length*8;
System.out.println(bits); //gives 256
So i checked the CMac.java and they basically just copy the key with System.arraycopyso there is no error there.
They check the key in https://github.com/bcgit/bc-java/blob/master/core/src/main/java/org/bouncycastle/crypto/engines/AESFastEngine.java with
int keyLen = key.length;
if (keyLen < 16 || keyLen > 32 || (keyLen & 7) != 0)
{
throw new IllegalArgumentException("Key length not 128/192/256 bits.");
}
And since key.length is equal to 32 and 100000 & 111 is obviously 0 I really don't see anything wrong.
Please try to use a key with only 16 characters and tell us if the error still applies. Maybe you should also check if you really use the latest version of org.bouncycastle.crypto
If this doesn't help try just key.getBytes() without the charset UTF-8 for once.
I have some json I'm trying to return as part of a BasicHttpResponse.
A code snippit is below and I've annotated the resulting printouts.
I can print out the json in A and it looks good in 1. But when I print the entity at B I don't get any body in 2, though it does have the length. If I go into the debugger I see the data there.
If I change the content type and print the entity I can see that this change is reflected in 3 but again no actual String data.
I'm trying to push this data through a pipe and not getting the json body on a write is a bit of a problem.
My expectation is that when I add data to an entity and then print or write the entity using a HttpMessageWriter that the json would be shown/transferred. What am I missing? Is it unreasonable to expect the json to be printed on a toString?
BasicHttpResponse resp;
StringEntity entity = new StringEntity(json.toString(), "UTF-8");
A) logger.info("to str: " + json.toString());
B) logger.info("Entity: " + entity);
entity.setContentType("application/json; charset=UTF-8");
resp.setEntity(entity);
C) logger.info("set entity " + resp.getEntity());
1) to str: [{"id":"12","revision":"12","group":"12",
"remote":"12"}]
2) Entity: [Content-Type: text/plain;
charset=UTF-8,Content-Length: 81,Chunked: false]
3) set entity [Content-Type: application/json;
charset=UTF-8,Content-Length: 81,Chunked: false]
The toString() method from StringEntity will only print the data that you are getting, that is the correct behaviour.
The String given to StringEntity is saved as a byte array in the object.
This is the constructor for StringEntity:
/**
* Creates a StringEntity with the specified content and content type.
*
* #param string content to be used. Not {#code null}.
* #param contentType content type to be used. May be {#code null}, in which case the default
* MIME type {#link ContentType#TEXT_PLAIN} is assumed.
*
* #throws IllegalArgumentException if the string parameter is null
* #throws UnsupportedCharsetException Thrown when the named charset is not available in
* this instance of the Java virtual machine
* #since 4.2
*/
public StringEntity(final String string, final ContentType contentType) throws UnsupportedCharsetException {
super();
Args.notNull(string, "Source string");
Charset charset = contentType != null ? contentType.getCharset() : null;
if (charset == null) {
charset = HTTP.DEF_CONTENT_CHARSET;
}
this.content = string.getBytes(charset);
if (contentType != null) {
setContentType(contentType.toString());
}
}
If you want to print your entity as json again (for logging for example, since it's already being set in the response) you have to do something like:
logger.info("Entity: " + IOUtils.toString(entity.getContent()));
Using IOUtils.toString since entity.getContent() brings an InputStream object, you can use it as you wish.
I want to download a PNG from a remote web server through the tor proxy. The proxy is up and running and I can send the necessary GET and receive the OK using a tor socket. I can also receive the content. However, I don't manage to parse the content. My goal is to have a method I can pass a connected socket (preconfigured by me to use the Tor network) and get back an BufferedImage. How can I do that?
My GET request looks like the following:
PrintStream httpOut = new PrintStream(socket.getOutputStream());
httpOut.print("GET /" + path + " HTTP/1.0\r\n");
httpOut.print("Host: " + host + ":" + port + "\r\n");
httpOut.print("\r\n");
httpOut.flush();
And an example response would be:
HTTP/1.1 200 OK
Date: Mon, 23 Feb 2015 20:54:25 GMT
Server: Apache
Set-Cookie: PHPSESSID=thesessionideeee; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Last-Modified: Mon, 23 Feb 2015 20:54:25GMT
Cache-Control: post-check=0, pre-check=0
Content-Length: 3884
Connection: close
Content-Type: image/png
Thank you for help,
Turakar
You can check this class from http://web.mit.edu/foley/www/TinFoil/src/tinfoil/TorLib.java
that explain how to connect:
package tinfoil;
/**
* Tinfoil is an RFID Privacy and Security Enhancement library.
*
* Copyright (c) 2005 Joe Foley, MIT AutoID Labs
Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the "Software"),
to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.
*/
// $HeadURL: $
// $Id: $
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.net.InetAddress;
import java.net.Socket;
/**
* The Onion Router Java Library routines<br />
* These methods allow us to setup an anonymized TCP socket through the Tor
* network and do safe anonymized DNS lookups.<br />
* This code was written with the help of Roger Dingledine and Nick Mathewson.<br />
* The code is open-source under the MIT X11 license.
* <ul>
* <li> http://tor.eff.org
* <li><a href =
* "http://tor.eff.org/cvs/control/doc/howto.txt">http://tor.eff.org
* /cvs/control/doc/howto.txt</a>
* <li><a href =
* "http://www.mokabyte.it/2000/06/firewallutil.htm">http://www.mokabyte
* .it/2000/06/firewallutil.htm</a>
* </ul>
*
* #author Joe Foley<foley at MIT dot EDU>, MIT AutoID Labs
* #version 1.0
* <p>
*/
public class TorLib {
/**
* Default TOR Proxy port.
*/
static int proxyPort = 9050;
/**
* Default TOR Proxy hostaddr.
*/
static String proxyAddr = "localhost";
/**
* Constant tells SOCKS4/4a to connect. Use it in the <i>req</i> parameter.
*/
final static byte TOR_CONNECT = (byte) 0x01;
/**
* Constant tells TOR to do a DNS resolve. Use it in the <i>req</i>
* parameter.
*/
final static byte TOR_RESOLVE = (byte) 0xF0;
/**
* Constant indicates what SOCKS version are talking Either SOCKS4 or
* SOCKS4a
*/
final static byte SOCKS_VERSION = (byte) 0x04;
/**
* SOCKS uses Nulls as field delimiters
*/
final static byte SOCKS_DELIM = (byte) 0x00;
/**
* Setting the IP field to 0.0.0.1 causes SOCKS4a to be enabled.
*/
final static int SOCKS4A_FAKEIP = 0x01;
/**
* This method allows you to demo/access the resolver/socket generation from
* the command line. Run with "-h" to get the help menu.
*
* #param args
* Command line arguments for test main method.
*/
public static void main(final String[] args) {
String req = "-r";
String targetHostname = "tor.eff.org";
String targetDir = "index.html";
int targetPort = 80;
if ((args.length > 0) && args[0].equals("-h")) {
System.out.println("Tinfoil/TorLib - interface for using Tor from Java\n" + "By Joe Foley<foley#mit.edu>\n"
+ "Usage: java Tinfoil.TorLib <cmd> <args>\n" + "<cmd> can be: -h for help\n"
+ " -r for resolve\n" + " -w for wget\n" + "For -r, the arg is:\n"
+ " <hostname> Hostname to DNS resolve\n" + "For -w, the args are:\n"
+ " <host> <path> <optional port>\n"
+ " for example, http://tor.eff.org:80/index.html would be\n" + " tor.eff.org index.html 80\n"
+ " Since this is a demo, the default is the tor website.\n");
System.exit(2);
}
if (args.length >= 4) {
targetPort = new Integer(args[2]).intValue();
}
if (args.length >= 3) {
targetDir = args[2];
}
if (args.length >= 2) {
targetHostname = args[1];
}
if (args.length >= 1) {
req = args[0];
}
if (req.equals("-r")) {
System.out.println(TorResolve(targetHostname));
} else if (req.equals("-w")) {
try {
Socket s = TorSocket(targetHostname, targetPort);
DataInputStream is = new DataInputStream(s.getInputStream());
PrintStream out = new java.io.PrintStream(s.getOutputStream());
// Construct an HTTP request
out.print("GET /" + targetDir + " HTTP/1.0\r\n");
out.print("Host: " + targetHostname + ":" + targetPort + "\r\n");
out.print("Accept: */*\r\n");
out.print("Connection: Keep-Aliv\r\n");
out.print("Pragma: no-cache\r\n");
out.print("\r\n");
out.flush();
// this is from Java Examples In a Nutshell
final InputStreamReader from_server = new InputStreamReader(is);
char[] buffer = new char[1024];
int chars_read;
// read until stream closes
while ((chars_read = from_server.read(buffer)) != -1) {
// loop through array of chars
// change \n to local platform terminator
// this is a nieve implementation
for (int j = 0; j < chars_read; j++) {
if (buffer[j] == '\n') {
System.out.println();
} else {
System.out.print(buffer[j]);
}
}
System.out.flush();
}
s.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
/**
* This method Creates a socket, then sends the inital SOCKS request info It
* stops before reading so that other methods may differently interpret the
* results. It returns the open socket.
*
* #param targetHostname
* The hostname of the destination host.
* #param targetPort
* The port to connect to
* #param req
* SOCKS/TOR request code
* #return An open Socket that has been sent the SOCK4a init codes.
* #throws IOException
* from any Socket problems
*/
static Socket TorSocketPre(final String targetHostname, final int targetPort, final byte req) throws IOException {
Socket s;
// System.out.println("Opening connection to "+targetHostname+":"+targetPort+
// " via proxy "+proxyAddr+":"+proxyPort+" of type "+req);
s = new Socket(proxyAddr, proxyPort);
DataOutputStream os = new DataOutputStream(s.getOutputStream());
os.writeByte(SOCKS_VERSION);
os.writeByte(req);
// 2 bytes
os.writeShort(targetPort);
// 4 bytes, high byte first
os.writeInt(SOCKS4A_FAKEIP);
os.writeByte(SOCKS_DELIM);
os.writeBytes(targetHostname);
os.writeByte(SOCKS_DELIM);
return (s);
}
/**
* This method creates a socket to the target host and port using
* TorSocketPre, then reads the SOCKS information.
*
* #param targetHostname
* Hostname of destination host.
* #param targetPort
* Port on remote destination host.
* #return Fully initialized TCP Socket that tunnels to the target Host/Port
* via the Tor Proxy host/port.
* #throws IOException
* when Socket and Read/Write exceptions occur.
*/
static Socket TorSocket(final String targetHostname, final int targetPort) throws IOException {
Socket s = TorSocketPre(targetHostname, targetPort, TOR_CONNECT);
DataInputStream is = new DataInputStream(s.getInputStream());
// only the status is useful on a TOR CONNECT
byte version = is.readByte();
byte status = is.readByte();
if (status != (byte) 90) {
// failed for some reason, return useful exception
throw (new IOException(ParseSOCKSStatus(status)));
}
// System.out.println("status: "+ParseSOCKSStatus(status));
int port = is.readShort();
int ipAddr = is.readInt();
return (s);
}
/**
* This method opens a TOR socket, and does an anonymous DNS resolve through
* it. Since Tor caches things, this is a very fast lookup if we've already
* connected there The resolve does a gethostbyname() on the exit node.
*
* #param targetHostname
* String containing the hostname to look up.
* #return String representation of the IP address: "x.x.x.x"
*/
static String TorResolve(final String targetHostname) {
int targetPort = 0; // we dont need a port to resolve
try {
Socket s = TorSocketPre(targetHostname, targetPort, TOR_RESOLVE);
DataInputStream is = new DataInputStream(s.getInputStream());
byte version = is.readByte();
byte status = is.readByte();
if (status != (byte) 90) {
// failed for some reason, return useful exception
throw (new IOException(ParseSOCKSStatus(status)));
}
int port = is.readShort();
byte[] ipAddrBytes = new byte[4];
is.read(ipAddrBytes);
InetAddress ia = InetAddress.getByAddress(ipAddrBytes);
// System.out.println("Resolved into:"+ia);
is.close();
String addr = ia.toString().substring(1); // clip off the "/"
return (addr);
} catch (Exception e) {
e.printStackTrace();
}
return (null);
}
/**
* This helper method allows us to decode the SOCKS4 status codes into Human
* readible input.<br />
* Based upon info from
* http://archive.socks.permeo.com/protocol/socks4.protocol
*
* #param status
* Byte containing the status code.
* #return String human-readible representation of the error.
*/
static String ParseSOCKSStatus(final byte status) {
// func to turn the status codes into useful output
// reference
String retval;
switch (status) {
case 90:
retval = status + " Request granted.";
break;
case 91:
retval = status + " Request rejected/failed - unknown reason.";
break;
case 92:
retval = status + " Request rejected: SOCKS server cannot connect to identd on the client.";
break;
case 93:
retval = status + " Request rejected: the client program and identd report different user-ids.";
break;
default:
retval = status + " Unknown SOCKS status code.";
}
return (retval);
}
}
General Use-Case
Imagine a client that is uploading large amounts of JSON. The Content-Type should remain application/json because that describes the actual data. Accept-Encoding and Transfer-Encoding seem to be for telling the server how it should format the response. It appears that responses use the Content-Encoding header explicitly for this purpose, but it is not a valid request header.
Is there something I am missing? Has anyone found an elegant solution?
Specific Use-Case
My use-case is that I have a mobile app that is generating large amounts of JSON (and some binary data in some cases but to a lesser extent) and compressing the requests saves a large amount of bandwidth. I am using Tomcat as my Servlet container. I am using Spring for its MVC annotations primarily just to abstract away some of the JEE stuff into a much cleaner, annotation-based interface. I also use Jackson for auto (de)serialization.
I also use nginx, but I am not sure if thats where I want the decompression to take place. The nginx nodes simply balance the requests which are then distributed through the data center. It would be just as nice to keep it compressed until it actually got to the node that was going to process it.
Thanks in advance,
John
EDIT:
The discussion between myself and #DaSourcerer was really helpful for those that are curious about the state of things at the time of writing this.
I ended up implementing a solution of my own. Note that this specifies the branch "ohmage-3.0", but it will soon be merged into the master branch. You might want to check there to see if I have made any updates/fixes.
https://github.com/ohmage/server/blob/ohmage-3.0/src/org/ohmage/servlet/filter/DecompressionFilter.java
It appears [Content-Encoding] is not a valid request header.
That is actually not quite true. As per RFC 2616, sec 14.11, Content-Encoding is an entity header which means it can be applied on the entities of both, http responses and requests. Through the powers of multipart MIME messages, even selected parts of a request (or response) can be compressed.
However, webserver support for compressed request bodies is rather slim. Apache supports it to a degree via the mod_deflate module. It's not entirely clear to me if nginx can handle compressed requests.
Because the original code is not available any more. In case someone come here need it.
I use "Content-Encoding: gzip" to identify the filter need to decompression or not.
Here's the codes.
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException
{
HttpServletRequest httpServletRequest = (HttpServletRequest) request;
String contentEncoding = httpServletRequest.getHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.indexOf("gzip") > -1)
{
try
{
final InputStream decompressStream = StreamHelper.decompressStream(httpServletRequest.getInputStream());
httpServletRequest = new HttpServletRequestWrapper(httpServletRequest)
{
#Override
public ServletInputStream getInputStream() throws IOException
{
return new DecompressServletInputStream(decompressStream);
}
#Override
public BufferedReader getReader() throws IOException
{
return new BufferedReader(new InputStreamReader(decompressStream));
}
};
}
catch (IOException e)
{
mLogger.error("error while handling the request", e);
}
}
chain.doFilter(httpServletRequest, response);
}
Simple ServletInputStream wrapper class
public static class DecompressServletInputStream extends ServletInputStream
{
private InputStream inputStream;
public DecompressServletInputStream(InputStream input)
{
inputStream = input;
}
#Override
public int read() throws IOException
{
return inputStream.read();
}
}
Decompression stream code
public class StreamHelper
{
/**
* Gzip magic number, fixed values in the beginning to identify the gzip
* format <br>
* http://www.gzip.org/zlib/rfc-gzip.html#file-format
*/
private static final byte GZIP_ID1 = 0x1f;
/**
* Gzip magic number, fixed values in the beginning to identify the gzip
* format <br>
* http://www.gzip.org/zlib/rfc-gzip.html#file-format
*/
private static final byte GZIP_ID2 = (byte) 0x8b;
/**
* Return decompression input stream if needed.
*
* #param input
* original stream
* #return decompression stream
* #throws IOException
* exception while reading the input
*/
public static InputStream decompressStream(InputStream input) throws IOException
{
PushbackInputStream pushbackInput = new PushbackInputStream(input, 2);
byte[] signature = new byte[2];
pushbackInput.read(signature);
pushbackInput.unread(signature);
if (signature[0] == GZIP_ID1 && signature[1] == GZIP_ID2)
{
return new GZIPInputStream(pushbackInput);
}
return pushbackInput;
}
}
Add to your header when you are sending:
JSON : "Accept-Encoding" : "gzip, deflate"
Client code :
HttpUriRequest request = new HttpGet(url);
request.addHeader("Accept-Encoding", "gzip");
#JulianReschke pointed out that there can be a case of:
"Content-Encoding" : "gzip, gzip"
so extended server code will be:
InputStream in = response.getEntity().getContent();
Header encodingHeader = response.getFirstHeader("Content-Encoding");
String gzip = "gzip";
if (encodingHeader != null) {
String encoding = encodingHeader.getValue().toLowerCase();
int firstGzip = encoding.indexOf(gzip);
if (firstGzip > -1) {
in = new GZIPInputStream(in);
int secondGzip = encoding.indexOf(gzip, firstGzip + gzip.length());
if (secondGzip > -1) {
in = new GZIPInputStream(in);
}
}
}
I suppose that nginx is used as load balancer or proxy, so you need to set tomcat to do decompression.
Add following attributes to the Connector in server.xml on Tomcat,
<Connector
compression="on"
compressionMinSize="2048"
compressableMimeType="text/html,application/json"
... />
Accepting gziped requests in tomcat is a different story. You'll have to put a filter in front of your servlets to enable request decompression. You can find more about that here.
The method below is hit or miss if it parses the request properly...
Here is what the request looks like:
POST /addEvent/ HTTP/1.1
Host: localhost:1234
Content-Type: multipart/form-data; boundary=Boundary+0xAbCdEfGbOuNdArY
Accept-Encoding: gzip, deflate
Content-Length: 201
Accept-Language: en;q=1, fr;q=0.9, de;q=0.8, ja;q=0.7, nl;q=0.6, it;q=0.5
Accept: application/json
Connection: keep-alive
User-Agent: XXXXXXXXXXXXXXXXXX
--Boundary+0xAbCdEfGbOuNdArY
Content-Disposition: form-data; name="userInfo"
{ "user_id" : 1, "value" : "Water", "typeCode" : "Searched" }
Here is how we are extracting it now...
//Key where the request begins
String keyString = "\"userInfo\"";
//Get the index of the key
int end = bufferedJson.lastIndexOf("\"userInfo\"");
//Create substring at beginning of the json
String json = bufferedJson.substring(end+keyString.length(), bufferedJson.length());
//Convert json to feed item
Gson gson = new Gson();
Event eventItem = gson.fromJson(json, Event.class);
I get this error pretty often:
Expected BEGIN_OBJECT but was STRING at line 1 column 1
How can we parse this efficiently?
Use Apache HTTP Client 4 to read Http response body in a convenient way. If you need to marshall your json further to a java object then make use of jackson. Here is the sample code:
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
/**
* This example demonstrates the use of the {#link ResponseHandler} to simplify
* the process of processing the HTTP response and releasing associated resources.
*/
public class ClientWithResponseHandler {
public final static void main(String[] args) throws Exception {
HttpClient httpclient = new DefaultHttpClient();
try {
HttpGet httpget = new HttpGet("http://www.google.com/");
System.out.println("executing request " + httpget.getURI());
// Create a response handler
ResponseHandler<String> responseHandler = new BasicResponseHandler();
// Body contains your json stirng
String responseBody = httpclient.execute(httpget, responseHandler);
System.out.println("----------------------------------------");
System.out.println(responseBody);
System.out.println("----------------------------------------");
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}
To parse this better:
First, it seems like you're taking a "raw" HTTP POST request, and then reading it line by line using BufferedReader (your comment suggests this), this way you'll lose the new line chars; if you are going to do so, add a new line ("\n") every time you read a line to your final String, this way it doesn't lose the new lines and facilitates the things for the next step.
Now, with this final String, you can use this:
String json = null;
Pattern pattern = Pattern.compile("\n\n");
Matcher matcher = pattern.matcher(myString); // myString is the String you built from your header
if(matcher.find() && matcher.find()) {
json = myString.substring(matcher.start() + 2);
} else {
// Handle error: json string wasn't found
}
CAVEATS: this works if:
POST Request will always be multipart/form-data
There are not other parameters in the request
you STOP reading the request as soon as you find your json data
you included "\n" every time you read a line as I said in the first step
Personally I wouldn't read the raw HTTP header, I'd rather use Apache commons FileUpload or the like, but if your are going to do it this way, I think this is the less terrible solution.
You can use
gson().fromJson(request.getReader(), Event.class);
or
String json = request.getReader().readLine();
gson().fromJson(json, Event.class);