I have written a code to send a HTTP request through a socket in java. Now I want to get the HTTP response that was sent by the server to which I sent HTTP request.
It's not totally clear what you're asking for. Assuming you've written the request to the socket, the next thing you'll want to do is:
Call shutdownOutput() on the socket to tell the server that the request is done (not necessary if you've sent the content length)
Read the response from the socket's input stream, parsing according to the HTTP spec.
This is a bunch of work, so I'd suggest that rather than rolling your own HTTP request logic, use URLConnection which is built-in to Java and includes methods for retrieving the content of a response as well as any headers set by the server.
As Jon said, read the HTTP spec. However Internet protocols are generally line oriented, so you can read the response a line a time. The first line will be the response. Following this will be the headers, one per line (unless there's a continuation). If there's a body one of the headers will the content-type, this will tell you what the content is. If you want to read the content you will need to understand the different ways the content can be sent. There may be a content length header (or not) or the content maybe chunked (can't remember the header off the top of my head). And of course the content may be binary rather than text.
yup!
that's right!
the respond should be clearly readed by the inputstream into
a few chunk of bytes...
thus we could translate it into a readable format.
But that also take longer time.... :(
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 years ago.
Improve this question
I'm trying to get the HTTP request from Google Chrome to get it's data. For that I use readLine() from BufferedReader but for some reason I think it gets stuck at the last line because the buffer stays open and it stays waiting for more input. Here is the code that I use in the while loop:
String line;
ArrayList<String> request = new ArrayList<String>();
while ((line = inFromClient.readLine()) != null) {
request.add(line);
}
If I forcely break the loop it works, basically im trying to get an efficient read of all lines but without the inconsistencies of ready()
HTTP seems like a crazy simple protocol but it is not; you should use an HTTP client library such as the built-in java.net.http client.
The problem is that the concept of 'give me my data, then close it down' is HTTP/1.0, and that's a few decades out of date. HTTP/2.0 and HTTP/3.0 are binary protocols, and HTTP/1.1 tends to leave the connection open. In general, 'read lines', and even 'use Reader' (as in, read characters instead of bytes) is the wrong way to go about it, as HTTP is not a textual protocol. I know. It looks like one. It's not.
Here is a highly oversimplified overview of how e.g. a browser reads HTTP/1.1 responses:
Use raw byte processing because HTTP body content is raw (or can be), therefore wrapping the whole thing into e.g. an InputStreamReader or BufferedReader is a non-starter.
Keep reading until an 0x0A byte (in ASCII, the newline symbol), or X bytes have been read and your buffer for this is full, where X is not extraordinarily large. Wouldn't want a badly behaving server or a misunderstanding where you connect to a different (non-HTTP) service to cause a memory issue! Parse this first line as an HTTP/1.1 response.
Keep doing this loop to pick up all headers. Use the same 'my buffer has limits' trick to avoid memory issues.
Then check the response code in order to figure out if a body will be forthcoming. It's HTTP/1.1, so you can't just go: "Well, if the connection is closed, I guess no body is forthcoming". Whether one will be coming or not depends primarily on the response code.
Assuming a body exists, read the double-newline that separates headers from the body.
If the content is transfered as chunked encoding (common), start blitting data into a buffer, but check if you read the entire chunk. Reading chunked encoding is its own game, really.
Alternatively, HTTP/1.1 DEMANDS that if chunked encoding isn't used that Content-Length is present. Use this header to know precisely how many bytes to read.
Neither 'a newline' nor 'close connection' can ever serve as a meaningful marker of 'end of data' in HTTP/1.1, so, don't.
Then either pass the content+headers+returncode verbatim to the requesting code, or dress it up a bit. For example, if the Content-Type header is present and has value text/html; encoding=UTF-8 you can consider taking the body data and turning it into a string via UTF-8 (new String(byteArray, StandardCharsets.UTF_8);).
Note that I've passed right over some bizarre behaviour that servers do because in ye olden days some dumb browser did weird things and it's now the status quo (for example, range requests are quite bizarre) and there's of course HTTP2 and HTTP3 which are completely different protocols.
Also, of course, HTTP servers are rare these days; HTTPS is where its at, and that's quite different too.
Using Spring MVC, for each incoming request id like to set the statusCode and headers.
Once they are set, I need to pad the response body to make the entire response size, including all headers and content (actual data wired to the client), to be exactly X bytes (e.g. 300 bytes). The response size will vary per request, but all responses will have to padded.
There are no limitations regarding the manipulation of the response.
Using HttpServletResponse I can set the statusCose and headers and also maybe get the response size. But I couldn't find a way to set the body content/length in accordance to the required size.
If I use a ResponseEntity I can set the body but can't tell the size of the response.
How can I pad the response to the required size while setting the fields above?
First of all, as JB Nizet said, this requirement is fully outside of the HTTP protocol and you'd better fix the client side to only control the size of the body part (what the content-length header is made for).
Now assuming you really need to do that, I can imagine 2 ways to fulfil this requirement (both of them seem equally ugly...):
Use a dedicated proxy to post-process the HTTP response
Consistently add a custom header to the response that indicates the total size the response should have. Put a dedicated proxy between the clients and the server. That proxy should listen on its own port and forward everything to the server. For the response part, it should:
read the header part line by line (delimited with \r\n), store the required total size without transmitting it and forward all other headers storing the number of bytes sent
once the header part is over (empty line), read the body part and trunk or pad it to the correct size before transmitting it.
This would be a low level program that should use directly the socket interfaces, or that could directly be written in C. This really looks like plumbing, but it should be usable even for different servlet containers.
Compute the body part size and guess the header one
Control how your servlet container processes headers. The protocol requires that they are written as NAME=value\r\n and that the header part is followed by an empty line (r\n). But you should control twice whether the containers adds its own headers or whether it automatically adds some headers if you do not provide them. That should allow you to compute the header size from the headers you added to the response, but is clear coupled to a single servlet container, and when used the same way.
Alternatively, you could try to ask the response what headers it contains. Normally (at least Tomcat does it), it actually computes the header part when it commits the response (*). So you could:
set the status and the headers you need
commit the response by a call to flushBuffer()
get the generated headers through:
for (String name: resp.getHeaderNames()) {
for (String value: resp.getHeaders(name)) {
...
If you take care of the status line (should be HTTP/1.1 200 OK\r\n but here again control twice its actual size), it should be enough to compute the total header size. Provided you have enough control on the body to know what you want to write, you should be able to compute how much padding you need.
This avoids a dedicated proxy, but it can only be guaranteed to work on a (version of a) particular servlet container
(*) Beware: when you commit the response manually, the servlet container can add a Transfer-Encoding=chunked header because it cannot guess the body size. Whether it is acceptable for you use case or not, I cannot know...
Considering the comment discussion above, you could create an HttpResponse Wrapper.
Using your wrapper you can then intercept the response and override the actual size.
More info: your response wrapper writes response content to an internal byte array. It is not actually written to the "real" http output stream. In the filter you make sure 'write' and 'flush' called on the wrapper don't exceed your preferred max length.
Having said this, I agree with #jb-nijet that this is not common and looks like a flaw.
I am building a small client server connection where I need to send some payload and get a reply back.
I am using Java 7 Async IO.
My requirement is
to receive the payload
parse the request
do some processing
write the response back
How can I mark the end of request from the client without closing the connection. I can see two options,
Send -1,this implies another read step from server
parse the request bytes as they come to get to know the end, this I want to avoid.
Is there any better way?
Usually you will have either data separator (your option 1), but you have to be ABSOLUTELY sure that this data separator will NEVER be present in your data packet
or
you can prepend the packet with size information (i.e. have an equivalent of packet header)
I have created a HTTP server which works perfectly! No problems there,
using Sockets and ServerSockets.
Now I need to make the Form entries - which are done in the web browser and when they are submitted, they are processed by Java.
I've tried making a form, then pressing sumbit, except it's given a 501 error... The first part is 'POST /'. I suppose that means a hidden variable passed... But how do I make the server recognise the hidden variable, and make it process it?
Is there any way to do this?
How you decode the parameters depends on a number of things. If the request is a POST request then the values are part of the request body (this is, the values are sent after a blank link that separates the head and the body of the request). The encoding of the request body is then either "application/x-www-form-urlencoded" or "multipart/form-data". You can read how these two encodings work here: http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4
I assume that this is a toy project and that you want to write this code yourself. Otherwise there are many server implementations of the HTTP protocol for Java out there...
I'm using org.apache.commons.httpclient.HttpClient and need to setup response encoding (for some reason server returns incorrect encoding in Content-Type). My way is to get response as raw bytes and convert to String with desired encoding. I'm wondering if there is some better way to do this (eg. setup HttpClient). Thanks for suggestions.
I don't think there's a better answer using HttpClient 3.x APIs.
The HTTP 1.1 spec says clearly that a client "must" respect the character set specified in the response header, and use ISO-8859-1 if no character set is specified. The HttpClient APIs are designed on the assumption that the programmer wants to conform to the HTTP specs. Obviously, you need to break the rules in the spec so that you can talk to the non-compliant server. Not withstanding, this is not a use-case that the API designers saw a need to support explicitly.
If you were using the HttpClient 4.x, you could write your own ResponseHandler to convert the body into an HttpEntity, ignoring the response message's notional character set.
A few notes:
Server serves data, so it's up to server to serve it in an appropriate format. So response encoding is set by server not client. However, client could suggest to server what format it would like via Accept and Accept-Charset:
Accept: text/plain
Accept-Charset: utf-8
However, http servers usually do not convert between formats.
If option 1. does not work, then you should look at the configuration of the server.
When String is sent as raw bytes (and it always is, because this is what networks transmit), there is always the encoding defined. Since server produces this raw bytes, it defines the encoding. So, you can not take raw bytes and use encoding of your choice to create a String. You must use encoding that was used when converted from String to bytes.
Disclaimer: I'm not really knowing HttpClient, only reading the API.
I would use the execute method returning a HttpResponse, then .getEntity().getContent(). This is a pure byte stream, so if you want to ignore the encoding told by the server, you can simply wrap your own InputStreamReader around it.
Okay, looks like I had the wrong version (obviously, there are too much HttpClient classes out there).
But same as before, just located on other classes: the HttpMethod has a getResponseBodyAsStream() method, around which you can now wrap your own InputStreamReader. (Or get the whole array at once, if it is not too big, and convert it to String, as you wrote.)
I think trying to change the response and letting the HttpClient analyze it is not the right way here.
I suggest sending a message to the server administrator/webmaster about the wrong charset, though.
Greetings folks,
Jus in case someone finds this post googling for setting HttpClient to write in UTF-8.
This line of code should be handy...
response.setContentType("text/html; charset=UTF-8");
Best