I am trying to read the http header from the client using input stream but it does not seem to terminate. I wonder if there is any problem to read the request this way. I wanted this to work on both GET and POST request. Here is my code
private static byte[] readClientRequest(byte[] clientRequest, InputStream fromClient) throws IOException{
int bytesRead;
ByteArrayOutputStream clientByteArray = new ByteArrayOutputStream();
while((bytesRead = fromClient.read()) != -1) {
clientByteArray.write(bytesRead);
}
clientByteArray.flush();
clientRequest = clientByteArray.toByteArray();
return clientRequest;
}
Thanks a lot.
In general network socket does not known where the end is. It may be application protocol specific. For HTTP you can read one header line until end of line (bytesRead != '\n'). Then you should use available method to check that there is next part of data.
Related
I have a problem with socket programming in Java.
There is a server which has been written in python like this which I shouldn't not change.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send('from server\nnewline\0')
data = s.recv(BUFFER_SIZE)
s.close()
Now I want to write a code in Java which read the string from a server. Something like this:
public static String readStr(Socket client) throws IOException {
InputStreamReader inStream = new InputStreamReader(
client.getInputStream());
BufferedReader inBuff = new BufferedReader(inStream);
String answer = new String();
String str = inBuff.readLine();
while (str!=null) {
answer = answer.concat(str + "\n");
str = inBuff.readLine();
}
answer = answer.substring(0, answer.length() - 1);
System.out.println("answer:\n "+answer);
return answer;
}
But it seems that it blocks at line str = inBuff.readLine(); at the last line of the message. I tried the read() method but it was blocked too.
When designing a protocol over tcp, the best way is to include some kind of framer. This is done in the current protocol by the usage of a NUL byte.
When reading the data from the socket, you should first divide it into blocks/frames by some operations, before parsing the individual blocks.
A crude way to divide the packets into blocks is reading until you find a NUL byte, then returning that block as a byte array. (This is not the most efficient implementation)
public byte[] readPacket(InputStream in) throws IOException {
ByteArrayOutputStream tempStr = new ByteArrayOutputStream();
int read;
read=in.read();
while(read > 0){
tempStr.write(read);
read=in.read();
}
if(read == -1)
throw new EOFException();
return tempStr.toByteArray();
}
Because you now have proper frames for your data, you can now easily read the data:
byte[] frame = readPacket(in);
String[] lines = new String(frame, StandardCharsets.UTF8).split("\n");
// Do something with the lines
This is probably because the last line sent by the server does not end and readLine() method only returns when it reaches end of the line. Since you change the server's code. I recommend you use another method for reading from the stream. You may also use InputStreamReader class.
Apart from already mentioned inconsistent message/line ending - once with \n, second with \0, at the server there is no detection of end of the message. So the server will loop as long as the socket is not closed (or shut down for writing) at the client side. And as you have this line before closing the socket:
data = s.recv(BUFFER_SIZE)
in other words the client is waiting for some response from the server. But the server is stuck forever reading the message from the client in a loop.
So you need to either close the socket prior to that recv call or send some message (like empty line) and detect in on the server and eventually exit the loop.
This question already has answers here:
Determine the size of an InputStream
(13 answers)
Closed 3 years ago.
I'm having an InputStream from a ProcessBuilder that acutally reads the stdout stream.
Question: how can I know the size of that inmemory InputStream, so I can write it to a HttpResponse http header?
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
OutputStream out = response.getOutputStream();
int bytes;
while ((bytes = br.read()) != -1) {
out.write(bytes);
}
//how can I know the size of the inmemory stream/file written?
//response.setContentLength((int) pdfFile.length());
There is no such thing as the size of an input stream. Consider a program which never exits, or a socket peer which never stops sending. And you don't need to know to write it to an HttpResponse header. The Content-length is managed automatically for you.
Try this
InputStream is = process.getInputStream();
ByteArrayOutputStream os = new ByteArrayOutputStream();
int b;
while ((b = is.read()) != -1)
os.write(b);
response.setContentLength(os.size());
response.getOutputStream().write(os.toByteArray());
If you really want to set the content length header, you'll need to read the entire stream before writing to the response OutputStream
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] bytes = new byte[1024];
int count;
while ((count = in.read(bytes)) > 0) {
out.write(bytes, 0, count);
}
response.setContentLength(out.size();
out.writeTo(response.getOutputStream());
Note: With this approach you've now read the entire stream into memory, this will have an impact on available memory and likely won't scale well.
import org.apache.commons.io.IOUtils;
byte[] bytes = IOUtils.toByteArray(inputStream);
log.message("bytes .lenght "+bytes.length);
if (bytes.length > 400000)
//some byte range limit`enter code can add any byte range
{
throw new Exception("File Size is larger than 40 MB ..");
}
An InputStream inherently doesn't have a size. It could conceivably keep delivering bytes forever. Or the producing end could end the stream without warning.
If you must find out the length, then you have to read to the end, counting the bytes, and report the length when you finish.
You're worrying about HTTP's Content-length header, and you've got a point. The fact is that the original version of HTTP was not designed for large, dynamically generated content. The protocol inherently expects you to know the size of the content before you start writing it - yet how is that possible if it's (for example) an ongoing chat, or the output of a video camera?
The solution is HTTP's chunked transfer encoding. Here you don't set a Content-Length header. You set Transfer-Encoding: chunked, then write the content as chunks, each of which has a size header.
The HTTP RFC has details one this, or https://en.wikipedia.org/wiki/Chunked_transfer_encoding is slightly more friendly.
However most HTTP APIs hide this detail from you. Unless you are developing a web library from scratch (perhaps for academic reasons), you shouldn't have to think about Content-Length or Transfer-Encoding.
I am trying to read a stream on a SocketChannel without defining the number of bytes.
The alternate solution i thought about is storing different ByteBuffers of a pre-defined size into a list which will allow me afterwards to allocate a new ByteBuffer of the received size and put the result inside.
The problem is that i am on blocking-mode and cannot find a valid condition to leave the loop i made on the read method check the code:
public static final Charset charsetUTF8 = Charset.forName("UTF-8");
public static final int BUFFER_SIZE = 1024;
public static String getUnbounded(String st, SocketAddress address) throws IOException {
SocketChannel sc = SocketChannel.open(address);
sc.write(charsetUTF8.encode(st));
List<ByteBuffer> listBuffers = new ArrayList<>();
ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
while( sc.read(buff) > -1){
if(buff.remaining() == 0){
listBuffers.add(buff);
buff.clear();
}
}
listBuffers.add(buff);
ByteBuffer finalBuffer = ByteBuffer.allocate(BUFFER_SIZE * listBuffers.size());
for(ByteBuffer tempBuff: listBuffers){
finalBuffer.put(tempBuff);
tempBuff.clear();
}
finalBuffer.flip();
return charsetUTF8.decode(finalBuffer).toString();
}
Any idea on how to solve this?
You can't just clear() the byte buffer. You need to allocate a new one; otherwise the same buffer is being added to listBuffers repeatedly.
ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
while( sc.read(buff) > -1){
if(buff.remaining() == 0){
listBuffers.add(buff);
buff = ByteBuffer.allocate(BUFFER_SIZE);
}
}
if (buff.position() > 0) {
listBuffers.add(buff);
}
Since the last buffer might not (probably will not) be full, you should calculate the finalBuffer size taking this into account.
The number of bytes in an HTTP response stream is not 'undefined'. See the RFC. It is defined by either:
EOS in the case of a connection which is closed (HTTP 1.0 or Connection: close),
The Content-Length header, or
The result of decoding the chunked-encoding format.
It is essential that it be defined in one of these ways, and maybe there are others, so that HTTP persistent connections can work, where there may be another response following this one.
I would like to know why you are implementing this at all, when the HttpURLConnection class already exists, along with various third-party HTTP clients, which already implement all this correctly, and many other things besides.
The solution is that to get out of the loop i had to call:
sc.shutdownOutput();
Which closes the writing stream without closing the reading stream and set the sc.read(buff) to -1
I have an ipcamera that whenever multiple of users are connecting to it it becomes too slow.
I was thinking about getting the stream from the camera with my server and multiple of clients should be able to stream from the server instead of the poor ipcamera.
i set up a quick and dirty servlet just too see if it works :
#RequestMapping(value = "/", method = RequestMethod.GET, produces = "application/x-shockwave-flash")
public String getVideoStream(Locale locale, Model model, HttpServletRequest request, HttpServletResponse response) throws IOException {
logger.info("Start");
// An IPcamera stream example
URL url = new URL("http://www.earthcam.com/swf/ads5.swf");
URLConnection yc = url.openConnection();
OutputStream out = response.getOutputStream();
InputStream in = yc.getInputStream();
String mimeType = "application/x-shockwave-flash";
byte[] bytes = new byte[100000];
int bytesRead;
response.setContentType(mimeType);
while ((bytesRead = in.read(bytes)) != -1) {
out.write(bytes, 0, bytesRead);
}
logger.info("End");
I believe this might work, my problem right now is that :
bytesRead = in.read(bytes)
reads only 61894 bytes and that's it :( why is that happening? am i trying to get the stream wrong?
btw: i tried to do this with xuggler, but i had an error that compressed-SWF not supported.
thanks
Your code is working perfectly. I just fetched ads5.swf from your server and it is, indeed, 61894 bytes in length. The problem you're facing is that the SWF file is just the movie player. After being downloaded, the player then fetches the video stream from the server. By default (if this is some kind of turn-key streaming solution), it's probably trying to get the stream from the same server where the SWF comes from.
Can any one of you solve this problem !
Problem Description:
i have received content-encoding: gzip header from http web-server.
now i want to decode the content but when i use GZIP classes from jdk 1.6.12, it gives null.
does it means that contents are not in gzip format ? or are there some another classes for decompress http response content?
Sample Code:
System.out.println("Reading InputStream");
InputStream in = httpuc.getInputStream();// httpuc is an object of httpurlconnection<br>
System.out.println("Before reading GZIP inputstream");
System.out.println(in);
GZIPInputStream gin = new GZIPInputStream(in));
System.out.println("After reading GZIP inputstream");
Output:
Reading InputStream
Before reading GZIP inputstream
sun.net.www.protocol.http.HttpURLConnection$HttpInputStream#8acf6e
null
I have found one error in code, but don't able to understand it properly. what does it indicates.
Error ! java.io.EOFException
Thanks
I think you should have a look at HTTPClient, which will handle a lot of the HTTP issues for you. In particular, it allows access to the response body, which may be gzipped, and then you simply feed that through a GZIPInputStream
e.g.
Header hce = postMethod.getResponseHeader("Content-Encoding");
InputStream in = null;
if(null != hce)
{
if(hce.getValue().equals(GZIP)) {
in = new GZIPInputStream(postMethod.getResponseBodyAsStream());
}
// etc...
I second Brian's suggestion. Whenever u need to deal with getting/posting stuff via HTTP don't bother with low-level access use the Apache HTTP client.
InputStream is = con.getInputStream();
InputStream bodyStream = new GZIPInputStream(is);
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int length;
while ((length = bodyStream.read(buffer)) > 0) {
outStream.write(buffer, 0, length);
}
String body = new String(outStream.toByteArray(), "UTF-8");