Let's assume I have SocketChannel is waiting for read.
I don't want to read the first K bytes (as in, I don't want to copy them to a byte buffer, just skip them).
Is there anyway of doing so? there is no skip() method available.
I'm not sure if this is what you want to achieve, or even if it works as you want to, but you can get a Socket and then an InputStream from it.
Since you want to skip bytes, seems that work with I/O streams is the best solution.
SocketChannel sock = SocketChannel.open();
sock.connect(new InetSocketAddress("your addres or something", 80));
InputStream ins = sock.socket().getInputStream();
//Skip here
ins.skip(5);
Edit
No, you cannot reuse what you have consumed with the input stream.
By the way, could you be more precise with "The allocation" problem? You can pipe the InputStream, so why do you say that you have to allocate a buffer? What is the target at the end?
Edit 2
There's an example that does not need to allocate K bytes
ins.skip(1);
ArrayList<Byte> bts = new ArrayList<>();
int last = 0;
while (last != -1) {
last = in.read();
bts.add((byte)last);
}
System.out.println(bts.stream().map(b -> new String(new byte[]{b.byteValue()})).collect(Collectors.joining()));
Related
My goal is to read the n number of bytes from a Socket.
Is it better to directly read from the InputStream, or wrap it into a BufferedReader?
Throughout the net you find both approaches, but none states which to use when.
Socket socket;
is = socket.getInputStream();
BufferedReader br = new BufferedReader(new InputStreamReader(is));
char[] buffer = new char[CONTENT_LENGTH];
//what is better?
is.read(buffer);
br.read(buffer);
Since your goal is to "read the n number of bytes" there is little point creating a character Reader from your input as this might mean the nth byte is part way into a character - and assuming that the stream is character based.
Since JDK11 there is handy call for reading n bytes:
byte[] input = is.readNBytes(n);
If n is small and you repeat the above often, consider reading the stream using one of bis = new BufferedInputStream(is), in.transferTo(out) or len = read(byteArray) which may be more effective for longer streams.
I have a problem with socket programming in Java.
There is a server which has been written in python like this which I shouldn't not change.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send('from server\nnewline\0')
data = s.recv(BUFFER_SIZE)
s.close()
Now I want to write a code in Java which read the string from a server. Something like this:
public static String readStr(Socket client) throws IOException {
InputStreamReader inStream = new InputStreamReader(
client.getInputStream());
BufferedReader inBuff = new BufferedReader(inStream);
String answer = new String();
String str = inBuff.readLine();
while (str!=null) {
answer = answer.concat(str + "\n");
str = inBuff.readLine();
}
answer = answer.substring(0, answer.length() - 1);
System.out.println("answer:\n "+answer);
return answer;
}
But it seems that it blocks at line str = inBuff.readLine(); at the last line of the message. I tried the read() method but it was blocked too.
When designing a protocol over tcp, the best way is to include some kind of framer. This is done in the current protocol by the usage of a NUL byte.
When reading the data from the socket, you should first divide it into blocks/frames by some operations, before parsing the individual blocks.
A crude way to divide the packets into blocks is reading until you find a NUL byte, then returning that block as a byte array. (This is not the most efficient implementation)
public byte[] readPacket(InputStream in) throws IOException {
ByteArrayOutputStream tempStr = new ByteArrayOutputStream();
int read;
read=in.read();
while(read > 0){
tempStr.write(read);
read=in.read();
}
if(read == -1)
throw new EOFException();
return tempStr.toByteArray();
}
Because you now have proper frames for your data, you can now easily read the data:
byte[] frame = readPacket(in);
String[] lines = new String(frame, StandardCharsets.UTF8).split("\n");
// Do something with the lines
This is probably because the last line sent by the server does not end and readLine() method only returns when it reaches end of the line. Since you change the server's code. I recommend you use another method for reading from the stream. You may also use InputStreamReader class.
Apart from already mentioned inconsistent message/line ending - once with \n, second with \0, at the server there is no detection of end of the message. So the server will loop as long as the socket is not closed (or shut down for writing) at the client side. And as you have this line before closing the socket:
data = s.recv(BUFFER_SIZE)
in other words the client is waiting for some response from the server. But the server is stuck forever reading the message from the client in a loop.
So you need to either close the socket prior to that recv call or send some message (like empty line) and detect in on the server and eventually exit the loop.
I am trying to read a stream on a SocketChannel without defining the number of bytes.
The alternate solution i thought about is storing different ByteBuffers of a pre-defined size into a list which will allow me afterwards to allocate a new ByteBuffer of the received size and put the result inside.
The problem is that i am on blocking-mode and cannot find a valid condition to leave the loop i made on the read method check the code:
public static final Charset charsetUTF8 = Charset.forName("UTF-8");
public static final int BUFFER_SIZE = 1024;
public static String getUnbounded(String st, SocketAddress address) throws IOException {
SocketChannel sc = SocketChannel.open(address);
sc.write(charsetUTF8.encode(st));
List<ByteBuffer> listBuffers = new ArrayList<>();
ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
while( sc.read(buff) > -1){
if(buff.remaining() == 0){
listBuffers.add(buff);
buff.clear();
}
}
listBuffers.add(buff);
ByteBuffer finalBuffer = ByteBuffer.allocate(BUFFER_SIZE * listBuffers.size());
for(ByteBuffer tempBuff: listBuffers){
finalBuffer.put(tempBuff);
tempBuff.clear();
}
finalBuffer.flip();
return charsetUTF8.decode(finalBuffer).toString();
}
Any idea on how to solve this?
You can't just clear() the byte buffer. You need to allocate a new one; otherwise the same buffer is being added to listBuffers repeatedly.
ByteBuffer buff = ByteBuffer.allocate(BUFFER_SIZE);
while( sc.read(buff) > -1){
if(buff.remaining() == 0){
listBuffers.add(buff);
buff = ByteBuffer.allocate(BUFFER_SIZE);
}
}
if (buff.position() > 0) {
listBuffers.add(buff);
}
Since the last buffer might not (probably will not) be full, you should calculate the finalBuffer size taking this into account.
The number of bytes in an HTTP response stream is not 'undefined'. See the RFC. It is defined by either:
EOS in the case of a connection which is closed (HTTP 1.0 or Connection: close),
The Content-Length header, or
The result of decoding the chunked-encoding format.
It is essential that it be defined in one of these ways, and maybe there are others, so that HTTP persistent connections can work, where there may be another response following this one.
I would like to know why you are implementing this at all, when the HttpURLConnection class already exists, along with various third-party HTTP clients, which already implement all this correctly, and many other things besides.
The solution is that to get out of the loop i had to call:
sc.shutdownOutput();
Which closes the writing stream without closing the reading stream and set the sc.read(buff) to -1
Ok, So I'm making a Java program that has a server and client and I'm sending a Zip file from server to client. I have sending the file down, almost. But recieving I've found some inconsistency. My code isn't always getting the full archive. I'm guessing it's terminating before the BufferedReader has the full thing. Here's the code for the client:
public void run(String[] args) {
try {
clientSocket = new Socket("jacob-custom-pc", 4444);
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedInputStream(clientSocket.getInputStream());
BufferedReader inRead = new BufferedReader(new InputStreamReader(in));
int size = 0;
while(true) {
if(in.available() > 0) {
byte[] array = new byte[in.available()];
in.read(array);
System.out.println(array.length);
System.out.println("recieved file!");
FileOutputStream fileOut = new FileOutputStream("out.zip");
fileOut.write(array);
fileOut.close();
break;
}
}
}
} catch(IOException e) {
e.printStackTrace();
System.exit(-1);
}
}
So how can I be sure the full archive is there before it writes the file?
On the sending side write the file size before you start writing the file. On the reading side Read the file size so you know how many bytes to expect. Then call read until you have gotten everything you expect. With network sockets it may take more than one call to read to get everything that was sent. This is especially true as your data gets larger.
HTTP sends a content-length: x+\n in bytes. This is elegant, it might throw a TimeoutException if the conn is broken.
You are using a TCP socket. The ZIP file is probably larger than the network MTU, so it will be split up into multiple packets and reassembled at the other side. Still, something like this might happen:
client connects
server starts sending. The ZIP file is bigger than the MTU and therefore split up into multiple packets.
client busy-waits in the while (true) until it gets the first packets.
client notices that data has arrived (in.available() > 0)
client reads all available data, writes it to the file and exits
the last packets arrive
So as you can see: Unless the client machine is crazily slow and the network is crazily fast and has a huge MTU, your code simply won't receive the entire file by design. That's how you built it.
A different approach: Prefix the data with the length.
Socket clientSocket = new Socket("jacob-custom-pc", 4444);
DataInputStream dataReader = new DataInputStream(clientSocket.getInputStream());
FileOutputStream out = new FileOutputStream("out.zip");
long size = dataReader.readLong();
long chunks = size / 1024;
int lastChunk = (int)(size - (chunks * 1024));
byte[] buf = new byte[1024];
for (long i = 0; i < chunks; i++) {
dataReader.read(buf);
out.write(buf);
}
dataReader.read(buf, 0, lastChunk);
out.write(buf, 0, lastChunk);
And the server uses DataOutputStream to send the size of the file before the actual file. I didn't test this, but it should work.
How can I make sure I received whole file through socket stream?
By fixing your code. You are using InputStream.available() as a test for end of stream. That's not what it's for. Change your copy loop to this, which is also a whole lot simpler:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Use with any buffer size greater than zero, typically 8192.
In.available() just tells you that there is no data to be consumed by in.read() without blocking (waiting) at the moment but it does not mean the end of stream. But, they may arrive into your PC at any time, with TCP/IP packet. Normally, you never use in.available(). In.read() suffices everything for the reading the stream entirely. The pattern for reading the input streams is
byte[] buf;
int size;
while ((size = in.read(buf)) != -1)
process(buf, size);
// end of stream has reached
This way you will read the stream entirely, until its end.
update If you want to read multiple files, then chunk you stream into "packets" and prefix every one with an integer size. You then read until size bytes is received instead of in.read = -1.
update2 Anyway, never use in.available for demarking between the chunks of data. If you do that, you imply that there is a time delay between incoming data pieces. You can do this only in the real-time systems. But Windows, Java and TCP/IP are all these layers incompatible with real-time.
I would like to send image file from java server to android app using this code:
Server(Java):
File file = new File("./clique.jpg");
FileInputStream stream = new FileInputStream(file);
DataOutputStream writer = new DataOutputStream(socket.getOutputStream());
byte[] contextB = new byte[4096];
int n;
int i = 0;
while ( (n=stream.read(contextB))!=-1 ){
writer.write(contextB, 0, n);
writer.flush();
System.out.println(n);
i+=n;
}
writer.flush();
stream.close();
android app:
DataInputStream reader = new DataInputStream(socket.getInputStream());
byte[] buffer = new byte[4096];
ByteArrayOutputStream content = new ByteArrayOutputStream();
int n;
int i = 0;
reader = new DataInputStream(socket.getInputStream());
while ( (n=reader.read(buffer)) != null){
content.write(buffer, 0, n);
content.flush();
}
Utility.CreateImageFile(content.toByteArray());
What I noticed is that in the android app n which is the number of read bytes is not 4096 while I am sending from server byte blocks of 4096 size,also I can not get n=-1 which is the end of stream,it blocks until I close the app then I get n=-1.
Regarding the number of bytes you read at a time has nothing to do with the number of bytes you write -it very much depends on the network conditions and will be variable with every chunk (basically as many bytes managed to be transmitted in short period of time between your reads as many you will get in the read chunk.
Regarding the end of stream - in your server code you have forgotten to close the output stream (you only close the stream which is input stream - you should also close the writer which in turn will close the underlying output stream.
Two comments:
1) I would really recommend to use Buffered Readers/Writers wrapping the writes/readers - the code you will get will be nicer and you will not have to create/manage buffers yourself.
2) Use try {} finally and close your streams in finally clauses - this is the best practice that will make sure that you will close the streams and free resources even in case of problems while reading/writing.
You got a problem in your android code:
while ( (n=reader.read(buffer)) != null) {
n can not be null.
Use writer.close() instead of writer.flush() after your loop on the server.