In my application I upload a byte[] (serialized Object) to my FTP server, which is working perfectly. However when I try to download it only the first part (like 3000 bytes) of the array are correct, the rest is filled with zeros.
I can't seem to figure out what is wrong, any help would be appreciated.
I am using the package
org.apache.commons.net.*
public static byte[] downloadBoard( String host, int port, String usr, String pwd) throws IOException {
FTPClient ftpClient = new FTPClient();
byte[] buf = new byte[20000];
try {
ftpClient.connect( host, port );
ftpClient.login( usr, pwd );
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
InputStream is = ftpClient.retrieveFileStream("asdf.board");
is.read(buf);
is.close();
ftpClient.completePendingCommand();
ftpClient.logout();
} finally {
ftpClient.disconnect();
}
return buf;
}
is.read() may not return the full content. You'll need to put read() into a loop similar to this:
int pos = 0;
while (true) {
int count = is.read(buf, pos, buf.length - pos);
if (count <= 0) {
break;
}
pos += count;
}
P.S.:
If you know the size of the file, you can use a DataInputStream to read the buffer without a loop:
byte[] buf = new byte[exactFileSize];
DataInputStream dis = new DataInputStream(is);
dis.readFully(buf);
InputStream.read() typically doesn't read the entire stream for you, only some part of it. Note that InputStream.read() returns the number of bytes actually read, which you'll need to check.
The typical pattern is to loop until the InputStream has reported that no more bytes are available.
Related
I need to upload a very large file from my machine to a server. (a few GB)
Currently, I tried the below approach but I keep getting.
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
I can increase the memory but this is not something I want to do because not sure where my code will run. I want to read a few MB/kb send them to the server and release the memory and repeat. tried other approaches like Files utils or IOUtils.copyLarge but I get the same problem.
URL serverUrl =
new URL(url);
HttpURLConnection urlConnection = (HttpURLConnection) serverUrl.openConnection();
urlConnection.setConnectTimeout(Configs.TIMEOUT);
urlConnection.setReadTimeout(Configs.TIMEOUT);
File fileToUpload = new File(file);
urlConnection.setDoOutput(true);
urlConnection.setRequestMethod("POST");
urlConnection.addRequestProperty("Content-Type", "application/octet-stream");
urlConnection.connect();
OutputStream output = urlConnection.getOutputStream();
FileInputStream input = new FileInputStream(fileToUpload);
upload(input, output);
//..close streams
private static long upload(InputStream input, OutputStream output) throws IOException {
try (
ReadableByteChannel inputChannel = Channels.newChannel(input);
WritableByteChannel outputChannel = Channels.newChannel(output)
) {
ByteBuffer buffer = ByteBuffer.allocateDirect(10240);
long size = 0;
while (inputChannel.read(buffer) != -1) {
buffer.flip();
size += outputChannel.write(buffer);
buffer.clear();
}
return size;
}
}
I think it has something to do with this but I can't figure out what I am doing wrong.
Another approach was but I get the same issue:
private static long copy(InputStream source, OutputStream sink)
throws IOException {
long nread = 0L;
byte[] buf = new byte[10240];
int n;
int i = 0;
while ((n = source.read(buf)) > 0) {
sink.write(buf, 0, n);
nread += n;
i++;
if (i % 10 == 0) {
log.info("flush");
sink.flush();
}
}
return nread;
}
Use setFixedLengthStreamingMode as per this answer on the duplicate question Denis Tulskiy linked to:
conn.setFixedLengthStreamingMode((int) fileToUpload.length());
From the docs:
This method is used to enable streaming of a HTTP request body without internal buffering, when the content length is known in advance.
At the moment, your code is attempting to buffer the file into Java's heap memory in order to compute the Content-Length header on the HTTP request.
I am using Commons-IO to read and write from Socket. Things all works till payload size is either 1448/2896 max.
Below is the code snippet. Really unsure how to handle it.
Checked system buffer size too
$ cat /proc/sys/net/ipv4/tcp_wmem
4096 16384 4194304
public static void usingCommonsIO(){
Socket socket = null;
try {
socket = new Socket(serverIP, 55000);
IOUtils.write(request.getBytes(), socket.getOutputStream());
System.out.println("Message Sent....");
StringBuilder response = new StringBuilder();
String resp =IOUtils.toString(socket.getInputStream(), "UTF-8");
System.out.println(resp);
} catch (IOException e) {
e.printStackTrace();
}
}
Alternatively tried using DataInputStream but no luck. Code snipped is below.
public static void usingDataIOStream(String requestStr){
Socket socket = null;
try {
socket = new Socket("192.168.1.6", 55000);
System.out.println("Request Length -:" + request.length());
DataOutputStream out = new DataOutputStream(socket.getOutputStream());
out.write(requestStr.getBytes("UTF-8"), 0, requestStr.length());
out.flush();
System.out.println("Message Sent....");
DataInputStream din = new DataInputStream(socket.getInputStream());
byte[] response = new byte[16*1024];
int responseLength = din.read(response);
System.out.println("Response -:" + new java.lang.String(response, 0, responseLength));
} catch (IOException e) {
e.printStackTrace();
}
}
Confusing part is that the same code works with only 1448 bytes sometimes and max of 2896 bytes sometimes. There are no specific patterns.
Update 1
To simulate it, tried writing Server socket on my own and code is as below. Strange thing noticed with this is, on first request payload of size 6500 was read and received properly. Connection Reset from second request onwards. Am I missing something here?
public static void usingBAOS() throws IOException {
server = new ServerSocket(port);
Socket socket = null;
DataInputStream din = null;
DataOutputStream dos = null;
while (true) {
System.out.println("Waiting for Client...");
try {
// Accepting Client's connection
socket = server.accept();
System.out.println("Connnected to client " + socket.getInetAddress());
din = new DataInputStream(socket.getInputStream());
// Read request payload from Socket
String requestString = readRequest(din);
System.out.println("Request Read.....");
System.out.println("Writing Response.....");
// Writing response to socket
dos = writeResponse(socket, requestString);
} catch (IOException e) {
e.printStackTrace();
}finally {
//close resources
din.close();
System.out.println("InputStream is closed......");
dos.close();
System.out.println("OutputStream is closed......");
socket.close();
System.out.println("Socket is closed......");
}
}
}
private static DataOutputStream writeResponse(Socket socket, String requestString) throws IOException {
String responseString = "Hi Client" + requestString;
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
//write object to Socket
dos.write(responseString.getBytes(),0, responseString.getBytes().length);
dos.flush();
return dos;
}
private static String readRequest(DataInputStream din) throws IOException {
byte[] response = new byte[16*1024];
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int n = 0;
boolean read = true;
while(read){
n = din.read(response);
baos.write(response, 0, n);
if(baos.toString().length() == n){
read = false;
}
}
baos.flush();
String requestString = baos.toString();
return requestString;
}
Although this question is old at the time of writing this answer I'm putting this here for others in case it solves their problem. I encountered the same issue when using buffered data input and output streams on Android 8.0 devices where I had naively assumed that doing this:
int len = 2304;
byte[] data = new byte[len];
inputStream.read(data, 0, len);
would read all the data I sent down the socket. But as suggested by #Kayaman in the comments, this does not guarantee that len bytes of data are actually read from the buffer even if there are bytes available. In fact, this is in the documentation:
public final int read(byte[] b, int off, int len) throws IOException
Reads up to len bytes of data from the contained input stream into an array of bytes. An attempt is made to read as many as len bytes, but a smaller number may be read, possibly zero. The number of bytes actually read is returned as an integer.
In fact, if it doesn't read all the data, the only way to tell is to capture the returned value. My solution was then to monitor the amount of bytes actually read from the stream and just call read() in a loop as:
int i = 0;
len = 2304;
byte[] data = new byte[len];
while (i < len)
{
i += socket.inputStream.read(data, i, len - i);
}
Hope this helps someone.
So I was implementing client and socket for java. I wanted to send huge files on tcp through sockets and I was able to send files too but the only problem was the files on the other end were either not complete or not working. I have checked the bits are being transfered then what is the error.
Client side:
Socket sock = new Socket("127.0.0.1", 1056);
byte[] mybytearray = new byte[1024];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream("abc.mp3");
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(mybytearray, 0, mybytearray.length);
int len = 0;
while((len = is.read(mybytearray)) != -1)
{
bos.write(mybytearray, 0, len);
System.out.println("sending");
}
bos.close();
sock.close();
Server side:
ServerSocket ss = new ServerSocket(1056);
while (true) {
Socket s = ss.accept();
PrintStream out = new PrintStream(s.getOutputStream());
BufferedReader in = new BufferedReader(new InputStreamReader(s.getInputStream()));
String info = null;
String request = null;
System.out.println("sending");
String filename = "abc.mp3";
File fi = new File(filename);
InputStream fs = new FileInputStream(fi);
int n = fs.available();
byte buf[] = new byte[1024];
out.println("Content_Length:" + n);
out.println("");
while ((n = fs.read(buf)) >= 0) {
out.write(buf, 0, n);
System.out.println("sending");
}
out.close();
s.close();
in.close();
}
When you are connected via TCP you create a network stream which you can read and write in, similar to all other streams you worked with. Writing a large amount of data to the stream is not a good idea, so I suggest you break the selected file into smaller packets in which each packet length is 1024 bytes (1KB) and then send all the packets to the server. The SendTCP function is as follows:(I have used Windows Forms to make things more obvious)
public void SendTCP(string M, string IPA, Int32 PortN)
{
byte[] SendingBuffer = null
TcpClient client = null;
lblStatus.Text = "";
NetworkStream netstream = null;
try
{
client = new TcpClient(IPA, PortN);
lblStatus.Text = "Connected to the Server...\n";
netstream = client.GetStream();
FileStream Fs = new FileStream(M, FileMode.Open, FileAccess.Read);
int NoOfPackets = Convert.ToInt32
(Math.Ceiling(Convert.ToDouble(Fs.Length) / Convert.ToDouble(BufferSize)));
progressBar1.Maximum = NoOfPackets;
int TotalLength = (int)Fs.Length, CurrentPacketLength, counter = 0;
for (int i = 0; i < NoOfPackets; i++)
{
if (TotalLength > BufferSize)
{
CurrentPacketLength = BufferSize;
TotalLength = TotalLength - CurrentPacketLength;
}
else
CurrentPacketLength = TotalLength;
SendingBuffer = new byte[CurrentPacketLength];
Fs.Read(SendingBuffer, 0, CurrentPacketLength);
netstream.Write(SendingBuffer, 0, (int)SendingBuffer.Length);
if (progressBar1.Value >= progressBar1.Maximum)
progressBar1.Value = progressBar1.Minimum;
progressBar1.PerformStep();
}
lblStatus.Text=lblStatus.Text+"Sent "+Fs.Length.ToString()+"
bytes to the server";
Fs.Close();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
netstream.Close();
client.Close();
}
}
As you can see, a TCP client and a network stream are being constructed and a network connection is initiated. After opening the selected file according to the buffer size which is 1024 bytes, the number of packets that are going to be sent is calculated. There are two other variables CurrentPacketLength and TotalLength. If the total length of the selected file is more than the buffer size the CurrentPacketLength is set to the buffer size, otherwise why send some empty bytes, so CurrentPacketLength is set to the total length of the file. After that, I subtract the current from the total length, so actually we can say total length is showing the total amount of data that has not been sent yet. The rest is pretty much straight forward, reading the data from the file stream and writing it to the SendingBuffer according to the CurrentPacketLength and writing the buffer to the network stream.
At the server side, the application is listening for an incoming connection:
public void ReceiveTCP(int portN)
{
TcpListener Listener = null;
try
{
Listener = new TcpListener(IPAddress.Any, portN);
Listener.Start();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
byte[] RecData = new byte[BufferSize];
int RecBytes;
for (; ; )
{
TcpClient client = null;
NetworkStream netstream = null;
Status = string.Empty;
try
{
string message = "Accept the Incoming File ";
string caption = "Incoming Connection";
MessageBoxButtons buttons = MessageBoxButtons.YesNo;
DialogResult result;
if (Listener.Pending())
{
client = Listener.AcceptTcpClient();
netstream = client.GetStream();
Status = "Connected to a client\n";
result = MessageBox.Show(message, caption, buttons);
if (result == System.Windows.Forms.DialogResult.Yes)
{
string SaveFileName=string.Empty;
SaveFileDialog DialogSave = new SaveFileDialog();
DialogSave.Filter = "All files (*.*)|*.*";
DialogSave.RestoreDirectory = true;
DialogSave.Title = "Where do you want to save the file?";
DialogSave.InitialDirectory = #"C:/";
if (DialogSave.ShowDialog() == DialogResult.OK)
SaveFileName = DialogSave.FileName;
if (SaveFileName != string.Empty)
{
int totalrecbytes = 0;
FileStream Fs = new FileStream
(SaveFileName, FileMode.OpenOrCreate, FileAccess.Write);
while ((RecBytes = netstream.Read
(RecData, 0, RecData.Length)) > 0)
{
Fs.Write(RecData, 0, RecBytes);
totalrecbytes += RecBytes;
}
Fs.Close();
}
netstream.Close();
client.Close();
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
//netstream.Close();
}
}
}
A TCP listener is created and starts listening to the specified port. Again the buffer size is set to 1024 bytes. A TCP listener can pre check to see if there are any connections pending before calling the AcceptTcpClient method. It returns true if there are any pending connections. This method is a good way of avoiding the socket being blocked. Before reading anything from the network stream, a message box asks you if you want to accept the incoming connection, then a SaveFileDialog will be opened, and when you enter the file name plus extension, a file stream will be constructed and you start reading from the network stream and writing to the file stream. Create a thread in your code and run the receiving method in the created thread. I have sent more than 100 MB files in a LAN with the application.
For more details, check this article.
So, first you do this
int bytesRead = is.read(mybytearray, 0, mybytearray.length);
That reads up to 1024 bytes into mybytearray.
You don't do anything with that and I don't understand why you are doing it. You never write those bytes so they get overwritten if the while loop reads anything.
Just delete that. The while loop should cover all of this.
I'm working on a Java application which will stream video from an IP Camera. The video streams from the IP Camera in MJPEG format. The protocol is the following...
--ipcamera (\r\n)
Content-Type: image/jpeg (\r\n)
Content-Length: {length of frame} (\r\n)
(\r\n)
{frame}
(\r\n)
--ipcamera (\r\n)
etc.
I've tried using classes such as BufferedReader and Scanner to read until the "\r\n", however those are meant for text and not binary data, so it becomes corrupt. Is there any way to read the binary stream until it encounters a "\r\n"? Here is my current (broken) code.
EDIT: I've gotten it to work. I updated the code below. However, it's really slow in doing so. I'm not sure if it has anything to do with the ArrayList or not, but it could be the culprit. Any pointers to speed up the code? It's currently taking 500ms to 900ms for a single frame.
public void run() {
long startTime = System.currentTimeMillis();
try {
URLConnection urlConn = url.openConnection();
urlConn.setReadTimeout(15000);
urlConn.connect();
urlStream = urlConn.getInputStream();
DataInputStream dis = new DataInputStream(urlStream);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ArrayList<Byte> bytes = new ArrayList<Byte>();
byte cur;
int curi;
byte[] curBytes;
int length = 0;
while ((curi = dis.read()) != -1) {
cur = (byte) curi;
bytes.add(cur);
curBytes = getPrimativeArray(bytes);
String curBytesString = new String(curBytes, "UTF-8");
if (curBytesString.equals("--ipcamera\r\n")) {
bytes.clear();
continue;
} else if (curBytesString.equals("Content-Type: image/jpeg\r\n")) {
bytes.clear();
continue;
} else if (curBytesString.matches("^Content-Length: ([0-9]+)\r\n$")) {
length = Integer.parseInt(curBytesString.replace("Content-Length: ", "").trim());
bytes.clear();
continue;
} else if (curBytesString.equals("\r\n")) {
if (length == 0) {
continue;
}
byte[] frame = new byte[length];
dis.readFully(frame, 0, length);
writeFrame(frame);
bytes.clear();
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
long curTime = System.currentTimeMillis() - startTime;
System.out.println(curTime);
}
private byte[] getPrimativeArray(ArrayList<Byte> array) {
byte[] bytes = new byte[array.size()];
for (int i = 0; i < array.size(); i++) {
bytes[i] = array.get(i).byteValue();
}
return bytes;
}
private void writeFrame(byte[] bytes) throws IOException {
File file = new File("C:\\test.jpg");
FileOutputStream fos = new FileOutputStream(file);
fos.write(bytes);
fos.close();
System.out.println("done");
}
Currently you do not cope with the case when data is read in the frame part.
A rough assumption is:
Current version:
else if (line.equals("") && length != 0)
Probably more correct version:
else if (!line.equals("") && length != 0)
You cannot use BufferedReader to read binary, it will corrupt it. I you want to keep things simple, use DataInputStream.readLine(). Though not ideal, it may be the simplest in your case.
Other than using some bad practices and assuming that your URLConnection correctly delivers the data, the example you posted seems to work if you reset the length to zero after reading the frame data.
} else if (line.equals("") && length != 0) {
char[] buf = new char[length];
reader.read(buf, 0, length);
baos.write(new String(buf).getBytes());
//break;
length = 0; // <-- reset length
}
Please note this way all the frame data are written in the same ByteArrayOutputStream consecutively. If you don't want that, you should create a new ByteArrayOutputStream for every new frame you encounter.
You can't use a BufferedReader for part of the transmission and then some other stream for the rest of it. The BufferedReader will fill its buffer and steal some of the data you want to read with the other stream. Use DataInputStream.readLine(), noting that it's deprecated, or else roll your own line-reading code, using the input stream provided by the URLConnection.
Surely you don't have to? URLConnection reads the headers for you. If you want the content-length, use the API to get it. The stuff you get to read starts at the body of the transmission.
For the second time I have this extremely anoying problem with an InputStream.
This InputStream belongs to a Socket that is supposed to receive an image. The code for reading this image is as below:
InputStream input = socket.getInputStream();
InputStreamReader reader = new InputStreamReader(input);
BufferedReader bufferedReader = new BufferedReader(reader);
int total = Integer.parseInt(bufferedReader.readLine());
int bytesRead = 0;
byte[] buffer = new byte[total]; // total is the total size of the image
while (bytesRead < total) {
int next = input.read(buffer, bytesRead, total-bytesRead);
if (next > 0) {
bytesRead += next;
System.out.println("Read: " + bytesRead);
}
}
Now the strange thing is that this code skips the first 1182 bytes of the image, and then reads the remaining part. So when the total size is 15000 bytes, it reads byte 1182-15000.
I checked Wireshark and the whole image is transmitted. The code throws no exceptions. input.read() returns -1 as usual.
Pervious data has been readed from the stream using a BufferedReader. This data is only 5 characters long so it can't contain the missing 1K, but my guess is that the BufferedReader.readLine() method reads (buffers) more bytes from the InputStream than needed. Could this be correct?
I've had the same problem a few months ago but I absolutely have no clue on how I solved it.
Hope anyone can help.
Thanks in advance.
EDIT: I can solve the problem by adding a 100 ms sleep between sending the image size and the image data. It solves the problem but I would still realy like to know a more appropriate solution
As the name **Buffererd**Reader indicates it will snarf more bytes than just the first line from the underlying reader and hence also from the stream. Otherwise if would not be called "buffered".
Unfortunately I'm not aware of any non-deprecated class in Java which allows mixing of binary and textual data in the way you want.
I suggest, that you modify your protocol and transfer the length of the image also in some binary encoding. Then you can stick to InputStream.
My guess is the BufferedReader will assume that whatever reading operations you're performing afterwards will go through it, so it will happily consume input in increments of its buffer size.
One thing you could do is use a BufferedInputStream on top of input, instantiate a DataInputStream on top of that to do the readLine(), and then use the BufferedInputStream in your loop. The documentation says readLine is deprecated because it doesn't convert bytes to characters properly, but I'm hoping that with your first line containing only decimal digits, it shouldn't run into that problem.
I've written a short test, I hope it covers your use case:
import java.net.*;
import java.io.*;
class test{
static byte[] b;
static {
b = new byte[123456];
java.util.Random r = new java.util.Random();
for (int i=0;i<b.length;i++)
b[i] = (byte)(r.nextInt());
}
static void client() throws Exception{
Socket socket = new Socket("127.0.0.1", 9000);
InputStream input = socket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(input);
//read in length
DataInputStream dais = new DataInputStream(bis);
int total = Integer.parseInt(dais.readLine());
System.out.println("Total: "+total);
int bytesRead = 0;
byte[] buffer = new byte[total];
while (bytesRead < total) {
int next = bis.read(buffer, bytesRead, total-bytesRead);
if (next > 0) {
bytesRead += next;
System.out.println("Read: " + bytesRead);
}
}
for (int i=0;i<buffer.length;i++)
if (buffer[i]!=b[i]){
System.err.println("Error");
System.exit(1);
}
System.out.println("OK");
bis.close();
socket.close();
}
static void server() throws Exception{
ServerSocket srv = new ServerSocket(9000);
Socket sock = srv.accept();
OutputStream os = sock.getOutputStream();
BufferedOutputStream bos =new BufferedOutputStream(os);
DataOutputStream daos = new DataOutputStream(bos);
//we're sending the b buffer
//send the length in plain text, followed by newline
byte[]num = String.valueOf(b.length).getBytes();
daos.write(num,0,num.length);
daos.write(10);
//send actual buffer contents
bos.write(b, 0, b.length);
os.close();
srv.close();
sock.close();
}
public static void main(String[]args){
new Thread(new Runnable(){
public void run(){
try{server();}catch(Exception e){e.printStackTrace();}
}
}).start();
new Thread(new Runnable(){
public void run(){
try{client();}catch(Exception e){e.printStackTrace();}
}}).start();
}
}
Have you considered just sending the image data (without the preceding image size) and using an external library such as Apache Commons IO to handle this for you? In particular, I think that you will find IOUtils.toByteArray(InputStream input) interesting, e.g.
InputStream input = socket.getInputStream();
byte[] buffer = IOUtils.toByteArray(input);