I need help on my homework, any help will be much appreciated. I can send small files without a problem. But when i try to send let’s say a 1GB file byte array sends OutOfMemoryError so i need a better solution to send file from server to client. How can i improve this code and send big files, please help me.
Server Code:
FileInputStream fis = new FileInputStream(file);
byte[] fileByte = new byte[fis.available()]; //This causes the problem.
bytesRead = fis.read(fileByte);
oos = new ObjectOutputStream(sock.getOutputStream());
oos.writeObject(fileByte);
Client Code:
ois = new ObjectInputStream(sock.getInputStream());
byte[] file = (byte[]) ois.readObject();
fos = new FileOutputStream(file);
fos.write(file);
Don't read the whole file into memory, use a small buffer and write while you are reading the file:
BufferedOutputStream bos = new BufferedOutputStream(sock.getOutputStream())
File file = new File("asd");
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
byte[] buffer = new byte[1024*1024*10];
int n = -1;
while((n = bis.read(buffer))!=-1) {
bos.write(buffer,0,n):
}
Use Buffered* to optimize the writing and reading from Streams
Just split the array to smaller chunks so that you don't need to allocate any big array.
For example you could split the array into 16Kb chunks, eg new byte[16384] and send them one by one. On the receiving side you would have to wait until a chunk can be fully read and then store them somewhere and start with next chunk.
But if you are not able to allocate a whole array of the size you need on server side you won't be able to store all the data that you are going to receive anyway.
You could also compress the data before sending it to save bandwidth (and time), take a look at ZipOutputStream and ZipInputStream.
Here's how I solved it:
Client Code:
bis=new BufferedInputStream(sock.getInputStream());
fos = new FileOutputStream(file);
int n;
byte[] buffer = new byte[8192];
while ((n = bis.read(buffer)) > 0){
fos.write(buffer, 0, n);}
Server Code:
bos= new BufferedOutputStream(sock.getOutputStream());
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
int n=-1;
byte[] buffer = new byte[8192];
while((n = bis.read(buffer))>-1)
bos.write(buffer,0,n);
Depending on whether or not you have to write the code yourself, there are existing libraries which solve this problem, e.g. rmiio. If you are not using RMI, just plain java serialization, you can use the DirectRemoteInputStream, which is kind of like a Serializable InputStream. (this library also has support for things like auto-magically compressing the data).
Actually, if you are only sending file data, you would be better off ditching the Object streams and use DataInput/DataOutput streams. first write an integer indicating the file length, then copy the bytes directly to the stream. on the receiving side, read the integer file length, then read exactly that many bytes.
when you copy the data between streams, use a small, fixed size byte[] to move chunks of data between the input and output streams in a loop. there are numerous examples of how to do this correctly available online (e.g. #ErikFWinter's answer).
Related
Using Java, I am trying to send some file data over a DatagramSocket. I need to read a file in 1000-byte chunks and send them over as packets. My code:
reads a file into a byte array wrapped in a byte buffer
places the data in a packet and sends it
has the receiver open the packet and re-write the contents to a new file.
I am having a problem with writing the byte array back to a file. Please see my code below.
Client/Sender:
byte[] data = new byte[1000];
ByteBuffer b = ByteBuffer.wrap(data);
DatagramPacket pkt;
File file = new File(sourceFile);
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
CRC32 crc = new CRC32();
while(true){
b.clear();
b.putLong(0); // I need to put the checksum at the beginning for easy retrieval
bytesRead = bis.read(data);
if(bytesRead==-1) { break; }
crc.reset();
crc.update(data, 8, data.length-8);
long chksum = crc.getValue();
b.rewind();
b.putLong(chksum);
pkt = new DatagramPacket(data, 1000, addr); // addr is valid, works fine
sk.send(pkt);
}
bis.close();
fis.close();
Server/Receiver:
DatagramSocket sk = new DatagramSocket(port);
File destfile = new File("hello.txt");
FileOutputStream fos = new FileOutputStream(destfile);
BufferedOutputStream bos = new BufferedOutputStream(fos);
PrintStream ps = new PrintStream(fos);
byte[] data = new byte[1000];
DatagramPacket pkt = new DatagramPacket(data, data.length);
ByteBuffer b = ByteBuffer.wrap(data);
CRC32 crc = new CRC32();
while(true) {
pkt.setLength(data.length);
sk.receive(pkt);
b.rewind();
// compare checksum, print error if checksum is different
// if checksum is the same:
bos.write(data); // Where the problem seems to be occurring.
// send acknowledgement packet.
}
bos.close();
fos.close();
Here, I am mainly having issues with writing the file back. With a small text file that says Hello World!, I get a strange output that says vˇ]rld!. Also, the input file is only 12 bytes but the file that the receiver creates is 1KB.
I think my issue is dealing with a byte buffer - I've written a program that copies files using file streams and buffered streams, which worked well. But I'm confused with how streams work in this sort of situation, and I would really appreciate any help. Thanks!
In the sender's data[] you overwrite the text, which was read from the file by the crc! You have to read the text in a position after the long. When correcting this in the Sender, it works:
//int bytesRead = bis.read(data); --old code
int bytesRead=bis.read(data,8,data.length-8);
Furthermore you send 1000 bytes, so will receive 1000 bytes, which will go into the destfile.
BTW: you do not check the crc in the server.... so why send it ?
I've been trying to get this working for a few days now, but I've had no success.
I want to send a file over socket client/server. The only difference is: I want to send an object that contains the file bytes.
So the client loads a file, reads chunks of 1024 bytes, store them in a object, and send the object to the server. Since the file can be larger than 1024 bytes, I want to send the object repeatedly but with different bytes stored in them (as the buffer reads it). On the server, I want to compose the array of bytes and save it as a file.
The reason I'm using 1024 is because I want to avoid any sort of out of memory error, if the file is, let's say, 4 GB in size.
I tried doing the following on the client:
File file = new File("C:\\test\\test.txt");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
ObjectOutputStream oos = new ObjectOutputStream(socket.getOutputStream());
byte[] bytes = new byte[1024];
FileTest ft = new FileTest();
ft.setName("Testing");
int counttest = 1;
while (bis.read(bytes) > 0) {
ft.setCounttest(counttest);
ft.setBytes(bytes);
oos.writeObject(ft);
counttest += 1;
}
On the server:
int bufferSize = socket.getReceiveBufferSize();
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("C:\\test\\test2.txt"));
ObjectInputStream ois = new ObjectInputStream(socket.getInputStream());
byte[] bytes = new byte[bufferSize];
while (true) {
FileTest ft = (FileTest) ois.readObject();
if (ft != null) {
System.out.println(ft.getName());
bos.write(ft.getBytes());
}
}
So I tested sending a txt file with a sequence of numbers and the test2.txt file produced by the server came out only with the first 1024 chunk of bytes repeated twice. Also, the counttest integer never increases when received in the server.
Any idea how to accomplish this?
Thanks in advance.
You are running into the effects of the ObjectOutputStream's attempt to preserve object identity. Repeatedly writing the same object instance will result on the same instance on the receiver's end. This is generally a good thing, but confusing if you are modifying the object on the sender's end and expecting those modifications to show up on the receiver's end.
Thus, you have two issues:
In order to send the data each time, you either need to create a new FileTest instance each time, or use the writeUnshared() method.
Due to this identity preserving behavior, you will need to periodically reset() the ObjectOutputStream in order to keep all these instances from being held forever (and potentially leading to an OOME on the client or server).
To process some images in my android application I currently use code like this:
FileOutputStream fileOuputStream = new FileOutputStream(imgpath);
[..DO SOME STUFF..]
Bitmap data = BitmapFactory.decodeByteArray(bFile, 0, bFile.length, options);
data.compress(Bitmap.CompressFormat.JPEG, 90, fileOuputStream);
[..DO SOME STUFF..]
File file = new File(imgpath);
FileInputStream imageInFile = new FileInputStream(file);
byte imageData[] = new byte[(int) file.length()];
imageInFile.read(imageData);
[..DO SOME STUFF..]
file.delete();
//NOTE: The code is all in the same method
the problem is that passing my image from one part of the code to another using this method creates a temporary file.
I was looking for a way to read / write the file data using a memory variable, something like "generic stream" in which store data in order to replace use of "FileInputStream " and "FileOutputStream " and do not write temporary file.
If you are able to use an InputStream or OutputStream you can use ByteArrayInputStream or ByteArrayOutputStream for in memory handling of the data.
If you have two thread you can also use PipedInputStream and PipedOutputStream together to communicate between the threads.
You could write your data to a ByteArrayOutputStream and use the byte array of that stream:
ByteArrayOutputStream out = new ByteArrayOutputStream();
data.compress(Bitmap.CompressFormat.JPEG, 90, out);
// now take the bytes out of your Stream
byte[] imgData = out.toByteArray();
I read from here that a big difference from Java IO and Java NIO is that in the first we can navigate from the data only after the creation of a buffer (I think with, for example, a BufferedInputStreamer object).
In the second the data read from a channel is stored directly in a buffer.
Please, can anyone write some code snippets that show how to navigate, back and forth from an old IO buffer and the same translate with the new IO API?
Thanks.
Example of skipping 1024, reading the next 1024, and seeking back to 0;
nio:
int i=1024;
Path p = Paths.get("./","file.txt");
SeekableByteChannel sbc = Files.newByteChannel(p, StandardOpenOption.READ);
sbc.position((long)i);
ByteBuffer bf = ByteBuffer.allocate(i);
sbc.read(bf);
byte[] b = bf.array();
sbc.position(0L);
io:
int i=1024;
File f = new File("./file.txt");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f));
bis.mark(i*2);
bis.skip((long)i);
byte[] b = new byte[i];
bis.read(byte[] b);
bis.reset();
I'm creating a TFTP server. I've got it tranfering files fine but most of the files wont open when they arrive at the other end. This is because the output of the ArrayList im using to store file bytes from every packet received adds a load of bytes to the start of the file. eg. "¬í sr java.util.ArrayListxÒ™Ça I sizexp w ur [B¬óøTà xp ü!". The reason for using the List in the first place is that the server im creating has no way to tell the file size of the file which is being received. Therefore as far as I can tell I cant use a byte[] as this needs to be initialised with a set length. Is there any way round this?
WRQ WRQ = new WRQ();
ACK ACK = new ACK();
DatagramPacket outPacket;
byte[] bytes;
byte[] fileOut;
List fileBytes = new ArrayList();
outPacket = WRQ.firstPacket(packet);
socket.send(outPacket);
socket.receive(packet);
while (packet.getLength() == 516){
bytes = WRQ.doWRQ(packet);
fileBytes.add(bytes);
outPacket = ACK.doACK(packet);
socket.send(outPacket);
socket.receive(packet);
}
bytes = WRQ.doWRQ(packet);
fileBytes.add(bytes);
outPacket = ACK.doACK(packet);
socket.send(outPacket);
ObjectOutputStream os;
ByteArrayOutputStream byteStream = new ByteArrayOutputStream(5000);
os = new ObjectOutputStream(new BufferedOutputStream(byteStream));
os.flush();
os.writeObject(fileBytes);
os.flush();
byte[] outFile = byteStream.toByteArray();
os.close();
FileOutputStream foStream = new FileOutputStream(filename);
foStream.write(outFile);
foStream.close();
You store byte arrays in an ArrayList, and then you write the whole ArrayList to a ByteArrayOutputStream wrapped in an ObjectOutputStream, using the writeObject() method.
This uses the native Object serialization mechanism to save the ArrayList object. It doesn't write every byte array in the list one after the other. To make it clear: it writes the class name, and the internal structure of the ArrayList, using the object serialization protocol.
You don't need an ArrayList. Write directly to a ByteArrayOutputStream, or even directly to a FileOutputStream. As is, you're trying to
write bytes to a list
write the bytes in the list to a byte array
write the byte array to a file.
It would be much more straightforward (and efficient) to write directly to the output file (wrapped in a BufferedOutputStream for buffering)