I want to create a client-server program that allows the client to send a file to the server along with some information about the file (sender name, description, etc.).
The file could potentially be quite large as it could be either a text, picture, audio or video file, and because of that I do not want to have to read the whole file into a byte array before sending, I would rather read the file in blocks, sending them over the network and then allowing the server to append the blocks to the file at it's end.
However I am faced with the problem of how to best send the file along with a few bits of information about the file itself. I would like at a minimum to send the sender's name and a description both of which will be input to the client program by the user, but this may change in the future so should be flexible.
What is a good way of doing this that would also allow me to "stream" the file being sent rather than reading it in as a whole and then sending?
Sockets are natively streams of bytes so you shouldn't have a problem there. I suggest you have a protocol which looks like this.
This will allow you to send arbitrary properties as long as the total length is less than 64 KB. Followed by the file which can be any 63-bit length and is sent a block at a time. (with a buffer of 8 KB)
The Socket can be used to send more files if you wish.
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
Properties fileProperties = new Properties();
File file = new File(filename);
// send the properties
StringWriter writer = new StringWriter();
fileProperties.store(writer, "");
writer.close();
dos.writeUTF(writer.toString());
// send the length of the file
dos.writeLong(file.length());
// send the file.
byte[] bytes = new byte[8*1024];
FileInputStream fis = new FileInputStream(file);
int len;
while((len = fis.read(bytes))>0) {
dos.write(bytes, 0, len);
}
fis.close();
dos.flush();
to read
DataInputStream dis = new DataInputStream(socket.getInputStream());
String propertiesText = dis.readUTF();
Properties properties = new Properties();
properties.load(new StringReader(propertiesText));
long lengthRemaining = dis.readLong();
FileOutputStream fos = new FileOutputStream(outFilename);
int len;
while(lengthRemaining > 0
&& (len = dis.read(bytes,0, (int) Math.min(bytes.length, lengthRemaining))) > 0) {
fos.write(bytes, 0, len);
lengthRemaining -= len;
}
fos.close();
You could build up program around a well known protocol as FTP.
And to send the meta information you could just create a special file with a unique name that contains the info. Afterwards transfer both the user file and the meta file with FTP.
Otherwise, again using FTP for the file you could transfer the meta data in the client-server stream of your hand-written program.
I recommend using the http protocol for this. The server can be implemented using a servlet and Apache HttpClient can be used for the client. This article has some good examples. You can send both the file and the parameters in the same request. And that too with very little code!
Related
I am very new to Hadoop file system.
I have two different hadoop file systems(one is client and another is server), both of them are in different domain and not having direct access to one another.
I want to copy files(in GB) from server to client.
Since I don't have direct access to server(from client), I followed below method to copy the files.
I wrote server java program which reads the file with server configuration and writing as bytes to stdout.
System.out.write(buf.array(), 0, length);
System.out.flush();
then, I wrote cgi script which call this server jar.
then, I wrote a client java program which calls above cgi script to read the data
FSDataOutputStream dataOut = fs.create(client_file, true, bufSize, replica, blockSize);
URL url = new URL("http://xxx.company.com/cgi/my_cgi_script?" + "file=" + server_file);
InputStream is = url.openStream();
byte[] byteChunk = new byte[1024 * 1024];
int n = 0;
while ( (n = is.read(byteChunk)) > 0 ) {
dataOut.write(byteChunk, 0, n);
received += n;
}
dataOut.close();
now, copying file done without any issue and I see the same file size on server and client.
When I do FileChecksum for same file on client and server file systems, I am getting different values.
MD5-of-262144MD5-of-512CRC32C:86094f4043b9592a49ec7f6ef157e0fe
MD5-of-262144MD5-of-512CRC32C:a83a0b3f182db066da7520b36c79e696
Can you please help me to fix this issue?
Note: I am using the same blockSize on client and server file systems
I have simple java-server via sockets.
Server is read from client url of file which need to download.
FileOutputStream outStream= new FileOutputStream(SERVER_PATH + file.getName());
BufferedOutputStream out = new BufferedOutputStream(outStream);
byte buf[] = new byte[BATCH];
int read = 0;
while ((read = in.read(buf,0,BATCH))>=0){
out.write(buf,0,read);
}
how to continue to download file?
Your Question is a little ambiguous .!
After looking at the code, it looks like you are reading from a File in Client machine and Writing the same to the Server URL.
Assuming this situation,
The points that can help you resolve this are,
1. There will an IOException if the connection is lost. That means you have to handle the exception and reconnect to the Socket. May be after waiting for some time (!!)
2. Then you need to open the server File in Append mode and continue with out.write. As the out is not reset or lost with the Disconnection.
Thanks, Sunil
I write a client-server application which will be sending an .xml file from the client to the server. I have a problem with sending large data. I notice that the server can get at most 1460 bytes. When I send a file with more than 1460 bytes the server gets only first 1460 bytes and nothng more. In effect I get uncompleted file. Here is my code:
client send:
public void sendToServer(File file) throws Exception
{
OutputStream output = sk.getOutputStream();
FileInputStream fileInputStream = new FileInputStream(file);
byte[] buffer = new byte[1024*1024];
int bytesRead = 0;
while((bytesRead = fileInputStream.read(buffer))>0)
{
output.write(buffer,0,bytesRead);
}
fileInputStream.close();
}
server get:
public File getFile(String name) throws Exception
{
File file=null;
InputStream input = sk.getInputStream();
file = new File("C://protokolPliki/" + name);
FileOutputStream out = new FileOutputStream(file);
byte[] buffer = new byte[1024*1024];
int bytesReceived = 0;
while((bytesReceived = input.read(buffer))>0) {
out.write(buffer,0,bytesReceived);
System.out.println(bytesReceived);
break;
}
return file;
}
Do anyone know what is wrong with this code? Thanks for any help.
EDIT:
Nothing help :(. I google about that and I think its may connected with TCP MSS with is equal 1460 bytes.
Make sure you call flush() on the streams.
A passerby asks: isn't close() enough?
You linked to the docs for Writer, and the info. on the close() method states..
Closes the stream, flushing it first. ..
So you are partly right, OTOH, the OP is clearly using an OutputStream and the docs for close() state:
Closes this output stream and releases any system resources associated with this stream. The general contract of close is that it closes the output stream. A closed stream cannot perform output operations and cannot be reopened.
The close method of OutputStream does nothing.
(Emphasis mine.)
So to sum up. No, calling close() on a plain OutputStream will have no effect, and might as well be removed by the compiler.
Although not relate to your question, the API document said FileInputStream.read returns -1 for end of file. You should use >=0 for the while loop.
The MTU (Maximum Transmission Unit) for Ethernet is around 1500 bytes. Consider sending the file in chunks (i.e. one line at a time or 1024 bytes at a time).
See if using 1024 instead of 1024 * 1024 for the byte buffer solves your problem.
In the code executed on the server side, there is a break instruction in the while loop. Therefore the code in the loop will only get executed once. Remove the break instruction and the code should work just fine.
I am attempting to send an image from my android device to my computer via a socket. The problem is the input stream on my computer reads in every single byte but the last set of them. I have tried trimming the byte array down and sending it, I've manually written out -1 to the outputstream multiple times but the inputstream never reads -1. It just hangs waiting for data. I've also tried not closing the stream or sockets to see if it was some sort of timing issue, but that didn't work as well.
Client side (Android Phone)
//This has to be an objectoutput stream because I write objects to it first
InputStream is = An image's input stream android
ObjectOutputStream objectOutputStream = new ObjectOutputStream(socket.getOutputStream());
objectOutputStream.writeObject(object);
objectOutputStream.flush();
byte[] b = new byte[socket.getSendBufferSize()];
int read = 0;
while ((read = is.read(b)) != -1) {
objectOutputStream.write(b, 0, read);
objectOutputStream.flush();
b = new byte[socket.getSendBufferSize()];
}
//Tried manually writing -1 and flushing here
objectOutputStream.close();
is.close();
socket.close();
Server Side (Computer) This bit of code takes place after the object input stream reads in the objects sent. It only starts to read when the file starts to send
File loc = Location of where the file is stored on the computer
loc.createNewFile();
FileOutputStream os = new FileOutputStream(loc);
Socket gSocket = The socket
ObjectInputStream gInputStream = Object Input stream created from the sockets input stream already used to read in the previous objects
byte[] b = new byte[gSocket.getReceiveBufferSize()];
int read = 0;
while ((read = gInputStream.read(b)) != -1) {
os.write(b, 0, read);
os.flush();
b = new byte[gSocket.getReceiveBufferSize()];
}
os.close();
This code never reads in -1 even if I write -1 directly and flush the stream. The outcome is java.net.SocketException: Connection reset when the stream or socket from the android device is closed. The picture is almost completely sent but the very last pixels of the picture are gray. I also even tried using the out/input stream directly from the socket instead of using the already created objectinputstream/objectoutputstream and it still doesn't work.
Firstly, I think you misunderstood the meaning of EOF (-1). It doesn't mean the server wrote a -1, it means the server closed the stream.
I think your main problem though is that both the server and the client are reading in a loop, and neither get to the point where they close the stream. They are deadlocked - both are waiting for the other one to close first.
Your client:
Your server:
If you know that you have no more data to write then just close the stream.
Since you're already using ObjectInputStream and ObjectOutputStream, you can use their respective readObject and writeObject methods to read/write entire objects at a time. Maybe you could send/receive the entire byte array as an object?
On your android:
1) byte[] imageBytes = ...; // contains the Image
2) objectOutputStream.writeObject(imageBytes);
On your computer:
1) byte[] imageBytes = (byte[])readObject();
2) get image from imageBytes
Of course, you'll have to use readObject from within a thread since it'll block.
You are writing byte[] arrays as objects, bur reading bytes. You should be reading Objects and casting them to byte[]. EOS will cause an EOFException to be thrown.
i need to send file from server to client through some sockt (lets say port 8478) and also massage(in middel of file transfer ) (somthing like "hi", or "you reach to your limt"or "you reach to your 50% limt").
now to send only file it's easy im using
BufferedInputStrear and BufferedOutputStream in the client and server side.
now how can i send also massage in middel of file transfer in same port (8478).
thank you all..
this how i transfer the file
server side:
BufferedInputStream d=new BufferedInputStream(new FileInputStream(s));
BufferedOutputStream outStream = new BufferedOutputStream(cs.getOutputStream());
ObjectOutputStream msgoutStream = new ObjectOutputStream(cs.getOutputStream());
byte buffer[] = new byte[1024];
int read;
while((read = d.read(buffer))!=-1)
{
//msgoutStream.writeUTF("hjlhkhjk");
outStream.write(buffer, 0, read);
outStream.flush();
}
client side:
byte buffer[] = new byte[1024];
int read;
int f=0;
while((read = d.read(buffer))!=-1)
{
if(ifContinun)
{
System.out.println("strat write to file...");
}
//String s1=msgInPutStream.readLine();
//String s2=msgInPutStream.readUTF();
outStream.write(buffer, 0, read);
outStream.flush();
if(ifContinun)
{
System.out.println("after write to file...");
ifContinun=false;
}
}
You need to send the file in parts. You can invent a protocol like
short stream-id
short length of message
bytes of the message
This will allow you to interleave multiple streams of data in the same socket and have the other end break up the different streams.
However, its likely to be much simpler to open two connections which avoids the need for a protocol like this. e.g. FTP does this. ;)
In order to do this, you need to define a protocol on top of TCP. For example, the protocol can be:
There are a series of messages
Each message has a type
Each message is preceded by 4 bytes that carry the size of the next message
Each message starts with a type byte
The types are: 1 -- StartFile, 2 -- NextFileChunk, 3 -- TextMessage
The the second byte onwards contains the body of the message
For StartFile, the rest of the bytes constitute the filename and whatever other properties you want to send. (You can choose to use regular Java serialization.)
For NextFileChunk, you just have the next n bytes of the file being transferred
For TestMessage, the rest of the bytes would carry the text message
One way is to use some escape code to indicate when changing from file transfer to text transefer, and vice versa. Because a binary file may contain your escape codes, you must hand those some how.
But message protocol is preferable.