Develop Reference

How to calculate checksum with InputStream and then use it again

I want to calculate the CRC3 checksum of a given InputStream and then use to get the string out of it. Here's what I've tried so far
private long calculateChecksum(InputStream stream) throws IOException {
CRC32 crc = new CRC32();
byte[] buffer = new byte[8192];
int length;
while ((length = stream.read(buffer)) > 0) {
crc.update(buffer, 0, length);
}
return crc.getValue();
}
and then
String text = IOUtils.toString(inputStream, UTF_8);
I also tried to reverse the order. First use it as string and then calculate the checksum. But it didn't work.
What seems to be my issue is that the index goes to the end while calculating the checksum and then doesn't reset. Any idea how to use InputStream after calculating the checksum?

As others said, a stream can be consumed only once. But you can consume it and calculate the CRC value at the same time by wrapping your InputStream with a java.util.zip.CheckedInputStream.
Here is a complete example, assuming the text file "test.txt" is in the current directory and contains only this one line: These are german umlauts: äöüÄÖÜß
import org.apache.commons.io.IOUtils;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.zip.CRC32;
import java.util.zip.CheckedInputStream;
public class App {
private static final String INPUT_FILE = "test.txt";
public static void main( String[] args ) {
final CRC32 crc32 = new CRC32();
try(InputStream in = new CheckedInputStream(new BufferedInputStream(
new FileInputStream(INPUT_FILE)), crc32))
{
final String text = IOUtils.toString(in, StandardCharsets.UTF_8);
System.out.println(text);
System.out.println(String.format("CRC32: %x", crc32.getValue()));
} catch (IOException e) {
e.printStackTrace();
}
}
}
Output:
These are german umlauts: äöüÄÖÜß
CRC32: 84bcd851

Yes, an InputStream is consumed. You have a few options:
mark
mark() / reset() are optional methods of inputstreams; mark sets a mark (this does, by itself, nothing), and reset 'rewinds back' to the mark, replaying everything that was provided since the last time you called mark().
However, your average inputstream either does not support it, or, if it does, supports it by storing in memory all the bytes that are received since setting the mark. Meaning, if you do this to an inputstream that contains a few GB worth of data, you're going to get an OutOfMemoryError.
If there isn't a lot of data, just use mark and reset. Wrap in a BufferedInputStream which is specced to support mark/reset:
private void example(InputStream in) {
BufferedInputStream buffered = new BufferedInputStream(in);
in.mark();
long crc = calculateChecksum(buffered);
in.reset();
String text = IOUtils.toString(buffered, UTF_8);
}
Duplicate
Your second option is to duplicate the inputstream, sending each retrieved byte both to IOUtils as well as to the CRC algorithm.
This is complicated and not recommended.
Checksum the string instead.
You already have a string of data. Just checksum that:
private void example(InputStream in) {
String text = IOUtils.toString(in, UTF_8);
CRC32 crc = new CRC32();
crc.update(text.getBytes(UTF_8));
long checksum = crc.getValue();
}
Or, ditching IOUtils:
private void example(InputStream in) {
byte[] data = in.readAllBytes();
CRC32 crc = new CRC32();
crc.update(data);
long checksum = crc.getValue();
String text = new String(data, UTF_8);
}

InputStream is a read-once stream. Once you've read it, you can't go back to start again. This is because InputStream is general-purpose: it could be the stream of bytes read from a keyboard, for example, or read from a real-time data feed.
If your input stream is in fact a FileInputStream, then you could use
inputStream.getChannel.position(0);
to reset it to the start of the file.
If it's a ByteArrayInputStream, then you already have a byte array so you might as well just use that instead.
If you want to write a general-purpose function that doesn't know what kind of InputStream it is given, then you can wrap it in a BufferedInputStream and use its mark() method. This will use extra memory to buffer the whole of the stream.

Java - Converting data from a DataOutputStream to a File where DataOutputStream size is unknown

This is a simplified version of my program.
I have a Server.java file where I read in "b.txt" into a byte array and then write it to the DataOutputStream.
import java.io.*;
import java.net.*;
class Server {
public static void main(String[] args) {
try {
ServerSocket ss = new ServerSocket(4000);
Socket s = ss.accept();
DataOutputStream dout = new DataOutputStream(s.getOutputStream());
byte[] byteArray = new byte[32];
File tempFile = new File("Files/b.txt");
FileInputStream fis = new
FileInputStream(tempFile);
fis.read(byteArray);
fis.close();
dout.write(byteArray);
dout.flush();
dout.close();
s.close();
} catch (Exception e) {
//handle exceptions
}
}
}
In Client.java I read the DataInputStream into a byteArray. Then, I create a FileOutputStream. In the commented out code fos.write(byteArray), this writes to the file, however since the actual file may not be exactly 32 bytes I am left with a corrupt file since the remaining bytes just get filled with random data (or whatever happens).
import java.io.*;
import java.net.*;
class Client {
public static void main(String[] args) {
try {
Socket s = new Socket("localhost", 4000);
DataInputStream din = new DataInputStream(s.getInputStream());
byte[] byteArray = new byte[32];
FileOutputStream fos = new FileOutputStream("Downloaded/testfile.txt");
//fos.write(byteArray);
int count;
while ((count = din.read(byteArray)) > 0) {
fos.write(byteArray, 0, count);
}
fos.close();
din.close();
s.close();
} catch (Exception e) {
//handle exceptions
}
}
}
So I looked up how to fix this and a lot of places told me to do the code I have inside the while loop: while ((count = din.read... etc.
However when I try this, testfile.txt is blank and contains 0 bytes. Am I missing something big here? I have tried sending in the correct amount of bytes from the server, however this doesn't change anything.
I would like to be able to transfer any file type, so it is important that I am able to transfer the correct amount of bytes without corrupting the data by having bytes "hanging over". Thank you.

how to send file formats other than txt using tcp

I am trying to send pdf files over to clients using tcp connections. Somehow, the file is corrupted. I wonder if tcp support non-txt files?
this is a simple server program
import java.io.*;
import java.net.*;
class server{
public static void main (String args[]) throws Exception
{
Socket s = null;
int firsttime=1;
while(true) {
ServerSocket s_sock = new ServerSocket(3248);
Socket socket = s_sock.accept();
BufferedReader bf = new BufferedReader(new InputStreamReader(socket.getInputStream()));
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
File myFile = new File ("/Users/wchen48/Desktop/123.pdf");
byte [] mybyte = new byte[(int)myFile.length()];
OutputStream os = socket.getOutputStream();
System.out.println("sending......" + mybyte.length + "Byte");
os.write(mybyte,0,mybyte.length);
os.flush();
socket.close();
s_sock.close();
}
} //end of main
This is the client:
import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.Socket;
public class client{
public static void main(String[] argv) throws Exception {
Socket sock = new Socket("127.0.0.1", 3248);
byte[] mybytearray = new byte[11536];
InputStream is = sock.getInputStream();
FileOutputStream fos = new
FileOutputStream("/Users/wchen48/Documents/123.pdf");
bos = new BufferedOutputStream(fos);
int bytesRead = is.read(mybytearray, 0, mybytearray.length);
bos.write(mybytearray, 0, mybytearray.length);
bos.close();
sock.close();
}
}
Thank you in advance.

What Zielu and Juned said, plus:
You need loops at both server and client.
At the server, choose a reasonable-size buffer and loop around. Try to read [buffer length] bytes into it from the file. Then USE THE RETURNED VALUE FROM THE READ CALL to both check for errors, (result<0), or end-of-file, (result 0), or use the value in the network write call as the number of bytes to write. DO NOT use the buffer length - it may not be full. When you get 0, close the file and socket, else loop back and read some more.
At the client, choose a reasonable-size buffer and loop around. Try to read [buffer length] bytes into it from the socket. Then USE THE RETURNED VALUE FROM THE READ CALL to both check for errors, (result<0), or socket close, (result 0), or use the value in the file write call as the number of bytes to write. DO NOT use the buffer length - it may not be full. When you get 0, exit the loop and close the file, else loop back and read some more.

you dont read content of your file into the buffer mybyte you later send through socket, so you send 00000....

The most likely reason for getting the corrupted file is the fixed number of bytes(11536) that you are trying to read. You should try to read all the bytes received on the socket rather than fixed number of bytes.

File transfer via Sockets [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have write a code that transfer file in Java socket but I want to expand it. I want to select the file that I want. And I don't understand one line of code:
byte[] mybytearray = new byte[(int) myFile.length()];
Here's my real code:
import import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
public class Main {
public static void main(String[] args) throws IOException {
ServerSocket servsock = new ServerSocket(123456);
File myFile = new File("s.pdf");
while (true) {
Socket sock = servsock.accept();
byte[] mybytearray = new byte[(int) myFile.length()];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
bis.read(mybytearray, 0, mybytearray.length);
OutputStream os = sock.getOutputStream();
os.write(mybytearray, 0, mybytearray.length);
os.flush();
sock.close();
}
}
}
The client module:
import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.Socket;
public class Main {
public static void main(String[] argv) throws Exception {
Socket sock = new Socket("127.0.0.1", 123456);
byte[] mybytearray = new byte[1024];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream("s.pdf");
BufferedOutputStream bos = new BufferedOutputStream(fos);
int bytesRead = is.read(mybytearray, 0, mybytearray.length);
bos.write(mybytearray, 0, bytesRead);
bos.close();
sock.close();
}
}

That line of code is attempting to make a byte array that has the same size as the file you are reading, allowing it to read the entire file into the byte array in a single read.
The second block of code 'assumes' that the file is no longer than 1024 bytes long, and reads that length maximum, no more. Do you know that the file is <= 1024 bytes long?
For a number of reasons, it is a bad idea to have a variable buffer size when dealing with streams. The beauty of streams is that you don't have the whole file in memory. Instead, use this pattern ... it is easy enough:
{
Socket sock = servsock.accept();
byte[] mybytearray = new byte[4096];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
OutputStream os = sock.getOutputStream();
int amt = bis.read(mybytearray, 0, 4096);
while (amt>0) {
os.write(mybytearray, 0, amt);
amt = bis.read(mybytearray, 0, 4096);
}
os.flush();
sock.close();
}
I chose 4096 as a random arbitrary size, but there is no real reason to choose any size bigger than this. This will copy a file of any size (even gigabytes), the entire file, and it will not cause an out of memory problem when the file gets big.

Have a look at The Java™ Tutorials, Trail: Learning the Java Language, Arrays.
In short: it creates an array for elements of type byte and with a size of the number of bytes in myFile. Since array creation expects an int value for the size and java.io.File offers long length() the returned long value ist cast to (int). Do a DDuckinG and look into the offered tutorials.

Transform InputStream to FileItem in Java

How can I do to transform from InputStream to FileItem in Java?
Thanks.

Here is a working example. Note that you must change the InputStream from the example with your InputStream, and also you might want to change the location of your work/tmp dir().
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.disk.DiskFileItem;
public class TestFile {
public static void main(String args[]) throws IOException {
// This is a sample inputStream, use your own.
InputStream inputStream = new FileInputStream("c:\\Kit\\Apache\\geronimo-tomcat6-javaee5-2.1.6\\README.txt");
int availableBytes = inputStream.available();
// Write the inputStream to a FileItem
File outFile = new File("c:\\tmp\\newfile.xml"); // This is your tmp file, the code stores the file here in order to avoid storing it in memory
FileItem fileItem = new DiskFileItem("fileUpload", "plain/text", false, "sometext.txt", availableBytes, outFile); // You link FileItem to the tmp outFile
OutputStream outputStream = fileItem.getOutputStream(); // Last step is to get FileItem's output stream, and write your inputStream in it. This is the way to write to your FileItem.
int read = 0;
byte[] bytes = new byte[1024];
while ((read = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
// Don't forget to release all the resources when you're done with them, or you may encounter memory/resource leaks.
inputStream.close();
outputStream.flush(); // This actually causes the bytes to be written.
outputStream.close();
// NOTE: You may also want to delete your outFile if you are done with it and dont want to take space on disk.
}
}