I am writing code for uploading a file from a client to my server and the performance isn't as fast as I think it should be.
I have the current code snippet that is doing the file transfer and I was wondering how I could speed up the transfer.
Sorry about all of the code:
InputStream fileItemInputStream ;
OutputStream saveFileStream;
int[] buffer;
while (fileItemInputStream.available() > 0) {
buffer = Util.getBytesFromStream(fileItemInputStream);
Util.writeIntArrToStream(saveFileStream, buffer);
}
saveFileStream.close();
fileItemInputStream.close();
The Util methods are as follows:
public static int[] getBytesFromStream(InputStream in, int size) throws IOException {
int[] b = new int[size];
int count = 0;
while (count < size) {
b[count++] = in.read();
}
return b;
}
and:
public static void writeIntArrToStream(OutputStream out, int[] arrToWrite) throws IOException {
for (int i = 0; i < arrToWrite.length; i++) {
out.write(arrToWrite[i]);
}
}
Reading a single byte at a time will be horribly inefficient. You're also relying on available, which is rarely a good idea. (It will return 0 if there are no bytes currently available, but there may be more to come.)
This is the right sort of code to copy a stream:
public void copyStream(InputStream input, OutputStream output) throws IOException
{
byte[] buffer = new byte[32*1024];
int bytesRead;
while ((bytesRead = input.read(buffer, 0, buffer.length)) > 0)
{
output.write(buffer, 0, bytesRead);
}
}
(The caller should close both streams.)
Related
I'm working on a Java project to store and retrieve files from MongoDB using GridFS specification. I'm using the code snippets provided in MongoDB Java driver documentation from https://mongodb.github.io/mongo-java-driver/4.1/driver/tutorials/gridfs/.
While using OpenDownloadStream to retrieve the file, I noticed that if the file is divided into more than one chunks, it returns only the first chunk, and not the full file.
ObjectId fileId;
GridFSDownloadStream downloadStream = gridFSBucket.openDownloadStream(fileId);
int fileLength = (int) downloadStream.getGridFSFile().getLength();
byte[] bytesToWriteTo = new byte[fileLength];
downloadStream.read(bytesToWriteTo); /*read file contents */
downloadStream.close();
System.out.println(new String(bytesToWriteTo, StandardCharsets.UTF_8));
Any solutions to this?
Looking at the class GridFSDownloadStreamImpl which implements GridFSDownloadStream, it looks like the method read(byte[]) reads chunk by chunk:
#Override
public int read(final byte[] b) {
return read(b, 0, b.length);
}
#Override
public int read(final byte[] b, final int off, final int len) {
checkClosed();
if (currentPosition == length) {
return -1;
} else if (buffer == null) {
buffer = getBuffer(chunkIndex);
} else if (bufferOffset == buffer.length) {
chunkIndex += 1;
buffer = getBuffer(chunkIndex);
bufferOffset = 0;
}
int r = Math.min(len, buffer.length - bufferOffset);
System.arraycopy(buffer, bufferOffset, b, off, r);
bufferOffset += r;
currentPosition += r;
return r;
}
Therefore, you have to loop until all expected bytes are actually read:
byte[] bytesToWriteTo = new byte[fileLength];
int bytesRead = 0;
while(bytesRead < fileLength) {
int newBytesRead = downloadStream.read(bytesToWriteTo);
if(newBytesRead == -1) {
throw new Exception();
}
bytesRead += newBytesRead;
}
downloadStream.close();
Note that I was not able to test above code so please use with caution.
I ended up using readAllBytes() method and it returns the whole file.
GridFSDownloadStream downloadStream = gridFSBucket.openDownloadStream(fileId);
int fileLength = (int) downloadStream.getGridFSFile().getLength();
byte[] bytesToWriteTo = new byte[fileLength];
bytesToWriteTo = downloadStream.readAllBytes();
downloadStream.close();
so i am trying to write some ints on a file per 1000 ints using a byte Array but it only does it one time(i have confirmed it using a hexeditor) when i would like about 1000 and i have no idea why it actually does the first but then stops.these are the errors that come up: Exception in thread "main" java.lang.IndexOutOfBoundsException at java.io.ByteArrayOutputStream.write(Unknown Source) at java.io.DataOutputStream.write(Unknown Source) at test.Class1.main(Class1.java:32)
public class Class1 {
static byte[] buf=new byte[1000];
public static void main(String[] args) throws IOException {
ByteArrayOutputStream BOS = new ByteArrayOutputStream() ;
DataOutputStream DOS = new DataOutputStream(BOS);
RandomAccessFile MyFile = new RandomAccessFile ("myfile.dat", "rw");
int acc=0;
int k=0;
for( k=0;k<100;k++) {
buf=bufferFiller( buf);
DOS.write(buf, k*1000, buf.length);
acc++;
}
}
public static byte[] bufferFiller(byte[] buf) {
int i=0;
int randomNum=0;
for(i=0;i<1000;i++) {
buf[i]=(byte) (randomNum = -50000 + (int)(Math.random() * ((100000) + 1)));
}
return buf;
}
}
From the OP's question, the issue was with the use of the offset in the buffer.
Javadoc for DataOutputStream.write (emphasis added)
public void write(byte[] b, int off, int len) throws IOException
Writes len bytes from the specified byte array starting at offset off to the underlying output stream. If no exception is thrown, the counter written is incremented by len.
So, the java.lang.IndexOutOfBoundsException was a result of this line:
DOS.write(buf, k*1000, buf.length);
By always using a "0" for the offset, the .write(...) will write the number of bytes (in this case the length of the buf), starting from the offset (which should be 0). The OP was nicely always re-filling the buffer in each loop iteration.
So changing to something like:
for( k=0;k<100;k++) {
buf=bufferFiller( buf);
DOS.write(buf, 0, buf.length);
acc++;
}
should resolve the issue.
While transfering images over the network using sockets, I had a -for me- strange issue:
When I wrote images to the OutputStream of one socket with ImageIO.write() and read the same images from the InputStream of the other socket with ImageIO.read() I noticed, that 16 bytes per image were sent more than read.
To be able to send multiple images in a row I had to read these bytes after every call of ImageIO.read() to not receive null because the input could not be parsed.
Does anybody know, why this is so and what these bytes are?
In this piece of code I have extracted the issue:
public class Test implements Runnable
{
public static final int COUNT = 5;
public void run()
{
try(ServerSocket server = new ServerSocket(3040))
{
Socket client = server.accept();
for(int i = 0; i < COUNT; i++)
{
final BufferedImage image = readImage(client.getInputStream());
System.out.println(image);
}
}
catch(IOException e)
{
e.printStackTrace();
}
}
private BufferedImage readImage(InputStream stream) throws IOException
{
BufferedImage image = ImageIO.read(stream);
dontKnowWhy(stream);
return image;
}
private void dontKnowWhy(InputStream stream) throws IOException
{
stream.read(new byte[16]);
}
public static void main(String... args)
{
new Thread(new Test()).start();
try(Socket server = new Socket("localhost", 3040))
{
for(int i = 0; i < COUNT; i++)
{
BufferedImage image = new BufferedImage(300, 300, BufferedImage.TYPE_INT_ARGB); //
int[] vals = new int[image.getWidth() * image.getHeight()]; //
Arrays.fill(vals, new Random().nextInt()); // Create random image
image.setRGB(0, 0, image.getWidth(), image.getHeight(), vals, 0, 1); //
ImageIO.write(image, "png", server.getOutputStream()); //send image to server
long time = System.currentTimeMillis(); //
while(time + 1000 > System.currentTimeMillis()); //wait a second
}
}
catch(IOException e)
{
e.printStackTrace();
}
}
}
I am glad about any answers, already thank you!
The "extra" bytes you see, is not read, simply because they are not needed to correctly decode the image (they are, however, most likely needed to form a fully compliant file in the chosen file format, so they are not just random "garbage" bytes).
For any given ImageIO plugin, the number of bytes left in the stream after a read may be 0, 16or any other number. It might depend on the format, the writer that wrote it, the reader, the number of images in the input, the metadata in the file, etc. In other words, relying on this behavior would be an error.
The easies way to fix this, is to prepend each image with a byte count, containing the length of the output image. This typically means you need to buffer the response on the client, to either a ByteArrayOutputStream (in-memory) or a FileOutputStream (disk).
The client then needs to read the byte count for the image, and make sure you skip any remaining bytes after the read. This can be accomplished by wrapping the input (see FilterInputStream) and keep track of the byte count internally.
(You can also read all the bytes up front, and wrapping them in a ByteArrayInputStream, before passing the data to ImageIO.read(), which is simpler but does more in-memory buffering).
After this, the client is ready do start over, by reading a new byte count, and a new image.
Another approach if you'd like less buffering on the server, could be to implement something like HTTP chunked transfer encoding, where you have multiple smaller blocks (chunks) sent to the client for each image, each prepended with its own byte count. You would need to handle the last chunk of each image especially, or insert special delimiter chunks to mark end of stream or start of a new stream.
Code below implements the buffering approach on the server, while using direct reading on the client.
Server:
DataOutputStream stream = new DataOutputStream(server.getOutputStream());
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
for (...) {
buffer.reset();
ImageIO.write(image, "png", buffer);
stream.writeInt(buffer.size());
buffer.writeTo(stream); // Send image to server
}
Client:
DataInputStream stream = new DataInputStream(client.getInputStream());
for (...) {
int size = stream.readInt();
try (InputStream imageData = new SubStream(stream, size)) {
return ImageIO.read(imageData);
}
// Note: imageData implicitly closed using try-with-resources
}
...
// Util class
private static final class SubStream extends FilterInputStream {
private final long length;
private long pos;
public SubStream(final InputStream stream, final long length) {
super(stream);
this.length = length;
}
#Override
public boolean markSupported() {
return false;
}
#Override
public int available() throws IOException {
return (int) Math.min(super.available(), length - pos);
}
#Override
public int read() throws IOException {
if (pos++ >= length) {
return -1;
}
return super.read();
}
#Override
public int read(byte[] b, int off, int len) throws IOException {
if (pos >= length) {
return -1;
}
int count = super.read(b, off, (int) Math.min(len, length - pos));
if (count < 0) {
return -1;
}
pos += count;
return count;
}
#Override
public long skip(long n) throws IOException {
if (pos >= length) {
return -1;
}
long skipped = super.skip(Math.min(n, length - pos));
if (skipped < 0) {
return -1;
}
pos += skipped;
return skipped;
}
#Override
public void close() throws IOException {
// Don't close wrapped stream, just consume any bytes left
while (pos < length) {
skip(length - pos);
}
}
}
I have to fill a byte[] in my Android application. Sometime, this one is bigger than 4KB.
I initialize my byte[] like this :
int size = ReadTools.getPacketSize(ptr.dataInputStream);
byte[] myByteArray = new byte[size];
Here, my size = 22625. But when I fill up my byte[] like this :
while (i != size) {
myByteArray[i] = ptr.dataInputStream.readByte();
i++;
}
But when I print the content of my byte[], I have a byte[] with size = 4060.
Does Java split my byte[] if this one is bigger than 4060 ? And if yes, how can I have a byte[] superior to 4060 ?
Here is my full code:
public class ReadSocket extends Thread{
DataInputStream inputStream;
BufferedReader reader;
GlobalContent ptr;
public ReadSocket(DataInputStream inputStream, GlobalContent ptr)
{
this.inputStream = inputStream;
this.ptr = ptr;
}
public void run() {
int i = 0;
int j = 0;
try {
ptr.StatusThreadReadSocket = 1;
while(ptr.dataInputStream.available() == 0)
{
if(ptr.StatusThreadReadSocket == 0)
{
ptr.dataInputStream.close();
break;
}
}
if(ptr.StatusThreadReadSocket == 1)
{
int end = ReadTools.getPacketSize(ptr.dataInputStream);
byte[] buffer = new byte[end];
while (i != end) {
buffer[j] = ptr.dataInputStream.readByte();
i++;
j++;
}
ptr.StatusThreadReadSocket = 0;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
...
}
Java doesn't split anything. You should post the minimal code which reproduces your error, and tell where ReadTools comes from.
There are two options here:
ReadTools.getPacketSize() returns 4096
You inadevertedly reassign myByteArray to another array
You should really post your full code and tell what library you use. Likely, it will have a method like
read(byte[] buffer, int offset, int length);
Which will save you some typing and also give better performance if all you need is bulk reading the content of the input in memory
The following code doesn't work to download a file (btw clen is file's length):
int pos = 0, total_pos = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while (pos != -1) {
pos = in.read(buffer, 0, BUFFER_SIZE);
total_pos += pos;
out.write(buffer);
setProgress((int) (total_pos * 100 / clen));
}
...but this works fine:
int buf;
while ((buf = in.read()) != -1)
out.write(buf);
I'm wondering why, even though the second code segment works quickly. On that note, is there any particular reason to use a byte[] buffer (since it doesn't seem to be faster, and BufferedInputStream already uses a buffer of its own....?)
Here's how it should be done.
public static void copyStream(InputStream is, OutputStream os)
{
byte[] buff = new byte[4096];
int count;
try {
while((count = is.read(buff)) > 0)
os.write(buff, 0, count);
}catch (Exception e) {
e.printStackTrace();
}finally {
try {
if(is != null)
is.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
if(os != null)
os.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
I've tried to make the minimum changes necessary to your code to get it working. st0le did a good job of providing a neater version of stream copying.
public class Test {
private static final String FORMAT = "UTF-8";
private static final int BUFFER_SIZE = 10; // for demonstration purposes.
public static void main(String[] args) throws Exception {
String string = "This is a test of the public broadcast system";
int clen = string.length();
ByteArrayInputStream in = new ByteArrayInputStream(string.getBytes(FORMAT));
OutputStream out = System.out;
int pos = 0, total_pos = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while (pos != -1) {
pos = in.read(buffer, 0, BUFFER_SIZE);
if (pos > 0) {
total_pos += pos;
out.write(buffer, 0, pos);
setProgress((int) (total_pos * 100 / clen));
}
}
}
private static void setProgress(int i) {
}
}
You were ignoring the value of pos when you were writing out the buffer to the output stream.
You also need to re-check the value of pos because it may have just read the end of the file. You don't increment the total_pos in that case (although you should probably report that you are 100% complete)
Be sure to handle your resources correctly with close()s in the appropriate places.
-edit-
The general reason for using an array as a buffer is so that the output stream can do as much work as it can with a larger set of data.
Writing to a console there might not be much of a delay, but it might be a network socket being written to or some other slow device. As the JavaDoc states
The write method of OutputStream calls the write method of one argument on each of the bytes to be written out. Subclasses are encouraged to override this method and provide a more efficient implementation.
The benefit of using it when using a Buffered Input/Output Stream are probably minimal.