Develop Reference

Fast way to read file from input stream when we know the size with low memory usage

Is there any faster way to read from an inputstream when we know the size of the data?
This code that I have is very slow:
File file = new File("file.jar");
if(!file.exists)file.createNewFile();
String url = "https://launcher.mojang.com/v1/objects/3870888a6c3d349d3771a3e9d16c9bf5e076b908/client.jar";
int len = 8461484;
InputStream is = new URL(url).openStream();
if(!file.exists())
file.createNewFile();
PrintWriter writer = new PrintWriter(file);
for(long i = 0;i < len;i ++) {
writer.write(is.read());
writer.flush();
System.out.println(i);
}
writer.close();

Use Buffered Input & Output Streams together with try-with-resources
(which ensures the Streams are all closed at EOJ)
Something like this:
try(final InputStream ist = new URL(url).openStream ();
final InputStream bis = new BufferedInputStream (ist);
final OutputStream ost = new FileOutputStream(file);
final OutputStream bos = new BufferedOutputStream(ost))
{
final byte[] bytes = new byte[64_000]; // <- as large as possible!
/**/ int count;
while ((count = bis.read(bytes)) != -1) {
bos.write(bytes, 0, count);
}
}

How to get BufferedInputStream from Multipart?

I'm trying to get a BufferedInputStream from an uploaded cvs file.
I'm working with a Multipart derived from the cvs file.
When I first get the Multipart, it's a BufferedInputStream, but the buffer is all null.
But if I look deeper down, there's another buffer in the CoyoteInputStream and that has data.
How can I get at this second buffer? My code is below.
And of course it's throwing a null exception when it gets to
while ((multiPartDataPos = stream.read(buffer)) >= 0)
What am I doing wrong? Am I mistaken that the CoyoteInputStream is the data I want?
public byte[] handleUploadedFile(Multipart multiPart) throws EOFException {
Multipart multiPartData = null;
BufferedInputStream stream = null;
int basicBufferSize = 0x2000;
byte[] buffer = new byte[basicBufferSize];
int bufferPos = 0;
try {
while (multiPart.hasNext()) {
int multiPartDataPos = bufferPos;
multiPartData = (Multipart) multiPart.next();
stream = new BufferedInputStream(multiPartData.getInputStream());
while ((multiPartDataPos = stream.read(buffer)) >= 0) {
int len = stream.read(buffer, multiPartDataPos, buffer.length - multiPartDataPos);
multiPartDataPos += len;
}
bufferPos = bufferPos + multiPartDataPos;
}
} ...

Your code doesn't make any sense.
while ((multiPartDataPos = stream.read(buffer)) >= 0) {
At this point you have read multiPartDataPos bytes into buffer, so that buffer[0..multiPartDataPos-1] contains the data just read.
int len = stream.read(buffer, multiPartDataPos, buffer.length - multiPartDataPos);
At this point you are doing another read, which could return -1, which will otherwise add some data from multiPartPos to multiPartDataPos+len-.
multiPartDataPos += len;
This step is only valid if len > 0.
And you are doing nothing with the buffer; and next time around the loop you will clobber whatever you just read.
The correct way to read any stream in Java is as follows:
while ((count = in.read(buffer)) > 0)
{
// use buffer[9..count-1], for example out.write(buffer, 0, count);
}
I don't understand why you think access to an underlying stream is required or what it's going to give you that you don't already have.

Turns out the better solution was to use move the data from an InputStream to a ByteArrayOutputStream and then return ByteArrayOutputStream.toByteArray()
Multipart multiPartData = null;
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
int read;
byte[] input = new byte[4096];
InputStream is;
try {
multiPartData = (Multipart)multipart.next();
is = multiPartData.getInputStream();
while ((read = is.read(input, 0, input.length)) != -1) {
buffer.write(input, 0, read);
}
buffer.flush();
return buffer.toByteArray(); // just a test right now
}

data loss in file transfer - JAVA?

i have a FileServer and a FileClient, server sends a file, when client connects. it is a simple program, just to understand the concept behind.
I am able to send the file from Server to Client, with a buffer of 1024.
The problem is that the recieved file is always around 0.01 MB less than the original file.Thus mp3 files lose some info, and video files just doesn't play.
I put some prinln statements in my while loops in both Server & Client. i found that my server is not sending the whole file.
//Server
byte [] mybytearray = new byte [1024];
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
bis.read(mybytearray,0,mybytearray.length);
OutputStream os = sock.getOutputStream();
System.out.println("Sending...\n mybytearray length:"+mybytearray.length+"file length:"+(int)myFile.length());
int read, readTotal=0;
while ((read = fis.read(mybytearray,0,mybytearray.length)) != -1) {
os.write(mybytearray, 0, read);
System.out.println("File REad:"+read+"readtotal:"+readTotal); //*
readTotal += read;
}
System.out.println("Final File Read:"+read+" Final readtotal:"+readTotal);
os.flush();
sock.close();
}
The Println statement output is:
Sending...
mybytearray length:1024file length:12767554
File REad:1024readtotal:0
File REad:1024readtotal:1024
.............and a lot of it...and then
File REad:1024readtotal:12756992
File REad:1024readtotal:12758016
File REad:322readtotal:12759040
Final File Read:-1 Final readtotal:12759362
The file length:12767554 & Last readTotal: 12759362 shud be equal. i don't understand why the last read value is lower [322], while it can still have 1024.
Any kind of help wud be appreciated.
Thanks.
[EDIT]
//Client
int read;
int totalRead = 0;
while ((read = is.read(mybytearray,0,mybytearray.length)) != -1) {
bos.write(mybytearray, 0 , read);
totalRead += read;
System.out.println("\nread:"+read+"\ntotalread: "+totalRead);
}
System.out.println("Final File Read:"+read+" Final readtotal:"+totalRead);
bos.write(mybytearray, 0 , read); //57 Line in FileClient.java
bos.flush();
i again tried to send a file. a txt this time.
this is my server's output
Sending...
mybytearray length:1024file length:1232
File REad:1024readtotal:0
File REad:208readtotal:1024
Final File Read:-1 Final readtotal:1232
and this one on client
read:208
totalread: 1232
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException
Final File Read:-1 Final readtotal:1232
at java.lang.System.arraycopy(Native Method)
at java.io.BufferedOutputStream.write(Unknown Source)
at FileClient.main(FileClient.java:57)
readtotal values are same, but sometimes i get this error, sometimes i don't.
[BIG EDIT--COMPLETE CLIENT CODE]
public class FileClient{
public static void main (String [] args ) throws IOException {
long start = System.currentTimeMillis();
int bytesRead;
int current = 0;
final JFrame f = new JFrame("Sample");
f.getContentPane().setLayout(new FlowLayout());
f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
f.setSize(590,490);
f.setVisible(true);
// localhost for testing
Socket sock = new Socket("localhost",13267);
System.out.println("Connecting...");
File f1=new File(RecieveObject(sock));
// receive file
byte [] mybytearray = new byte [1024];
InputStream is = sock.getInputStream();
FileOutputStream fos = new FileOutputStream(f1);
ProgressMonitorInputStream nn= new ProgressMonitorInputStream(f,"reading",is);
BufferedOutputStream bos = new BufferedOutputStream(fos);
/* bytesRead = is.read(mybytearray,0,mybytearray.length);
current = bytesRead;
do {
bytesRead =
is.read(mybytearray, current, (mybytearray.length-current));
System.out.println("mybytesarray length: "+mybytearray.length+"\ncurrent:"+current+"\nbytesread: "+bytesRead);
if(bytesRead >= 0) current += bytesRead;
} while(bytesRead > -1);
bos.write(mybytearray, 0 , current);
bos.flush();
*/
int read;
int totalRead = 0;
//InputStream clientInputStream = clientSocket.getInputStream();
while ((read = is.read(mybytearray,0,mybytearray.length)) != -1) {
bos.write(mybytearray, 0 , read);
totalRead += read;
System.out.println("\nread:"+read+"\ntotalread: "+totalRead);
}
System.out.println("Final File Read:"+read+" Final readtotal:"+totalRead);
// bos.write(mybytearray, 0 , read);
bos.flush();
long end = System.currentTimeMillis();
System.out.println(end-start);
bos.close();
sock.close();
}
public static String RecieveObject(Socket s) {
String str = null;
try{
ObjectInputStream is = new ObjectInputStream(s.getInputStream());
str =(String)is.readUTF();
}
catch(IOException ex){}
return str;
}
}

You're missing the beginning of the file (up to the first 1024 bytes), due to your 5th line (including the comment). You read from the input and advanced it, without sending it to the client. Remove this:
bis.read(mybytearray,0,mybytearray.length);
Additionally, you're not using the BufferedInputStream in your loop. Either use bis.read here instead of fis.read (if you still want buffered reads) - or remove the BufferedInputStream all together.
Your other problem is that you're reading the final bytes on the client, then entering the loop once more. is.read is again called. Instead of returning -1, it's throwing an IOException as the other side of the socket has been closed. As such, bos.flush() and bos.close() aren't getting called, and your final bytes are never written to disk. To assist with this, try calling sock.shutdownOutput before closing it. Regardless, you'll want to add some proper exception handling around this.

byte [] mybytearray = new byte [1024];
FileInputStream fis = new FileInputStream(myFile);
BufferedInputStream bis = new BufferedInputStream(fis);
OutputStream os = sock.getOutputStream();
System.out.println("Sending...\n mybytearray length:"+mybytearray.length+"filelength:"+(int)myFile.length());
int read, readTotal=0;
while ((read = bis.read(mybytearray,0,mybytearray.length)) != -1) {
os.write(mybytearray, 0, read);
System.out.println("File REad:"+read+"readtotal:"+readTotal); //*
readTotal += read;
}
System.out.println("Final File Read:"+read+" Final readtotal:"+readTotal);
os.flush();
sock.close();
}

In java, how to read a fixed length from the inputstream and save as a file?

In java, how to read a fixed length from the inputstream and save as a file?
eg. I want to read 5M from inputStream, and save as downloadFile.txt or whatever.(BUFFERSIZE=1024)
FileOutputStream fos = new FileOutputStream(downloadFile);
byte buffer [] = new byte[BUFFERSIZE];
int temp = 0;
while ((temp = inputStream.read(buffer)) != -1)
{
fos.write(buffer, 0, temp);
}

Two options:
Just keep reading and writing until you either reach the end of the input or you've copied enough:
byte[] buffer = new byte[1024];
int bytesLeft = 5 * 1024 * 1024; // Or whatever
FileInputStream fis = new FileInputStream(input);
try {
FileOutputStream fos = new FileOutputStream(output);
try {
while (bytesLeft > 0) {
int read = fis.read(buffer, 0, Math.min(bytesLeft, buffer.length);
if (read == -1) {
throw new EOFException("Unexpected end of data");
}
fos.write(buffer, 0, read);
bytesLeft -= read;
}
} finally {
fos.close(); // Or use Guava's Closeables.closeQuietly,
// or try-with-resources in Java 7
}
} finally {
fis.close();
}
Read all 5M into memory in one call, e.g. using DataInputStream.readFully, and then write it out in one go. Simpler, but obviously uses more memory.

How to read a bin file to a byte array?

I have a bin file that I need to convert to a byte array. Can anyone tell me how to do this?
Here is what I have so far:
File f = new File("notification.bin");
is = new FileInputStream(f);
long length = f.length();
/*if (length > Integer.MAX_VALUE) {
// File is too large
}*/
// Create the byte array to hold the data
byte[] bytes = new byte[(int)length];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length && (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+f.getName());
}
But it's not working...
Kaddy

try using this
public byte[] readFromStream(InputStream inputStream) throws Exception
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
byte[] data = new byte[4096];
int count = inputStream.read(data);
while(count != -1)
{
dos.write(data, 0, count);
count = inputStream.read(data);
}
return baos.toByteArray();
}
Btw, do you want a Java code or C++ code. Seeing the code in your question, I assumed it to be a java code and hence gave a java answer to it

You're probably better off using a memory mapped file. See this question

In Java, a simple solution is:
InputStream is = ...
ByteArrayOutputStream os = new ByteArrayOutputStream();
byte[] data = new byte[4096]; // A larger buffer size would probably help
int count;
while ((count = is.read(data)) != -1) {
os.write(data, 0, count);
}
byte[] result = os.toByteArray();
If the input is a file, we can preallocate a byte array of the right size:
File f = ...
long fileSize = f.length();
if (fileSize > Integer.MAX_VALUE) {
// file too big
}
InputStream is = new FileInputStream(f);
byte[] data = new byte[fileSize];
if (is.read(data)) != data.length) {
// file truncated while we were reading it???
}
However, there is probably a more efficient way to do this task using NIO.

Unless you really need to do it just that way, maybe simplify what you're doing.
Doing everything in the for loop may seem like a very slick way of doing it, but it's shooting yourself in the foot when you need to debug and don't immediately see the solution.

In this answer I read from an URL
You could modify it so the InputStream is from a File instead of a URLConnection.
Something like:
FileInputStream inputStream = new FileInputStream("your.binary.file");
ByteArrayOutputStream output = new ByteArrayOutputStream();
byte [] buffer = new byte[ 1024 ];
int n = 0;
while (-1 != (n = inputStream.read(buffer))) {
output.write(buffer, 0, n);
}
inputStream.close();
etc

Try open source library apache commons-io
IOUtils.toByteArray(inputStream)
You are not the first and not the last developer who needs to read a file, no need to reinvent it each time.