Develop Reference

Limit file size while writing in java

I need to limit the file size to 1 GB while writing preferably using BufferedWriter.
Is it possible using BufferedWriter or I have to use other libraries ?
like
try (BufferedWriter writer = Files.newBufferedWriter(path)) {
//...
writer.write(lines.stream());
}

You can always write your own OutputStream to limit the number of bytes written.
The following assumes you want to throw exception if size is exceeded.
public final class LimitedOutputStream extends FilterOutputStream {
private final long maxBytes;
private long bytesWritten;
public LimitedOutputStream(OutputStream out, long maxBytes) {
super(out);
this.maxBytes = maxBytes;
}
#Override
public void write(int b) throws IOException {
ensureCapacity(1);
super.write(b);
}
#Override
public void write(byte[] b) throws IOException {
ensureCapacity(b.length);
super.write(b);
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
ensureCapacity(len);
super.write(b, off, len);
}
private void ensureCapacity(int len) throws IOException {
long newBytesWritten = this.bytesWritten + len;
if (newBytesWritten > this.maxBytes)
throw new IOException("File size exceeded: " + newBytesWritten + " > " + this.maxBytes);
this.bytesWritten = newBytesWritten;
}
}
You will of course now have to set up the Writer/OutputStream chain manually.
final long SIZE_1GB = 1073741824L;
try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
new LimitedOutputStream(Files.newOutputStream(path), SIZE_1GB),
StandardCharsets.UTF_8))) {
//
}

Exact bytes to 1 GB is very difficult in cases where you are writing lines. Each line may contain unknown number of bytes in it. I am assuming you want to write data line by line in file.
However, you can check how many bytes does line has before writing it to the file and another approach is to check file size after writing each line.
Following basic example writes one same line each time. Here This is just a test ! text takes 21 bytes on file in UTF-8 encoding. Ultimately after 49 writes it reaches to 1029 Bytes and stops writing.
public class Test {
private static final int ONE_KB = 1024;
public static void main(String[] args) {
File file = new File("D:/test.txt");
try (BufferedWriter writer = Files.newBufferedWriter(file.toPath())) {
while (file.length() < ONE_KB) {
writer.write("This is just a test !");
writer.flush();
}
System.out.println("1 KB Data is written to the file.!");
} catch (IOException e) {
e.printStackTrace();
}
}
}
As you can see we have already written out of the limit of 1KB as above program writes 1029 Bytes and not less than 1024 Bytes.
Second approach is checking the bytes according to specific encoding before writing it to file.
public class Test {
private static final int ONE_KB = 1024;
public static void main(String[] args) throws UnsupportedEncodingException {
File file = new File("D:/test.txt");
String data = "This is just a test !";
int dataLength = data.getBytes("UTF-8").length;
try (BufferedWriter writer = Files.newBufferedWriter(file.toPath())) {
while (file.length() + dataLength < ONE_KB) {
writer.write(data);
writer.flush();
}
System.out.println("1 KB Data written to the file.!");
} catch (IOException e) {
e.printStackTrace();
}
}
}
In this approach we check length of bytes prior to writing it to the file. So, it will write 1008 Bytes and it will stop writing.
Problems with both the approaches,
Write and Check : You may end up with some extra bytes and file size may cross the limit
Check and Write : You may have less bytes than the limit if next line has lot of data in it. You should be careful about the encoding.
However, there are other ways to do this validations with some third party library like apache io and I find it more cumbersome then conventional java ways.

int maxSize = 1_000_000_000;
Charset charset = StandardCharsets.UTF_F);
int size = 0;
int lineCount = 0;
while (lineCount < lines.length) {
long size2 = size + (lines[lineCount] + "\r\n").getBytes(charset).length;
if (size2 > maxSize) {
break;
}
size = size2;
++lineCount;
}
List<String> linesToWrite = lines.substring(0, lineCount);
Path path = Paths.get("D:/test.txt");
Files.write(path, linesToWrite , charset);
Or a bit faster while decoding only once:
int lineCount = 0;
try (FileChannel channel = new RandomAccessFile("D:/test.txt", "w").getChannel()) {
ByteBuffer buf = channel.map(FileChannel.MapMode.WRITE, 0, maxSize);
lineCount = lines.length;
for (int i = 0; i < lines.length; i++) {
bytes[] line = (lines.get(i) + "\r\n").getBytes(charset);
if (line.length > buffer.remaining()) {
lineCount = i;
break;
}
buffer.put(line);
}
}

IIUC, there are various ways to do it.
Keep writing data in chucks and flushing it and keep checking the file size after every flush.
Use log4j (or some logging framework) which can let us rollover to new file after certain size or time or some other trigger point.
While BufferedReader is great, there are some new APIs in java which could make it faster. Fastest way to write huge data in text file Java

Stream of short[]

Hi I need to calculate the entropy of order m of a file where m is the number of bit (m <= 16).
So:
H_m(X)=-sum_i=0 to i=2^m-1{(p_i,m)(log_2 (p_i,m))}
So, I thought to create an input stream to read the file and then calculate the probability of each sequence composed by m bit.
For m = 8 it's easy because I consider a byte.
Since that m<=16 I tought to consider as primitive type short, save each short of the file in an array short[] and then manipulate bits using bitwise operators to obtain all the sequences of m bit in the file.
Is this a good idea?
Anyway, I'm not able to create a stream of short. This is what I've done:
public static void main(String[] args) {
readFile(FILE_NAME_INPUT);
}
public static void readFile(String filename) {
short[] buffer = null;
File a_file = new File(filename);
try {
File file = new File(filename);
FileInputStream fis = new FileInputStream(filename);
DataInputStream dis = new DataInputStream(fis);
int length = (int)file.length() / 2;
buffer = new short[length];
int count = 0;
while(dis.available() > 0 && count < length) {
buffer[count] = dis.readShort();
count++;
}
System.out.println("length=" + length);
System.out.println("count=" + count);
for(int i = 0; i < buffer.length; i++) {
System.out.println("buffer[" + i + "]: " + buffer[i]);
}
fis.close();
}
catch(EOFException eof) {
System.out.println("EOFException: " + eof);
}
catch(FileNotFoundException fe) {
System.out.println("FileNotFoundException: " + fe);
}
catch(IOException ioe) {
System.out.println("IOException: " + ioe);
}
}
But I lose a byte and I don't think this is the best way to proced.
This is what I think to do using bitwise operator:
int[] list = new int[l];
foreach n in buffer {
for(int i = 16 - m; i > 0; i-m) {
list.add( (n >> i) & 2^m-1 );
}
}
I'm assuming in this case to use shorts.
If I use bytes, how can I do a cycle like that for m > 8?
That cycle doesn't work because I have to concatenate multiple bytes and each time varying the number of bits to be joined..
Any ideas?
Thanks

I think you just need to have a byte array:
public static void readFile(String filename) {
ByteArrayOutputStream outputStream=new ByteArrayOutputStream();
try {
FileInputStream fis = new FileInputStream(filename);
byte b=0;
while((b=fis.read())!=-1) {
outputStream.write(b);
}
byte[] byteData=outputStream.toByteArray();
fis.close();
}
catch(IOException ioe) {
System.out.println("IOException: " + ioe);
}
Then you can manipulate byteData as per your bitwise operations.
--
If you want to work with shorts you can combine bytes read this way
short[] buffer=new short[(int)(byteData.length/2.)+1];
j=0;
for(i=0; i<byteData.length-1; i+=2) {
buffer[j]=(short)((byteData[i]<<8)|byteData[i+1]);
j++;
}
To check for odd bytes do this
if((byteData.length%2)==1) last=(short)((0x00<<8)|byteData[byteData.length-1]]);
last is a short so it could be placed in buffer[buffer.length-1]; I'm not sure if that last position in buffer is available or occupied; I think it is but you need to check j after exiting the loop; if j's value is buffer.length-1 then it is available; otherwise might be some problem.
Then manipulate buffer.
The second approach with working with bytes is more involved. It's a question of its own. So try this above.

Unknown bytes while reading an image from ImageIO after sending it with ImageIO

While transfering images over the network using sockets, I had a -for me- strange issue:
When I wrote images to the OutputStream of one socket with ImageIO.write() and read the same images from the InputStream of the other socket with ImageIO.read() I noticed, that 16 bytes per image were sent more than read.
To be able to send multiple images in a row I had to read these bytes after every call of ImageIO.read() to not receive null because the input could not be parsed.
Does anybody know, why this is so and what these bytes are?
In this piece of code I have extracted the issue:
public class Test implements Runnable
{
public static final int COUNT = 5;
public void run()
{
try(ServerSocket server = new ServerSocket(3040))
{
Socket client = server.accept();
for(int i = 0; i < COUNT; i++)
{
final BufferedImage image = readImage(client.getInputStream());
System.out.println(image);
}
}
catch(IOException e)
{
e.printStackTrace();
}
}
private BufferedImage readImage(InputStream stream) throws IOException
{
BufferedImage image = ImageIO.read(stream);
dontKnowWhy(stream);
return image;
}
private void dontKnowWhy(InputStream stream) throws IOException
{
stream.read(new byte[16]);
}
public static void main(String... args)
{
new Thread(new Test()).start();
try(Socket server = new Socket("localhost", 3040))
{
for(int i = 0; i < COUNT; i++)
{
BufferedImage image = new BufferedImage(300, 300, BufferedImage.TYPE_INT_ARGB); //
int[] vals = new int[image.getWidth() * image.getHeight()]; //
Arrays.fill(vals, new Random().nextInt()); // Create random image
image.setRGB(0, 0, image.getWidth(), image.getHeight(), vals, 0, 1); //
ImageIO.write(image, "png", server.getOutputStream()); //send image to server
long time = System.currentTimeMillis(); //
while(time + 1000 > System.currentTimeMillis()); //wait a second
}
}
catch(IOException e)
{
e.printStackTrace();
}
}
}
I am glad about any answers, already thank you!

The "extra" bytes you see, is not read, simply because they are not needed to correctly decode the image (they are, however, most likely needed to form a fully compliant file in the chosen file format, so they are not just random "garbage" bytes).
For any given ImageIO plugin, the number of bytes left in the stream after a read may be 0, 16or any other number. It might depend on the format, the writer that wrote it, the reader, the number of images in the input, the metadata in the file, etc. In other words, relying on this behavior would be an error.
The easies way to fix this, is to prepend each image with a byte count, containing the length of the output image. This typically means you need to buffer the response on the client, to either a ByteArrayOutputStream (in-memory) or a FileOutputStream (disk).
The client then needs to read the byte count for the image, and make sure you skip any remaining bytes after the read. This can be accomplished by wrapping the input (see FilterInputStream) and keep track of the byte count internally.
(You can also read all the bytes up front, and wrapping them in a ByteArrayInputStream, before passing the data to ImageIO.read(), which is simpler but does more in-memory buffering).
After this, the client is ready do start over, by reading a new byte count, and a new image.
Another approach if you'd like less buffering on the server, could be to implement something like HTTP chunked transfer encoding, where you have multiple smaller blocks (chunks) sent to the client for each image, each prepended with its own byte count. You would need to handle the last chunk of each image especially, or insert special delimiter chunks to mark end of stream or start of a new stream.
Code below implements the buffering approach on the server, while using direct reading on the client.
Server:
DataOutputStream stream = new DataOutputStream(server.getOutputStream());
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
for (...) {
buffer.reset();
ImageIO.write(image, "png", buffer);
stream.writeInt(buffer.size());
buffer.writeTo(stream); // Send image to server
}
Client:
DataInputStream stream = new DataInputStream(client.getInputStream());
for (...) {
int size = stream.readInt();
try (InputStream imageData = new SubStream(stream, size)) {
return ImageIO.read(imageData);
}
// Note: imageData implicitly closed using try-with-resources
}
...
// Util class
private static final class SubStream extends FilterInputStream {
private final long length;
private long pos;
public SubStream(final InputStream stream, final long length) {
super(stream);
this.length = length;
}
#Override
public boolean markSupported() {
return false;
}
#Override
public int available() throws IOException {
return (int) Math.min(super.available(), length - pos);
}
#Override
public int read() throws IOException {
if (pos++ >= length) {
return -1;
}
return super.read();
}
#Override
public int read(byte[] b, int off, int len) throws IOException {
if (pos >= length) {
return -1;
}
int count = super.read(b, off, (int) Math.min(len, length - pos));
if (count < 0) {
return -1;
}
pos += count;
return count;
}
#Override
public long skip(long n) throws IOException {
if (pos >= length) {
return -1;
}
long skipped = super.skip(Math.min(n, length - pos));
if (skipped < 0) {
return -1;
}
pos += skipped;
return skipped;
}
#Override
public void close() throws IOException {
// Don't close wrapped stream, just consume any bytes left
while (pos < length) {
skip(length - pos);
}
}
}

create an ArrayList of bytes

I want to read bytes from a wave file into an array. Since the number of bytes read depends upon the size of the wave file, I'm creating a byte array with a maximum size of 1000000. But this is resulting in empty values at the end of the array. So, I wanted to create a dynamically increasing array and I found that ArrayList is the solution. But the read() function of the AudioInputStream class reads bytes only into a byte array! How do I pass the values into an ArrayList instead?

ArrayList isn't the solution, ByteArrayOutputStream is the solution. Create a ByteArrayOutputStream write your bytes to it, and then invoke toByteArray() to get the bytes.
Example of what your code should look like:
in = new BufferedInputStream(inputStream, 1024*32);
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] dataBuffer = new byte[1024 * 16];
int size = 0;
while ((size = in.read(dataBuffer)) != -1) {
out.write(dataBuffer, 0, size);
}
byte[] bytes = out.toByteArray();

You can have an array of byte like:
List<Byte> arrays = new ArrayList<Byte>();
To convert it back to arrays
Byte[] soundBytes = arrays.toArray(new Byte[arrays.size()]);
(Then, you will have to write a converter to transform Byte[] to byte[]).
EDIT: You are using List<Byte> wrong, I'll just show you how to read AudioInputStream simply with ByteArrayOutputStream.
AudioInputStream ais = ....;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int read;
while((read = ais.read()) != -1) {
baos.write(read);
}
byte[] soundBytes = baos.toByteArray();
PS An IOException is thrown if frameSize is not equal to 1. Hence use a byte buffer to read data, like so:
AudioInputStream ais = ....;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int bytesRead = 0;
while((bytesRead = ais.read(buffer)) != -1) {
baos.write(buffer, 0, bytesRead);
}
byte[] soundBytes = baos.toByteArray();

Something like this should do:
List<Byte> myBytes = new ArrayList<Byte>();
//assuming your javax.sound.sampled.AudioInputStream is called ais
while(true) {
Byte b = ais.read();
if (b != -1) { //read() returns -1 when the end of the stream is reached
myBytes.add(b);
} else {
break;
}
}
Sorry if the code is a bit wrong. I haven't done Java for a while.
Also, be careful if you do implement it as a while(true) loop :)
Edit: And here's an alternative way of doing it that reads more bytes each time:
int arrayLength = 1024;
List<Byte> myBytes = new ArrayList<Byte>();
while(true) {
Byte[] aBytes = new Byte[arrayLength];
int length = ais.read(aBytes); //length is the number of bytes read
if (length == -1) { //read() returns -1 when the end of the stream is reached
break; //or return if you implement this as a method
} else if (length == arrayLength) { //Array is full
myBytes.addAll(aBytes);
} else { //Array has been filled up to length
for (int i = 0; i < length; i++) {
myBytes.add(aBytes[i]);
}
}
}
Note that both read() methods throw an IOException - handling this is left as an exercise for the reader!

What is the best way to transfer a file using Java?

I am writing code for uploading a file from a client to my server and the performance isn't as fast as I think it should be.
I have the current code snippet that is doing the file transfer and I was wondering how I could speed up the transfer.
Sorry about all of the code:
InputStream fileItemInputStream ;
OutputStream saveFileStream;
int[] buffer;
while (fileItemInputStream.available() > 0) {
buffer = Util.getBytesFromStream(fileItemInputStream);
Util.writeIntArrToStream(saveFileStream, buffer);
}
saveFileStream.close();
fileItemInputStream.close();
The Util methods are as follows:
public static int[] getBytesFromStream(InputStream in, int size) throws IOException {
int[] b = new int[size];
int count = 0;
while (count < size) {
b[count++] = in.read();
}
return b;
}
and:
public static void writeIntArrToStream(OutputStream out, int[] arrToWrite) throws IOException {
for (int i = 0; i < arrToWrite.length; i++) {
out.write(arrToWrite[i]);
}
}

Reading a single byte at a time will be horribly inefficient. You're also relying on available, which is rarely a good idea. (It will return 0 if there are no bytes currently available, but there may be more to come.)
This is the right sort of code to copy a stream:
public void copyStream(InputStream input, OutputStream output) throws IOException
{
byte[] buffer = new byte[32*1024];
int bytesRead;
while ((bytesRead = input.read(buffer, 0, buffer.length)) > 0)
{
output.write(buffer, 0, bytesRead);
}
}
(The caller should close both streams.)