Develop Reference

Java Open BufferedWriter only once but rewrite content many times

I am running a long running operation, Say 100k jobs. i want to update the progress of it in a file once every 100 such jobs are completed.
i am opening the file using bufferedWriter with append mode as false. Writing it and then closing it. this is done once every 100 jobs are completed. So the file open and close would have happened 1000 times. Can i optimise it further by opening and closing the file only once?
public static void writeMetaData(String writeDir, JSONObject jsonObject) throws Exception {
String filePath = writeDir.concat("/").concat("metadata.txt");
BufferedWriter metaDataWriter = Files.newBufferedWriter(Paths.get(filePath), StandardCharsets.UTF_8, StandardOpenOption.TRUNCATE_EXISTING);
metaDataWriter.write(jsonObject.toString());
IOUtils.closeQuietly(metaDataWriter);
}
for(int i =0 ; i < 100000; i++) {
// do Something;
if(i % 100 == 0) {
writeMetaData(writeDir, jsonObject);
}
}
File should only have a single line.
Expected file content after 100 jobs:
progress: 100
Expected file content after 200 jobs:
progress: 200
Can this be optimised further?

First of all, an expression like writeDir.concat("/").concat("metadata.txt") is reducing readability and performance. A straight-forward writeDir + "/" + "metadata.txt" will provide better performance. But since you’re constructing a string merely for constructing a Path, it’s even more straight-forward not to do the Path’s job in your code but rather use Paths.get(writeDir, "metadata.txt").
You can not rewind a BufferedWriter but you can rewind a FileChannel. Therefore, to keep the channel open and rewind it when needed, you have to construct a new writer after rewinding:
public static void writeMetaData(FileChannel ch, JSONObject jsonObj) throws IOException {
ch.position(0);
if(ch.size() > 0) ch.truncate(0);
Writer w = Channels.newWriter(ch, StandardCharsets.UTF_8.newEncoder(), 8192);
w.write(jsonObj.toString());
w.flush();
}
try(FileChannel ch = FileChannel.open(Paths.get(writeDir, "metadata.txt"),
StandardOpenOption.WRITE, StandardOpenOption.CREATE)) {
for(int i = 0; i < 100000; i++) {
// do Something;
if(i % 100 == 0) {
writeMetaData(ch, jsonObject);
}
}
}
It’s important that the use of the Writer ends with flush() to force the write of all buffered data, but not close() as that would also close the underlying channel. Note that this code does not wrap the writer into a BufferedWriter; encoding text as UTF-8 is already a buffered operation and by requesting a larger buffer for the encoder, matching BufferedWriter’s default buffer size, we get the same effect of buffering without the copying overhead.
Since writing is not an end in itself, there’s a question left regarding your reading side. If the reader is trying to read the data in some intervals, there’s the risk of overlapping with the write, getting incomplete data.
You could use
public static void writeMetaData(FileChannel ch, JSONObject jsonObj) throws IOException {
try(FileLock lock = ch.lock()) {
ch.position(0);
if(ch.size() > 0) ch.truncate(0);
Writer w = Channels.newWriter(ch, StandardCharsets.UTF_8.newEncoder(), 8192);
w.write(jsonObj.toString());
w.flush();
}
}
to lock the file during the write. But depending on the system, file locking might not be mandatory but only affect readers also trying to get a read lock.
When you use JDK 11 or newer, you may consider using
for(int i = 0; i < 100000; i++) {
// do Something;
if(i % 100 == 0) {
Files.writeString(Paths.get(writeDir, "metadata.txt"), jsonObject.toString());
}
}
which clearly wins on simplicity (yes, that’s the complete code, no additional method required). The default options do already include the desired StandardCharsets.UTF_8 and StandardOpenOption.TRUNCATE_EXISTING.
While it does open and close the file internally, it has some other performance tweaks which may compensate. Especially in the likely case that the string consists of ASCII characters only, as the implementation will simply write the string’s internal array directly to the file then.

A Stream does not allow to go back and rewrite content. A way to achieve what you want is using a RandomAccessFile.
Its setLength() method will truncate the file if you pass 0.
Here is a simple example:
import java.io.*;
public class Test
{
public static void updateFile(RandomAccessFile raf, String content) throws IOException
{
raf.setLength(0);
raf.write(content.getBytes("UTF-8"));
}
public static void main(String[] args) throws IOException
{
try(RandomAccessFile raf = new RandomAccessFile("metadata.txt", "rw"))
{
updateFile(raf, "progress: 100");
updateFile(raf, "progress: 200");
}
}
}

File operations are typically buffered by the underlying kernel, and so you're unlikely to see much of a performance benefit by keeping an open file descriptor for this kind of low throughput application.
Keeping your code as a single operation that gets to leave no state after it finishes, rather than designing it as a continuous rewindable stream, makes for an elegant, simple, and unless you've specifically requested synchronous IO, then also sufficiently performant implementation that gets to benefit from the optimizations of all of the layers that sit beneath it.
When you do get measurable impedance to performance by this, which I suspect you never will, you could use the RandomAccessFile API, or go unnecessarily lower level by using FileChannel as others already specified.
I think you shouldn't compromise the simplicity/elegance of your design for this kind of micro-optimization, which in the grand scheme of things, is guaranteed to be insignificant (one tiny write operation per 100 jobs processed).

Limit the content available from a Java NIO Channel (File or Socket)

I'm pretty new to NIO and wanted to implement some feature with it, instead of typical Streams (which can do all sort of things).
What I'm not sure I can get is reading from a file into a buffer and limiting the content that I will transfer. Let's say from position 100 to 200 (even if file length is 1000). It also would be nice to do on network sockets.
I know that NIO keeps things basic to leverage OS capabilities that's why I'm not sure it can be done.
I was thinking that a tricky way to do it would be a 'LimitedReadChannel' that when it's should return less than the available buffer size it uses another byte-buffer and then transfer to the original one (1). But seems more tricky than necessary. I also don't want to use anything related to streams because it would defeat the purpose of using NIO.
(1) So far....
LimitedChannel.read(buffer) {
if (buffer.available?? > contentLeft) {
delegateChannel.read(smallerBuffer);
// transfer from smallerBuffer to buffer
} else {
delegateChannel.read(buffer);
}
}

I've found that Buffers admit to ask for the current limit or set a new one. So that wrapper channel (the one that limits the effective number of bytes read) could modify the buffer limit to avoid reading more...
Something like:
// LimitedChannel.java
// private int bytesLeft; // remaining amount of bytes to read
public int read(ByteBuffer buffer) {
if (bytesLeft <= 0) {
return -1;
}
int oldLimit = buffer.limit();
if (bytesLeft < buffer.remaining()) {
// ensure I'm not reading more than allowed
buffer.limit(buffer.position() + bytesLeft);
}
int bytesRead = delegateChannel.read(buffer);
bytesLeft -= bytesRead;
buffer.limit(oldLimit);
return bytesRead;
}
Anyway not sure if this already exists somewhere. It's difficult to find documentation about this use case...

how to divide huge text file into small in java [duplicate]

I need the advice from someone who knows Java very well and the memory issues.
I have a large file (something like 1.5GB) and I need to cut this file in many (100 small files for example) smaller files.
I know generally how to do it (using a BufferedReader), but I would like to know if you have any advice regarding the memory, or tips how to do it faster.
My file contains text, it is not binary and I have about 20 character per line.

To save memory, do not unnecessarily store/duplicate the data in memory (i.e. do not assign them to variables outside the loop). Just process the output immediately as soon as the input comes in.
It really doesn't matter whether you're using BufferedReader or not. It will not cost significantly much more memory as some implicitly seem to suggest. It will at highest only hit a few % from performance. The same applies on using NIO. It will only improve scalability, not memory use. It will only become interesting when you've hundreds of threads running on the same file.
Just loop through the file, write every line immediately to other file as you read in, count the lines and if it reaches 100, then switch to next file, etcetera.
Kickoff example:
String encoding = "UTF-8";
int maxlines = 100;
BufferedReader reader = null;
BufferedWriter writer = null;
try {
reader = new BufferedReader(new InputStreamReader(new FileInputStream("/bigfile.txt"), encoding));
int count = 0;
for (String line; (line = reader.readLine()) != null;) {
if (count++ % maxlines == 0) {
close(writer);
writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream("/smallfile" + (count / maxlines) + ".txt"), encoding));
}
writer.write(line);
writer.newLine();
}
} finally {
close(writer);
close(reader);
}

First, if your file contains binary data, then using BufferedReader would be a big mistake (because you would be converting the data to String, which is unnecessary and could easily corrupt the data); you should use a BufferedInputStream instead. If it's text data and you need to split it along linebreaks, then using BufferedReader is OK (assuming the file contains lines of a sensible length).
Regarding memory, there shouldn't be any problem if you use a decently sized buffer (I'd use at least 1MB to make sure the HD is doing mostly sequential reading and writing).
If speed turns out to be a problem, you could have a look at the java.nio packages - those are supposedly faster than java.io,

You can consider using memory-mapped files, via FileChannels .
Generally a lot faster for large files. There are performance trade-offs that could make it slower, so YMMV.
Related answer: Java NIO FileChannel versus FileOutputstream performance / usefulness

This is a very good article:
http://java.sun.com/developer/technicalArticles/Programming/PerfTuning/
In summary, for great performance, you should:
Avoid accessing the disk.
Avoid accessing the underlying operating system.
Avoid method calls.
Avoid processing bytes and characters individually.
For example, to reduce the access to disk, you can use a large buffer. The article describes various approaches.

Does it have to be done in Java? I.e. does it need to be platform independent? If not, I'd suggest using the 'split' command in *nix. If you really wanted, you could execute this command via your java program. While I haven't tested, I imagine it perform faster than whatever Java IO implementation you could come up with.

You can use java.nio which is faster than classical Input/Output stream:
http://java.sun.com/javase/6/docs/technotes/guides/io/index.html

Yes.
I also think that using read() with arguments like read(Char[], int init, int end) is a better way to read a such a large file
(Eg : read(buffer,0,buffer.length))
And I also experienced the problem of missing values of using the BufferedReader instead of BufferedInputStreamReader for a binary data input stream. So, using the BufferedInputStreamReader is a much better in this like case.

package all.is.well;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import junit.framework.TestCase;
/**
* #author Naresh Bhabat
*
Following implementation helps to deal with extra large files in java.
This program is tested for dealing with 2GB input file.
There are some points where extra logic can be added in future.
Pleasenote: if we want to deal with binary input file, then instead of reading line,we need to read bytes from read file object.
It uses random access file,which is almost like streaming API.
* ****************************************
Notes regarding executor framework and its readings.
Please note :ExecutorService executor = Executors.newFixedThreadPool(10);
* for 10 threads:Total time required for reading and writing the text in
* :seconds 349.317
*
* For 100:Total time required for reading the text and writing : seconds 464.042
*
* For 1000 : Total time required for reading and writing text :466.538
* For 10000 Total time required for reading and writing in seconds 479.701
*
*
*/
public class DealWithHugeRecordsinFile extends TestCase {
static final String FILEPATH = "C:\\springbatch\\bigfile1.txt.txt";
static final String FILEPATH_WRITE = "C:\\springbatch\\writinghere.txt";
static volatile RandomAccessFile fileToWrite;
static volatile RandomAccessFile file;
static volatile String fileContentsIter;
static volatile int position = 0;
public static void main(String[] args) throws IOException, InterruptedException {
long currentTimeMillis = System.currentTimeMillis();
try {
fileToWrite = new RandomAccessFile(FILEPATH_WRITE, "rw");//for random write,independent of thread obstacles
file = new RandomAccessFile(FILEPATH, "r");//for random read,independent of thread obstacles
seriouslyReadProcessAndWriteAsynch();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Thread currentThread = Thread.currentThread();
System.out.println(currentThread.getName());
long currentTimeMillis2 = System.currentTimeMillis();
double time_seconds = (currentTimeMillis2 - currentTimeMillis) / 1000.0;
System.out.println("Total time required for reading the text in seconds " + time_seconds);
}
/**
* #throws IOException
* Something asynchronously serious
*/
public static void seriouslyReadProcessAndWriteAsynch() throws IOException {
ExecutorService executor = Executors.newFixedThreadPool(10);//pls see for explanation in comments section of the class
while (true) {
String readLine = file.readLine();
if (readLine == null) {
break;
}
Runnable genuineWorker = new Runnable() {
#Override
public void run() {
// do hard processing here in this thread,i have consumed
// some time and ignore some exception in write method.
writeToFile(FILEPATH_WRITE, readLine);
// System.out.println(" :" +
// Thread.currentThread().getName());
}
};
executor.execute(genuineWorker);
}
executor.shutdown();
while (!executor.isTerminated()) {
}
System.out.println("Finished all threads");
file.close();
fileToWrite.close();
}
/**
* #param filePath
* #param data
* #param position
*/
private static void writeToFile(String filePath, String data) {
try {
// fileToWrite.seek(position);
data = "\n" + data;
if (!data.contains("Randomization")) {
return;
}
System.out.println("Let us do something time consuming to make this thread busy"+(position++) + " :" + data);
System.out.println("Lets consume through this loop");
int i=1000;
while(i>0){
i--;
}
fileToWrite.write(data.getBytes());
throw new Exception();
} catch (Exception exception) {
System.out.println("exception was thrown but still we are able to proceeed further"
+ " \n This can be used for marking failure of the records");
//exception.printStackTrace();
}
}
}

Don't use read without arguments.
It's very slow.
Better read it to buffer and move it to file quickly.
Use bufferedInputStream because it supports binary reading.
And it's all.

Unless you accidentally read in the whole input file instead of reading it line by line, then your primary limitation will be disk speed. You may want to try starting with a file containing 100 lines and write it to 100 different files one line in each and make the triggering mechanism work on the number of lines written to the current file. That program will be easily scalable to your situation.

How to convert OutputStream to InputStream?

I am on the stage of development, where I have two modules and from one I got output as a OutputStream and second one, which accepts only InputStream. Do you know how to convert OutputStream to InputStream (not vice versa, I mean really this way) that I will be able to connect these two parts?
Thanks

There seem to be many links and other such stuff, but no actual code using pipes. The advantage of using java.io.PipedInputStream and java.io.PipedOutputStream is that there is no additional consumption of memory. ByteArrayOutputStream.toByteArray() returns a copy of the original buffer, so that means that whatever you have in memory, you now have two copies of it. Then writing to an InputStream means you now have three copies of the data.
The code using lambdas (hat-tip to #John Manko from the comments):
PipedInputStream in = new PipedInputStream();
final PipedOutputStream out = new PipedOutputStream(in);
// in a background thread, write the given output stream to the
// PipedOutputStream for consumption
new Thread(() -> {originalOutputStream.writeTo(out);}).start();
One thing that #John Manko noted is that in certain cases, when you don't have control of the creation of the OutputStream, you may end up in a situation where the creator may clean up the OutputStream object prematurely. If you are getting the ClosedPipeException, then you should try inverting the constructors:
PipedInputStream in = new PipedInputStream(out);
new Thread(() -> {originalOutputStream.writeTo(out);}).start();
Note you can invert the constructors for the examples below too.
Thanks also to #AlexK for correcting me with starting a Thread instead of just kicking off a Runnable.
The code using try-with-resources:
// take the copy of the stream and re-write it to an InputStream
PipedInputStream in = new PipedInputStream();
new Thread(new Runnable() {
public void run () {
// try-with-resources here
// putting the try block outside the Thread will cause the
// PipedOutputStream resource to close before the Runnable finishes
try (final PipedOutputStream out = new PipedOutputStream(in)) {
// write the original OutputStream to the PipedOutputStream
// note that in order for the below method to work, you need
// to ensure that the data has finished writing to the
// ByteArrayOutputStream
originalByteArrayOutputStream.writeTo(out);
}
catch (IOException e) {
// logging and exception handling should go here
}
}
}).start();
The original code I wrote:
// take the copy of the stream and re-write it to an InputStream
PipedInputStream in = new PipedInputStream();
final PipedOutputStream out = new PipedOutputStream(in);
new Thread(new Runnable() {
public void run () {
try {
// write the original OutputStream to the PipedOutputStream
// note that in order for the below method to work, you need
// to ensure that the data has finished writing to the
// ByteArrayOutputStream
originalByteArrayOutputStream.writeTo(out);
}
catch (IOException e) {
// logging and exception handling should go here
}
finally {
// close the PipedOutputStream here because we're done writing data
// once this thread has completed its run
if (out != null) {
// close the PipedOutputStream cleanly
out.close();
}
}
}
}).start();
This code assumes that the originalByteArrayOutputStream is a ByteArrayOutputStream as it is usually the only usable output stream, unless you're writing to a file. The great thing about this is that since it's in a separate thread, it also is working in parallel, so whatever is consuming your input stream will be streaming out of your old output stream too. That is beneficial because the buffer can remain smaller and you'll have less latency and less memory usage.
If you don't have a ByteArrayOutputStream, then instead of using writeTo(), you will have to use one of the write() methods in the java.io.OutputStream class or one of the other methods available in a subclass.

An OutputStream is one where you write data to. If some module exposes an OutputStream, the expectation is that there is something reading at the other end.
Something that exposes an InputStream, on the other hand, is indicating that you will need to listen to this stream, and there will be data that you can read.
So it is possible to connect an InputStream to an OutputStream
InputStream----read---> intermediateBytes[n] ----write----> OutputStream
As someone metioned, this is what the copy() method from IOUtils lets you do. It does not make sense to go the other way... hopefully this makes some sense
UPDATE:
Of course the more I think of this, the more I can see how this actually would be a requirement. I know some of the comments mentioned Piped input/ouput streams, but there is another possibility.
If the output stream that is exposed is a ByteArrayOutputStream, then you can always get the full contents by calling the toByteArray() method. Then you can create an input stream wrapper by using the ByteArrayInputStream sub-class. These two are pseudo-streams, they both basically just wrap an array of bytes. Using the streams this way, therefore, is technically possible, but to me it is still very strange...

As input and output streams are just start and end point, the solution is to temporary store data in byte array. So you must create intermediate ByteArrayOutputStream, from which you create byte[] that is used as input for new ByteArrayInputStream.
public void doTwoThingsWithStream(InputStream inStream, OutputStream outStream){
//create temporary bayte array output stream
ByteArrayOutputStream baos = new ByteArrayOutputStream();
doFirstThing(inStream, baos);
//create input stream from baos
InputStream isFromFirstData = new ByteArrayInputStream(baos.toByteArray());
doSecondThing(isFromFirstData, outStream);
}
Hope it helps.

ByteArrayOutputStream buffer = (ByteArrayOutputStream) aOutputStream;
byte[] bytes = buffer.toByteArray();
InputStream inputStream = new ByteArrayInputStream(bytes);

You will need an intermediate class which will buffer between. Each time InputStream.read(byte[]...) is called, the buffering class will fill the passed in byte array with the next chunk passed in from OutputStream.write(byte[]...). Since the sizes of the chunks may not be the same, the adapter class will need to store a certain amount until it has enough to fill the read buffer and/or be able to store up any buffer overflow.
This article has a nice breakdown of a few different approaches to this problem:
http://blog.ostermiller.org/convert-java-outputstream-inputstream

The easystream open source library has direct support to convert an OutputStream to an InputStream: http://io-tools.sourceforge.net/easystream/tutorial/tutorial.html
// create conversion
final OutputStreamToInputStream<Void> out = new OutputStreamToInputStream<Void>() {
#Override
protected Void doRead(final InputStream in) throws Exception {
LibraryClass2.processDataFromInputStream(in);
return null;
}
};
try {
LibraryClass1.writeDataToTheOutputStream(out);
} finally {
// don't miss the close (or a thread would not terminate correctly).
out.close();
}
They also list other options: http://io-tools.sourceforge.net/easystream/outputstream_to_inputstream/implementations.html
Write the data the data into a memory buffer (ByteArrayOutputStream) get the byteArray and read it again with a ByteArrayInputStream. This is the best approach if you're sure your data fits into memory.
Copy your data to a temporary file and read it back.
Use pipes: this is the best approach both for memory usage and speed (you can take full advantage of the multi-core processors) and also the standard solution offered by Sun.
Use InputStreamFromOutputStream and OutputStreamToInputStream from the easystream library.

I encountered the same problem with converting a ByteArrayOutputStream to a ByteArrayInputStream and solved it by using a derived class from ByteArrayOutputStream which is able to return a ByteArrayInputStream that is initialized with the internal buffer of the ByteArrayOutputStream. This way no additional memory is used and the 'conversion' is very fast:
package info.whitebyte.utils;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
/**
* This class extends the ByteArrayOutputStream by
* providing a method that returns a new ByteArrayInputStream
* which uses the internal byte array buffer. This buffer
* is not copied, so no additional memory is used. After
* creating the ByteArrayInputStream the instance of the
* ByteArrayInOutStream can not be used anymore.
* <p>
* The ByteArrayInputStream can be retrieved using <code>getInputStream()</code>.
* #author Nick Russler
*/
public class ByteArrayInOutStream extends ByteArrayOutputStream {
/**
* Creates a new ByteArrayInOutStream. The buffer capacity is
* initially 32 bytes, though its size increases if necessary.
*/
public ByteArrayInOutStream() {
super();
}
/**
* Creates a new ByteArrayInOutStream, with a buffer capacity of
* the specified size, in bytes.
*
* #param size the initial size.
* #exception IllegalArgumentException if size is negative.
*/
public ByteArrayInOutStream(int size) {
super(size);
}
/**
* Creates a new ByteArrayInputStream that uses the internal byte array buffer
* of this ByteArrayInOutStream instance as its buffer array. The initial value
* of pos is set to zero and the initial value of count is the number of bytes
* that can be read from the byte array. The buffer array is not copied. This
* instance of ByteArrayInOutStream can not be used anymore after calling this
* method.
* #return the ByteArrayInputStream instance
*/
public ByteArrayInputStream getInputStream() {
// create new ByteArrayInputStream that respects the current count
ByteArrayInputStream in = new ByteArrayInputStream(this.buf, 0, this.count);
// set the buffer of the ByteArrayOutputStream
// to null so it can't be altered anymore
this.buf = null;
return in;
}
}
I put the stuff on github: https://github.com/nickrussler/ByteArrayInOutStream

The library io-extras may be useful. For example if you want to gzip an InputStream using GZIPOutputStream and you want it to happen synchronously (using the default buffer size of 8192):
InputStream is = ...
InputStream gz = IOUtil.pipe(is, o -> new GZIPOutputStream(o));
Note that the library has 100% unit test coverage (for what that's worth of course!) and is on Maven Central. The Maven dependency is:
<dependency>
<groupId>com.github.davidmoten</groupId>
<artifactId>io-extras</artifactId>
<version>0.1</version>
</dependency>
Be sure to check for a later version.

From my point of view, java.io.PipedInputStream/java.io.PipedOutputStream is the best option to considere. In some situations you may want to use ByteArrayInputStream/ByteArrayOutputStream. The problem is that you need to duplicate the buffer to convert a ByteArrayOutputStream to a ByteArrayInputStream. Also ByteArrayOutpuStream/ByteArrayInputStream are limited to 2GB. Here is an OutpuStream/InputStream implementation I wrote to bypass ByteArrayOutputStream/ByteArrayInputStream limitations (Scala code, but easily understandable for java developpers):
import java.io.{IOException, InputStream, OutputStream}
import scala.annotation.tailrec
/** Acts as a replacement for ByteArrayOutputStream
*
*/
class HugeMemoryOutputStream(capacity: Long) extends OutputStream {
private val PAGE_SIZE: Int = 1024000
private val ALLOC_STEP: Int = 1024
/** Pages array
*
*/
private var streamBuffers: Array[Array[Byte]] = Array.empty[Array[Byte]]
/** Allocated pages count
*
*/
private var pageCount: Int = 0
/** Allocated bytes count
*
*/
private var allocatedBytes: Long = 0
/** Current position in stream
*
*/
private var position: Long = 0
/** Stream length
*
*/
private var length: Long = 0
allocSpaceIfNeeded(capacity)
/** Gets page count based on given length
*
* #param length Buffer length
* #return Page count to hold the specified amount of data
*/
private def getPageCount(length: Long) = {
var pageCount = (length / PAGE_SIZE).toInt + 1
if ((length % PAGE_SIZE) == 0) {
pageCount -= 1
}
pageCount
}
/** Extends pages array
*
*/
private def extendPages(): Unit = {
if (streamBuffers.isEmpty) {
streamBuffers = new Array[Array[Byte]](ALLOC_STEP)
}
else {
val newStreamBuffers = new Array[Array[Byte]](streamBuffers.length + ALLOC_STEP)
Array.copy(streamBuffers, 0, newStreamBuffers, 0, streamBuffers.length)
streamBuffers = newStreamBuffers
}
pageCount = streamBuffers.length
}
/** Ensures buffers are bug enough to hold specified amount of data
*
* #param value Amount of data
*/
private def allocSpaceIfNeeded(value: Long): Unit = {
#tailrec
def allocSpaceIfNeededIter(value: Long): Unit = {
val currentPageCount = getPageCount(allocatedBytes)
val neededPageCount = getPageCount(value)
if (currentPageCount < neededPageCount) {
if (currentPageCount == pageCount) extendPages()
streamBuffers(currentPageCount) = new Array[Byte](PAGE_SIZE)
allocatedBytes = (currentPageCount + 1).toLong * PAGE_SIZE
allocSpaceIfNeededIter(value)
}
}
if (value < 0) throw new Error("AllocSpaceIfNeeded < 0")
if (value > 0) {
allocSpaceIfNeededIter(value)
length = Math.max(value, length)
if (position > length) position = length
}
}
/**
* Writes the specified byte to this output stream. The general
* contract for <code>write</code> is that one byte is written
* to the output stream. The byte to be written is the eight
* low-order bits of the argument <code>b</code>. The 24
* high-order bits of <code>b</code> are ignored.
* <p>
* Subclasses of <code>OutputStream</code> must provide an
* implementation for this method.
*
* #param b the <code>byte</code>.
*/
#throws[IOException]
override def write(b: Int): Unit = {
val buffer: Array[Byte] = new Array[Byte](1)
buffer(0) = b.toByte
write(buffer)
}
/**
* Writes <code>len</code> bytes from the specified byte array
* starting at offset <code>off</code> to this output stream.
* The general contract for <code>write(b, off, len)</code> is that
* some of the bytes in the array <code>b</code> are written to the
* output stream in order; element <code>b[off]</code> is the first
* byte written and <code>b[off+len-1]</code> is the last byte written
* by this operation.
* <p>
* The <code>write</code> method of <code>OutputStream</code> calls
* the write method of one argument on each of the bytes to be
* written out. Subclasses are encouraged to override this method and
* provide a more efficient implementation.
* <p>
* If <code>b</code> is <code>null</code>, a
* <code>NullPointerException</code> is thrown.
* <p>
* If <code>off</code> is negative, or <code>len</code> is negative, or
* <code>off+len</code> is greater than the length of the array
* <code>b</code>, then an <tt>IndexOutOfBoundsException</tt> is thrown.
*
* #param b the data.
* #param off the start offset in the data.
* #param len the number of bytes to write.
*/
#throws[IOException]
override def write(b: Array[Byte], off: Int, len: Int): Unit = {
#tailrec
def writeIter(b: Array[Byte], off: Int, len: Int): Unit = {
val currentPage: Int = (position / PAGE_SIZE).toInt
val currentOffset: Int = (position % PAGE_SIZE).toInt
if (len != 0) {
val currentLength: Int = Math.min(PAGE_SIZE - currentOffset, len)
Array.copy(b, off, streamBuffers(currentPage), currentOffset, currentLength)
position += currentLength
writeIter(b, off + currentLength, len - currentLength)
}
}
allocSpaceIfNeeded(position + len)
writeIter(b, off, len)
}
/** Gets an InputStream that points to HugeMemoryOutputStream buffer
*
* #return InputStream
*/
def asInputStream(): InputStream = {
new HugeMemoryInputStream(streamBuffers, length)
}
private class HugeMemoryInputStream(streamBuffers: Array[Array[Byte]], val length: Long) extends InputStream {
/** Current position in stream
*
*/
private var position: Long = 0
/**
* Reads the next byte of data from the input stream. The value byte is
* returned as an <code>int</code> in the range <code>0</code> to
* <code>255</code>. If no byte is available because the end of the stream
* has been reached, the value <code>-1</code> is returned. This method
* blocks until input data is available, the end of the stream is detected,
* or an exception is thrown.
*
* <p> A subclass must provide an implementation of this method.
*
* #return the next byte of data, or <code>-1</code> if the end of the
* stream is reached.
*/
#throws[IOException]
def read: Int = {
val buffer: Array[Byte] = new Array[Byte](1)
if (read(buffer) == 0) throw new Error("End of stream")
else buffer(0)
}
/**
* Reads up to <code>len</code> bytes of data from the input stream into
* an array of bytes. An attempt is made to read as many as
* <code>len</code> bytes, but a smaller number may be read.
* The number of bytes actually read is returned as an integer.
*
* <p> This method blocks until input data is available, end of file is
* detected, or an exception is thrown.
*
* <p> If <code>len</code> is zero, then no bytes are read and
* <code>0</code> is returned; otherwise, there is an attempt to read at
* least one byte. If no byte is available because the stream is at end of
* file, the value <code>-1</code> is returned; otherwise, at least one
* byte is read and stored into <code>b</code>.
*
* <p> The first byte read is stored into element <code>b[off]</code>, the
* next one into <code>b[off+1]</code>, and so on. The number of bytes read
* is, at most, equal to <code>len</code>. Let <i>k</i> be the number of
* bytes actually read; these bytes will be stored in elements
* <code>b[off]</code> through <code>b[off+</code><i>k</i><code>-1]</code>,
* leaving elements <code>b[off+</code><i>k</i><code>]</code> through
* <code>b[off+len-1]</code> unaffected.
*
* <p> In every case, elements <code>b[0]</code> through
* <code>b[off]</code> and elements <code>b[off+len]</code> through
* <code>b[b.length-1]</code> are unaffected.
*
* <p> The <code>read(b,</code> <code>off,</code> <code>len)</code> method
* for class <code>InputStream</code> simply calls the method
* <code>read()</code> repeatedly. If the first such call results in an
* <code>IOException</code>, that exception is returned from the call to
* the <code>read(b,</code> <code>off,</code> <code>len)</code> method. If
* any subsequent call to <code>read()</code> results in a
* <code>IOException</code>, the exception is caught and treated as if it
* were end of file; the bytes read up to that point are stored into
* <code>b</code> and the number of bytes read before the exception
* occurred is returned. The default implementation of this method blocks
* until the requested amount of input data <code>len</code> has been read,
* end of file is detected, or an exception is thrown. Subclasses are encouraged
* to provide a more efficient implementation of this method.
*
* #param b the buffer into which the data is read.
* #param off the start offset in array <code>b</code>
* at which the data is written.
* #param len the maximum number of bytes to read.
* #return the total number of bytes read into the buffer, or
* <code>-1</code> if there is no more data because the end of
* the stream has been reached.
* #see java.io.InputStream#read()
*/
#throws[IOException]
override def read(b: Array[Byte], off: Int, len: Int): Int = {
#tailrec
def readIter(acc: Int, b: Array[Byte], off: Int, len: Int): Int = {
val currentPage: Int = (position / PAGE_SIZE).toInt
val currentOffset: Int = (position % PAGE_SIZE).toInt
val count: Int = Math.min(len, length - position).toInt
if (count == 0 || position >= length) acc
else {
val currentLength = Math.min(PAGE_SIZE - currentOffset, count)
Array.copy(streamBuffers(currentPage), currentOffset, b, off, currentLength)
position += currentLength
readIter(acc + currentLength, b, off + currentLength, len - currentLength)
}
}
readIter(0, b, off, len)
}
/**
* Skips over and discards <code>n</code> bytes of data from this input
* stream. The <code>skip</code> method may, for a variety of reasons, end
* up skipping over some smaller number of bytes, possibly <code>0</code>.
* This may result from any of a number of conditions; reaching end of file
* before <code>n</code> bytes have been skipped is only one possibility.
* The actual number of bytes skipped is returned. If <code>n</code> is
* negative, the <code>skip</code> method for class <code>InputStream</code> always
* returns 0, and no bytes are skipped. Subclasses may handle the negative
* value differently.
*
* The <code>skip</code> method of this class creates a
* byte array and then repeatedly reads into it until <code>n</code> bytes
* have been read or the end of the stream has been reached. Subclasses are
* encouraged to provide a more efficient implementation of this method.
* For instance, the implementation may depend on the ability to seek.
*
* #param n the number of bytes to be skipped.
* #return the actual number of bytes skipped.
*/
#throws[IOException]
override def skip(n: Long): Long = {
if (n < 0) 0
else {
position = Math.min(position + n, length)
length - position
}
}
}
}
Easy to use, no buffer duplication, no 2GB memory limit
val out: HugeMemoryOutputStream = new HugeMemoryOutputStream(initialCapacity /*may be 0*/)
out.write(...)
...
val in1: InputStream = out.asInputStream()
in1.read(...)
...
val in2: InputStream = out.asInputStream()
in2.read(...)
...

As some here have answered already, there is no efficient way to just ‘convert’ an OutputStream to an InputStream. The trick to solve a problem like yours is to execute all code that requires the OutputStream into its own thread. By using piped streams, we can then transfer the data out of the created thread over into an InputStream.
Example usage:
public static InputStream downloadFileAsStream(final String uriString) throws IOException {
final InputStream inputStream = runInOwnThreadWithPipedStreams((outputStream) -> {
try {
downloadUriToStream(uriString, outputStream);
} catch (final Exception e) {
LOGGER.error("Download of uri '{}' has failed", uriString, e);
}
});
return inputStream;
}
Helper function:
public static InputStream runInOwnThreadWithPipedStreams(
final Consumer<OutputStream> outputStreamConsumer) throws IOException {
final PipedInputStream inputStream = new PipedInputStream();
final PipedOutputStream outputStream = new PipedOutputStream(inputStream);
new Thread(new Runnable() {
public void run() {
try {
outputStreamConsumer.accept(outputStream);
} finally {
try {
outputStream.close();
} catch (final IOException e) {
LOGGER.error("Closing outputStream has failed. ", e);
}
}
}
}).start();
return inputStream;
}
Unit Test:
#Test
void testRunInOwnThreadWithPipedStreams() throws IOException {
final InputStream inputStream = LoadFileUtil.runInOwnThreadWithPipedStreams((OutputStream outputStream) -> {
try {
IOUtils.copy(IOUtils.toInputStream("Hello World", StandardCharsets.UTF_8), outputStream);
} catch (final IOException e) {
LoggerFactory.getLogger(LoadFileUtilTest.class).error(e.getMessage(), e);
}
});
final String actualResult = IOUtils.toString(inputStream, StandardCharsets.UTF_8);
Assertions.assertEquals("Hello World", actualResult);
}

If you want to make an OutputStream from an InputStream there is one basic problem. A method writing to an OutputStream blocks until it is done. So the result is available when the writing method is finished. This has 2 consequences:
If you use only one thread, you need to wait until everything is written (so you need to store the stream's data in memory or disk).
If you want to access the data before it is finished, you need a second thread.
Variant 1 can be implemented using byte arrays or filed.
Variant 1 can be implemented using pipies (either directly or with extra abstraction - e.g. RingBuffer or the google lib from the other comment).
Indeed with standard java there is no other way to solve the problem. Each solution is an implementataion of one of these.
There is one concept called "continuation" (see wikipedia for details). In this case basically this means:
there is a special output stream that expects a certain amount of data
if the ammount is reached, the stream gives control to it's counterpart which is a special input stream
the input stream makes the amount of data available until it is read, after that, it passes back the control to the output stream
While some languages have this concept built in, for java you need some "magic". For example "commons-javaflow" from apache implements such for java. The disadvantage is that this requires some special bytecode modifications at build time. So it would make sense to put all the stuff in an extra library whith custom build scripts.

Though you cannot convert an OutputStream to an InputStream, java provides a way using PipedOutputStream and PipedInputStream that you can have data written to a PipedOutputStream to become available through an associated PipedInputStream. Sometime back I faced a similar situation when dealing with third party libraries that required an InputStream instance to be passed to them instead of an OutputStream instance. The way I fixed this issue is to use the PipedInputStream and PipedOutputStream. By the way they are tricky to use and you must use multithreading to achieve what you want. I recently published an implementation on github which you can use. Here is the link . You can go through the wiki to understand how to use it.

Old post but might help others, Use this way:
OutputStream out = new ByteArrayOutputStream();
...
out.write();
...
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(out.toString().getBytes()));

Questions related to writing your own file downloader using multiple threads java

In my current company, i am doing a PoC on how we can write a file downloader utility. We have to use socket programming(TCP/IP) for downloading the files. One of the requirements of the client is that a file(which will be large in size) should be transfered in chunks for example if we have a file of 5Mb size then we can have 5 threads which transfer 1 Mb each. I have written a small application which downloads a file. You can download the eclipe project
from http://www.fileflyer.com/view/QM1JSC0
A brief explanation of my classes
FileSender.java : This class provides the bytes of file. It has a method called
sendBytesOfFile(long start,long end, long sequenceNo) which gives the number of bytes.
import java.io.File;
import java.io.IOException;
import java.util.zip.CRC32;
import org.apache.commons.io.FileUtils;
public class FileSender {
private static final String FILE_NAME = "C:\\shared\\test.pdf";
public ByteArrayWrapper sendBytesOfFile(long start,long end, long sequenceNo){
try {
File file = new File(FILE_NAME);
byte[] fileBytes = FileUtils.readFileToByteArray(file);
System.out.println("Size of file is " +fileBytes.length);
System.out.println();
System.out.println("Start "+start +" end "+end);
byte[] bytes = getByteArray(fileBytes, start, end);
ByteArrayWrapper wrapper = new ByteArrayWrapper(bytes, sequenceNo);
return wrapper;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
private byte[] getByteArray(byte[] bytes, long start, long end){
long arrayLength = end-start;
System.out.println("Start : "+start +" end : "+end + " Arraylength : "+arrayLength +" length of source array : "+bytes.length);
byte[] arr = new byte[(int)arrayLength];
for(int i = (int)start, j =0; i < end;i++,j++){
arr[j] = bytes[i];
}
return arr;
}
public static long fileSize(){
File file = new File(FILE_NAME);
return file.length();
}
}
FileReceiver.java - This class receives the file.
Small Explanation what this file does
This class finds the size of the file to be fetched from Sender
Depending upon the size of the file it finds the start and end position till the bytes needs to be read.
It starts n number of threads giving each thread start,end, sequence number and a list which all the threads share.
Each thread reads the number of bytes and creates a ByteArrayWrapper.
ByteArrayWrapper objects are added to the list
Then i have while loop which basically make sure that all threads have done their work
finally it sorts the list based on the sequence number.
then the bytes are joined, and a complete byte array is formed which is converted to a file.
Code of File Receiver
package com.filedownloader;
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.zip.CRC32;
import org.apache.commons.io.FileUtils;
public class FileReceiver {
public static void main(String[] args) {
FileReceiver receiver = new FileReceiver();
receiver.receiveFile();
}
public void receiveFile(){
long startTime = System.currentTimeMillis();
long numberOfThreads = 10;
long filesize = FileSender.fileSize();
System.out.println("File size received "+filesize);
long start = filesize/numberOfThreads;
List<ByteArrayWrapper> list = new ArrayList<ByteArrayWrapper>();
for(long threadCount =0; threadCount<numberOfThreads ;threadCount++){
FileDownloaderTask task = new FileDownloaderTask(threadCount*start,(threadCount+1)*start,threadCount,list);
new Thread(task).start();
}
while(list.size() != numberOfThreads){
// this is done so that all the threads should complete their work before processing further.
//System.out.println("Waiting for threads to complete. List size "+list.size());
}
if(list.size() == numberOfThreads){
System.out.println("All bytes received "+list);
Collections.sort(list, new Comparator<ByteArrayWrapper>() {
#Override
public int compare(ByteArrayWrapper o1, ByteArrayWrapper o2) {
long sequence1 = o1.getSequence();
long sequence2 = o2.getSequence();
if(sequence1 < sequence2){
return -1;
}else if(sequence1 > sequence2){
return 1;
}
else{
return 0;
}
}
});
byte[] totalBytes = list.get(0).getBytes();
byte[] firstArr = null;
byte[] secondArr = null;
for(int i = 1;i<list.size();i++){
firstArr = totalBytes;
secondArr = list.get(i).getBytes();
totalBytes = concat(firstArr, secondArr);
}
System.out.println(totalBytes.length);
convertToFile(totalBytes,"c:\\tmp\\test.pdf");
long endTime = System.currentTimeMillis();
System.out.println("Total time taken with "+numberOfThreads +" threads is "+(endTime-startTime)+" ms" );
}
}
private byte[] concat(byte[] A, byte[] B) {
byte[] C= new byte[A.length+B.length];
System.arraycopy(A, 0, C, 0, A.length);
System.arraycopy(B, 0, C, A.length, B.length);
return C;
}
private void convertToFile(byte[] totalBytes,String name) {
try {
FileUtils.writeByteArrayToFile(new File(name), totalBytes);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
Code of ByteArrayWrapper
package com.filedownloader;
import java.io.Serializable;
public class ByteArrayWrapper implements Serializable{
private static final long serialVersionUID = 3499562855188457886L;
private byte[] bytes;
private long sequence;
public ByteArrayWrapper(byte[] bytes, long sequenceNo) {
this.bytes = bytes;
this.sequence = sequenceNo;
}
public byte[] getBytes() {
return bytes;
}
public long getSequence() {
return sequence;
}
}
Code of FileDownloaderTask
import java.util.List;
public class FileDownloaderTask implements Runnable {
private List<ByteArrayWrapper> list;
private long start;
private long end;
private long sequenceNo;
public FileDownloaderTask(long start,long end,long sequenceNo,List<ByteArrayWrapper> list) {
this.list = list;
this.start = start;
this.end = end;
this.sequenceNo = sequenceNo;
}
#Override
public void run() {
ByteArrayWrapper wrapper = new FileSender().sendBytesOfFile(start, end, sequenceNo);
list.add(wrapper);
}
}
Questions related to this code
Does file downloading becomes fast when multiple threads is used? In this code i am not able to see the benefit.
How should i decide how many threads should i create ?
Are their any opensource libraries which does that
The file which file receiver receives is valid and not corrupted but checksum (i used FileUtils of common-io) does not match. Whats the problem?
This code gives out of memory when used with large file(above 100 Mb) i.e. because byte array which is created. How can i avoid?
I know this is a very bad code but i have to write this in one day -:). Please suggest any other good way to do this?

There's a bunch of questions here to answer. I'm not going to go through all of the code, but I can give you some tips.
First off, what some download accelerators do is indeed using the HTTP Range header to download parts of a file in parallel. Why does this work? TCP tries to allocate bandwidth fairly per connection. So if you're downloading a file from a server whose bandwidth is swamped, then you can receive a bigger share of the bandwidth by adding more connections. The same principle applies to servers that restrict outgoing bandwidth, which is usually also applied per connection (sometimes taking the IP into consideration).
Obviously if everybody was doing this, we'd be left with a whole lot of TCP connections and their overhead, and not a lot of bandwidth to do the actual downloading, which is why even these download accelerators will only use 2-4 connections. Moreover, if you are the one writing the server, you really don't need to worry about this, as you will only be slowing yourself down (by adding more overhead).
Going out of memory: don't use a bytearray, use a (buffered) InputStream (or if you have some time, learn how to use java.nio and the byte buffers) and read chunks as you are sending the file. The java tutorials cover all the basics.

1) Another reason why multiple connections may be faster is related to TCP window size.
throughput <= window size / roundtrip time
See http://en.wikipedia.org/wiki/TCP_tuning#Window_size for details.
You wont see that much difference if you run tests on a local network, because roundtrip time is small enough.
2) The only way to know for sure is to try. And the right number of threads will depend on environnment. If you need to download really big files, it might be worth it to first run a small calibration program that will try to download with different number of threads.
3) I havent looked there for a long time, but Azureus (now called Vuze) has a pretty complete API to download anything from torrent files to FTP ... And they probably have a quite efficient implementation...
Good luck !
Edit (clarification on window size) :
What you are trying to do is maximize throughput (download files faster). There is not much you can do about roundtime trip, it depends on the network. What you can do is increase window size. The window size is automagically adjusted (there is plenty of documentation on this, but I'm too lazy to google it) to best fit the current state of the network. Basically a larger window means better throughput as long as there isnt congestion or packet loss.
In the best case, you will get a window size of 64Kbits, at this point, unless you use some tricks (Jumbo frame / window scaling) which are not cupported by all routers on the internet, you get stuck at a maximum throughput of :
throughput >= 64Kbit / roundtrip time
As you cant get a bigger window, you have to open multiple windows to get around this limitation.
Notes :
As aioobe said, UDP isnt subject to the same limitations, this is one of the reason why it is more efficient.
A very efficient and scalable protocol to distribute large files is Bittorrent. As long as you dont need authentication / authorization of the downloads, it might work for you. And if you do need authorization, you can always encrypt the files ...

1 Does file downloading becomes fast when multiple threads is used? In this code i am not able to see the benefit.
No. I would be very surprised if that was the case. The CPU would never have a problem of keeping up with the feeding the network-buffer.
2 How should i decide how many threads should i create ?
In my opinion, 0 extra threads.
4 The file which file receiver receives is valid and not corrupted
but checksum (i used FileUtils of
common-io) does not match. Whats the
problem?
Make sure you don't accidentally rely on strings and specific encodings.
5 This code gives out of memory when
used with large file(above 100 Mb)
i.e. because byte array which is
created. How can i avoid?
The obvious solution would be to read smaller chunks of the file. Have a look at the read method of DataInputStream
http://java.sun.com/j2se/1.4.2/docs/api/java/io/DataInputStream.html#read%28byte[],%20int,%20int%29
And, finally, some general pointers in the matter: Instead of using multiple threads for this kind of thing, I strongly encourage you to have a look at the java.nio package, specifically java.nio.channels and the Selector class.
EDIT:
If you're really keen on getting it super-efficient, and have very large files, you could benefit from using UDP, and handle packet order and acknowledgements yourself. TCP does for instance guarantee that the packets received come in the same order as the packets sent. This is not something that you rely heavily on (since you could easily encode the "byte-offset" for each datagram yourself) and thus don't need to "pay" for.

Don't read huge file chunks into memory. No wonder you're running out. Just seek to the required position in the file and start copying via a sensibly sized buffer:
int count;
byte[] buffer = new byte[8192];
// or whatever takes your fancy, but sizes > the socket send buffer size are pointless
while ((count = in.read(buffer)) > 0)
out.write(buffer, 0, count);
out.close();
in.close();
Same logic can be used at both ends - when writing the file at the receiver, use a RandomAccessFile and seek to the appropriate offset before starting this loop.
However as other respondents have noted, the client's requirement is really pretty pointless. It doesn't buy anything much except expense and risk. I would just stream the file via a single connection.
What you should do is set a large socket send and receive buffers at both ends, e.g. 60k. The default is 8k on Windows which is uselessly low.