Related
I have to write a program which can split and merge files with various extensions. While splitting and merging it should use multiple threads. My code can do only a half of the task - if I don't use multithreading, it splits the file perfectly. If I do use multithreading, it splits the file, but saves only the first part several times.
What should I fix to make it work?
A method of Splitter.class
public void splitFile(CustomFile customFile, int dataSize) {
for (int i = 1; i <= partsNumber; i++) {
FileSplitterThread thread = new FileSplitterThread(customFile, i, dataSize);
thread.start();
}
}
Run method of my thread:
#Override
public void run() {
try {
fileInputStream = new FileInputStream(initialFile.getData());
byte[] b = new byte[dataSize];
String fileName = initialFile.getName() + "_part_" + index + "." + initialFile.getExtension();
fileOutputStream = new FileOutputStream(fileName);
int i = fileInputStream.read(b);
fileOutputStream.write(b, 0, i);
fileOutputStream.close();
fileOutputStream = null;
} catch (IOException e) {
e.printStackTrace();
}
}
The reason is you cannot achieve multi-threaded file splitting with just InputStream. And you are reading the file from the beginning always, you are getting the same bytes
For a simple file splitting mechanism, the following could be the general steps:
Get the size of the file (data size)
Chunk it into offsets for each thread to read. Example, if you have 2 threads and the data is 1000 bytes, the offsets will be 0,1000/2, where the read length is 500. the first thread will read position from 0 to 499, the next thread will start at 500 and read till 999
Get two InputStreams and position them using Channel (here is a good post, Java how to read part of file from specified position of bytes?)
Encapsulate the above info: InputStream, offset, length to read, output file name etc. and provide it to each of the threads
i create java downloader and download 900MB file from my wampserver , and it works correctly , but when i check the Ram usage , it increase a lot , i dont know why?
i used IDM to download same file form my wampserver and it didnt use a lot of Ram
when i see process and disk usage of my java downloader it take about 50MB and 5% cup but when i look to performance in Performance TAB , my RAM increase alot.
this is performance pic : http://i.stack.imgur.com/zfxNv.png
and this my code that my app create 8 thread to download this file simultaneously:
private void downloadFile() {
try {
this.response = this.con.getInputStream();
this.bis = new BufferedInputStream(this.response, 32 * 1024);
this.responseContentSize = this.con.getContentLength();
if (this.responseContentSize == (this.end_range - this.start_range) + 1) {
int MAX_BUFFER_SIZE = 32*1024;
byte buffer[] = new byte[MAX_BUFFER_SIZE];
makeTmpFile();
out = new FileOutputStream(this.tmpdir + this.tmpFilename);
bos = new BufferedOutputStream(out, 32 * 1024);
while (true) {
int r = this.bis.read(buffer, 0, this.MAX_BUFFER_SIZE);
if (r == -1)
break;
bos.write(buffer, 0, r);
downloadedBytes += r;
}
if (bos != null)
bos.close();
if (out != null)
out.close();
if (bis != null)
bis.close();
if (response != null)
response.close();
}
} catch (IOException e) {
sharedDownloadStatus.setCell(this.threadIndex, STATUS, 0);//error
log.setLog("func : downloadFile =>\n couldnt download file\n" + e.getMessage());
}
}
You allocate data buffer for each chunk. They temporarily occupy a lot of memory befor garbage collector take care of them.
Try holding only one data buffer per thread. (Using ThreadLocal, for example).
You can use the jvisualvm.exe program to look at your memory consumption and see where its being allocated. jvisualvm is located in the bin folder of your jdk installation.
Heres what I would do to change your code:
Get rid of the BufferedInput/BufferedOutput streams. These buffered streams are unnecessary in your program and are just adding in an additional memory allocation (of 32kb per buffered stream) but arent needed.
Dont use 8 threads, I've build file download application to transfer very large files across networks and I have found the optimal amount of threads is 2 or 3. You cant download the file faster than your internet connection therefore adding in extra thread doesnt do anything but waste memory and time.
Use a smaller buffer, a 32kb buffer is quite large, I would test using a smaller buffer like 1kb.
Dont confuse wammp memory/cpu usage with your Java applications memory/cpu usage. Use the 'Processes' tab of the Windows Task Manager instead of the Performance tab.
With that said I dont have your entire program and cant test and debug performance for you but these are the things I would do. Below is your code that I have modified as much as possible.
private void downloadFile() {
try {
this.response = this.con.getInputStream();
this.responseContentSize = this.con.getContentLength();
if (this.responseContentSize == (this.end_range - this.start_range) + 1) {
int MAX_BUFFER_SIZE = 1024;
byte buffer[] = new byte[MAX_BUFFER_SIZE];
makeTmpFile();
out = new FileOutputStream(this.tmpdir + this.tmpFilename);
int r = 0;
while ((r = response.read(buffer)) != -1) {
out.write(buffer, 0, r);
downloadedBytes = r;
}
}
} catch (IOException e) {
sharedDownloadStatus.setCell(this.threadIndex, STATUS, 0);//error
log.setLog("func : downloadFile =>\n couldnt download file\n" + e.getMessage());
} finally {
if (out != null)
out.close();
if (response != null)
response.close();
}
}
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 8 years ago.
Improve this question
This should show the internal and external memory speeds, but unfortunately it says it's 0.02 or 0.04 MB/s? Is this an enormous inefficiency in Android, or coding error?
findViewById(R.id.buttonStorageSpeed).setOnClickListener(new OnClickListener() {
#Override
public void onClick(View arg0) {
double SDSpeed = MBPSTest(getExternalCacheDir()); // /sdcard
double MemSpeed = MBPSTest(getCacheDir()); // /data/data
final AlertDialog dialog = new AlertDialog.Builder(thisContext)
.setTitle("Memory speed")
.setMessage( "Internal:"+String.valueOf(MemSpeed) + "\n" + "SD:"+String.valueOf(SDSpeed))
.create();
dialog.show();
}
});
/**
* Test MB/s write speed in some directory.
* Writes 4MB, deletes it after.
* #param outdir
* #return
*/
private double MBPSTest(File outDir) {
long start = System.currentTimeMillis();
try {
for (int fnum = 0; fnum < 1024; fnum++) {
File out = new File(outDir,"TESTspeed"+String.valueOf(fnum));
FileOutputStream fos = new FileOutputStream(out);
//Write 4k files
for (int i=0; i < 1024; i++) {
fos.write(65);//A
fos.write(69);//E
fos.write(73);//I
fos.write(79);//O
}
fos.flush();
fos.close();
//System.out.println("Wrote file.");
}
//Wrote 4MB
//Toast.makeText(getApplicationContext(), "Wrote External at: "+String.valueOf(4.0 / (elapsed/1000.0) )+" MB/S", Toast.LENGTH_LONG).show();
} catch (FileNotFoundException e) {
e.printStackTrace();
return 0;
} catch (IOException e) {
e.printStackTrace();
return 0;
}
long elapsed = System.currentTimeMillis() - start;
//Clean up:
for (int fnum = 0; fnum < 1024; fnum++) {
File out = new File(outDir, "TESTspeed"+String.valueOf(fnum));
out.delete();
}
// 4 MB / (seconds)
return 4.0 / (elapsed/1000.0);
}
As well as the overhead Jonathon mentioned of opening and closing the file a lot, you're also calling write(int) for every single byte.
Either use write(byte[]) with a big buffer, or use a BufferedOutputStream to wrap the FileOutputStream. There may be some buffering in FileOutputStream already, but equally there may not be. You may well find that once you've got fewer write operations (but still the same amount of data) it's much faster.
You're introducing a ton of overhead here:
for (int fnum = 0; fnum < 1024; fnum++) {
File out = new File(outDir,"TESTspeed"+String.valueOf(fnum));
FileOutputStream fos = new FileOutputStream(out);
//Write 4k files
for (int i=0; i < 1024; i++) {
fos.write(65);//A
fos.write(69);//E
fos.write(73);//I
fos.write(79);//O
}
fos.flush();
fos.close();
//System.out.println("Wrote file.");
}
//Wrote 4MB
You're closing and opening the file every 4k (1024 times per file). Instead you should open it just once outside the loop.
This is still far from being a scientific test. You're making a bunch of API calls that aren't going to show the real speed of the device. Also, you might have a bunch of filesystem re-sizing overhead going on.
A better method might be:
Open the file
Write the desired size of data
Seek to the beginning of the file
Flush
Start timer
Write desired size of data in as big of a chunk as possible
Stop timer
Cleanup
For my application, I had to write a custom "readline" method since I wanted to detect and preserve the newline endings in an ASCII text file. The Java readLine() method does not tell which newline sequence (\r, \n, \r\n) or EOF was encountered, so I cannot put the exact same newline sequence when writing to the modified file.
Here is the SSCE of my test example.
public class TestLineIO {
public static java.util.ArrayList<String> readLineArrayFromFile1(java.io.File file) {
java.util.ArrayList<String> lineArray = new java.util.ArrayList<String>();
try {
java.io.BufferedReader br = new java.io.BufferedReader(new java.io.FileReader(file));
String strLine;
while ((strLine = br.readLine()) != null) {
lineArray.add(strLine);
}
br.close();
} catch (java.io.IOException e) {
System.err.println("Could not read file");
System.err.println(e);
}
lineArray.trimToSize();
return lineArray;
}
public static boolean writeLineArrayToFile1(java.util.ArrayList<String> lineArray, java.io.File file) {
try {
java.io.BufferedWriter out = new java.io.BufferedWriter(new java.io.FileWriter(file));
int size = lineArray.size();
for (int i = 0; i < size; i++) {
out.write(lineArray.get(i));
out.newLine();
}
out.close();
} catch (java.io.IOException e) {
System.err.println("Could not write file");
System.err.println(e);
return false;
}
return true;
}
public static java.util.ArrayList<String> readLineArrayFromFile2(java.io.File file) {
java.util.ArrayList<String> lineArray = new java.util.ArrayList<String>();
try {
java.io.FileInputStream stream = new java.io.FileInputStream(file);
try {
java.nio.channels.FileChannel fc = stream.getChannel();
java.nio.MappedByteBuffer bb = fc.map(java.nio.channels.FileChannel.MapMode.READ_ONLY, 0, fc.size());
char[] fileArray = java.nio.charset.Charset.defaultCharset().decode(bb).array();
if (fileArray == null || fileArray.length == 0) {
return lineArray;
}
int length = fileArray.length;
int start = 0;
int index = 0;
while (index < length) {
if (fileArray[index] == '\n') {
lineArray.add(new String(fileArray, start, index - start + 1));
start = index + 1;
} else if (fileArray[index] == '\r') {
if (index == length - 1) { //last character in the file
lineArray.add(new String(fileArray, start, length - start));
start = length;
break;
} else {
if (fileArray[index + 1] == '\n') {
lineArray.add(new String(fileArray, start, index - start + 2));
start = index + 2;
index++;
} else {
lineArray.add(new String(fileArray, start, index - start + 1));
start = index + 1;
}
}
}
index++;
}
if (start < length) {
lineArray.add(new String(fileArray, start, length - start));
}
} finally {
stream.close();
}
} catch (java.io.IOException e) {
System.err.println("Could not read file");
System.err.println(e);
e.printStackTrace();
return lineArray;
}
lineArray.trimToSize();
return lineArray;
}
public static boolean writeLineArrayToFile2(java.util.ArrayList<String> lineArray, java.io.File file) {
try {
java.io.BufferedWriter out = new java.io.BufferedWriter(new java.io.FileWriter(file));
int size = lineArray.size();
for (int i = 0; i < size; i++) {
out.write(lineArray.get(i));
}
out.close();
} catch (java.io.IOException e) {
System.err.println("Could not write file");
System.err.println(e);
return false;
}
return true;
}
public static void main(String[] args) {
System.out.println("Begin");
String fileName = "test.txt";
long start = 0;
long stop = 0;
start = java.util.Calendar.getInstance().getTimeInMillis();
java.io.File f = new java.io.File(fileName);
java.util.ArrayList<String> javaLineArray = readLineArrayFromFile1(f);
stop = java.util.Calendar.getInstance().getTimeInMillis();
System.out.println("Total time = " + (stop - start) + " ms");
java.io.File oj = new java.io.File(fileName + "_readline.txt");
writeLineArrayToFile1(javaLineArray, oj);
start = java.util.Calendar.getInstance().getTimeInMillis();
java.util.ArrayList<String> myLineArray = readLineArrayFromFile2(f);
stop = java.util.Calendar.getInstance().getTimeInMillis();
System.out.println("Total time = " + (stop - start) + " ms");
java.io.File om = new java.io.File(fileName + "_custom.txt");
writeLineArrayToFile2(myLineArray, om);
System.out.println("End");
}
}
Version 1 uses readLine(), whereas version 2 is my version, which preserves newline characters.
On a text file with about 500K lines, version1 takes about 380 ms, whereas version2 takes 1074 ms.
How can I speed-up the performance of version2?
I checked Google guava and apache-commons libraries but cannot find a suitable replacement for "readLine()" that will tell which newline character was encountered when reading a text file.
Whenever the issue regards a program's speed, the main thing you should keep in mind is that, for any continuous process within that program, the speed is nearly always limited by one of two things: CPU (processing power) or IO (memory allocation and transfer speed).
Usually either your CPU is faster than your IO, or the contrary. Because of this, your program's speed-limit is almost always dictated by one of them, and it's usually easy to know which:
A program that does a lot of calculations but makes only a few, small operations with files, is almost certainly CPU-bound.
A program that reads a lot of data from files, or writes a lot of data to them, but is not very demanding towards processing, is almost certainly IO-bound.
Things are kinda straightforward when trying to improve an CPU-bounded program's speed. It mostly comes down to achieving the same goal or effect while making less operations.
This, on the other hand, does not make the process any easier. In fact, it's usually much harder to optimize CPU-bounded programs than to optimize IO-bounded ones, because each CPU-related operation is usually unique, and has to be revised individually.
Although generally easier once you have the experience, things are not so straightforward with IO-bound programs. There are a lot more stuff to consider when dealing with IO-bound processes.
I'll be using Hard-Disk Drives (HDDs) as the basis, since the characteristics I'll mention affect HDDs the strongest (because they are mechanical), but you should keep in mind that many of the same concepts apply, to some extent, to almost every memory-storage hardware, including Solid-State Drives (SSDs) and even RAM!
These are the main performance characteristics of most memory-storage hardware:
Access time: Also known as response time, it is the time it takes before the hardware can actually transfer data.
For mechanical hardware such as HDDs, this is mostly related to the mechanical nature of the drive, in other words, it's rotating disk and moving "heads". As such, access time of mechanical drives can vary significantly between each-other.
For circuital hardware such as SSDs and RAM, this time is not dependent on moving parts, but rather electrical connections, so the access time is very quick and consistent, and you shouldn't worry about it.
Seek time: The time it takes for the hardware to seek (reach) the correct position within it's internal subdivisions, in order to read from or write to addresses in that section.
For mechanical drives, mainly rotary ones, the seek time measures the time it takes the head assembly on the actuator arm to travel to the track of the disk where the data will be read from or written to.
Average seek time ranges from 3 ms (~) for high-end server drives, to 15 ms (~) for mobile drives, with the most common desktop drives typically having a seek time around 9 ms (~).
With RAM and SSDs, there are no moving parts, so a measurement of the seek time is only testing the electronic circuits, and preparing a particular location on the memory in the device for the operation.
Typical SSDs will have a seek time between 0.08 to 0.16 ms (~), with RAM being even faster.
Command-Processing time: Also known as command overhead, it is the time it takes for the drive's electronics to set up the necessary communication between the various internal components, so it can read or write the data.
This is in the range of 0.003 ms (~) for both, mechanical and circuital devices, and is usually ignored in benchmarks.
Settle time: It is the time it takes for the heads to settle on the target track and stop vibrating, so that they do not read or write off-track.
This amount is usually very small (typically less than 0.1 ms), and typically included in benchmarks as part of the seek time.
Data-Transfer rate: Also called throughput, it covers both: The internal rate, which is the time it takes to move data between the disk surface and the controller on the drive. And the external rate, which is the time to move data between the controller on the drive and an external component in the host system. It has a few sub-factors within:
Media rate: Speed at which the drive can read bits from the media. In other words, the actual read/write speed.
Sector overhead: Additional time (bytes) needed for control structures and other information necessary to manage the drive, locate and validate data and perform other support functions.
Allocation speed: Similar to sector overhead, it's the time taken for the drive to determine the slots that will be written to, and to register them on it's address dictionary. Only needed for write operations.
Head-Switch time: Time required to electrically switch from one head to another; Only applies to multi-head drives and is about 1 to 2 ms.
Cylinder-switch time: Time required to move to an adjacent track; The name cylinder is used because typically all the tracks of a drive with more than one head or data surface are read before moving the actuator, implying the image of a circle or cylinder rather than a track. This time is exclusive to rotary mechanical drives, and is typically about about 2 to 3 ms.
This means that the main performance issues regarding IO are caused by going back-and-forth between IO and processing. An issue that can be enormously diminished by using buffers, and processing and reading/writhing in bigger chunks of data, rather than every byte.
As you can also see, although many of the speed characteristics are still present, RAM and SSDs do not have the same internal limits of HDDs, so their internal and external transfer rates often reach the maximum capabilities of the drive-to-host interface.
Chunk approach example:
This example will create a Test folder on the desktop, and generate a Test.txt file within.
The file is generated with an specified number of lines, each line containing the word "Test" repeated for an specific number of times (for file-size purposes). Each line is ended by "\r", "\n" or "\r\n", sequentially.
It's meaningless to save the results of each chunk in-memory cumulatively, as doing so would lead the whole file end up in-memory eventually, which is nearly the same problem of not using chunks to begin with.
As such, an output file is created in the same Test folder, to which the result of every chunk is stored at, once that chunk is finished.
The base file is read using buffers, and those buffers are additionally used as the chunks.
The process here is simply printing a textual version of the line-separator ("\\r", "\\n" or "\\r\\n"), followed by ": ", followed by the line contents; But for the last line, "EOF" is used instead.
To actually operate with chunks, it's probably easier to manage with a class-based approach, rather than a purely function-based one.
Anyways, here goes the code:
public static void main(String[] args) throws FileNotFoundException, IOException {
File file = new File(TEST_FOLDER, "Test.txt");
//These settings create a 122 MB file.
generateTestFile(file, 500000, 50);
long clock = System.nanoTime();
processChunks(file, 8 * (int) Math.pow(1024, 2));
clock = System.nanoTime() - clock;
float millis = clock / 1000000f;
float seconds = millis / 1000f;
System.out.printf(""
+ "%12d nanos\n"
+ "%12.3f millis\n"
+ "%12.3f seconds\n",
clock, millis, seconds);
}
public static File prepareResultFile(File source) {
String ofn = source.getName(); //Original File Name.
int extPos = ofn.lastIndexOf('.'); //Extension index.
String ext = ofn.substring(extPos); //Get extension.
ofn = ofn.substring(0, extPos); //Get name without extension reusing 'ofn'.
return new File(source.getParentFile(), ofn + "_Result" + ext);
}
public static void processChunks(File file, int buffSize)
throws FileNotFoundException, IOException {
//No need for buffers bigger than the file itself.
if (file.length() < buffSize) {
buffSize = (int)file.length();
}
byte[] buffer = new byte[buffSize];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file), buffSize);
BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(
prepareResultFile(file)), buffSize);
StringBuilder sb = new StringBuilder();
while (bis.read(buffer) > (-1)) {
//Check if a "\r\n" was split between chunks.
boolean skipFirst = false;
if (sb.length() > 0 && sb.charAt(sb.length() - 1) == '\r') {
if (buffer[0] == '\n') {
bos.write(("\\r\\n: " + sb.toString() + System.lineSeparator()).getBytes());
sb = new StringBuilder();
skipFirst = true;
}
}
for (int i = skipFirst ? 1 : 0; i < buffer.length; i++) {
if (buffer[i] == '\r') {
if (i + 1 < buffer.length) {
if (buffer[i + 1] == '\n') {
bos.write(("\\r\\n: " + sb.toString() + System.lineSeparator()).getBytes());
i++; //Skip '\n'.
} else {
bos.write(("\\r: " + sb.toString() + System.lineSeparator()).getBytes());
}
sb = new StringBuilder(); //Reset accumulator.
} else {
//A "\r\n" might be split between two chunks.
}
} else if (buffer[i] == '\n') {
bos.write(("\\n: " + sb.toString() + System.lineSeparator()).getBytes());
sb = new StringBuilder(); //Reset accumulator.
} else {
sb.append((char) buffer[i]);
}
}
}
bos.write(("EOF: " + sb.toString()).getBytes());
bos.flush();
bos.close();
bis.close();
System.out.println("Finished!");
}
public static boolean generateTestFile(File file, int lines, int elements)
throws IOException {
String[] lineBreakers = {"\r", "\n", "\r\n"};
BufferedOutputStream bos = null;
try {
bos = new BufferedOutputStream(new FileOutputStream(file));
for (int i = 0; i < lines; i++) {
for (int ii = 1; ii < elements; ii++) {
bos.write("test ".getBytes());
}
bos.write("test".getBytes());
bos.write(lineBreakers[i % 3].getBytes());
}
bos.flush();
System.out.printf("LOG: Test file \"%s\" created.\n", file.getName());
return true;
} catch (IOException ex) {
System.err.println("ERR: Could not write file.");
throw ex;
} finally {
try {
bos.close();
} catch (IOException ex) {
System.err.println("WRN: Could not close stream.");
Logger.getLogger(Q_13458142_v2.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
I don't know what IDE you are using, but if it's NetBeans, make a memory-profile of your code and compare to a profile of this one. You should notice a big difference in the amount of memory needed during processing.
Here, the chunk approach's memory usage, which includes not only the chunk itself but also the program's own variables and structures, does not go over 40 MB even tough we are dealing with a file bigger than 100 MB. As you can see:
It also spends very little time in GB, mostly less than 5% at any given point:
The second version doesn't seem to use BufferedReader or another form of buffer. It might be the cause of slow down.
Since you seem to read the whole file in memory, you can perhaps read it as a big string (with a buffer) then parse it in memory to analyze the line endings.
Your are doubling the out statements(one for line and one for newline):
Can you try below(use lineSeparator() to get the line separator and append before writing):
out.write(lineArray.get(i)+System.lineSeparator());
Don't reinvent the wheel.
Check the BufferedReader#readLine() code
Copy, paste, and make the changes you need to keep the line separator inside the line
As the title says, I'm looking for the fastest possible way to write integer arrays to files. The arrays will vary in size, and will realistically contain anywhere between 2500 and 25 000 000 ints.
Here's the code I'm presently using:
DataOutputStream writer = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(filename)));
for (int d : data)
writer.writeInt(d);
Given that DataOutputStream has a method for writing arrays of bytes, I've tried converting the int array to a byte array like this:
private static byte[] integersToBytes(int[] values) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
for (int i = 0; i < values.length; ++i) {
dos.writeInt(values[i]);
}
return baos.toByteArray();
}
and like this:
private static byte[] integersToBytes2(int[] src) {
int srcLength = src.length;
byte[] dst = new byte[srcLength << 2];
for (int i = 0; i < srcLength; i++) {
int x = src[i];
int j = i << 2;
dst[j++] = (byte) ((x >>> 0) & 0xff);
dst[j++] = (byte) ((x >>> 8) & 0xff);
dst[j++] = (byte) ((x >>> 16) & 0xff);
dst[j++] = (byte) ((x >>> 24) & 0xff);
}
return dst;
}
Both seem to give a minor speed increase, about 5%. I've not tested them rigorously enough to confirm that.
Are there any techniques that will speed up this file write operation, or relevant guides to best practice for Java IO write performance?
I had a look at three options:
Using DataOutputStream;
Using ObjectOutputStream (for Serializable objects, which int[] is); and
Using FileChannel.
The results are
DataOutputStream wrote 1,000,000 ints in 3,159.716 ms
ObjectOutputStream wrote 1,000,000 ints in 295.602 ms
FileChannel wrote 1,000,000 ints in 110.094 ms
So the NIO version is the fastest. It also has the advantage of allowing edits, meaning you can easily change one int whereas the ObjectOutputStream would require reading the entire array, modifying it and writing it out to file.
Code follows:
private static final int NUM_INTS = 1000000;
interface IntWriter {
void write(int[] ints);
}
public static void main(String[] args) {
int[] ints = new int[NUM_INTS];
Random r = new Random();
for (int i=0; i<NUM_INTS; i++) {
ints[i] = r.nextInt();
}
time("DataOutputStream", new IntWriter() {
public void write(int[] ints) {
storeDO(ints);
}
}, ints);
time("ObjectOutputStream", new IntWriter() {
public void write(int[] ints) {
storeOO(ints);
}
}, ints);
time("FileChannel", new IntWriter() {
public void write(int[] ints) {
storeFC(ints);
}
}, ints);
}
private static void time(String name, IntWriter writer, int[] ints) {
long start = System.nanoTime();
writer.write(ints);
long end = System.nanoTime();
double ms = (end - start) / 1000000d;
System.out.printf("%s wrote %,d ints in %,.3f ms%n", name, ints.length, ms);
}
private static void storeOO(int[] ints) {
ObjectOutputStream out = null;
try {
out = new ObjectOutputStream(new FileOutputStream("object.out"));
out.writeObject(ints);
} catch (IOException e) {
throw new RuntimeException(e);
} finally {
safeClose(out);
}
}
private static void storeDO(int[] ints) {
DataOutputStream out = null;
try {
out = new DataOutputStream(new FileOutputStream("data.out"));
for (int anInt : ints) {
out.write(anInt);
}
} catch (IOException e) {
throw new RuntimeException(e);
} finally {
safeClose(out);
}
}
private static void storeFC(int[] ints) {
FileOutputStream out = null;
try {
out = new FileOutputStream("fc.out");
FileChannel file = out.getChannel();
ByteBuffer buf = file.map(FileChannel.MapMode.READ_WRITE, 0, 4 * ints.length);
for (int i : ints) {
buf.putInt(i);
}
file.close();
} catch (IOException e) {
throw new RuntimeException(e);
} finally {
safeClose(out);
}
}
private static void safeClose(OutputStream out) {
try {
if (out != null) {
out.close();
}
} catch (IOException e) {
// do nothing
}
}
I would use FileChannel from the nio package and ByteBuffer. This approach seems (on my computer) give 2 to 4 times better write performance:
Output from program:
normal time: 2555
faster time: 765
This is the program:
public class Test {
public static void main(String[] args) throws IOException {
// create a test buffer
ByteBuffer buffer = createBuffer();
long start = System.currentTimeMillis();
{
// do the first test (the normal way of writing files)
normalToFile(new File("first"), buffer.asIntBuffer());
}
long middle = System.currentTimeMillis();
{
// use the faster nio stuff
fasterToFile(new File("second"), buffer);
}
long done = System.currentTimeMillis();
// print the result
System.out.println("normal time: " + (middle - start));
System.out.println("faster time: " + (done - middle));
}
private static void fasterToFile(File file, ByteBuffer buffer)
throws IOException {
FileChannel fc = null;
try {
fc = new FileOutputStream(file).getChannel();
fc.write(buffer);
} finally {
if (fc != null)
fc.close();
buffer.rewind();
}
}
private static void normalToFile(File file, IntBuffer buffer)
throws IOException {
DataOutputStream writer = null;
try {
writer =
new DataOutputStream(new BufferedOutputStream(
new FileOutputStream(file)));
while (buffer.hasRemaining())
writer.writeInt(buffer.get());
} finally {
if (writer != null)
writer.close();
buffer.rewind();
}
}
private static ByteBuffer createBuffer() {
ByteBuffer buffer = ByteBuffer.allocate(4 * 25000000);
Random r = new Random(1);
while (buffer.hasRemaining())
buffer.putInt(r.nextInt());
buffer.rewind();
return buffer;
}
}
Benchmarks should be repeated every once in a while, shouldn't they?
:) After fixing some bugs and adding my own writing variant, here are
the results I get when running the benchmark on an ASUS ZenBook UX305
running Windows 10 (times given in seconds):
Running tests... 0 1 2
Buffered DataOutputStream 8,14 8,46 8,30
FileChannel alt2 1,55 1,18 1,12
ObjectOutputStream 9,60 10,41 11,68
FileChannel 1,49 1,20 1,21
FileChannel alt 5,49 4,58 4,66
And here are the results running on the same computer but with Arch
Linux and the order of the write methods switched:
Running tests... 0 1 2
Buffered DataOutputStream 31,16 6,29 7,26
FileChannel 1,07 0,83 0,82
FileChannel alt2 1,25 1,71 1,42
ObjectOutputStream 3,47 5,39 4,40
FileChannel alt 2,70 3,27 3,46
Each test wrote an 800mb file. The unbuffered DataOutputStream took
way to long so I excluded it from the benchmark.
As seen, writing using a file channel still beats the crap out of all
other methods, but it matters a lot whether the byte buffer is
memory-mapped or not. Without memory-mapping the file channel write
took 3-5 seconds:
var bb = ByteBuffer.allocate(4 * ints.length);
for (int i : ints)
bb.putInt(i);
bb.flip();
try (var fc = new FileOutputStream("fcalt.out").getChannel()) {
fc.write(bb);
}
With memory-mapping, the time was reduced to between 0.8 to 1.5
seconds:
try (var fc = new RandomAccessFile("fcalt2.out", "rw").getChannel()) {
var bb = fc.map(READ_WRITE, 0, 4 * ints.length);
bb.asIntBuffer().put(ints);
}
But note that the results are order-dependent. Especially so on
Linux. It appears that the memory-mapped methods doesn't write the
data in full but rather offloads the job request to the OS and returns
before it is completed. Whether that behaviour is desirable or not
depends on the situation.
Memory-mapping can also lead to OutOfMemory problems so it is not
always the right tool to
use. Prevent OutOfMemory when using java.nio.MappedByteBuffer.
Here is my version of the benchmark code:
https://gist.github.com/bjourne/53b7eabc6edea27ffb042e7816b7830b
I think you should consider using file channels (the java.nio library) instead of plain streams (java.io). A good starting point is this interesting discussion: Java NIO FileChannel versus FileOutputstream performance / usefulness
and the relevant comments below.
Cheers!
The main improvement you can have for writing int[] is to either;
increase the buffer size. The size is right for most stream, but file access can be faster with a larger buffer. This could yield a 10-20% improvement.
Use NIO and a direct buffer. This allows you to write 32-bit values without converting to bytes. This may yield a 5% improvement.
BTW: You should be able to write at least 10 million int values per second. With disk caching you increase this to 200 million per second.
Array is Serializable - can't you just use writer.writeObject(data);? That's definitely going to be faster than individual writeInt calls.
If you have other requirements on the output data format than retrieval into int[], that's a different question.