I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?
I read these records in blocks of 40k. So first I read 40k then another 40k and so on.
After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file.
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
FileWriter writer = new FileWriter(file.getAbsoluteFile(), append);
PrintWriter out = new PrintWriter(writer);
try {
out.print(sb);
out.flush();
writer.flush();
} finally {
writer.close();
out.close();
}
}
I read this other example but it is equally slow: Fastest way to write huge data in text file Java
I also tried it with NIO api:
private static void write(StringBuilder sb, Boolean append)) throws Exception {
FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel();
ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8"));
rwChannel.write(bb);
rwChannel.close();
}
Which is the best method to write/append huge data into file?
You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) {
writer.append(sb);
}
}
On my system I encountered a small performance improvement using a Channel rather than an OutputStream:
private static void write0a(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
try(Writer writer = Channels.newWriter(new FileOutputStream(
file.getAbsoluteFile(), append).getChannel(), "UTF-8")) {
writer.append(sb);
}
}
However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.
If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.
This is the best scenario:
1) Before you start process on the data create FileInputStream
FileOutputStream fos = new FileOutputStream("/path/of/your/file");
2) Create and OutputStreamWriter from this file
OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");
3) Create BufferedWriter (Improve file writing performance)
BufferedWriter bw = new BufferedWriter(w);
4) Pass bw to your process function and then flush/close
bw.flush();
bw.close();
The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.
In this way, it doesn't matter how large data is it
You are using a FileWriter (or a FileOutputStream in the second example). These are not buffered! So they write single chars resp. bytes to the disk.
That means, you should wrap the FileWriter in a BufferedWriter (or the FileOutputSystem in a BufferedOutputSystem).
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
Writer writer = new BufferedWriter(new FileWriter(file.getAbsoluteFile(), append));
PrintWriter out = new PrintWriter(writer);
try {
out.print(sb);
out.flush();
writer.flush();
} finally {
writer.close();
out.close();
}
}
You are opening the file, writing one line, then closing it. It's the opening and closing that takes the time here. Find a way to keep the output file open.
Did you try Apache IO, is the performance still the same?
Related
There are so many Input/Output Classes in Java.
It is really a mess. You do not know which to use.
Which functions does operating system offer ? There will be one
to read one byte of a file or many bytes of a file I guess.
So for example if I use this.
String path = "C:\\Users\\myName\\test.txt";
FileOutputStream fos = new FileOutputStream(path);
fos.write(333);
If I open it with a text editor it shows me letter "G" . Already I do not understand this.
And this code does not write anything, the file is empty weirdly.
String path = "C:\\Users\\myName\\test.txt";
FileOutputStream fos = new FileOutputStream(path);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(fos));
out.write("something");
All these I/O classes just confuse me. What does buffered mean. It reads 1000 Bytes at once. So
there is operating function to straight away read 1000 Bytes of a file I guess.
You need to close the instances of BufferedWriter out and FileOutputStream fos, after invoking the out.write("something"), then only the file gets created successfully with the contents you are trying to write.
public static void main(String[] args) throws IOException {
String path = "C:\\Users\\myName\\test.txt";
FileOutputStream fos = new FileOutputStream(path);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(fos));
out.write("something");
out.close(); // Closes the stream, flushing it first.
fos.close(); // Closes this file output stream and releases any system resources associated with this stream.
}
Closing the instances of BufferedWriter and FileOutputStream will solve the issue.
fos.write(333) => The number has been written to the file and when you open the file it opens in ASCII format. You can use below code.
public static void main(String[] args) throws IOException {
FileWriter fw=new FileWriter("D:\\test.txt");
fw.write("Hello! This is a sample text");
System.out.println("Writing successful");
fw.close();
/* your code
String path = "D:\\test1.txt";
FileOutputStream fos = new FileOutputStream(path);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(fos));
out.write("Hello! This is a sample text");
out.close();
fos.close();
*/
}
There are so many Input/Output Classes in Java. It is really a mess. You do not know which to use.
The Files class is by far the easiest to use. For instance,
Files.writeString(Paths.get("test.txt"), "hello world!");
creates a text file named "test.txt" containing the text "hello world!".
The other classes are only needed if you want to do something fancy (for instance, deal with files too big to fit in main memory). For instance, suppose you wanted to read a huge log file (hundreds of gigabytes long) and wanted to write each line containing a particular word to another file. If you were to open the file with
Files.readAllLines(Paths.get("huge.log"));
you'd receive an OutOfMemoryError because the file doesn't fit in main memory. To work around that, we must read the file piece-wise, and that is what all those Reader and Writer classes (or InputStream and OutputStream, if you're dealing with binary files) are good for:
try (
var reader = Files.newBufferedReader(Paths.get("huge.log"));
var writer = Files.newBufferedWriter(Paths.get("interesting.log"));
) {
String line;
while ((line = reader.readLine()) != null) {
if (line.contains(searchWord)) {
writer.write(line);
writer.write('\n');
}
}
}
As you can see, their use is quite a bit more complicated. For one, we must close the Reader and Writer once we are done with them, which is easiest accomplished with the try with resources statement shown above.
Closing is necessary because most operating systems limit the number of files that can be open at once. Closing also gives any Buffered* classes the opportunity to empty their buffers, ensuring that any data still in buffers is passed on to the file system.
If we fail to close, as you did in your example code, the file remains open until our program exits, upon which time any data in the buffers is lost, resulting in the incomplete file you found.
My current problems lie with the fact that no matter what solution I attempt at creating a file in Java, the file never, ever is created or shows up.
I've searched StackOverflow for solutions and tried many, many different pieces of code all to no avail. I've tried using BufferedWriter, PrintWriter, FileWriter, wrapped in try and catch and thrown IOExceptions, and none of it seems to be working. For every field that requires a path, I've tried both the name of the file alone and the name of the file in a path. Nothing works.
//I've tried so much I don't know what to show. Here is what remains in my method:
FileWriter fw = new FileWriter("testFile.txt", false);
PrintWriter output = new PrintWriter(fw);
fw.write("Hello");
I don't get any errors thrown whenever I've run my past code, however, the files never actually show up. How can I fix this?
Thank you in advance!
There are several ways to do this:
Write with BufferedWriter:
public void writeWithBufferedWriter()
throws IOException {
String str = "Hello";
BufferedWriter writer = new BufferedWriter(new FileWriter(fileName));
writer.write(str);
writer.close();
}
If you want to append to a file:
public void appendUsingBufferedWritter()
throws IOException {
String str = "World";
BufferedWriter writer = new BufferedWriter(new FileWriter(fileName, true));
writer.append(' ');
writer.append(str);
writer.close();
}
Using PrintWriter:
public void usingPrintWriteru()
throws IOException {
FileWriter fileWriter = new FileWriter(fileName);
PrintWriter printWriter = new PrintWriter(fileWriter);
printWriter.print("Some String");
printWriter.printf("Product name is %s and its price is %d $", "iPhone", 1000);
printWriter.close();
}
Using FileOutputStream:
public void usingFileOutputStream()
throws IOException {
String str = "Hello";
FileOutputStream outputStream = new FileOutputStream(fileName);
byte[] strToBytes = str.getBytes();
outputStream.write(strToBytes);
outputStream.close();
}
Note:
If you try to write to a file that doesn’t exist, the file will be created first and no exception will be thrown.
It is very important to close the stream after using it, as it is not closed implicitly, to release any resources associated with it.
In output stream, the close() method calls flush() before releasing the resources which forces any buffered bytes to be written to the stream.
Source and More Examples: https://www.baeldung.com/java-write-to-file
Hope this helps. Good luck.
A couple of things worth trying:
1) In case you haven't (it's not in the code you've shown) make sure you close the file after you're done with it
2) Use a File instead of a String. This will let you double check where the file is being created
File file = new File("testFile.txt");
System.out.println("I am creating the file at '" + file.getAbsolutePath() + "');
FileWriter fw = new FileWriter(file, false);
fw.write("Hello");
fw.close();
As a bonus, Java's try-with-resource will automatically close the resource when it's done, you might want to try
File file = new File("testFile.txt");
System.out.println("I am creating the file at '" + file.getAbsolutePath() + "');
try (FileWriter fw = new FileWriter(file, false)) {
fw.write("Hello");
}
in the api,java said that,'Unless prompt output is required, it is advisable to wrap a BufferedWriter around any Writer whose write() operations may be costly, such as FileWriters and OutputStreamWriters'.I tried the comparation many times,it turns out that,it costs less time if I do not use PrintWriter to wrap BufferedWriter.
so does it mean that,I do not have to wrap on the 'console'?
there is my simple code that compare the two.
public static void PrintWrite(String path, String content) throws IOException {
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(path)));
pw.write(content);
pw.close();
}
public static void BufferedWrite(String path, String content) throws IOException {
BufferedWriter bw = new BufferedWriter(new FileWriter(path));
bw.write(content);
bw.close();
}
private static void main(String[] args)
{
String s = "";
String path = "D:\\out.txt";
for (int i = 0; i < 10000; i++)
{
s += "kjuatlistened";
}
try
{
long start = System.nanoTime();
//PrintWrite(path,s);
BufferedWrite(path,s);
long end = System.nanoTime();
Printer.println("done,elapse " + (end - start) + " s");
}
catch (IOException ex)
{
ex.printStackTrace();
}
}
You're certainly adding a layer of processing when you wrap one object in another.
The main reason for using PrintWriter is the wealth of convenience functions which do intelligent output of various things. If you don't need that capability, you wouldn't use a PrintWriter at all.
The point of a BufferedWriter is to allow output to be buffered and then written in a batch. I/O to file systems, the Internet, etc. is costly, and a large fraction of that cost is in generating the connection in the first place, and then waiting while each I/O operation completes. If you batch I/O to and from external devices, you can really reduce the overall cost because output (or input) can be batched--and you end up waiting for completion just once per batch, rather than once per I/O operation. For an "interesting" test, try writing characters to your FileWriter one at a time, then do the same thing with a BufferedWriter wrapping your FileWriter. I think in the latter case you'll see a large improvement.
I don't fully understand your underlying question, but...
Wrapping a BufferedWriter with a PrintWriter pretty much has to to be slower, since it is added a layer of calls.
System.out and .err are PrintStream objects, not PrintWriter. Is that what you mean by the console? I'm pretty sure you should not wrap either of them.
I am working on a project. For the project I am using GUI and I want to write a number to a file. I have been successful, and I can write the number to the file that i want to. My problem that hopefully someone can give insite to is that everytime i write a number to a file the new number replaces the old one. How would i go about keeping the current info in the file. My code is:
public static void writeCodeFile (String filename, int x, String userName) throws IOException{
BufferedWriter outputWriter = null;
outputWriter = new BufferedWriter(new FileWriter(filename));
outputWriter.newLine();
outputWriter.write(userName +":"+ Integer.toString(x));
outputWriter.newLine();
outputWriter.flush();
outputWriter.close();
}
Use append mode:
outputWriter = new BufferedWriter(new FileWriter(filename, true));
When you create the FileWriter, add a second parameter "true" to go into append mode.
I have a StringWriter variable, sw, which is populated by a FreeMarker template. Once I have populated the sw, how can I print it to a text file?
I have a for loop as follows:
for(2 times)
{
template.process(data, sw);
out.println(sw.toString());
}
Right now, I am just outputting to the screen only. How do I do this for a file? I imagine that with each loop, my sw will get changed, but I want the data from each loop appended together in the file.
Edit:
I tried the code below. When it runs, it does show that the file.txt has been changed, but when it reloads, the file still has nothing in it.
sw.append("CheckText");
PrintWriter out = new PrintWriter("file.txt");
out.println(sw.toString());
How about
FileWriter fw = new FileWriter("file.txt");
StringWriter sw = new StringWriter();
sw.write("some content...");
fw.write(sw.toString());
fw.close();
and also you could consider using an output stream which you can directly pass to template.process(data, os); instead of first writing to a StringWriter then to a file.
Look at the API-doc for the template.process(...) to find out if such a facility is available.
Reply 2
template.process(Object, Writer) can also take a FileWriter object, witch is a subclass of Writer, as parameter, so you probably can do something like that:
FileWriter fw = new FileWriter("file.txt");
for(2 times)
{
template.process(data, fw);
}
fw.close();
You can use many different streams to write to file.
I personally like to work with PrintWriter here
You can flag to append in the FileWriter (the true in the following example):
try {
PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("outfilename", true)));
out.println(sw.toString());
out.close();
} catch (IOException e) {
// Do something
}
Why not use a FileWriter ?
Open it before you loop and generate your required output. As you write to the FileWriter it'll append to the buffer and write out your accumulated output upon a close()
Note that you can open a FileWriter in overwrite or append mode, so you can append to existing files.
Here's a simple tutorial.
If you don't mind using Apache commons IO :
FileUtils.write(new File("file.txt"), sw.toString(), /*append:*/ true);