Develop Reference

Why does this Java code not write to file?

There are so many Input/Output Classes in Java.
It is really a mess. You do not know which to use.
Which functions does operating system offer ? There will be one
to read one byte of a file or many bytes of a file I guess.
So for example if I use this.
String path = "C:\\Users\\myName\\test.txt";
FileOutputStream fos = new FileOutputStream(path);
fos.write(333);
If I open it with a text editor it shows me letter "G" . Already I do not understand this.
And this code does not write anything, the file is empty weirdly.
String path = "C:\\Users\\myName\\test.txt";
FileOutputStream fos = new FileOutputStream(path);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(fos));
out.write("something");
All these I/O classes just confuse me. What does buffered mean. It reads 1000 Bytes at once. So
there is operating function to straight away read 1000 Bytes of a file I guess.

You need to close the instances of BufferedWriter out and FileOutputStream fos, after invoking the out.write("something"), then only the file gets created successfully with the contents you are trying to write.
public static void main(String[] args) throws IOException {
String path = "C:\\Users\\myName\\test.txt";
FileOutputStream fos = new FileOutputStream(path);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(fos));
out.write("something");
out.close(); // Closes the stream, flushing it first.
fos.close(); // Closes this file output stream and releases any system resources associated with this stream.
}

Closing the instances of BufferedWriter and FileOutputStream will solve the issue.
fos.write(333) => The number has been written to the file and when you open the file it opens in ASCII format. You can use below code.
public static void main(String[] args) throws IOException {
FileWriter fw=new FileWriter("D:\\test.txt");
fw.write("Hello! This is a sample text");
System.out.println("Writing successful");
fw.close();
/* your code
String path = "D:\\test1.txt";
FileOutputStream fos = new FileOutputStream(path);
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(fos));
out.write("Hello! This is a sample text");
out.close();
fos.close();
*/
}

There are so many Input/Output Classes in Java. It is really a mess. You do not know which to use.
The Files class is by far the easiest to use. For instance,
Files.writeString(Paths.get("test.txt"), "hello world!");
creates a text file named "test.txt" containing the text "hello world!".
The other classes are only needed if you want to do something fancy (for instance, deal with files too big to fit in main memory). For instance, suppose you wanted to read a huge log file (hundreds of gigabytes long) and wanted to write each line containing a particular word to another file. If you were to open the file with
Files.readAllLines(Paths.get("huge.log"));
you'd receive an OutOfMemoryError because the file doesn't fit in main memory. To work around that, we must read the file piece-wise, and that is what all those Reader and Writer classes (or InputStream and OutputStream, if you're dealing with binary files) are good for:
try (
var reader = Files.newBufferedReader(Paths.get("huge.log"));
var writer = Files.newBufferedWriter(Paths.get("interesting.log"));
) {
String line;
while ((line = reader.readLine()) != null) {
if (line.contains(searchWord)) {
writer.write(line);
writer.write('\n');
}
}
}
As you can see, their use is quite a bit more complicated. For one, we must close the Reader and Writer once we are done with them, which is easiest accomplished with the try with resources statement shown above.
Closing is necessary because most operating systems limit the number of files that can be open at once. Closing also gives any Buffered* classes the opportunity to empty their buffers, ensuring that any data still in buffers is passed on to the file system.
If we fail to close, as you did in your example code, the file remains open until our program exits, upon which time any data in the buffers is lost, resulting in the incomplete file you found.

How to use a regular expression to parse a text file and write the result on another file in Java

I used a regular expression to parse a text file to use the resulted group one and two as follows:
write group two in another file
make its name to be group one
Unfortunately, No data is written on the file!
I did not figure out where is the problem, here is my code:
package javaapplication5;
import java.io.*;
import java.util.regex.*;
public class JavaApplication5 {
public static void main(String[] args) {
// TODO code application logic here
try {
FileInputStream fstream = new FileInputStream("C:/Users/Welcome/Desktop/End-End-Delay.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
File newFile1= new File("C:/Users/Welcome/Desktop/AUV1.txt");
FileOutputStream fos1= new FileOutputStream(newFile1);
BufferedWriter bw1= new BufferedWriter(new OutputStreamWriter(fos1));
String strLine;
while ((strLine = br.readLine()) != null) {
Pattern p = Pattern.compile("sender\\sid:\\s(\\d+).*?End-End\\sDelay:(\\d+(?:\\.\\d+)?)");
Matcher m = p.matcher(strLine);
while (m.find()) {
String b = m.group(1);
String c = m.group(2);
int i = Integer.valueOf(b);
if(i==0){
System.out.println(b);
bw1.write(c);
bw1.newLine();
}
System.out.println(b);
// System.out.println(c);
}
}
}
catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
}
}
Can anyone here help me to solve this problem and Identify it?

You are using BufferedWriter, and never flush (flushing writer pushes the contents on disk) your writer or even close it at the end of your program.
Due to which, before your content gets written in actual file on disk from BufferedWriter, the program exits and the contents get lost.
To avoid this, either you can call flush just after writing contents in bw1,
bw1.write(c);
bw1.newLine();
bw1.flush();
OR
Before your program ends, you should call,
bw1.close(); // this ensures all content in buffered writer gets push to disk before jvm exists
Calling flush every time you write the data is not really recommended, as it defeats the purpose of buffered writing.
So best is to close the buffered writer object. You can do it in two ways,
Try-with-resources
Manually close the buffered writer object in the end, likely in the finally block so as to ensure it gets called.
Besides all this, you need to ensure that your regex matches and your condition,
if(i==0){
gets executed else code that is writing data in file won't get executed and of course in that case no write will happen in file.
Also, it is strongly recommended to close any of the resources you open like file resources, database (Connection, Statements, ResultSets) resources etc.
Hope that helps.

bufferedwriter truncate the string [duplicate]

This question already has answers here:
How to set the buffer size on a BufferedWriter over a FileWriter
(4 answers)
Closed 7 years ago.
I write a String to a file, the string is very long, about 100K, here is my code:
public static void main(String[] args) throws IOException {
String s = "kkkk";
BufferedWriter bw = new BufferedWriter(new FileWriter("/Users/liaoliuqing/Downloads/1.txt",true),1024);
bw.write(s);
bw.flush();
bw.close();
}
when the string is short, it works well. but when the string is very long, the string will be truncate, and only some character written to the file.
what is the problem?
here is my code, and it works in many threads.
private void writeToFile(String filterName, String messageToWrite)
throws IOException {
if(!messageToWrite.contains("2015")){
LOG.warn(messageToWrite);
}
bufferWritter = getBufferWriter(filterName, messageToWrite);
bufferWritter.write(messageToWrite);
if(!messageToWrite.contains("2015")){
LOG.warn(messageToWrite);
}
bufferWritter.flush();
}
I pretty sure messageToWrite is the full string, since I log it. but just the latter part of the string is written to the file.
It works in about 10 threads.
I find the problem. when one thread is writting the file, then it is the time for another thread to run, the content of first thread will confuse with 2rd thread.
I try to incread the buffer size of the bufferwriter by
returnWriter = new BufferedWriter(new FileWriter(fileName, true),64*1024);
but it is not work. it still write just 8192(the default buffer size) to the file.
how to solve this?

You need to ensure that only a single thread can use the buffered writer at a time. A simple solution is to make the writeToFile method synchronized. Otherwise you can use some sort of locking mechanism. You can either use Object.wait/notify or use a semaphore from java.util.concurrent package.

Java - GC a large string

I have a method to read and parse an extremely long xml file.
The xml file is read into a string, which then is parsed by a different class. However, this causes the Java to use a large amount of memory (~500 MB).
Normally, the program runs at around 30 MB, but when parse() is called, it increases to 500 MB. When parse() is done running, however, the memory usage doesn't go back down to 30 MB; instead it stays at 500 MB.
I've tried setting s = null and calling System.gc() but the memory usage still stays at 500 MB.
public void parse(){
try {
System.out.println("parsing data...");
String path = dir + "/data.xml";
InputStream i = new FileInputStream(path);
BufferedReader reader = new BufferedReader(new InputStreamReader(i));
String line;
String s = "";
while ((line = reader.readLine()) != null){
s += line + "\n";
}
... parse ...
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Any ideas?
Thanks.

Solution for your memory leak
You should Close the BufferReader at the end in order to close the stream and releases any system resources associated with it. You can close both InputStream and BufferReader. However, closing the BufferReader actually closes its stream as well.
Generally it's better to add a finally and close it.
finally
{
i.Close();
reader.Close();
}
Better approach try-with-resources Statement
try (BufferedReader br = new BufferedReader(new FileReader(path)))
{
return br.readLine();
}
Bonus Note
Use a StringBuilder instead of concatenating strings
String does not allow appending. Each append/concatenate on a String creates a new object and returns it. This is because String is immutable - it cannot change its internal state.
On the other hand StringBuilder is mutable. When you call Append, it alters the internal char array, rather than creating a new string object.
Thus it is more memory efficient to use a StringBuilder when you want to append many strings.

Just a note: a try-with-resources block will help you a lot with IO objects like those readers.
try(InputStream i = new FileInputStream(path);
BufferedReader reader = new BufferedReader(new InputStreamReader(i))) {
//your reading here
}
This will make sure these objects are disposed of by calling close() on them, regardless of how your method block exits (success, exception...). Closing these objects may also help to free up some memory.
The thing that's probably causing a big slowdown and probably blowup of memory usage, though, is your string concatenation. Calling s += line + "\n" is fine for a single concatenation, but the + operator actually has to create a new String instance each time, and copy the characters from the ones being concatenated. The StringBuilder class was designed just for this purpose. :)

The 500MB is caused by parsing, so it has nothing to do with the string, or the BufferedReader either. It is the DOM of the parsed XML. Release that and your memory usage will revert.
But why read the entire file into a string? This is a waste of time and space. Just parse the input directly from the file.

You should keep in mind that calling System.gc(); will not definitely do the Garbage collection but it suggest GC to do it's thing and it can ignore doing that if GC dont want to garbage collect. it is better to use StringBuilder do reduce the number of Strings you create in memory because it only creates String when you call toString() on it.

How to append/write huge data file text in Java

I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?
I read these records in blocks of 40k. So first I read 40k then another 40k and so on.
After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file.
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
FileWriter writer = new FileWriter(file.getAbsoluteFile(), append);
PrintWriter out = new PrintWriter(writer);
try {
out.print(sb);
out.flush();
writer.flush();
} finally {
writer.close();
out.close();
}
}
I read this other example but it is equally slow: Fastest way to write huge data in text file Java
I also tried it with NIO api:
private static void write(StringBuilder sb, Boolean append)) throws Exception {
FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel();
ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8"));
rwChannel.write(bb);
rwChannel.close();
}
Which is the best method to write/append huge data into file?

You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) {
writer.append(sb);
}
}
On my system I encountered a small performance improvement using a Channel rather than an OutputStream:
private static void write0a(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
try(Writer writer = Channels.newWriter(new FileOutputStream(
file.getAbsoluteFile(), append).getChannel(), "UTF-8")) {
writer.append(sb);
}
}
However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.

If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.
This is the best scenario:
1) Before you start process on the data create FileInputStream
FileOutputStream fos = new FileOutputStream("/path/of/your/file");
2) Create and OutputStreamWriter from this file
OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");
3) Create BufferedWriter (Improve file writing performance)
BufferedWriter bw = new BufferedWriter(w);
4) Pass bw to your process function and then flush/close
bw.flush();
bw.close();
The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.
In this way, it doesn't matter how large data is it

You are using a FileWriter (or a FileOutputStream in the second example). These are not buffered! So they write single chars resp. bytes to the disk.
That means, you should wrap the FileWriter in a BufferedWriter (or the FileOutputSystem in a BufferedOutputSystem).
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
Writer writer = new BufferedWriter(new FileWriter(file.getAbsoluteFile(), append));
PrintWriter out = new PrintWriter(writer);
try {
out.print(sb);
out.flush();
writer.flush();
} finally {
writer.close();
out.close();
}
}

You are opening the file, writing one line, then closing it. It's the opening and closing that takes the time here. Find a way to keep the output file open.

Did you try Apache IO, is the performance still the same?