This question already has answers here:
Writing a file using multiple threads
(3 answers)
Closed 7 years ago.
I am trying to write some content to a file, through multiple threads in Java. Each thread reads a different input file, does some computation and writes some (different) content to the common output file. The problem is that in the end, the output file only contains the content written by the last terminating thread and not the content from the other threads. Relevant code for the threads -
public void run()
{
try
{
File file = new File("/home/output.txt");
if (!file.exists())
{
file.createNewFile();
}
FileWriter fw = new FileWriter(file.getAbsoluteFile());
BufferedWriter bw = new BufferedWriter(fw);
BufferedReader br = new BufferedReader(new FileReader(inputfile)); // each thread reads a different input file
String line="";
while((line=br.readLine())!=null)
{
String id = line.trim(); // fetch id
StringBuffer sb = processId(userId); // process id
synchronized(this){
bw.write(sb.toString() + "\n"); // write to file
}
}
bw.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
How do i make all threads write their content to the common file ?
Use the constructor for FileWriter that uses append mode
FileWriter fw = new FileWriter(file.getAbsoluteFile(), true);
Have a single file writer thread which keeps reading from blocking queue and writing to a file. All other 20 threads will just put the data in blocking queue. This will prevent contention between 20 writer threads.
Few points related to your code :
1. The way you are creating the FileWriter is not correct . If you want to append data to the file, use the constructor that contains an additional boolean argument ( Make it true ):
public FileWriter(File file,boolean append) throws IOException
For example :
FileWriter fw = new FileWriter(file.getAbsoluteFile(),true);
2. You are talking about multiple threads that will share a common file , but I couldn't see any synchronized block in your code . Use synchronization to ensure that only one thread can access the shared resource at a time.
FileWriter fw = new FileWriter(file.getAbsoluteFile(),true);
should be used which indicates append mode.
I'm a newbie. SO I MIGHT BE WRONG! But your code MAY or MAY NOT work. Is that the run of a Runnable or thread? Note this:
MyClass implements Runnable {
public void run() {
....synchronized(this)..
}
}
Thread t1 = new Thread(new MyClass());
Thread t2 = new Thread(new MyClass());
Many of us do it like that(I did it, do it:), and yey! Each thread will obtain a lock on different obj, and you can end up with unexpected results, unless the OS uses some mechanism to sync writes to the same file (I don't know). If it does, then your synchonized is useless(useless anyway in my example), if it doesn't you're pretty.... Also as a note, there may be another process using this output.txt of which we have no idea...complicated stuff, at least for me
My opinion is that synchronized is useful if you have one instance of the class which SHARES, MUTABLE, STATE. (Let's forget about synchronized and static for now)
Does MyClass have state? No. Is it shared? NO, not in my example. Will it work? Not like this. Probably like this?
MyClass mc = new MyClass()
Thread t1 = new Thread(mc);
Thread t2 = new Thread(mc);
PS: I forgot to call start on every thread.
Related
I used a regular expression to parse a text file to use the resulted group one and two as follows:
write group two in another file
make its name to be group one
Unfortunately, No data is written on the file!
I did not figure out where is the problem, here is my code:
package javaapplication5;
import java.io.*;
import java.util.regex.*;
public class JavaApplication5 {
public static void main(String[] args) {
// TODO code application logic here
try {
FileInputStream fstream = new FileInputStream("C:/Users/Welcome/Desktop/End-End-Delay.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
File newFile1= new File("C:/Users/Welcome/Desktop/AUV1.txt");
FileOutputStream fos1= new FileOutputStream(newFile1);
BufferedWriter bw1= new BufferedWriter(new OutputStreamWriter(fos1));
String strLine;
while ((strLine = br.readLine()) != null) {
Pattern p = Pattern.compile("sender\\sid:\\s(\\d+).*?End-End\\sDelay:(\\d+(?:\\.\\d+)?)");
Matcher m = p.matcher(strLine);
while (m.find()) {
String b = m.group(1);
String c = m.group(2);
int i = Integer.valueOf(b);
if(i==0){
System.out.println(b);
bw1.write(c);
bw1.newLine();
}
System.out.println(b);
// System.out.println(c);
}
}
}
catch (Exception e) {
System.err.println("Error: " + e.getMessage());
}
}
}
Can anyone here help me to solve this problem and Identify it?
You are using BufferedWriter, and never flush (flushing writer pushes the contents on disk) your writer or even close it at the end of your program.
Due to which, before your content gets written in actual file on disk from BufferedWriter, the program exits and the contents get lost.
To avoid this, either you can call flush just after writing contents in bw1,
bw1.write(c);
bw1.newLine();
bw1.flush();
OR
Before your program ends, you should call,
bw1.close(); // this ensures all content in buffered writer gets push to disk before jvm exists
Calling flush every time you write the data is not really recommended, as it defeats the purpose of buffered writing.
So best is to close the buffered writer object. You can do it in two ways,
Try-with-resources
Manually close the buffered writer object in the end, likely in the finally block so as to ensure it gets called.
Besides all this, you need to ensure that your regex matches and your condition,
if(i==0){
gets executed else code that is writing data in file won't get executed and of course in that case no write will happen in file.
Also, it is strongly recommended to close any of the resources you open like file resources, database (Connection, Statements, ResultSets) resources etc.
Hope that helps.
This question already has answers here:
How to set the buffer size on a BufferedWriter over a FileWriter
(4 answers)
Closed 7 years ago.
I write a String to a file, the string is very long, about 100K, here is my code:
public static void main(String[] args) throws IOException {
String s = "kkkk";
BufferedWriter bw = new BufferedWriter(new FileWriter("/Users/liaoliuqing/Downloads/1.txt",true),1024);
bw.write(s);
bw.flush();
bw.close();
}
when the string is short, it works well. but when the string is very long, the string will be truncate, and only some character written to the file.
what is the problem?
here is my code, and it works in many threads.
private void writeToFile(String filterName, String messageToWrite)
throws IOException {
if(!messageToWrite.contains("2015")){
LOG.warn(messageToWrite);
}
bufferWritter = getBufferWriter(filterName, messageToWrite);
bufferWritter.write(messageToWrite);
if(!messageToWrite.contains("2015")){
LOG.warn(messageToWrite);
}
bufferWritter.flush();
}
I pretty sure messageToWrite is the full string, since I log it. but just the latter part of the string is written to the file.
It works in about 10 threads.
I find the problem. when one thread is writting the file, then it is the time for another thread to run, the content of first thread will confuse with 2rd thread.
I try to incread the buffer size of the bufferwriter by
returnWriter = new BufferedWriter(new FileWriter(fileName, true),64*1024);
but it is not work. it still write just 8192(the default buffer size) to the file.
how to solve this?
You need to ensure that only a single thread can use the buffered writer at a time. A simple solution is to make the writeToFile method synchronized. Otherwise you can use some sort of locking mechanism. You can either use Object.wait/notify or use a semaphore from java.util.concurrent package.
I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?
I read these records in blocks of 40k. So first I read 40k then another 40k and so on.
After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file.
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
FileWriter writer = new FileWriter(file.getAbsoluteFile(), append);
PrintWriter out = new PrintWriter(writer);
try {
out.print(sb);
out.flush();
writer.flush();
} finally {
writer.close();
out.close();
}
}
I read this other example but it is equally slow: Fastest way to write huge data in text file Java
I also tried it with NIO api:
private static void write(StringBuilder sb, Boolean append)) throws Exception {
FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel();
ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8"));
rwChannel.write(bb);
rwChannel.close();
}
Which is the best method to write/append huge data into file?
You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) {
writer.append(sb);
}
}
On my system I encountered a small performance improvement using a Channel rather than an OutputStream:
private static void write0a(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
try(Writer writer = Channels.newWriter(new FileOutputStream(
file.getAbsoluteFile(), append).getChannel(), "UTF-8")) {
writer.append(sb);
}
}
However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.
If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.
This is the best scenario:
1) Before you start process on the data create FileInputStream
FileOutputStream fos = new FileOutputStream("/path/of/your/file");
2) Create and OutputStreamWriter from this file
OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");
3) Create BufferedWriter (Improve file writing performance)
BufferedWriter bw = new BufferedWriter(w);
4) Pass bw to your process function and then flush/close
bw.flush();
bw.close();
The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.
In this way, it doesn't matter how large data is it
You are using a FileWriter (or a FileOutputStream in the second example). These are not buffered! So they write single chars resp. bytes to the disk.
That means, you should wrap the FileWriter in a BufferedWriter (or the FileOutputSystem in a BufferedOutputSystem).
private static void write(StringBuilder sb, Boolean append) throws Exception {
File file = File.createTempFile("foo", ".txt");
Writer writer = new BufferedWriter(new FileWriter(file.getAbsoluteFile(), append));
PrintWriter out = new PrintWriter(writer);
try {
out.print(sb);
out.flush();
writer.flush();
} finally {
writer.close();
out.close();
}
}
You are opening the file, writing one line, then closing it. It's the opening and closing that takes the time here. Find a way to keep the output file open.
Did you try Apache IO, is the performance still the same?
So I have the following code where I should read a Text File (This is just the Main Class):
import gui.MenuWindow;
import java.io.IOException;
import javax.swing.JOptionPane;
public class Assessor {
public static void main(String args[]) throws IOException {
FileLoader file = new FileLoader("Example.txt");
try{
new MenuWindow(file.loader());
} catch(Exception exc) {
JOptionPane.showMessageDialog(null, "Error Reading File");
}
}
}
Then I'd have to load the Text into a ListBox using Swing. The thing is that I've found this new code to read a Text File:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.io.IOException;
public class ReadTextFileExample {
public static void main(String[] args) {
File file = new File("test.txt");
StringBuffer contents = new StringBuffer();
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(file));
String text = null;
// repeat until all lines is read
while ((text = reader.readLine()) != null) {
contents.append(text)
.append(System.getProperty(
"line.separator"));
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (reader != null) {
reader.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
// show file contents here
System.out.println(contents.toString());
}
}
So I'd like to know what is the difference between the following two lines:
FileLoader file = new FileLoader("Example.txt"); //First Code
File file = new File("test.txt"); //Second Code
And... What's the StringBuffer and BufferedReader used to? Thanks!
So I'd like to know what is the difference between the following two lines:
FileLoader file = new FileLoader("Example.txt"); //First Code
File file = new File("test.txt"); //Second Code
The first creates a java.io.FileLoader which Andreas discusses. Since the javadoc says "The constructors of this class assume that the default character encoding and the default byte-buffer size are appropriate", it should never be used.
The second creates a java.io.File which is just a file path with some utility methods that can be used to read directory trees, delete, create, and move files, etc., or it can be used with FileInputStream and other classes to actually access the file contents.
And... What's the StringBuffer and BufferedReader used to? Thanks!
The StringBuffer is used to collect the contents of the file.
The BufferedReader is used to speed up reading of the file. Instead of reading one character at a time, the BufferedReader batches reads using an internal buffer.
This is an exemplary question about learning Java SE, especially regarding the java.io package. I was a bit puzzled in the beginning, but now I am quite sure that you want to compare the FileReader to the File class, which both belong to the same package java.io.
File in the Java SE API:"An abstract representation of file and directory pathnames."In other words, it is there to handle files and directories on the file system within Java. Since Java is an object-oriented language, they made a class for it. Files, i.e. binary and text files, share some attributes in common with directories, as there are: absolute, canonical path and simple name, etc.Of course, File is one of the base classes in the java.io package and many classes like FileReader make use of it for object construction.
FileReader:"Convenience class for reading character files."It comes with a handy constructor that takes a file name or file path as a String. Originally, it was meant to be constructed by a File instance. A Reader instance in general is practical to read text files, in contrast to InputStream, which is used to read binary files. A Reader instance in general is connected to a character set, e.g. "UTF-8" to translate byte to character streams.
Please also have a look at the excellent Java Tutorials provided by Oracle.
I hope the difference between File and FileReader becomes a little clearer. Especially note that there is no I/O, when you instantiate a File instance. To answer your question, the interconnection of the two classes would be:
File file = new File("test.txt"); // 1) Instaniate the file
Reader reader = new FileReader(file); // 2) Instantiate the Reader using the File instance
When you wrap a BufferedReader around a Reader instance, you can read the text file linewise, as:
BufferedReader bufferedReader = new BufferedReader(reader); // 3) Get a "buffered reader" to have access line by line.
StringBuffer comes in, when you want to chain a large number of String objects, since String objects is immutable and string operations like
String s1 = "Star ";
String s2 = "Wars";
s1 = s1 + s2;
are very costly, especially in loops, since at every addition a new String object (left side result) is created, with practically no size limits, apart from the reserved Java VM heap space.
Let me point out that you should better use the StringBuilder class, which is even faster, and is the unsynchronized counter-part of StringBuffer, introduced in the Java 5 release. The feature that StringBuffer is guaranteed to be synchronized among different Thread's is hardly ever used. I never came across it in my whole life as Java programmer.
import java.lang.*;
import java.io.*;
class test
{
public static void main(String[] a) throws Exception
{
int i;
String[] str = new String[]{"javac","example.java"};
String[] str1 = new String[]{"java","example"};
Runtime r = Runtime.getRuntime();
Process p = null;
Process p1 = null;
p=r.exec(str);
p1=r.exec(str1);
InputStreamReader reader = new InputStreamReader (p1.getInputStream ());
BufferedReader br = new BufferedReader(reader);
FileWriter fw = new FileWriter("this.txt",true);
char[] c = new char[1];
while ((i=br.read())!=-1)
{
c[0] = (char) i ;
fw.write(c);
c = new char[1];
}
fw.close();
}
}
this a simple program using runtime class.
is there any termination of 'process' code need to be employed?
Thanks in advance!!!
1 . There's no reason to initialize the process objects to null.
Just do:
Process p = r.exec(str);
Process p1 = r.exec(str1);
While you're at it, better variable names would help
2 . You can improve performance by reading and writing more than 1 character at a time:
3 . You may want to explicitly specify encodings, rather than using the platform default.
InputStreamReader reader = new InputStreamReader (p1.getInputStream (), inputCharsetName);
BufferedReader br = new BufferedReader(reader);
FileOutputStream fos = new FileOutputStream("this.txt", true);
Writer writer = new OutputStreamWriter(fos, outputCharsetName);
The input stream is never closed, and I recommend closing the writer in a finally block.
You should use Process.waitFor() in order to let the compiling process complete. Otherwise you will likely try to run a non-existing class file because compilation took too long.
The buffer is too small, everytime you do something like char[] c = new char[1]; // lolz array with exactly one element you should look at least twice.
Your program will not work the way as you would expect it to. ie. it will try to run the example.class before even the compilation of example.java is completed.
is there any termination of 'process' code need to be employed?
I think that depends on the processes - will they terminate on their own? (And do you want them to terminate when this program does?)
You can terminate easily with:
p.destroy();
p1.destroy();
First, you need to wait for the compilation to complete before executing the second process. Use the waitFor() method:
int compilationResult = -1;
try {
compilationresult = p.waitFor();
}
catch (InterruptedException ie) {
...
}
Then check compilationResult (a value of 0 indicates successful compilation) to determine whether to continue with the rest of the code.
Secondly, if you're finished with the p1 process after processing its output, then you should either wait for it to finish, or kill it yourself - depending on whether you expect the process to complete by itself or not.
To wait for the process to complete normally, use a similar block of code to above, by enclosing p1.waitFor() in a try/catch block.
To kill it yourself, use:
p1.destroy();
After you're sure the process has completed, you might then want to check the exit value with p1.exitValue(). Why be so careful about this? Well, you probably shouldn't trust the output you're collecting and writing to this.txt until you're sure the process executed correctly.