Develop Reference

Reading large numbers of text files in java

I have an application that needs to read only specific content from a text file. I have to read the text from 10,000 different text files arranged in a folder and have to populate the content from all those text files into a single CSV file.
My application runs fine, but it is reading up to file number 999 only. No error, but is not reading file after 999.
Any ideas?
public void calculate(String location) throws IOException{
String mylocation = location;
File rep = new File(mylocation);
File f2 = new File (mylocation + "\\" + "metricvalue.csv");
FileWriter fw = new FileWriter(f2);
BufferedWriter bw = new BufferedWriter (fw);
if(rep.exists() && rep.isDirectory()){
File name[] = rep.listFiles();
for(int j = 0; j < name.length; j++){
if(name[j].isFile()){
String filename = name[j].getPath();
String nameinfo = name[j].getName();
File f1= new File (filename);
FileReader fr = new FileReader(f1);
BufferedReader br = new BufferedReader (fr);
String line = null;
while((line = br.readLine()) != null){
if(line.contains(" | #1 #2 % Correct")){
bw.write(nameinfo + ",");
while((line=br.readLine()) != null) {
if((line.indexOf("#" ) != -1)){
String info[] = line.split("\\s+");
String str = info[2] + "," + info[3] + ",";
bw.write(str);
}
}
}
}
bw.newLine();
br.close();
}
}
}
bw.close();
}

Your platform's file system is limited to 999 open files. You may need to increase the limit or close() the FileReader explicitly:
fr.close();

How to debug:
Put a breakpoint at File name[] = rep.listFiles();
Open variables when Eclipse pauses and check that your array contains all of the file names you want. This will tell you if your problem is there or in your parsing.

You need to debug your code. Here are a couple of pointers to get you started:
File name[] = rep.listFiles();
for(int j =0;j<name.length; j++) {
if(name[j].isFile()) {
What is the size of the array? Figure it out. If there are 10000 elements in the array, that's how many iterations your loop will do, there is simply no other way. Just adding
System.out.println(name.length) will answer this question for you
If the array is shorter than 10000, that's your answer, you simply counted your files incorrectly. If it is not, then your problem must be that one of the "files" isn't really a file (and the test of the if statement fails). Add an else statement to it, and print out the name ... Or better yet, remove this if at all (in general, avoid nested conditionals encompassing the entire body of an outer structure, especially, huge ones like this, it makes your code fragile, and logic very hard to follow), and replace it with
if(!name[j].isFile()) {
System.out.println("Skipping " + name[j] + " because it is not a plain file.");
continue;
}
This will tell you which of 10000 files you are skipping. If it does not print anything, that means, that you do in fact read all 10000 files, as you expect, and the actual problem causing the symptom you are investigating, is elsewhere.

Having problems with removing first line from txt file

I know this is very basic stuff but for some reason I'm having problems with a bufferedReader/ Writer. I am trying to get the first line of text and return it to another method. However, for some reason the writer doesn't seem to be writing to the temp file and it isn't changing the name of the temp file either.
By throwing a few print statements I have been able to figure out:
The while loop is operating correctly
The if else statement is operating correctly
The tempFile is not writing to a text file correctly
The tempFile is not renaming correctly
There are no errors being thrown
private static String wavFinder() throws IOException{
String currentWav=null;
int x = 1;
File inputFile = new File("C:\\convoLists/unTranscribed.txt");
File tempFile = new File("C:\\convoLists/unTranscribedtemp.txt");
BufferedReader reader = new BufferedReader(new FileReader(inputFile));
BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile));
String currentLine = null;
while((currentLine = reader.readLine()) != null) {
if(x == 1){
currentWav = currentLine;
}
else{
writer.write(currentLine);
}
x = 2;
}
boolean successful = tempFile.renameTo(inputFile);
System.out.println("Success: " + successful);
System.out.println("currentWav = " + currentWav);
return currentWav;
}
Here is the method I am using. If you notice anything please let me know and if you have any questions I will be sure to answer them quickly. Thank you :)

First flush the steam(writer) and close them.
You can not have two files with same name. You are trying to rename the temp file with input file. You need to delete input file and then rename it to that.
reader.close();
writer.flush();
writer.close();
inputFile.delete();
Add these lines before rename and it will work

Close your buffers before trying to call renameTo.
reader.close()
writer.close()

File inputFile = new File("C:\convoLists/unTranscribed.txt");
File tempFile = new File("C:\convoLists/unTranscribedtemp.txt");
Why you have different signs for path?
Always should be //.

how to read from a huge file and write to a new file by java

What I am doing is to read one file line by line, format every line, then write to a new file. But the problem is that the file is huge, nearly 178 MB. But always getting error message: IO console updater error, java heap space. Here is my code:
public class fileFormat {
public static void main(String[] args) throws IOException{
String strLine;
FileInputStream fstream = new FileInputStream("train_final.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(fstream));
BufferedWriter writer = new BufferedWriter(new FileWriter("newOUTPUT.txt"));
while((strLine = reader.readLine()) != null){
List<String> numberBox = new ArrayList<String>();
StringTokenizer st = new StringTokenizer(strLine);
while(st.hasMoreTokens()){
numberBox.add(st.nextToken());
}
for (int i=1; i< numberBox.size(); i++){
String head = numberBox.get(0);
String tail = numberBox.get(i);
String line = head + " "+tail ;
System.out.println(line);
writer.write(line);
writer.newLine();
}
numberBox.clear();
}
reader.close();
writer.close();
}
}
How can I avoid this error message? Moreover, I have set the VM preference: -xms1024m

Remove the line
System.out.println(line);
This is a workaround the fialing console updater, which otherwise runs out of memory.

The program looks okay. I suspect the problem is that you run this inside of Eclipse, and System.out is collected by Eclipse in memory (to be displayed in that Console window).
System.out.println(line);
Try to run it outside of Eclipse, change Eclipse settings to pipe System.out somewhere, or remove the line.

This part of the code:
for (int i=1; i< numberBox.size(); i++){
String head = numberBox.get(0);
String tail = numberBox.get(i);
String line = head + " "+tail ;
System.out.println(line);
writer.write(line);
writer.newLine();
}
Can be translated to:
String head = numberBox.get(0);
for (int i=1; i< numberBox.size(); i++){
String tail = numberBox.get(i);
System.out.print(head);
System.out.print(" ");
System.out.println(tail);
writer.write(head);
writer.write(" ");
writer.write(tail);
writer.newLine();
}
This may add a little code duplication but it avoids creating a lot of objects.
Also there if you merge this for loop with the loop contructing the numberBox, you won't need numberBox structure at all.

If you read whole file the heap memory will occupy so better option in to read the file in chuck. See my below code. It will start reading from the offset given in argument and will return the end offset . You need to pass number of lines to be read.
Please remember: You can use any collection to store these read lines and clear the collection before calling this method to read next chunk.
FileInputStream fis = new FileInputStream(file);
InputStreamReader streamReader = new InputStreamReader(fis, "UTF-8");
LineNumberReader reader = new LineNumberReader(streamReader);
//call this below method recursively until the file does not reaches to the end
public int getParsedLines(LineNumberReader reader, int iLineNumber_Start, int iNumberOfLinesToBeRead) {
int iLineNumber_End = 0;
int iReadUptoLines = iLineNumber_Start + iNumberOfLinesToBeRead;
try {
reader.mark(iLineNumber_Start);
reader.setLineNumber(iLineNumber_Start);
do {
String str = reader.readLine();
if (str == null) {
break;
}
// your code
iLineNumber_End = reader.getLineNumber();
} while (iLineNumber_End != iReadUptoLines);
} catch (Exception ex) {
// exception handling
}
return iLineNumber_End;
}

Correct way of accessing a file with name

Ok I need someone to clear things up for me.
I saw a hundred different ways of accessing a file to read in it (FileReader).
I tried all of them and can't find a way to do it correctly.
If I try :
String path = Engine.class.getResource("Words.txt").toString();
or
URL url = getClass().getResource("Words.txt");
String path = url.getFile();
File myFile = new File(path);
I go directly to :
dispatchUncaughtException
I just don't know where to look anymore since no one seems to agree on the good way to do it. Also, what is that kind of exception ?There must be an easy way to do this since it is such an easy task. I just want my program to see my Words.txt file that is in the SRC folder of my project.
Full code if it helps :
public String GetWord()
{
String [] Words = new String [10];
int random = (int)(Math.random() * 10);
URL url = getClass().getResource("Words.txt");
String path = url.getFile();
File myFile = new File(path);
try
{
FileReader myReader = new FileReader(myFile);
BufferedReader textReader = new BufferedReader(myReader);
for(int i = 0; i < 10; i++)
{
Words[i] = textReader.readLine();
}
}
catch(Exception e)
{
System.out.println(e.getMessage());
}
return Words[random];
}

String path = Engine.class.getResource("Words.txt").toString();
For that to work, your file has to be in the same package as the Engine class. So, you probably want to move your file to the package where the class is at.
If you want to move the file into some other package then you need to specify the location starting from the root of the classpath. e.g. /some/other/pkg/Words.txt.
For a file which is not in the classpath, you need the full path along with the file name, to be able to read the file. The SRC folder itself is not a package and not in the classpath.
In that case, you can do something as follows:
FileInputStream fis = new FileInputStream("C:\\path\\to\\file\\Words.txt");
BufferedReader br = new BufferedReader(new InputStreamReader(fis));

If you use Java 7 I recommend using newBufferedReader. It's more efficient and easier to use than the BufferedReader. I also modified your code to match the Java Code Conventions.
Working exmaple:
public String getWord() {
String[] words = new String[10];
int random = (int) (Math.random() * 10);
Path path = Paths.get("src" + System.getProperty("file.separator")
+ "Words.txt");
try {
BufferedReader textReader = Files.newBufferedReader(path,
StandardCharsets.UTF_8);
for (int i = 0; i < 10; i++) {
words[i] = textReader.readLine();
}
} catch (Exception e) {
System.out.println(e.getMessage());
}
return words[random];
}

Java project can't find file

I have a project that finds a text file and makes it into an array of characters. However, for some reason or another it isn't finding the file. This is all the code involving opening/reading the file:
public void initialize(){
try{
File file = new File(getClass().getResource("/worlds/world1.txt").toString());
BufferedReader reader = new BufferedReader(
new InputStreamReader(
new FileInputStream(file),
Charset.forName("UTF-8")));
int c;
for(int i = 0; (c = reader.read()) != -1; i ++) {
for(int x = 0; x < 20; x++){
worlds[1][x][i] = (char) c;
c = reader.read();
}
}
}catch(IOException e){
e.printStackTrace();
}
}
When ran, it shows in the console that it is pointing to the correct file, but claims nothing exists there. I've checked, and the file is completely intact and in existence. What could be going wrong here?

You should not get a resource like that. You can use
BufferedReader reader = new BufferedReader(new InputStreamReader(
getClass().getResourceAsStream("/worlds/world1.txt")
));
Also, be careful when you package your application if you develop it inside an IDE, otherwise you'll run into common CLASSPATH troubles

File path for embedded resources is calculated from the package root folder. Assuming that src folder is the root package folder, make sure, that world1.txt file is located at src/worlds/ folder and full path is src/worlds/world1.txt
Second point, use the following code to obtain embedded file reader object:
// we do not need this line anymore
// File file = new File(getClass().getResource("/worlds/world1.txt").toString());
// use this approach
BufferedReader reader = new BufferedReader(
new InputStreamReader(
getClass().getResourceAsStream("/worlds/world1.txt"),
Charset.forName("UTF-8")));

You haven't indicated where your file lives.
getClass().getResource is used to locate a resource/file on your classpath; the resource may be packaged in your jar, for example. In this case, you can't open it as a File; see Raffaele's response.
If you want to locate the resource/file on the file system, then create the File object directly without getResource():
new File("/worlds/world1.txt")

I was using Netbeans and I was getting similar results. When I defined the file Path from the C drive and ran my code it stated: Access has been denied.
The following code ran fine, just back track your file location to the source (src) file.
//EXAMPLE FILE PATH
String filePath = "src\\solitaire\\key.data";
try {
BufferedReader lineReader = new BufferedReader(new FileReader(filePath));
String lineText = null;
while ((lineText = lineReader.readLine()) != null) {
hand.add(lineText);
System.out.println(lineText); // Test print of the lines
}
lineReader.close(); // Closes the bufferReader
System.out.print(hand); // Test print of the Array list
} catch(IOException ex) {
System.out.println(ex);
}