I have an application that needs to read only specific content from a text file. I have to read the text from 10,000 different text files arranged in a folder and have to populate the content from all those text files into a single CSV file.
My application runs fine, but it is reading up to file number 999 only. No error, but is not reading file after 999.
Any ideas?
public void calculate(String location) throws IOException{
String mylocation = location;
File rep = new File(mylocation);
File f2 = new File (mylocation + "\\" + "metricvalue.csv");
FileWriter fw = new FileWriter(f2);
BufferedWriter bw = new BufferedWriter (fw);
if(rep.exists() && rep.isDirectory()){
File name[] = rep.listFiles();
for(int j = 0; j < name.length; j++){
if(name[j].isFile()){
String filename = name[j].getPath();
String nameinfo = name[j].getName();
File f1= new File (filename);
FileReader fr = new FileReader(f1);
BufferedReader br = new BufferedReader (fr);
String line = null;
while((line = br.readLine()) != null){
if(line.contains(" | #1 #2 % Correct")){
bw.write(nameinfo + ",");
while((line=br.readLine()) != null) {
if((line.indexOf("#" ) != -1)){
String info[] = line.split("\\s+");
String str = info[2] + "," + info[3] + ",";
bw.write(str);
}
}
}
}
bw.newLine();
br.close();
}
}
}
bw.close();
}
Your platform's file system is limited to 999 open files. You may need to increase the limit or close() the FileReader explicitly:
fr.close();
How to debug:
Put a breakpoint at File name[] = rep.listFiles();
Open variables when Eclipse pauses and check that your array contains all of the file names you want. This will tell you if your problem is there or in your parsing.
You need to debug your code. Here are a couple of pointers to get you started:
File name[] = rep.listFiles();
for(int j =0;j<name.length; j++) {
if(name[j].isFile()) {
What is the size of the array? Figure it out. If there are 10000 elements in the array, that's how many iterations your loop will do, there is simply no other way. Just adding
System.out.println(name.length) will answer this question for you
If the array is shorter than 10000, that's your answer, you simply counted your files incorrectly. If it is not, then your problem must be that one of the "files" isn't really a file (and the test of the if statement fails). Add an else statement to it, and print out the name ... Or better yet, remove this if at all (in general, avoid nested conditionals encompassing the entire body of an outer structure, especially, huge ones like this, it makes your code fragile, and logic very hard to follow), and replace it with
if(!name[j].isFile()) {
System.out.println("Skipping " + name[j] + " because it is not a plain file.");
continue;
}
This will tell you which of 10000 files you are skipping. If it does not print anything, that means, that you do in fact read all 10000 files, as you expect, and the actual problem causing the symptom you are investigating, is elsewhere.
Related
I'm trying to delete a line of text from a text file without copying to a temporary file. I am trying to do this by using a Printwriter and a Scanner and having them traverse the file at the same time, the writer writing what the Scanner reads and overwriting each line with the same thing, until it gets to the line that I wish to delete. Then, I advance the Scanner but not the writer, and continue as before. Here is the code:
But first, the parameters: My file names are numbers, so this would read 1.txt or 2.txt, etc, and so f specifies the file name. I convert it to a String in the constructor for a file. Int n is the index of the line that I want to delete.
public void deleteLine(int f, int n){
try{
Scanner reader = new Scanner(new File(f+".txt"));
PrintWriter writer = new PrintWriter(new FileWriter(new File(f+".txt")),false);
for(int w=0; w<n; w++)
writer.write(reader.nextLine());
reader.nextLine();
while(reader.hasNextLine())
writer.write(reader.nextLine());
} catch(Exception e){
System.err.println("Enjoy the stack trace!");
e.printStackTrace();
}
}
It gives me strange errors. It says "NoSuchElementException" and "no line found" in the stack trace. It points to different lines; it seems that any of the nextLine() calls can do this. Is it possible to delete a line this way? If so, what am I doing wrong? If not, why? (BTW, just in case you'd want this, the text file is about 500 lines. I don't know if that counts as large or even matters, though.)
As others have pointed out, you might be better off using a temporary file, if there's a slightest risk that your program crashes mid way:
public static void removeNthLine(String f, int toRemove) throws IOException {
File tmp = File.createTempFile("tmp", "");
BufferedReader br = new BufferedReader(new FileReader(f));
BufferedWriter bw = new BufferedWriter(new FileWriter(tmp));
for (int i = 0; i < toRemove; i++)
bw.write(String.format("%s%n", br.readLine()));
br.readLine();
String l;
while (null != (l = br.readLine()))
bw.write(String.format("%s%n", l));
br.close();
bw.close();
File oldFile = new File(f);
if (oldFile.delete())
tmp.renameTo(oldFile);
}
(Beware of the sloppy treatment of encodings, new-line characters and exception handling.)
However, I don't like answering questions with "I won't tell you how, because you shouldn't do it anyway.". (In some other situation for instance, you may be working with a file that's larger than half your hard drive!) So here goes:
You need to use a RandomAccessFile instead. Using this class you can both read and write to the file using the same object:
public static void removeNthLine(String f, int toRemove) throws IOException {
RandomAccessFile raf = new RandomAccessFile(f, "rw");
// Leave the n first lines unchanged.
for (int i = 0; i < toRemove; i++)
raf.readLine();
// Shift remaining lines upwards.
long writePos = raf.getFilePointer();
raf.readLine();
long readPos = raf.getFilePointer();
byte[] buf = new byte[1024];
int n;
while (-1 != (n = raf.read(buf))) {
raf.seek(writePos);
raf.write(buf, 0, n);
readPos += n;
writePos += n;
raf.seek(readPos);
}
raf.setLength(writePos);
raf.close();
}
You cannot do it this way. FileWriter can only append to a file, rather than write in the middle of it - You need RandomAccessFile if you want to write in the middle. What you do now - you override the file the first time you write to it (and it gets empty - that's why you get the exception). You can create FileWriter with append flag set to true - but this way you would append to a file rather than write in the middle of it.
I'd really recommend to write to a new file and then rename it at the end.
#shelley: you can't do what you are trying to do and what's more, you shouldn't. You should read the file and write to a temporary file for several reasons, for one, it's possible to do it this way (as opposed to what you're trying to do) and for another, if the process gets corrupted, you could bale out without loss of the original file. Now you could update a specific location of a file using a RandomAccessFile, but this is usually done (in my experience) when you are dealing with fixed sized records rather than typical text files.
I want to be able to read files from a directory in java without the worry of getting an out of memory exception because files.listfiles() and files.list() can only hold so much. I also want to know if there is a way to store the offset of the file in the directory so I can store that number and not have to iterate through the directory again to find it, is this possible?
Right now I'm using Jaime Hablutzel's answer from another question to go through the directory but wondering if I can store the file offset in the directory to go directly to it next time.
My thought was to store the filenames in a text file then store a count to record the number of characters until each new line was encountered which is another filename then use the RandomAccessFile seek() method to go directory to that line from the saved count.
How to list 2 million files in a directory in java
Okay I solved my problem on what I wanted to do. I created an offset to each file I stored in a textfile by taking the length + 1 of each filename and adding it to the offset then writing it to a file.
public static void readOffsets()
{
try {
File file = new File("indexFile.txt");
FileReader fr = new FileReader(file);
BufferedReader br = new BufferedReader(fr);
FileWriter fw = new FileWriter("indexFile2.txt");
PrintWriter pw = new PrintWriter(fw);
String line;
int offset = 0;
pw.write(offset + "\n");
while((line = br.readLine()) != null)
{
int length = line.length();
offset += length + 1;
pw.write(offset + "\n");
}
pw.close();
br.close();
}
catch(IOException ioe)
{
ioe.printStackTrace();
}
}
Then when seeking the offset using the randomaccessfile I would get the correct filename.
public static void seekOffset()
{
try {
RandomAccessFile file = new RandomAccessFile("IndexFile.txt", "r");
file.seek(693);
System.out.println(file.readLine());
}
catch(IOException ioe)
{
}
}
I'm currently writing my project for school in which requires me to read and write to txt files. I can read them correctly but I can only write to them at the end from an appended FileWriter. I would like to be able to overwrite things in my txt files on line numbers by first deleting the data on the line and then writing in the new data. I attempted to use this method...
public void overWriteFile(String dataType, String newData) throws IOException
{
ReadFile file = new ReadFile(path);
RandomAccessFile ra = new RandomAccessFile(path, "rw");
int line = file.lineNumber(path, dataType);
ra.seek(line);
ra.writeUTF(dataType.toUpperCase() + ":" + newData);
}
but I believe that the seek method moves along in bytes rather than line numbers. Can anyone help. Thanks in advance :)
P.S. the file.lineNumber method returns the exact line that the old data was on so I already have the line number that needs to be written to.
EDIT: Soloution found! Thanks guys :) I'll post the soloution below if anyone is interested
public void overWriteFile(String dataType, String newData, Team team, int dataOrder) throws IOException
{
try
{
ReadFile fileRead = new ReadFile(path);
String data = "";
if(path == "res/metadata.txt")
{
data = fileRead.getMetaData(dataType);
}
else if(path == "res/squads.txt")
{
data = fileRead.getSquadData(dataType, dataOrder);
}
else if(path == "res/users.txt")
{
data = fileRead.getUsernameData(dataType, dataOrder);
}
else if(path == ("res/playerdata/" + team.teamname + ".txt"))
{
//data = fileRead.getPlayerData(dataType, team.teamname, dataOrder);
}
BufferedReader file = new BufferedReader(new FileReader(path));
String line;
String input = "";
while((line = file.readLine()) != null)
{
input += line + '\n';
}
input = input.replace(dataType.toUpperCase() + ":" + data, dataType.toUpperCase() + ":" + newData);
FileOutputStream out = new FileOutputStream(path);
out.write(input.getBytes());
}
catch(Exception e)
{
System.out.println("Error overwriting file: " + path);
e.printStackTrace();
}
}
A quick and dirty solution would be to use the Files.readAllLines and Files.write methods to read all lines, change the one you want to change, and overwrite the whole file:
List<String> lines = Files.readAllLines(file.toPath());
lines.set(line, dataType.toUpperCase() + ":" + newData);
Files.write(file.toPath(), lines); // You can add a charset and other options too
Of course, that's not a good idea if it's a very big file. See this answer for some ideas on how to copy the file line by line in that case.
Regardless of how you do it, though, if you are changing the byte length of the line, you will need to rewrite the whole file (AFAIK). RandomAcessFile allows you to move around the file and overwrite data, but not to insert new bytes or removes existing ones, so the length of the file (in bytes) will stay the same.
Here is a link to a question just like this with a great answer:
I want to open a text file and edit a specific line in java
Basically, you can't just edit that line, unless it'll be the exact same length.
Instead, you'll want to copy over every line, and then when you reach the line number of the line you want to change, instead of copying over the old line, just put in your new line.
The link I gave you has a great example on how to do this.
I hope this helps...if not, let me know, and I'll elaborate further on the post. Good luck :)
I know this is very basic stuff but for some reason I'm having problems with a bufferedReader/ Writer. I am trying to get the first line of text and return it to another method. However, for some reason the writer doesn't seem to be writing to the temp file and it isn't changing the name of the temp file either.
By throwing a few print statements I have been able to figure out:
The while loop is operating correctly
The if else statement is operating correctly
The tempFile is not writing to a text file correctly
The tempFile is not renaming correctly
There are no errors being thrown
private static String wavFinder() throws IOException{
String currentWav=null;
int x = 1;
File inputFile = new File("C:\\convoLists/unTranscribed.txt");
File tempFile = new File("C:\\convoLists/unTranscribedtemp.txt");
BufferedReader reader = new BufferedReader(new FileReader(inputFile));
BufferedWriter writer = new BufferedWriter(new FileWriter(tempFile));
String currentLine = null;
while((currentLine = reader.readLine()) != null) {
if(x == 1){
currentWav = currentLine;
}
else{
writer.write(currentLine);
}
x = 2;
}
boolean successful = tempFile.renameTo(inputFile);
System.out.println("Success: " + successful);
System.out.println("currentWav = " + currentWav);
return currentWav;
}
Here is the method I am using. If you notice anything please let me know and if you have any questions I will be sure to answer them quickly. Thank you :)
First flush the steam(writer) and close them.
You can not have two files with same name. You are trying to rename the temp file with input file. You need to delete input file and then rename it to that.
reader.close();
writer.flush();
writer.close();
inputFile.delete();
Add these lines before rename and it will work
Close your buffers before trying to call renameTo.
reader.close()
writer.close()
File inputFile = new File("C:\convoLists/unTranscribed.txt");
File tempFile = new File("C:\convoLists/unTranscribedtemp.txt");
Why you have different signs for path?
Always should be //.
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
I guess this comes down to reading and writing to the same file. I would like to be able to return the same text file as is input, but with all integer values quadrupled. Should I even be attempting this with Java, or is it better to write to a new file and overwrite the original .txt file?
In essence, I'm trying to transform This:
12
fish
55 10 yellow 3
into this:
48
fish
220 40 yellow 12
Here's what I've got so far. Currently, it doesn't modify the .txt file.
import java.io.*;
import java.util.Scanner;
public class CharacterStretcher
{
public static void main(String[] args)
{
Scanner keyboard = new Scanner( System.in );
System.out.println("Copy and paste the path of the file to fix");
// get which file you want to read and write
File file = new File(keyboard.next());
File file2 = new File("temp.txt");
BufferedReader reader;
BufferedWriter writer;
try {
// new a writer and point the writer to the file
FileInputStream fstream = new FileInputStream(file);
// Use DataInputStream to read binary NOT text.
reader = new BufferedReader(new InputStreamReader(fstream));
writer = new BufferedWriter(new FileWriter(file2, true));
String line = "";
String temp = "";
int var = 0;
int start = 0;
System.out.println("000");
while ((line = reader.readLine()) != null)
{
System.out.println("a");
if(line.contains("="))
{
System.out.println("b");
var = 0;
temp = line.substring(line.indexOf('='));
for(int x = 0; x < temp.length(); x++)
{
System.out.println(temp.charAt(x));
if(temp.charAt(x)>47 && temp.charAt(x)<58) //if 0<=char<=9
{
if(start==0)
start = x;
var*=10;
var+=temp.indexOf(x)-48; //converts back into single digit
}
else
{
if(start!=0)
{
temp = temp.substring(0, start) + var*4 + temp.substring(x);
//writer.write(line.substring(0, line.indexOf('=')) + temp);
//TODO: Currently writes a bunch of garbage to the end of the file, how to write in the middle?
//move x if var*4 has an extra digit
if((var<10 && var>2)
|| (var<100 && var>24)
|| (var<1000 && var>249)
|| (var<10000 && var>2499))
x++;
}
//start = 0;
}
System.out.println(temp + " " + start);
}
if(start==0)
writer.write(line);
else
writer.write(temp);
}
}
System.out.println("end");
// writer the content to the file
//writer.write("I write something to a file.");
// always remember to close the writer
writer.close();
//writer = null;
file2.renameTo(file); //TODO: Not sure if this works...
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Given that this is a pretty quick and simple hack of a formatted text file, I don't think you need to be too clever about it.
Your logic for deciding whether you are looking at a number is pretty complex and I'd say it's overkill.
I've written up a basic outline of what I'd do in this instance.
It's not very clever or impressive, but should get the job done I think.
I've left out the overwriting and reading the input form the console so you get to do some of the implementation yourself ;-)
import java.io.*;
public class CharacterStretcher {
public static void main(String[] args) {
//Assumes the input is at c:\data.txt
File inputFile = new File("c:\\data.txt");
//Assumes the output is at c:\temp.txt
File outputFile = new File("c:\\temp.txt");
try {
//Construct a file reader and writer
final FileInputStream fstream = new FileInputStream(inputFile);
final BufferedReader reader = new BufferedReader(new InputStreamReader(fstream));
final BufferedWriter writer = new BufferedWriter(new FileWriter(outputFile, false));
//Read the file line by line...
String line;
while ((line = reader.readLine()) != null) {
//Create a StringBuilder to build our modified lines that will
//go into the output file
StringBuilder newLine = new StringBuilder();
//Split each line from the input file by spaces
String[] parts = line.split(" ");
//For each part of the input line, check if it's a number
for (String part : parts) {
try {
//If we can parse the part as an integer, we assume
//it's a number because it almost certainly is!
int number = Integer.parseInt(part);
//We add this to out new line, but multiply it by 4
newLine.append(String.valueOf(number * 4));
} catch (NumberFormatException nfEx) {
//If we couldn't parse it as an integer, we just add it
//to the new line - it's going to be a String.
newLine.append(part);
}
//Add a space between each part on the new line
newLine.append(" ");
}
//Write the new line to the output file remembering to chop the
//trailing space off the end, and remembering to add the line
//breaks
writer.append(newLine.toString().substring(0, newLine.toString().length() - 1) + "\r\n");
writer.flush();
}
//Close the file handles.
reader.close();
writer.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
You may want to consider one of these:
Build the new file in memory, rather than trying to write to the same file you are reading from. You could use StringBuilder for this.
Write to a new file, then overwrite the old file with the new one. This SO Question may help you there.
With both of these, you will be able to see your whole output, separate from the input file.
Additionally, with option (2), you don't have the risk of the operation failing in the middle and giving you a messed up file.
Now, you certainly can modify the file in-place. But it seems like unnecessary complexity for your case, unless you have really huge input files.
At the very least, if you try it this way first, you can narrow down on why the more complicated version is failing.
You cannot read and simultaneously write to the same file, because this would modify the text you currently read. This means, you must first write a modified new file and later rename it to the original one. You probably need to remove the original file before renameing.
For renaming, you can use File.renameTo or see one of the many SO's questions
You seem to parse integers in your code by collecting single digits and adding them up. You should consider using either a Scanner.nextInt or employ Integer.parseInt.
You can read your file line by line, split the words at white space and then parse them and check if it is either an integer or some other word.