Develop Reference

Remove stop words from file - going over it multiple times causes content duplication and does not remove the words

I am trying to go over a bunch of files, read each of them, and remove all stopwords from a specified list with such words. The result is a disaster - the content of the whole file copied over and over again.
What I tried:
- Saving the file as String and trying to look with regex
- Saving the file as String and going over line by line and comparing tokens to the stopwords that are stored in a LinkedHashSet, I can also store them in a file
- tried to twist the logic below in multiple ways, getting more and more ridiculous output.
- tried looking into text / line with the .contains() method, but no luck
My general logic is as follows:
for every word in the stopwords set:
while(file has more lines):
save current line into String
while (current line has more tokens):
assign current token into String
compare token with current stopword:
if(token equals stopword):
write in the output file "" + " "
else: write in the output file the token as is
Tried what's in this question and many other SO questions, but just can't achieve what I need.
Real code below:
private static void removeStopWords(File fileIn) throws IOException {
File stopWordsTXT = new File("stopwords.txt");
System.out.println("[Removing StopWords...] FILE: " + fileIn.getName() + "\n");
// create file reader and go over it to save the stopwords into the Set data structure
BufferedReader readerSW = new BufferedReader(new FileReader(stopWordsTXT));
Set<String> stopWords = new LinkedHashSet<String>();
for (String line; (line = readerSW.readLine()) != null; readerSW.readLine()) {
// trim() eliminates leading and trailing spaces
stopWords.add(line.trim());
}
File outp = new File(fileIn.getPath().substring(0, fileIn.getPath().lastIndexOf('.')) + "_NoStopWords.txt");
FileWriter fOut = new FileWriter(outp);
Scanner readerTxt = new Scanner(new FileInputStream(fileIn), "UTF-8");
while(readerTxt.hasNextLine()) {
String line = readerTxt.nextLine();
System.out.println(line);
Scanner lineReader = new Scanner(line);
for (String curSW : stopWords) {
while(lineReader.hasNext()) {
String token = lineReader.next();
if(token.equals(curSW)) {
System.out.println("---> Removing SW: " + curSW);
fOut.write("" + " ");
} else {
fOut.write(token + " ");
}
}
}
fOut.write("\n");
}
fOut.close();
}
What happens most often is that it looks for the first word from the stopWords set and that's it. The output contains all the other words even if I manage to remove the first one. And the first will be there in the next appended output in the end.
Part of my stopword list
about
above
after
again
against
all
am
and
any
are
as
at
With tokens I mean words, i.e. getting every word from the line and comparing it to the current stopword

After awhile of debugging I believe I have found the solution. This problem is very tricky as you have to use several different scanners and file readers etc. Here is what I did:
I changed how you added to your StopWords set, as it wasn't adding them correctly. I used a buffered reader to read each line, then a scanner to read each word, then added it to the set.
Then when you compared them I got rid of one of your loops as you can easily use the .contains() method to check if the word was a stopWord.
I left you to do the part of writing to the file to take out the stop words, as I'm sure you can figure that out now that everything else is working.
-My sample stop words txt file:
Stop words
Words
-My samples input file was the exact same, so it should catch all three words.
The code:
// create file reader and go over it to save the stopwords into the Set data structure
BufferedReader readerSW = new BufferedReader(new FileReader("stopWords.txt"));
Set<String> stopWords = new LinkedHashSet<String>();
String stopWordsLine = readerSW.readLine();
while (stopWordsLine != null) {
// trim() eliminates leading and trailing spaces
Scanner words = new Scanner(stopWordsLine);
String word = words.next();
while(word != null) {
stopWords.add(word.trim()); //Add the stop words to the set
if(words.hasNext()) {
word = words.next(); //If theres another line, read it
}
else {
break; //else break the inner while loop
}
}
stopWordsLine = readerSW.readLine();
}
BufferedReader outp = new BufferedReader(new FileReader("Words.txt"));
String line = outp.readLine();
while(line != null) {
Scanner lineReader = new Scanner(line);
String line2 = lineReader.next();
while(line2 != null) {
if(stopWords.contains(line2)) {
System.out.println("removing " + line2);
}
if(lineReader.hasNext()) { //If theres another line, read it
line2 = lineReader.next();
}
else {
break; //else break the first while loop
}
}
lineReader.close();
line = outp.readLine();
}
OutPut:
removing Stop
removing words
removing Words
Let me know if I can elaborate any more on my code or why I did something!

How to check if reading the last line AND if at last line, how to add a new line at the end of file in JAVA?

I am relatively new to programming, especially in Java, so bear that in mind when answering.
I'm programming a simple collectible card game deck building program, but file reading/writing proved to be problematic.
Here is the code for "addDeck" method that I'm trying to get working:
/**
* Adds a deckid and a deckname to decks.dat file.
*/
public static void AddDeck() throws IOException {
// Opens the decks.dat file.
File file = new File("./files/decks.dat");
BufferedReader read = null;
BufferedWriter write = null;
try {
read = new BufferedReader(new FileReader(file));
write = new BufferedWriter(new FileWriter(file));
String line = read.readLine();
String nextLine = read.readLine();
String s = null; // What will be written to the end of the file as a new line.
String newDeck = "Deck ";
int newInd = 00; // Counter index to indicate the new deckid number.
// If there are already existing deckids in the file,
// this will be the biggest existing deckid number + 1.
// If the first line (i.e. the whole file) is initially empty,
// the following line will be created: "01|Deck 01", where the
// number before the '|' sign is deckid, and the rest is the deckname.
if (line == null) {
s = "01" + '|' + newDeck + "01";
write.write(s);
}
// If the first line of the file isn't empty, the following happens:
else {
// A loop to find the last line and the biggest existing deckid of the file.
while (line != null) {
// The following if clause should determine whether or not the next
// line is the last line of the file.
if ((nextLine = read.readLine()) == null) {
// Now the reader should be at the last line of the file.
for (int i = 0; Character.isDigit(line.charAt(i)); i++) {
// Checks the deckid number of the last line and stores it.
s += line.charAt(i);
}
// The value of the last existing deckid +1 will be stored to newInd.
// Also, the divider sign '|' and the new deckname will be added.
// e.g. If the last existing deckid of decks.dat file is "12",
// the new line to be added would read "13|Deck 13".
newInd = (Integer.parseInt(s)) + 1;
s += '|' + newDeck + newInd;
write.newLine();
write.write(s);
}
else {
// If the current line isn't the last line of the file:
line = nextLine;
nextLine = read.readLine();
}
}
}
} finally {
read.close();
write.close();
}
}
The addDeck method should make the decks.dat file longer by one line each time when invoked. But no matter how many times I invoke this method, the
decks.dat has only one line that reads "01|Deck 01".
Also, I need to make a method removeDeck, which removes one whole line from the decks.dat file, and I'm even more at a loss there.
I would be so very grateful for any help!

For starters, this line will create a new file called decks.dat each time the program runs. That is, it will overwrite the contents of the file always.
File file = new File("./files/decks.dat");
As a result, if (line == null) { computes to true always and you end up with "01|Deck 01" in the file always.
To solve this problem, remove the above line and just open the BufferedReader like so:
read = new BufferedReader(new FileReader("./files/decks.dat"));
The second problem is, you cannot really open the same file to read and write at the same time, so you should not open up write like you did. I suggest you collect the updated version into a variable (I suggest StringBuilder) and finally write the contents of this variable into the decks.dat file.
Once you work on these issues, you should be able to make progress with what you intend to do.

How to read a string of characters from a text file until a certain character, print them in the console, then continue?

i have a question. I have a text file with some names and numbers arranged like this :
Cheese;10;12
Borat;99;55
I want to read the chars and integers from the file until the ";" symbol, println them, then continue, read the next one, println etc. Like this :
Cheese -> println , 10-> println, 99 -> println , and on to the next line and continue.
I tried using :
BufferedReader flux_in = new BufferedReader (
new InputStreamReader (
new FileInputStream ("D:\\test.txt")));
while ((line = flux_in.readLine())!=null &&
line.contains(terminator)==true)
{
text = line;
System.out.println(String.valueOf(text));
}
But it reads the entire line, doesn`t stop at the ";" symbol. Setting the 'contains' condition to false does not read the line at all.
EDIT : Partially solved, i managed to write this code :
StringBuilder sb = new StringBuilder();
// while ((line = flux_in.readLine())!=null)
int c;
String terminator_char = ";";
while((c = flux_in.read()) != -1) {
{
char character = (char) c;
if (String.valueOf(character).contains(terminator_char)==false)
{
// System.out.println(String.valueOf(character) + " : Char");
sb.append(character);
}
else
{
continue;
}
}
}
System.out.println(String.valueOf(sb) );
Which returns a new string formed out of the characters from the read one, but without the ";". Still need a way to make it stop on the first ";", println the string and continue.

This simple code does the trick, thanks to Stefan Vasilica for the ideea :
Scanner scan = new Scanner(new File("D:\\testfile.txt"));
// Printing the delimiter used
scan.useDelimiter(";");
System.out.println("Delimiter:" + scan.delimiter());
// Printing the tokenized Strings
while (scan.hasNext()) {
System.out.println(scan.next());
}
// closing the scanner stream
scan.close();

Read the characters from file 1 by 1
Delete the 'contains' condition
Use a stringBuilder() to build yourself the strings 1 by 1
Each stringBuilder stops when facing a ';' (say you use an if clause)
I didn't test it because I'm on my phone. Hope this helps

Find String in text file, delete line and ones below

I have a bit of code to find a string in a text file, print the line the string is on and then print the 5 lines below it. However, I need to modify it so that instead of printing, it deletes/removes the line after the string is found. How would I go about doing this?
File file = new File("./output.txt");
Scanner in = null;
try {
in = new Scanner(file);
while (in.hasNext()) {
String line = in.nextLine();
if (line.contains("(1)")) {
for (int a = 0; in.hasNextLine() && a < 6; a++) {
System.out.println(line);
line = in.nextLine();
}
}
}
} catch (Exception e) {
}

Find a small snippet you can start with.
Assuming your question.txt has the following input.
line 1
line 2
line 3 (1)
line 4
line 5
line 6
line 7
line 8
line 9
line 10
This snippet will print all lines and skip the line line 3 (1) as well the five lines after.
List<String> lines = Files.readAllLines(Paths.get("question.txt"), Charset.defaultCharset());
for (int i = 0; i < lines.size(); i++) {
if (lines.get(i).contains("(1)")) {
i = i + 6;
}
System.out.println(lines.get(i));
}
output
line 1
line 2
line 9
line 10
To store the lines into the file is left for you.

My Suggestion is you first declare and initialise a StringBuilder say output before your above code like:
StringBuilder output = new StringBuilder();
Now after the close of the if statement before the closing of the while loop append the line to the output and add a "\n" at the end like this:
output.append(line+"\n");
Now finally after your code that you have posted create a FileWriter say writer and then use the writer to write the output as shown below:
try(FileWriter writer = new FileWriter(file, false)){
writer.write(output);
}catch IOException(e){
e.printStackTrace();
}
Also don't forget to remove or comment out the following line if you do not want them printed in the output.
System.out.println(line);

SubOtimal has a good, concise answer that will work for most cases. The following is more complex but avoids loading the whole file into memory. That probably isn't an issue for you but just in case...
public void deleteAfter(File file, String searchString, int lineCountToDelete) {
// Create a temporary file to write to
File temp = new File(file.getAbsolutePath() + ".tmp");
try (BufferedReader reader = new BufferedReader(new FileReader(file));
PrintWriter writer = new PrintWriter(new FileWriter(temp)) ) {
// Read up to the line we are searching for
// and write each to the temp file
String line;
while((line = reader.readLine()) != null && !line.equals(searchString)){
writer.println(line);
}
// Skip over the number of lines we want to "delete"
// as well as watching out for hitting the end of the file
for(int i=0;i < lineCountToDelete && line != null; i++){
line = reader.readLine();
}
// Write the remaining lines to the temp file.
while((line = reader.readLine()) != null){
writer.println(line);
}
} catch (IOException e) {
throw new IllegalStateException("Unable to delete the lines",e);
}
// Delete the original file
if(!file.delete()){
throw new IllegalStateException("Unable to delete file: " + file.getAbsolutePath());
}
// Rename the temp file to the original name
if(!temp.renameTo(file)){
throw new IllegalStateException("Unable to rename " +
temp.getAbsolutePath() + " to " + file.getAbsolutePath());
}
}
I tested this with multiple conditions, including a line that doesn't exist, a line at the end and a line with fewer lines left than the number to skip. All worked and gave the appropriate results.

How can I recognize a special delimiter string when reading from a file of strings?

I want to read strings from a file. When a certain string (><) is found, I want to start reading integers instead, and convert them to binary strings.
My program is reading the strings in and saving them in an ArrayList successfully, but
it does not recognise the >< symbol and therefore the reading of the binary strings is not successful.
The Code
try {
FileInputStream fstream = new FileInputStream(fc.getSelectedFile().getPath());
// Get the object of DataInputStream
DataInputStream ino = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(ino));
String ln;
String str, next;
int line, c =0;
while ((ln = br.readLine()) != null) {
character = ln;
System.out.println(character);
iname.add(ln); // arraylist that holds the strings
if (iname.get(c).equals("><")) {
break; // break and moves
// on with the following while loop to start reading binary strings instead.
}
c++;
}
String s = "";
// System.out.println("SEQUENCE of bytes");
while ((line = ino.read()) != -1) {
String temp = Integer.toString(line, 2);
arrayl.add(temp);
System.out.println("telise? oxii");
System.out.println(line);
}
ino.close();
} catch (Exception exc) { }
The file I'm trying to read is for example:
T
E
a
v
X
L
A
.
x
"><"
sequence of bytes.
Where the last part is saved as bytes and in the textfile appears like that. no worries this bit works. all the strings are saved in a new line.

< is two characters and iname.get(c) is only one character.
What u should do is test if ln equals > and then another test if the next character equals < . If both test pass then break out of the loop.
you will have to becarefull

Use a Scanner. It allows you to specify a delimiter, and has methods for reading input tokens as String or int.

Could you not do something like:
while ((ln = br.readLine()) != null){
character=ln;
System.out.println(character);
//
// Look for magic characters >< and stop reading if found
//
if (character.indexOf("><") >= 0) {
break;
}
iname.add(ln);
}
This would work if you didn't want to add the magic symbol to your ArrayList. Your code sample is incomplete - if you're still having trouble you'd need to post the whole class.