I want to remove stop word after tokenizing string. I have external file .txt and read it then compare it to the tokenized string. if the tokenized word is equal with the stop word, then remove it.
here is the code for tokenizing
try{
while ((msg =readBufferData.readLine()) != null) {
int numberOfTokens;
System.out.println("Before: "+msg);
StringTokenizer tokens = new StringTokenizer(msg);
numberOfTokens = tokens.countTokens();
System.out.println("Tokens: "+numberOfTokens);
System.out.print("After : ");
while (tokens.hasMoreTokens()) {
msg = tokens.nextToken();
String msgLower = msg.toLowerCase();
String punctuationremove = punctuationRemover(msgLower);
// buffWriter.write(punctuationremove+" "); --> write into file .txt
System.out.print(punctuationremove+" ");
removingStopWord(punctuationremove, readStopWordsFile());
numberOfTotalTokens++;
}
// buffWriter.newLine(); make a new line after tokening new message
System.out.println("\n");
numberOfMessages++;
}
// write close buffWriter.close();
System.out.println("Total Tokens: "+numberOfTotalTokens);
System.out.println("Total Messages: "+numberOfMessages);
}
catch (Exception e){
System.out.println("Error Exception: "+e.getMessage());
}
Then I have a code for reading the stop word file
public static Set<String> readStopWordsFile() throws FileNotFoundException, IOException{
String fileStopWords = "\\stopWords.txt";
Set<String> stopWords = new LinkedHashSet<String>();
FileReader readFileStopWord = new FileReader(fileStopWords);
BufferedReader stopWordsFile = new BufferedReader(readFileStopWord);
String line;
while((line = stopWordsFile.readLine())!=null){
line = line.trim();
stopWords.add(line);
}
stopWordsFile.close();
return stopWords;
}
How can I compare the token with the set of stop word and delete the token that same with the stop word. Can you help me, thank you
You can simply read the stop words first and then check whether your token is a stopword.
Set<String> stopWords = readStopWordsFile();
// some file reading logic
while (tokens.hasMoreTokens()) {
msg = tokens.nextToken();
if(stopWords.contains(msg)){
continue; // skip over a stopword token
}
}
Related
In my code I have two files in my drive those two files have some text and I want to display those string in the console and also remove the repeated string and display the repeated string once rather than displaying it twice.
Code:
public class read {
public static void main(String[] args) {
try{
File file = new File("D:\\file1.txt");
FileReader fileReader = new FileReader(file);
BufferedReader br = new BufferedReader(fileReader);
StringBuffer stringBuffer = new StringBuffer();
String line;
while((line = br.readLine()) != null){
stringBuffer.append(line);
stringBuffer.append("\n");
}
fileReader.close();
System.out.println("Contents of file1:");
String first = stringBuffer.toString();
System.out.println(first);
File file1 = new File("D:\\file2.txt");
FileReader fileReader1 = new FileReader(file1);
BufferedReader br1 = new BufferedReader(fileReader1);
StringBuffer stringBuffer1 = new StringBuffer();
String line1;
while((line1 = br1.readLine()) != null){
stringBuffer1.append(line1);
stringBuffer1.append("\n");
}
fileReader1.close();
System.out.println("Contents of file2:");
String second = stringBuffer1.toString();
System.out.println(second);
System.out.println("answer:");
System.out.println(first+second);
}catch (IOException e) {
// TODO: handle exception
e.printStackTrace();
}
}
}
Output is:
answer:
hi hello
how are you
hi ya
i am fine
But I want to compare both the strings and if the same string repeated then that string should be displayed once.
Output I expect is like this:
answer:
hi hello
how are you
ya
i am fine
Where the "hi" is found in both the strings so that I need to delete the one duplicate string.
How can I do that please help.
Thanks in advance.
You can pass your lines through this method to parse out duplicate words:
// store unique previous words
static Set<String> words = new HashSet<>();
static String removeDuplicateWords(String line) {
StringJoiner sj = new StringJoiner(" ");
// split on whitespace to get distinct words
for (String word : line.split("\\s+")) {
// try to add word to the set
if (words.add(word)) {
// if the word was added (=not seen before), append to the result
sj.add(word);
}
}
return sj.toString();
}
I am new in java. I just wants to read each string in java and print it on console.
Code:
public static void main(String[] args) throws Exception {
File file = new File("/Users/OntologyFile.txt");
try {
FileInputStream fstream = new FileInputStream(file);
BufferedReader infile = new BufferedReader(new InputStreamReader(
fstream));
String data = new String();
while ((data = infile.readLine()) != null) { // use if for reading just 1 line
System.out.println(""+data);
}
} catch (IOException e) {
// Error
}
}
If file contains:
Add label abc to xyz
Add instance cdd to pqr
I want to read each word from file and print it to a new line, e.g.
Add
label
abc
...
And afterwards, I want to extract the index of a specific string, for instance get the index of abc.
Can anyone please help me?
It sounds like you want to be able to do two things:
Print all words inside the file
Search the index of a specific word
In that case, I would suggest scanning all lines, splitting by any whitespace character (space, tab, etc.) and storing in a collection so you can later on search for it. Not the question is - can you have repeats and in that case which index would you like to print? The first? The last? All of them?
Assuming words are unique, you can simply do:
public static void main(String[] args) throws Exception {
File file = new File("/Users/OntologyFile.txt");
ArrayList<String> words = new ArrayList<String>();
try {
FileInputStream fstream = new FileInputStream(file);
BufferedReader infile = new BufferedReader(new InputStreamReader(
fstream));
String data = null;
while ((data = infile.readLine()) != null) {
for (String word : data.split("\\s+") {
words.add(word);
System.out.println(word);
}
}
} catch (IOException e) {
// Error
}
// search for the index of abc:
for (int i = 0; i < words.size(); i++) {
if (words.get(i).equals("abc")) {
System.out.println("abc index is " + i);
break;
}
}
}
If you don't break, it'll print every index of abc (if words are not unique). You could of course optimize it more if the set of words is very large, but for a small amount of data, this should suffice.
Of course, if you know in advance which words' indices you'd like to print, you could forego the extra data structure (the ArrayList) and simply print that as you scan the file, unless you want the printings (of words and specific indices) to be separate in output.
Split the String received for any whitespace with the regex \\s+ and print out the resultant data with a for loop.
public static void main(String[] args) { // Don't make main throw an exception
File file = new File("/Users/OntologyFile.txt");
try {
FileInputStream fstream = new FileInputStream(file);
BufferedReader infile = new BufferedReader(new InputStreamReader(fstream));
String data;
while ((data = infile.readLine()) != null) {
String[] words = data.split("\\s+"); // Split on whitespace
for (String word : words) { // Iterate through info
System.out.println(word); // Print it
}
}
} catch (IOException e) {
// Probably best to actually have this on there
System.err.println("Error found.");
e.printStackTrace();
}
}
Just add a for-each loop before printing the output :-
while ((data = infile.readLine()) != null) { // use if for reading just 1 line
for(String temp : data.split(" "))
System.out.println(temp); // no need to concatenate the empty string.
}
This will automatically print the individual strings, obtained from each String line read from the file, in a new line.
And afterwards, I want to extract the index of a specific string, for
instance get the index of abc.
I don't know what index are you actually talking about. But, if you want to take the index from the individual lines being read, then add a temporary variable with count initialised to 0.
Increment it till d equals abc here. Like,
int count = 0;
for(String temp : data.split(" ")){
count++;
if("abc".equals(temp))
System.out.println("Index of abc is : "+count);
System.out.println(temp);
}
Use Split() Function available in Class String.. You may manipulate according to your need.
or
use length keyword to iterate throughout the complete line
and if any non- alphabet character get the substring()and write it to the new line.
List<String> words = new ArrayList<String>();
while ((data = infile.readLine()) != null) {
for(String d : data.split(" ")) {
System.out.println(""+d);
}
words.addAll(Arrays.asList(data));
}
//words List will hold all the words. Do words.indexOf("abc") to get index
if(words.indexOf("abc") < 0) {
System.out.println("word not present");
} else {
System.out.println("word present at index " + words.indexOf("abc"))
}
I need to assign the elements present in a CSV file into an arraylist. CSV file contains filenames with extension .tar. I need to trim those elements before i read it into an array list or trim the whole arraylist. Please help me with it
try
{
String strFile1 = "D:\\Ramakanth\\PT2573\\target.csv"; //csv file containing data
BufferedReader br1 = new BufferedReader( new FileReader(strFile1)); //create BufferedReader
String strLine1 = "";
StringTokenizer st1 = null;
while( (strLine1 = br1.readLine()) != null) //read comma separated file line by line
{
st1 = new StringTokenizer(strLine1, ","); //break comma separated line using ","
while(st1.hasMoreTokens())
{
array1.add(st1.nextToken()); //store csv values in array
}
}
}
catch(Exception e)
{
System.out.println("Exception while reading csv file: " + e);
}
If you want to remove the ".tar" string from your tokens, you can use:
String nextToken = st1.nextToken();
if (nextToken.endsWith(".tar")) {
nextToken = nextToken.replace(".tar", "");
}
array1.add(nextToken);
You shouldn't be using StringTokenizer the JavaDoc says (in part) StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead. You should close your BufferedReader. You could use a try-with-resources statement to do that. And, you might use a for-each loop to iterate the array produced by String.split(String) the regular expression below optionally matches whitespace before or after your , and you might continue the loop if the token endsWith ".tar" like
String strFile1 = "D:\\Ramakanth\\PT2573\\target.csv";
try (BufferedReader br1 = new BufferedReader(new FileReader(strFile1)))
{
String strLine1 = "";
while( (strLine1 = br1.readLine()) != null) {
String[] parts = strLine1.split("\\s*,\\s*");
for (String token : parts) {
if (token.endsWith(".tar")) continue; // <-- don't add "tar" files.
array1.add(token);
}
}
}
catch(Exception e)
{
System.out.println("Exception while reading csv file: " + e);
}
if(str.indexOf(".tar") >0)
str = str.subString(0, str.indexOf(".tar")-1);
while(st1.hasMoreTokens())
{
String input = st1.nextToken();
int index = input.indexOf("."); // Get the position of '.'
if(index >= 0){ // To avoid StringIndexOutOfBoundsException, when there is no match with '.' then the index position set to -1.
array1.add(input.substring(0, index)); // Get the String before '.' position.
}
}
What is the best way to split a string containing three words?
My code looks like this right now (see below for updated code):
BufferedReader infile = new BufferedReader(new FileReader("file.txt"));
String line;
int i = 0;
while ((line = infile.readLine()) != null) {
String first, second, last;
//Split line into first, second and last (word)
//Do something with words (no help needed)
i++;
}
Here is the full file.txt:
Allegrettho Albert 0111-27543
Brio Britta 0113-45771
Cresendo Crister 0111-27440
Dacapo Dan 0111-90519
Dolce Dolly 0116-31418
Espressivo Eskil 0116-19042
Fortissimo Folke 0118-37547
Galanto Gunnel 0112-61805
Glissando Gloria 0112-43918
Grazioso Grace 0112-43509
Hysterico Hilding 0119-71296
Interludio Inga 0116-22709
Jubilato Johan 0111-47678
Kverulando Kajsa 0119-34995
Legato Lasse 0116-26995
Majestoso Maja 0116-80308
Marcato Maria 0113-25788
Molto Maja 0117-91490
Nontroppo Maistro 0119-12663
Obligato Osvald 0112-75541
Parlando Palle 0112-84460
Piano Pia 0111-10729
Portato Putte 0112-61412
Presto Pelle 0113-54895
Ritardando Rita 0117-20295
Staccato Stina 0112-12107
Subito Sune 0111-37574
Tempo Kalle 0114-95968
Unisono Uno 0113-16714
Virtuoso Vilhelm 0114-10931
Xelerando Axel 0113-89124
New code as #Pshemo suggested:
public String load() {
try {
Scanner scanner = new Scanner(new File("reg.txt"));
while (scanner.hasNextLine()) {
String firstname = scanner.next();
String lastname = scanner.next();
String number = scanner.next();
list.add(new Entry(firstname, lastname, number));
}
msg = "The file reg.txt has been opened";
return msg;
} catch (NumberFormatException ne) {
msg = ("Can't find reg.txt");
return msg;
} catch (IOException ie) {
msg = ("Can't find reg.txt");
return msg;
}
}
I receive multiple errors, what's wrong?
Assuming that each line always contains exactly three words instead of split you can simply use Scanners method next three times for each line.
Scanner scanner = new Scanner(new File("file.txt"));
int i = 0;
while (scanner.hasNextLine()) {
String first = scanner.next();
String second = scanner.next();
String last = scanner.next();
//System.out.println(first+": "+second+": "+last);
i++;
}
line.split("\\s+"); // don't use " ". use "\\s+" for more than one whitespace
Assuming the line has 3+ words, use the split(delimiter) method:
String line = ...;
String[] parts = line.split("\\s+"); // Assuming words are separated by whitespaces, use another if required
then you can access to the first, second and last respectively:
String first = parts[0];
String second = parts[1];
String last = parts[parts.length() - 1];
Remember that indexes starts with 0.
String []parts=line.split("\\s+");
System.out.println(parts[0]);
System.out.println(parts[1]);
System.out.println(parts[parts.length-1]);
My method takes a file, and tries to extract the text between the header ###Title### and closing ###---###. I need it to extract multiple lines and put each line into an array. But since readAllLines() converts all lines into an array, I don't know how to compare and match it.
public static ArrayList<String> getData(File f, String title) throws IOException {
ArrayList<String> input = (ArrayList<String>) Files.readAllLines(f.toPath(), StandardCharsets.US_ASCII);
ArrayList<String> output = new ArrayList<String>();
//String? readLines = somehow make it possible to match
System.out.println("Checking entry.");
Pattern p = Pattern.compile("###" + title + "###(.*)###---###", Pattern.DOTALL);
Matcher m = p.matcher(readLines);
if (m.matches()) {
m.matches();
String matched = m.group(1);
System.out.println("Contents: " + matched);
String[] array = matched.split("\n");
ArrayList<String> array2 = new ArrayList<String>();
for (String j:array) {
array2.add(j);
}
output = array2;
} else {
System.out.println("No matches.");
}
return output;
}
Here is my file, and I'm 100% sure that the compiler is reading the correct one.
###Test File###
Entry 1
Entry 2
Data 1
Data 2
Test 1
Test 2
###---###
The output says "No matches." instead of the entries.
You don't need regex for that. It's enough to loop through the array and compare items line by line, taking those between the start and end tags.
ArrayList<String> input = (ArrayList<String>) Files.readAllLines(f.toPath(), StandardCharsets.US_ASCII);
ArrayList<String> output = new ArrayList<String>();
boolean matched = false;
for (String line : input) {
if (line.equals("###---###") && matched) matched = false; //needed parentheses
if (matched) output.add(line);
if (line.equals("###Test File###") && !matched) matched = true;
}
As per your comment, if they are going to be in the same way as posted, then i don't think regex is needed for this requirement. You can read line by line and do a contains of '###'
public static void main(String args[])
{
ArrayList<String> dataList = new ArrayList<String>();
try{
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream("textfile.txt");
// Get the object of DataInputStream
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
//Read File Line By Line
while ((strLine = br.readLine()) != null) {
// this line will skip the header and footer with '###'
if(!strLine.contains("###");
dataList.add(strLine);
}
//Close the input stream
in.close();
}catch (Exception e){//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
}
//Now dataList has all the data between ###Test File### and ###---###
}
You can also change the contains method parameter according to your requirement to ignore lines!