I am trying to read contents of a file using string tokenizer and store all the tokens in an array but i keep getting exception in main error. I need advise on how to do this.Below is the code am using for that;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.Reader;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.StringTokenizer;
public class FileTokenizer
{
private static final String DEFAULT_DELIMITERS = "< , { } >";
private static final String DEFAULT_TEST_FILE = "trans1.txt";
public List<String> tokenize(Reader reader) throws IOException
{
List<String> tokens = new ArrayList<String>();
BufferedReader br = null;
try
{
int i = 0;
br = new BufferedReader(reader);
Scanner scanner = new Scanner(br);
while (scanner.hasNext())
{
StringTokenizer st = new StringTokenizer(scanner.next(), DEFAULT_DELIMITERS, true);
while (st.hasMoreElements())
{
String[] t = new String[200];
tokens.add(st.nextToken());
t[i] = st.nextToken();
System.out.println(t[i]);
i++;
}
}
}
finally
{
close(br);
}
return tokens;
}
public static void close(Reader r)
{
try
{
if (r != null)
{
r.close();
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
public static void main(String[] args)
{
try
{
String fileName = ((args.length > 0) ? args[0] : DEFAULT_TEST_FILE);
FileReader fileReader = new FileReader(new File(fileName));
FileTokenizer fileTokenizer = new FileTokenizer();
List<String> tokens = fileTokenizer.tokenize(fileReader);
//System.out.println(tokens);
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
My file looks like;
PDA = (
{ q1, q2, q3, q4},
{ 0, 1 },
{ 0, $ },
{ (q1, #, #) -> { (q2, $) }, (q2, 0, #) -> { (q2, 0) },
(q2, 1, 0) -> { (q3, #) }, (q3, 1, 0) -> { (q3, #) },
(q3, #, $) -> { (q4, #) } },
q1,
{ q1, q4}
)
You will get the java.util.NoSuchElementException since you are calling st.nextToken() twice within the loop
while (st.hasMoreElements())
Modifying harigm's example, you can then add t[i] to tokens as you require
String[] t = new String[200];
System.out.println(t[i]);
tokens.add(t[i]);
Delimiters shouldn't be separated by spaces:
private static final String DEFAULT_DELIMITERS = "<,{}>";
Also, keep the following in mind (from the Javadoc):
StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.
String.split() was introduced in JDK 1.4.
That said:
Using a Scanner to tokenize a stream together with a StringTokenizer looks a bit weird to me;
You call st.nextToken() twice in the inner loop;
t is useless. You re-create it each time in your inner loop and use only one element of it.
It seems that what you are trying to build is a lexical analyzer. Maybe you should look up some documentation on the subject.
HI,
I have modified your code and Now works perfectly fine, check this
package org.sample;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.Reader;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
import java.util.StringTokenizer;
public class FileTokenizer
{
private static final String DEFAULT_DELIMITERS = "< , { } >";
// private static final String DEFAULT_TEST_FILE = "trans1.txt";
public List<String> tokenize(Reader reader) throws IOException
{
List<String> tokens = new ArrayList<String>();
BufferedReader br = null;
try
{
int i = 0;
br = new BufferedReader(reader);
Scanner scanner = new Scanner(br);
while (scanner.hasNext())
{
StringTokenizer st = new StringTokenizer(scanner.next(), DEFAULT_DELIMITERS, true);
while (st.hasMoreElements())
{
String[] t = new String[200];
// tokens.add(st.nextToken());
// t[i] = st.nextToken();
System.out.println(t[i]);
i++;
}
}
}
finally
{
close(br);
}
return tokens;
}
public static void close(Reader r)
{
try
{
if (r != null)
{
r.close();
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
public static void main(String[] args)
{
try
{
// String fileName = ((args.length > 0) ? args[0] : DEFAULT_TEST_FILE);
FileReader fileReader = new FileReader(new File("c:\\DevTest\\1.txt"));
FileTokenizer fileTokenizer = new FileTokenizer();
List<String> tokens = fileTokenizer.tokenize(fileReader);
//System.out.println(tokens);
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
Looking at your input file, I should point out that its hierarchical and irregular structure makes it more suited to be parsed by an actual parser. You may have to learn how to use a parser generator and write a lexer and grammar for it etc, but in the end you'll end up with a much more maintainable code. Doing this yourself is rather painstaking and error-prone.
I recommend ANTLR. It's quite mature, and it has a wide enough user base that I'm sure you can get help easily.
Related
The code should do a reverse and output the result to out.txt, but this does not happen, can you explain my mistake in the code. Thanks in advance
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class Main {
public static void main(String[] args) throws IOException {
FileReader input = new FileReader("in.txt");
FileWriter output = new FileWriter("out.txt");
BufferedReader sb = new BufferedReader(input);
String data;
while ((data = sb.readLine()) != null) {
String[] words = data.split(" ");
for (String a : words) {
StringBuilder builder = new StringBuilder(a);
builder.reverse();
while ((sb.read()) != -1) {
output.write(String.valueOf(builder.reverse()));
}
}
}
}
}
You are trying to reverse the string twice because of that the string is getting back to the original string. Also, there is an unnecessary (as per my understanding) while loop inside the for loop (I have removed that in my answer).
Try the below code:
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class Main {
public static void main(String[] args) throws IOException {
FileReader input = new FileReader("in.txt");
FileWriter output = new FileWriter("out.txt");
BufferedReader sb = new BufferedReader(input);
String data;
while ((data = sb.readLine()) != null) {
String[] words = data.split(" ");
// above statement can be replaced with
// String[] words = data.split(" {34}");
for (String a : words) {
StringBuilder builder = new StringBuilder(a);
// why while loop is required?
//while ((sb.read()) != -1) {
output.write(builder.reverse().toString());
output.flush(); // flush data to the file
//}
}
}
output.close();
}
}
Read about File writer here on how to flush data and also close the writer after writing is completed.
I have a file called "marathon", where I have 7 keys:
sex
time
athlete
athlete's nationality
date
city
country
splitted by a comma ",". I have to put the second key (time) in a Treemap.
At the moment I am just trying to show only the time in the console.
So here is my code:
public class Text {
#SuppressWarnings("resource")
public static void main(String[] args) throws FileNotFoundException {
try {
BufferedReader in = new BufferedReader(new FileReader("marathon"));
String str;
str = in.readLine();
while ((str = in.readLine()) != null) {
//System.out.println(str);
String[] ar=str.split(",");
System.out.println(ar[0]);
}
in.close();
} catch (IOException e) {
System.out.println("File Read Error");
}
}
}
This is what a line of the text looks like:
M, 2:30:57.6, Harry Payne, GBR, 1929-07-05, Stamford Bridge, England
When I start the program of my code example and put in System.out.println(ar[0]); a[0] shows me the first line in the console so M's and F's. But when I put a[1] there is an exception:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
As others have pointed out, you do readline twice before you get into the body of the loop, so you will miss the first line.
But you are also not checking that readline resulted in a properly formatted line. It may be an empty line or a line that in some other way does not result in an array that you expect.
So you should add an if-statement that checks that you have what you expected, like so...
public class Text {
#SuppressWarnings("resource")
public static void main(String[] args) throws FileNotFoundException {
try {
BufferedReader in = new BufferedReader(new FileReader("marathon"));
String str = "";
while ((str = in.readLine()) != null) {
String[] ar=str.split(",");
if(ar.length >= 7) {
System.out.println(ar[0] + ", " + ar[1]);
}
}
in.close();
} catch (IOException e) {
System.out.println("File Read Error");
}
}
}
Please try the code below. It's working for me.
You should have read the line only once in the while loop.
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class Text {
#SuppressWarnings("resource")
public static void main(String[] args) throws FileNotFoundException {
try {
BufferedReader in = new BufferedReader(new FileReader("marathon"));
String str;
while ((str = in.readLine()) != null) {
//System.out.println(str);
String[] ar=str.split(",");
System.out.println(ar[0]);
System.out.println(ar[1]);
}
in.close();
} catch (IOException e) {
System.out.println("File Read Error");
}
}
}
SortedMap<String, String[]> map = new TreeMap<>();
Path path = Paths.get("marathon");
Files.lines(path, Charsets.defaultCharset())
.map(line -> line.split(",\\s*"))
.peek(words -> {
if (words.length != 7) {
Logger.getLogger(getClass().getName()).info("Line wrong: " + line);
}
})
.filter(words -> words.length == 7)
.forEach(words -> map.put(word[1], words));
However there are CSV reader classes out there, that can handle quoted fields with commas and such.
Java 8 just for fun
import java.io.IOException;
import java.nio.file.*;
import java.util.*;
import java.util.stream.Stream;
import static java.nio.charset.Charset.defaultCharset;
import static java.lang.System.out;
import static java.nio.file.Files.lines;
public class Main {
public static void main(String[] args) throws IOException {
Map<String, String[]> map = new TreeMap<>( );
try( Stream<String> lines = lines(Paths.get("marathon"), defaultCharset())){
lines.map(line -> line.split( "," )).forEach( entry -> map.put(entry[1], entry ));
map.values().forEach( entry -> out.println(Arrays.toString( entry )) );
}
}
}
This question already has answers here:
What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it?
(26 answers)
Closed 5 years ago.
I have an exercise where are given a list of 850 basic words in English in the file basicWords.txt. I need to compose a text of 10000 words by randomly selecting words from the basic words list and write it to another file. I generated successfully the words, but I have a problem: I get an exception when the words are generated: ArrayIndexOutOfBoundsException at line 35. Also, how can I print the result into another text file?
I have a final solution for this:
package randomstring;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
/**
*
* #author robi1
*/
public class RandomString {
public static void main(String[] args){
List<String> dictionary = readDictionaryFrom("basicWordsInEnglish.txt");
List<String> monkeyText = generateTextFrom(dictionary);
writeTextToFile(monkeyText, "final.txt");
String letters = "abcdefghijklmnopqrstuvwxyz ";
Object[] wrds = readFile("basicWordsInEnglish.txt");
int x = wrds.length;
String[] words = new String[x];
for(int i =0;i<x;i++){
words[i] = wrds[i].toString();
}
char[] let = letters.toCharArray();
String n ="";
Random r = new Random();
char t;
}
public static Object[] readFile(String name){
ArrayList<String> al = new ArrayList<String>();
FileInputStream fstream;
try {
fstream = new FileInputStream(name);
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
while((strLine=br.readLine())!=null){
if(strLine.length()>4)
al.add(strLine);
}
fstream.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Object[] array = al.toArray();
return array;
}
public static List<String> readDictionaryFrom(String path) {
try {
return Files.readAllLines(new File(path).toPath());
} catch(IOException e) {
throw new RuntimeException(e);
}
}
public RandomString(List<String> text, String path) {
try(BufferedWriter file = new BufferedWriter(new FileWriter(new File(path)))){
for(String word : text) {
file.write(word+" ");
}
} catch(IOException e) {
throw new RuntimeException(e);
}
}
public static List<String> generateTextFrom(List<String> words) {
Random generator = new Random();
List<String> result = new ArrayList<>();
for(int i = 0; i < 10000; i++) {
int random = generator.nextInt(words.size());
result.add(words.get(random));
}
return result;
}
public static void writeTextToFile(List<String> text, String path) {
try(BufferedWriter file = new BufferedWriter(new FileWriter(new File(path)))){
for(String word : text) {
file.write(word+" ");
}
} catch(IOException e) {
throw new RuntimeException(e);
}
}
}
Why do you not use collections? According to description this task is very easy especially when don't use bunch for, while loops and meaningless variables like n,t,j. etc.
public void main(String... args) {
List<String> dictionary = readDictionaryFrom("path to dictionary");
List<String> monkeyText = generateTextFrom(dictionary);
writeTextToFile(monkeyText, "path to destination file");
}
public List<String> readDictionaryFrom(String path) {
try {
return Files.readAllLines(new File(path).toPath());
} catch(IOException e) {
throw new RuntimeException(e);
}
}
public static void writeTextToFile(List<String> text, String path) {
try(BufferedWriter file = new BufferedWriter(new FileWriter(new File(path)))){
for(String word : text) {
file.write(word);
}
} catch(IOException e) {
throw new RuntimeException(e);
}
}
public static List<String> generateTextFrom(List<String> words) {
Random generator = new Random();
List<String> result = new ArrayList<>();
for(int i = 0; i < 10_000; i++) {
int random = generator.nextInt(words.size());
result.add(words.get(random));
}
return result;
}
Use the debugging feature of your favorite IDE (might be Eclipse), set an exception breakpoint on ArrayIndexOutOfBoundsException, run your program in debug mode.
When it hits the exception, Eclipse will halt your program. Look at your variable values, especially which array you are accessing and what value the index has, and why it got a value outside of the array size.
By the way, your code line if(n.length()>4){ cannot produce an ArrayIndexOutOfBoundsException, as there's no array indexing in that line.
I had a file to read and with this code I succeeded my JUnit tests. As you can see, I pass the String line as parameter to the readPrevisione(...) method.
package oroscopo.persistence;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.Reader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.NoSuchElementException;
import java.util.Set;
import java.util.StringTokenizer;
import oroscopo.model.Previsione;
import oroscopo.model.SegnoZodiacale;
public class TextFileOroscopoRepository implements OroscopoRepository {
private HashMap<String, List<Previsione>> mapSettore = new HashMap<>();
public TextFileOroscopoRepository(Reader baseReader) throws IOException, BadFileFormatException{
if (baseReader == null)
throw new IllegalArgumentException("baseReader is null");
BufferedReader bufReader = new BufferedReader(baseReader);
String line;
while((line=bufReader.readLine()) != null){
readPrevisione(line,bufReader);
}
}
private void readPrevisione(String line, BufferedReader bufReader) throws IOException, BadFileFormatException{
String nomeSettore = line.trim();
if (!Character.isUpperCase(nomeSettore.charAt(0)))
throw new BadFileFormatException();
List<Previsione> listaPrev = new ArrayList<>();
while (!(line = bufReader.readLine()).equalsIgnoreCase("FINE")){
try{
StringTokenizer st1 = new StringTokenizer(line, "\t");
if(st1.countTokens() < 2)
throw new BadFileFormatException();
String prev = st1.nextToken("\t").trim();
int val = Integer.parseInt(st1.nextToken("\t").trim());
Set<SegnoZodiacale> segni = new HashSet<>();
if (st1.hasMoreTokens()){
while(st1.hasMoreTokens()){
try{
segni.add(SegnoZodiacale.valueOf(st1.nextToken(",").trim()));
}
catch (IllegalArgumentException e){
throw new BadFileFormatException();
}
}
Previsione p = new Previsione(prev,val,segni);
listaPrev.add(p);
}
else{
Previsione p2 = new Previsione(prev,val);
listaPrev.add(p2);
}
}
catch (NumberFormatException e){
throw new BadFileFormatException();
}
catch (NoSuchElementException e){
throw new BadFileFormatException();
}
}
mapSettore.put(nomeSettore, listaPrev);
}
#Override
public Set<String> getSettori() {
return mapSettore.keySet();
}
#Override
public List<Previsione> getPrevisioni(String settore) {
return mapSettore.get(settore.toUpperCase());
}
}
Here with the same code, instead passing the read line as parameter, I pass the StringTokenizer that already has read the line. It should work like above but my JUnit tests fail. What did I do wrong?
package oroscopo.persistence;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.Reader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.NoSuchElementException;
import java.util.Set;
import java.util.StringTokenizer;
import oroscopo.model.Previsione;
import oroscopo.model.SegnoZodiacale;
public class TextFileOroscopoRepository implements OroscopoRepository {
private HashMap<String, List<Previsione>> mapSettore = new HashMap<>();
public TextFileOroscopoRepository(Reader baseReader) throws IOException, BadFileFormatException{
if (baseReader == null)
throw new IllegalArgumentException("baseReader is null");
BufferedReader bufReader = new BufferedReader(baseReader);
String line;
while((line=bufReader.readLine()) != null){
StringTokenizer st = new StringTokenizer(line);
readPrevisione(st,bufReader);
}
}
private void readPrevisione(StringTokenizer st, BufferedReader bufReader) throws IOException, BadFileFormatException{
String nomeSettore = st.nextToken().trim();
if (!Character.isUpperCase(nomeSettore.charAt(0)))
throw new BadFileFormatException();
List<Previsione> listaPrev = new ArrayList<>();
String line;
while (!(line = bufReader.readLine()).equalsIgnoreCase("FINE")){
try{
StringTokenizer st1 = new StringTokenizer(line, "\t");
if(st1.countTokens() < 2)
throw new BadFileFormatException();
String prev = st1.nextToken("\t").trim();
int val = Integer.parseInt(st1.nextToken("\t").trim());
Set<SegnoZodiacale> segni = new HashSet<>();
if (st1.hasMoreTokens()){
while(st1.hasMoreTokens()){
try{
segni.add(SegnoZodiacale.valueOf(st1.nextToken(",").trim()));
}
catch (IllegalArgumentException e){
throw new BadFileFormatException();
}
}
Previsione p = new Previsione(prev,val,segni);
listaPrev.add(p);
}
else{
Previsione p2 = new Previsione(prev,val);
listaPrev.add(p2);
}
}
catch (NumberFormatException e){
throw new BadFileFormatException();
}
catch (NoSuchElementException e){
throw new BadFileFormatException();
}
}
mapSettore.put(nomeSettore, listaPrev);
}
#Override
public Set<String> getSettori() {
return mapSettore.keySet();
}
#Override
public List<Previsione> getPrevisioni(String settore) {
return mapSettore.get(settore.toUpperCase());
}
}
EDIT: Here is the File.txt that I want to read.
And here is an example of one of my JUnit test:
#Test
public void testLetturaCorrettaPrevisioni1() throws IOException, BadFileFormatException {
Reader mr = new StringReader(
"NOMESEZIONE\navrai la testa un po' altrove\t\t4\tARIETE,TORO,GEMELLI\ngrande intimita'\t9\nFINE\n"
+ "SEZIONE2\ntesto di prova\t\t\t\t\t66\t\nFINE");
OroscopoRepository or = new TextFileOroscopoRepository(mr);
assertEquals("avrai la testa un po' altrove", or.getPrevisioni("nomesezione").get(0).getPrevisione());
assertEquals(4, or.getPrevisioni("nomesezione").get(0).getValore());
Set<SegnoZodiacale> validi = new HashSet<SegnoZodiacale>() {
private static final long serialVersionUID = 1L;
{
add(SegnoZodiacale.ARIETE);
add(SegnoZodiacale.TORO);
add(SegnoZodiacale.GEMELLI);
}
};
for (SegnoZodiacale s : SegnoZodiacale.values()) {
if (validi.contains(s))
assertTrue(or.getPrevisioni("nomesezione").get(0).validaPerSegno(s));
else
assertFalse(or.getPrevisioni("nomesezione").get(0).validaPerSegno(s));
}
assertEquals("grande intimita'", or.getPrevisioni("nomesezione").get(1).getPrevisione());
assertEquals(9, or.getPrevisioni("nomesezione").get(1).getValore());
for (SegnoZodiacale s : SegnoZodiacale.values()) {
assertTrue(or.getPrevisioni("nomesezione").get(1).validaPerSegno(s));
}
}
You are creating StringTokenizer with default delimiter, that is, "the space character, the tab character, the newline character, the carriage-return character, and the form-feed character."
So in the first case you set as value of the "nomeSettore" variable the whole line but when you use StringTokenizer.nextToken() you are giving to "nomeSettore" just the value of the first token. So, "nomeSettore" can have different values if your String "line" contains whitespaces and you will have different key-value pairs inside you map.
You can take a look at this example:
public class TestSO {
public static void main(String[] args) {
String line = "abcdfs faf afd fa";
StringTokenizer st = new StringTokenizer(line);
readPrevisione(st, null);
readPrevisione(line, null);
}
private static void readPrevisione(StringTokenizer st, BufferedReader bufReader) {
String nomeSettore = st.nextToken().trim();
System.out.println(nomeSettore);
}
private static void readPrevisione(String st, BufferedReader bufReader) {
String nomeSettore = st.trim();
System.out.println(nomeSettore);
}
}
It prints as output:
abcdfs
abcdfs faf afd fa
I've understood why it didn't work..
The String line was : "EXAMPLE\n"
but after
while((line=bufReader.readLine()) != null){
...}
line = "EXAMPLE" because the readLine() eats the newline.
So I passed to the readPrevisione() a StringTokenizer as parameter
while((line=bufReader.readLine()) != null){
StringTokenizer st = new StringTokenizer(line);
readPrevisione(st,bufReader);
}
private void readPrevisione(StringTokenizer st, BufferedReader bufReader) throws IOException, BadFileFormatException{
String nomeSettore = st.nextToken().trim();
...}
And st.nextToken() search for a \n that is not contained in "EXAMPLE". That's why it didn't work.
I was wondering if anyone has logic in java that removes duplicate lines while maintaining the lines order.
I would prefer no regex solution.
public class UniqueLineReader extends BufferedReader {
Set<String> lines = new HashSet<String>();
public UniqueLineReader(Reader arg0) {
super(arg0);
}
#Override
public String readLine() throws IOException {
String uniqueLine;
if (lines.add(uniqueLine = super.readLine()))
return uniqueLine;
return "";
}
//for testing..
public static void main(String args[]) {
try {
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream(
"test.txt");
UniqueLineReader br = new UniqueLineReader(new InputStreamReader(fstream));
String strLine;
// Read File Line By Line
while ((strLine = br.readLine()) != null) {
// Print the content on the console
if (strLine != "")
System.out.println(strLine);
}
// Close the input stream
in.close();
} catch (Exception e) {// Catch exception if any
System.err.println("Error: " + e.getMessage());
}
}
}
Modified Version:
public class UniqueLineReader extends BufferedReader {
Set<String> lines = new HashSet<String>();
public UniqueLineReader(Reader arg0) {
super(arg0);
}
#Override
public String readLine() throws IOException {
String uniqueLine;
while (lines.add(uniqueLine = super.readLine()) == false); //read until encountering a unique line
return uniqueLine;
}
public static void main(String args[]) {
try {
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream(
"/home/emil/Desktop/ff.txt");
UniqueLineReader br = new UniqueLineReader(new InputStreamReader(fstream));
String strLine;
// Read File Line By Line
while ((strLine = br.readLine()) != null) {
// Print the content on the console
System.out.println(strLine);
}
// Close the input stream
in.close();
} catch (Exception e) {// Catch exception if any
System.err.println("Error: " + e.getMessage());
}
}
}
If you feed the lines into a LinkedHashSet, it ignores the repeated ones, since it's a set, but preserves the order, since it's linked. If you just want to know whether you've seena given line before, feed them into a simple Set as you go on, and ignore those which the Set already contains/contained.
It can be easy to remove duplicate line from text or File using new java Stream API. Stream support different aggregate feature like sort,distinct and work with different java's existing data structures and their methods. Following example can use to remove duplicate or sort the content in File using Stream API
package removeword;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.OpenOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Scanner;
import java.util.stream.Stream;
import static java.nio.file.StandardOpenOption.*;
import static java.util.stream.Collectors.joining;
public class Java8UniqueWords {
public static void main(String[] args) throws IOException {
Path sourcePath = Paths.get("C:/Users/source.txt");
Path changedPath = Paths.get("C:/Users/removedDouplicate_file.txt");
try (final Stream<String> lines = Files.lines(sourcePath )
// .map(line -> line.toLowerCase()) /*optional to use existing string methods*/
.distinct()
// .sorted()) /*aggregrate function to sort disctincted line*/
{
final String uniqueWords = lines.collect(joining("\n"));
System.out.println("Final Output:" + uniqueWords);
Files.write(changedPath , uniqueWords.getBytes(),WRITE, TRUNCATE_EXISTING);
}
}
}
Read the text file using a BufferedReader and store it in a LinkedHashSet. Print it back out.
Here's an example:
public class DuplicateRemover {
public String stripDuplicates(String aHunk) {
StringBuilder result = new StringBuilder();
Set<String> uniqueLines = new LinkedHashSet<String>();
String[] chunks = aHunk.split("\n");
uniqueLines.addAll(Arrays.asList(chunks));
for (String chunk : uniqueLines) {
result.append(chunk).append("\n");
}
return result.toString();
}
}
Here's some unit tests to verify ( ignore my evil copy-paste ;) ):
import org.junit.Test;
import static org.junit.Assert.*;
public class DuplicateRemoverTest {
#Test
public void removesDuplicateLines() {
String input = "a\nb\nc\nb\nd\n";
String expected = "a\nb\nc\nd\n";
DuplicateRemover remover = new DuplicateRemover();
String actual = remover.stripDuplicates(input);
assertEquals(expected, actual);
}
#Test
public void removesDuplicateLinesUnalphabetized() {
String input = "z\nb\nc\nb\nz\n";
String expected = "z\nb\nc\n";
DuplicateRemover remover = new DuplicateRemover();
String actual = remover.stripDuplicates(input);
assertEquals(expected, actual);
}
}
Here's another solution. Let's just use UNIX!
cat MyFile.java | uniq > MyFile.java
Edit: Oh wait, I re-read the topic. Is this a legal solution since I managed to be language agnostic?
For better/optimum performance, it's wise to use Java 8's API features viz. Streams & Method references with LinkedHashSet for Collection as below:
import java.io.IOException;
import java.io.PrintWriter;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.LinkedHashSet;
import java.util.stream.Collectors;
public class UniqueOperation {
private static PrintWriter pw;
enter code here
public static void main(String[] args) throws IOException {
pw = new PrintWriter("abc.txt");
for(String p : Files.newBufferedReader(Paths.get("C:/Users/as00465129/Desktop/FrontEndUdemyLinks.txt")).
lines().
collect(Collectors.toCollection(LinkedHashSet::new)))
pw.println(p);
pw.flush();
pw.close();
System.out.println("File operation performed successfully");
}
here I'm using a hashset to store seen lines
Scanner scan;//input
Set<String> lines = new HashSet<String>();
StringBuilder strb = new StringBuilder();
while(scan.hasNextLine()){
String line = scan.nextLine();
if(lines.add(line)) strb.append(line);
}