I am trying to run a mapreduce job on hadoop which reads the fifth entry of a tab delimited file (fifth entry are user reviews) and then do some sentiment analysis and word count on them.
However, as you know with user reviews, they usually include line breaks and empty lines. My code iterates through the words of each review to find keywords and check sentiment if keyword is found.
The problem is as the code iterates through the review, it gives me ArrayIndexOutofBoundsException Error because of these line breaks and empty lines in one review.
I have tried using replaceAll("\r", " ") and replaceAll("\n", " ") to no avail.
I have also tried if(tokenizer.countTokens() == 2){
word.set(tokenizer.nextToken());}
else {
}
also to no avail. Below is my code:
public class KWSentiment_Mapper extends Mapper<LongWritable, Text, Text, IntWritable> {
ArrayList<String> keywordsList = new ArrayList<String>();
ArrayList<String> posWordsList = new ArrayList<String>();
ArrayList<String> tokensList = new ArrayList<String>();
int e;
#Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String[] line = value.toString().split("\t");
String Review = line[4].replaceAll("[\\-\\+\\\\)\\.\\(\"\\{\\$\\^:,]", "").toLowerCase();
StringTokenizer tokenizer = new StringTokenizer(Review);
while (tokenizer.hasMoreTokens()) {
// 1- first read the review line and store the tokens in an arraylist, 2-
// iterate through review to check for KW if found
// 3-check if there's PosWord near (upto +3 and -2)
// 4- setWord & context.write 5- null the review line arraylist
String CompareString = tokenizer.nextToken();
tokensList.add(CompareString);
}
{
for (int i = 0; i < tokensList.size(); i++)
{
for (int j = 0; j < keywordsList.size(); j++) {
boolean flag = false;
if (tokensList.get(i).startsWith(keywordsList.get(j)) == true) {
for (int e = Math.max(0, i - 2); e < Math.min(tokensList.size(), i + 4); e++) {
if (posWordsList.contains(tokensList.get(e))) {
word.set(keywordsList.get(j));
context.write(word, one);
flag = true;
break; // breaks out of e loop }}
}
}
}
if (flag)
break;
}
}
tokensList.clear();
}
}
Expected results are such that:
Take these two cases of reviews where error occurs:
Case 1: "Beautiful and spacious!
I highly recommend this place and great host."
Case 2: "The place in general was really silent but we didn't feel stayed.
Aside from this, the bathroom is big and the shower is really nice but there problem. "
The system should read the whole review as one line and iterate through the words in it. However, it just stops as it finds a line break or an empty line as in case 2.
Case 1 should be read such as: "Beautiful and spacious! I highly recommend this place and great host."
Case 2 should be:"The place in general was really silent but we didn't feel stayed. Aside from this, the bathroom is big and the shower is really nice but there problem. "
I am running out of time and would really appreciate help here.
Thanks!
So, I hope I am understanding what what you are trying to do....
If I am reading what you have above correctly, the value of 'value' passed into your map function above contains the delimited value that you would like to parse the user reviews out of. If that is the case, I believe we can make use of the escaping functionality in the opencsv library using tabs as your delimiting character instead of commas to correctly populate the user review field:
http://opencsv.sourceforge.net
In this example we are reading one line from the input that is passed in and parsing it into 'columns' base on the tab character and placing the results in the 'nextLine' array. This will allow us to use the escaping functionality of the CSVReader without reading an actual file and instead using the value of the text passed into your map function.
StringReader reader = new StringReader(value.toString());
CSVReader csvReader = new CSVReader(reader, '\t', '\"', '\\', 0);
String [] nextLine = csvReader.readNext();
if(nextLine != null && nextLine.length >= 5) {
// Do some stuff
}
In the example that you pasted above, I think even that split("\n") will be problematic as tabs within a user review split into two results in the result in addition to new lines being treated as new records. But, both of these characters are legal as long as they are inside a quoted value (as they should be in a properly escaped file and as they are in your example). CSVReader should handle all of these.
Validate each line at the start of the map method, so that you know line[4] exists and isn't null.
if (value == null || value.toString == null) {
return;
}
String[] line = value.toString().split("\t");
if (line == null || line.length() < 5 || line[4] == null) {
return;
}
As for line breaks, you'll need to show some sample input. By default MapReduce passes each line into the map method independently, so if you do want to read multiple lines as one message, you'll have to write a custom InputSplit, or pre-format your data so that all data for each review is on the same line.
Related
My csv is getting read into the System.out, but I've noticed that any text with a space gets moved into the next line (as a return \n)
Here's how my csv starts:
first,last,email,address 1, address 2
john,smith,blah#blah.com,123 St. Street,
Jane,Smith,blech#blech.com,4455 Roger Cir,apt 2
After running my app, any cell with a space (address 1), gets thrown onto the next line.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class main {
public static void main(String[] args) {
// -define .csv file in app
String fileNameDefined = "uploadedcsv/employees.csv";
// -File class needed to turn stringName to actual file
File file = new File(fileNameDefined);
try{
// -read from filePooped with Scanner class
Scanner inputStream = new Scanner(file);
// hashNext() loops line-by-line
while(inputStream.hasNext()){
//read single line, put in string
String data = inputStream.next();
System.out.println(data + "***");
}
// after loop, close scanner
inputStream.close();
}catch (FileNotFoundException e){
e.printStackTrace();
}
}
}
So here's the result in the console:
first,last,email,address
1,address
2
john,smith,blah#blah.com,123
St.
Street,
Jane,Smith,blech#blech.com,4455
Roger
Cir,apt
2
Am I using Scanner incorrectly?
Please stop writing faulty CSV parsers!
I've seen hundreds of CSV parsers and so called tutorials for them online.
Nearly every one of them gets it wrong!
This wouldn't be such a bad thing as it doesn't affect me but people who try to write CSV readers and get it wrong tend to write CSV writers, too. And get them wrong as well. And these ones I have to write parsers for.
Please keep in mind that CSV (in order of increasing not so obviousness):
can have quoting characters around values
can have other quoting characters than "
can even have other quoting characters than " and '
can have no quoting characters at all
can even have quoting characters on some values and none on others
can have other separators than , and ;
can have whitespace between seperators and (quoted) values
can have other charsets than ascii
should have the same number of values in each row, but doesn't always
can contain empty fields, either quoted: "foo","","bar" or not: "foo",,"bar"
can contain newlines in values
can not contain newlines in values if they are not delimited
can not contain newlines between values
can have the delimiting character within the value if properly escaped
does not use backslash to escape delimiters but...
uses the quoting character itself to escape it, e.g. Frodo's Ring will be 'Frodo''s Ring'
can have the quoting character at beginning or end of value, or even as only character ("foo""", """bar", """")
can even have the quoted character within the not quoted value; this one is not escaped
If you think this is obvious not a problem, then think again. I've seen every single one of these items implemented wrongly. Even in major software packages. (e.g. Office-Suites, CRM Systems)
There are good and correctly working out-of-the-box CSV readers and writers out there:
opencsv
Ostermiller Java Utilities
Apache Commons CSV
If you insist on writing your own at least read the (very short) RFC for CSV.
scanner.useDelimiter(",");
This should work.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class TestScanner {
public static void main(String[] args) throws FileNotFoundException {
Scanner scanner = new Scanner(new File("/Users/pankaj/abc.csv"));
scanner.useDelimiter(",");
while(scanner.hasNext()){
System.out.print(scanner.next()+"|");
}
scanner.close();
}
}
For CSV File:
a,b,c d,e
1,2,3 4,5
X,Y,Z A,B
Output is:
a|b|c d|e
1|2|3 4|5
X|Y|Z A|B|
Scanner.next() does not read a newline but reads the next token, delimited by whitespace (by default, if useDelimiter() was not used to change the delimiter pattern). To read a line use Scanner.nextLine().
Once you read a single line you can use String.split(",") to separate the line into fields. This enables identification of lines that do not consist of the required number of fields. Using useDelimiter(","); would ignore the line-based structure of the file (each line consists of a list of fields separated by a comma). For example:
while (inputStream.hasNextLine())
{
String line = inputStream.nextLine();
String[] fields = line.split(",");
if (fields.length >= 4) // At least one address specified.
{
for (String field: fields) System.out.print(field + "|");
System.out.println();
}
else
{
System.err.println("Invalid record: " + line);
}
}
As already mentioned, using a CSV library is recommended. For one, this (and useDelimiter(",") solution) will not correctly handle quoted identifiers containing , characters.
I agree with Scheintod that using an existing CSV library is a good idea to have RFC-4180-compliance from the start. Besides the mentioned OpenCSV and Oster Miller, there are a series of other CSV libraries out there. If you're interested in performance, you can take a look at the uniVocity/csv-parsers-comparison. It shows that
uniVocity CSV parser
SimpleFlatMapper CSV parser
Jackson CSV parser
are consistently the fastest using either JDK 6, 7, 8, or 9. The study did not find any RFC 4180 compatibility issues in any of those three. Both OpenCSV and Oster Miller are found to be about twice as slow as those.
I'm not in any way associated with the author(s), but concerning the uniVocity CSV parser, the study might be biased due to its author being the same as of that parser.
To note, the author of SimpleFlatMapper has also published a performance comparison comparing only those three.
Split nextLine() by this delimiter:
(?=([^\"]*\"[^\"]*\")*[^\"]*$)").
I have seen many production problems caused by code not handling quotes ("), newline characters within quotes, and quotes within the quotes; e.g.: "he said ""this""" should be parsed into: he said "this"
Like it was mentioned earlier, many CSV parsing examples out there just read a line, and then break up the line by the separator character. This is rather incomplete and problematic.
For me and probably those who prefer build verses buy (or use somebody else's code and deal with their dependencies), I got down to classic text parsing programming and that worked for me:
/**
* Parse CSV data into an array of String arrays. It handles double quoted values.
* #param is input stream
* #param separator
* #param trimValues
* #param skipEmptyLines
* #return an array of String arrays
* #throws IOException
*/
public static String[][] parseCsvData(InputStream is, char separator, boolean trimValues, boolean skipEmptyLines)
throws IOException
{
ArrayList<String[]> data = new ArrayList<String[]>();
ArrayList<String> row = new ArrayList<String>();
StringBuffer value = new StringBuffer();
int ch = -1;
int prevCh = -1;
boolean inQuotedValue = false;
boolean quoteAtStart = false;
boolean rowIsEmpty = true;
boolean isEOF = false;
while (true)
{
prevCh = ch;
ch = (isEOF) ? -1 : is.read();
// Handle carriage return line feed
if (prevCh == '\r' && ch == '\n')
{
continue;
}
if (inQuotedValue)
{
if (ch == -1)
{
inQuotedValue = false;
isEOF = true;
}
else
{
value.append((char)ch);
if (ch == '"')
{
inQuotedValue = false;
}
}
}
else if (ch == separator || ch == '\r' || ch == '\n' || ch == -1)
{
// Add the value to the row
String s = value.toString();
if (quoteAtStart && s.endsWith("\""))
{
s = s.substring(1, s.length() - 1);
}
if (trimValues)
{
s = s.trim();
}
rowIsEmpty = (s.length() > 0) ? false : rowIsEmpty;
row.add(s);
value.setLength(0);
if (ch == '\r' || ch == '\n' || ch == -1)
{
// Add the row to the result
if (!skipEmptyLines || !rowIsEmpty)
{
data.add(row.toArray(new String[0]));
}
row.clear();
rowIsEmpty = true;
if (ch == -1)
{
break;
}
}
}
else if (prevCh == '"')
{
inQuotedValue = true;
}
else
{
if (ch == '"')
{
inQuotedValue = true;
quoteAtStart = (value.length() == 0) ? true : false;
}
value.append((char)ch);
}
}
return data.toArray(new String[0][]);
}
Unit Test:
String[][] data = parseCsvData(new ByteArrayInputStream("foo,\"\",,\"bar\",\"\"\"music\"\"\",\"carriage\r\nreturn\",\"new\nline\"\r\nnext,line".getBytes()), ',', true, true);
for (int rowIdx = 0; rowIdx < data.length; rowIdx++)
{
System.out.println(Arrays.asList(data[rowIdx]));
}
generates the output:
[foo, , , bar, "music", carriage
return, new
line]
[next, line]
If you absolutely must use Scanner, then you must set its delimiter via its useDelimiter(...) method. Else it will default to using all white space as its delimiter. Better though as has already been stated -- use a CSV library since this is what they do best.
For example, this delimiter will split on commas with or without surrounding whitespace:
scanner.useDelimiter("\\s*,\\s*");
Please check out the java.util.Scanner API for more on this.
Well, I do my coding in NetBeans 8.1:
First: Create a new project, select Java application and name your project.
Then modify your code after public class to look like the following:
/**
* #param args the command line arguments
* #throws java.io.FileNotFoundException
*/
public static void main(String[] args) throws FileNotFoundException {
try (Scanner scanner = new Scanner(new File("C:\\Users\\YourName\\Folder\\file.csv"))) {
scanner.useDelimiter(",");
while(scanner.hasNext()){
System.out.print(scanner.next()+"|");
}}
}
}
I am working on a Comp Sci assignment. In the end, the program will determine whether a file is written in English or French. Right now, I'm struggling with the method that counts the frequency of words that appears in a .txt file.
I have a set of text files in both English and French in their respective folders labeled 1-20. The method asks for a directory (which in this case is "docs/train/eng/" or "docs/train/fre/") and for how many files that the program should go through (there are 20 files in each folder). Then it reads that file, splits all the words apart (I don't need to worry about capitalization or punctuation), and puts every word in a HashMap along with how many times they were in the file. (Key = word, Value = frequency).
This is the code I came up with for the method:
public static HashMap<String, Integer> countWords(String directory, int nFiles) {
// Declare the HashMap
HashMap<String, Integer> wordCount = new HashMap();
// this large 'for' loop will go through each file in the specified directory.
for (int k = 1; k < nFiles; k++) {
// Puts together the string that the FileReader will refer to.
String learn = directory + k + ".txt";
try {
FileReader reader = new FileReader(learn);
BufferedReader br = new BufferedReader(reader);
// The BufferedReader reads the lines
String line = br.readLine();
// Split the line into a String array to loop through
String[] words = line.split(" ");
int freq = 0;
// for loop goes through every word
for (int i = 0; i < words.length; i++) {
// Case if the HashMap already contains the key.
// If so, just increments the value
if (wordCount.containsKey(words[i])) {
wordCount.put(words[i], freq++);
}
// Otherwise, puts the word into the HashMap
else {
wordCount.put(words[i], freq++);
}
}
// Catching the file not found error
// and any other errors
}
catch (FileNotFoundException fnfe) {
System.err.println("File not found.");
}
catch (Exception e) {
System.err.print(e);
}
}
return wordCount;
}
The code compiles. Unfortunately, when I asked it to print the results of all the word counts for the 20 files, it printed this. It's complete gibberish (though the words are definitely there) and is not at all what I need the method to do.
If anyone could help me debug my code, I would greatly appreciate it. I've been at it for ages, conducting test after test and I'm ready to give up.
Let me combine all the good answers here.
1) Split up your methods to handle one thing each. One to read the files into strings[], one to process the strings[], and one to call the first two.
2) When you split think deeply about how you want to split. As #m0skit0 suggest you should likely split with \b for this problem.
3) As #jas suggested you should first check if your map already has the word. If it does increment the count, if not add the word to the map and set it's count to 1.
4) To print out the map in the way you likely expect, take a look at the below:
Map test = new HashMap();
for (Map.Entry entry : test.entrySet()){
System.out.println(entry.getKey() + " " + entry.getValue());
}
I would have expected something more like this. Does it make sense?
if (wordCount.containsKey(words[i])) {
int n = wordCount.get(words[i]);
wordCount.put(words[i], ++n);
}
// Otherwise, puts the word into the HashMap
else {
wordCount.put(words[i], 1);
}
If the word is already in the hashmap, we want to get the current count, add 1 to that and replace the word with the new count in the hashmap.
If the word is not yet in the hashmap, we simply put it in the map with a count of 1 to start with. The next time we see the same word we'll up the count to 2, etc.
If you split by space only, then other signs (parenthesis, punctuation marks, etc...) will be included in the words. For example: "This phrase, contains... funny stuff", if you split it by space you get: "This" "phrase," "contains..." "funny" and "stuff".
You can avoid this by splitting by word boundary (\b) instead.
line.split("\\b");
Btw your if and else parts are identical. You're always incrementing freq by one, which doesn't make much sense. If the word is already in the map, you want to get the current frequency, add 1 to it, and update the frequency in the map. If not, you put it in the map with a value of 1.
And pro tip: always print/log the full stacktrace for the exceptions.
Basically I want to create a program which simulates the 'Countdown' game on Channel 4. In effect a user must input 9 letters and the program will search for the largest word in the dictionary that can be made from these letters.I think a tree structure would be better to go with rather than hash tables. I already have a file which contains the words in the dictionary and will be using file io.
This is my file io class:
public static void main(String[] args){
FileIO reader = new FileIO();
String[] contents = reader.load("dictionary.txt");
}
This is what I have so far in my Countdown class
public static void main(String[] args) throws IOException{
Scanner scan = new Scanner(System.in);
letters = scan.NextLine();
}
I get totally lost from here. I know this is only the start but I'm not looking for answers. I just want a small bit of help and maybe a pointer in the right direction. I'm only new to java and found this question in an interview book and thought I should give it a .
Thanks in advance
welcome to the world of Java :)
The first thing I see there that you have two main methods, you don't actually need that. Your program will have a single entry point in most cases then it does all its logic and handles user input and everything.
You're thinking of a tree structure which is good, though there might be a better idea to store this. Try this: http://en.wikipedia.org/wiki/Trie
What your program has to do is read all the words from the file line by line, and in this process build your data structure, the tree. When that's done you can ask the user for input and after the input is entered you can search the tree.
Since you asked specifically not to provide answers I won't put code here, but feel free to ask if you're unclear about something
There are only about 800,000 words in the English language, so an efficient solution would be to store those 800,000 words as 800,000 arrays of 26 1-byte integers that count how many times each letter is used in the word, and then for an input 9 characters you convert to similar 26 integer count format for the query, and then a word can be formed from the query letters if the query vector is greater than or equal to the word-vector component-wise. You could easily process on the order of 100 queries per second this way.
I would write a program that starts with all the two-letter words, then does the three-letter words, the four-letter words and so on.
When you do the two-letter words, you'll want some way of picking the first letter, then picking the second letter from what remains. You'll probably want to use recursion for this part. Lastly, you'll check it against the dictionary. Try to write it in a way that means you can re-use the same code for the three-letter words.
I believe, the power of Regular Expressions would come in handy in your case:
1) Create a regular expression string with a symbol class like: /^[abcdefghi]*$/ with your letters inside instead of "abcdefghi".
2) Use that regular expression as a filter to get a strings array from your text file.
3) Sort it by length. The longest word is what you need!
Check the Regular Expressions Reference for more information.
UPD: Here is a good Java Regex Tutorial.
A first approach could be using a tree with all the letters present in the wordlist.
If one node is the end of a word, then is marked as an end-of-word node.
In the picture above, the longest word is banana. But there are other words, like ball, ban, or banal.
So, a node must have:
A character
If it is the end of a word
A list of children. (max 26)
The insertion algorithm is very simple: In each step we "cut" the first character of the word until the word has no more characters.
public class TreeNode {
public char c;
private boolean isEndOfWord = false;
private TreeNode[] children = new TreeNode[26];
public TreeNode(char c) {
this.c = c;
}
public void put(String s) {
if (s.isEmpty())
{
this.isEndOfWord = true;
return;
}
char first = s.charAt(0);
int pos = position(first);
if (this.children[pos] == null)
this.children[pos] = new TreeNode(first);
this.children[pos].put(s.substring(1));
}
public String search(char[] letters) {
String word = "";
String w = "";
for (int i = 0; i < letters.length; i++)
{
TreeNode child = children[position(letters[i])];
if (child != null)
w = child.search(letters);
//this is not efficient. It should be optimized.
if (w.contains("%")
&& w.substring(0, w.lastIndexOf("%")).length() > word
.length())
word = w;
}
// if a node its end-of-word we add the special char '%'
return c + (this.isEndOfWord ? "%" : "") + word;
}
//if 'a' returns 0, if 'b' returns 1...etc
public static int position(char c) {
return ((byte) c) - 97;
}
}
Example:
public static void main(String[] args) {
//root
TreeNode t = new TreeNode('R');
//for skipping words with "'" in the wordlist
Pattern p = Pattern.compile(".*\\W+.*");
int nw = 0;
try (BufferedReader br = new BufferedReader(new FileReader(
"files/wordsEn.txt")))
{
for (String line; (line = br.readLine()) != null;)
{
if (p.matcher(line).find())
continue;
t.put(line);
nw++;
}
// line is not visible here.
br.close();
System.out.println("number of words : " + nw);
String res = null;
// substring (1) because of the root
res = t.search("vuetsrcanoli".toCharArray()).substring(1);
System.out.println(res.replace("%", ""));
}
catch (Exception e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Output:
number of words : 109563
counterrevolutionaries
Notes:
The wordlist is taken from here
the reading part is based on another SO question : How to read a large text file line by line using Java?
My csv is getting read into the System.out, but I've noticed that any text with a space gets moved into the next line (as a return \n)
Here's how my csv starts:
first,last,email,address 1, address 2
john,smith,blah#blah.com,123 St. Street,
Jane,Smith,blech#blech.com,4455 Roger Cir,apt 2
After running my app, any cell with a space (address 1), gets thrown onto the next line.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class main {
public static void main(String[] args) {
// -define .csv file in app
String fileNameDefined = "uploadedcsv/employees.csv";
// -File class needed to turn stringName to actual file
File file = new File(fileNameDefined);
try{
// -read from filePooped with Scanner class
Scanner inputStream = new Scanner(file);
// hashNext() loops line-by-line
while(inputStream.hasNext()){
//read single line, put in string
String data = inputStream.next();
System.out.println(data + "***");
}
// after loop, close scanner
inputStream.close();
}catch (FileNotFoundException e){
e.printStackTrace();
}
}
}
So here's the result in the console:
first,last,email,address
1,address
2
john,smith,blah#blah.com,123
St.
Street,
Jane,Smith,blech#blech.com,4455
Roger
Cir,apt
2
Am I using Scanner incorrectly?
Please stop writing faulty CSV parsers!
I've seen hundreds of CSV parsers and so called tutorials for them online.
Nearly every one of them gets it wrong!
This wouldn't be such a bad thing as it doesn't affect me but people who try to write CSV readers and get it wrong tend to write CSV writers, too. And get them wrong as well. And these ones I have to write parsers for.
Please keep in mind that CSV (in order of increasing not so obviousness):
can have quoting characters around values
can have other quoting characters than "
can even have other quoting characters than " and '
can have no quoting characters at all
can even have quoting characters on some values and none on others
can have other separators than , and ;
can have whitespace between seperators and (quoted) values
can have other charsets than ascii
should have the same number of values in each row, but doesn't always
can contain empty fields, either quoted: "foo","","bar" or not: "foo",,"bar"
can contain newlines in values
can not contain newlines in values if they are not delimited
can not contain newlines between values
can have the delimiting character within the value if properly escaped
does not use backslash to escape delimiters but...
uses the quoting character itself to escape it, e.g. Frodo's Ring will be 'Frodo''s Ring'
can have the quoting character at beginning or end of value, or even as only character ("foo""", """bar", """")
can even have the quoted character within the not quoted value; this one is not escaped
If you think this is obvious not a problem, then think again. I've seen every single one of these items implemented wrongly. Even in major software packages. (e.g. Office-Suites, CRM Systems)
There are good and correctly working out-of-the-box CSV readers and writers out there:
opencsv
Ostermiller Java Utilities
Apache Commons CSV
If you insist on writing your own at least read the (very short) RFC for CSV.
scanner.useDelimiter(",");
This should work.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class TestScanner {
public static void main(String[] args) throws FileNotFoundException {
Scanner scanner = new Scanner(new File("/Users/pankaj/abc.csv"));
scanner.useDelimiter(",");
while(scanner.hasNext()){
System.out.print(scanner.next()+"|");
}
scanner.close();
}
}
For CSV File:
a,b,c d,e
1,2,3 4,5
X,Y,Z A,B
Output is:
a|b|c d|e
1|2|3 4|5
X|Y|Z A|B|
Scanner.next() does not read a newline but reads the next token, delimited by whitespace (by default, if useDelimiter() was not used to change the delimiter pattern). To read a line use Scanner.nextLine().
Once you read a single line you can use String.split(",") to separate the line into fields. This enables identification of lines that do not consist of the required number of fields. Using useDelimiter(","); would ignore the line-based structure of the file (each line consists of a list of fields separated by a comma). For example:
while (inputStream.hasNextLine())
{
String line = inputStream.nextLine();
String[] fields = line.split(",");
if (fields.length >= 4) // At least one address specified.
{
for (String field: fields) System.out.print(field + "|");
System.out.println();
}
else
{
System.err.println("Invalid record: " + line);
}
}
As already mentioned, using a CSV library is recommended. For one, this (and useDelimiter(",") solution) will not correctly handle quoted identifiers containing , characters.
I agree with Scheintod that using an existing CSV library is a good idea to have RFC-4180-compliance from the start. Besides the mentioned OpenCSV and Oster Miller, there are a series of other CSV libraries out there. If you're interested in performance, you can take a look at the uniVocity/csv-parsers-comparison. It shows that
uniVocity CSV parser
SimpleFlatMapper CSV parser
Jackson CSV parser
are consistently the fastest using either JDK 6, 7, 8, or 9. The study did not find any RFC 4180 compatibility issues in any of those three. Both OpenCSV and Oster Miller are found to be about twice as slow as those.
I'm not in any way associated with the author(s), but concerning the uniVocity CSV parser, the study might be biased due to its author being the same as of that parser.
To note, the author of SimpleFlatMapper has also published a performance comparison comparing only those three.
Split nextLine() by this delimiter:
(?=([^\"]*\"[^\"]*\")*[^\"]*$)").
I have seen many production problems caused by code not handling quotes ("), newline characters within quotes, and quotes within the quotes; e.g.: "he said ""this""" should be parsed into: he said "this"
Like it was mentioned earlier, many CSV parsing examples out there just read a line, and then break up the line by the separator character. This is rather incomplete and problematic.
For me and probably those who prefer build verses buy (or use somebody else's code and deal with their dependencies), I got down to classic text parsing programming and that worked for me:
/**
* Parse CSV data into an array of String arrays. It handles double quoted values.
* #param is input stream
* #param separator
* #param trimValues
* #param skipEmptyLines
* #return an array of String arrays
* #throws IOException
*/
public static String[][] parseCsvData(InputStream is, char separator, boolean trimValues, boolean skipEmptyLines)
throws IOException
{
ArrayList<String[]> data = new ArrayList<String[]>();
ArrayList<String> row = new ArrayList<String>();
StringBuffer value = new StringBuffer();
int ch = -1;
int prevCh = -1;
boolean inQuotedValue = false;
boolean quoteAtStart = false;
boolean rowIsEmpty = true;
boolean isEOF = false;
while (true)
{
prevCh = ch;
ch = (isEOF) ? -1 : is.read();
// Handle carriage return line feed
if (prevCh == '\r' && ch == '\n')
{
continue;
}
if (inQuotedValue)
{
if (ch == -1)
{
inQuotedValue = false;
isEOF = true;
}
else
{
value.append((char)ch);
if (ch == '"')
{
inQuotedValue = false;
}
}
}
else if (ch == separator || ch == '\r' || ch == '\n' || ch == -1)
{
// Add the value to the row
String s = value.toString();
if (quoteAtStart && s.endsWith("\""))
{
s = s.substring(1, s.length() - 1);
}
if (trimValues)
{
s = s.trim();
}
rowIsEmpty = (s.length() > 0) ? false : rowIsEmpty;
row.add(s);
value.setLength(0);
if (ch == '\r' || ch == '\n' || ch == -1)
{
// Add the row to the result
if (!skipEmptyLines || !rowIsEmpty)
{
data.add(row.toArray(new String[0]));
}
row.clear();
rowIsEmpty = true;
if (ch == -1)
{
break;
}
}
}
else if (prevCh == '"')
{
inQuotedValue = true;
}
else
{
if (ch == '"')
{
inQuotedValue = true;
quoteAtStart = (value.length() == 0) ? true : false;
}
value.append((char)ch);
}
}
return data.toArray(new String[0][]);
}
Unit Test:
String[][] data = parseCsvData(new ByteArrayInputStream("foo,\"\",,\"bar\",\"\"\"music\"\"\",\"carriage\r\nreturn\",\"new\nline\"\r\nnext,line".getBytes()), ',', true, true);
for (int rowIdx = 0; rowIdx < data.length; rowIdx++)
{
System.out.println(Arrays.asList(data[rowIdx]));
}
generates the output:
[foo, , , bar, "music", carriage
return, new
line]
[next, line]
If you absolutely must use Scanner, then you must set its delimiter via its useDelimiter(...) method. Else it will default to using all white space as its delimiter. Better though as has already been stated -- use a CSV library since this is what they do best.
For example, this delimiter will split on commas with or without surrounding whitespace:
scanner.useDelimiter("\\s*,\\s*");
Please check out the java.util.Scanner API for more on this.
Well, I do my coding in NetBeans 8.1:
First: Create a new project, select Java application and name your project.
Then modify your code after public class to look like the following:
/**
* #param args the command line arguments
* #throws java.io.FileNotFoundException
*/
public static void main(String[] args) throws FileNotFoundException {
try (Scanner scanner = new Scanner(new File("C:\\Users\\YourName\\Folder\\file.csv"))) {
scanner.useDelimiter(",");
while(scanner.hasNext()){
System.out.print(scanner.next()+"|");
}}
}
}
My program reads lines from a plain text file w/ lines formatted: <integer>;<integer>%n, where ; is the delimiter. It compares the two parsed integers against 2 other known values and increments tallyArray[i] if they match.
I currently use:
try {
scan = new Scanner(new BufferedReader(new FileReader("LogFileToBeRead.txt")));
for (int i = 0; i < tallyArraySize; i++) {
explodedLogLine = scan.nextLine().split(";");
if (IntReferenceVal1 == Integer.parseInt(explodedLogLine[0]) && IntReferenceVal2 == Integer.parseInt(explodedLogLine[1])) {
tallyArray[i]++;
}
}
} finally {
if (scan != null) { scan.close(); }
}
I was wondering if there were any serious faults with this method. It does not need to be production-quality.
Also, is there a standard way of parsing a string like this?
EDIT: We can assume the text file is perfectly formatted. But I see the importance for accounting for possible exceptions.
You are not handling NumberFormatExceptions thrown by the Integer.parseInt() method calls. If there's one bad line, execution exits your for loop.
You aren't vetting the integrity of the file you are reading from. If there isn't a ; character or if the Strings aren't actually numbers, execution simply exits the code block you posted.
If you can assume the file is perfectly formatted, and you're set on using a Scanner, you can add ; as a delimiter to the Scanner:
scan = new Scanner(new BufferedReader(new FileReader("LogFileToBeRead.txt")));
scan.useDelimiter(Pattern.compile("(;|\\s)"));
for (int i = 0; i < tallyArraySize; i++) {
int ref1 = scan.nextInt();
int ref2 = scan.nextInt();
if (IntReferenceVal1 == ref1 &&
IntReferenceVal2 == ref2) {
tallyArray[i]++;
}
}
And simply call Scanner.nextInt() twice for each line.
According to me There are three flaws in the program.
Delimiter ; what if there is delimiter is removed by accident or added by accident
There should be check on explodedLogLine that it is of length 2 and it is not null otherwise it will result in unexpected runtime error
You should catch NumberFormatException format exception since you can never be sure that Input is always a number
A simple illustration below gives you idea how things will go wrong.
String str = "3;;3";
System.out.println(Arrays.toString(str.split(";")));
This code will print [3, , 3] in such case your program will produce NumberFormatException as "" string can not be parsed to Integer.