Develop Reference

How to determine the delimiter in CSV file

I have a scenario at which i have to parse CSV files from different sources, the parsing code is very simple and straightforward.
String csvFile = "/Users/csv/country.csv";
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
// use comma as separator
String[] country = line.split(cvsSplitBy);
System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
}
} catch (IOException e) {
e.printStackTrace();
}
my problem come from the CSV delimiter character, i have many different formats, some time it is a , sometimes it is a ;
is there is any way to determine the delimiter character before parsing the file

univocity-parsers supports automatic detection of the delimiter (also line endings and quotes). Just use it instead of fighting with your code:
CsvParserSettings settings = new CsvParserSettings();
settings.detectFormatAutomatically();
CsvParser parser = new CsvParser(settings);
List<String[]> rows = parser.parseAll(new File("/path/to/your.csv"));
// if you want to see what it detected
CsvFormat format = parser.getDetectedFormat();
Disclaimer: I'm the author of this library and I made sure all sorts of corner cases are covered. It's open source and free (Apache 2.0 license)
Hope this helps.

Yes, but only if the delimiter characters are not allowed to exist as regular text
The most simple answer is to have a list with all the available delimiter characters and try to identify which character is being used. Even though, you have to place some limitations on the files or the person/people that created them. Look a the following two scenarios:
Case 1 - Contents of file.csv
test,test2,test3
Case 2 - Contents of file.csv
test1|test2,3|test4
If you have prior knowledge of the delimiter characters, then you would split the first string using , and the second one using |, getting the same result. But, if you try to identify the delimiter by parsing the file, both strings can be split using the , character, and you would end up with this:
Case 1 - Result of split using ,
test1
test2
test3
Case 2 - Result of split using ,
test1|test2
3|test4
By lacking the prior knowledge of which delimiter character is being used, you cannot create a "magical" algorithm that will parse every combination of text; even regular expressions or counting the number of appearance of a character will not save you.
Worst case
test1,2|test3,4|test5
By looking the text, one can tokenize it by using | as the delimiter. But the frequency of appearance of both , and | are the same. So, from an algorithm's perspective, both results are accurate:
Correct result
test1,2
test3,4
test5
Wrong result
test1
2|test3
4|test5
If you pose a set of guidelines or you can somehow control the generation of the CSV files, then you could just try to find the delimiter used with String.contains() method, employing the aforementioned list of characters. For example:
public class MyClass {
private List<String> delimiterList = new ArrayList<>(){{
add(",");
add(";");
add("\t");
// etc...
}};
private static String determineDelimiter(String text) {
for (String delimiter : delimiterList) {
if(text.contains(delimiter)) {
return delimiter;
}
}
return "";
}
public static void main(String[] args) {
String csvFile = "/Users/csv/country.csv";
String line = "";
String cvsSplitBy = ",";
String delimiter = "";
boolean firstLine = true;
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
if(firstLine) {
delimiter = determineDelimiter(line);
if(delimiter.equalsIgnoreCase("")) {
System.out.println("Unsupported delimiter found: " + delimiter);
return;
}
firstLine = false;
}
// use comma as separator
String[] country = line.split(delimiter);
System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Update
For a more optimized way, in determineDelimiter() method instead of the for-each loop, you can employ regular expressions.

If the delimiter can appear in a data column, then you are asking for the impossible. For example, consider this first line of a CSV file:
one,two:three
This could be either a comma-separated or a colon-separated file. You can't tell which type it is.
If you can guarantee that the first line has all its columns surrounded by quotes, for example if it's always this format:
"one","two","three"
then you may be able to use this logic (although it's not 100% bullet-proof):
if (line.contains("\",\""))
delimiter = ',';
else if (line.contains("\";\""))
delimiter = ';';
If you can't guarantee a restricted format like that, then it would be better to pass the delimiter character as a parameter.
Then you can read the file using a widely-known open-source CSV parser such as Apache Commons CSV.

While I agree with Lefteris008 that it is not possible to have the function that correctly determine all the cases, we can have a function that is both efficient and give mostly correct result in practice.
def head(filename: str, n: int):
try:
with open(filename) as f:
head_lines = [next(f).rstrip() for x in range(n)]
except StopIteration:
with open(filename) as f:
head_lines = f.read().splitlines()
return head_lines
def detect_delimiter(filename: str, n=2):
sample_lines = head(filename, n)
common_delimiters= [',',';','\t',' ','|',':']
for d in common_delimiters:
ref = sample_lines[0].count(d)
if ref > 0:
if all([ ref == sample_lines[i].count(d) for i in range(1,n)]):
return d
return ','
My efficient implementation is based on
Prior knowledge such as list of common delimiter you often work with ',;\t |:' , or even the likely hood of the delimiter to be used so that I often put the regular ',' on the top of the list
The frequency of the delimiter appear in each line of the text file are equal. This is to resolve the problem that if we read a single line and see the frequency to be equal (false detection as Lefteris008) or even the right delimiter to appear less frequent as the wrong one in the first line
The efficient implementation of a head function that read only first n lines from the file
As you increase the number of test sample n, the likely hood that you get a false answer reduce drastically. I often found n=2 to be adequate

Add a condition like this,
String [] country;
if(line.contains(",")
country = line.split(",");
else if(line.contains(";"))
country=line.split(";");

That depends....
If your datasets are always the same length and/or the separator NEVER occurs in your datacolumns, you could just read the first line of the file, look at it for the longed for separator, set it and then read the rest of the file using that separator.
Something like
String csvFile = "/Users/csv/country.csv";
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
// use comma as separator
if (line.contains(",")) {
cvsSplitBy = ",";
} else if (line.contains(";")) {
cvsSplitBy = ";";
} else {
System.out.println("Wrong separator!");
}
String[] country = line.split(cvsSplitBy);
System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
}
} catch (IOException e) {
e.printStackTrace();
}
Greetz Kai

BufferedWriter not writing to my text file

File outFile = new File("D:\\output.txt");
BufferedWriter wb = new BufferedWriter(new FileWriter(outFile));
while (resultSet.next()) {
int attr_id = resultSet.getInt("int_id");
String stringValue = resultSet.getString("StringValue");
String name = resultSet.getString("Name");
int index = stringValue.indexOf(".");
int valueLength = stringValue.length();
if(isNumeric(stringValue)) {
//if(index != -2 ) {
if(index != (valueLength - 2)) {
String string1 = Double.valueOf(stringValue).toString();
System.out.println("converted values : " +string1);
System.out.println("stringValue : " +stringValue);
System.out.println("intValue : " +int_id);
wb.write( stringValue + "," + int_id + "," + string1 );
wb.newLine();
}
}
}
Above is my part of the code, from resultset i'm writing the data into a file. However the code is not printing values in output.txt file but i could see the result in console.
if i remove the commented line and comment if(index != (valueLength - 2)) { this line, the java code is creating output.txt with values.
What's wrong?

probably need to just call flush and/or close on your BufferedWriter when you are done.

The reason why your code is not writing to the file is because you never push the data from the Writer to the actual file.
Think of BufferedWriters as an email you are sending to a coworker. When you use:
wb.write( stringValue + "," + int_id + "," + string1 );
it is like you are typing the email to your coworker, and defining the message to them. BUT, since you did not press "send email", your coworker will never see the message. The same concept can be applied here.
You are filling the writer with a bunch of data, but you are never sending the data to the file that you are trying to write to. There are two ways that you can do this, the first being to "flush" the writer (see documentation HERE). You can do this by calling
wb.flush();
This is the equivalent of pressing "send email", and will write the data to the file that you are editing. The second method would be to "close" the writer (see documenation HERE). The reason that this works is because the BufferedWriter's close method flushes the stream first, before closing the stream. This can be done by calling
wb.close();
Although, it would not be wise to do so until you are fully done editing the file, as it can no longer be accessed.
The following is your code edited to flush the stream after every record you are writing, and then close the stream after all records have been processed. Note the locations of wb.flush() and wb.close().
File outFile = new File("D:\\output.txt");
BufferedWriter wb = new BufferedWriter(new FileWriter(outFile));
while (resultSet.next()) {
int attr_id = resultSet.getInt("int_id");
String stringValue = resultSet.getString("StringValue");
String name = resultSet.getString("Name");
int index = stringValue.indexOf(".");
int valueLength = stringValue.length();
if(isNumeric(stringValue)) {
//if(index != -2 ) {
if(index != (valueLength - 2)) {
String string1 = Double.valueOf(stringValue).toString();
System.out.println("converted values : " +string1);
System.out.println("stringValue : " +stringValue);
System.out.println("intValue : " +int_id);
wb.write( stringValue + "," + int_id + "," + string1 );
wb.newLine();
wb.flush();
}
}
}
wb.close();

Why does a blank line in a text file cause java.util.NoSuchElementException?

If you have a text file:
AAA:123:123:AAAA
BBB:456:456:BBBB
At first when there is no blank line in the text file and you read and retrieve data. Everything is fine.
When you write the file into a new file and replaces the data or update
AAA:9993:9993:AAAA
BBB:456:456:BBBB
-------- This is a blank line-----------
After this happens, NoSuchElementException will pop out. If the blank line is not removed, the error will always pop out.
try {
File fileCI = new File("CI.txt");
FileWriter fileWriter = new FileWriter(fileCI);
BufferedWriter bw = new BufferedWriter(fileWriter);
for (Customer ci : custList){
if (inputUser.equals(ci.getUserName()) && inputPass.equals(ci.getPassword())) {
ci.setCardNo(newCardNo);
ci.setCardType(newCardType);
}
String text = ci.getRealName() + ";" + ci.getUserName() + ";" + ci.getPassword() + ";" + ci.getAddress() + ";" + ci.getContact() + ";" + ci.getcardType() + ";" + ci.getcardNo() + System.getProperty("line.separator");
bw.write(text);
}
bw.close();
fileWriter.close();
}
catch (IOException e) {
e.printStackTrace();
}
if I dont add the System.getProperty("line.separator"); The String will be added and everything will be combined together into a line without a new separator. But this separator adds a blank line at the end of the text file. Is there anything I can do to avoid this problem?
Should I solve at the place where I read the file? Or solve it at the place where I write the file into a new file.
try {
Scanner read = new Scanner(file);
read.useDelimiter(";|\n");
String tmp = "";
while (read.hasNextLine()){
if (read.hasNext()){
custList.add(new Customer(read.next(), read.next(), read.next(), read.next(), read.next(), read.next(), read.next()));
} else {
break;
}
}
read.close();
}
catch (IOException e) {
e.printStackTrace();
}
Edit: This above read works perfectly now!

I think you reach at the end of file(EOF) where there is no remaining line and you still trying to read line . So you getting NoSuchElementException(if no line was found ).
Try this:
String tmp="";
while (reader.hasNextLine()){
tmp = s.nextLine();
// then do something
}
I think you dont have to use \n in delimiter. Since we are using scanner.hasNextLine(). If you want to use scanner.next(). Then
read.useDelimiter(";|\n");
Above line should be:
read.useDelimiter(";|\\n");// use escape character.
And loop this way.
while(s.hasNext()){
//do something.
}

java - Can't get "append mode" on Filewriter to work for me

I've been all over the search and it seems like the given answers just don't work for me.
My code is relatively simple, it generates an array of Objects, populates it with some random strings and then tries to output to a file. The idea is basically to generate a CSV file with some names, login names, passwords, etc. and the names are strings of random letters (long story, it's for mass-populating an environment with users...)
I have a "Writer" class like this:
public class Writer {
public static void log(String message) throws IOException {
PrintWriter out = new PrintWriter(new FileWriter("testlog.txt"), true);
out.println(message);
out.close();
}
}
And a loop like this:
for (int y=0; y < num_names; y++) {
try {
Writer.log(arrayTest[y].first + "," + arrayTest[y].last + "," + arrayTest[y].loginName + "," + arrayTest[y].password +
"," + arrayTest[y].email);
} catch (IOException ex) {
Logger.getLogger(Csvgenerator.class.getName()).log(Level.SEVERE, null, ex);
}
System.out.println(arrayTest[y].first + "," + arrayTest[y].last + "," + arrayTest[y].loginName + "," + arrayTest[y].password +
"," + arrayTest[y].email);
}
My expectation is that I will loop through and for each object in arrayTest[] I output a single line of data to the file.
I included the System.out.println just for debugging.
When I run my code, the System.out.println proves that it works properly -- I get a list of 10 rows. (num_names = 10 here) so this proves that each time I get to this line of code, I have a unique "row" of data which gets printed out.
However, at the end of the run, the file "testlog.txt" only contains a single line -- the very last row in my output.
I've tried "out.append" instead of "out.println" but no difference. It appears as though every time I call the logger it's creating the file anew for some reason.
So in other words if my console output (from the system.out.println) looks like this:
nxoayISPaX,aNQWbAjvWE,nanqwbajvwe,P#ssw0rd!,nanqwbajvwe#mylab.com
RpZDZAovgv,QOfyNRtIAN,rqofynrtian,P#ssw0rd!,rqofynrtian#mylab.com
SajEwHhfZz,VziPeyXmAc,svzipeyxmac,P#ssw0rd!,svzipeyxmac#mylab.com
sifahXTtBx,MRmewORtGZ,smrmewortgz,P#ssw0rd!,smrmewortgz#mylab.com
PlepqHzAxE,MQUJsHgEgy,pmqujshgegy,P#ssw0rd!,pmqujshgegy#mylab.com
VKYjYGLCfV,nuRKBJUuxW,vnurkbjuuxw,P#ssw0rd!,vnurkbjuuxw#mylab.com
YgvgeWmomA,ysKLVSZvaI,yysklvszvai,P#ssw0rd!,yysklvszvai#mylab.com
feglvfOBUX,UTIPxdEriq,futipxderiq,P#ssw0rd!,futipxderiq#mylab.com
RAQPPNajxR,vzdIwzFHJY,rvzdiwzfhjy,P#ssw0rd!,rvzdiwzfhjy#mylab.com
DeXgVFClyg,IEuUuvdWph,dieuuuvdwph,P#ssw0rd!,dieuuuvdwph#mylab.com
Then testlog.txt only contains a single line:
DeXgVFClyg,IEuUuvdWph,dieuuuvdwph,P#ssw0rd!,dieuuuvdwph#mylab.com
How do I force this to keep using the same file and just append new lines?

On the constructor PrintWriter(Writer out, boolean autoFlush), the second boolean argument is actually for autoflush, not append mode.
I think you intended to use FileWriter(File file, boolean append) constructor instead, ie:
PrintWriter out = new PrintWriter(new FileWriter("testlog.txt", true));
instead of
PrintWriter out = new PrintWriter(new FileWriter("testlog.txt"), true);

Reading multiple records from a flat file in Java

I have a text file dump that I need to convert to a delimited file. The file contains a series of "records" (for lack of a better word) formatted like this:
User: abc123
Date: 7/3/12
Subject: the foo is bar
Project: 123456
Problem: foo bar in multiple lines of text
Resolution: foo un-barred in multiple lines of text
User: abc123
Date: 7/3/12
Subject: the foo is bar
Project: 234567
Problem: foo bar in multiple lines of text
Resolution: foo un-barred in multiple lines of text
...
My end result is to get a flat file of delimited values. Using the records above, we would see:
abc123;7/3/12;the foo is bar;123456;foo bar in multiple lines of text;foo un-barred in multiple lines of text
abc123;7/3/12;the foo is bar;234567;foo bar in multiple lines of text;foo un-barred in multiple lines of text
Code appears below, and following that, the problem I'm experiencing.
import java.util.*;
import java.io.*;
import java.nio.file.*;
//
public class ParseOutlookFolderForSE
{
public static void main(String args[])
{
String user = "";
String PDLDate = "";
String name = "";
String PDLNum = "";
String problemDesc = "test";
String resolutionDesc = "test";
String delim = ";";
int recordCounter = 0;
//
try
{
Path file = Paths.get("testfile2.txt");
FileInputStream fstream = new FileInputStream("testfile2.txt");
// Get the object of DataInputStream
/* DataInputStream in = new DataInputStream(fstream); */
BufferedReader br = new BufferedReader(new InputStreamReader(fstream)); //Buffered Reader
String inputLine = null; //String
StringBuffer theText = new StringBuffer(); //StringBuffer
// problem: output contains last record ONLY. program is cycling through the entire file, overwriting records until the end.
// add a for loop based on recordCounter
for(recordCounter=0;recordCounter<10;recordCounter++)
{
while((inputLine=br.readLine())!=null)
{
if(inputLine.toLowerCase().startsWith("from:"))
{
/* recordCounter = recordCounter++; */ // commented out when I added recordCounter++ to the for loop
user = inputLine.trim().substring(5).trim();
}
else
if(inputLine.toLowerCase().startsWith("effective date"))
{
PDLDate = inputLine.trim().substring(15).trim();
}
else
if(inputLine.toLowerCase().startsWith("to:"))
{
name = inputLine.trim().substring(3).trim();
}
else
if(inputLine.toLowerCase().startsWith("sir number"))
{
PDLNum = inputLine.trim().substring(12).trim();
}
} //close for loop
} // close while
System.out.println(recordCounter + "\n" + user + "\n" + name + "\n" + PDLNum + "\n" + PDLDate + "\n" + problemDesc + "\n" + resolutionDesc);
System.out.println(recordCounter + ";" + user + ";" + name + ";" + PDLNum + ";" + PDLDate + ";" + problemDesc + ";" + resolutionDesc);
String lineForFile = (recordCounter + ";" + user + ";" + name + ";" + PDLNum + ";" + PDLDate + ";" + problemDesc + ";" + resolutionDesc + System.getProperty("line.separator"));
System.out.println(lineForFile);
try
{
BufferedWriter out = new BufferedWriter(new FileWriter("testfileoutput.txt"));
out.write(lineForFile);
out.close();
}
catch (IOException e)
{
System.out.println("Exception ");
}
} //close try
catch (Exception e)
{
System.err.println("Error: " + e.getMessage());
}
}
}
My final output is ONLY the last record. I believe that what's happening is that the program is reading every line, but only the LAST one doesn't get overwritten with the next record. Makes sense. So I added a FOR loop, incrementing by 1 if(inputLine.toLowerCase().startsWith("user:")) and outputting the counter variable with my data to validate what's happening.
My FOR loop begins after step 3 in my pseudocode...after BufferedReader but before my IF statements. I terminate it after I write to the file in step 6. I'm using for(recCounter=0;recCounter<10;recCounter++) and while I get ten records in my output file, they are all instances of the LAST record of the input file, numbered 0-9.
Leaving the for loop in the same place, I modified it to read for(recCounter=0;recCounter<10;) and placed recCounter's increment WITHIN the IF statement, incrementing every time the line starts with User:. In this case, I also got ten records in my output file, they were ten instances of the last record in the input file, and all the counters are 0.
EDIT: Given how the file is formatted, the ONLY way to determine w=one record from the next is a subsequent instance of the word "User:" at the start of the line. Each time that occurs, until the NEXT time it occurs is what constitutes a single record.
It appears as though I'm not setting my "recCounter" appropriately, or I'm not interpreting the results of what IS being set as "start a new record".
Anyone have any suggestions for how to read this file as multiple records?

Okay, so your pseudo-code should go something like this:
declare variables
open file
while not eof
read input
if end of set
format output
write output
clear variables
figure out which variable
store in correct variable
end-while
There might be a trick to figuring out when you've finished one set and can start the next. If a set is supposed to be terminated by a blank line as appears from your example, then you could just check for the blank line. Otherwise, how do you know? Does a set always start with "user"?
Also, don't forget to write the last record. You don't want to leave unwritten stuff in your buffer/table.

From your description it sounds like the following is the case: you are actually not writing the output strings as you complete them, but instead doing all of the writing at the end. It does not sound like you are saving the output strings outside of the loop, and so each time you find a record, you are overwriting the output string you previously calculated.
You should test that you are actually writing to the file after each record is found and has its output string created.
Without posting your code, I am not sure I can help you much further.