Develop Reference

How to determine the delimiter in CSV file

I have a scenario at which i have to parse CSV files from different sources, the parsing code is very simple and straightforward.
String csvFile = "/Users/csv/country.csv";
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
// use comma as separator
String[] country = line.split(cvsSplitBy);
System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
}
} catch (IOException e) {
e.printStackTrace();
}
my problem come from the CSV delimiter character, i have many different formats, some time it is a , sometimes it is a ;
is there is any way to determine the delimiter character before parsing the file

univocity-parsers supports automatic detection of the delimiter (also line endings and quotes). Just use it instead of fighting with your code:
CsvParserSettings settings = new CsvParserSettings();
settings.detectFormatAutomatically();
CsvParser parser = new CsvParser(settings);
List<String[]> rows = parser.parseAll(new File("/path/to/your.csv"));
// if you want to see what it detected
CsvFormat format = parser.getDetectedFormat();
Disclaimer: I'm the author of this library and I made sure all sorts of corner cases are covered. It's open source and free (Apache 2.0 license)
Hope this helps.

Yes, but only if the delimiter characters are not allowed to exist as regular text
The most simple answer is to have a list with all the available delimiter characters and try to identify which character is being used. Even though, you have to place some limitations on the files or the person/people that created them. Look a the following two scenarios:
Case 1 - Contents of file.csv
test,test2,test3
Case 2 - Contents of file.csv
test1|test2,3|test4
If you have prior knowledge of the delimiter characters, then you would split the first string using , and the second one using |, getting the same result. But, if you try to identify the delimiter by parsing the file, both strings can be split using the , character, and you would end up with this:
Case 1 - Result of split using ,
test1
test2
test3
Case 2 - Result of split using ,
test1|test2
3|test4
By lacking the prior knowledge of which delimiter character is being used, you cannot create a "magical" algorithm that will parse every combination of text; even regular expressions or counting the number of appearance of a character will not save you.
Worst case
test1,2|test3,4|test5
By looking the text, one can tokenize it by using | as the delimiter. But the frequency of appearance of both , and | are the same. So, from an algorithm's perspective, both results are accurate:
Correct result
test1,2
test3,4
test5
Wrong result
test1
2|test3
4|test5
If you pose a set of guidelines or you can somehow control the generation of the CSV files, then you could just try to find the delimiter used with String.contains() method, employing the aforementioned list of characters. For example:
public class MyClass {
private List<String> delimiterList = new ArrayList<>(){{
add(",");
add(";");
add("\t");
// etc...
}};
private static String determineDelimiter(String text) {
for (String delimiter : delimiterList) {
if(text.contains(delimiter)) {
return delimiter;
}
}
return "";
}
public static void main(String[] args) {
String csvFile = "/Users/csv/country.csv";
String line = "";
String cvsSplitBy = ",";
String delimiter = "";
boolean firstLine = true;
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
if(firstLine) {
delimiter = determineDelimiter(line);
if(delimiter.equalsIgnoreCase("")) {
System.out.println("Unsupported delimiter found: " + delimiter);
return;
}
firstLine = false;
}
// use comma as separator
String[] country = line.split(delimiter);
System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Update
For a more optimized way, in determineDelimiter() method instead of the for-each loop, you can employ regular expressions.

If the delimiter can appear in a data column, then you are asking for the impossible. For example, consider this first line of a CSV file:
one,two:three
This could be either a comma-separated or a colon-separated file. You can't tell which type it is.
If you can guarantee that the first line has all its columns surrounded by quotes, for example if it's always this format:
"one","two","three"
then you may be able to use this logic (although it's not 100% bullet-proof):
if (line.contains("\",\""))
delimiter = ',';
else if (line.contains("\";\""))
delimiter = ';';
If you can't guarantee a restricted format like that, then it would be better to pass the delimiter character as a parameter.
Then you can read the file using a widely-known open-source CSV parser such as Apache Commons CSV.

While I agree with Lefteris008 that it is not possible to have the function that correctly determine all the cases, we can have a function that is both efficient and give mostly correct result in practice.
def head(filename: str, n: int):
try:
with open(filename) as f:
head_lines = [next(f).rstrip() for x in range(n)]
except StopIteration:
with open(filename) as f:
head_lines = f.read().splitlines()
return head_lines
def detect_delimiter(filename: str, n=2):
sample_lines = head(filename, n)
common_delimiters= [',',';','\t',' ','|',':']
for d in common_delimiters:
ref = sample_lines[0].count(d)
if ref > 0:
if all([ ref == sample_lines[i].count(d) for i in range(1,n)]):
return d
return ','
My efficient implementation is based on
Prior knowledge such as list of common delimiter you often work with ',;\t |:' , or even the likely hood of the delimiter to be used so that I often put the regular ',' on the top of the list
The frequency of the delimiter appear in each line of the text file are equal. This is to resolve the problem that if we read a single line and see the frequency to be equal (false detection as Lefteris008) or even the right delimiter to appear less frequent as the wrong one in the first line
The efficient implementation of a head function that read only first n lines from the file
As you increase the number of test sample n, the likely hood that you get a false answer reduce drastically. I often found n=2 to be adequate

Add a condition like this,
String [] country;
if(line.contains(",")
country = line.split(",");
else if(line.contains(";"))
country=line.split(";");

That depends....
If your datasets are always the same length and/or the separator NEVER occurs in your datacolumns, you could just read the first line of the file, look at it for the longed for separator, set it and then read the rest of the file using that separator.
Something like
String csvFile = "/Users/csv/country.csv";
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
while ((line = br.readLine()) != null) {
// use comma as separator
if (line.contains(",")) {
cvsSplitBy = ",";
} else if (line.contains(";")) {
cvsSplitBy = ";";
} else {
System.out.println("Wrong separator!");
}
String[] country = line.split(cvsSplitBy);
System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
}
} catch (IOException e) {
e.printStackTrace();
}
Greetz Kai

Java compare strings from two places and exclude any matches

I'm trying to end up with a results.txt minus any matching items, having successfully compared some string inputs against another .txt file. Been staring at this code for way too long and I can't figure out why it isn't working. New to coding so would appreciate it if I could be steered in the right direction! Maybe I need a different approach? Apologies in advance for any loud tutting noises you may make. Using Java8.
//Sending a String[] into 'searchFile', contains around 8 small strings.
//Example of input: String[]{"name1","name2","name 3", "name 4.zip"}
^ This is my exclusions list.
public static void searchFile(String[] arr, String separator)
{
StringBuilder b = new StringBuilder();
for(int i = 0; i < arr.length; i++)
{
if(i != 0) b.append(separator);
b.append(arr[i]);
String findME = arr[i];
searchInfo(MyApp.getOptionsDir()+File.separator+"file-to-search.txt",findME);
}
}
^This works fine. I'm then sending the results to 'searchInfo' and trying to match and remove any duplicate (complete, not part) strings. This is where I am currently failing. Code runs but doesn't produce my desired output. It often finds part strings rather than complete ones. I think the 'results.txt' file is being overwritten each time...but I'm not sure tbh!
file-to-search.txt contains: "name2","name.zip","name 3.zip","name 4.zip" (text file is just a single line)
public static String searchInfo(String fileName, String findME)
{
StringBuffer sb = new StringBuffer();
try {
BufferedReader br = new BufferedReader(new FileReader(fileName));
String line = null;
while((line = br.readLine()) != null)
{
if(line.startsWith("\""+findME+"\""))
{
sb.append(line);
//tried various replace options with no joy
line = line.replaceFirst(findME+"?,", "");
//then goes off with results to create a txt file
FileHandling.createFile("results.txt",line);
}
}
} catch (Exception e) {
e.printStackTrace();
}
return sb.toString();
}
What i'm trying to end up with is a result file MINUS any matching complete strings (not part strings):
e.g. results.txt to end up with: "name.zip","name 3.zip"

ok with the information I have. What you can do is this
List<String> result = new ArrayList<>();
String content = FileUtils.readFileToString(file, "UTF-8");
for (String s : content.split(", ")) {
if (!s.equals(findME)) { // assuming both have string quotes added already
result.add(s);
}
}
FileUtils.write(newFile, String.join(", ", result), "UTF-8");
using apache commons file utils for ease. You may add or remove spaces after comma as per your need.

Java - take name from string

I'm developing a Java application that make some statistic stuff.
This application take all data from a .txt file which is supplied by the user.
The first line of that file contains the name of the sets of data that follows like this:
velx,vely,velz
//various data
I need to analyze that first line and retrieve the three name of variables, I correctly get the first two but I'm not able to get the last one.
There the code to get names:
public ArrayList<String> getTitle(){
// the ArrayList originally is not here but in the class intestation
// I copied it here to simplify code's understanding
ArrayList<String> title = new ArrayList<String>();
try {
InputStreamReader isr = new InputStreamReader(in);
BufferedReader br = new BufferedReader(isr);
StringBuilder sb = new StringBuilder();
int titleN = 0;
String line = br.readLine(); //read the first line of file
String temp;
System.out.println(ManageTable.class.getName() + " Line: " + line);
int c = line.length();
for(int i = 0; i <c; i++){
if((line.charAt(i) == ',') || **ANOTHER CONDITION** ){
temp = sb.toString();
System.out.println(ManageTable.class.getName() +" Temp is: " + temp);
title.add(temp);
System.out.println(ManageTable.class.getName() + " Title added");
sb.delete(0, sb.length());
}else{
sb.append(line.charAt(i));
}
}
} catch (IOException ex) {
Logger.getLogger(ManageTable.class.getName()).log(Level.SEVERE, null, ex);
}
return title;
}
I need to add a second condition to the if statement in order to find out when the line is ended and save the last name, even if its not followed by ','
I tried using:
if((line.charAt(i) == ',') || (i==c))
but from the name I get, always miss a character.
How can I check the end of the line and so get the full name?

If line contains just three names separated by comma, you can do
String[] names = line.split(",");

No need for all this looping. You can just split the line around the comma to get an array:
String[] names = line.split(",");

How to trim the elements before assigning it into an array list?

I need to assign the elements present in a CSV file into an arraylist. CSV file contains filenames with extension .tar. I need to trim those elements before i read it into an array list or trim the whole arraylist. Please help me with it
try
{
String strFile1 = "D:\\Ramakanth\\PT2573\\target.csv"; //csv file containing data
BufferedReader br1 = new BufferedReader( new FileReader(strFile1)); //create BufferedReader
String strLine1 = "";
StringTokenizer st1 = null;
while( (strLine1 = br1.readLine()) != null) //read comma separated file line by line
{
st1 = new StringTokenizer(strLine1, ","); //break comma separated line using ","
while(st1.hasMoreTokens())
{
array1.add(st1.nextToken()); //store csv values in array
}
}
}
catch(Exception e)
{
System.out.println("Exception while reading csv file: " + e);
}

If you want to remove the ".tar" string from your tokens, you can use:
String nextToken = st1.nextToken();
if (nextToken.endsWith(".tar")) {
nextToken = nextToken.replace(".tar", "");
}
array1.add(nextToken);

You shouldn't be using StringTokenizer the JavaDoc says (in part) StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead. You should close your BufferedReader. You could use a try-with-resources statement to do that. And, you might use a for-each loop to iterate the array produced by String.split(String) the regular expression below optionally matches whitespace before or after your , and you might continue the loop if the token endsWith ".tar" like
String strFile1 = "D:\\Ramakanth\\PT2573\\target.csv";
try (BufferedReader br1 = new BufferedReader(new FileReader(strFile1)))
{
String strLine1 = "";
while( (strLine1 = br1.readLine()) != null) {
String[] parts = strLine1.split("\\s*,\\s*");
for (String token : parts) {
if (token.endsWith(".tar")) continue; // <-- don't add "tar" files.
array1.add(token);
}
}
}
catch(Exception e)
{
System.out.println("Exception while reading csv file: " + e);
}

if(str.indexOf(".tar") >0)
str = str.subString(0, str.indexOf(".tar")-1);

while(st1.hasMoreTokens())
{
String input = st1.nextToken();
int index = input.indexOf("."); // Get the position of '.'
if(index >= 0){ // To avoid StringIndexOutOfBoundsException, when there is no match with '.' then the index position set to -1.
array1.add(input.substring(0, index)); // Get the String before '.' position.
}
}

Splitting a String without spaces

I have the following string which is generated by an external program (OpenVAS) and returned to my program successfully as a string.
<create_target_response id="b4c8de55-94d8-4e08-b20e-955f97a714f1" status_text="OK, resource created" status="201"></create_target_response>
I am trying to split the string to give me the "b4c8d....14f1" without the inverted commas. I have tried all sorts of escape methods and keep getting the else method "String does not contain a Target ID". I have tried removing the IF statement checking for the string, but continue to have the same issue. The goal is to get my id string into jTextField6. String Lob contains the full string as above.
if (Lob.contains("id=\"")){
// put the split here
String[] parts = Lob.split("id=\"");
String cut1 = parts[1];
String[] part2 = cut1.split("\"");
String TaskFinal = part2[0];
jTextField6.setText(TaskFinal);
}
else {
throw new IllegalArgumentException("String does not contain a Target ID");
}
} catch (IOException e) {
e.printStackTrace();
}
It seems I only need to escape the " and not the = (Java kicks up an error if i do)
Thanks in advance
EDIT: Code as it stands now using jSoup lib - The 'id' string won't display. Any ideas?
Thanks
private void jButton1ActionPerformed(java.awt.event.ActionEvent evt) {
// TODO add your handling code here:
String TargIP = jTextField1.getText(); // Get IP Address
String TargName = jTextField5.getText(); // Get Target Name
String Vag = "8d32ad99-ac84-4fdc-b196-2b379f861def";
String Lob = "";
final String dosCommand = "cmd /c omp -u admin -w admin --xml=\"<create_target><name>" + TargName + "</name><hosts>" + TargIP + "</hosts></create_target>\"";
3</comment><config id='daba56c8-73ec-11df-a475-002264764cea'/><target id='" + Vag + "'/></create_task>\"";
final String location = "C:\\";
try {
final Process process = Runtime.getRuntime().exec(
dosCommand + " " + location);
final InputStream in = process.getInputStream();
int ch;
while((ch = in.read()) != -1) {
System.out.print((char)ch);
Lob = String.valueOf((char)ch);
jTextArea2.append(Lob);
}
} catch (IOException e) {
e.printStackTrace();
}
String id = Jsoup.parse(Lob).getAllElements().attr("id");
System.out.println(id); // This doesn't output?
}

Split on the basis of ". You can get all the key values.
String str = "<create_target_response id=\"b4c8de55-94d8-4e08-b20e-955f97a714f1\" status_text=\"OK, resource created\" status=\"201\"></create_target_response>";
String[] tokens = str.split("\\\"");
System.out.println(tokens[1]);
System.out.println(tokens[5]);
output:
b4c8de55-94d8-4e08-b20e-955f97a714f1
201

This will get you your job id more easily:
int idStart = Lob.indexOf("id=")+("id=\"").length();
System.out.println(Lob.substring(idStart,Lob.indexOf("\"",idStart)));

Everyone's telling you to use an XML parser (and they're right) but noone's showing you how.
Here goes:
String lob = ...
Using Jsoup from http://jsoup.org, actually an HTML parser but also handles XML neatly:
String id = Jsoup.parse(lob).getAllElements().attr("id");
// b4c8de55-94d8-4e08-b20e-955f97a714f1
With built-in Java XML APIs, less concise but no addtional libraries:
Document dom = DocumentBuilderFactory.newInstance().newDocumentBuilder()
.parse(new InputSource(new StringReader(lob)));
String id = dom.getDocumentElement().getAttribute("id");
// b4c8de55-94d8-4e08-b20e-955f97a714f1

This is a lot simpler than you're making it, to my mind. First, split on space, then check if an = is present. If it is, split on the =, and finally remove the " from the second token.
The tricky bit is the spaces inside of the "". This will require some regular expressions, which you can work out from this question.
Example
String input; // Assume this contains the whole string.
String pattern; // Have fun working out the regex.
String[] values = input.split(pattern);
for(String value : values)
{
if(value.contains("=")) {
String[] pair = value.split("=");
String key = pair[0];
String value = pair[1].replaceAll("\"");
// Do something with the values.
}
}
Advantage of my approach
Is that provided the input follows the format of key="value" key="value", you can parse anything that comes through, rather than hard coding the name of the attributes.
And if this is XML..
Then use an XML parser. There is a good (awesome) answer that explains why you shouldn't be using Stringmanipulation to parse XML/HTML. Here is the answer.

You can use a regex to extract what is needed; what is more, it looks like the value of id is a UUID. Therefore:
private static final Pattern PATTERN
= Pattern.compile("\\bid=\"([^\"]+)\"");
// In code...
public String getId(final String input)
{
final Matcher m = PATTERN.matcher(input);
if (!m.find())
throw new IllegalArgumentException("String does not contain a Target ID");
final String uuid = m.group(1);
try {
UUID.fromString(uuid);
} catch (IllegalArgumentException ignored) {
throw new IllegalArgumentException("String does not contain a Target ID");
}
return uuid;
}