Develop Reference

Parsing csv data into object with different column length

I am new to Java and practicing parsing csv file into the object. I've tried but cannot figure it out.
The file looks like this:
[0], [1], [2], [3] , [4] , [5] , [6] , [7] , [8] , [9]
class, gender, age, bodyType, profession, pregnant, isYou ,species, isPet, role
scenario:green, , , , , , , ,
person, female, 24, average , , FALSE , , , , passenger
animal, male , 4, , , FALSE , , dog , TRUE , pedestrian
scenario:red
person, male , 16, athletic, boxer , FALSE , TRUE , , , passenger
person, female, 25, athletic, doctor , TRUE , FALSE , , , pedestrian
I need to parse it by any number of passengers and pedestrians with any scenarios. Finally, add these scenarios into an ArrayList for analyzing.
What I think is to:
loop through each line, stops when reaches to the next scenario:red, adds the passengers and the pedestrians to the Character ArrayList. (I've done adding, but don't how to stop).
Create a scenario using constructor scenario(ArrayList<Character> passenger, ArrayList<Character> pedestrians, boolean redOrGreen);
The ArrayList scenarios add the created scenarios.
What I've done is put everything together instead of separate them. Any help or hint is highly appreciated.
Thanks for this community who helped me, here is what I've got so far.
public void loadCsv() throws IOException {
String csvFile = "config.csv";
String line = "";
String csvSplit = "\\s*,\\s*";
Scenario scenario = new Scenario();
Person person = new Person();
Animal animal = new Animal();
ArrayList<Scenario> scenaios = new ArrayList<Scenario>();
ArrayList<String> csvContents = new ArrayList<String>();
ArrayList<Character> passengers = new ArrayList<Character>();
ArrayList<Character> pedestrians = new ArrayList<Character>();
try (BufferedReader csvReader = new BufferedReader(new FileReader(csvFile));) {
String headerLine = csvReader.readLine(); //get rid of the header
//add each line to the arrayList
while ((line = csvReader.readLine()) != null) {
csvContents.add(line);
}
for(String csvLine : csvContents) {
String[] data = csvLine.split(csvSplit); // split by comma and remove redundant spaces
if (data.length == NO_OF_FIELD) { //check and avoid indexOutOfBoundException
String clazz = data[0].toLowerCase();// cannot use word "class" as a variable
if (clazz.startsWith("scenario") && data.length == 1) {
scenario = new Scenario();
scenario.setLegalCrossing(clazz.endsWith("green"));
continue;
}
else if ("person".equals(clazz) && data.length ==10) {
person = loadCsvPerson(data);
addCharacter(person, data);
}
else if ("animal".equals(clazz) && data.length ==10) {
animal = loadCsvAnimal(data);
addCharacter(animal, data);
}
}
}
}
//passenger and pedestrians are in position
System.out.println("passengers: " + passengers);
System.out.println("pedestrians: " + pedestrians);
if (null != scenario) {
scenario.setPassengers(passengers);
scenario.setPedestrians(pedestrians);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}

If it is possible to change the csv file format, I would add scenario type column (and scenario id or name if required), so you can work with csv file as a result set from database when you join tables (scenario + passenger + pedestrian) and return plain rows.
With this approach you will be able to delegate parsing to any csv library and do your logic (group by scenario id/name/type) separately. With surrogate rows you have (scenario:green...) you have to write your custom parser.
For example, you can use univocity to simply parse file into your model (even using annotations) and iteratively group it and handle.

Or if you need to work with existing file format do something like that:
if (clazz.startsWith("scenario") && data.length == 1) {
// collect existing scenario before starting processing new one
if (scenario != null) {
scenario.setPassengers(passengers);
scenario.setPedestrians(pedestrians);
passengers = new ArrayList();
pedestrians = new ArrayList();
scenarios.add(scenario);
}
// now start new group (scenario)
scenario = new Scenario();
scenario.setLegalCrossing(clazz.endsWith("green"));
continue;
}

Following things need to be addressed in your code:
Strive to avoid using the name of a class which is already used by the standard library (and especially when it is in the default package, java.lang) e.g. there is already a class Character in Java library and therefore you should use a different name for your custom class.
Use continue to skip the line, scenario:red
for(String csvLine : csvContents) {
if(csvLine.equals("scenario:red")){
continue;
}
String[] data = csvLine.split(csvSplit); // split by comma and remove redundant spaces
if (data.length == NO_OF_FIELD) {
//..
}
//..
}
If you have already defined final int NO_OF_FIELD = 10, you can use the same instead of using the value 10 directly i.e. you should use NO_OF_FIELD instead of 10 in the following code:
if (data.length == NO_OF_FIELD) { //check and avoid indexOutOfBoundException
String clazz = data[0].toLowerCase();// cannot use word "class" as a variable
//...
else if ("person".equals(clazz) && data.length ==10) {
However, you also need to understand that && data.length ==10 is unnecessary here as you have already checked data.length == NO_OF_FIELD in the enclosing if condition.
I couldn't understand the rest of your points. If you clarify them, I'll be able to help you further.

I need to add the previous scenario in the second round.
Since the last set of data won't be captured, I need to set another new scenario to add it in. Thanks for the art sir.
Character character = null;
try (BufferedReader csvReader = new BufferedReader(new FileReader(csvFile));) {
String headerLine = csvReader.readLine(); //get rid of the header
//add each line to the arrayList
while ((line = csvReader.readLine()) != null) {
csvContents.add(line);
}
final int NO_OF_FIELDS = 10;
for(String csvLine : csvContents) {
String[] data = csvLine.split(csvSplit); // split by comma and remove redundant spaces
String clazz = data[0].toLowerCase();// cannot use word "class" as a variable
if (clazz.startsWith("scenario") && data.length == 1) {
// adding scenario after one set of data
// i.e second round adding the first round data
if (passengers.size() != 0 && pedestrians.size() != 0) {
Scenario scenario = new Scenario();
scenario.setPassengers(passengers);
scenario.setPedestrians(pedestrians);
scenarios.add(scenario);
}
passengers = new ArrayList<Character>();
pedestrians = new ArrayList<Character>();
if (clazz.endsWith("green")) {
scenario.setLegalCrossing(true);
System.out.println("green light");
}
else if (clazz.endsWith("red")){
scenario.setLegalCrossing(false);
System.out.println("red light");
}
continue;
}
//...
Scenario scenario = new Scenario();
scenario.setPassengers(passengers);
scenario.setPedestrians(pedestrians);
scenarios.add(scenario);
scenario.setPassengers(passengers);
scenario.setPedestrians(pedestrians);
Audit audit = new Audit();
audit.setScenario(scenarios);

Modifying YAML in java while preserving comments

How can we modify an existing YAML and preserve comments in it.
Is there any Java parser which does that ?
For example if i have following YAML:
#This is a test YAML
name: abcd
age: 23
#Test YAML ends here.
Is there a way I can edit this Yaml using a java parser and preserve the comments.

As of the time of writing, there is no round-tripping YAML parser for Java. There is the well-known SnakeYAML, which does not preserve comments (see the author's comment here), and a newer project named camel, which I know little of; but it definitely is not round-tripping.
What you can theoretically do is to use SnakeYaml's Yaml.parse and then iterate over the events. Each event has a start and an end mark, giving the start and end line & column of the event. This makes it possible to map the events back into the source and discover the portions of the source that were not parsed into events (presumably comments). Having this mapping, you can now modify the event list and write it back. Finally, you read the result in a second time and discover the gaps between your events where there were comments in the original YAML, but not in the modified YAML, and re-insert those comments, giving you the final YAML with your modifications and the comments.
Of course, this is very complex. I would not advice to do it unless you a) have either a solid understanding of how YAML is structured or are willing to learn it, and b) your use-case justifies this amount of work.

I wrote a groovy script to solve this. The Java version is very similar:
def key = "name"
def value = "efgh"
def yamlFile = new File("file.yaml")
def yamlFileLines = new StringBuilder()
def foundKey = false
yamlFile.text.eachLine { line ->
if (!foundKey && line.contains("$key:")) {
line = line.replaceAll(/$key:.*/, "$key: $value")
foundKey = true
}
yamlFileLines.append("$line\n")
}
if (foundKey) {
yamlFile.text = yamlFileLines.toString()
} else {
throw new StopExecutionException("Could not find key '$key' in file ${yamlFile.getAbsolutePath()}")
}

if you use snakeyaml , you should modify the ScannerImpl file.
notice: read the in-line comment as text
private Token scanPlain() {
StringBuilder chunks = new StringBuilder();
Mark startMark = reader.getMark();
Mark endMark = startMark;
int indent = this.indent + 1;
String spaces = "";
while (true) {
int c;
int length = 0;
// A comment indicates the end of the scalar.
// read the in-line comment as text
// if (reader.peek() == '#' && ) {
// break;
// }
while (true) {
c = reader.peek(length);
if (Constant.NULL_BL_T_LINEBR.has(c)
|| (c == ':' && Constant.NULL_BL_T_LINEBR.has(reader.peek(length + 1), flowLevel != 0 ? ",[]{}":""))
|| (this.flowLevel != 0 && ",?[]{}".indexOf(c) != -1)) {
break;
}
length++;
}
if (length == 0) {
break;
}
this.allowSimpleKey = false;
chunks.append(spaces);
chunks.append(reader.prefixForward(length));
endMark = reader.getMark();
spaces = scanPlainSpaces();
// System.out.printf("spaces[%s]\n", spaces);
if (spaces.length() == 0
// read the in-line comment as text
// || reader.peek() == '#'
|| (this.flowLevel == 0 && this.reader.getColumn() < indent)) {
break;
}
}
return new ScalarToken(chunks.toString(), startMark, endMark, true);
}

Java - Parse delimited file and find column datatypes

Is it possible to parse a delimited file and find column datatypes? e.g
Delimited file:
Email,FirstName,DOB,Age,CreateDate
test#test1.com,Test User1,20/01/2001,24,23/02/2015 14:06:45
test#test2.com,Test User2,14/02/2001,24,23/02/2015 14:06:45
test#test3.com,Test User3,15/01/2001,24,23/02/2015 14:06:45
test#test4.com,Test User4,23/05/2001,24,23/02/2015 14:06:45
Output:
Email datatype: email
FirstName datatype: Text
DOB datatype: date
Age datatype: int
CreateDate datatype: Timestamp
The purpose of this is to read a delimited file and construct a table creation query on the fly and insert data into that table.
I tried using apache validator, I believe we need to parse the complete file in order to determine each column data type.
EDIT: The code that I've tried:
CSVReader csvReader = new CSVReader(new FileReader(fileName),',');
String[] row = null;
int[] colLength=(int[]) null;
int colCount = 0;
String[] colDataType = null;
String[] colHeaders = null;
String[] header = csvReader.readNext();
if (header != null) {
colCount = header.length;
}
colLength = new int[colCount];
colDataType = new String[colCount];
colHeaders = new String[colCount];
for (int i=0;i<colCount;i++){
colHeaders[i]=header[i];
}
int templength=0;
String tempType = null;
IntegerValidator intValidator = new IntegerValidator();
DateValidator dateValidator = new DateValidator();
TimeValidator timeValidator = new TimeValidator();
while((row = csvReader.readNext()) != null) {
for(int i=0;i<colCount;i++) {
templength = row[i].length();
colLength[i] = templength > colLength[i] ? templength : colLength[i];
if(colHeaders[i].equalsIgnoreCase("email")){
logger.info("Col "+i+" is Email");
} else if(intValidator.isValid(row[i])){
tempType="Integer";
logger.info("Col "+i+" is Integer");
} else if(timeValidator.isValid(row[i])){
tempType="Time";
logger.info("Col "+i+" is Time");
} else if(dateValidator.isValid(row[i])){
tempType="Date";
logger.info("Col "+i+" is Date");
} else {
tempType="Text";
logger.info("Col "+i+" is Text");
}
logger.info(row[i].length()+"");
}
Not sure if this is the best way of doing this, any pointers in the right direction would be of help

If you wish to write this yourself rather than use a third party library then probably the easiest mechanism is to define a regular expression for each data type and then check if all fields satisfy it. Here's some sample code to get you started (using Java 8).
public enum DataType {
DATETIME("dd/dd/dddd dd:dd:dd"),
DATE("dd/dd/dddd",
EMAIL("\\w+#\\w+"),
TEXT(".*");
private final Predicate<String> tester;
DateType(String regexp) {
tester = Pattern.compile(regexp).asPredicate();
}
public static Optional<DataType> getTypeOfField(String[] fieldValues) {
return Arrays.stream(values())
.filter(dt -> Arrays.stream(fieldValues).allMatch(dt.tester)
.findFirst();
}
}
Note that this relies on the order of the enum values (e.g. testing for datetime before date).

Yes it is possible and you do have to parse the entire file first. Have a set of rules for each data type. Iterate over every row in the column. Start of with every column having every data type and cancel of data types if a row in that column violates a rule of that data type. After iterating the column check what data type is left for the column. Eg. Lets say we have two data types integer and text... rules for integer... well it must only contain numbers 0-9 and may begin with '-'. Text can be anything.
Our column:
345
-1ab
123
The integer data type would be removed by the second row so it would be text. If row two was just -1 then you would be left with integer and text so it would be integer because text would never be removed as our rule says text can be anything... you dont have to check for text basically if you left with no other data type the answer is text. Hope this answers your question

I have slight similar kind of logic needed for my project. Searched lot but did not get right solution. For me i need to pass string object to the method that should return datatype of the obj. finally i found post from #sprinter, it looks similar to my logic but i need to pass string instead of string array.
Modified the code for my need and posted below.
public enum DataType {
DATE("dd/dd/dddd"),
EMAIL("#gmail"),
NUMBER("[0-9]+"),
STRING("^[A-Za-z0-9? ,_-]+$");
private final String regEx;
public String getRegEx() {
return regEx;
}
DataType(String regEx) {
this.regEx = regEx;
}
public static Optional<DataType> getTypeOfField(String str) {
return Arrays.stream(DataType.values())
.filter(dt -> {
return Pattern.compile(dt.getRegEx()).matcher(str).matches();
})
.findFirst();
}
}
For example:
Optional<DataType> dataType = getTypeOfField("Bharathiraja");
System.out.println(dataType);
System.out.println(dataType .get());
Output:
Optional[STRING]
STRING
Please note, regular exp pattern is vary based on requirements, so modify the pattern as per your need don't take as it is.
Happy Coding !

Dynamically Read Properties file and draw Text boxes in javascript

I have a properties file with all the fields. dynamically I need to draw the text box with fields as read from the properties file and enter values and post it to controller in spring - java !
Example Properties File
name=String
age=int
address=string
How can I do this from java code..

For my idea, I will do it as below:
Using ajax to get fields from property file on server and return a list of field and type of field in a json format (key, value).
Now we have the data of those fields, then we generate them to your form using jquery or javascript.
Submit the form to server to get value.
Step 1 and 2 are quite easy, so I do not post the code; for step 3, you can try the method below to parse the params in query string to a map.
public static Map getMapFromQueryString(String queryString) {
Map returnMap = new HashMap();
StringTokenizer stringTokenizer = new StringTokenizer(queryString, "&");
while (stringTokenizer.hasMoreTokens()) {
String key, value;
String keyAndValue = stringTokenizer.nextToken();
int indexOfEqual = keyAndValue.indexOf("=");
if (indexOfEqual >= 0) {
key = keyAndValue.substring(0, indexOfEqual);
if ((indexOfEqual + 1) < keyAndValue.length()) {
value = keyAndValue.substring(indexOfEqual + 1);
} else {
value = "";
}
} else {
key = keyAndValue;
value = "";
}
if (key.length() > 0) returnMap.put(key, value);
}
return returnMap;
}
Now you can get all the value of dynamic fields on the form.
Hope this solution is helpful for you.

How to check if the subdomain is also from same domain using java

i have a list of url's i need to filter specific domain and subdomain. say i have some domains like
http://www.example.com
http://test.example.com
http://test2.example.com
I need to extract urls which from domain example.com.

Working on project that required me to determine if two URLs are from the same sub domain (even when there are nested domains). I worked up a modification from the guide above. This holds out pretty well thus far:
public static boolean isOneSubdomainOfTheOther(String a, String b) {
try {
URL first = new URL(a);
String firstHost = first.getHost();
firstHost = firstHost.startsWith("www.") ? firstHost.substring(4) : firstHost;
URL second = new URL(b);
String secondHost = second.getHost();
secondHost = secondHost.startsWith("www.") ? secondHost.substring(4) : secondHost;
/*
Test if one is a substring of the other
*/
if (firstHost.contains(secondHost) || secondHost.contains(firstHost)) {
String[] firstPieces = firstHost.split("\\.");
String[] secondPieces = secondHost.split("\\.");
String[] longerHost = {""};
String[] shorterHost = {""};
if (firstPieces.length >= secondPieces.length) {
longerHost = firstPieces;
shorterHost = secondPieces;
} else {
longerHost = secondPieces;
shorterHost = firstPieces;
}
//int longLength = longURL.length;
int minLength = shorterHost.length;
int i = 1;
/*
Compare from the tail of both host and work backwards
*/
while (minLength > 0) {
String tail1 = longerHost[longerHost.length - i];
String tail2 = shorterHost[shorterHost.length - i];
if (tail1.equalsIgnoreCase(tail2)) {
//move up one place to the left
minLength--;
} else {
//domains do not match
return false;
}
i++;
}
if (minLength == 0) //shorter host exhausted. Is a sub domain
return true;
}
} catch (MalformedURLException ex) {
ex.printStackTrace();
}
return false;
}
Figure I'd leave it here for future reference of a similar problem.

I understand you are probably looking for a fancy solution using URL class or something but it is not required. Simply think of a way to extract "example.com" from each of the urls.
Note: example.com is essentially a different domain than say example.net. Thus extracting just "example" is technically the wrong thing to do.
We can divide a sample url say:
http://sub.example.com/page1.html
Step 1: Split the url with delimiter " / " to extract the part containing the domain.
Each such part may be looked at in form of the following blocks (which may be empty)
[www][subdomain][basedomain]
Step 2: Discard "www" (if present). We are left with [subdomain][basedomain]
Step 3: Split the string with delimiter " . "
Step 4: Find the total number of strings generated from the split. If there are 2 strings, both of them are the target domain (example and com). If there are >=3 strings, get the last 3 strings. If the length of last string is 3, then the last 2 strings comprise the domain (example and com). If the length of last string is 2, then the last 3 strings comprise the domain (example and co and uk)
I think this should do the trick (I do hope this wasn't a homework :D )
//You may clean this method to make it more optimum / better
private String getRootDomain(String url){
String[] domainKeys = url.split("/")[2].split("\\.");
int length = domainKeys.length;
int dummy = domainKeys[0].equals("www")?1:0;
if(length-dummy == 2)
return domainKeys[length-2] + "." + domainKeys[length-1];
else{
if(domainKeys[length-1].length == 2) {
return domainKeys[length-3] + "." + domainKeys[length-2] + "." + domainKeys[length-1];
}
else{
return domainKeys[length-2] + "." + domainKeys[length-1];
}
}
}