Develop Reference

Iterating over a CSV file within a TestNG DataProvider

I'm a little confused as to how best to implement a simple DataProvider, having not done so before.
I have a very simple comma delimited .csv file:
978KAL,625JBH,876SSH,452GSH
I simply need to read it in and iterate over the records, running the same test for each record until done.
My code so far:
String csvFile = "src/test/resources/registrationsData.csv";
BufferedReader br = null;
String line = "";
String cvsSplitBy = ",";
#DataProvider(name="getRegistrations")
private Object[] getCSVTestData() {
Object [] registrationsObject;
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
// use comma as separator
String [] registrations = line.split(cvsSplitBy);
System.out.println( registrations[0] + "," + registrations[1]);
}
} catch//File not found & IOException handling here
registrationsObject = new Object[][]{registrations};
return registrationsObject;
}
#Test(dataProvider = "getRegistrations")
public void getRegistrations(String registration){
Object[] objRegArray = getCSVTestData();
for(int i=0; objRegArray.length>i; i++){
//run tests for every record in the array (csv file)
}
}
I know that I need to use an Object array return type for the Data Provider method.
I'm unclear as to how (and/or the best way) to retrieve each record from the objRegArray object.
This is a basic Collections question I guess; can anyone point me the right way?

Check this code with my explanation below:
package click.webelement.testng;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Iterator;
import java.util.Scanner;
public class OneLineCSV {
final static String CSV_FILE = "/path_to_file/oneline.csv";
final static String DELIMETER = ",";
#DataProvider(name = "test")
public Iterator<Object[]> testDP(){
try {
Scanner scanner = new Scanner(new File(CSV_FILE)).useDelimiter(DELIMETER);
return new Iterator<Object[]>() {
#Override
public boolean hasNext() {
return scanner.hasNext();
}
#Override
public Object[] next() {
return new Object[]{scanner.next()};
}
};
} catch (FileNotFoundException e) {
e.printStackTrace();
return null;
}
}
#Test(dataProvider = "test")
public void testOneLineCSV(String value){
System.out.println(value);
}
}
So I would use Scanner class hence it has the convenient facility to parse a string into tokens.
I would also use the capability to return Iterator<Object[]> in your data provider since Scanner is designed in that way. You simply wrap it with new Iterator that converts String that is returned by Scanner.next() to new Object[]{scanner.next}.
Using Iterator with Scanner is really more comfortable since you may not know how many values you will have to provide. So you shouldn't care of defining array size.

Java - File Reading into an Array Error

I'm doing an assignment for school (so I can unfortunately not use third party libraries) and the goal is to read a csv file into an array, process it in a different method, and print it in another. This is what I have so far but I get the error:
Type mismatch: cannot convert from List<String> to Collection<? extends String[]>.
Here is my code:
package client.java;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public abstract class Client{
String file = "bank-Detail.csv";
ArrayList<String[]> bank = new ArrayList<>();
public Client(String file) {
this.file = file;
}
public void readData() throws IOException {
int count = 0;
String file = "bank-Detail.txt";
try {
BufferedReader br = new BufferedReader(new FileReader(file));
String line = "";
while ((line = br.readLine()) != null) {
bank.addAll(Arrays.asList(line.split(",")));
The line (Arrays.asList(line.split(","))); is where I get the error.
String[] entries = line.split(",");
String[][] numbers = (String[][]) bank.toArray(new String[bank.size()][12]);
}
} catch (FileNotFoundException e) {
}
}
public void processData() {
}
public void printData() {
}

ArrayList<String[]> bank is a list of arrays, so instead of doing
bank.addAll(Arrays.asList(line.split(",")));
you just need to do
bank.add(line.split(","));
Here's javadoc for split method, it returns an array of String which is what we need to add into the list.

Each entry of your list bank is an arrray of String. But
But at this line
bank.addAll(Arrays.asList(line.split(",")));
You are trying to add a list of string List<String> to bank using addAll. But if you want to use addAll you have to add List<String []>.
There is a small fix to your problem:
bank.add(line.split(","))
As line.split(",") will return an array of String.
And you are good to go.

How to convert the free text into Json String Array in Java using GSON library? [duplicate]

This question already has an answer here:
directly convert CSV file to JSON file using the Jackson library
(1 answer)
Closed 7 years ago.
I am having free text available into the file.
I am stucked while convert it into the json string array
The columns names are variable and can be n number of columns
email_from,email_to,DATE_CHANGED
samwilliams#gmail.com, mike_haley#gmail.com, 1447666867
smithpaul#gmail.com, angierussell#gmail.com, 1447668867
The first line is of headers, and the rest of all the lines are their values.
So, every line would contain, same number of parameters with respect to each column.
Columns are comma separated.
The json string response should be looked like this
{
"data": [
{
"email_from": "samwilliams#gmail.com",
"email_to": "mike_haley#gmail.com",
"DATE_CHANGED": "1447666867"
},
{
"email_from": "smithpaul#gmail.com",
"email_to": "angierussell#gmail.com",
"DATE_CHANGED": "1447668867"
}
]
}

The following code opens a file with comma-delimited strings and uses while loop to construct JsonObject and then keeps adding them to the JsonArray and finally prints it (Please add your validations as well you could move majority of the code out of try block if you wish to make the code perform better).
It addresses the need for having n number of columns in the file.
package gjson;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import com.google.gson.FieldNamingPolicy;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
public class GJSONTest {
public static void main(String[] args) {
// create an array called datasets
JsonArray datasets = new JsonArray();
File file = new File("C:\\test_stackoverflow\\list.txt");
try (BufferedReader br = new BufferedReader(new FileReader(file))) {
String line;
boolean flag = true;
List<String> columns = null;
while ((line = br.readLine()) != null) {
if (flag) {
flag = false;
//process header
columns = Arrays.asList(line.split(","));
} else {
//to store the object temporarily
JsonObject obj = new JsonObject();
List<String> chunks = Arrays.asList(line.split(","));
for(int i = 0; i < columns.size(); i++) {
obj.addProperty(columns.get(i), chunks.get(i));
}
datasets.add(obj);
}
}
} catch(FileNotFoundException fnfe) {
System.out.println("File not found.");
} catch(IOException io) {
System.out.println("Cannot read file.");
}
Gson gson = new GsonBuilder().setPrettyPrinting().serializeNulls().setFieldNamingPolicy(FieldNamingPolicy.UPPER_CAMEL_CASE).create();
System.out.println(gson.toJson(datasets));
}
}
See the following screenshots (below with 3 columns)
Added another column to the text file and following is the output.
Here is the sample .txt file containing your data

private String getParsedData(String data){
String[] lines = data.split("\\r?\\n");
List<Map> dataList = new ArrayList<Map>();
int colCount = 0;
if(lines.length > 1){
String keyLine = lines[0];
String[] keys = keyLine.split(",");
for(int i = 1; i < lines.length; i++){
colCount = 0;
Map<String, Object> rawObj = new HashMap<String, Object>();
try {
String[] values = lines[i].split(",");
for(String value: values){
rawObj.put(keys[colCount], value);
colCount++;
}
} catch (Exception e) {
}
dataList.add(rawObj);
}
}
Map<String, Object> rawObj = new HashMap<String, Object>();
rawObj.put("data", dataList);
Gson gson = new Gson();
String res = gson.toJson(rawObj);
return res;
}
String data = "email_from,email_to,DATE_CHANGED\r\nsamwilliams#gmail.com, mike_haley#gmail.com, 1447666867\r\nsmithpaul#gmail.com, angierussell#gmail.com, 1447668867";
But I am not sure whether its an efficient code or not.

Try this code:
public class ContactObject {
private String emailFrom;
private String emailTo;
private String dateChanged;
public ContactObject(String emailFrom, String emailTo, String dateChanged) {
this.emailFrom = emailFrom;
this.emailTo = emailTo;
this.dateChanged = dateChanged;
}
#Override
public String toString() {
return "{email_from:" + emailFrom + ", email_to:" + emailTo + ", DATE_CHANGED:" + dateChanged;
}
}
public class ContactJSON {
private List<ContactObject> data;
public ContactJSON(List<ContactObject> contactList) {
this.data = contactList;
}
}
Then in your main() method you can make use of these classes:
public static void main(String[] args) {
List<ContactObject> contactList = new ArrayList<ContactObject>();
ContactObject obj1 = new ContactObject("samwilliams#gmail.com", "mike_haley#gmail.com", "1447666867");
ContactObject obj2 = new ContactObject("smithpaul#gmail.com", "angierussell#gmail.com", "1447668867");
contactList.add(obj1);
contactList.add(obj2);
Gson gson = new Gson();
String json = gson.toJson(new ContactJSON(contactList));
System.out.println(json);
}
Output:
{"data":[{"emailFrom":"samwilliams#gmail.com","emailTo":"mike_haley#gmail.com","dateChanged":"1447666867"},{"emailFrom":"smithpaul#gmail.com","emailTo":"angierussell#gmail.com","dateChanged":"1447668867"}]}

putting text file data into a hash map

my data in the text file looks like this...
3
movie title
4
movie title
1
movie title
the number on top is the movie rating and the text under it is the movie title.
The code I have so far is below. But It's not printing anything out except empty brackets! Sample code would be appreciated!
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
public class MovieReview {
public static void main(String[] args) {
Map<Integer, String> map = new HashMap<Integer, String>();
BufferedReader br = new BufferedReader(new FileReader("C:/Users/sgoetz/Desktop/movieReviews.txt"));
String line = br.readLine();
while (line != null) {
line = br.readLine();
System.out.println(map);
}
}

Try This
public static void main(String[] args) throws IOException {
Map<Integer, String> map = new HashMap<Integer, String>();
BufferedReader br = new BufferedReader(new FileReader("movieReviews.txt"));
String line="";
int i=0;
while (line != null) {
line = br.readLine();
map.put(i,line);
i++;
}
for(int j=0;j<map.size();j++){
System.out.println(map.get(j));
}
}

Your code is printing nothing because you are printing the map on which you put nothing. So the map remains empty. Also your first while iteration is buggy, you read one line from the stream before the while loop then you enter it an immediately read the next line. The first line is lost.
For reading from a buffered stream the following pattern should be considered:
while(null != (line = br.readLine())) {
// do what ever you want with the line.
}

If you want to store moving names of against its rating you have to declare map as Map>. Following code populates the movie title against its rating. Hope this is helpful.
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.List;
public class MovieReview {
public static void main(String[] args) {
Map<Integer, List<String>> map = new HashMap<Integer, List<String>>();
BufferedReader br = new BufferedReader(new FileReader("C:/Users/sgoetz/Desktop/movieReviews.txt"));
String line = null;
while ((line = br.readLine()) != null) {
//Rating
int rating = Integer.parseInt(line);
//Movie name
line = br.readLine();
List<String> movieList = map.get(rating);
if(movieList == null) {
movieList = new ArrayList<String>();
map.put(rating, movieList);
}
//Adding movie name to list
movieList.add(line);
}
}
}
}

here we go
File:
3,movie title,rating,other,other
4,movie title,rating,other,other
1,movie title,rating,other,other
code:
public static void main(String[] args) throws IOException {
Map<Integer, String> map = new HashMap<Integer, String>();
BufferedReader br = new BufferedReader(new FileReader("movieReviews.txt"));
String line="";
int i=0;
while (line != null) {
line = br.readLine();
map.put(i,line);
i++;
}
String movieNumber="";
String movieTitle="";
String movieRating="";
String movieOther1="";
String movieOther2="";
for(int j=0;j<map.size();j++){
if(!(map.get(j)== null)){
String[] getData=map.get(j).toString().split("\\,");
movieNumber = getData[0];
movieTitle = getData[1];
movieRating = getData[2];
movieOther1 = getData[3];
movieOther2 = getData[4];
System.out.println("|"+movieNumber+"|"+movieTitle+"|"+movieRating+"|"+movieOther1+"|"+movieOther2);
}
}
}

A more example:
while (true) {
// Number line
String value = br.readLine();
if (value == null || value.trim().isEmpty()) {
break;
}
Integer valueInt = Integer.parseInt(value);
// Name line
String title = br.readLine();
if (title == null || value.trim().isEmpty()) {
break;
}
map.put(valueInt, title);
}
System.out.println(map);
And the output is:
{1=movie title, 3=movie title, 4=movie title}

Remove Duplicate Lines from Text using Java

I was wondering if anyone has logic in java that removes duplicate lines while maintaining the lines order.
I would prefer no regex solution.

public class UniqueLineReader extends BufferedReader {
Set<String> lines = new HashSet<String>();
public UniqueLineReader(Reader arg0) {
super(arg0);
}
#Override
public String readLine() throws IOException {
String uniqueLine;
if (lines.add(uniqueLine = super.readLine()))
return uniqueLine;
return "";
}
//for testing..
public static void main(String args[]) {
try {
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream(
"test.txt");
UniqueLineReader br = new UniqueLineReader(new InputStreamReader(fstream));
String strLine;
// Read File Line By Line
while ((strLine = br.readLine()) != null) {
// Print the content on the console
if (strLine != "")
System.out.println(strLine);
}
// Close the input stream
in.close();
} catch (Exception e) {// Catch exception if any
System.err.println("Error: " + e.getMessage());
}
}
}
Modified Version:
public class UniqueLineReader extends BufferedReader {
Set<String> lines = new HashSet<String>();
public UniqueLineReader(Reader arg0) {
super(arg0);
}
#Override
public String readLine() throws IOException {
String uniqueLine;
while (lines.add(uniqueLine = super.readLine()) == false); //read until encountering a unique line
return uniqueLine;
}
public static void main(String args[]) {
try {
// Open the file that is the first
// command line parameter
FileInputStream fstream = new FileInputStream(
"/home/emil/Desktop/ff.txt");
UniqueLineReader br = new UniqueLineReader(new InputStreamReader(fstream));
String strLine;
// Read File Line By Line
while ((strLine = br.readLine()) != null) {
// Print the content on the console
System.out.println(strLine);
}
// Close the input stream
in.close();
} catch (Exception e) {// Catch exception if any
System.err.println("Error: " + e.getMessage());
}
}
}

If you feed the lines into a LinkedHashSet, it ignores the repeated ones, since it's a set, but preserves the order, since it's linked. If you just want to know whether you've seena given line before, feed them into a simple Set as you go on, and ignore those which the Set already contains/contained.

It can be easy to remove duplicate line from text or File using new java Stream API. Stream support different aggregate feature like sort,distinct and work with different java's existing data structures and their methods. Following example can use to remove duplicate or sort the content in File using Stream API
package removeword;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.OpenOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Scanner;
import java.util.stream.Stream;
import static java.nio.file.StandardOpenOption.*;
import static java.util.stream.Collectors.joining;
public class Java8UniqueWords {
public static void main(String[] args) throws IOException {
Path sourcePath = Paths.get("C:/Users/source.txt");
Path changedPath = Paths.get("C:/Users/removedDouplicate_file.txt");
try (final Stream<String> lines = Files.lines(sourcePath )
// .map(line -> line.toLowerCase()) /*optional to use existing string methods*/
.distinct()
// .sorted()) /*aggregrate function to sort disctincted line*/
{
final String uniqueWords = lines.collect(joining("\n"));
System.out.println("Final Output:" + uniqueWords);
Files.write(changedPath , uniqueWords.getBytes(),WRITE, TRUNCATE_EXISTING);
}
}
}

Read the text file using a BufferedReader and store it in a LinkedHashSet. Print it back out.
Here's an example:
public class DuplicateRemover {
public String stripDuplicates(String aHunk) {
StringBuilder result = new StringBuilder();
Set<String> uniqueLines = new LinkedHashSet<String>();
String[] chunks = aHunk.split("\n");
uniqueLines.addAll(Arrays.asList(chunks));
for (String chunk : uniqueLines) {
result.append(chunk).append("\n");
}
return result.toString();
}
}
Here's some unit tests to verify ( ignore my evil copy-paste ;) ):
import org.junit.Test;
import static org.junit.Assert.*;
public class DuplicateRemoverTest {
#Test
public void removesDuplicateLines() {
String input = "a\nb\nc\nb\nd\n";
String expected = "a\nb\nc\nd\n";
DuplicateRemover remover = new DuplicateRemover();
String actual = remover.stripDuplicates(input);
assertEquals(expected, actual);
}
#Test
public void removesDuplicateLinesUnalphabetized() {
String input = "z\nb\nc\nb\nz\n";
String expected = "z\nb\nc\n";
DuplicateRemover remover = new DuplicateRemover();
String actual = remover.stripDuplicates(input);
assertEquals(expected, actual);
}
}

Here's another solution. Let's just use UNIX!
cat MyFile.java | uniq > MyFile.java
Edit: Oh wait, I re-read the topic. Is this a legal solution since I managed to be language agnostic?

For better/optimum performance, it's wise to use Java 8's API features viz. Streams & Method references with LinkedHashSet for Collection as below:
import java.io.IOException;
import java.io.PrintWriter;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.LinkedHashSet;
import java.util.stream.Collectors;
public class UniqueOperation {
private static PrintWriter pw;
enter code here
public static void main(String[] args) throws IOException {
pw = new PrintWriter("abc.txt");
for(String p : Files.newBufferedReader(Paths.get("C:/Users/as00465129/Desktop/FrontEndUdemyLinks.txt")).
lines().
collect(Collectors.toCollection(LinkedHashSet::new)))
pw.println(p);
pw.flush();
pw.close();
System.out.println("File operation performed successfully");
}

here I'm using a hashset to store seen lines
Scanner scan;//input
Set<String> lines = new HashSet<String>();
StringBuilder strb = new StringBuilder();
while(scan.hasNextLine()){
String line = scan.nextLine();
if(lines.add(line)) strb.append(line);
}