Hi I am trying to convert a CSV file into a JSON array using A dependency called csvReader, but when I run the code it prints out the JSON response incorrectly and I ament sure why would anyone be able to point me in the right direction.
#GetMapping("/convert")
public List<List<String>> convertCSV() throws FileNotFoundException {
List<List<String>> records = new ArrayList<List<String>>();
try (CSVReader csvReader = new CSVReader(new FileReader("C:/Download/cities.csv"));) {
String[] values = null;
while ((values = csvReader.readNext()) != null) {
records.add(Arrays.asList(values));
}
} catch (IOException e) {
e.printStackTrace();
}
return values;
}
Your case is not a big deal.
You can read that csv and build json.
Read first row and determine columns. The rest of rows are values.
public class Foo{
public static void main(String[] args) throws Exception{
List<String> csvRows = null;
try(var reader = Files.lines(Paths.get("dataFile.csv"))){
csvRows = reader.collect(Collectors.toList());
}catch(Exception e){
e.printStackTrace();
}
if(csvRows != null){
String json = csvToJson(csvRows);
System.out.println(json);
}
}
public static String csvToJson(List<String> csv){
//remove empty lines
//this will affect permanently the list.
//be careful if you want to use this list after executing this method
csv.removeIf(e -> e.trim().isEmpty());
//csv is empty or have declared only columns
if(csv.size() <= 1){
return "[]";
}
//get first line = columns names
String[] columns = csv.get(0).split(",");
//get all rows
StringBuilder json = new StringBuilder("[\n");
csv.subList(1, csv.size()) //substring without first row(columns)
.stream()
.map(e -> e.split(","))
.filter(e -> e.length == columns.length) //values size should match with columns size
.forEach(row -> {
json.append("\t{\n");
for(int i = 0; i < columns.length; i++){
json.append("\t\t\"")
.append(columns[i])
.append("\" : \"")
.append(row[i])
.append("\",\n"); //comma-1
}
//replace comma-1 with \n
json.replace(json.lastIndexOf(","), json.length(), "\n");
json.append("\t},"); //comma-2
});
//remove comma-2
json.replace(json.lastIndexOf(","), json.length(), "");
json.append("\n]");
return json.toString();
}
}
Tested on:
fname,lname,note
Shaun,Curtis,a
Kirby,Beil,b
-----------------------
[
{
"fname" : "Shaun",
"lname" : "Curtis",
"note" : "a"
}, {
"fname" : "Kirby",
"lname" : "Beil",
"note" : "b"
}
]
This method work on any structure of csv. Don't need to map columns.
That is because of your reading data in String and printing the List of String. If you want to map the CSV to Object ( JSON Object), You need to read the CSV as bean object please find below code snippet to print as JSON, override toString method as JSON format.
User.java
public class User {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#NotNull
private String name;
#NotNull
private String surname;
//Getter and Setters
}
CsvReaderUtil.java
public static List<User> readCsvFile() throws IOException {
List<User> list = null;
CSVReader reader = null;
InputStream is = null;
try {
File initialFile = new File("C:\\Users\\ER\\Desktop\\test.csv");
is = new FileInputStream(initialFile);
reader = new CSVReader(new InputStreamReader(is), ',', '"', 1);
ColumnPositionMappingStrategy strat = new ColumnPositionMappingStrategy();
strat.setType(User.class);
String[] columns = new String[]{"id", "name", "surname"};
strat.setColumnMapping(columns);
CsvToBean csv = new CsvToBean();
list = csv.parse(strat, reader);
} catch (Exception e) {
e.printStackTrace();
} finally {
is.close();
reader.close();
}
return list;
}
Now print this List Of Users as a JSON object.
Here is a useful example of how to transform CSV to JSON using Java 11+:
private String fromCsvToJson(String csvFile) {
String[] lines = file.split("\n");
if (lines.length <= 1) {
return List.of();
}
var headers = lines[0].split(",");
var jsonFormat = Arrays.stream(lines)
.skip(1)
.map(line -> line.split(","))
.filter(line -> headers.length == line.length)
.map(line -> IntStream.range(0, headers.length).boxed().collect(toMap(i -> headers[i], i -> line[i], (a, b) -> b)))
.toList();
return new ObjectMapper().writeValueAsString(jsonFormat);
}
Related
I'm trying to read from a csv file but it's slow. Here's the code roughly explained:
private static Film[] readMoviesFromCSV() {
// Regex to split by comma without splitting in double quotes.
// https://regexr.com/3s3me <- example on this data
var pattern = Pattern.compile(",(?=(?:[^\\\"]*\\\"[^\\\"]*\\\")*[^\\\"]*$)");
Film[] films = null;
try (var br = new BufferedReader(new FileReader(FILENAME))) {
var start = System.currentTimeMillis();
var temparr = br.lines().skip(1).collect(Collectors.toList()); // skip first line and read into List
films = temparr.stream().parallel()
.map(pattern::split)
.filter(x -> x.length == 24 && x[7].equals("en")) // all fields(total 24) and english speaking movies
.filter(x -> (x[14].length() > 0)) // check if it has x[14] (date)
.map(movieData -> new Film(movieData[8], movieData[9], movieData[14], movieData[22], movieData[23], movieData[7]))
// movieData[8] = String title, movieData[9] = String overview
// movieData[14] = String date (constructor parses it to LocalDate object)
// movieData[22] = String avgRating
.toArray(Film[]::new);
System.out.println(MessageFormat.format("Execution time: {0}", (System.currentTimeMillis() - start)));
System.out.println(films.length);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return films;
}
File is about 30 MB big and it takes about 3-4 seconds avg. I'm using streams but it's still really slow. Is it because of that splitting each time?
EDIT: I've managed to speed up reading and processing time by 3x with uniVocity-parsers library. On average it takes 950 ms to finish. That's pretty impressive.
private static Film[] readMoviesWithLib() {
Film[] films = null;
CsvParserSettings parserSettings = new CsvParserSettings();
parserSettings.setLineSeparatorDetectionEnabled(true);
RowListProcessor rowProcessor = new RowListProcessor();
parserSettings.setProcessor(rowProcessor);
parserSettings.setHeaderExtractionEnabled(true);
CsvParser parser = new CsvParser(parserSettings);
var start = System.currentTimeMillis();
try {
parser.parse(new BufferedReader(new FileReader(FILENAME)));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
List<String[]> rows = rowProcessor.getRows();
films = rows.stream()
.filter(Objects::nonNull)
.filter(x -> x.length == 24 && x[14] != null && x[7] != null)
.filter(x -> x[7].equals("en"))
.map(movieData -> new Film(movieData[8], movieData[9], movieData[14], movieData[22], movieData[23], movieData[7]))
.toArray(Film[]::new);
System.out.printf(MessageFormat.format("Time: {0}",(System.currentTimeMillis()-start)));
return films;
}
Author of the univocity-parsers library here. You can speed up the code you posted in your edit a little bit further by rewriting it like this:
//initialize an arraylist with a good size to avoid reallocation
final ArrayList<Film> films = new ArrayList<Film>(20000);
CsvParserSettings parserSettings = new CsvParserSettings();
parserSettings.setLineSeparatorDetectionEnabled(true);
parserSettings.setHeaderExtractionEnabled(true);
//don't generate strings for columns you don't want
parserSettings.selectIndexes(7, 8, 9, 14, 22, 23);
//keep generating rows with the same number of columns found in the input
//indexes not selected will have nulls as they are not processed.
parserSettings.setColumnReorderingEnabled(false);
parserSettings.setProcessor(new AbstractRowProcessor(){
#Override
public void rowProcessed(String[] row, ParsingContext context) {
if(row.length == 24 && "en".equals(row[7]) && row[14] != null){
films.add(new Film(row[8], row[9], row[14], row[22], row[23], row[7]));
}
}
});
CsvParser parser = new CsvParser(parserSettings);
long start = System.currentTimeMillis();
try {
parser.parse(new File(FILENAME), "UTF-8");
} catch (FileNotFoundException e) {
e.printStackTrace();
}
System.out.printf(MessageFormat.format("Time: {0}",(System.currentTimeMillis()-start)));
return films.toArray(new Film[0]);
For convenience, if you have to process stuff into different classes you can also use annotations in your Film class.
Hope this helps.
I am trying to read from one CSV file using OpenCSV. I then want to copy all the data from the input csv and output it to another csv file while adding a new column with information.
public void run_streets_tsv( String tsvIn, String tsvOut) throws Exception
{
CSVReader reader = null;
CSVWriter writer = null;
try
{
reader = new CSVReader((new FileReader(tsvIn)));
writer = new CSVWriter(new FileWriter(tsvOut), '\t');
String element [] = null;
List<String[]> a = new ArrayList<String[]>();
while((element = reader.readNext()) != null){
for(int i = 0; i<element.length; i++){
a.add(i, element);
//a.add("JSON"); need to add this json element at the end of each column
}
}
writer.writeAll(a);
}
catch(Exception e)
{
throw e;
}
finally
{
reader.close();
writer.close();
}
}
Another method I am trying is like this (changing the while loop, all other code remains the same):
String element [] = null;
while((element = reader.readNext()) != null){
ArrayList list = new ArrayList(Arrays.asList(reader));
list.add(element);
list.add("JSON");
writer.writeNext(element);
}
This does correctly print all the lines, but it just copies. I want to add that extra "JSON" column with its data.
The following "enlarges" the element-Array by one, enabling you to put something in the newly created last index. Then just save that array.
import java.util.Arrays;
String element[] = null;
while((element = reader.readNext()) != null){
element = Arrays.copyOf(element, element.length + 1);
element[element.length - 1] = "JSON";
writer.writeNext(element);
}
OK, you are close although I see a few errors.
'reader.readNext()' return a line from the input as a String array, we basically need to add an element to this for the output.
while((element = reader.readNext()) != null) {
String[] output = getExpandedArray(element);
a.add(output);
}
You will need to implement getExpandedArray, I will start it off.
private String[] getExpandedArray(String[] input) {
String[] output = null;
//Populate/create output from input, but with the array 1 bigger.
output[output.length -1] = "JSON";
return output;
}
I have a JSON file airports.json which contains 5720 objects, I want to parse the file in Java into Objects Airport.
The code below is how I parse it, the problem is, it is taking too much time to completely parse all the file, about 1 min and 50 sec.
ArrayList<Airport> arrayAirports = new ArrayList<>();
String json_response = loadJSONFromAsset();
try {
JSONObject airports = new JSONObject(json_response.trim());
Log.d("Length Array 0", String.valueOf(airports.names().length()));
Log.d("Length Array 1", String.valueOf(arrayAirports.size()));
for(int i = 0; i<airports.names().length(); i++){
JSONObject jsonAirport = airports.getJSONObject(airports.names().getString(i));
Airport newAirport = new Airport();
newAirport.name = jsonAirport.getString("name");
newAirport.city = jsonAirport.getString("city");
newAirport.country = jsonAirport.getString("country");
newAirport.latitude = jsonAirport.getString("latitude");
newAirport.longitude = jsonAirport.getString("longitude");
newAirport.altitude = jsonAirport.getString("altitude");
newAirport.iata = jsonAirport.getString("iata");
newAirport.icao = jsonAirport.getString("icao");
newAirport.timeZone = jsonAirport.getString("timezone");
newAirport.dst = jsonAirport.getString("dst");
arrayAirports.add(newAirport);
}
} catch (JSONException e1) {
e1.printStackTrace();
}
Log.d("Length Array 2", String.valueOf(arrayAirports.size()));
Is there a way to parse it quicklier.
By the way my friend is parsing it with no time at all using Objective C.
Link to JSON file
Use GSON instead and measure the speed with it. While it is possible possible that just reading the file could take a long time, a whole minute seems pretty bad.
Since you mentioned you don't know how to use GSON, here's a tutorial I wrote for a student on how to use GSON. It assume you're getting the file from a network call though, so you need to apply it to use your local JSON file.
Don't know if it matters for performance, but you shouldn't call names() repeatedly. Assign to variable before loop, then use it.
JSONArray names = airports.names();
Log.d("Length Array 0", String.valueOf(names.length()));
Log.d("Length Array 1", String.valueOf(arrayAirports.size()));
for(int i = 0; i<names.length(); i++){
JSONObject jsonAirport = airports.getJSONObject(names.getString(i));
// code
}
Better yet, use length() and keys():
Log.d("Length Array 0", String.valueOf(airports.length()));
Log.d("Length Array 1", String.valueOf(arrayAirports.size()));
for (Iterator<String> nameIter = airports.keys(); nameIter.hasNext(); ){
String name = nameIter.next();
JSONObject jsonAirport = airports.getJSONObject(name);
// code
}
You are putting a (I guess?) large JSON data file into a JSONObject directly.
In this case it would be recommended to use a token based reader, such as JsonReader.
Pasted directly from the docs:
public List readJsonStream(InputStream in ) throws IOException {
JsonReader reader = new JsonReader(new InputStreamReader( in , "UTF-8"));
try {
return readMessagesArray(reader);
finally {
reader.close();
}
}
public List readMessagesArray(JsonReader reader) throws IOException {
List messages = new ArrayList();
reader.beginArray();
while (reader.hasNext()) {
messages.add(readMessage(reader));
}
reader.endArray();
return messages;
}
public Message readMessage(JsonReader reader) throws IOException {
long id = -1;
String text = null;
User user = null;
List geo = null;
reader.beginObject();
while (reader.hasNext()) {
String name = reader.nextName();
if (name.equals("id")) {
id = reader.nextLong();
} else if (name.equals("text")) {
text = reader.nextString();
} else if (name.equals("geo") && reader.peek() != JsonToken.NULL) {
geo = readDoublesArray(reader);
} else if (name.equals("user")) {
user = readUser(reader);
} else {
reader.skipValue();
}
}
reader.endObject();
return new Message(id, text, user, geo);
}
public List readDoublesArray(JsonReader reader) throws IOException {
List doubles = new ArrayList();
reader.beginArray();
while (reader.hasNext()) {
doubles.add(reader.nextDouble());
}
reader.endArray();
return doubles;
}
public User readUser(JsonReader reader) throws IOException {
String username = null;
int followersCount = -1;
reader.beginObject();
while (reader.hasNext()) {
String name = reader.nextName();
if (name.equals("name")) {
username = reader.nextString();
} else if (name.equals("followers_count")) {
followersCount = reader.nextInt();
} else {
reader.skipValue();
}
}
reader.endObject();
return new User(username, followersCount);
}
}
I have a CSV file which contains rules and ruleversions. The CSV file looks like this:
CSV FILE:
#RULENAME, RULEVERSION
RULE,01-02-01
RULE,01-02-02
RULE,01-02-34
OTHER_RULE,01-02-04
THIRDRULE, 01-02-04
THIRDRULE, 01-02-04
As you can see, 1 rule can have 1 or more rule versions. What I need to do is read this CSV file and put them in an array. I am currently doing that with the following script:
private static List<String[]> getRulesFromFile() {
String csvFile = "rulesets.csv";
BufferedReader br = null;
String line = "";
String delimiter = ",";
List<String[]> input = new ArrayList<String[]>();
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
if (!line.startsWith("#")) {
String[] rulesetEntry = line.split(delimiter);
input.add(rulesetEntry);
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return input;
}
But I need to adapt the script so that it saves the information in the following format:
ARRAY (
=> RULE => 01-02-01, 01-02-02, 01-02-04
=> OTHER_RULE => 01-02-34
=> THIRDRULE => 01-02-01, 01-02-02
)
What is the best way to do this? Multidimensional array? And how do I make sure it doesn't save the rulename more than once?
You should use a different data structure, for example an HashMap, like this.
HashMap<String, List<String>> myMap = new HashMap<>();
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
if (!line.startsWith("#")) {
String[] parts = string.split(delimiter);
String key = parts[0];
String value = parts[1];
if (myMap.containsKey(key)) {
myMap.get(key).add(value);
} else {
List<String> values = new ArrayList<String>();
values.add(value);
myMap.put(key, values);
}
}
}
This should work!
See using an ArrayList is not a good data structure of choice here.
I would personally suggest you to use a HashMap> for this particular purpose.
The rules will be your keys and rule versions will be your values which will be a list of strings.
While traversing your original file, just check if the rule (key) is present, then add the value to the list of rule versions (values) already present, otherwise add a new key and add the value to it.
For instance like this:
public List<String> removeDuplicates(List<String> myList) {
Hashtable<String, String> hashtable=new Hashtable<String, String>();
for(String s:myList) {
hashtable.put(s, s);
}
return new ArrayList<String>(hashtable.values());
}
This is exactly what key - value pairs can be used for. Just take a look at the Map Interface. There you can define a unique key containing various elements as value, perfectly for your issue.
Code:
// This collection will take String type as a Key
// and Prevent duplicates in its associated values
Map<String, HashSet<String>> map = new HashMap<String,HashSet<String>>();
// Check if collection contains the Key you are about to enter
// !REPLACE! -> "rule" with the Key you want to enter into your collection
// !REPLACE! -> "whatever" with the Value you want to associate with the key
if(!map.containsKey("rule")){
map.put("rule", new HashSet<String>());
}
else{
map.get("rule").add("whatever");
}
Reference:
Set
Map
Hello is there any way to convert vector into arraylist ? I want to make search options for my table but looks like it would be much easier with arraylist for me. I used search but surprising there was nothing about vectors.
public Vector<Vector<Object>> InfoForTheTable() {
Scanner s = null;
Vector<Vector<Object>> data = new Vector<Vector<Object>>();
try {
s = new Scanner(new File("info.txt"));
while (s.hasNextLine()) {
String line = s.nextLine();
if (line.startsWith("")) {
String[] atoms = line.split("[#]");
Vector<Object> row = new Vector<Object>();
row.add(atoms[0]);
row.add(atoms[1]);
row.add(atoms[2]);
row.add(atoms[3]);
row.add(atoms[4]);
row.add(atoms[5]);
data.add(row);
}
}
}
catch(IOException e) {
e.printStackTrace();
}
finally {
if (s != null) {
s.close();
}
}
return data;
}
Instead of
Vector<Vector<Object>> data = new Vector<Vector<Object>>();
I recommend to use:
List<List<Object>> data = new ArrayList<>();
Or even better: create a class MyInfo and assign the atoms values to its properties
Then use:
List<MyInfo> data = new ArrayList<>();
So here is a more modern version of your code:
public List<Info> readInfoFromFile() {
List<Info> infoList= new ArrayList<>();
try (Scanner s = new Scanner(Paths.get("info.txt")){
while (s.hasNextLine()) {
String line = s.nextLine();
if (line.startsWith("")) {
String[] atoms = line.split("[#]");
Info info = new Info();
info.setA(atoms[0]);
info.setB(atoms[1]);
info.setC(atoms[2]);
info.setD(atoms[3]);
info.setE(atoms[4]);
info.setG(atoms[5]);
infoList.add(rowinfo);
}
} catch(IOException e) {
e.printStackTrace();
}
return data;
}
Replace the properties and their types as needed.
I'd recommend you to don't use that type of collections, first of all the main difference between ArrayList and Vector is that all vector operations are synchhronized. Second, you have a Vector< Vector<Object> > and you want this to be ArrayList< ArrayList<Object> > So in my opinion you should create your own class (bean class).
Example with at least Java 1.7:
public class myTableModel{
private String somePropertyName1;
.
.
.
private String somePropertyNameN;
public MyTableModel(String ... array){
//assign values to instance attributes.
}
//getters and setters
}
//remember method names in java starts with lower-case
public List<MyTableModel> infoForTheTable() {
List<MyTableModel> data = new ArrayList<>(); //diamond inference
//use try-with-resources
try (Scanner s = new Scanner(Paths.getPath("info.txt"))){
while (s.hasNextLine()) {
String line = s.nextLine();
if (line.startsWith("")) {
String[] atoms = line.split("[#]");
data.add(new MyTableModel(atoms[0],atoms[1],atoms[2],atoms[3],atoms[4],atoms[5]));
}
}
}
catch(IOException e) {
//handle exception or throw it up!
}
return data;
}
Yes, ArrayList has a constructor that takes a Collection (which Vector obviously is) to do so:
ArrayList(Collection<? extends E> c)
just use the constructor which takes a collection as its parameter:
ArrayList<String> list = new ArrayList<String>(row);
Note that it only does a shallow copy.
for more details see this