I have a JSON file airports.json which contains 5720 objects, I want to parse the file in Java into Objects Airport.
The code below is how I parse it, the problem is, it is taking too much time to completely parse all the file, about 1 min and 50 sec.
ArrayList<Airport> arrayAirports = new ArrayList<>();
String json_response = loadJSONFromAsset();
try {
JSONObject airports = new JSONObject(json_response.trim());
Log.d("Length Array 0", String.valueOf(airports.names().length()));
Log.d("Length Array 1", String.valueOf(arrayAirports.size()));
for(int i = 0; i<airports.names().length(); i++){
JSONObject jsonAirport = airports.getJSONObject(airports.names().getString(i));
Airport newAirport = new Airport();
newAirport.name = jsonAirport.getString("name");
newAirport.city = jsonAirport.getString("city");
newAirport.country = jsonAirport.getString("country");
newAirport.latitude = jsonAirport.getString("latitude");
newAirport.longitude = jsonAirport.getString("longitude");
newAirport.altitude = jsonAirport.getString("altitude");
newAirport.iata = jsonAirport.getString("iata");
newAirport.icao = jsonAirport.getString("icao");
newAirport.timeZone = jsonAirport.getString("timezone");
newAirport.dst = jsonAirport.getString("dst");
arrayAirports.add(newAirport);
}
} catch (JSONException e1) {
e1.printStackTrace();
}
Log.d("Length Array 2", String.valueOf(arrayAirports.size()));
Is there a way to parse it quicklier.
By the way my friend is parsing it with no time at all using Objective C.
Link to JSON file
Use GSON instead and measure the speed with it. While it is possible possible that just reading the file could take a long time, a whole minute seems pretty bad.
Since you mentioned you don't know how to use GSON, here's a tutorial I wrote for a student on how to use GSON. It assume you're getting the file from a network call though, so you need to apply it to use your local JSON file.
Don't know if it matters for performance, but you shouldn't call names() repeatedly. Assign to variable before loop, then use it.
JSONArray names = airports.names();
Log.d("Length Array 0", String.valueOf(names.length()));
Log.d("Length Array 1", String.valueOf(arrayAirports.size()));
for(int i = 0; i<names.length(); i++){
JSONObject jsonAirport = airports.getJSONObject(names.getString(i));
// code
}
Better yet, use length() and keys():
Log.d("Length Array 0", String.valueOf(airports.length()));
Log.d("Length Array 1", String.valueOf(arrayAirports.size()));
for (Iterator<String> nameIter = airports.keys(); nameIter.hasNext(); ){
String name = nameIter.next();
JSONObject jsonAirport = airports.getJSONObject(name);
// code
}
You are putting a (I guess?) large JSON data file into a JSONObject directly.
In this case it would be recommended to use a token based reader, such as JsonReader.
Pasted directly from the docs:
public List readJsonStream(InputStream in ) throws IOException {
JsonReader reader = new JsonReader(new InputStreamReader( in , "UTF-8"));
try {
return readMessagesArray(reader);
finally {
reader.close();
}
}
public List readMessagesArray(JsonReader reader) throws IOException {
List messages = new ArrayList();
reader.beginArray();
while (reader.hasNext()) {
messages.add(readMessage(reader));
}
reader.endArray();
return messages;
}
public Message readMessage(JsonReader reader) throws IOException {
long id = -1;
String text = null;
User user = null;
List geo = null;
reader.beginObject();
while (reader.hasNext()) {
String name = reader.nextName();
if (name.equals("id")) {
id = reader.nextLong();
} else if (name.equals("text")) {
text = reader.nextString();
} else if (name.equals("geo") && reader.peek() != JsonToken.NULL) {
geo = readDoublesArray(reader);
} else if (name.equals("user")) {
user = readUser(reader);
} else {
reader.skipValue();
}
}
reader.endObject();
return new Message(id, text, user, geo);
}
public List readDoublesArray(JsonReader reader) throws IOException {
List doubles = new ArrayList();
reader.beginArray();
while (reader.hasNext()) {
doubles.add(reader.nextDouble());
}
reader.endArray();
return doubles;
}
public User readUser(JsonReader reader) throws IOException {
String username = null;
int followersCount = -1;
reader.beginObject();
while (reader.hasNext()) {
String name = reader.nextName();
if (name.equals("name")) {
username = reader.nextString();
} else if (name.equals("followers_count")) {
followersCount = reader.nextInt();
} else {
reader.skipValue();
}
}
reader.endObject();
return new User(username, followersCount);
}
}
Related
So currently, my code iterates over huge amount of files (.ndjson.gzip).
During the iteration, each file converted into string and this string inserted to an hashset.
My Originally idea was to stream all files, so that the end user could manipulate this stream as he wishes.
Also, it is a huge chunk of data, so I though, it will be much faster to stream all the files one after the other.
1.How can I implement my idea?
2.For example, how can I return from a function ( getAllFiles()) a stream, so that I could print the content?
public class ServiceGzipped {
HashSet<String> getAllFiles() throws IOException {
HashSet<String> hashSet = new HashSet<>();
ListObjectsV2Request request = new ListObjectsV2Request().withBucketName("some-bucket-name").withPrefix("some-prefix");
ListObjectsV2Result result;
do {
result = client.listObjectsV2(request);
for (S3ObjectSummary summary : result.getObjectSummaries()) {
System.out.println(summary.getKey() + " : " + summary.getSize());
String s = downloadFromAWS(summary.getKey());
hashSet.add(s);
}
String token = result.getNextContinuationToken();
System.out.println(token);
request.setContinuationToken(token);
} while (result.isTruncated());
return hashSet ;
}
String downloadFromAWS(String file) {
S3Object s3Object = client.getObject(bucketName, file);
String output = "";
try {
GZIPInputStream gzipInputStream= new GZIPInputStream(s3Object.getObjectContent());
InputStreamReader reader = new InputStreamReader(gzipInputStream);
BufferedReader in = new BufferedReader(reader);
String readed;
while ((readed = in.readLine())!= null){
System.out.println(readed);
output +=readed;
}
return output;
}catch (Exception e){
System.out.println(e);
}
return null;
}
Thanks in advance for your help :)
Hi I am trying to convert a CSV file into a JSON array using A dependency called csvReader, but when I run the code it prints out the JSON response incorrectly and I ament sure why would anyone be able to point me in the right direction.
#GetMapping("/convert")
public List<List<String>> convertCSV() throws FileNotFoundException {
List<List<String>> records = new ArrayList<List<String>>();
try (CSVReader csvReader = new CSVReader(new FileReader("C:/Download/cities.csv"));) {
String[] values = null;
while ((values = csvReader.readNext()) != null) {
records.add(Arrays.asList(values));
}
} catch (IOException e) {
e.printStackTrace();
}
return values;
}
Your case is not a big deal.
You can read that csv and build json.
Read first row and determine columns. The rest of rows are values.
public class Foo{
public static void main(String[] args) throws Exception{
List<String> csvRows = null;
try(var reader = Files.lines(Paths.get("dataFile.csv"))){
csvRows = reader.collect(Collectors.toList());
}catch(Exception e){
e.printStackTrace();
}
if(csvRows != null){
String json = csvToJson(csvRows);
System.out.println(json);
}
}
public static String csvToJson(List<String> csv){
//remove empty lines
//this will affect permanently the list.
//be careful if you want to use this list after executing this method
csv.removeIf(e -> e.trim().isEmpty());
//csv is empty or have declared only columns
if(csv.size() <= 1){
return "[]";
}
//get first line = columns names
String[] columns = csv.get(0).split(",");
//get all rows
StringBuilder json = new StringBuilder("[\n");
csv.subList(1, csv.size()) //substring without first row(columns)
.stream()
.map(e -> e.split(","))
.filter(e -> e.length == columns.length) //values size should match with columns size
.forEach(row -> {
json.append("\t{\n");
for(int i = 0; i < columns.length; i++){
json.append("\t\t\"")
.append(columns[i])
.append("\" : \"")
.append(row[i])
.append("\",\n"); //comma-1
}
//replace comma-1 with \n
json.replace(json.lastIndexOf(","), json.length(), "\n");
json.append("\t},"); //comma-2
});
//remove comma-2
json.replace(json.lastIndexOf(","), json.length(), "");
json.append("\n]");
return json.toString();
}
}
Tested on:
fname,lname,note
Shaun,Curtis,a
Kirby,Beil,b
-----------------------
[
{
"fname" : "Shaun",
"lname" : "Curtis",
"note" : "a"
}, {
"fname" : "Kirby",
"lname" : "Beil",
"note" : "b"
}
]
This method work on any structure of csv. Don't need to map columns.
That is because of your reading data in String and printing the List of String. If you want to map the CSV to Object ( JSON Object), You need to read the CSV as bean object please find below code snippet to print as JSON, override toString method as JSON format.
User.java
public class User {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#NotNull
private String name;
#NotNull
private String surname;
//Getter and Setters
}
CsvReaderUtil.java
public static List<User> readCsvFile() throws IOException {
List<User> list = null;
CSVReader reader = null;
InputStream is = null;
try {
File initialFile = new File("C:\\Users\\ER\\Desktop\\test.csv");
is = new FileInputStream(initialFile);
reader = new CSVReader(new InputStreamReader(is), ',', '"', 1);
ColumnPositionMappingStrategy strat = new ColumnPositionMappingStrategy();
strat.setType(User.class);
String[] columns = new String[]{"id", "name", "surname"};
strat.setColumnMapping(columns);
CsvToBean csv = new CsvToBean();
list = csv.parse(strat, reader);
} catch (Exception e) {
e.printStackTrace();
} finally {
is.close();
reader.close();
}
return list;
}
Now print this List Of Users as a JSON object.
Here is a useful example of how to transform CSV to JSON using Java 11+:
private String fromCsvToJson(String csvFile) {
String[] lines = file.split("\n");
if (lines.length <= 1) {
return List.of();
}
var headers = lines[0].split(",");
var jsonFormat = Arrays.stream(lines)
.skip(1)
.map(line -> line.split(","))
.filter(line -> headers.length == line.length)
.map(line -> IntStream.range(0, headers.length).boxed().collect(toMap(i -> headers[i], i -> line[i], (a, b) -> b)))
.toList();
return new ObjectMapper().writeValueAsString(jsonFormat);
}
I am reading multiple JSONObject from a file and converting into a string using StringBuilder.
These are the JSON Objects.
{"Lng":"-1.5908601","Lat":"53.7987816"}
{"Lng":"-2.5608601","Lat":"54.7987816"}
{"Lng":"-3.5608601","Lat":"55.7987816"}
{"Lng":"-4.5608601","Lat":"56.7987816"}
{"Lng":"-5.560837","Lat":"57.7987816"}
{"Lng":"-6.5608294","Lat":"58.7987772"}
{"Lng":"-7.5608506","Lat":"59.7987823"}
How to convert into a string?
Actual code is:-
BufferedReader reader = new BufferedReader(new InputStreamReader(contents.getInputStream()));
StringBuilder builder = new StringBuilder();
String line;
try {
while ((line = reader.readLine()) != null) {
builder.append(line);
}
}
catch(IOException e)
{
msg.Log(e.toString());
}
String contentsAsString = builder.toString();
//msg.Log(contentsAsString);
I tried this code
JSONObject json = new JSONObject(contentsAsString);
Iterator<String> iter = json.keys();
while(iter.hasNext())
{
String key = iter.next();
try{
Object value = json.get(key);
msg.Log("Value :- "+ value);
}catch(JSONException e)
{
//error
}
}
It just gives first object. How to loop them?
try this and see how it works for you,
BufferedReader in
= new BufferedReader(new FileReader("foo.in"));
ArrayList<JSONObject> contentsAsJsonObjects = new ArrayList<JSONObject>();
while(true)
{
String str = in.readLine();
if(str==null)break;
contentsAsJsonObjects.add(new JSONObject(str));
}
for(int i=0; i<contentsAsJsonObjects.size(); i++)
{
JSONObject json = contentsAsJsonObjects.get(i);
String lat = json.getString("Lat");
String lng = json.getString("Lng");
Log.i("TAG", lat + lng)
}
What you do is you are loading multiple JSON objects into one JSON object. This does not make sense -- it is logical that only the first object is parsed, the parser does not expect anything after the first }. Since you want to loop over the loaded objects, you should load those into a JSON array.
If you can edit the input file, convert it to the array by adding braces and commas
[
{},
{}
]
If you cannot, append the braces to the beginning of the StringBuilder and append comma to each loaded line. Consider additional condition to eliminate exceptions caused by inpropper input file.
Finally you can create JSON array from string and loop over it with this code
JSONArray array = new JSONArray(contentsAsString);
for (int i = 0; i < array.length(); ++i) {
JSONObject object = array.getJSONObject(i);
}
public static void main(String[] args) throws IOException {
try {
JSONObject json = new JSONObject(readUrl("http://api.wunderground.com" +"/api/106c4dee47162999/history_20060405/q/CA/San_Francisco.json"));
JSONObject data = json.getJSONObject("observations.tempm");
System.out.println(data);
} catch (JSONException e) {
e.printStackTrace();
}
}
private static String readUrl(String string) throws IOException {
BufferedReader reader = null;
try {
String urlString = string;
URL url = new URL(urlString);
reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer buffer = new StringBuffer();
int read;
char[] chars = new char[1024];
while ((read = reader.read(chars)) != -1)
buffer.append(chars, 0, read);
return buffer.toString();
} finally {
if (reader != null)
reader.close();
}
}
}
2.errors :
org.json.JSONException: JSONObject["observations.tempm"] not found.
at org.json.JSONObject.get(JSONObject.java:454)
at org.json.JSONObject.getJSONObject(JSONObject.java:553)
at com.parser.ParserObject.main(ParserObject.java:17)
The intersting part of the JSON you use is:
{'history':
{'observations':
[
{'tempm':'10.0'}
]
}
}
Use this:
JSONObject json = ...;
JSONObject history = (JSONObject) json.get("history");
JSONArray observations = (JSONArry) json.get("observations");
JSONObject observation0 = (JSONObject) observations.get(0);
String tempm = observation0.get("tempm");
If you are interested in other array elements, use a different index.
'observations' isnt in root. you need to get 'history' first. 'observations' is a part of 'history'. and 'observations' is a list/array.
i suggest using 'jackson' or 'gson' library.parsing native json is really a pain.
you can even go to a extent of un/marshalling only required fields/objects from JSON.
I have to read line by line an Xml file in java.
The file has lines of the format :
<CallInt xsi:type="xsd:int">124</CallInt>
I need to pick up only tag name CallInt and the value 124 from the above line.
I tried using String Tokenizer, Split etc. But nothing to the rescue.
Can anyone help me with this?
Some code
BufferedReader buf = new BufferedReader(new FileReader(myxmlfile));
while((line = buf.readLine())!=null)
{
String s = line;
// Scanning for the tag and the integer value code???
}
You should really use a small xml parser.
If you have to read line-by-line, and the format is guaranteed to be line-based, search for delimiters around the content you want to extract with indexOf() and then use substring()...
int cut0 = line.indexOf('<');
if (cut0 != -1) {
int cut1 = line.indexOf(' ', cut0);
if (cut1 != -1) {
String tagName = line.substring(cut0 + 1, cut1);
int cut2 = line.indexOf('>', cut1); // insert more ifs as needed...
int cut3 = line.indexOf('<', cut2);
String value = line.substring(cut2 + 1, cut2);
}
}
Here's a small example with StaX.
Note I've removed the reference to the schema for simplicity (it'll fail as is otherwise).
XML file called "test", in path "/your/path"
<thingies>
<thingie foo="blah"/>
<CallInt>124</CallInt>
</thingies>
Code
XMLInputFactory factory = null;
XMLStreamReader reader = null;
// code is Java 6 style, no try with resources
try {
factory = XMLInputFactory.newInstance();
// coalesces all characters in one event
factory.setProperty(XMLInputFactory.IS_COALESCING, true);
reader = factory.createXMLStreamReader(new FileInputStream(new File(
"/your/path/test.xml")));
boolean readCharacters = false;
while (reader.hasNext()) {
int event = reader.next();
switch (event) {
case (XMLStreamConstants.START_ELEMENT): {
if (reader.getLocalName().equals("CallInt")) {
readCharacters = true;
}
break;
}
case (XMLStreamConstants.CHARACTERS): {
if (readCharacters) {
System.out.println(reader.getText());
readCharacters = false;
}
break;
}
}
}
}
catch (Throwable t) {
t.printStackTrace();
}
finally {
try {
reader.close();
}
catch (Throwable t) {
t.printStackTrace();
}
}
Output
124
Here is an interesting SO thread on schemas and StaX.