Develop Reference

How do I map a heterogeneous JSON array to a Java object?

I've downloaded a large amount of historic crypto market data via an API. It is formatted like this:
[
[1601510400000,"4.15540000","4.16450000","4.15010000","4.15030000","4483.01000000",1601510459999,"18646.50051400",50,"2943.27000000","12241.83706500","0"],
...
[1609490340000,"4.94020000","4.95970000","4.93880000","4.94950000","5307.62000000",1609490399999,"26280.03711000",98,"3751.46000000","18574.22402400","0"]
]
I take that to be an array of arrays, the inner one containing heterogeneous types (always the same types in the same order). As an intermediate step I've saved it to text files but I'd like to read it back and map it onto an array of objects of this type:
public class MinuteCandle {
private long openTime;
private double openValue;
private double highValue;
private double lowValue;
private double closeValue;
private double volume;
private long closeTime;
private double quoteAssetVolume;
private int numberOfTrades;
private double takerBuyBaseAssetVolume;
private double takerBuyQuoteAssetVolume;
private double someGarbageData;
//...
}
I'm using the Spring Framework and the jackson library for json mapping. Is it doable with that or should I manually parse the text somehow?

Use JsonFormat and annotate your class with it where you specify shape as an ARRAY:
#JsonFormat(shape = JsonFormat.Shape.ARRAY)
class MinuteCandle
Also, consider to use BigDecimal instead of double if you want to store a price.
See also:
A realistic example where using BigDecimal for currency is strictly
better than using double
How to deserialise anonymous array of mixed types with Jackson

I would do this in two steps:
Read the JSON content into a list of List<Object> with Jackson.
Convert each List<Object> into a MinuteCandle object
and collect these objects into a list of MinuteCandles.
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
public class Main {
public static void main(String[] args) throws Exception {
ObjectMapper objectMapper = new ObjectMapper();
File file = new File("example.json");
List<List<Object>> lists = objectMapper.readValue(file, new TypeReference<List<List<Object>>>() {});
List<MinuteCandle> minuteCandles = new ArrayList<>();
for (List<Object> list : lists) {
minuteCandles.add(MinuteCandle.createFromList(list));
}
}
}
The conversion from List<Object> to MinuteCandle (step 2 from above)
could be achieved by adding a static method in your MinuteCandle class.
public static MinuteCandle createFromList(List<Object> list) {
MinuteCandle m = new MinuteCandle();
m.openTime = (Long) list.get(0);
m.openValue = Double.parseDouble((String) list.get(1));
m.highValue = Double.parseDouble((String) list.get(2));
m.lowValue = Double.parseDouble((String) list.get(3));
m.closeValue = Double.parseDouble((String) list.get(4));
m.volume = Double.parseDouble((String) list.get(5));
m.closeTime = (Long) list.get(6);
m.quoteAssetVolume = Double.parseDouble((String) list.get(7));
m.numberOfTrades = (Integer) list.get(8);
m.takerBuyBaseAssetVolume = Double.parseDouble((String) list.get(9));
m.takerBuyQuoteAssetVolume = Double.parseDouble((String) list.get(10));
m.someGarbageData = Double.parseDouble((String) list.get(11));
return m;
}

Assuming the text stored in the file is valid JSON, similar to the solution in How to Read JSON data from txt file in Java? one can use com.google.gson.Gson as follows :
import com.google.gson.Gson;
import java.io.FileReader;
import java.io.Reader;
public class Main {
public static void main(String[] args) throws Exception {
try (Reader reader = new FileReader("somefile.txt")) {
Gson gson = new Gson();
MinuteCandle[] features = gson.fromJson(reader, MinuteCandle[].class);
}
}
}

JSONObject to ArrayList square brackets missing

I am trying to get the rates from https://api.ratesapi.io/api/latest into an ArrayList<Currency> of a custom Currency class:
public class Currency {
private String shortName;
private double rate;
...
}
The JSON looks like:
{"base":"EUR","rates":{"GBP":0.90033,"HKD":9.1786,"IDR":17304.0,
"ILS":4.0309,"DKK":7.45,"INR":88.765,"CHF":1.0759,"MXN":26.615,
"CZK":26.202,"SGD":1.6236,"THB":36.832,"HRK":7.468,"MYR":4.9604,
"NOK":10.6538,"CNY":8.2325,"BGN":1.9558,"PHP":58.136,"SEK":10.3165,
"PLN":4.4073,"ZAR":20.7655,"CAD":1.5748,"ISK":160.2,"BRL":6.334,
"RON":4.836,"NZD":1.7828,"TRY":8.5853,"JPY":124.96,"RUB":86.9321,
"KRW":1404.99,"USD":1.1843,"HUF":346.23,"AUD":1.6492},"date":"2020-08-06"}
Using org.json I managed to get the data into a JSONObject:
JSONObject obj = new JSONObject(getJSON("https://api.ratesapi.io/api/latest"));
As far as I understand, the normal procedure is now to convert the JSONObject into a JSONArray. However trying:
JSONArray jsonArray = obj.getJSONArray("rates");
fails with the error message:
Exception in thread "main" org.json.JSONException: JSONObject["rates"]
is not a JSONArray.
How do I fix this error or is there another way to make an ArrayList out of the JSON?
I suspect that the problem are missing square brackets in the JSON string.

If you take a look at the JSON returned by the API, you get a JSON object:
{"base":"EUR","rates":{"GBP":0.90033,"HKD":9.1786, ... },"date":"2020-08-06"}
You probably want to do something like this:
JSONObject obj = new JSONObject(getJSON("https://api.ratesapi.io/api/latest"));
JSONObject rates = obj.getJSONObject("rates");
final Iterator<String> keys = rates.keys();
while (keys.hasNext()) {
final String key = keys.next();
final Currency currency = new Currency(key, rates.getDouble(key));
// do something with the Currency
}

The object "rates" is not a JSONArray, is a JSONObject.
So you have to do obj.getJSONObject(rates");then iterate on the fields of the JSONObject using map methods (for examply using keySet() )

A working solution using Jackson library and Lombok may be as follows:
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import lombok.*;
import java.util.*;
import java.util.stream.Collectors;
public class CcyApiParser {
#Getter
#Setter
#AllArgsConstructor
#NoArgsConstructor
#ToString
public static class Currency {
private String shortName;
private double rate;
}
#Getter
#Setter
public static class RatesApiResponse {
private String base;
private Map<String, Double> rates;
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd")
private LocalDate date;
}
public static void main(String[] args) throws IOException {
ObjectMapper mapper = new ObjectMapper()
.registerModule(new JavaTimeModule()); // to parse date
URL apiUrl = new URL("https://api.ratesapi.io/api/latest");
// read proper api response
RatesApiResponse rates = mapper.readValue(apiUrl, RatesApiResponse.class);
// convert inner rates into list of Currency objects
List<Currency> ccys = rates.getRates().entrySet().stream()
.map(e -> new Currency(e.getKey(), e.getValue()))
.collect(Collectors.toList());
ccys.forEach(ccy -> System.out.printf("%s=%s%n", ccy.getShortName(), ccy.getRate()));
}
}
Output
GBP=0.90033
HKD=9.1786
IDR=17304.0
ILS=4.0309
... etc.
Update
It is also possible to customize deserialization of RatesApiResponse and move mapping of "rates" into this class to convert immediately into list of currencies.
#Getter
#Setter
public static class RatesApiResponse {
private String base;
#JsonProperty(access = JsonProperty.Access.READ_ONLY)
private List<Currency> ccys;
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd")
private LocalDate date;
// no getter for rates
// this customized setter for the map of rates converts into a list
#JsonProperty("rates")
public void setRates(Map<String, Double> rates) {
ccys = rates.entrySet().stream()
.map(e -> new Currency(e.getKey(), e.getValue()))
.collect(Collectors.toList());
}
}
// Updates in the test method
RatesApiResponse rates = mapper.readValue(src, RatesApiResponse.class);
rates.getCcys().forEach(ccy -> System.out.printf("%s=%s%n", ccy.getShortName(), ccy.getRate()));

You can use ObjectMapper class to convert json from some URL to some kind of object. In this case (if json structure is always the same) it can be Map<String, Object>.
ObjectMapper mapper = new ObjectMapper();
URL url = new URL("https://api.ratesapi.io/api/latest");
Map<String, Object> map = mapper.readValue(url, Map.class);
System.out.println(map);
// {base=EUR, rates={GBP=0.90373, HKD=9.1585, ... , AUD=1.6403}, date=2020-08-07}
Then you can get inner rates map, and (if it is needed) convert it to list using java stream api:
Map<String, Double> rates = (Map<String, Double>) map.get("rates");
System.out.println(rates); // {GBP=0.90373, HKD=9.1585, ... , AUD=1.6403}
Convert Map<String, Object> to ArrayList<Currency>:
ArrayList<Currency> list = rates.entrySet().stream()
.map(entry -> new Currency(entry.getKey(), entry.getValue()))
.collect(ArrayList::new, ArrayList::add, ArrayList::addAll);
System.out.println(list); // [GBP=0.90373, HKD=9.1585, ... , AUD=1.6403]
Note: add a constructor with two fields shortName and rate;
Note: override the toString method as follows: shortName + "=" + rate;
Maven dependency:
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
<version>2.11.2</version>
</dependency>
See also: «Formatting Json Response into an Array Java».

Exception in thread "main" org.json.JSONException: JSONObject["rates"]
is not a JSONArray.
You got this error because rates is not in the form of an array. It is simply an element like base and date but looks like an array. Get it from the JSON string like you get base and date from it and then process it to create the required List<Currency>.
Given below is the working code with the explanation added as comments in the code:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
import org.json.JSONException;
import org.json.JSONObject;
class Currency {
private String shortName;
private double rate;
public Currency(String shortName, double rate) {
this.shortName = shortName;
this.rate = rate;
}
#Override
public String toString() {
return shortName + ":" + rate;
}
}
public class Main {
public static JSONObject getJSON(String url) throws IOException, JSONException {
// Create a URLConnection for the given URL
URLConnection connection = new URL(url).openConnection();
// Add header to avoid 403 Forbidden HTTP status code
connection.addRequestProperty("User-Agent",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0" + "");
StringBuilder jsonStr = new StringBuilder();
// Get InputStream from connection and read the response
try (InputStream is = connection.getInputStream();) {
Reader reader = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
int ch;
while ((ch = reader.read()) != -1) {
jsonStr.append((char) ch);
}
}
return new JSONObject(jsonStr.toString());
}
public static void main(String[] args) throws IOException, JSONException {
JSONObject jsonObj = getJSON("https://api.ratesapi.io/api/latest");
// Get rates from jsonObj
String rates = jsonObj.get("rates").toString();
// Remove {, }, and " from the string
String[] keyValArr = rates.replaceAll("[\\{\\\"}]", "").split(",");
// List object to hold Currency objects
List<Currency> list = new ArrayList<>();
for (String keyVal : keyValArr) {
// Split each key:value string on ':'
String[] curRate = keyVal.split(":");
// Add Currency object to List
list.add(new Currency(curRate[0], Double.parseDouble(curRate[1])));
}
// Display list
list.forEach(System.out::println);
}
}
Output:
CHF:1.0804
HRK:7.4595
MXN:26.5127
...
...
...
NZD:1.7786
BRL:6.3274

is there a way to use gson well to get an list with arrays that have 4 variables in java

I'm making an 3D engine using lwjgl.
I have tried to make a class to using a list of HashMap but the HashMap only accepts 2 variables so that does not work.
Part of my code for getting the JSON file
Gson().fromJson(string.toString(), BlockIndexFile.class);
the BlockIndexFile class
public class BlockIndexFile {
List<HashMap<String, String>> blocks = new ArrayList<HashMap<String, String>>();
public void setBlocks(List<HashMap<String, String>> blocks) {
this.blocks = blocks;
}
public List<HashMap<String, String>> getBlocks(){
return this.blocks;
}
}
and the json file
{
"blocks":
[
{
"name": "Foo",
"id": "foo",
"model": "cube1",
"texture": "foo"
}
]
}
I expected to be able to use a HashMap to get the id and then use that to get the other variables like the texture and model.

HashMap can contain more than 2 variables. See below example how you could use it:
import java.io.File;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
public class GsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
Gson gson = new GsonBuilder().create();
BlockIndexFile blockIndexFile;
try (FileReader fileReader = new FileReader(jsonFile)) {
blockIndexFile = gson.fromJson(fileReader, BlockIndexFile.class);
}
HashMap<String, String> node0 = blockIndexFile.getBlocks().get(0);
System.out.println("id => " + node0.get("id"));
System.out.println("model => " + node0.get("id"));
System.out.println("texture => " + node0.get("id"));
}
}
Above code prints:
id =>foo
model =>foo
texture =>foo
Instead Map you can create POJO and code should be much easier and concise:
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import java.io.File;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
public class GsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
Gson gson = new GsonBuilder().create();
BlockIndexFile blockIndexFile;
try (FileReader fileReader = new FileReader(jsonFile)) {
blockIndexFile = gson.fromJson(fileReader, BlockIndexFile.class);
}
Block node0 = blockIndexFile.getBlocks().get(0);
System.out.println(node0);
}
}
class BlockIndexFile {
private List<Block> blocks = new ArrayList<>();
// getters, setters
}
class Block {
private String id;
private String name;
private String model;
private String texture;
// getters, setters, toString
}
Above code prints:
Block{id='foo', name='Foo', model='cube1', texture='foo'}

how to convert text file to parquet with java spark

I'm trying to convert a text file into a parquet file. I can only find "how to convert to parquet" from other file format or code written in scala/python.
Here is what I came up with
import org.apache.parquet.schema.MessageType;
import org.apache.parquet.schema.MessageTypeParser;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.*;
private static final StructField[] fields = new StructField[]{
new StructField("timeCreate", DataTypes.StringType, false, Metadata.empty()),
new StructField("cookieCreate", DataTypes.StringType, false,Metadata.empty())
};//simplified
private static final StructType schema = new StructType(fields);
public static void main(String[] args) throws IOException {
SparkSession spark = SparkSession
.builder().master("spark://levanhuong:7077")
.appName("Convert text file to Parquet")
.getOrCreate();
spark.conf().set("spark.executor.memory", "1G");
WriteParquet(spark, args);
}
public static void WriteParquet(SparkSession spark, String[] args){
JavaRDD<String> data = spark.read().textFile(args[0]).toJavaRDD();
JavaRDD<Row> output = data.map((Function<String, Row>) s -> {
DataModel model = new DataModel(s);
return RowFactory.create(model);
});
Dataset<Row> df = spark.createDataFrame(output.rdd(),schema);
df.printSchema();
df.show(2);
df.write().parquet(args[1]);
}
args[0] is a path to input file, args[1] is a path to the output file. here is the simplified DataModel. DateTime fields are properly formated in set() function
public class DataModel implements Serializable {
DateTime timeCreate;
DateTime cookieCreate;
public DataModel(String data){
String model[] = data.split("\t");
setTimeCreate(model[0]);
setCookieCreate(model[1]);
}
And here is the error. Error log point to df.show(2) but i think the error was caused by map(). I'm not sure why since I don't see any casting in the code
>java.lang.ClassCastException: cannot assign instance of
java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.fun$1
of type org.apache.spark.api.java.function.Function in instance
of org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1
I think this is enough to recreate the error, please tell me if I need to provide any more information.

A little bit other approach can be used, working fine:
JavaRDD<String> data = spark().read().textFile(args[0]).toJavaRDD();
JavaRDD<DataModel> output = data.map(s -> {
String[] parts = s.split("\t");
return new DataModel(parts[0], parts[1]);
});
Dataset<Row> result = spark().createDataFrame(output, DataModel.class);
Class "DataModel" is better looks as simple TO, without functionality:
public class DataModel implements Serializable {
private final String timeCreate;
private final String cookieCreate;
public DataModel(String timeCreate, String cookieCreate) {
this.timeCreate = timeCreate;
this.cookieCreate = cookieCreate;
}
public String getTimeCreate() {
return timeCreate;
}
public String getCookieCreate() {
return cookieCreate;
}
}

JSON text to Java conversion issue with random number generation

For example my JSON text is coming like this.
"pages":{"42010":{"pageid":42010,"ns":0,"title":"Queen (band)"}}
Because everytime my json text is coming with different number which is inside pages tag.
How do i convert this to Java equivalent class?
Currently my generated java class is something like this.
#Generated("org.jsonschema2pojo")
public class Pages {
#SerializedName("42010")
#Expose
private _42010 _42010;
}
That _42010 class contains the inner fields like "pageid":42010,"ns":0,"title":"Queen (band)", since i am getting everytime new number inside pages, its not working. its working only for the specific json text.

You can use a custom deserialiser that ignored the changing number. For example:
package jacksonTest;
import java.io.IOException;
import java.lang.reflect.Type;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonDeserializationContext;
import com.google.gson.JsonDeserializer;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
public class CustomDeserialiser {
public static void main(String[] args) throws JsonParseException, JsonMappingException, IOException {
String json = "{\"42010\":{\"pageid\":42010,\"ns\":0,\"title\":\"Queen (band)\"}}";
String json2 = "{\"12345\":{\"pageid\":12345,\"ns\":0,\"title\":\"Queen (band)\"}}";
Gson g = new GsonBuilder().registerTypeAdapter(Pages.class, new PagesDeserialiser()).create();
Pages fromJson = g.fromJson(json, Pages.class);
System.out.println(fromJson);
fromJson = g.fromJson(json2, Pages.class);
System.out.println(fromJson);
}
public static class PagesDeserialiser implements JsonDeserializer<Pages> {
#Override
public Pages deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context)
throws com.google.gson.JsonParseException {
JsonObject object = json.getAsJsonObject();
Pages p = new Pages();
object.entrySet().forEach( e -> {
JsonObject tmp = e.getValue().getAsJsonObject();
if(tmp.get("pageid") != null) {
// right object
p._42010 = new _42010();
p._42010.ns = tmp.get("ns").getAsInt();
p._42010.pageid = tmp.get("pageid").getAsInt();
p._42010.title = tmp.get("title").getAsString();
}
});
return p;
}
}
public static class Pages {
_42010 _42010;
#Override
public String toString() {
return _42010.toString();
}
}
public static class _42010 {
int pageid;
int ns;
String title;
#Override
public String toString() {
return title + " " + pageid + " " + ns;
}
}
}
The deserialiser for type pages simply checks the entries to find the one that contains a pageId and then populates the class.
Running my test gives you:
Queen (band) 42010 0
Queen (band) 12345 0
I am assuming that you are using Gson as your json library.
Regards,
Artur

Why do not use an JSON library like jackson or org.json?
Make your json correct like
{
"pages":{
"42010":{
"pageid":42010,
"ns":0,
"title":"Queen (band)"
}
}
}
And you will be able to use it like :
JSONObject jsonObjet = new JSONObject(yourJson);
jsonObjet.get("pages");

Ideally it should be using Map.
This helps in forming the values as Map<Integer, Pojo>.
Lets say
public class Pojo{
private int pageid;
private String title;
private int ns;
// getter and setter
}
This suffices the requirement of holding the random digits, generated at runtime,