Develop Reference

Java Jackson how to parse large REST API JSON?

Api I'm trying to get info from https://prices.runescape.wiki/api/v1/osrs/latest
HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
.GET()
.uri(URI.create("https://prices.runescape.wiki/api/v1/osrs/latest"))
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
ObjectMapper mapper = new ObjectMapper();
JsonNode json = mapper.readTree(response.body());
JsonNode data = json.get("data");
List<Data> items = new ArrayList<>();
data.forEach(item -> {
Data d = new Data(
item.get("high").asInt(),
item.get("highTime").asInt(),
item.get("low").asInt(),
item.get("lowTime").asInt()
);
items.add(d);
});
Problem is the object is the itemID. So if I want the item id 6. I can't do it since it's not an attribute of the object.
"2":{"high":164,"highTime":1672078170,"low":160,"lowTime":1672078164} ``
2 is the itemID and the object.
Below is when I loop through the arraylist and print it out
Data{high=157, highTime=1672071161, low=150, lowTime=1672071151}
Data{high=187987, highTime=1672071066, low=182005, lowTime=1672070881}
Data{high=189903, highTime=1672071052, low=186820, lowTime=1672070884}
Data{high=190000, highTime=1672070957, low=184882, lowTime=1672070984}

Your JSON has the following structure:
{
"data":{
"2":{ ... },
"6":{ ... },
"8":{ ... },
"10":{ ... },
"12":{ ... }
...
}
}
And property "data" is associated not with an object, or a List<Data>, but with a Map<Integer,Data> (or Map<String,Data>.
There are several ways how you can parse:
You can define an object with a single property Map<Integer,Data> data.
public static class DataWrapper {
private Map<Integer, Data1> data;
// getters, setters
}
Usage example:
String json = """
{
"data":{ ... }
}
""";
ObjectMapper mapper12 = new ObjectMapper();
DataWrapper dataWrapper = mapper12.readValue(json12, DataWrapper.class);
Map<Integer, Data1> dataMap = dataWrapper.getData();
System.out.println("item 6: " + dataMap.get(6));
Another approach would be to create a JsonNode by parsing the given JSON, access the node mapped to the property "data" and parse it as a Map using ObjectReader and its method ObjectReader.readValue(). To generate ObjectReader we can make use of the method ObjectMapper.readerFor() which expects a TypeReference.
That's how the second approach might be implemented:
String json = """
{
"data":{ ... }
}
""";
ObjectMapper mapper = new ObjectMapper();
JsonNode tree = mapper.readTree(json);
ObjectReader reader = mapper.readerFor(new TypeReference<Map<Integer, Data>>() {});
Map<Integer, Data> itemsMap = reader.readValue(tree.get("data"));
System.out.println("item 6: " + itemsMap.get(6));
Output:
item 6: Data{high=195500, highTime=1672079035, low=182009, lowTime=1672078518}

How to model JSON with key value pairs?

This simple JSON is returned by https://httpbin.org/get which is handy for testing.
{
"args": {},
"headers": {
"Content-Length": "0",
"Host": "httpbin.org",
"User-Agent": "AemSConnector v1.0",
"X-Amzn-Trace-Id": "Root=1-606c333f-338353e14fc31e375617f4ba"
},
"origin": "81.40.159.142",
"url": "https://httpbin.org/get"
}
I'm trying to figure out how to build a Java class to model this.
I have tried:
public class ModelTest {
public String origin;
public String url;
public HashMap<String, String> headers;
public HashMap<String, String> args;
// getters and setters and default constructor here...
}
}
And also just this:
public class ModelTest {
public String origin;
public String url;
// getters and setters and default constructor here...
}
}
But when I try to convert the JSON string to this model, I just get a null point exception in the logs, no helpful info.
The code I am using is this:
// ModelTest model = null;
ModelTest model = new ModelTest();
model = (ModelTest) getObjectFromJson(reply, model);
}
:
public static Object getObjectFromJson(String jsonString, Object obj) {
Gson gson = new Gson();
Object returnValue = null;
try {
returnValue = gson.fromJson(jsonString, obj.getClass());
} catch (Exception e) {
log.error("Exception occured in Something :: getObjectFromJson --> ", e);
}
return returnValue;
}
exception:
2021-04-06 12:09:04.245 ERROR [com.adobe.aem.guides.wknd.core.util.MyConnector] Exception occured in Something :: getObjectFromJson -->
java.lang.NullPointerException: null
at com.adobe.aem.guides.wknd.core.util.SpineConnector.getObjectFromJson(MyConnector.java:77) [aem-guides-wknd.core:0.2.1.SNAPSHOT]
at com.adobe.aem.guides.wknd.core.util.SpineConnector.get(MyConnector.java:50) [aem-guides-wknd.core:0.2.1.SNAPSHOT]
at com.adobe.aem.guides.wknd.core.servlets.SpineServlet.doGet(MyServlet.java:64) [aem-guides-wknd.core:0.2.1.SNAPSHOT]

I found a solution. The Model was fine, it was the getObjectFromJson method which was causing the issues, even with a non-null object (as the commenters pointed out)
I scrapped it, and did the mapping in-line and it worked as expected:
Gson gson = new Gson();
model = gson.fromJson(reply, ModelTest.class);

Mapping a trimmed response using ObjectMapper Or wrapping the response obtained

Details ---
One of my POJO SomeResponseObject for an api response has attribute
#JsonProperty("s_summary")
private Map<String, SummaryObject> summary
which further has few more attributes. These are summed in json as follows :
{
"s_summary": {
"rewardSubscription": {
"accountId": "XYZ",
"startDate": "2015-12-29T19:00:00+05:30",
"endDate": "2017-06-21T00:00:00+05:30",
"isActive": true,
"entityId": "ID123",
"status": "ACTIVE"
}
}
}
This POJO(json) is further modified by our service to return a RESPONSE as :
{
"rewardSubscription": {
"accountId": "XYZ",
"startDate": "2015-12-29T19:00:00+05:30",
"endDate": "2017-06-21T00:00:00+05:30",
"isActive": true,
"entityId": "ID123",
"status": "ACTIVE"
}
}
Narrowing Down ---
Now when we are writing tests against this API call. We end up being unable to map the response to any specific POJOs(java response class). Test code -
JSONObject responseObject = new JSONObject(responseFromService.getResponseBody())
.getJSONObject("RESPONSE");
ObjectMapper objectMapper = new ObjectMapper();
SomeResponseObject summaryResponse = objectMapper.getObjectMapper()
.readValue(responseObject.toString(), SomeResponseObject.class); // And this wouldn't work.
Question --
Is there any way we can cast the current API response or wrap it somehow to be mapped to the actual POJO(SomeResponseObject.class)?
Thanks in advance.

Problem
You receive an object with a rewardSubscription field, or, in your case, a map, with a rewardSubscription key. You can't convert a map to an object of SomeResponseObject type directly.
Solution
Option 1
Convert json to a map manually and set it to the SomeResponseObject instance:
JSONObject responseObject = new JSONObject(responseFromService.getResponseBody())
.getJSONObject("RESPONSE");
ObjectMapper objectMapper = new ObjectMapper();
Map<String, SummaryObject> summaryMap = objectMapper.readValue(responseObject.toString(), new TypeReference<Map<String, SummaryObject>>() {});
SomeResponseObject response = new SomeResponseObject();
response.setSummaryMap(summaryMap);
Option 2
So as not to manually convert map each time, write a custom deserializer that will handle both cases. The deserialize method should be similar to this:
#Override
public SomeResponseObject deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException {
ObjectMapper objectMapper = new ObjectMapper();
JsonNode rootNode = jp.readValueAsTree();
JsonNode sSummaryNode = rootNode.get("s_summary");
if (sSummaryNode != null) {
// Usual case.
return objectMapper.treeToValue(sSummaryNode, SomeResponseObject.class);
} else {
// Special case - when received a map.
Map<String, SummaryObject> summaryMap = objectMapper.readValue(rootNode.toString(), new TypeReference<Map<String, SummaryObject>>() {});
SomeResponseObject response = new SomeResponseObject();
response.setSummaryMap(summaryMap);
return response;
}
}
And then in the code you don't care:
ObjectMapper objectMapper = new ObjectMapper();
SomeResponseObject response = objectMapper.readValue(json, SomeResponseObject.class);

MappingJackson2HttpMessageConverter Mistake on Converting Complex Object

i have Class Response that return to client for every request from client my Response is:
public class Response<T> extends Request{
private ResponseType responseType;
private String message;
private ArrayList<T> result;
private int activationCode;
.
.
.
}
in my server side i have method that return Response that contains results with arraylist of InstagramUser
my method:
public Response getUserByUserName(#RequestBody List<Request> requests){
.
.
.
RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
try {
String searchResponse = restTemplate.getForObject(uri, String.class);
JSONObject rawObject = new JSONObject(searchResponse);
JSONArray searchResults = rawObject.getJSONArray("data");
ArrayList<InstagramUser> users = new ArrayList<InstagramUser>();
for (int i = 0; i < searchResults.length(); i++) {
JSONObject result = searchResults.getJSONObject(i);
users.add(new InstagramUser(result, AccessToken.getTokenString()));
}
response = new Response(requests.get(0).getUserId(), ResponseType.success, "users find successfully on: " + new Date().toString());
response.setResult(users);
.
.
.
return response;
}
and my InstagramUser:
public class InstagramUser extends InstagramModel {
protected long id;
protected String userName;
protected String fullName;
protected String profilePictureURI;
protected String bio;
protected String website;
protected int mediaCount = -1;
protected int followerCount = -1;
protected int followingCount = -1;
...
}
but in client side when i get Response from server my results is ArrayList of LinkedHashMap insted of ArrayList of InstagramUser:
restTemplate.postForObject(URL + conditions, params,Response.class);
and this is my json response from server for calling this method:
{
"id": 6151638910251304448,
"userId": 2,
"instagramId": null,
"searchId": null,
"mediaId": null,
"phoneNumber": null,
"name": null,
"date": 1466665008687,
"responseType": "success",
"message": "users find successfully on: Thu Jun 23 11:26:48 IRDT 2016",
"result": [
{
"id": 110000004535,
"userName": "______etabdar",
"fullName": "________dar",
"profilePictureURI": "https://igcdn-photos-h-a.akamaihd.net/hphotos-ak-xap1/t51.2885-19/s150x150/13183XXXXXXXX0231_1584363729_a.jpg",
"bio": "🎓 XXXXX 90",
"website": "",
"mediaCount": -1,
"followerCount": -1,
"followingCount": -1
}
],
"activationCode": 0
}
how can i fix this?

you'll have to use parameterized type reference. Works only with rest template exchange methods.
List<YourObjectType>> res = template.exchange(
rootUrl,
HttpMethod.POST,
null,
new ParameterizedTypeReference<List<YourObjectType>>() {});
Adjust parameters based on your inputs.

It looks like
restTemplate.postForObject(URL + conditions, params,Response.class)
doesn't know the specific type this part of your servers response:
"result": [
{
"id": 110000004535,
"userName": "______etabdar",
"fullName": "________dar",
"profilePictureURI": "https://igcdn-photos-h-a.akamaihd.net/hphotos-ak-xap1/t51.2885-19/s150x150/13183XXXXXXXX0231_1584363729_a.jpg",
"bio": "🎓 XXXXX 90",
"website": "",
"mediaCount": -1,
"followerCount": -1,
"followingCount": -1
}
],
should be mapped to when assigning
private ArrayList<T> result;
of your Response class, therefore switching to a default type (probably LinkedHashMap).
Maybe the answer to the following SO-question is of any help:
Using Spring RestTemplate in generic method with generic parameter

Deserializing json and delimiting newlines NDJ

I have a simple JSON file I'd like to parse (units.json) using gson and Java:
{"user_name": "happyman", "status": {"hp": 20, "karma": 7, "mp": 10}, "gold": 5.25,}
{"user_name": "sadman", "status": {"hp": 10, "karma": 2, "mp": 6}, "gold": 0.5,}
...
The problem I have is, how do I parse this file as it is not in an array format?
Currently I have a Filereader to the path being called by the JsonReader
FileReader fileReader = new FileReader(jsonFilePath);
JsonReader jsonReader = new JsonReader(fileReader);
jsonReader.beginObject();
while (jsonReader.hasNext()) {
String name = jsonReader.nextName();
if (name.equals("user_name")) {
System.out.println("user_id: "+jsonReader.nextString());
}
//...checks for other equals
However, with this code, I'm only able to get the first line.
I have a feeling it has something to do with the "hasNext()" not being the right method call in the while loop.
I appreciate the help! Thanks.

You can use this code with GSON library:
ObjectMapper mapr = new ObjectMapper();
Map<String,Object> map = mapr.readValue(jsonString, Map.class);
or
public class Units {
#JsonProperty("user_name")
public String user_name;
#JsonProperty("status")
public List<Attr> attrs;
..
}
public class Attr {
#JsonProperty("hp")
public String hp;
#JsonProperty("karma")
public String karma;
#JsonProperty("mp")
public String mp;
}
ObjectMapper mapr = new ObjectMapper();
Units unit = mapr.readValue(jsonString, Units.class);
for both you can use jsonString to define each json unit in your file.