I have written a program which inserts in bulk to Elasticsearch in batch of around 3000. The problem is that I need to convert these object to json before executing the bulk insert request. But there is major downside with json convertion and it is becoming a bottle neck of my whole computation.
Can any one suggest a super fast way to convert object to json in java. My code looks like this:
private String getESValueAsString(ElasticSearchValue elasticSearchValue) throws JsonProcessingException {
ElasticSearchValue prevValue = null;
if (stateType == StateType.OPAQUE) {
prevValue = (ElasticSearchValue) elasticSearchValue.getPrevious();
}
elasticSearchValue.setPrevious(null);
ObjectMapper om = new ObjectMapper();
Map<String, Object> props = om.convertValue(elasticSearchValue, Map.class);
if (stateType == stateType.OPAQUE) {
props.put("previous", prevValue);
}
return om.writeValueAsString(props);
}
Just found the issue, I am creating too many ObjectMapper for each serialization and that is making my whole processing slow.
This is a very good guide and it improved my performance 100x
http://wiki.fasterxml.com/JacksonBestPracticesPerformance
why not just insert into BulkRequestBuilder json records in the first place, something like this
Client client = new TransportClient().addTransportAddress(new InetSocketTransportAddress("localhost", 9300));
BulkRequestBuilder bulk = client.prepareBulk();
.....
bulk.add(client.prepareIndex(<your index>, <your type>)
.setSource(<your object>.toJson());
....
and in <your object> class
create Gson like this:
Gson gson = new GsonBuilder().excludeFieldsWithoutExposeAnnotation().create();
and method:
public String toJson(){
return gson.toJson(this, <you class>.class);
}
Related
Im trying to get a key:value pair from a simple jsonString to add it after into a memory tab. If facing an issue cause my input is a string. and it looks like my loop isnot able to read the key value pair.
I read many topics about it, and im still in trouble with it. As you can see below
{"nom":"BRUN","prenom":"Albert","date_naiss":"10-10-1960","adr_email":"abrun#gmail.com","titre":"Mr","sexe":"F"}
and my method, find only on object... the result is the same in my loop
public static ArrayHandler jsonSimpleObjectToTab(String data) throws ParseException {
if( data instanceof String) {
final var jsonParser = new JSONParser();
final var object = jsonParser.parse(data);
final var array = new JSONArray();
array.put(object);
final var handler = new ArrayHandler("BW_funct_Struct");
for( KeyValuePair element : array) {
handler.addCell(element);
Log.warn(handler);
}
return handler;
} else {
throw new IllegalArgumentException("jsonSimpleObjectToTab: do not support complex object" + data + "to Tab");
}
}
i also tryed before to type my array as a List, Object etc, without the keyValuePair object, i would appreciate some help.
Thanks again dear StackOverFlowers ;)
You can try this :
const json = '{"nom":"BRUN","prenom":"Albert","date_naiss":"10-10-1960","adr_email":"abrun#gmail.com","titre":"Mr","sexe":"F"}';
map = new Map();
const obj = JSON.parse(json,(key,value) => {
map.set(key,value)
});
and you'll have every pair stored in map
Simply split the whole line at the commas and then split the resulting parts at the colon. This should give you the individual parts for your names and values.
Try:
supposing
String input = "\"nom\":\"BRUN\",\"prenom\":\"Albert\"";
then
String[] nameValuePairs = input.split(",");
for(String pair : nameValuePairs)
{
String[] nameValue = pair.split(":");
String name = nameValue[0]; // use it as you need it ...
String value = nameValue[1]; // use it as you need it ...
}
You can use TypeReference to convert to Map<String,String> so that you have key value pair.
String json = "{\"nom\":\"BRUN\",\"prenom\":\"Albert\",\"date_naiss\":\"10-10-1960\",\"adr_email\":\"abrun#gmail.com\",\"titre\":\"Mr\",\"sexe\":\"F\"}";
ObjectMapper objectMapper = new ObjectMapper();
TypeReference<Map<String,String>> typeReference = new TypeReference<Map<String, String>>() {
};
Map<String,String> map = objectMapper.readValue(json, typeReference);
I just answered a very similar question. The gist of it is that you need to parse your Json String into some Object. In your case you can parse it to Map. Here is the link to the question with my answer. But here is a short version: you can use any Json library but the recommended ones would be Jackson Json (also known as faster XML) or Gson(by Google) Here is their user guide site. To parse your Json text to a class instance you can use ObjectMapper class which is part of Jackson-Json library. For example
public <T> T readValue(String content,
TypeReference valueTypeRef)
throws IOException,
JsonParseException,
JsonMappingException
See Javadoc. But also I may suggest a very simple JsonUtils class which is a thin wrapper over ObjectMapper class. Your code could be as simple as this:
Map<String, Object> map;
try {
map = JsonUtils.readObjectFromJsonString(input , Map.class);
} catch(IOException ioe) {
....
}
Here is a Javadoc for JsonUtils class. This class is a part of MgntUtils open source library written and maintained by me. You can get it as Maven artifacts or from the Github
I have a stable SpringBoot project that runs. I want to add a end point that reads a json file from classpath and passes it through to the response without having to create any Model objects (pass thru).
I have no issues reading the json file into JsonNode or ObjectNode, I'm struggling with where to go next to set the data in my response object.
Added this caveat later, I do need to update the json from a database.
Ok, paired up with a colleague at work and we tried two things, return a string (escapes strings in REST output - returns a big String.) not good. What worked is setting the response object to a and calling mapper.readValue(jsonFeed, Map.class), that returned the JSON in proper object notation.
#Value("${metadata.json.file}") //defined in application.context
private Resource metaJsonFileName;
public String getJsonFromFile(List<UnitUiItem> uiitems){
JsonNode root;
ObjectMapper mapper = new ObjectMapper();
InputStream stream = metaJsonFileName.getInputStream();
root = mapper.readTree(stream);
JsonNode dataNode = root.get("data");
JsonNode optionDataNode = dataNode.get("storelocation");
((ObjectNode)optionDataNode).putArray("units");
for(UnitUiItem item : uiitems){
JsonNode unitNode = ((ObjectNode)optionDataNode).withArray("units").addObject();
((ObjectNode)unitNode).put("code",item.getCode());
((ObjectNode)unitNode).put("displayName",item.getDisplayName());
}
LOGGER.info("buildMetaJson exit");
return root.toString();
}
//calling method
String jsonFeed = getJsonFromFile();
ObjectMapper mapper = new ObjectMapper();
response.setData(mapper.readValue(jsonFeed, Map.class));
I have some code cleanup to do.. any cleaner ways of doing this?
I have got stuck trying to insert a JSONArray into a Jackson ObjectNode. This is what I am trying to do:
public void myMethod(JSONArray jsonArray) {
ObjectNode payload = objectMapper.createObjectNode(0);
payload.put("array", /* jsonArray */);
payload.put(/* some other things */);
...
}
It feels like something really silly but what is actually the best way to do it?!
EDIT: I am sorry beacause I did not mention an important point, that is I have to serialize the ObjectNode once I finished building it, so using putPOJO() is not a possibility.
I like aribeiro's approach more. You can use the putPOJO() method to do this. For example:
// Incoming org.json.JSONArray.
JSONArray incomingArray = new JSONArray("[\"Value1\",\"Value2\"]");
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode payload = objectMapper.createObjectNode();
// Adds the JSONArray node to the payload as POJO (plain old Java object).
payload.putPOJO("array", incomingArray);
System.out.println(objectMapper.writeValueAsString(payload));
Javadoc can be found here.
Note: here's a previous implementation that I submitted using readTree():
// Incoming org.json.JSONArray.
JSONArray incomingArray = new JSONArray("[\"Value1\",\"Value2\"]");
ObjectMapper objectMapper = new ObjectMapper();
ObjectNode payload = objectMapper.createObjectNode();
// Reads the JSON array into a Jackson JsonNode.
JsonNode jsonNode = objectMapper.readTree(incomingArray.toString());
// Sets the Jackson node on the payload.
payload.set("array", jsonNode);
System.out.println(objectMapper.writeValueAsString(payload));
The documentation in Jackson is horrible and it only states how it is possible to create JSON from objects and streams, not piece by piece. I cannot find the term to be able to search for more examples/documentation.
I would like to do something like the following.
JsonObject jsonObject = new JsonObject().add("type", "unregister").add("id",id);
return jsonObject.toString();
Which would return {"type":"unregister","id":2} as an example. Can anyone nudge me in the right direction?
The corresponding JsonObject in Jackson is ObjectNode. You can use its various xyzNode(...) factory methods to produce JsonNode subtype instances. You can then use ObjectNode#set(String, JsonNode) to add them.
JsonNodeFactory nc = new JsonNodeFactory(false); // or true, depending
ObjectNode root = new ObjectNode(nc);
TextNode text = root.textNode("unregister");
NumericNode id = root.numberNode(2);
root.set("type", text);
root.set("id", id);
Then
System.out.println(node);
produces
{"type":"unregister","id":2}
You can also create an ObjectNode through the ObjectMapper which uses the ObjectMapper's JsonNodeFactory.
ObjectMapper mapper = new ObjectMapper();
ObjectNode root = mapper.createObjectNode();
Jackson maps Java Objects to JSON, so their documentation is telling you what it does (it does not provide a sort of builder API to arbitrarily compose JSON documents)
If you want to map arbitrary key/values, use a Map<String, Object> and use Jackson to turn it into JSON. You can mix heterogeneous map values and get the result you want this way, pseudo:
Map<String, Object> map = new HashMap<>();
map.put("type", "unregister");
map.put("id", id);
StringWriter sw = new StringWriter();
mapper.writeValue(sw, map); // mapper is a ObjectMapper instance
System.out.println(sw.toString());
I have always used XML for returning data from my servlets and i have decided to have a go with JSON.Simple.
I currently have;
public JSONObject loadDetails() throws IOException {
JSONArray list = new JSONArray();
list.add(new Car("car1",3,4,3,2,4,5));
list.add(new Car("car2",3,4,3,2,4,5));
StringWriter out = new StringWriter();
list.writeJSONString(out);
System.out.println(out.toString());
return null;
}
I am trying to store some car details, however when i try and print i just get the object names rather than the correct JSON value. My car class implements JSONStreamAware eg;
public void writeJSONString (Writer out) throws IOException{
LinkedHashMap obj = new LinkedHashMap();
obj.put("carname", carname);
obj.put("engine", engine);
JSONValue.writeJSONString(obj, out);
}
Have i missed something here?
I am sorry, I know that this is not what you are explicitly asking for but I will definitely recommend using Gson for json parsing. You jsut specify several annotations and everything is magically serialized / deserialized. Read about gson here. Again sorry that this is not exact answer of the question, but still I hope it will help you get started in json.