I'm learning java at the moment and building a small app in Kafka Streams to ingest data and create a new stream essentially.
I have a main class like this:
public class Main {
public static void main(final String[] args) throws Exception {
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put("application.id", "streams-consumer-logs");
streamsConfiguration.put("client.id", "consumer-client");
streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, WallclockTimestampExtractor.class);
streamsConfiguration.put("servers");
final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();
System.out.println(streamsConfiguration);
final KStreamBuilder builder = new KStreamBuilder();
final KStream<String, String> textLines = builder.stream( stringSerde, stringSerde,"sdp_logs");
//textLines.print();
System.out.println("test");
KStream<String, String> wordCounts = textLines
.map((key, value) -> {
//how do I manipulate this value which is a json object?
Gson gson = new Gson();
String json = gson.toJson(value);
FilterLogs filterlogs = new FilterLogs(json);
String filterlogs = filterlogs.execute();
return KeyValue.pair(key, json);
});
//KStream<String, Long> streamWrite = wordCounts.toStream();
wordCounts.print();
final KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
streams.cleanUp();
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
}
I then have a FilterLogs class that looks like this:
public class FilterLogs {
private type? jsoonObj;
public FilterLogs (type? jsonObj) {
this.jsonObj = jsonObj;
}
public type? getResult() { return result; }
public void execute () {
//manipulation of jsonObj goes in here?
}
}
}
My json object looks like this that value returns KStream<String, String> wordCounts = textLines.map((key, value) -> {}:
{
"tags": [
"dgtl"
],
"beat": {
"hostname": "host1",
"name": "host`",
"version": "5.4.1"
},
"input_type": "log",
"#timestamp": "2017-09-27T20:01:52.282Z",
"source": "/log/state-change.log",
"offset": 1725163,
"message": "message of the log is here"
"type": "log"
}
From my understanding in Java (as I come from the a functional programming paradigm) is that I have to take value, pass it to the class FilterLogs in my case, and then that class should return the new value...right?
Where I'm confused is what type to use for the class and how do I actually parse through the json and do stuff with it?
Normally, in javascript, I can just parse through it with a bunch of loops and do what I want but I don't quite get how to do this in Java.
What if I want my json object to return like this?
{
"tags": [
"dgtl"
],
"beat": {
"hostname": "host1"
},
"offset": 1725163,
"message": "message of the log is here"
}
You should checkout the new
JSON-P( for JSON processing & parsing) - http://json-b.net/
JSON-B (for JSON Binding) apis that was released recently - https://github.com/javaee/jsonp
They were made purposely for all of your usecases
Related
I'm receiving JSON from REST API looks like:
{
"items": [
{
"id": 60659,
"name": "Display",
"active": true,
"account_id": 235
},
{
"id": 36397,
"name": " Mail Display",
"active": true,
"account_id": 107
}
]
}
I'm using this method to parse it:
Mono<List<Item>> getItems(String token) {
return webCLient
.get()
.headers(httpHeaders -> httpHeaders.setBearerAuth(token))
.retrieve()
.bodyToMono(ItemResponse.class)
.map(ItemResponse::getResponse)
.retryBackoff(RetrySettings.RETRIES, RetrySettings.FIRST_BACKOFF, RetrySettings.MAX_BACKOFF)
.doOnError(e -> log.error("error: " + e.getCause().toString()))
Response:
public class ItemResponse {
#JsonProperty("items")
private List<Item> response;
}
But sometimes 3rd party API returns different response without top level items property and looks like:
[
{
"id": 60659,
"name": "Display",
"active": true,
"account_id": 235
},
{
"id": 36397,
"name": " Mail Display",
"active": true,
"account_id": 107
}
]
At this point my app is crashing with JSON decoding error. I used for this case:
bodyToMono(new ParameterizedTypeReference<List<Item>>() {})
But I can't always refactoring this part of code just to handle their json. How to do it in dynamical way with Spring WebFlux? Like try -> parse#1 -> catch -> parse#2. So i need to parse json in way#1 and if error occurs app should try to parse it with way#2.
You can get the response as a string .bodyToMono(String.class) and do whatever you want, with multiple try catches... but I think your best bet is to create a custom Deserializer and use it with your WebClient via ExchangeStrategies like described here: How to customize SpringWebFlux WebClient JSON deserialization?
.
class MyResponse {
List<Object> data;
MyResponse(List<Object> data) {
this.data = data;
}
}
class MyResponseDeserializer extends JsonDeserializer<MyResponse> {
#Override
public MyResponse deserialize(JsonParser jsonParser, DeserializationContext deserializationContext) throws IOException {
TreeNode treeNode = jsonParser.getCodec().readTree(jsonParser);
List<Object> data = new ArrayList<>();
if (treeNode.isArray()) {
// parse it as array
} else {
// parse it as object and put inside list
}
MyResponse myResponse = new MyResponse(data);
return myResponse;
}
}
And then
WebClient getWebClient() {
ObjectMapper objectMapper = new ObjectMapper();
SimpleModule simpleModule = new SimpleModule();
simpleModule.addDeserializer(MyResponse.class, new MyResponseDeserializer());
objectMapper.registerModule(simpleModule);
ExchangeStrategies strategies = ExchangeStrategies
.builder()
.codecs(clientDefaultCodecsConfigurer -> {
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonEncoder(new Jackson2JsonEncoder(objectMapper, MediaType.APPLICATION_JSON));
clientDefaultCodecsConfigurer.defaultCodecs().jackson2JsonDecoder(new Jackson2JsonDecoder(objectMapper, MediaType.APPLICATION_JSON));
}).build();
return WebClient.builder().exchangeStrategies(strategies).build();
}
Mono<List<Item>> getItems(String token) {
return getWebClient()
.get()
.headers(httpHeaders -> httpHeaders.setBearerAuth(token))
.retrieve()
.bodyToMono(MyResponse.class)
.map(MyResponse::data)
.retryBackoff(RetrySettings.RETRIES, RetrySettings.FIRST_BACKOFF, RetrySettings.MAX_BACKOFF)
.doOnError(e -> log.error("error: " + e.getCause().toString()))
}
The rest is the same as in your example just change the class name and add appropriate fields.
And of course this is just a fast written demo and everything hardcoded and within a one method, better to have them injected
Code sample:-
public List<UserDto> getUserCandidates(String taskId) {
List<UserCandidates> listResponse;
ResponseEntity<String> response=restTemplate.getForEntity(configProperties.getUrl()+"/task/"+taskId+"/identity-links",
String.class);
listResponse =new Gson().fromJson(response.getBody(), new TypeToken<ArrayList<UserCandidates>>(){}.getType());
listResponse.forEach(result->{
if(!StringUtils.isEmpty(result.getUserId())){
ResponseEntity<UserRefer> userResponse=restTemplate.getForEntity(configProperties.getUrl()+"/user/"+result.getUserId()+"/profile", UserRefer.class);
userDtoList.add(new UserDto(result.getUserId(), Arrays.asList(result.getGroupId()), Arrays.asList(result.getType()), userResponse.getBody().getFirstName(),
userResponse.getBody().getLastName(), userResponse.getBody().getEmail()));
}
else if(!StringUtils.isEmpty(result.getGroupId())) {
ResponseEntity<String> responseGroup=restTemplate.getForEntity(configProperties.getUrl()+"/user"+"?memberOfGroup="+result.getGroupId(), String.class);
List<UserResponse> listGroup=new Gson().fromJson(responseGroup.getBody(), new TypeToken<ArrayList<UserResponse>>(){}.getType());
listGroup.forEach(resultGroup->{
userDtoList.add(new UserDto(resultGroup.getId(),Arrays.asList(result.getGroupId()),
Arrays.asList(result.getType()),resultGroup.getFirstName(),resultGroup.getLastName(),resultGroup.getEmail()));
});
}
});
return userDtoList;
}
So in if condition the response from API I'm getting is
UserRefer(id=demo, firstName=Demo, lastName=Demo, email=demo#camunda.org) - userResponse object
And from listResponse object data is [UserCandidates(userId=null, groupId=accounting, type=candidate), UserCandidates(userId=null, groupId=sales, type=candidate), UserCandidates(userId=demo, groupId=null, type=assignee)]
next in else if condition the response for listGroup is [UserResponse(status=null, id=demo, firstName=Demo, lastName=Demo, email=demo#camunda.org), UserResponse(status=null, id=mary, firstName=Mary, lastName=Anne, email=mary#camunda.org)]
So now you can see the data is duplicate. The output i want is for when userId is not empty from the data it should take type and merge the array
else if grouped not empty the data it should take for groupType and merge in the array removing duplicte and merging in same object
Output :-
[
{
"userId": "demo",
"name": "Demo Demo",
"type": [
"candidate",
"assignee"
],
"email": "demo#camunda.org",
"groupId": [
"accounting",
"sales"
]
},
{
"userId": "mary",
"name": "Mary Anne",
"type": [
"candidate"
],
"email": "mary#camunda.org",
"groupId": [
"accounting",
"sales"
]
}
]
You need some fundamental changes in your code.
1- instead of using ResponseEntity<String> use ResponseEntity<UserCandidates[]> response by this changing you don't need use Gson() dependency.
2- You don't need to use StringUtils to check to be empty. there is same method for both string and list objects.
3- For the duplicate date I define a Map<String,UserDto> with id as key and userDto object as a value. and where the userDto data is created I store it in the map with the id. as you see for storing userDto object in the map I used merge method that for the duplicate key(id) it has a merge function.
Tip: for readability would be nice to separate the restTemplate call in other class may you reuse it too.
mergeFunction is somthing like this:
private UserDto mergeFunction(UserDto u1,UserDto u2){
u1.getType().addAll(u2.getType());
u1.getGroupId().addAll(u2.getGroupId());
return u1;
}
and complete code is:
public List<UserDto> getUserCandidates(String taskId) {
Map<String, UserDto> userDtoMap = new HashMap<>();
Map<String, String> params = new HashMap<>();
ResponseEntity<UserCandidates[]> response = restTemplate
.getForEntity(configProperties.getUrl() + "/task/" + taskId + "/identity-links",
UserCandidates[].class, params);
Arrays.asList(response.getBody()).forEach(result -> {
if (!result.getUserId().isEmpty()) {
ResponseEntity<UserRefer> userResponse = restTemplate
.getForEntity(configProperties.getUrl() + "/**", UserRefer.class);
userDtoMap.merge(result.getUserId(), new UserDto(result.getUserId(),
new ArrayList<>(Arrays.asList(result.getGroupId())), Arrays.asList(result.getType()),
userResponse.getBody().getFirstName(),
userResponse.getBody().getLastName(),
userResponse.getBody().getEmail()), (u1, u2) -> mergeFunction(u1,u2));
} else if (!result.getGroupId().isEmpty()) {
String requestUri = configProperties.getUrl() + "/user" +
"?memberOfGroup={memberOfGroup}";
Map<String, String> userResParam = new HashMap<>();
userResParam.put("memberOfGroup", result.getGroupId());
ResponseEntity<UserResponse[]> responseGroup = restTemplate
.getForEntity(requestUri, UserResponse[].class, userResParam);
Arrays.asList(responseGroup.getBody()).forEach(resultGroup -> {
userDtoMap.merge(resultGroup.getId(), new UserDto(resultGroup.getId(),
Arrays.asList(result.getGroupId()),
Arrays.asList(result.getType()), resultGroup.getFirstName(),
resultGroup.getLastName(),
resultGroup.getEmail()), (u1, u2) -> mergeFunction(u1,u2));
});
}
});
return new ArrayList<>(userDtoMap.values());
}
I have below json response as part of a webservice response ,where inner object name is value which is dynamic.Not able to understand how to declare a equivalent java bean...
{
"error": [
{
"fea09b93175b626e4bb3d248ccb890bc": {
"name": "80_5595_10.zip (5.0)",
"error": [
"Location path does not exist"
]
}
},
{
"5a8745c1967adabe9d492d3595f37ce1": {
"name": "80_7208_01.zip (5.0)",
"error": [
"Location path does not exist"
]
}
}
]
You could do it that way (example using Jackson).
I declared a bean as follow:
class Error {
String id;
String name;
List<String> error;
// Getters / Setters
#Override
public String toString() {
return "id="+id+", name="+name+", error="+error;
}
}
Your JSON structure is Map<String, List<Map<String, Error>>>
// Parse your JSON into a Map<String, List<Map<String, Error>>>
ObjectMapper mapper = new ObjectMapper();
Map<String, List<Map<String, Error>>> map =
mapper.readValue(json, new TypeReference<Map<String, List<Map<String, Error>>>>() {});
You now have this Map structure
{ error = [
{ fea09b93175b626e4bb3d248ccb890bc = Error#36d64342 },
{ 5a8745c1967adabe9d492d3595f37ce1 = Error#39ba5a14 }
]
}
Then take each error element of the Map and set their id to be the same as their key. And add them into a List
List<Map<String, Error>> errorsMap = map.get("error");
List<Error> errors = new ArrayList<>();
for(Map<String, Error> errorMap : errorsMap) {
String key = new ArrayList<>(errorMap.keySet()).get(0);
Error error = errorMap.get(key);
error.setId(key);
errors.add(error);
}
errors.forEach(System.out::println);
Outputs:
id=fea09b93175b626e4bb3d248ccb890bc, name=80_5595_10.zip (5.0), error=[Location path does not exist]
id=5a8745c1967adabe9d492d3595f37ce1, name=80_7208_01.zip (5.0), error=[Location path does not exist]
I have the following Java Map. Map<String, String> containing values such as:
876234876, google
mike#hotmail, hotmail
9879892, google
I need to convert it to the following JSON object structure and Java JSON objects are not my friend.
"addresses" : [
{ "address":"876234876", "domain":"google" },
{ "address":"mike#hotmail", "domain":"hotmail" },
{ "address":"9879892", "domain":"google" }
]
To create the JSON you ask, you need to insert a JSONObject into a JSONArray. So for each Entry of your Map<String, String>, create a JSONObject like {"address": entry.key, "domain": entry.value} and add those to a JSONArray.
Let's use a Stream.map to create that object and insert the result into the array directly:
public static JSONObject createJson(Map<String, String> value) {
JSONObject result = new JSONObject();
JSONArray addresses = new JSONArray();
result.put("addresses", addresses);
value.entrySet().stream() //iterate the map
.map(e -> { //build an object
JSONObject address = new JSONObject();
address.put("address", e.getKey());
address.put("domain", e.getValue());
return address;
})
.forEach(addresses::put); //insert into the array
return result;
}
And test it with :
public static void main(String[] args) {
Map<String, String> values = new HashMap<>();
values.put("876234876", "google");
values.put("mike#hotmail", "hotmail");
values.put("9879892", "google");
System.out.println(createJson(values).toString(4));
}
And the result :
{"addresses": [
{
"address": "9879892",
"domain": "google"
},
{
"address": "876234876",
"domain": "google"
},
{
"address": "mike#hotmail",
"domain": "hotmail"
}
]}
Using the API : JSON In Java
<!-- https://mvnrepository.com/artifact/org.json/json -->
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20180130</version>
</dependency>
Check this solution:
List<String> addresses = new ArrayList<>();
map.forEach((k, v) -> {
Map<String, String> nn = new HashMap<>();
nn.put("address", k);
nn.put("domain", v);
addresses.add(JSONValue.toJSONString(nn));
});
JSONObject result = new JSONObject(Collections.singletonMap("addresses", new JSONArray(addresses)));
Your JAVA objects should look like this :
List of addresses
#XmlRootElement
public class Foo {
private List<Address> addresses;
// Getter, Setter, ...
}
Address
public class Address {
private String address;
private String domain;
// Getters, setters, ...
}
I'm new to NoSQL databases. I want to save the following JSON data into CouchBase Lite. Can someone guide me on the best way to do this?
{"widget": {
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
"text": {
"data": "Click Here",
"size": 36,
"style": "bold",
"name": "text1",
"hOffset": 250,
"vOffset": 100,
"alignment": "center",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
}
}}
I tried to do this using the following code.
public void insertSample(String widget,String control,ArrayList<eModel> details){
Map<String, Object> properties = new HashMap<String, Object>();
properties.put("control", control);
properties.put("details", details);
Document document = database.getDocument(widget);
try {
document.putProperties(properties);
} catch (CouchbaseLiteException e) {
Log.e("", "Cannot save document", e);
}
}
But this code is creating a new id each time. I want to insert the same widget value in multiple times.
This is run time data, not static data I want to insert one by one.
For example, given a widget Map as follows:
{"widget": {
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
}
then I want to append the following field under the "window" field:
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
}
The constraint on documents in Couchbase Lite is that the _id property should be unique. As the document gets updated, it creates new revisions.
There are two possible methods to update a document:
putProperties: given a new JSON object, it replaces the document's body with that object.
update: takes a callback function or block. It loads the current revision's properties, then calls this function, passing it an UnsavedRevision object, whose, properties are a mutable copy of the current ones. Your callback code can modify this object's properties as it sees fit; after it returns, the modified revision is saved and becomes the current one.
So, to update a widget document with the image dictionary, you should use the update method:
final Map<String, Object> imageProperties = new HashMap<String, Object>();
imageProperties.put("src", "Images/Sun.png");
imageProperties.put("name", "sun1");
// ...
Document document = database.getDocument(widgetId);
document.update(new Document.DocumentUpdater() {
#Override
public boolean update(UnsavedRevision newRevision) {
Map<String, Object> properties = newRevision.getUserProperties();
properties.put("image", imageProperties);
newRevision.setUserProperties(properties);
return true;
}
});
Note: It's recommended to use a library like Jackson to serialize/deserialize the JSON and POJO models in a Java application (you can read this blog post to find more info).
You need to construct your map like this to save like above json format.
Map<String, Object> windowMap = new HashMap<String, Object>();
windowMap.put("title","Sample Konfabulator Widget");
windowMap.put("name","main_window");
windowMap.put("width",500);
windowMap.put("height",500);
Map<String, Object> imageMap = new HashMap<String, Object>();
imageMap.put("src","Images/Sun.png");
imageMap.put("name","sun1");
imageMap.put("hOffset",250);
imageMap.put("vOffset",250);
Map<String, Object> textMap = new HashMap<String, Object>();
textMap.put("data","Click Here");
textMap.put("size",36);
Map<String, Object> memberMap = new HashMap<String, Object>();
memberMap.put("window",windowMap);
memberMap.put("image",imageMap);
memberMap.put("text",textMap);
Map<String, Object> widgetMap = new HashMap<String, Object>();
widgetMap.put("widget",memberMap);
try {
document.putProperties(widgetMap);
} catch (CouchbaseLiteException e) {
Log.e("", "Cannot save document", e);
}
I was having the same problem. You need to traverse the keys of the json and add each object using putproperties. However, for a JSONArray, you need to use an ArrayList. I am traversing the keys using the Jackson library(which is also used by couchbase internally)
try {
Map<String, Object> properties = new HashMap<String, Object>();
JsonFactory factory = new JsonFactory();
ObjectMapper mapper = new ObjectMapper(factory);
// convert you json string to Jackson JsonNode
JsonNode rootNode = mapper.readTree(doc.toString());
Iterator<Map.Entry<String, JsonNode>> it = rootNode.getFields();
while (it.hasNext()) {
Map.Entry<String, JsonNode> pair = it.next();
String key = pair.getKey().toString();
String value = pair.getValue().toString();
if (JSONDiffUtil.isJSONArray(value)) {
Debug.logInfo("its a json array: " + key, MODULE);
ArrayNode arrNode = (ArrayNode) pair.getValue();
properties.put(key, arrNode);
}
else if (key.startsWith("_")) {
Debug.logInfo("skipping _rev" + key, MODULE);
}
else {
Debug.logInfo("its a json object: " + key, MODULE);
properties.put(key, pair.getValue());
}
}
document.putProperties(properties);
}
catch (CouchbaseLiteException e) {
Debug.logInfo("Error", MODULE);
Debug.logInfo(e.getMessage(), MODULE);
} catch (JsonProcessingException e) {
Debug.logInfo(e.getMessage(), MODULE);
} catch (IOException e) {
Debug.logInfo(e.getMessage(), MODULE);
}
There is a very useful link where you can find good info
https://blog.couchbase.com/object-mapping-couchbase-mobile-android/