Develop Reference

Parse a single POJO from multiple YAML documents representing different classes

I want to use a single YAML file which contains several different objects - for different applications. I need to fetch one object to get an instance of MyClass1, ignoring the rest of docs for MyClass2, MyClass3, etc. Some sort of selective de-serializing: now this class, then that one... The structure of MyClass2, MyClass3 is totally unknown to the application working with MyClass1. The file is always a valid YAML, of course.
The YAML may be of any structure we need to implement such a multi-class container. The preferred parsing tool is snakeyaml.
Is it sensible? How can I ignore all but one object?
UPD: replaced all "document" with "object". I think we have to speak about the single YAML document containing several objects of different structure. More of it, the parser knows exactly only 1 structure and wants to ignore the rest.
UDP2: I think it is impossible with snakeyaml. We have to read all objects anyway - and select the needed one later. But maybe I'm wrong.
UPD2: sample config file
---
-
exportConfiguration781:
attachmentFieldName: "name"
baseSftpInboxPath: /home/user/somedir/
somebool: false
days: 9999
expected:
- ABC w/o quotes
- "Cat ABC"
- "Some string"
dateFormat: yyyy-MMdd-HHmm
user: someuser
-
anotherConfiguration:
k1: v1
k2:
- v21
- v22

This is definitely possible with SnakeYAML, albeit not trivial. Here's a general rundown what you need to do:
First, let's have a look what loading with SnakeYAML does. Here's the important part of the YAML class:
private Object loadFromReader(StreamReader sreader, Class<?> type) {
Composer composer = new Composer(new ParserImpl(sreader), resolver, loadingConfig);
constructor.setComposer(composer);
return constructor.getSingleData(type);
}
The composer parses YAML input into Nodes. To do that, it doesn't need any knowledge about the structure of your classes, since every node is either a ScalarNode, a SequenceNode or a MappingNode and they just represent the YAML structure.
The constructor takes a root node generated by the composer and generates native POJOs from it. So what you want to do is to throw away parts of the node graph before they reach the constructor.
The easiest way to do that is probably to derive from Composer and override two methods like this:
public class MyComposer extends Composer {
private final int objIndex;
public MyComposer(Parser parser, Resolver resolver, int objIndex) {
super(parser, resolver);
this.objIndex = objIndex;
}
public MyComposer(Parser parser, Resolver resolver, LoaderOptions loadingConfig, int objIndex) {
super(parser, resolver, loadingConfig);
this.objIndex = objIndex;
}
#Override
public Node getNode() {
return strip(super.getNode());
}
private Node strip(Node input) {
return ((SequenceNode)input).getValue().get(objIndex);
}
}
The strip implementation is just an example. In this case, I assumed your YAML looks like this (object content is arbitrary):
- {first: obj}
- {second: obj}
- {third: obj}
And you simply select the object you actually want to deserialize by its index in the sequence. But you can also have something more complex like a searching algorithm.
Now that you have your own composer, you can do
Constructor constructor = new Constructor();
// assuming we want to get the object at index 1 (i.e. second object)
Composer composer = new MyComposer(new ParserImpl(sreader), new Resolver(), 1);
constructor.setComposer(composer);
MyObject result = (MyObject)constructor.getSingleData(MyObject.class);

The answer of #flyx was very helpful for me, opening the way to workaround the library (in our case - snakeyaml) limitations by overriding some methods. Thanks a lot! It's quite possible there is a final solution in it - but not now. Besides, the simple solution below is robust and should be considered even if we'd found the complete library-intruding solution.
I've decided to solve the task by double distilling, sorry, processing the configuration file. Imagine the latter consisting of several parts and every part is marked by the unique token-delimiter. For the sake of keeping the YAML-likenes, it may be
---
#this is a unique key for the configuration A
<some YAML document>
---
#this is another key for the configuration B
<some YAML document
The first pass is pre-processing. For the given String fileString and String key (and DELIMITER = "\n---\n". for example) we select a substring with the key-defined configuration:
int begIndex;
do {
begIndex= fileString.indexOf(DELIMITER);
if (begIndex == -1) {
break;
}
if (fileString.startsWith(DELIMITER + key, begIndex)) {
fileString = fileString.substring(begIndex + DELIMITER.length() + key.length());
break;
}
// spoil alien delimiter and repeat search
fileString = fileString.replaceFirst(DELIMITER, " ");
} while (true);
int endIndex = fileString.indexOf(DELIMITER);
if (endIndex != -1) {
fileString = fileString.substring(0, endIndex);
}
Now we feed the fileString to the simple YAML parsing
ExportConfiguration configuration = new Yaml(new Constructor(ExportConfiguration.class))
.loadAs(fileString, ExportConfiguration.class);
This time we have a single document that must co-respond to the ExportConfiguration class.
Note 1: The structure and even the very content of the rest of configuration file plays absolutely no role. This was the main idea, to get independent configurations in a single file
Note 2: the rest of configurations may be JSON or XML or whatever. We have a method-preprocessor that returns a String configuration - and the next processor parses it properly.

Strategies for iterating through JsonPath array

I just started exploring JsonPath today. I want to explore not just what's possible to do with it, but some effective strategies.
For instance, let's say I have to iterate through an array contained within one element in the json string.
I'm using the "store" example from https://github.com/jayway/JsonPath#path-examples .
To get the list of books itself, I would imagine I could do something like this:
List<?> allBooks = JsonPath.<List<?>>read(context, "$.store.book");
Does it make sense to think about it this way?
It's the logic for iterating through this that I'm uncertain about.
I would have thought I could define a "Book" pojo and then do something like this:
for (int ctr = 0; ctr < allBooks.size(); ++ ctr) {
Book book = JsonPath.<Book>read(context, ".[" + ctr + "]");
System.out.println("book[" + book + "]");
}
However, this doesn't work. The "read" method at this point returns a JSONArray.
The last line in the code sample at https://github.com/jayway/JsonPath#what-is-returned-when is close to what I'm looking at, but this requires parsing the json in every iteration. It seems like the "DocumentContext" class has "read" methods that can take a type parameter, but not "JsonPath".
What are some reasonable strategies for navigating something like this?

JSON path will just return you a list of Maps as you've no doubt already seen. You need a way to tell it how to map these values to an object - for this you will need a custom configuration. There are other providers like Gson etc., but I've only used Jackson.
Configuration configuration = Configuration
.builder()
.jsonProvider(new JacksonJsonProvider())
.mappingProvider(new JacksonMappingProvider())
.build();
The second step is to specify generic type information with a TypeRef and pass it along when reading the tag.
List<Book> allBooks = JsonPath.using(configuration)
.parse(context)
.read("$.store.book", new TypeRef<List<Book>>() {});
As a result you get a nice list of Book objects.

Elasticsearch Update indexdocument

I need to update an index document for an elasticsearch table and this is the code I have implemented. But it is not working, what's wrong and how should I implement this?
My code.
Map<String, Object> matching_result;
for (SearchHit hit : response_text.getHits()) {
matching_result = hit.getSource();
String flag_value = matching_result.get("flag").toString();
matching_result.put("flag", true);
}
String indexString = JSONConverter.toJsonString(matching_result);
IndexResponse response = client.prepareIndex("index_name", "data").setSource(indexString).execute().actionGet();
boolean created = response.isCreated();
System.out.println("created or updated--------------------->" + created);
System.out.println("flag value==========" + matching_result.get("flag"));
return actual_theme;
(JSONConverter.toJsonString) is our library class for converting to json string.
What is wrong with this query?
Instead of updating the existing document it is creating a new one. I want to change the existing one.

Based on your example code, it looks like by "update" you mean you are trying to replace the entire document. In order to do this, you must specify the id of the document you wish to update.
Using the Java API, in addition to calling setSource on the IndexRequestBuilder, you would also need to supply the id by calling setId. For example:
IndexResponse response = client.prepareIndex("index_name", "data")
.setSource(indexString)
.setId(123) <----- supply the ID of the document you want to replace
.execute()
.actionGet();
Otherwise, just so you know, in ES you have the option to do a partial update. That is, only update certain fields in the document. This can be done with a script or by providing a partial document. Have a look at the documentation for the Update API.
In either case, you need to provide ES with the ID for the document you wish to modify.

document hierarchical JSON payload with Spring REST Docs

I started to use Spring REST Docs to document a simple REST API. I have a payload that has some hierarchical structure, for example like this (company with employees).
{
"companyName": "FooBar",
"employee":
[
{
"name": "Lorem",
"age": "42"
},
{
"name": "Ipsum",
"age": "24"
}
]
}
I would like to separate the documentation of the company object (name and array of employees) and the employee object (employee name and age).
Using the org.springframework.restdocs.payload.PayloadDocumentation.responseFields like explained here forces me to document all fields but in case I only want to document the employee fields - how can I achieve this?
I have no problem to document the company without the employee details because if a field is document the descendants are treated as been documented also. But I can not document the employee structure on its own and I have no dedicated payload for this structure without the company root object.

Inspired by this question, I've implemented an enhancement which makes the original answer (see below) obsolete.
If you use 1.0.0.BUILD-SNAPSHOT (available from https://repo.spring.io/libs-snapshot), you can now mark a field as ignored. Ignored fields count has having been documented without actually appearing in the documentation.
Given that you want to separate the documentation, having two document calls makes sense. In the first you can document the company name and the array of employees. In the second you document the employees array and mark the company name as ignored.
Your test would look something like this:
mockMvc.perform(get("/company/5").accept(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andDo(document("company",
responseFields(
fieldWithPath("companyName").description(
"The name of the company"),
fieldWithPath("employee").description(
"An array of the company's employees"))))
.andDo(document("employee",
responseFields(
fieldWithPath("companyName").ignored(),
fieldWithPath("employee[].name").description(
"The name of the employee"),
fieldWithPath("employee[].age").description(
"The age of the employee"))));
You'll end up with two directories of snippets, one named company and one named employee. You can then use the response-fields.adoc snippet from each.
Original answer
There's no explicit support for ignoring a field when you're documenting a request or a response, but I think you can probably achieve what you want by using a preprocessor to remove the fields that you don't want to document.
Given that you want to separate the documentation, having two document calls makes sense. In the first you can document the company name and the array of employees. In the second you need to preprocess the request to remove the company and then document the employees array.
Your test would look something like this:
mockMvc.perform(get("/company/5").accept(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andDo(document("company",
responseFields(
fieldWithPath("companyName").description(
"The name of the company"),
fieldWithPath("employee").description(
"An array of the company's employees"))))
.andDo(document("employee",
preprocessResponse(removeCompany()),
responseFields(
fieldWithPath("employee[].name").description(
"The name of the employee"),
fieldWithPath("employee[].age").description(
"The age of the employee"))));
Note the use of preprocessResponse in the second document call. removeCompany returns a preprocessor that uses a custom ContentModifier to remove company name from the response:
private OperationPreprocessor removeCompany() {
return new ContentModifyingOperationPreprocessor(new ContentModifier() {
#Override
public byte[] modifyContent(byte[] originalContent, MediaType contentType) {
ObjectMapper objectMapper = new ObjectMapper();
try {
Map<?, ?> map = objectMapper.readValue(originalContent, Map.class);
map.remove("companyName");
return objectMapper.writeValueAsBytes(map);
}
catch (IOException ex) {
return originalContent;
}
}
});
}
You'll end up with two directories of snippets, one named company and one named employee. You can then use the response-fields.adoc snippet from each.
While the above will work, it's harder than it needs to be. I've opened an issue so that the preprocessing to modify the response's content will no longer be necessary.

This answer is for those who came here at the present time with the same question. Using version 2.0.x of Spring Rest Docs, you can do the following:
Document the first level with PayloadDocumentation.relaxedResponseFields methods (Any undocumented field will be ignored):
Document the second level with PayloadDocumentation.beneathPath that will extract the subsection of the JSON payload found beneath the given path
This will generate a response-fields.snippet file and a response-fields-beneath-employee.snippet file
.consumeWith(document("company",
PayloadDocumentation.relaxedResponseFields(
PayloadDocumentation.fieldWithPath("companyName").description("The name of the company"),
PayloadDocumentation.fieldWithPath("employee").description("An array of the company's employees")
),
//Using PayloadDocumentation.beneathPath(..) to document separately employee parts
PayloadDocumentation.responseFields(
PayloadDocumentation.beneathPath("employee"),
PayloadDocumentation.fieldWithPath("employee.name").description("The name of the employee"),
PayloadDocumentation.fieldWithPath("employee.age").description("The age of the employee")
)
)

Creating a document in Java

I am creating a KML document in Java. Inside of it i have to add many similar elements, resulting in the need to add a function, where I can pass needed arguments.
The problem is that when I try to add a part of the document into the main document it shows error, or creates malformed document.
Here's a code snippet:
Element style = doc.createElement("Style");
style.setAttribute("id", "green");
dnode.appendChild(style);
Element polyStyle = doc.createElement("PolyStyle");
style.appendChild(polyStyle);
Element color = doc.createElement("color");
color.appendChild(doc.createTextNode("5014F064"));
polyStyle.appendChild(color);
Element iconStyle = doc.createElement("IconStyle");
style.appendChild(iconStyle);
color = doc.createElement("color");
color.appendChild(doc.createTextNode("5014F064"));
iconStyle.appendChild(color);
Element "dnode" is a Document element inside xml. I want to try something like this:
doc.appendChild(addFeatureStyle("red", "501400FA"));
Called three times with different parameters but have no idea how to include it. I want to add function written above, calling the code snippet.
Should the function "addFeatureStyle" return element, or a string, or something else?

I'm not sure I understand your question, but I'll try to answer:
Should the function "addFeatureStyle" return element, or a string, or something else?
You're calling the method appendChild() with the value returned by addFeatureStyle("red", "501400FA") as argument.
The documentation of appendChild() shows that it takes a Node as argument. So the return type of addFeatureStyle() can't be a String: String doesn't implement the Node interface. The return type of addFeatureStyle() must be Node, or a class implementing Node, or an interface extending Node.