I started to use Spring REST Docs to document a simple REST API. I have a payload that has some hierarchical structure, for example like this (company with employees).
{
"companyName": "FooBar",
"employee":
[
{
"name": "Lorem",
"age": "42"
},
{
"name": "Ipsum",
"age": "24"
}
]
}
I would like to separate the documentation of the company object (name and array of employees) and the employee object (employee name and age).
Using the org.springframework.restdocs.payload.PayloadDocumentation.responseFields like explained here forces me to document all fields but in case I only want to document the employee fields - how can I achieve this?
I have no problem to document the company without the employee details because if a field is document the descendants are treated as been documented also. But I can not document the employee structure on its own and I have no dedicated payload for this structure without the company root object.
Inspired by this question, I've implemented an enhancement which makes the original answer (see below) obsolete.
If you use 1.0.0.BUILD-SNAPSHOT (available from https://repo.spring.io/libs-snapshot), you can now mark a field as ignored. Ignored fields count has having been documented without actually appearing in the documentation.
Given that you want to separate the documentation, having two document calls makes sense. In the first you can document the company name and the array of employees. In the second you document the employees array and mark the company name as ignored.
Your test would look something like this:
mockMvc.perform(get("/company/5").accept(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andDo(document("company",
responseFields(
fieldWithPath("companyName").description(
"The name of the company"),
fieldWithPath("employee").description(
"An array of the company's employees"))))
.andDo(document("employee",
responseFields(
fieldWithPath("companyName").ignored(),
fieldWithPath("employee[].name").description(
"The name of the employee"),
fieldWithPath("employee[].age").description(
"The age of the employee"))));
You'll end up with two directories of snippets, one named company and one named employee. You can then use the response-fields.adoc snippet from each.
Original answer
There's no explicit support for ignoring a field when you're documenting a request or a response, but I think you can probably achieve what you want by using a preprocessor to remove the fields that you don't want to document.
Given that you want to separate the documentation, having two document calls makes sense. In the first you can document the company name and the array of employees. In the second you need to preprocess the request to remove the company and then document the employees array.
Your test would look something like this:
mockMvc.perform(get("/company/5").accept(MediaType.APPLICATION_JSON))
.andExpect(status().isOk())
.andDo(document("company",
responseFields(
fieldWithPath("companyName").description(
"The name of the company"),
fieldWithPath("employee").description(
"An array of the company's employees"))))
.andDo(document("employee",
preprocessResponse(removeCompany()),
responseFields(
fieldWithPath("employee[].name").description(
"The name of the employee"),
fieldWithPath("employee[].age").description(
"The age of the employee"))));
Note the use of preprocessResponse in the second document call. removeCompany returns a preprocessor that uses a custom ContentModifier to remove company name from the response:
private OperationPreprocessor removeCompany() {
return new ContentModifyingOperationPreprocessor(new ContentModifier() {
#Override
public byte[] modifyContent(byte[] originalContent, MediaType contentType) {
ObjectMapper objectMapper = new ObjectMapper();
try {
Map<?, ?> map = objectMapper.readValue(originalContent, Map.class);
map.remove("companyName");
return objectMapper.writeValueAsBytes(map);
}
catch (IOException ex) {
return originalContent;
}
}
});
}
You'll end up with two directories of snippets, one named company and one named employee. You can then use the response-fields.adoc snippet from each.
While the above will work, it's harder than it needs to be. I've opened an issue so that the preprocessing to modify the response's content will no longer be necessary.
This answer is for those who came here at the present time with the same question. Using version 2.0.x of Spring Rest Docs, you can do the following:
Document the first level with PayloadDocumentation.relaxedResponseFields methods (Any undocumented field will be ignored):
Document the second level with PayloadDocumentation.beneathPath that will extract the subsection of the JSON payload found beneath the given path
This will generate a response-fields.snippet file and a response-fields-beneath-employee.snippet file
.consumeWith(document("company",
PayloadDocumentation.relaxedResponseFields(
PayloadDocumentation.fieldWithPath("companyName").description("The name of the company"),
PayloadDocumentation.fieldWithPath("employee").description("An array of the company's employees")
),
//Using PayloadDocumentation.beneathPath(..) to document separately employee parts
PayloadDocumentation.responseFields(
PayloadDocumentation.beneathPath("employee"),
PayloadDocumentation.fieldWithPath("employee.name").description("The name of the employee"),
PayloadDocumentation.fieldWithPath("employee.age").description("The age of the employee")
)
)
Related
I have two indexes, say A and B, with the following structure:
A contains among other things a field called agent_key like below:
[...]
"_source": {
"id": "1599048000PbsPVjP2bRWK",
"property_time": 1616637650000,
"property_key": "YO88AN0000027654869134276550001",
"**agent_key**": "50001000002765",
[...]
B contains agents documents. It contains among other things the agent_key field like below:
[...]
},
"upload_time": 1616637650000,
"agent_key": "50001000002765",
[...]
Using the Java high-level rest client it is easy to search each index separately. I have the use case where I need to return a SearchResponse that contains both, or rather the response from A should be enriched with the response from B.
How can I achieve that without reinventing the wheel? I am sure this is not a new use case.
Here my very poor attempt to try to find a solution: (it is pseudocode)
public SearchResponse getEnhancedProperty() throws IOException {
SearchResponse property = client.search(searchRequest, RequestOptions.DEFAULT);
SearchResponse agent = getAgent(extractAgentKey(property));
return combineResponses(property,agent); //<=== No idea how to combine responses
}
I would suggest including document B as a part of document A, in your search -
first, you need to set ignore_unavailable to true in the query for A document, otherwise it will throw exception when it won't find doc b inside doc a.
second, you can set document b, as a nested field in document a, and set "includeInParent" to true, so you can access B fields directly from A document.
lets call B - a document from index B, and same for A.
for the first part, an example would explain better:
NativeSearchQuery query = new NativeSearchQueryBuilder().withQuery(queryBuilder)
.withIndicesOptions(IndicesOptions.LENIENT_EXPAND_OPEN).build()
please note that I used the LENIENT_EXPAND_OPEN option, since it includes the ignore_unavailable option.
for the second part, you cant use the following field annotation:
#Field(type = FieldType.Nested, includeInParent = true...)
then, you could set the B field inside A to the B search-result content:
A.setB(searchHit.getContent());
I want to use a single YAML file which contains several different objects - for different applications. I need to fetch one object to get an instance of MyClass1, ignoring the rest of docs for MyClass2, MyClass3, etc. Some sort of selective de-serializing: now this class, then that one... The structure of MyClass2, MyClass3 is totally unknown to the application working with MyClass1. The file is always a valid YAML, of course.
The YAML may be of any structure we need to implement such a multi-class container. The preferred parsing tool is snakeyaml.
Is it sensible? How can I ignore all but one object?
UPD: replaced all "document" with "object". I think we have to speak about the single YAML document containing several objects of different structure. More of it, the parser knows exactly only 1 structure and wants to ignore the rest.
UDP2: I think it is impossible with snakeyaml. We have to read all objects anyway - and select the needed one later. But maybe I'm wrong.
UPD2: sample config file
---
-
exportConfiguration781:
attachmentFieldName: "name"
baseSftpInboxPath: /home/user/somedir/
somebool: false
days: 9999
expected:
- ABC w/o quotes
- "Cat ABC"
- "Some string"
dateFormat: yyyy-MMdd-HHmm
user: someuser
-
anotherConfiguration:
k1: v1
k2:
- v21
- v22
This is definitely possible with SnakeYAML, albeit not trivial. Here's a general rundown what you need to do:
First, let's have a look what loading with SnakeYAML does. Here's the important part of the YAML class:
private Object loadFromReader(StreamReader sreader, Class<?> type) {
Composer composer = new Composer(new ParserImpl(sreader), resolver, loadingConfig);
constructor.setComposer(composer);
return constructor.getSingleData(type);
}
The composer parses YAML input into Nodes. To do that, it doesn't need any knowledge about the structure of your classes, since every node is either a ScalarNode, a SequenceNode or a MappingNode and they just represent the YAML structure.
The constructor takes a root node generated by the composer and generates native POJOs from it. So what you want to do is to throw away parts of the node graph before they reach the constructor.
The easiest way to do that is probably to derive from Composer and override two methods like this:
public class MyComposer extends Composer {
private final int objIndex;
public MyComposer(Parser parser, Resolver resolver, int objIndex) {
super(parser, resolver);
this.objIndex = objIndex;
}
public MyComposer(Parser parser, Resolver resolver, LoaderOptions loadingConfig, int objIndex) {
super(parser, resolver, loadingConfig);
this.objIndex = objIndex;
}
#Override
public Node getNode() {
return strip(super.getNode());
}
private Node strip(Node input) {
return ((SequenceNode)input).getValue().get(objIndex);
}
}
The strip implementation is just an example. In this case, I assumed your YAML looks like this (object content is arbitrary):
- {first: obj}
- {second: obj}
- {third: obj}
And you simply select the object you actually want to deserialize by its index in the sequence. But you can also have something more complex like a searching algorithm.
Now that you have your own composer, you can do
Constructor constructor = new Constructor();
// assuming we want to get the object at index 1 (i.e. second object)
Composer composer = new MyComposer(new ParserImpl(sreader), new Resolver(), 1);
constructor.setComposer(composer);
MyObject result = (MyObject)constructor.getSingleData(MyObject.class);
The answer of #flyx was very helpful for me, opening the way to workaround the library (in our case - snakeyaml) limitations by overriding some methods. Thanks a lot! It's quite possible there is a final solution in it - but not now. Besides, the simple solution below is robust and should be considered even if we'd found the complete library-intruding solution.
I've decided to solve the task by double distilling, sorry, processing the configuration file. Imagine the latter consisting of several parts and every part is marked by the unique token-delimiter. For the sake of keeping the YAML-likenes, it may be
---
#this is a unique key for the configuration A
<some YAML document>
---
#this is another key for the configuration B
<some YAML document
The first pass is pre-processing. For the given String fileString and String key (and DELIMITER = "\n---\n". for example) we select a substring with the key-defined configuration:
int begIndex;
do {
begIndex= fileString.indexOf(DELIMITER);
if (begIndex == -1) {
break;
}
if (fileString.startsWith(DELIMITER + key, begIndex)) {
fileString = fileString.substring(begIndex + DELIMITER.length() + key.length());
break;
}
// spoil alien delimiter and repeat search
fileString = fileString.replaceFirst(DELIMITER, " ");
} while (true);
int endIndex = fileString.indexOf(DELIMITER);
if (endIndex != -1) {
fileString = fileString.substring(0, endIndex);
}
Now we feed the fileString to the simple YAML parsing
ExportConfiguration configuration = new Yaml(new Constructor(ExportConfiguration.class))
.loadAs(fileString, ExportConfiguration.class);
This time we have a single document that must co-respond to the ExportConfiguration class.
Note 1: The structure and even the very content of the rest of configuration file plays absolutely no role. This was the main idea, to get independent configurations in a single file
Note 2: the rest of configurations may be JSON or XML or whatever. We have a method-preprocessor that returns a String configuration - and the next processor parses it properly.
I am taking a JSON file as input for a class and parsing the values using gson through respective data classes.
I want to call a function that takes a String value as an argument.
The string value allowed is decided from the values parsed from JSON file. Can I somehow check for that string value passed to the function at compile-time & give an error at compile-time?
Or If I can allow only certain values in the argument for the function based on the values from JSON
Detailed Explanation of use case:
I am building a SDK in which a the person using sdk inputs json String. The json is standardised and is parsed in my code.
{
"name": "Test",
"objects": [
{
"name": "object1",
"type": "object1"
}
]
}
Here name values and other values may vary based on the input by the developer using it but key remains same. But we need to call a function using the value in objects name parameter.
fun testMethod(objectName:String)
So developer calls the testMethod as testMethod(object1).
I need to validate object1 parameter based on json but is there any way possible restricting the test method parameter to object1 only & give error at compile time if the developer calls testMethod(obj1)
Right now I parse JSON & have checks inside the testMethod()
Sure it's possible to do, but somehow in different way, that you described. First of all, as you already mentioned this behavior could be done easily. For this purpose we have Objects.requireNotNull() or Guava.Preconditions(). At the same way you can define you checking but this will work on runtime only.
To do in compile time, you need to create Annotation Preprocessor. The same, as did in different libraries, and one of them, could be Lombok, with their NotNull and Nullable. Android annotation just provide mark and bound for IDE warning, but in their case they adding NotNull checking and throw exception for every annotation usage during compile time.
It's not an easy way, but it's what you are looking for.
No, it's impossible check it in compiler time. It's string handling, as numeric calculation.
In my app, I convert string to JSON and JSON to string, passing class descriptor. My aim is record JSON string in a text file to load in SQLite database. This code I've run in my desktop computer not in Android.
data class calcDescr (
...
)
val calc = CalcDescr(...)
// toJson: internal Kotlin data to JSON
val content = Gson().toJson(calc)
//==================
// Testing validity
// ================
// fromJson: JSON to internal Kotlin data.
// It needs pass the class descriptor. Uses *Java* token, but it's *Kotlin*
var testModel = Gson().fromJson(content, CalcDescr::class.java)
// toJson: internal Kotlin data to JSON again
var contentAgain = Gson().toJson(testModel)
// shoul be equal!
if (content == contentAgain) println("***ok***")
In my file, I write the variable content in a file
We have some web services which are being consumed by mobile clients , in which the mobile clients make some request's and we return response to them. Somehow if the client make any invalid request we throw Custom Exceptions .But recently the mobile client has made some request which was out of range for Long variable. The client has different variables for ex ::
{
"accountId":"343"
"Amount":"90909090909090909090"
}
In case of the value for accountId or Amount are made more than 19 digits we get HttpMessageNotReadable exception as the range is out of long value. But from exception i am not able to fetch for which variable the exception has been raised whether for accountId or Amount. From the exception i am getting this information in _path variable but i am not able to fetch it.
And in the path variable i get something like::
[com.Upload["AccountInfo"], com.Info["Account"]]
Does somebody know how to fetch this information.
The following code prints out the field which causes the exception.
InvalidFormatException invalidFormatException = (InvalidFormatException) exception
.getCause();
System.out.println(invalidFormatException.getPath().get(0)
.getFieldName());
#ArunM's answer works as long as the field is in 1st level viz the example given by OP.
But what happens when the field is in nested json? say paymentType has wrong value in the following example?
{
"userType": "CUSTOMER",
"payment": {
"amount": 123456,
"paymentType": "INTERNET_BANKING"
}
}
In the above example, if there's anything wrong with the value of userType, there will be just one element in _path.
But if any value inside the payment is wrong, say for example paymentType, there will be multiple elements in the _path variable. Each element representing the hierarchical attribute.
So for paymentType, _path will have 2 elements as illustrated below:
_path[0].fieldName = "payment"
_path[1].fieldName = "paymentType"
Hence the correct approach is to get the last element in _path as shown below. If needed, we can build the complete path by using all the elements.
InvalidFormatException ifx = (InvalidFormatException) exception.getCause();
System.out.println(ifx.getPath().get(ifx.size() - 1).getFieldName());
This I believe is the right approach to get the invalid attribute name.
Background
We are building a Restful API that should return data objects as JSON. In most of the cases it fine just to return the data object, but in some cases, f.ex. pagination or validation, we need to add some metadata to the response.
What we have so far
We have wrapped all json responses like this example:
{
"metadata" :{
"status": 200|500,
"msg": "Some message here",
"next": "http://api.domain.com/users/10/20"
...
},
"data" :{
"id": 1001,
"name": "Bob"
}
}
Pros
We can add helpful metadata to the response
Cons
In most cases we don't need the metadata field, and it adds complexity to the json format
Since it's not a data object any more, but more like a enveloped response, we can not use the response right away in f.ex backbone.js without extracting the data object.
Question
What is the best practices to add metadata to a json response?
UPDATE
What I've got so far from answers below:
Remove the metadata.status an return the http response code in the
http protocol instead (200, 500 ...)
Add error msg to body of an http 500 repsonse
For pagination i natural to have some metadata telling about the pagination structure, and the data nested in that structure
Small amount of meta data can be added to http header (X-something)
You have several means to pass metadata in a RESTful API:
Http Status Code
Headers
Response Body
For the metadata.status, use the Http Status Code, that's what's for!
If metadata is refers to the whole response you could add it as header fields.
If metadata refers only to part of the response, you will have to embed the metadata as part of the object.DON'T wrap the whole response in an artifical envelope and split the wrapper in data and metadata.
And finally, be consistent across your API with the choices you make.
A good example is a GET on a whole collection with pagination. GET /items
You could return the collection size, and current page in custom headers. And pagination links in standard Link Header:
Link: <https://api.mydomain.com/v1/items?limit=25&offset=25>; rel=next
The problem with this approach is when you need to add metadata referencing specific elements in the response. In that case just embed it in the object itself. And to have a consistent approach...add always all metadata to response. So coming back to the GET /items, imagine that each item has created and updated metadata:
{
items:[
{
"id":"w67e87898dnkwu4752igd",
"message" : "some content",
"_created": "2014-02-14T10:07:39.574Z",
"_updated": "2014-02-14T10:07:39.574Z"
},
......
{
"id":"asjdfiu3748hiuqdh",
"message" : "some other content",
"_created": "2014-02-14T10:07:39.574Z",
"_updated": "2014-02-14T10:07:39.574Z"
}
],
"_total" :133,
"_links" :[
{
"next" :{
href : "https://api.mydomain.com/v1/items?limit=25&offset=25"
}
]
}
Note that a collection response is an special case. If you add metadata to a collection, the collection can no longer be returned as an array, it must be an object with an array in it. Why an object? because you want to add some metadata attributes.
Compare with the metadata in the individual items. Nothing close to wrapping the entity. You just add some attributes to the resource.
One convention is to differentiate control or metadata fields. You could prefix those fields with an underscore.
Along the lines of #Charlie's comment: for the pagination part of your question you still need to bake the metadata into the response somhow, but the status and message attributes here are somewhat redundant, since they are already covered by the HTTP protocol itself (status 200 - model found, 404 - model not found, 403 - insufficient privs, you get the idea) (see spec). Even if your server returns an error condition you can still send the message part as the response body. These two fields will cover quite much of your metadata needs.
Personally, I have tended towards (ab)using custom HTTP headers for smaller pieces of metadata (with an X- prefix), but I guess the limit where that gets unpractical is pretty low.
I've expanded a bit about this in a question with a smaller scope, but I think the points are still valid for this question.
I suggest you to read this page https://www.odata.org/ You are not forced to use OData but the way they do the work is a good example of good practice with REST.
We had the same use case, in which we needed to add pagination metadata to a JSON response. We ended up creating a collection type in Backbone that could handle this data, and a lightweight wrapper on the Rails side. This example just adds the meta data to the collection object for reference by the view.
So we created a Backbone Collection class something like this
// Example response:
// { num_pages: 4, limit_value: 25, current_page: 1, total_count: 97
// records: [{...}, {...}] }
PageableCollection = Backbone.Collection.extend({
parse: function(resp, xhr) {
this.numPages = resp.num_pages;
this.limitValue = resp.limit_value;
this.currentPage = resp.current_page;
this.totalCount = resp.total_count;
return resp.records;
}
});
And then we created this simple class on the Rails side, to emit the meta data when paginated with Kaminari
class PageableCollection
def initialize (collection)
#collection = collection
end
def as_json(opts = {})
{
:num_pages => #collection.num_pages
:limit_value => #collection.limit_value
:current_page => #collection.current_page,
:total_count => #collection.total_count
:records => #collection.to_a.as_json(opts)
}
end
end
You use it in a controller like this
class ThingsController < ApplicationController
def index
#things = Thing.all.page params[:page]
render :json => PageableCollection.new(#things)
end
end
Enjoy. Hope you find it useful.
How about returning directly the object that you want in data, like return:
{
"id": 1001,
"name": "Bob"
}
And return in headers the metadata.
Option 1 (one header for all metadata JSON):
X-METADATA = '{"status": 200|500,"msg": "Some message here","next": "http://api.domain.com/users/10/20"...}'
Option 2 (one header per each metadata field):
X-METADATA-STATUS = 200|500
X-METADATA-MSG = "Some message here",
X-METADATA-NEXT = "http://api.domain.com/users/10/20"
...
Until now I was using like you, a complex JSON with two fields, one for data and one for metadata. But I'm thinking in starting using this way that I suggested, I think it will be more easy.
Remind that some server have size limit for HTTP headers, like this example: https://www.tutorialspoint.com/What-is-the-maximum-size-of-HTTP-header-values
JSON:API solves this by defining top-level meta and data properties.