Develop Reference

How To Extract Json Data Filed With RestApi?

i make a post to an api with rest assured. and than i try to make sure expected data from responsed data ,
but i got some errors like this -> "java.lang.IllegalArgumentException: The parameter "data" was used but not defined. Define parameters using the JsonPath.params(...) function"
my code:
String payload_data = "{" +
"\"Time\":1638057600, " +
"\"exampleType\":example, " +
"\"Id\":[2]}";
RestAssured.defaultParser = Parser.JSON;
given().
contentType(ContentType.JSON).
body(payload_data).
when().
post(api_url).
then().
statusCode(200).
body("data.examples.2.exampleData", equalTo("33"));
}
my json data
{
"success": true,
"data": {
"examples": {
"2": {
"ex_data": 0,
"exampleData": 33,
"data_ex": 0,
}
}
}

First, I tested path "data.examples.2.exampleData" with your json, it works fine. No problem.
You made some mistakes here.
Your payload is invalid json.
{
"Time": 1638057600,
"exampleType": example, //it must be number or String with double quote
"Id": [
2
]
}
You are comparing 2 things with different data types.
"data.examples.2.exampleData" --> int 33
equalTo("33") --> String "33"
Fix: body("data.examples.2.exampleData", equalTo(33));

How to get Task Token in AWS Lambda Function

I have an AWS Step Function with the handlers implemented in Java.
My step function definition:
definition:
Comment: Steps for issuing a card
StartAt: RecipientFraudChecks
States:
RecipientFraudChecks:
Type: Task
Next: SaveTaskToken
Resource: arn:aws:lambda:eu-west-1:099720403855:RecipientFraudChecks
SaveTaskToken:
Type: Task
Resource: arn:aws:lambda:eu-west-1:12345678:function:SaveTaskToken
End: true
I have a Java project and all the Lambda Function handlers are defined there:
public class SaveTaskToken implements RequestHandler<Map<String,String>, String> {
....
#Override
public String handleRequest(Map<String, String> input, final Context context) {
// do the fraud checks
System.out.println("the context is: " + gson.toJson(context));
System.out.println("input: " + gson.toJson(input));
}
I'm running the step function locally using AWS SAM, and triggering according to this: https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-lambda.html#install-sam
Within context I would expect to see the Task Token, but I do not. Logs show:
the context is: {
"memoryLimit": 512,
"awsRequestId": "5065a9aa-1a4a-46fe-9b58-7dc2194f92b7",
"logGroupName": "aws/lambda/SaveTaskToken",
"logStreamName": "$LATEST",
"functionName": "SaveTaskToken",
"functionVersion": "$LATEST",
"invokedFunctionArn": "",
"cognitoIdentity": {
"identityId": "",
"poolId": ""
},
"logger": {}
}
In fact its nothing like the global Context I should expect in the docs.
What am I doing wrong? How can I get the Task Token?
EDIT
I added Parameters property to 'SaveTaskToken' and changed resource to arn:aws:states:::lambda:invoke.waitForTaskToken and know I can get the Task Token:
definition:
Comment: Steps for issuing a card
StartAt: RecipientFraudChecks
States:
RecipientFraudChecks:
Type: Task
Next: SaveTaskToken
Resource: arn:aws:lambda:eu-west-1:099720403855:RecipientFraudChecks
SaveTaskToken:
Type: Task
Resource: arn:aws:states:::lambda:invoke.waitForTaskToken
Parameters:
FunctionName: arn:aws:lambda:eu-west-1:12345678:function:SaveTaskToken
Payload:
taskToken
End: true
In the logs I can see:
the input is: {
"taskToken": "5286"
}
It has caused another problem - it overrides the input to the state machine. Im passing in the input:
{"giftCode": "xxx"}
In the first Lambda function, RecipientFraudChecks, I can get the input. However, in the second, since adding the Parameters property, I now can no longer get the input to the state machine, only the task token...
EDIT
Have implemented the answer here: https://stackoverflow.com/a/66995869/1246159
{
"Comment": "Steps for issuing a card",
"StartAt": "RecipientFraudChecks",
"States": {
"RecipientFraudChecks": {
"Type": "Task",
"Next": "PauseCardIfNecessary",
"ResultPath": "$.firstLambdaOutput",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:RecipientFraudChecks"
},
"PauseCardIfNecessary": {
"Type": "Task",
"Next": "GetOrCreateClient",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:PauseCardIfNecessary",
"Payload": {
"token.$": "$$.Task.Token",
"otherInput.$": "$"
}
}
},
"GetOrCreateClient": {
"Type": "Task",
"Next": "GetOrAccountClient",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetOrCreateClient"
},
"GetOrAccountClient": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetOrAccountClient",
"End": true
}
}
}
But I get another error, here are the logs:
arn: aws: states: eu-west-1: 123456789012: execution: HelloWorld5: cardIssue: {
"Type": "TaskStateExited",
"PreviousEventId": 5,
"StateExitedEventDetails": {
"Name": "RecipientFraudChecks",
"Output": "{\"inputToStep\":\"xxxx\",\"firstLambdaOutput\":\"output of recipient lambda\"}"
}
} arn: aws: states: eu-west-1: 123456789012: execution: HelloWorld5: cardIssue: {
"Type": "TaskStateEntered",
"PreviousEventId": 6,
"StateEnteredEventDetails": {
"Name": "PauseCardIfNecessary",
"Input": "{\"inputToStep\":\"xxxx\",\"firstLambdaOutput\":\"output of recipient lambda\"}"
}
} arn: aws: states: eu-west-1: 123456789012: execution: HelloWorld5: cardIssue: {
"Type": "ExecutionFailed",
"PreviousEventId": 7,
"ExecutionFailedEventDetails": {
"Error": "States.Runtime",
"Cause": "An error occurred while executing the state 'PauseCardIfNecessary' (entered at the event id #7). The value for the field 'token.$' must be a valid JSONPath expression"
}
}

Task token is not automatically passed in lambda context, it needs to be passed as input.
The context here is not context of lambda function, but the context of the task and to grab details from context, we can use $$. within step function definition,
Ex:
To get Task Token $$.Task.Token
To get start time $$.Execution.StartTime
Example Step function:
First Task executes a Lambda with step function input.
Appends the output of first lambda with step function input.
Second Task executes another lambda with resource waitForTaskToken, right here, we grab the task token and pass it along with output of previous step as input.
Step functions waits until it gets a SendTaskSuccess or SendTaskFailure
{
"StartAt": "fist-lambda-invoke-sync",
"States": {
"fist-lambda-invoke-sync": {
"Next": "second-lambda-invoke-task-token",
"Type": "Task",
"ResultPath": "$.firstLambdaOutput",
"Resource": "arn:aws:lambda:us-east-1:11112223333:function:myfirstlambda"
},
"second-lambda-invoke-task-token": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:11112223333:function:mysecondlambda",
"Payload": {
"token.$": "$$.Task.Token",
"otherInput.$": "$"
}
}
}
}
}
Input to Step function:
{
"inputToStep": "myValue"
}
Output of First Lambda: assuming it is a string value "10". It will be appended to input Json because of "ResultPath": "$.firstLambdaOutput"
{
"inputToStep": "myValue",
"firstLambdaOutput": "10"
}
Input to Second Lambda: Will receive below json as input with task token appended, because of "token.$": "$$.Task.Token" and "otherInput.$": "$"
{ otherInput: { inputToStep: 'myValue', firstLambdaOutput: '10' },
token: 'This is where Task Token generated by step function will be sent' }

Regarding the input, there seems to be an easy solution in the docs [1]:
Instead of hard-coding the event payload in the state machine definition, you can use the input from the state machine execution. The following example uses the input specified when you run the state machine as the event payload:
"Payload.$": "$"
There is another example in the docs [2]:
{
"StartAt":"GetManualReview",
"States":{
"GetManualReview":{
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters":{
"FunctionName":"get-model-review-decision",
"Payload":{
"model.$":"$.new_model",
"token.$":"$$.Task.Token"
},
"Qualifier":"prod-v1"
},
"End":true
}
}
}
You can possibly change "model.$":"$.new_model" to something like "input.$":"$" and get the desired Lambda payload.
Which translates to the following in YAML:
Parameters:
Payload:
input.$: "$"
token.$: "$$.Task.Token"
[1] https://docs.aws.amazon.com/lambda/latest/dg/services-stepfunctions.html#services-stepfunctions-setup
[2] https://docs.amazonaws.cn/en_us/step-functions/latest/dg/connect-lambda.html

ElasticSearch Make Field non-searchable from java

I am currently working on elastic search through my java Application . I know how to index the Java pojo using RestHighLevelClient. How i can make search only on new fields not the complete pojo.?
public class Employee{
private long id;
private String name;
private String designation;
private String address; //want to index but not searchable in elastic search
}
My Code for indexing is below which is working fine:
public String saveToEs(Employee employee) throws IOException {
Map<String, Object> map = objectMapper.convertValue(employee, Map.class);
IndexRequest indexRequest =
new IndexRequest(INDEX, TYPE, employee.getId().toString()).source(map, XContentType.JSON);
IndexResponse indexResponse = client.index(indexRequest, RequestOptions.DEFAULT);
I need to do this in java .Any help please or good link ?

Writing another answer for RestHighLevelClient As another answer is useful for people not using the Rest client and adding this in the first answer makes it too long.
Note: you are passing the type which is deprecated in ES 7.X and I am using the ES 7.X version, so my code is according to 7.X.
CreateIndexRequest request = new CreateIndexRequest("employee");
Map<String, Object> name = new HashMap<>();
name.put("type", "text");
Map<String, Object> address = new HashMap<>();
address.put("type", "text");
address.put("index", false);
Map<String, Object> properties = new HashMap<>();
properties.put("name", name);
properties.put("address", address);
Map<String, Object> mapping = new HashMap<>();
mapping.put("properties", properties);
request.mapping(mapping);
CreateIndexResponse createIndexResponse = client.indices().create(request, RequestOptions.DEFAULT);
Important points
I've used only 2 fields for illustration purpose, one of which is address field which is not searchable, and to do that I used, address.put("index", false); , while name is searchable field and there this option isn't present.
I've created index mapping using the Map method which is available in this official ES doc.
you can check the mapping created by this code, using mapping REST API.
Below is the mapping generated for this code in my system and you can see, index: false is added in the address field.
{
"employee": {
"mappings": {
"properties": {
"address": {
"type": "text",
"index": false
},
"name": {
"type": "text"
}
}
}
}
}
You can just use the same search JSON mentioned in the previous answer, to test that it's not searchable.

Use the index option as false on the address field, which is by default true to make it unsearchable. As mention in the same official ES link:
The index option controls whether field values are indexed. It accepts
true or false and defaults to true. Fields that are not indexed are
not queryable.
Let me show you how can you test it using the REST API and then the java code(using rest-high level client) to accomplish it.
Mapping
{
"mappings": {
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "text"
},
"designation": {
"type": "text"
},
"address": {
"type": "text",
"index" : false --> made `index` to false
}
}
}
}
Index few docs
{
"address" : "USA",
"name" : "Noshaf",
"id" : 234567892,
"designation" : "software engineer"
}
{
"address" : "USA california state",
"name" : "opster",
"id" : 234567890,
"designation" : "software engineer"
}
A simple match search query in JSON format on address field
{
"query": {
"match" : {
"address" : "USA"
}
}
}
Exception from Elasticsearch clearly mention, it's not searchable
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Cannot search on field [address] since it is not indexed."
}

Protobuf map type JSON format uses string literal "key" and "value" not the actual values

I am trying to convert a protobuf object to JSON format using com.googlecode.protobuf.format.JsonFormat but the map type came out unexpected.
My message is like this
message Response {
repeated Candidate candidates = 1;
map<string, ErrorMessage> errors = 2;
}
message ErrorMessage {
string message = 0;
ErrorType type = 1;
}
enum ErrorType {
ERROR = 0;
WARNING = 1;
}
The issue is the JSON format of the Response object I created
Response response = ...
Return new ResponseEntity<>(new JsonFormat().printToString(response), HttpStatus.OK);
I expect the errors be formatted as a map keyed by the string value (of the map key)
...
"errors": {
"someID" : {
"message": "blah blah",
"type": "ERROR"
}
}
However the actual output is (I evaluated only the new JsonFormat().printToString(response) part in intellij)
...
"errors": {
"key": "someID",
"value": {
"message": "blah blah",
"type": "ERROR"
}
}
I hope it's some small configuration I missed to make protobuf (or Jackson?) to be aware of the actual key value ? not using "key" and "value".
BTW, what's the point of having literal "key" and "value" field in a map type ? You can't do constituent lookup with it and you might just use a custom type/object.

This code works perfectly for me:
test.proto
syntax = "proto2";
package by.dev.madhead;
option java_outer_classname = "SO";
message Candidate {
}
enum ErrorType {
ERROR = 0;
WARNING = 1;
}
message ErrorMessage {
required string message = 1;
required ErrorType type = 2;
}
message Response {
repeated Candidate candidates = 1;
map<string, ErrorMessage> errors = 2;
}
App.java
public class App {
public static void main(String[] args) throws InvalidProtocolBufferException {
SO.Response response = SO.Response.newBuilder()
.addCandidates(SO.Candidate.newBuilder().build())
.addCandidates(SO.Candidate.newBuilder().build())
.addCandidates(SO.Candidate.newBuilder().build())
.putErrors("error1", SO.ErrorMessage.newBuilder().setMessage("error1").setType(SO.ErrorType.ERROR).build())
.putErrors("error2", SO.ErrorMessage.newBuilder().setMessage("error2").setType(SO.ErrorType.WARNING).build())
.build();
System.out.println(JsonFormat.printer().print(response));
}
}
The output is:
{
"candidates": [{
}, {
}, {
}],
"errors": {
"error1": {
"message": "error1",
"type": "ERROR"
},
"error2": {
"message": "error2",
"type": "WARNING"
}
}
}
Which has no keys and value as you see. Make sure that you printed not the message itself, but the result of JsonFormat.printer().print(). Basically, key and values you've seen are from internal toString() implementation of Protobuf Message.
And the full class name for JsonFormat is com.google.protobuf.util.JsonFormat, not com.googlecode.protobuf.format.JsonFormat.

Gerrit rest api returns empty JSONArray to java environment, but returns non empty to the browser

I faced following problem:
I wanna get list of changes from gerrit repository using this api function https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#list-changes
When I sent GET request in browser I got response
[
{
"id": "business~develop~I65d58a2345346cb35a0ecbb63d6ed896c7",
"project": "business",
"branch": "develop",
"hashtags": [],
"change_id": "I65d58a2a045645636cb35a0ecbb63d6ed896c7",
"subject": "Leader property deleted from organizational unit classes. Org unit\u0027s employees cascad type removed. Employee\u0027s roles amount limited by 1 in orgUnit.xsd. Employees identifiers changed in orgUnit.xml. Updated domain model. DataUploadRestService created. Emp",
"status": "NEW",
"created": "2015-05-12 14:31:48.226000000",
"updated": "2015-07-07 07:34:35.195000000",
"mergeable": true,
"insertions": 3100,
"deletions": 1358,
"_number": 589,
"owner": {
"_account_id": 1003473
}
}
]
But when I send GET request using RestEasy or other way to send request using Java code:
try {
final ResteasyClient client = new ResteasyClientBuilder().build();
final ResteasyWebTarget target = client
.target("https://gml-jbpm.gomel.iba.by/gerrit/changes/");
String response = target.request().get(String.class);
final JSONArray array = new JSONArray(response);
System.out.println("RestEasy response: " + array);
} catch (final Exception e) {
e.printStackTrace();
}
I get empty response:
[]
How to solve this problem? Why I get empty JSONArray?