Expected
The ability to use the special character "$" in the naming of a Kotlin or Java variable name. The API for EventRegistry (Example 4) requires the naming of the sub query as "$query". Is there a Kotlin or Java workaround in order to use "$" in a variable name?
Documentation
Example 4 - Request Body
{
"action": "getArticles",
"query": {
"$query": {
"$and": [
{
"dateStart":"2017-04-22",
"dateEnd":"2017-04-22"
},
{
"$or":[
{
"conceptUri":{
"$or": ["http://en.wikipedia.org/wiki/Barack_Obama"]
}
},
{
"keyword":"Trump"
}
]
},
{
"categoryUri":"dmoz/Business"
}
]
}
},
"articlesPage": 1,
"articlesCount": 100,
"articlesSortBy": "socialScore",
"articlesSortByAsc": false,
"articlesArticleBodyLen": -1,
"includeArticleSocialScore": true,
"resultType": "articles",
"apiKey": "YOUR_API_KEY"
}
Result
Lint error is thrown when attempting the following:
data class Query(val $query: SubQuery)
For Kotlin
surround the name with backticks (ascii code 96):
`$query`
Related
I'm thinking of using painless script, but it's too much complicated, is there any simple way ?
Say I have docs like this :
{'id': 1, 'keywords': '21;34;50' ...},
{'id': 2, 'keywords': '1;2;3' ...},
...
with a given string '1;5'
How can I filter the doc contains keywords 1 or 5 (not 21, 50 or others) ?
Many thanks！
You can define a custom analyzer to tokenize on semicolon. For instance, you can define it as following (not tested)
{
"settings": {
"analysis": {
"analyzer": {
"my_custom_analyzer": {
"tokenizer": "punctuation"
}
},
"tokenizer": {
"punctuation": {
"type": "pattern",
"pattern": "[;]"
}
}
}
}
}
More information on https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html
I have an AWS Step Function with the handlers implemented in Java.
My step function definition:
definition:
Comment: Steps for issuing a card
StartAt: RecipientFraudChecks
States:
RecipientFraudChecks:
Type: Task
Next: SaveTaskToken
Resource: arn:aws:lambda:eu-west-1:099720403855:RecipientFraudChecks
SaveTaskToken:
Type: Task
Resource: arn:aws:lambda:eu-west-1:12345678:function:SaveTaskToken
End: true
I have a Java project and all the Lambda Function handlers are defined there:
public class SaveTaskToken implements RequestHandler<Map<String,String>, String> {
....
#Override
public String handleRequest(Map<String, String> input, final Context context) {
// do the fraud checks
System.out.println("the context is: " + gson.toJson(context));
System.out.println("input: " + gson.toJson(input));
}
I'm running the step function locally using AWS SAM, and triggering according to this: https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-lambda.html#install-sam
Within context I would expect to see the Task Token, but I do not. Logs show:
the context is: {
"memoryLimit": 512,
"awsRequestId": "5065a9aa-1a4a-46fe-9b58-7dc2194f92b7",
"logGroupName": "aws/lambda/SaveTaskToken",
"logStreamName": "$LATEST",
"functionName": "SaveTaskToken",
"functionVersion": "$LATEST",
"invokedFunctionArn": "",
"cognitoIdentity": {
"identityId": "",
"poolId": ""
},
"logger": {}
}
In fact its nothing like the global Context I should expect in the docs.
What am I doing wrong? How can I get the Task Token?
EDIT
I added Parameters property to 'SaveTaskToken' and changed resource to arn:aws:states:::lambda:invoke.waitForTaskToken and know I can get the Task Token:
definition:
Comment: Steps for issuing a card
StartAt: RecipientFraudChecks
States:
RecipientFraudChecks:
Type: Task
Next: SaveTaskToken
Resource: arn:aws:lambda:eu-west-1:099720403855:RecipientFraudChecks
SaveTaskToken:
Type: Task
Resource: arn:aws:states:::lambda:invoke.waitForTaskToken
Parameters:
FunctionName: arn:aws:lambda:eu-west-1:12345678:function:SaveTaskToken
Payload:
taskToken
End: true
In the logs I can see:
the input is: {
"taskToken": "5286"
}
It has caused another problem - it overrides the input to the state machine. Im passing in the input:
{"giftCode": "xxx"}
In the first Lambda function, RecipientFraudChecks, I can get the input. However, in the second, since adding the Parameters property, I now can no longer get the input to the state machine, only the task token...
EDIT
Have implemented the answer here: https://stackoverflow.com/a/66995869/1246159
{
"Comment": "Steps for issuing a card",
"StartAt": "RecipientFraudChecks",
"States": {
"RecipientFraudChecks": {
"Type": "Task",
"Next": "PauseCardIfNecessary",
"ResultPath": "$.firstLambdaOutput",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:RecipientFraudChecks"
},
"PauseCardIfNecessary": {
"Type": "Task",
"Next": "GetOrCreateClient",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:123456789012:function:PauseCardIfNecessary",
"Payload": {
"token.$": "$$.Task.Token",
"otherInput.$": "$"
}
}
},
"GetOrCreateClient": {
"Type": "Task",
"Next": "GetOrAccountClient",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetOrCreateClient"
},
"GetOrAccountClient": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetOrAccountClient",
"End": true
}
}
}
But I get another error, here are the logs:
arn: aws: states: eu-west-1: 123456789012: execution: HelloWorld5: cardIssue: {
"Type": "TaskStateExited",
"PreviousEventId": 5,
"StateExitedEventDetails": {
"Name": "RecipientFraudChecks",
"Output": "{\"inputToStep\":\"xxxx\",\"firstLambdaOutput\":\"output of recipient lambda\"}"
}
} arn: aws: states: eu-west-1: 123456789012: execution: HelloWorld5: cardIssue: {
"Type": "TaskStateEntered",
"PreviousEventId": 6,
"StateEnteredEventDetails": {
"Name": "PauseCardIfNecessary",
"Input": "{\"inputToStep\":\"xxxx\",\"firstLambdaOutput\":\"output of recipient lambda\"}"
}
} arn: aws: states: eu-west-1: 123456789012: execution: HelloWorld5: cardIssue: {
"Type": "ExecutionFailed",
"PreviousEventId": 7,
"ExecutionFailedEventDetails": {
"Error": "States.Runtime",
"Cause": "An error occurred while executing the state 'PauseCardIfNecessary' (entered at the event id #7). The value for the field 'token.$' must be a valid JSONPath expression"
}
}
Task token is not automatically passed in lambda context, it needs to be passed as input.
The context here is not context of lambda function, but the context of the task and to grab details from context, we can use $$. within step function definition,
Ex:
To get Task Token $$.Task.Token
To get start time $$.Execution.StartTime
Example Step function:
First Task executes a Lambda with step function input.
Appends the output of first lambda with step function input.
Second Task executes another lambda with resource waitForTaskToken, right here, we grab the task token and pass it along with output of previous step as input.
Step functions waits until it gets a SendTaskSuccess or SendTaskFailure
{
"StartAt": "fist-lambda-invoke-sync",
"States": {
"fist-lambda-invoke-sync": {
"Next": "second-lambda-invoke-task-token",
"Type": "Task",
"ResultPath": "$.firstLambdaOutput",
"Resource": "arn:aws:lambda:us-east-1:11112223333:function:myfirstlambda"
},
"second-lambda-invoke-task-token": {
"End": true,
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters": {
"FunctionName": "arn:aws:lambda:us-east-1:11112223333:function:mysecondlambda",
"Payload": {
"token.$": "$$.Task.Token",
"otherInput.$": "$"
}
}
}
}
}
Input to Step function:
{
"inputToStep": "myValue"
}
Output of First Lambda: assuming it is a string value "10". It will be appended to input Json because of "ResultPath": "$.firstLambdaOutput"
{
"inputToStep": "myValue",
"firstLambdaOutput": "10"
}
Input to Second Lambda: Will receive below json as input with task token appended, because of "token.$": "$$.Task.Token" and "otherInput.$": "$"
{ otherInput: { inputToStep: 'myValue', firstLambdaOutput: '10' },
token: 'This is where Task Token generated by step function will be sent' }
Regarding the input, there seems to be an easy solution in the docs [1]:
Instead of hard-coding the event payload in the state machine definition, you can use the input from the state machine execution. The following example uses the input specified when you run the state machine as the event payload:
"Payload.$": "$"
There is another example in the docs [2]:
{
"StartAt":"GetManualReview",
"States":{
"GetManualReview":{
"Type":"Task",
"Resource":"arn:aws:states:::lambda:invoke.waitForTaskToken",
"Parameters":{
"FunctionName":"get-model-review-decision",
"Payload":{
"model.$":"$.new_model",
"token.$":"$$.Task.Token"
},
"Qualifier":"prod-v1"
},
"End":true
}
}
}
You can possibly change "model.$":"$.new_model" to something like "input.$":"$" and get the desired Lambda payload.
Which translates to the following in YAML:
Parameters:
Payload:
input.$: "$"
token.$: "$$.Task.Token"
[1] https://docs.aws.amazon.com/lambda/latest/dg/services-stepfunctions.html#services-stepfunctions-setup
[2] https://docs.amazonaws.cn/en_us/step-functions/latest/dg/connect-lambda.html
I need to transform below Input JSON to output JSON and not sure about how to write spec for that. Need to re-position one field ("homePage") as a root element. Any help or suggestion would be appreciated.
Input JSON :
[{
"uuid": "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId": 123456,
"page": {
"indexable": true,
"rootLevel": false,
"homePage": false
}
}]
Output JSON :
[{
"uuid": "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId": 123456,
"homePage": false,
"page": {
"indexable": true,
"rootLevel": false
}
}]
This Jolt Spec should work for you. Tested with https://jolt-demo.appspot.com/
[
{
"operation": "shift",
"spec": {
"*": {
"uuid": "[&1].uuid",
"pageId": "[&1].pageId",
"page": {
"indexable": "[&2].page.indexable",
"rootLevel": "[&2].page.rootLevel",
"homePage": "[&2].homePage"
}
}
}
}
]
input:
{
"uuid" : "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId" : 123456,
"page" : {
"indexable" : true,
"rootLevel" : false
},
"homePage" : false
}
output:
[ {
"uuid" : "cac40601-ffc9-4fd0-c5a1-772ac65f0587",
"pageId" : 123456,
"page" : {
"indexable" : true,
"rootLevel" : false
},
"homePage" : false
} ]
Explanation:
From the javadoc
& Path lookup
As Shiftr processes data and walks down the spec, it maintains a data structure describing the path it has walked.
The & wildcard can access data from that path in a 0 major, upward oriented way.
Example:
{
"foo" : {
"bar": {
"baz": // &0 = baz, &1 = bar, &2 = foo
}
}
}
Next thing: How to wrap the output object into the array?
A good example can be found in this post.
So, in our case:
"[&1].uuid" says:
Place the uuid value in the object inside the array. The index of the array is indicated by the &1 wildcard. For uuid it will be the index of the array, where the object with uuid key is placed in the original json.
Next, [&2] is similar to [&1]. However, looking at the "indexable" key, it is one level deeper in the input json. Thats why instead of [&1] we used [&2] (have a look again at the foo-bar example from the docs).
How to transform JSON response retrieved from external system to meaningful data (key/value pairs) in ESQL?
Retrieved JSON:
{
"data": [
{
"name": "application.info.header",
"value": "headerValue"
},
{
"name": "entity.statistics.name.fullName",
"value": "fullNameValue"
},
{
"name": "application.info.matter",
"value": "matterValue"
},
{
"name": "entity.statistics.skill",
"value": "skillValue"
}
]
}
where,
name ~ hierarchy of JSON (last attribute being the key)
value ~ value against the key
Expected JSON:
{
"data": {
"application": {
"info": {
"header": "headerValue",
"matter": "matterValue"
}
},
"entity": {
"statistics": {
"name": {
"fullName": "fullNameValue"
},
"skill": "skillValue"
}
}
}
}
Needless to say this can be easily achieved in Java through Split method - I'm looking for a suitable method in ESQL.
Current ESQL Module:
CREATE COMPUTE MODULE getDetails_prepareResponse
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
DECLARE data REFERENCE TO InputRoot.JSON.Data.data.Item[1];
SET OutputRoot.JSON.Data = InputRoot.JSON.Data;
SET OutputRoot.JSON.Data.data = NULL;
WHILE LASTMOVE(data) DO
DECLARE keyA CHARACTER SUBSTRING(data.name BEFORE '.');
DECLARE name CHARACTER SUBSTRING(data.name AFTER '.');
DECLARE keyB CHARACTER SUBSTRING(name BEFORE '.');
DECLARE key CHARACTER SUBSTRING(name AFTER '.');
CREATE LASTCHILD OF OutputRoot.JSON.Data.data.{EVAL('keyA')}.{EVAL('keyB')}
NAME key VALUE data.value;
MOVE data NEXTSIBLING;
END WHILE;
RETURN TRUE;
END;
END MODULE;
This is currently handled through SUBSTRING method in ESQL (for 3 levels only), but now the JSON levels are dynamic (no limit to key/value pairs) as per requirements.
You could implement your own procedure to split a string. Take a look at this answer for an example.
ESQL for splitting a string into mulitple values
The method splits S on Delim into an array in Env (Environment.Split.Array[]) and removes Environment.Split before refilling it.
I am having a problem while querying elastic search. The below is my query
GET _search {
"query": {
"bool": {
"must": [{
"match": {
"name": "SomeName"
}
},
{
"match": {
"type": "SomeType"
}
},
{
"match": {
"productId": "ff134be8-10fc-4461-b620-79s51199c7qb"
}
},
{
"range": {
"request_date": {
"from": "2018-08-22T12:16:37,392",
"to": "2018-08-28T12:17:41,137",
"format": "YYYY-MM-dd'T'HH:mm:ss,SSS"
}
}
}
]
}
}
}
I am using three match queries and a range query in the bool query. My intention is getting docs with these exact matches and with in this date range. Here , if i change name and type value, i wont get the results. But for productId , if i put just ff134be8, i would get results. Anyone knows why is that ? . The exact match works on name and type but not for productId
You need to set the mapping of your productId to keyword to avoid the tokenization. With the standard tokenizer "ff134be8-10fc-4461-b620-79s51199c7qb" will create ["ff134be8", "10fc", "4461", "b620", "79s51199c7qb"] as tokens.
You have different options :
1/ use a term query to check without analyzing the content of the field
...
{
"term": {
"productId": "ff134be8-10fc-4461-b620-79s51199c7qb"
}
},
...
2/ if you are in Elasticsearch 6.X you could change your request to
...
{
"match": {
"productId.keyword": "ff134be8-10fc-4461-b620-79s51199c7qb"
}
},
...
As elasticsearch will create a subfield keyword with the type keyword for all string field
The best option is, of course, the first one. Always use term query if you are trying to match the exact content.