Develop Reference

Elasticsearch Multimatch substring not working

So I have a record with following field :
"fullName" : "Virat Kohli"
I have written the following multi_match query that should fetch this record :
GET _search
{
"query": {
"multi_match": {
"query": "*kohli*",
"fields": [
"fullName^1.0",
"team^1.0"
],
"type": "phrase_prefix",
"operator": "OR",
"slop": 0,
"prefix_length": 0,
"max_expansions": 50,
"zero_terms_query": "NONE",
"auto_generate_synonyms_phrase_query": true,
"fuzzy_transpositions": true,
"boost": 1
}
}
}
This works fine.
But when I remove the letter 'k' from query and change it to :
"query": "*ohli*"
It doesn't fetch any record.
Any reason why this is happening? How can I modify the query to get the record returned with the above modification?

first let me explain you why your existing query didn't work and then the solution of it.
Problem : you are using the multi_match query with type phrase_prefix and as explained in the documentation it makes a prefix query on the last search term and in your case you have only 1 search term so on that Elasticsearch will perform the phrase query.
And prefix query works on the exact tokens, and you are using standard analyzer mostly, default for text fields so for fullName field it will have virat and kohli and your search term also generates kohli(notice smallcase k) as standard analyzer also lowercase the tokens, above you can check with the explain API output in your first request as shown below.
"_explanation": {
"value": 0.2876821,
"description": "max of:",
"details": [
{
"value": 0.2876821,
"description": "weight(fullName:kohli in 0) [PerFieldSimilarity], result of:",
"details": [
{
(note he search term in the weight)
Solution
As you are trying to use the wildcard in your query, best solution is to use the wildcard query against your field as shown below to get results in both case.
{
"query": {
"wildcard": {
"fullName": {
"value": "*ohli",
"boost": 1.0,
"rewrite": "constant_score"
}
}
}
}
And SR
"hits": [
{
"_shard": "[match_query][0]",
"_node": "BKVyHFTiSCeq4zzD-ZqMbA",
"_index": "match_query",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"id": 2,
"fullName": "Virat Kohli",
"team": [
"Royal Challengers Bangalore",
"India"
]
},
"_explanation": {
"value": 1.0,
"description": "fullName:*ohli",
"details": []
}
}
]

MongoDb aggregation query using $addFields and $max in Java

I have a MongoDB aggregation that works in the mongo shell but returns an error when I attempt to run it from within Java application.
The aggregation query in question is as follows:
db.assets.aggregate([
{ $lookup: {
from: "vulnerabilities",
let: { id: "$_id", jdkId: "$jdkId" },
pipeline: [
{ $match:
{ $expr:
{ $and: [
{ $eq: [ "$resolved", false ] },
{ $or: [
{ $eq: [ "$assetId", "$$id" ] },
{ $eq: [ "$assetId", "$$jdkId" ] }
] }
] }
}
}
],
as: "unresolvedVulnerabilities"
} },
{ $addFields: {
maxCvssBaseScore: {
$max: {
$max: "$unresolvedVulnerabilities.cves.cvssBaseScore"
}
}
} },
{ $skip: 0 },
{ $limit: 100 }
] ).pretty()
When I attempt to run this query in a Java application, I get the following error:
{
"operationTime": {
"$timestamp": {
"t": 1645001907,
"i": 1
}
},
"ok": 0.0,
"errmsg": "Invalid $addFields :: caused by :: Unrecognized expression '$max '",
"code": 168,
"codeName": "InvalidPipelineOperator",
"$clusterTime": {
"clusterTime": {
"$timestamp": {
"t": 1645001907,
"i": 1
}
},
"signature": {
"hash": {
"$binary": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
"$type": "00"
},
"keyId": {
"$numberLong": "0"
}
}
}
}
The code for generating the addFields stage of the aggregation pipeline is as follows:
Bson addFieldsStage = new Document().append("$addFields",
new Document().append(PROPERTY_MAX_CVSS_BASE_SCORE,
new Document().append("$max",
new Document().append("$max ", "$" + PROPERTY_UNRESOLVED_VULNERABILITIES + "." + PROPERTY_CVES + "." + PROPERTY_CVSS_BASE_SCORE)
)
)
);
An example of the underlying data is as follows:
{
"id": "eb843e46-901a-3d0e-8d6e-1315c78cf5f7",
"name": "test-server1#test.a.b.c",
"type": "liberty",
"productName": "WebSphere Application Server Liberty Network Deployment",
"version": "20.0.0.9",
"features": [
"el-3.0",
"jsp-2.3",
"servlet-3.1",
"ssl-1.0",
"transportSecurity-1.0",
"usageMetering-1.0"
],
"apars": [],
"hostName": "test.a.b.c",
"serverName": "test-server1",
"installDirectory": "/opt/ibm/wlp/",
"profileDirectory": "/opt/ibm/wlp/usr/",
"operatingSystem": "Linux",
"operatingSystemVersion": "3.10.0-1160.53.1.el7.x86_64",
"jdkId": "579a03c6-76d0-3813-ad61-51428041987b",
"unresolvedVulnerabilities": [
{
"id": "156bb40b-826d-31e8-ab61-e0d3ee8b6446",
"name": "6520468 : IBM J9 VM#test.a.b.c",
"description": "There are multiple vulnerabilities in the IBM® SDK, Java™ Technology Edition that is shipped with IBM WebSphere Application Server. These might affect some configurations of IBM WebSphere Application Server Traditional, IBM WebSphere Application Server Liberty and IBM WebSphere Application Server Hypervisor Edition. These products have addressed the applicable CVEs. If you run your own Java code using the IBM Java Runtime delivered with this product, you should evaluate your code to determine whether the complete list of vulnerabilities is applicable to your code. For a complete list of vulnerabilities, refer to the link for \"IBM Java SDK Security Bulletin\" located in the References section for more information. HP fixes are on a delayed schedule.",
"assetId": "579a03c6-76d0-3813-ad61-51428041987b",
"securityBulletinId": "7e4684ae-8252-354e-be6d-4268af2d272e",
"resolved": false,
"cves": [
{
"id": "CVE-2021-35578",
"description": "An unspecified vulnerability in Java SE related to the JSSE component could allow an unauthenticated attacker to cause a denial of service resulting in a low availability impact using unknown attack vectors.",
"cvssBaseScore": 5.3
},
{
"id": "CVE-2021-35564",
"description": "An unspecified vulnerability in Java SE related to the Keytool component could allow an unauthenticated attacker to cause no confidentiality impact, low integrity impact, and no availability impact.",
"cvssBaseScore": 5.3
}
],
"remediations": [
{
"startVersion": "8.0.0.0",
"endVersion": "8.0.6.36",
"fixPack": "8.0.7.0"
}
],
"created": "2022-01-11T10:58:43Z",
"createdBy": "server-registration-processor",
"updated": "2022-01-11T10:58:43Z",
"updatedBy": "server-registration-processor",
"secondsExposed": 0
},
{
"id": "0f6006e6-a8ae-3cb6-bb7e-ba3afbf93996",
"name": "6489683 : test-server1#test.a.b.c",
"description": "There are multiple vulnerabilities in the Apache Commons Compress library that is used by WebSphere Application Server Liberty. This has been addressed.",
"assetId": "eb843e46-901a-3d0e-8d6e-1315c78cf5f7",
"securityBulletinId": "12de7238-ff4e-3252-a05f-19d51a3f8bf0",
"resolved": false,
"cves": [
{
"id": "CVE-2021-36090",
"description": "Apache Commons Compress is vulnerable to a denial of service, caused by an out-of-memory error when large amounts of memory are allocated. By reading a specially-crafted ZIP archive, a remote attacker could exploit this vulnerability to cause a denial of service condition against services that use Compress' zip package.",
"cvssBaseScore": 7.5
},
{
"id": "CVE-2021-35517",
"description": "Apache Commons Compress is vulnerable to a denial of service, caused by an out of memory error when allocating large amounts of memory. By persuading a victim to open a specially-crafted TAR archive, a remote attacker could exploit this vulnerability to cause a denial of service condition against services that use Compress' tar package.",
"cvssBaseScore": 5.5
}
],
"remediations": [
{
"startVersion": "20.0.0.1",
"endVersion": "20.0.0.12",
"operator": "OR",
"iFixes": [
"PH39418"
],
"fixPack": "21.0.0.10"
}
],
"created": "2022-01-11T10:58:43Z",
"createdBy": "vulnerability-manager",
"updated": "2022-01-11T10:58:43Z",
"updatedBy": "vulnerability-manager",
"secondsExposed": 0
}
],
"groups": [
"NO_GROUP"
],
"created": "2022-01-11T10:58:43Z",
"createdBy": "server-registration-processor",
"updated": "2022-01-25T15:49:16Z",
"updatedBy": "vulnerability-manager"
}
Is anyone able to tell me what I am doing wrong?

It turns out that it was a fat finger error... one of the $max strings included an additional space at the end.
The Java code for the working stage is:
Bson addFieldsStage = new Document().append("$addFields",
new Document().append(PROPERTY_MAX_CVSS_BASE_SCORE,
new Document().append("$max",
new Document().append("$max", "$" + PROPERTY_UNRESOLVED_VULNERABILITIES + "." + PROPERTY_CVES + "." + PROPERTY_CVSS_BASE_SCORE)
)
)
);
Many thanks to prasad_ for spotting this.

Spring boot aggregation with group Trim

Consider collection with whitespace in field, in DB if there is white space if we group them "Ravi", " Ravi " , consider as total 2 values. but it should consider as single value. So I have trim in Group. I had performed in DB. But I don't know how can I trim and group value in Springboot. We can use Aggregation for this in Springboot. but don't how to use trim. Kindly help on this
Sample Collection:
[
{
"name": "ravi",
"DOB": "04-02-2000",
"Blood": "A1+"
},
{
"name": "ravi ",
"DOB": "05-03-2000",
"Blood": "A1+"
},
{
"name": "kumar ",
"DOB": "02-04-2000",
"Blood": "A1+"
},
{
"name": "kumar",
"DOB": "03-05-2000",
"Blood": "A1+"
}
]
MongoDB Operation:
db.collection.aggregate({
"$group": {
_id: {
$trim: {
input: "$name"
}
},
doc: {
"$first": "$$ROOT",
}
}
},
{
"$replaceRoot": {
"newRoot": "$doc"
}
})
Output:
[
{
"Blood": "A1+",
"DOB": "04-02-2000",
"_id": ObjectId("5a934e000102030405000000"),
"name": "ravi"
},
{
"Blood": "A1+",
"DOB": "02-04-2000",
"_id": ObjectId("5a934e000102030405000002"),
"name": "kumar "
}
]

Not possible via standard API.
Workaround: We need to add extra $addFields to trim the name field before applying $group stage.
#Autowired
private MongoTemplate template;
...
Aggregation agg = Aggregation.newAggregation(
Aggregation.addFields()
.addFieldWithValue("name", Trim.valueOf("name")).build(),
Aggregation.group("name")
.first("$$ROOT").as("doc"),
Aggregation.replaceRoot("doc")
);
...
template.aggregate(agg, inputType, outputType);
Note: Since you were manipulating the name field in the first stage, MongoDB wouldn't use indexes, so we can add an extra stage to the pipeline.

How to pass String as a Property in Groovy. Multi-level JSON

I am really struggling to figure something out in a piece of code I'm working on to pass through a string to multi level JSON. I know that groovy will treat example ="multi.level "$example" as one single property and I have been trying to figure out how to split the string keep the delimiter and then add it to an if statement without any success. I have looked at other solutions but am unable to get them to work in my case... Any help would be appreciated.
JSON:
scriptOutput: [
{
"id": 1,
"data": {
"name": "fyre-discard-vm-1007"
},
"key": "ABCDE123"
},
{
"id": 2,
"data": {
"name": "fyre-discard-vm-1008"
},
"key": "ABCYRE123"
},
{
"id": 3,
"data": {
"name": "fyre-discard-vm-1009"
},
"key": "AZREDE123"
},
{
"id": 4,
"data": {
"name": "fyre-discard-vm-1010"
},
"key": "YTNER857"
},
{
"id": 5,
"data": {
"name": "fyre-discard-vm-1011"
},
"key": "YANT76563A"
},
{
"id": 6,
"data": {
"name": "fyre-discard-vm-1012"
},
"key": "DYAN31685"
},
{
"id": 7,
"data": {
"name": "fyre-discard-vm-1013"
},
"key": "SANF84923"
},
{
"id": 8,
"data": {
"name": "fyre-discard-vm-1014"
},
"key": "SADNS57985"
},
{
"id": 9,
"data": {
"name": "fyre-discard-vm-1015"
},
"key": "DYUIK89OP"
},
{
"id": 10,
"data": {
"name": "fyre-discard-vm-1016"
},
"key": "DHJMNNB6547"
}
]
CODE
#!/usr/local/bin/groovy
import groovy.json.*
def proc7 = "curl https://my-json-server.typicode.com/owainow/privateAPI/machines".execute().text
def slaveList = ["fyre-discard-vm-1007","fyre-discard-vm-1015","fyre-discard-vm-1016"]
def deleteMap=[:]
def jsonName = "data.name"
println "scriptOutput: ${proc7}"
def json = new groovy.json.JsonSlurper().parseText(proc7)
for (slave in slaveList)
{
println(slave);
if (json.find{it.data.name==slave}){ //This works as the properties are passed through as properties
id = json.find{it.data.name==slave}.key
deleteMap = deleteMap+[(slave):(id)]
// I would like to do something like this however I cannot figure out how to split my properties and pass them through in a way that works
/* if (json.find{it."$jsonName"==slave}){ SOMEHOW expand or split jsonName here
id = json.find{it."$jsonName"==slave}.key
deleteMap = deleteMap+[(slave):(id)] */
println "Match"
println (id)
println deleteMap;
}
else{
println"No"
}
}
Cheers guys.

why do you need this? why jsonName should be a string?
Asking this because to get deleteMap from json is quite simple groovy expression:
def deleteMap= json.findAll{ it.data.name in slaveList }.collectEntries{ [it.id, it.data.name] }
however, if you still need it - groovy supports dynamic expression evaluation:
def jsonName = "ROOT.data.name"
def deleteMap= json.findAll{ Eval.me('ROOT', it, jsonName) in slaveList }.collectEntries{ [it.id, Eval.me('ROOT', it, jsonName)] }

I was going to suggest to use the getAt() operator
it[jsonName]
but it turns out that you have a bug in your code, apparently:
id = json.find{it."${jsonName}"==slave}.key
should be
id = json.find{it.data."${jsonName}"==slave}.key
I would still probably use
id = json.find{it.data[jsonName]==slave}.key
Here's complete, corrected snippet:
import groovy.json.*
def proc7 = "curl https://my-json-server.typicode.com/owainow/privateAPI/machines".execute().text
def slaveList = ["fyre-discard-vm-1007","fyre-discard-vm-1015","fyre-discard-vm-1016"]
def deleteMap=[:]
println "scriptOutput: ${proc7}"
def json = new groovy.json.JsonSlurper().parseText(proc7)
for (slave in slaveList)
{
println(slave);
if (json.find{it.data.name==slave}){ //This works as the properties are passed through as properties
id = json.find{it.data.name==slave}.key
deleteMap = deleteMap+[(slave):(id)]
// I would like to do something like this however I cannot figure out how to split my properties and pass them through in a way that works
/* if (json.find{it."$jsonName"==slave}){ SOMEHOW expand or split jsonName here
id = json.find{it."$jsonName"==slave}.key
deleteMap = deleteMap+[(slave):(id)] */
def jsonName="name"
id = json.find{it.data."${jsonName}"==slave}?.key
if (id) {
deleteMap = deleteMap+[(slave):(id)]
println "Match"
println (id)
println deleteMap;
}
}
else{
println"No"
}
}

Get value from deep inside a JSONObject

I have this Bing Maps JSON file and I want to retrieve "++formattedAddress++" from inside it
{
"statusCode": 200,
"statusDescription": "OK",
"copyright": "Copyright © 2013 Microsoft and its suppliers. All rights reserved. This API cannot be accessed and the content and any results may not be used, reproduced or transmitted in any manner without express written permission from Microsoft Corporation.",
"authenticationResultCode": "ValidCredentials",
"resourceSets": [
{
"resources": [
{
"__type": "Location:http://schemas.microsoft.com/search/local/ws/rest/v1",
"point": {
"type": "Point",
"coordinates": [
63.8185213804245,
12.105498909950256
]
},
"matchCodes": [
"Good"
],
"address": {
"addressLine": "55 Stuff",
"locality": "Stuff",
"++formattedAddress++": "55 Stuff, 51512 Stuff",
"postalCode": "25521",
"adminDistrict2": "Stuff-Stuff",
"countryRegion": "UK",
"adminDistrict": "NL"
},
"bbox": [
84.81465866285382,
12.097347537264563,
50.822384097995176,
7.11365028263595
],
"name": "55 Stuff, 51122 Stuff",
"confidence": "Medium",
"entityType": "Address",
"geocodePoints": [
{
"calculationMethod": "Interpolation",
"type": "Point",
"usageTypes": [
"Display",
"Route"
],
"coordinates": [
50.8185213804245,
7.105498909950256
]
}
]
}
],
"estimatedTotal": 1
}
],
"traceId": "8a13f73cab93472db1253e4c1621c651|BL2M002306|02.00.83.1900|BL2MSNVM001274, BL2MSNVM003152",
"brandLogoUri": "http://dev.virtualearth.net/Branding/logo_powered_by.png"
}
What I have tried so far is like this:
final JSONArray jsonMainArr = locationData.getJSONArray("resourceSets").getJSONObject(0).getJSONArray("resources");
final JSONObject childJSONObject = jsonMainArr.getJSONObject(0);
return childJSONObject.getString("formattedAddress");
childJSONObject is still 2-3 levels over formattedAddress and the query is becoming highly inefficient

get formattedAddress address value as from current json String :
final JSONObject childJSONObject = jsonMainArr.getJSONObject(0)
.getJSONObject("address");
return childJSONObject.getString("++formattedAddress++");

There are so much online sites where you paste your complex code and get it in an easy way. e.g. http://json.parser.online.fr/