I am using Elastic LowLevelRestClient to interact with my elastic instance, when I query the elastic using my search query it returns the response which is wrapped as an HttpEntity.
As per documentation of Elastic Reading Responses EntityUtils class of Apache provides a way to convert this HttpEntity into String which gives me below response. I just want to map this response to an appropriate Object.
My Code Snippet:
Request request = new Request("GET", "/neeraj_party/_search");
request.setJsonEntity(searchQuery);
Response response = lowLevelClient.performRequest(request);
String responseBody = EntityUtils.toString(response.getEntity());
ResponseBody looks like this
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 32.986195,
"hits": [
{
"_index": "neeraj_party",
"_type": "party",
"_id": "28588489",
"_score": 32.986195,
"_source": {
"name": "MUST HK LTD",
"city_nm": "郑州",
"#timestamp": "2019-03-23T18:28:07.305Z",
"type": "json",
"legal_nm": "MUST HK Ltd",
"gr_id": "28588489",
"path": "/ssd/sdds",
"address": "郑州",
"state_province_cd": "180",
"country_iso2_cd": "CN",
"host": "neeraj.com",
"postal_cd": "450000",
"#version": "1"
}
}
]
}
}
My Question is simple
Do ElasticSearch provide any such bean which can represent this response, or Should I create my own CustomBean.
You could use SearchResponse Object to achieve this.
If you use the search(SearchRequest) method, It gives you back a SearchResponse object (including aggs).
Or you also could make the SearchResponse from that String using this method.
public static SearchResponse getSearchResponseFromJson(String jsonResponse){
try {
NamedXContentRegistry registry = new
NamedXContentRegistry(DashboardCuke.getDefaultNamedXContents());
XContentParser parser =
JsonXContent.jsonXContent.createParser(registry, jsonResponse);
return SearchResponse.fromXContent(parser);
}catch (IOException e) {
System.out.println("exception " + e);
}catch (Exception e){
System.out.println("exception " + e);
}
return new SearchResponse();
}
I got this information from here: ElasticSearch Forum
Related
I'm trying to create a Hit<ObjectNode> from a json string returned by Elastic (for unit-testing purposes). When I try to deserialize this using .withJson I get a ClassCastException with the following message: "class co.elastic.clients.json.JsonDataImpl cannot be cast to class com.fasterxml.jackson.databind.node.ObjectNode (co.elastic.clients.json.JsonDataImpl and com.fasterxml.jackson.databind.node.ObjectNode are in unnamed module of loader 'app')"
Full code:
String hitJson = "{\"_index\":\"my-index\",\"_id\":\"ID:MYID\",\"_score\":1.0,\"_source\":{\"productCode\":\"MYID\",\"productName\":\"my product\",\"EffDate\":\"01/01/1900\",\"ExpDate\":\"12/31/9999\",\"Status\":\"Active\",\"productId\":1234567,\"_type\":\"my type\",\"type\":\"my type\",\"updateTime\":\"2021-07-08T11:40:18Z\"}}";
InputStream hitInput = new ByteArrayInputStream(hitJson.getBytes());
Hit<ObjectNode> aHit = Hit.of(builder -> builder.withJson(hitInput));
try {
ObjectNode source = aHit.source();
} catch (Exception ex) {
String msg = ex.getMessage();
}
I can see that JsonDataImpl does have all the properties, but I'm not sure why it can't deserialize it to ObjectNode. I'm not sure where to go from here - all my other attemps to use .withJson have succeeded without issue.
Formatted json response for reference:
{
"_index": "my-index",
"_id": "ID:MYID",
"_score": 1.0,
"_source": {
"productCode": "MYID",
"productName": "my product",
"EffDate": "01/01/1900",
"ExpDate": "12/31/9999",
"Status": "Active",
"productId": 1234567,
"_type": "my type",
"type": "my type",
"updateTime": "2021-07-08T11:40:18Z"
}
}
I'm running a JUnit test, and I'm having some issues with a function where I have to get specific data from a query response. Here's a function I'm testing, note .get("data") at the end:
protected JsonNode getFunctionIds(FilterInputModel paramA) {
String paramB = "here's a query";
String idsQuery = setFunctionFieldsForQuery(paramA, paramB);
...
return Objects.requireNonNull(webClient.post()
.uri("/query")
.header(HttpHeaders.AUTHORIZATION, getSessionId())
.body(BodyInserters.fromMultipartData(queryMap))
.retrieve()
.bodyToMono(JsonNode.class)
.block())
.get("data");
}
Note that the inputModel is a class containing strings and string lists.
Here's a test I wrote. I'm getting an AssertionFailedError with it, and I vaguely know why, but I don't know how to get the right data.
#Test
void getFunctionIds() throws IOException {
Path filePath = Path.of("documents/__files/getFunctionIds.json");
String body = Files.read(filePath.toFile(), Charset.defaultCharset());
wireMockServer.stubFor(post(urlEqualTo("/api/query"))
.willReturn(aResponse()
.withStatus(200)
.withHeader("Content-Type", "application/json")
.withBody(body))
);
JsonNode jsonNode = new ObjectMapper().readValue(body, JsonNode.class);
String bodyData = new ObjectMapper().writeValueAsString(jsonNode);
JsonNode vaultData = serviceTest.getFunctionIds(inputModel);
String response = new ObjectMapper().writeValueAsString(vaultData);
assertEquals(bodyData, response);
}
Next is a query response (paramB) I got when sending it to the vault. It is stored as a String bodyData.
{
"someOtherData": ...
{
...
},
"data": [
{
"id": "V4600000002G003"
},
{
"id": "V4600000002H214"
},
{
"id": "V4600000002I001"
},
{
"id": "V4600000002J001"
},
{
"id": "V4600000002J062"
},
{
"id": "V4600000002K047"
},
{
"id": "V4600000002K071"
},
{
"id": "V4600000002K171"
}
]
}
And as I said, all I got is an AssertionFailedError, because I'm asserting the whole query response (bodyData) with just the "data" part of the JSON, which is stored in String response.
Here's the error:
Expected :{
"someOtherData"... + "data", actually the whole query response (paramB) i mentioned already.
Actual:
[
{
"id": "V4600000002G003"
},
{
"id": "V4600000002H214"
},
{
"id": "V4600000002I001"
},
{
"id": "V4600000002J001"
},
{
"id": "V4600000002J062"
},
{
"id": "V4600000002K047"
},
{
"id": "V4600000002K071"
},
{
"id": "V4600000002K171"
}
]
Is there a way to catch just the Actual part of the query response, to avoid the AssertionFailedError? That would solve my problem, obviously.
I'm Using RestAssured,
and i'm getting the response like this-
{
"Data": {
"Sub": {
"SubDetails": [
{
"OrgId": 5,
"SubId": 1,
"SubName": "Mathematics"
}
]
}
},
"RawData": {
"Url": "http://localhost:11111/cases/case-15",
"Type": "Rest",
"Request": {
"Details": {
"OrganizationId": 5,
"Student": {
"Age": 30,
"Religion": "Hindu",
"StudentId": 10
}
}
},
"Response": {
"SmartReturnObject": {
"Subject": [
{
"SubjectId": 1,
"SubjectName": "Mathematics"
}
],
"OrganizationId": 5
}
},
"IsApiError": false
},
"SessionId": "5q0",
"RequestUniqueId": "4543534",
"StatusCode": "4540000",
"StatusMessage": "Success",
"DataSource": "DD"
}
Now i need to extract the Data object separate and RawData separate but i'm failed.
I tried with JSONPath also but not getting it in JSON format.
I tried
JsonPath body= response.jsonPath();
Object value=body.get("Data");
System.out.println("Value is-"+value);
and getting it like this - {Sub={SubDetails=[{OrgId=5, SubId=1, SubName=Mathematics}]}}
I want the response as instanceof JSONObject or JSONArray type.
pom.xml
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
Try below -
Gson gson = new Gson();
try {
JSONObject obj = new JSONObject(gson.toJson(givenString));
}catch (Exception e){
System.out.println(e.toString());
}
Am fetching documents from elastic search using java api, i have the following code in my elastic search documents and am trying to search it with the following pattern.
code : MS-VMA1615-0D
Input : MS-VMA1615-0D -- Am getting the results (MS-VMA1615-0D).
Input : VMA1615 -- Am getting the results (MS-VMA1615-0D) .
Input : VMA -- Am getting the results (MS-VMA1615-0D) .
But, if i give input like below, am not getting results.
Input : V -- Am not getting the results.
INPUT : MS -- Am not getting the results.
INPUT : -V -- Am not getting the results.
INPUT : 615 -- Am not getting the results.
Am expecting to return the code MS-VMA1615-0D. In simple, am trying to search character by character instead of term (word).
It should not return the code MS-VMA1615-0D for the following cases, Because its not matching with my code.
Input : VK -- should not return the results.
INPUT : MS3 -- should not return the results.
Please find my below java code that am using
private final String INDEX = "products";
private final String TYPE = "doc";
SearchRequest searchRequest = new SearchRequest(INDEX);
searchRequest.types(TYPE);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
QueryStringQueryBuilder qsQueryBuilder = new QueryStringQueryBuilder(code);
qsQueryBuilder.defaultField("code");
searchSourceBuilder.query(qsQueryBuilder);
searchSourceBuilder.size(50);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = null;
try {
searchResponse = SearchEngineClient.getInstance().search(searchRequest);
} catch (IOException e) {
e.getLocalizedMessage();
}
Item item = null;
SearchHit[] searchHits = searchResponse.getHits().getHits();
Please find my mapping details :
PUT products
{
"settings": {
"analysis": {
"analyzer": {
"custom_analyzer": {
"type": "custom",
"tokenizer": "my_pattern_tokenizer",
"char_filter": [
"html_strip"
],
"filter": [
"lowercase",
"asciifolding"
]
}
},
"tokenizer": {
"my_pattern_tokenizer": {
"type": "pattern",
"pattern": "-|\\d"
}
}
}
},
"mappings": {
"doc": {
"properties": {
"code": {
"type": "text",
"analyzer": "custom_analyzer"
}
}
}
}
}
After Update with new Answer :
This is my request via Java API
'SearchRequest{searchType=QUERY_THEN_FETCH, indices=[products], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[doc], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, source={"size":50,"query":{"match_phrase":{"code":{"query":"1615","slop":0,"boost":1.0}}}}}
' . But am getting response as null
Follow up: ElasticSearch - JavaApi searching not happening without (*) in my input query
Your mapping should look like:
PUT products
{
"settings": {
"analysis": {
"analyzer": {
"custom_analyzer": {
"type": "custom",
"tokenizer": "ngram",
"char_filter": [
"html_strip"
],
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"doc": {
"properties": {
"code": {
"type": "text",
"analyzer": "custom_analyzer"
}
}
}
}
}
And you should be using a match_phrase query.
In Kibana:
GET products/_search
{
"query": {
"match_phrase": {
"code": "V"
}
}
}
will return the result:
"hits": [
{
"_index": "products",
"_type": "doc",
"_id": "EoGtdGQBqdof7JidJkM_",
"_score": 0.2876821,
"_source": {
"code": "MS-VMA1615-0D"
}
}
]
But this:
GET products/_search
{
"query": {
"match_phrase": {
"code": "VK"
}
}
}
wont:
{
"took": 10,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
Based on your comment:
Instead of using a Query string:
QueryStringQueryBuilder qsQueryBuilder = new QueryStringQueryBuilder(code);
qsQueryBuilder.defaultField("code");
searchSourceBuilder.query(qsQueryBuilder);
searchSourceBuilder.size(50);
searchRequest.source(searchSourceBuilder);
Use a match phrase query:
QueryBuilder query = QueryBuilders.matchPhraseQuery("code", code);
searchSourceBuilder.query(query);
searchSourceBuilder.size(50);
searchRequest.source(searchSourceBuilder);
Am fetching the indexed documents from elastic search using Java API. But am getting Null as a response from elastic search when Index having more number of document like (2k+).
If index doesnt have more documents less than 500 something, the below Java API code is working properly.
More number of documents in Index, creating issue. ( Is that something like performance issue while fetching ?)
I used ingest-attachment processor plugin for attachment, i attached PDF in my documents.
But if i search with the same query using kibana with curl script am getting response, and am able to see the results in Kibana
Please find my java code below
private final static String ATTACHMENT = "document_attachment";
private final static String TYPE = "doc";
public static void main(String args[])
{
RestHighLevelClient restHighLevelClient = null;
try {
restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http"),
new HttpHost("localhost", 9201, "http")));
} catch (Exception e) {
System.out.println(e.getMessage());
}
SearchRequest contentSearchRequest = new SearchRequest(ATTACHMENT);
SearchSourceBuilder contentSearchSourceBuilder = new SearchSourceBuilder();
contentSearchRequest.types(TYPE);
QueryStringQueryBuilder attachmentQB = new QueryStringQueryBuilder("Activa");
attachmentQB.defaultField("attachment.content");
contentSearchSourceBuilder.query(attachmentQB);
contentSearchSourceBuilder.size(50);
contentSearchRequest.source(contentSearchSourceBuilder);
SearchResponse contentSearchResponse = null;
try {
contentSearchResponse = restHighLevelClient.search(contentSearchRequest); // returning null response
} catch (IOException e) {
e.getLocalizedMessage();
}
System.out.println("Request --->"+contentSearchRequest.toString());
System.out.println("Response --->"+contentSearchResponse.toString());
SearchHit[] contentSearchHits = contentSearchResponse.getHits().getHits();
long contenttotalHits=contentSearchResponse.getHits().totalHits;
System.out.println("condition Total Hits --->"+contenttotalHits);
Please find my script that am using in kibana., am getting response for the below script.
GET document_attachment/_search?pretty
{
"query" :{
"match": {"attachment.content": "Activa"}
}
}
Please find the below search request from Java API
SearchRequest{searchType=QUERY_THEN_FETCH, indices=[document_attachment], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[doc], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, source={"size":50,"query":{"match":{"attachment.content":{"query":"Activa","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}}}
Please find my mapping details
{
"document_attachment": {
"mappings": {
"doc": {
"properties": {
"app_language": {
"type": "text"
},
"attachment": {
"properties": {
"author": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"content": {
"type": "text",
"analyzer": "custom_analyzer"
},
"content_length": {
"type": "long"
},
"content_type": {
"type": "text"
},
"date": {
"type": "date"
},
"language": {
"type": "text"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"catalog_description": {
"type": "text"
},
"fileContent": {
"type": "text"
}
}
}
}
}
}
}
Please find my settings details
PUT _ingest/pipeline/document_attachment
{
"description" : "Extract attachment information",
"processors" : [
{
"attachment" : {
"field" : "fileContent"
}
}
]
}
Am getting this error only when am trying to search based on attachment.content , If i search with some other field am able to get results.
Am using ElasticSearch 6.2.3 version
Please find the error below.
org.apache.http.ContentTooLongException: entity content is too long [105539255] for the configured buffer limit [104857600]
at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76)
at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:131)
at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:147)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:303)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.NullPointerException
at com.es.utility.DocumentSearch.main(DocumentSearch.java:88)