I am currently working on elastic search through my java Application . I know how to index the Java pojo using RestHighLevelClient. How i can make search only on new fields not the complete pojo.?
public class Employee{
private long id;
private String name;
private String designation;
private String address; //want to index but not searchable in elastic search
}
My Code for indexing is below which is working fine:
public String saveToEs(Employee employee) throws IOException {
Map<String, Object> map = objectMapper.convertValue(employee, Map.class);
IndexRequest indexRequest =
new IndexRequest(INDEX, TYPE, employee.getId().toString()).source(map, XContentType.JSON);
IndexResponse indexResponse = client.index(indexRequest, RequestOptions.DEFAULT);
I need to do this in java .Any help please or good link ?
Writing another answer for RestHighLevelClient As another answer is useful for people not using the Rest client and adding this in the first answer makes it too long.
Note: you are passing the type which is deprecated in ES 7.X and I am using the ES 7.X version, so my code is according to 7.X.
CreateIndexRequest request = new CreateIndexRequest("employee");
Map<String, Object> name = new HashMap<>();
name.put("type", "text");
Map<String, Object> address = new HashMap<>();
address.put("type", "text");
address.put("index", false);
Map<String, Object> properties = new HashMap<>();
properties.put("name", name);
properties.put("address", address);
Map<String, Object> mapping = new HashMap<>();
mapping.put("properties", properties);
request.mapping(mapping);
CreateIndexResponse createIndexResponse = client.indices().create(request, RequestOptions.DEFAULT);
Important points
I've used only 2 fields for illustration purpose, one of which is address field which is not searchable, and to do that I used, address.put("index", false); , while name is searchable field and there this option isn't present.
I've created index mapping using the Map method which is available in this official ES doc.
you can check the mapping created by this code, using mapping REST API.
Below is the mapping generated for this code in my system and you can see, index: false is added in the address field.
{
"employee": {
"mappings": {
"properties": {
"address": {
"type": "text",
"index": false
},
"name": {
"type": "text"
}
}
}
}
}
You can just use the same search JSON mentioned in the previous answer, to test that it's not searchable.
Use the index option as false on the address field, which is by default true to make it unsearchable. As mention in the same official ES link:
The index option controls whether field values are indexed. It accepts
true or false and defaults to true. Fields that are not indexed are
not queryable.
Let me show you how can you test it using the REST API and then the java code(using rest-high level client) to accomplish it.
Mapping
{
"mappings": {
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "text"
},
"designation": {
"type": "text"
},
"address": {
"type": "text",
"index" : false --> made `index` to false
}
}
}
}
Index few docs
{
"address" : "USA",
"name" : "Noshaf",
"id" : 234567892,
"designation" : "software engineer"
}
{
"address" : "USA california state",
"name" : "opster",
"id" : 234567890,
"designation" : "software engineer"
}
A simple match search query in JSON format on address field
{
"query": {
"match" : {
"address" : "USA"
}
}
}
Exception from Elasticsearch clearly mention, it's not searchable
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Cannot search on field [address] since it is not indexed."
}
Related
I currently receive the following JSON body
{
"productId": "90000011",
"offerId": "String",
"format": "String",
"sellerId": "String",
"sellerName": "String",
"shippingPrice[zone=BE,method=STD]": 0.0,
"deliveryTimeEarliestDays[zone=BE,method=STD]": 1,
"deliveryTimeLatestDays[zone=BE,method=STD]": 1,
"shippingPrice[zone=NL,method=STD]": 0.0,
"deliveryTimeEarliestDays[zone=NL,method=STD]": 1,
"deliveryTimeLatestDays[zone=NL,method=STD]": 1
}
As you can see, I have similar properties that differ by zone and method enclosed in square brackets. I don't want to change the code every time a new zone and/or method is introduced. I'm looking for a more dynamic way you deserialize this via Jackson.
Is there a way to automatically deserialize all properties starting with shippingPrice, deliveryTimeEarliestDays and deliveryTimeLatestDays into the following format?
{
"productId": "90000011",
"offerId": "String",
"format": "String",
"sellerId": "String",
"sellerName": "String",
"deliveryModes":[
{
"method":"STD"
"zone":"BE",
"shippingPrice":0.0,
"deliveryTimeEarliestDays":1,
"deliveryTimeLatestDays":1
},{
"method":"STD"
"zone":"NL",
"shippingPrice":0.0,
"deliveryTimeEarliestDays":1,
"deliveryTimeLatestDays":1
}]
}
My first idea was to use the #JsonAnySetter annotation and put everything in a Map but that still leaves me with manual parsing of the field name.
My Second Idea was to build a custom deserializer where I loop over all attributes and filter out all the ones that start with shippingPrice, deliveryTimeEarliestDays and deliveryTimeLatestDays and map them to the described format above.
In order to achieve the required result, you need to implement deserialization logic yourself, it can't be done only by sprinkling a couple of data binding annotations.
That's how it can be done.
Assume here's a POJO that corresponds to your input JSON (to avoid boilerplate code, I'll use Lombok annotations):
#Getter
#Setter
public static class MyPojo {
private String productId;
private String offerId;
private String format;
private String sellerId;
private String sellerName;
#JsonIgnore // we don't want to expose this field to Jackson as is
private Map<DeliveryZoneMethod, DeliveryMode> deliveryModes = new HashMap<>();
#JsonAnySetter
public void setDeliveryModes(String property, String value) {
DeliveryZoneMethod zoneMethod = DeliveryZoneMethod.parse(property);
DeliveryMode mode = deliveryModes.computeIfAbsent(zoneMethod, DeliveryMode::new);
String name = property.substring(0, property.indexOf('['));
switch (name) {
case "shippingPrice" -> mode.setShippingPrice(new BigDecimal(value));
case "deliveryTimeEarliestDays" -> mode.setDeliveryTimeEarliestDays(Integer.parseInt(value));
case "deliveryTimeLatestDays" -> mode.setDeliveryTimeLatestDays(Integer.parseInt(value));
}
}
public Collection<DeliveryMode> getModes() {
return deliveryModes.values();
}
}
Properties productId, offerId, format, sellerId, sellerName would be parsed by Jackson in a regular way.
And all other properties formatted like "shippingPrice[zone=BE,method=STD]" would be handled by the method annotated with #JsonAnySetter.
To facilitate extracting and storing information from such properties I've defined a couple of auxiliary classes:
DeliveryZoneMethod which contains information about a zone and delivery method as its name suggests (the purpose of this class is to serve as Key in the map deliveryModes).
DeliveryMode which is meant to contain all the need information that correspond to a particular zone and method of delivery.
For conciseness, DeliveryZoneMethod can be implemented as a Java 16 record:
public record DeliveryZoneMethod(String method, String zone) {
public static Pattern ZONE_METHOD = Pattern.compile(".+zone=(\\p{Alpha}+).*method=(\\p{Alpha}+)");
public static DeliveryZoneMethod parse(String str) {
// "shippingPrice[zone=BE,method=STD]" - assuming the given string has always the same format
Matcher m = ZONE_METHOD.matcher(str);
if (!m.find()) throw new IllegalArgumentException("Unable to parse: " + str);
return new DeliveryZoneMethod(m.group(1), m.group(2));
}
}
And here's how DeliveryMode might look like:
#Getter
#Setter
public static class DeliveryMode {
private String method;
private String zone;
private BigDecimal shippingPrice;
private int deliveryTimeEarliestDays;
private int deliveryTimeLatestDays;
public DeliveryMode(DeliveryZoneMethod zoneMethod) {
this.method = zoneMethod.method();
this.zone = zoneMethod.zone();
}
}
Usage example:
public static void main(String[] args) throws JsonProcessingException {
String json = """
{
"productId": "90000011",
"offerId": "String",
"format": "String",
"sellerId": "String",
"sellerName": "String",
"shippingPrice[zone=BE,method=STD]": 0.0,
"deliveryTimeEarliestDays[zone=BE,method=STD]": 1,
"deliveryTimeLatestDays[zone=BE,method=STD]": 1,
"shippingPrice[zone=NL,method=STD]": 0.0,
"deliveryTimeEarliestDays[zone=NL,method=STD]": 1,
"deliveryTimeLatestDays[zone=NL,method=STD]": 1
}
""";
ObjectMapper mapper = new ObjectMapper();
MyPojo myPojo = mapper.readValue(json, MyPojo.class);
String serializedJson = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(myPojo);
System.out.println(serializedJson);
}
Output:
{
"productId" : "90000011",
"offerId" : "String",
"format" : "String",
"sellerId" : "String",
"sellerName" : "String",
"modes" : [ {
"method" : "BE",
"zone" : "STD",
"shippingPrice" : 0.0,
"deliveryTimeEarliestDays" : 1,
"deliveryTimeLatestDays" : 1
}, {
"method" : "NL",
"zone" : "STD",
"shippingPrice" : 0.0,
"deliveryTimeEarliestDays" : 1,
"deliveryTimeLatestDays" : 1
} ]
}
I would go with your first idea to deserialize your JSON into a map. And yes you will still need to analyze the map keys. It is easy to deserialize Json into a map with Json Jackson, but there is an Open source library called MgntUtils that provides class JsonUtils which is a thin wrapper over Json-Jackson library. using it you can very simply deserialize your Json into a Map (or any other class). Your code would look like this:
try {
Map<String, Object> map = JsonUtils.readObjectFromJsonString(jsonStr, Map.class);
System.out.println(map);
} catch (IOException e) {
...
}
Here is Javadoc for JsonUtils. The library can be obtained as maven artifact or on Github (with source code and Javadoc).
Disclaimer: This library is written and maintained by me
I need to send following JSON in API BODY POST request:
{
"name": "",
"type": "TEMP",
"shared": false,
"search": {
},
"order": [
]
}
In my MainBody.java, declared
private String name;
private String type;
private boolean shared;
private JSON search;
private Object order;
and defined getters and setters.
In Payload.java,
MainBody mb = new MainBody();
mb.setName("");
mb.setType("TEMP");
mb.setShared(false);
mb.setSearch(null);
mb.setOrder(new ArrayList<>());
ObjectMapper om = new ObjectMapper();
String myData = om.writerWithDefaultPrettyPrinter().writeValueAsString(mb);
System.out.println(myData);
results
{
"name" : "",
"type" : "TEMP",
"shared" : false,
"search" : null,
"order" : [ ]
}
Please assist with how search as { } can be achieved as per expected JSON instead of null.
TIA.
Instead of setting search to null, you need to set it to an empty object. I'm not sure which JSON library you are using, but there should be an object constructor like new JsonObject(). Depending on what the allowed values for search are, you may also want to consider representing it in your class as Map<String, String> or something like that.
I would try something like this:
mb.setSearch(new JSON());
This way you create empty object and there should be only {}. It also depends on which JSON library do you use.
Issue resolved after using JSONMapper.
I have a document given below.
{
"my_id": "123",
"content": {
"name": "abc",
"designation": "engineer"
}
}
I have written Java code for elasticsearch to access the field name which is given below.
String field = "content.name";
String value = "abc"
SearchResponse response = esClient.prepareSearch("indexName")
.setTypes("data")
.setQuery(QueryBuilders.matchQuery(field, value))
.get();
But the output that I am getting for this multilevel object search empty hits. Is there a way to access multilevel objects in Java
The given query works from sense.
GET indexName/_search
{
"query" : {
"match" : {
"content.name" : "abc"
}
}
}
I'm trying to disable dynamic mapping creation for only specific indexes, not for all. For some reason I can't put default mapping with 'dynamic' : 'false'.
So, here left two options as I can see:
specify property 'index.mapper.dynamic' in file elasticsearch.yml.
put 'index.mapper.dynamic' at index creation time, as described here https://www.elastic.co/guide/en/kibana/current/setup.html#kibana-dynamic-mapping
First option may only accept values: true, false and strict. So there is no way to specify subset of specific indexes (like we do by pattern with property 'action.auto_create_index' https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation).
Second option just not works.
I've created index
POST http://localhost:9200/test_idx/
{
"settings" : {
"mapper" : {
"dynamic" : false
}
},
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
}
}
}
}
}
Then checked index settings:
GET http://localhost:9200/test_idx/_settings
{
"test_idx" : {
"settings" : {
"index" : {
"mapper" : {
"dynamic" : "false"
},
"creation_date" : "1445440252221",
"number_of_shards" : "1",
"number_of_replicas" : "0",
"version" : {
"created" : "1050299"
},
"uuid" : "5QSYSYoORNqCXtdYn51XfA"
}
}
}
}
and mapping:
GET http://localhost:9200/test_idx/_mapping
{
"test_idx" : {
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
}
}
}
}
}
}
so far so good, let's index document with undeclared field:
POST http://localhost:9200/test_idx/test_type/1
{
"field1" : "it's ok, field must be in mapping and in source",
"somefield" : "but this field must be in source only, not in mapping"
}
Then I've checked mapping again:
GET http://localhost:9200/test_idx/_mapping
{
"test_idx" : {
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
},
"somefield" : {
"type" : "string"
}
}
}
}
}
}
As you can see, mapping is extended regardless of index setting "dynamic" : false.
I've also tried to create index exactly as described in doc
PUT http://localhost:9200/test_idx
{
"index.mapper.dynamic": false
}
but got the same behavior.
Maybe I've missed something?
Thanks a lot in advance!
You're almost there: the value needs to be set to strict.
And the correct usage is the following:
PUT /test_idx
{
"mappings": {
"test_type": {
"dynamic":"strict",
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
And pushing this a bit further, if you want to forbid the creation even of new types, not only fields in that index, use this:
PUT /test_idx
{
"mappings": {
"_default_": {
"dynamic": "strict"
},
"test_type": {
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
Without _default_ template:
PUT /test_idx
{
"settings": {
"index.mapper.dynamic": false
},
"mappings": {
"test_type": {
"dynamic": "strict",
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
You must know about that the below part just mean that ES could'nt create a type dynamically.
"mapper" : {
"dynamic" : false
}
You should configure ES like this:
PUT http://localhost:9200/test_idx/_mapping/test_type
{
"dynamic":"strict"
}
Then you cant't index other field that without mapping any more ,and get an error as follow:
mapping set to strict, dynamic introduction of [hatae] within [data] is not allowed
If you wanna store the data,but make the field can't be index,you could take the setting like this:
PUT http://localhost:9200/test_idx/_mapping/test_type
{
"dynamic":false
}
Hope these can help the people with the same issue :).
The answer is in the doc (7x.): https://www.elastic.co/guide/en/elasticsearch/reference/7.x/dynamic.html
The dynamic setting controls whether new fields can be added
dynamically or not. It accepts three settings:
true
Newly detected fields are added to the mapping. (default)
false
Newly detected fields are ignored. These fields will not be indexed so
will not be searchable but will still appear in the _source field of
returned hits. These fields will not be added to the mapping, new
fields must be added explicitly.
strict
If new fields are detected, an exception is thrown and the document is
rejected. New fields must be explicitly added to the mapping.
PUT my_index
{
"mappings": {
"dynamic": "strict",
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
You cannot disable dynamic mapping in ES 7 anymore, what you can do if you have completely unstructured data is to disable completely the mapping for the index like this:
curl -X PUT "localhost:9200/my_index?pretty" -H 'Content-Type: application/json' -d'
{
"mappings": {
"enabled": false
}
}
'
if you are using python you can do this:
from elasticsearch import Elasticsearch
# Connect to the elastic cluster
es=Elasticsearch([{'host':'localhost','port':9200}])
request_body = {
"mappings": {
"enabled": False
}
}
es.indices.create(index = 'my_index', body = request_body)
For ES 7 if you want to update an existing index:
PUT customers/_mapping
{
"dynamic": "strict"
}
first, please be concern aboout value false or strict,they work in a different way.
using "dynamic": "false" and create documents with fields not covered by the mapping, those fields will be ignored (so they won't be stored) and wouldn't show up in _source when you GET the document.
where value strict will not allow you to create the document rather it will throw an exception
Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
PUT my-index-000001
{
"mappings": {
"dynamic": false,
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
if you are using node.js client
await this.client.indices.putMapping({
index: ElasticIndex.UserDataFactory,
body: {
dynamic: 'strict',
properties: {
...this.schema,
},
},
});
I have tried updating my mappings in elastic Search using Java Code.
But to my dismay the mappings are not getting updated
Following is my code
String settingsAsJson =getJsonString("settingFile");
client.admin().indices().close(new CloseIndexRequest(indexName)).actionGet().isAcknowledged();
boolean settingsAck = client.admin().indices().prepareUpdateSettings().setSettings(settingsAsJson).setIndices(indexName).execute().actionGet().isAcknowledged();
System.out.println("Applied SETTINGS "+ settingsAck);
String mappingAsJson = getJsonString("mappingFile");
boolean mappingack = client.admin().indices().preparePutMapping().setIndices(indexName).setType(indexType).setSource(mappingAsJson).execute().actionGet().isAcknowledged();
System.out.println("Applied Mappings "+ mappingack);
client.admin().indices().open(new OpenIndexRequest(indexName));
client.admin().indices().prepareFlush(indexName);
Following is my mapping file
{
"profiles": {
"dynamic" : "true",
"_all": {
"type": "string",
"analyzer": "standard"
},
"properties": {
"employee_id": {
"type": "integer",
"analyzer":"standard"
}
}
}
}
Also note, only the dynamic property changes the rest all new updation on mappings dont get updated.
Is there any other way to update mappings using java code?