I have tried updating my mappings in elastic Search using Java Code.
But to my dismay the mappings are not getting updated
Following is my code
String settingsAsJson =getJsonString("settingFile");
client.admin().indices().close(new CloseIndexRequest(indexName)).actionGet().isAcknowledged();
boolean settingsAck = client.admin().indices().prepareUpdateSettings().setSettings(settingsAsJson).setIndices(indexName).execute().actionGet().isAcknowledged();
System.out.println("Applied SETTINGS "+ settingsAck);
String mappingAsJson = getJsonString("mappingFile");
boolean mappingack = client.admin().indices().preparePutMapping().setIndices(indexName).setType(indexType).setSource(mappingAsJson).execute().actionGet().isAcknowledged();
System.out.println("Applied Mappings "+ mappingack);
client.admin().indices().open(new OpenIndexRequest(indexName));
client.admin().indices().prepareFlush(indexName);
Following is my mapping file
{
"profiles": {
"dynamic" : "true",
"_all": {
"type": "string",
"analyzer": "standard"
},
"properties": {
"employee_id": {
"type": "integer",
"analyzer":"standard"
}
}
}
}
Also note, only the dynamic property changes the rest all new updation on mappings dont get updated.
Is there any other way to update mappings using java code?
Related
I am currently working on elastic search through my java Application . I know how to index the Java pojo using RestHighLevelClient. How i can make search only on new fields not the complete pojo.?
public class Employee{
private long id;
private String name;
private String designation;
private String address; //want to index but not searchable in elastic search
}
My Code for indexing is below which is working fine:
public String saveToEs(Employee employee) throws IOException {
Map<String, Object> map = objectMapper.convertValue(employee, Map.class);
IndexRequest indexRequest =
new IndexRequest(INDEX, TYPE, employee.getId().toString()).source(map, XContentType.JSON);
IndexResponse indexResponse = client.index(indexRequest, RequestOptions.DEFAULT);
I need to do this in java .Any help please or good link ?
Writing another answer for RestHighLevelClient As another answer is useful for people not using the Rest client and adding this in the first answer makes it too long.
Note: you are passing the type which is deprecated in ES 7.X and I am using the ES 7.X version, so my code is according to 7.X.
CreateIndexRequest request = new CreateIndexRequest("employee");
Map<String, Object> name = new HashMap<>();
name.put("type", "text");
Map<String, Object> address = new HashMap<>();
address.put("type", "text");
address.put("index", false);
Map<String, Object> properties = new HashMap<>();
properties.put("name", name);
properties.put("address", address);
Map<String, Object> mapping = new HashMap<>();
mapping.put("properties", properties);
request.mapping(mapping);
CreateIndexResponse createIndexResponse = client.indices().create(request, RequestOptions.DEFAULT);
Important points
I've used only 2 fields for illustration purpose, one of which is address field which is not searchable, and to do that I used, address.put("index", false); , while name is searchable field and there this option isn't present.
I've created index mapping using the Map method which is available in this official ES doc.
you can check the mapping created by this code, using mapping REST API.
Below is the mapping generated for this code in my system and you can see, index: false is added in the address field.
{
"employee": {
"mappings": {
"properties": {
"address": {
"type": "text",
"index": false
},
"name": {
"type": "text"
}
}
}
}
}
You can just use the same search JSON mentioned in the previous answer, to test that it's not searchable.
Use the index option as false on the address field, which is by default true to make it unsearchable. As mention in the same official ES link:
The index option controls whether field values are indexed. It accepts
true or false and defaults to true. Fields that are not indexed are
not queryable.
Let me show you how can you test it using the REST API and then the java code(using rest-high level client) to accomplish it.
Mapping
{
"mappings": {
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "text"
},
"designation": {
"type": "text"
},
"address": {
"type": "text",
"index" : false --> made `index` to false
}
}
}
}
Index few docs
{
"address" : "USA",
"name" : "Noshaf",
"id" : 234567892,
"designation" : "software engineer"
}
{
"address" : "USA california state",
"name" : "opster",
"id" : 234567890,
"designation" : "software engineer"
}
A simple match search query in JSON format on address field
{
"query": {
"match" : {
"address" : "USA"
}
}
}
Exception from Elasticsearch clearly mention, it's not searchable
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Cannot search on field [address] since it is not indexed."
}
Is it possible to set the value of context variable from java. I have modified the node json and added a action in that action in the result property I want to set data from my local database. So I am getting the action in java code and trying to set the value of result property of action object, below is the code which i am trying but it is not working. can some one suggest a better approach for this.
if(response.getActions()!=null) {
System.out.println("actions : "+response.getActions());
for (int i = 0; i < response.getActions().size(); i++) {
System.out.println(" i : "+response.getActions().get(i).getName());
if(response.getActions().get(i).getName()=="Apply_For_Loan") {
response.getActions().get(i).setResultVariable("123");
}
}
}
For setting value of result_variable in to the context below is my code.
if(response.getActions().get(i).getName().equals("Apply_For_Loan")) {
System.out.println("in action");
Assistant service = new Assistant("2018-12-12");
service.setUsernameAndPassword(userId, password);
service.setEndPoint(endPoint);
String result=response.getActions().get(i).getResultVariable();
Context context = response.getContext();
context.put(result, "123");
MessageOptions msg=new MessageOptions.Builder(watsonId)
.input(new InputData.Builder("Apply For Loan").build())
.context(context)
.build();
response=service.message(msg).execute();
System.out.println("msg : "+response);
}
Below is the response which i am getting after re-executing the assistant call.
{
"output": {
"generic": [
{
"response_type": "text",
"text": "Hello RSR, you loan application of 5420 is created. Please note the Loan Number for future use "
}
],
"text": [
"Hello RSR, you loan application of 5420 is created. Please note the Loan Number for future use "
],
"nodes_visited": [
"node_1_1544613102320",
"node_1_1544613102320"
],
"log_messages": []
},
"input": {
"text": "Apply For Loan"
},
"intents": [
{
"intent": "ApplyForLoan",
"confidence": 1.0
}
],
"entities": [],
"context": {
"number": 9.971070056E9,
"$Loan_Number": "123",
"system": {
"initialized": true,
"dialog_stack": [
{
"dialog_node": "node_1_1544613102320"
}
],
"dialog_turn_counter": 3.0,
"dialog_request_counter": 3.0,
"_node_output_map": {
"node_1_1544613102320": {
"0": [
0.0
]
}
},
"branch_exited": true,
"branch_exited_reason": "completed"
},
"Mail_Id": "Email_Id",
"conversation_id": "b59c7a02-2cc6-4149-ae29-602796ab22e1",
"person": "RSR",
"rupees": 5420.0
},
"actions": [
{
"name": "Apply_For_Loan",
"type": "client",
"parameters": {
"pername": "RSR",
"loanamount": 5420.0
},
"result_variable": "$Loan_Number"
}
]
}
In above response $Loan_Number is the result variable which i have updated from java code and the same result_variable i am using in the output text to return the $Loan_Number, but in output text it is still coming blank and in actions also it is still coming blank?
The simple answer to your question is no, you can't set anything in the JSON Model using setResultVariable. It should work like this:
1) Define an action on your Dialognode
The result_variable defines the <result_variable_name>, e.g. where the result of that action should be stored in the conversation context. If you execute some server side action, you read the result from the specified context location. In this example, the location is context.result_of_action, the context might be omitted. More details are here.
2) Process the action in Java Client
DialogNodeAction action = response.getActions().get(0);
final String result_variable = action.getResultVariable();
The result_variable is like a key. You obtain the value from your DB and add it to the context:
Context context = response.getContext();
context.put(result_variable, "value from DB");
new MessageOptions.Builder("Your ID")
.input(new InputData.Builder("Your response").build())
.context(context) // setting context
.build();
Finally the Watson Assistant Dialog (or some other App) can get the result from the Client:
"context": {
"result_of_action": "value from DB",
The Objects you read using the Java API are being parsed from JSON using Gson and need to be re-parsed back to JSON when construction a new API Call.
Edit after question update
In your example:
"actions": [
{
"name": "Apply_For_Loan",
"type": "client",
"parameters": {
"pername": "RSR",
"loanamount": 5420.0
},
"result_variable": "$Loan_Number"
}
the key result_variable tells other modules in your app where to find the result of your action. So $Loan_Number should not be the value but the key to the value in the session context (a kind of method contract).
Let's say you set the value to context.my_result, then your Watson Assistant should be able to access the "123" from DB in the next dialog node under "context.my_result".
I have a query in valid JSON format which works well in kibana or Sense when I use GET request.I am also able to create this query using XContentBuilder, but I need to send this query using its JSON form as it is to ElasticSearch. Is it possible to store the query in a JSON file and query ElasticSearch using this JSON file.
My query -
{
"min_score":5,
"sort" : [
{
"_geo_distance" : {
"location" : [40.715, -73.988],
"order" : "asc",
"unit" : "km",
"mode" : "min",
"distance_type" : "arc"
}
}
],
"query": {
"bool": {
"must": {
"query_string": {
"query": "hospital",
"analyzer": "english"
}
},
"filter": {
"geo_distance": {
"distance": "50000km",
"location": {
"lat": 40.715,
"lon": -73.988
}
}
}
}
}
}
What I want is to store this query in a JSON file and use this JSON file to send a search request directly without using Query builder.
You can use a search template, and store this template in the cluster state, see the official documentation about search templates, especially about pre-registered templates.
I'm trying to disable dynamic mapping creation for only specific indexes, not for all. For some reason I can't put default mapping with 'dynamic' : 'false'.
So, here left two options as I can see:
specify property 'index.mapper.dynamic' in file elasticsearch.yml.
put 'index.mapper.dynamic' at index creation time, as described here https://www.elastic.co/guide/en/kibana/current/setup.html#kibana-dynamic-mapping
First option may only accept values: true, false and strict. So there is no way to specify subset of specific indexes (like we do by pattern with property 'action.auto_create_index' https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation).
Second option just not works.
I've created index
POST http://localhost:9200/test_idx/
{
"settings" : {
"mapper" : {
"dynamic" : false
}
},
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
}
}
}
}
}
Then checked index settings:
GET http://localhost:9200/test_idx/_settings
{
"test_idx" : {
"settings" : {
"index" : {
"mapper" : {
"dynamic" : "false"
},
"creation_date" : "1445440252221",
"number_of_shards" : "1",
"number_of_replicas" : "0",
"version" : {
"created" : "1050299"
},
"uuid" : "5QSYSYoORNqCXtdYn51XfA"
}
}
}
}
and mapping:
GET http://localhost:9200/test_idx/_mapping
{
"test_idx" : {
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
}
}
}
}
}
}
so far so good, let's index document with undeclared field:
POST http://localhost:9200/test_idx/test_type/1
{
"field1" : "it's ok, field must be in mapping and in source",
"somefield" : "but this field must be in source only, not in mapping"
}
Then I've checked mapping again:
GET http://localhost:9200/test_idx/_mapping
{
"test_idx" : {
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
},
"somefield" : {
"type" : "string"
}
}
}
}
}
}
As you can see, mapping is extended regardless of index setting "dynamic" : false.
I've also tried to create index exactly as described in doc
PUT http://localhost:9200/test_idx
{
"index.mapper.dynamic": false
}
but got the same behavior.
Maybe I've missed something?
Thanks a lot in advance!
You're almost there: the value needs to be set to strict.
And the correct usage is the following:
PUT /test_idx
{
"mappings": {
"test_type": {
"dynamic":"strict",
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
And pushing this a bit further, if you want to forbid the creation even of new types, not only fields in that index, use this:
PUT /test_idx
{
"mappings": {
"_default_": {
"dynamic": "strict"
},
"test_type": {
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
Without _default_ template:
PUT /test_idx
{
"settings": {
"index.mapper.dynamic": false
},
"mappings": {
"test_type": {
"dynamic": "strict",
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
You must know about that the below part just mean that ES could'nt create a type dynamically.
"mapper" : {
"dynamic" : false
}
You should configure ES like this:
PUT http://localhost:9200/test_idx/_mapping/test_type
{
"dynamic":"strict"
}
Then you cant't index other field that without mapping any more ,and get an error as follow:
mapping set to strict, dynamic introduction of [hatae] within [data] is not allowed
If you wanna store the data,but make the field can't be index,you could take the setting like this:
PUT http://localhost:9200/test_idx/_mapping/test_type
{
"dynamic":false
}
Hope these can help the people with the same issue :).
The answer is in the doc (7x.): https://www.elastic.co/guide/en/elasticsearch/reference/7.x/dynamic.html
The dynamic setting controls whether new fields can be added
dynamically or not. It accepts three settings:
true
Newly detected fields are added to the mapping. (default)
false
Newly detected fields are ignored. These fields will not be indexed so
will not be searchable but will still appear in the _source field of
returned hits. These fields will not be added to the mapping, new
fields must be added explicitly.
strict
If new fields are detected, an exception is thrown and the document is
rejected. New fields must be explicitly added to the mapping.
PUT my_index
{
"mappings": {
"dynamic": "strict",
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
You cannot disable dynamic mapping in ES 7 anymore, what you can do if you have completely unstructured data is to disable completely the mapping for the index like this:
curl -X PUT "localhost:9200/my_index?pretty" -H 'Content-Type: application/json' -d'
{
"mappings": {
"enabled": false
}
}
'
if you are using python you can do this:
from elasticsearch import Elasticsearch
# Connect to the elastic cluster
es=Elasticsearch([{'host':'localhost','port':9200}])
request_body = {
"mappings": {
"enabled": False
}
}
es.indices.create(index = 'my_index', body = request_body)
For ES 7 if you want to update an existing index:
PUT customers/_mapping
{
"dynamic": "strict"
}
first, please be concern aboout value false or strict,they work in a different way.
using "dynamic": "false" and create documents with fields not covered by the mapping, those fields will be ignored (so they won't be stored) and wouldn't show up in _source when you GET the document.
where value strict will not allow you to create the document rather it will throw an exception
Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
PUT my-index-000001
{
"mappings": {
"dynamic": false,
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
if you are using node.js client
await this.client.indices.putMapping({
index: ElasticIndex.UserDataFactory,
body: {
dynamic: 'strict',
properties: {
...this.schema,
},
},
});
I have a sample json which I want to index into elasticsearch.
Sample Json Indexed:
put test/names/1
{
"1" : {
"name":"abc"
},
"2" : {
"name":"def"
},
"3" : {
"name":"xyz"
}
}
where ,
index name : test,
type name : names,
id :1
Now the default mapping generated by elasticsearch is :
{
"test": {
"mappings": {
"names": {
"properties": {
"1": {
"properties": {
"name": {
"type": "string"
}
}
},
"2": {
"properties": {
"name": {
"type": "string"
}
}
},
"3": {
"properties": {
"name": {
"type": "string"
}
}
},
"metadataFieldDefinition": {
"properties": {
"name": {
"type": "string"
}
}
}
}
}
}
}
}
If the map size increases from 3 ( currently) to suppose thousand or million, then ElasticSearch will create a mapping for each which may cause a performance issue as the mapping collection will be huge .
I tried creating a mapping by setting :
"dynamic":false,
"type":object
but it was overriden by ES. since it didnt match the indexed data.
Please let me know how can I define a mapping so that ES. doesnot creates one like the above .
I think there might be a little confusion here in terms of how we index documents.
put test/names/1
{...
document
...}
This says: the following document belongs to index test and is of type name with id 1. The entire document is treated as type name. Using the PUT API as you currently are, you cannot index multiple documents at once. ES immediately interprets 1, 2, and 3 as a properties of type object, each containing a property name of type string.
Effectively, ES thinks you are trying to index ONE document, instead of three
To get many documents into index test with a type of name, you could do this, using the CURL syntax:
curl -XPUT"http://your-es-server:9200/test/names/1" -d'
{
"name": "abc"
}'
curl -XPUT"http://your-es-server:9200/test/names/2" -d'
{
"name": "ghi"
}'
curl -XPUT"http://your-es-server:9200/test/names/3" -d'
{
"name": "xyz"
}'
This will specify the document ID in the endpoint you are index to. Your mapping will then look like this:
"test": {
"mappings": {
"names": {
"properties": {
"name": {
"type": "string"
}
}
}
}
}
Final Word: Split your indexing up into discrete operations, or check out the Bulk API to see the syntax on how to POST multiple operations in a single request.