According to Elasticsearch documentation it is possible to exclude a field from _all field using include_in_all setting (set to false). I need to exclude a field from _all and I'm using spring data elasticsearch do define my mappings. I haven't found a way to do it this way.
Is this possible using spring data elasticsearch annotations?
Unfortunately, Spring Data Elasticsearch cannot do this thing. The improvement request is already a year old, but its priority is minor, so there is no hope for now:
https://jira.spring.io/browse/DATAES-226
In my last project I had to do a lot of simple searches (instead of one by "_all" fields, I used 1 search per each field), and then I united all the results. I assume that there is no nice solve for that problem by Spring Data Elasticsearch.
You can save the mapping of your type into a json file. Then you add the '"include_in_all": false' to the field you want to exclude. This should look something like this.
{
"my_type": {
"properties": {
"excludedField": {
"type": "text",
"include_in_all": false
}
}
}
}
Then you load the file using your favorite JSON reader, parse it as a string and change your mapping with the elasticsearch api .
client.admin().indices()
.preparePutMapping("my_index")
.setType("my_type")
.setSource(loadedFileString)
.get();
EDIT: I just noticed you wanted to use annotations for it. Maybe the #Field annotation has a parameter for it? I'm sorry, I have no experience with the annotations.
Related
We're in the process of converting our java application from Hibernate Search 5 to 6 with an Elasticsearch backend.
For some good background info, see How to do highlighting within HibernateSearch over Elasticsearch for a question we had when upgrading our highlighting code from a Lucene to Elasticsearch backend and how it was resolved.
Hibernate Search 6 seems to support using 2 backends at the same time, Lucene and Elasticsearch, so we'd like to use Elasticsearch for all our queries and Lucene for the highlighting, if that's possible.
Here is basically what we're trying to do:
public boolean matchPhoneNumbers() {
String phoneNumber1 = "603-436-1234";
String phoneNumber2 = "603-436-1234";
LuceneBackend luceneBackend =
Search.mapping(entityManager.getEntityManagerFactory())
.backend().unwrap(LuceneBackend.class);
Analyzer analyzer = luceneBackend.analyzer("phoneNumberKeywordAnalyzer").get();
//... builds a Lucene Query using the analyzer and phoneNumber1 term
Query phoneNumberQuery = buildQuery(analyzer, phoneNumber1, ...);
return isMatch("phoneNumberField", phoneNumber2, phoneNumberQuery, analyzer);
}
private boolean isMatch(String field, String target, Query sourceQ, Analyzer analyzer) {
Highlighter highlighter = new Highlighter(new QueryScorer(sourceQ, field));
highlighter.setTextFragmenter(new NullFragmenter());
try {
String result = highlighter.getBestFragment(analyzer, field, target);
return StringUtils.hasText(result);
} catch (IOException e) {
...
}
}
What I've attempted so far is to configure two separate backends in the configuration properties, per the documentation, like this:
properties.setProperty("hibernate.search.backends.elasticsearch.analysis.configurer", "com.bt.demo.search.AnalysisConfigurer");
properties.setProperty("hibernate.search.backends.lucene.analysis.configurer", "com.bt.demo.search.CustomLuceneAnalysisConfigurer");
properties.setProperty("hibernate.search.backends.elasticsearch.type", "elasticsearch");
properties.setProperty("hibernate.search.backends.lucene.type", "lucene");
properties.setProperty("hibernate.search.backends.elasticsearch.uris", "http://127.0.0.1:9200");
The AnalysisConfigurer class implements ElasticsearchAnalysisConfigurer and
CustomLuceneAnalysisConfigurer implements from LuceneAnalysisConfigurer.
Analyzers are defined twice, once in the Elasticsearch configurer and again in the Lucene configurer.
I don't know why both hibernate.search.backends.elasticsearch.type and hibernate.search.backends.lucene.type are necessary but if I don't include the lucene.type, I get Ambiguous backend type: configuration property 'hibernate.search.backends.lucene.type' is not set.
But if I do have both backend properties types set, I get
HSEARCH000575: No default backend. Check that at least one entity is configured to target the default backend, when attempting to retrieve the Lucene backend, like:
Search.mapping(entityManager.getEntityManagerFactory())
.backend().unwrap(LuceneBackend.class);
And the same error when trying to retrieve the Elasticsearch backend.
I've also added #Indexed(..., backend = "elasticsearch") to my entities since I wish to have them saved into Elasticsearch and don't need them in Lucene. I also tried adding a fake entity with #Indexed(..., backend = "lucene") but it made no difference.
What have I got configured wrong?
I don't know why both hibernate.search.backends.elasticsearch.type and hibernate.search.backends.lucene.type are necessary but if I don't include the lucene.type, I get Ambiguous backend type: configuration property 'hibernate.search.backends.lucene.type' is not set.
That's because the backend name is just that: a name. Hibernate Search doesn't infer particular information from it, even if you name your backend "lucene" or "elasticsearch". You could have multiple Elasticsearch backends for all it knows :)
But if I do have both backend properties types set, I get HSEARCH000575: No default backend. Check that at least one entity is configured to target the default backend, when attempting to retrieve the Lucene backend, like:
Search.mapping(entityManager.getEntityManagerFactory())
.backend().unwrap(LuceneBackend.class);
``
You called .backend(), which retrieves the default backend, i.e. the backend that doesn't have a name and is configured through hibernate.search.backend.* instead of hibernate.search.backends.<somename>.* (see https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#configuration-structure ).
But you are apparently mapping all your entities to a named backends, one named elasticsearch and one named lucene. So the default backend just doesn't exist.
You should call this:
Search.mapping(entityManager.getEntityManagerFactory())
.backend("lucene").unwrap(LuceneBackend.class);
I've also added #Indexed(..., backend = "elasticsearch") to my entities since I wish to have them saved into Elasticsearch
Since you obviously only want to use one backend for indexing, I would recommend reverting that change (keeping #Indexed without setting #Indexed.backend) and simply making using the default backend.
In short, remove the #Indexed.backend and replace this:
properties.setProperty("hibernate.search.backends.elasticsearch.analysis.configurer", "com.bt.demo.search.AnalysisConfigurer");
properties.setProperty("hibernate.search.backends.lucene.analysis.configurer", "com.bt.demo.search.CustomLuceneAnalysisConfigurer");
properties.setProperty("hibernate.search.backends.elasticsearch.type", "elasticsearch");
properties.setProperty("hibernate.search.backends.lucene.type", "lucene");
properties.setProperty("hibernate.search.backends.elasticsearch.uris", "http://127.0.0.1:9200");
With this
properties.setProperty("hibernate.search.backend.analysis.configurer", "com.bt.demo.search.AnalysisConfigurer");
properties.setProperty("hibernate.search.backends.lucene.analysis.configurer", "com.bt.demo.search.CustomLuceneAnalysisConfigurer");
properties.setProperty("hibernate.search.backend.type", "elasticsearch");
properties.setProperty("hibernate.search.backends.lucene.type", "lucene");
properties.setProperty("hibernate.search.backend.uris", "http://127.0.0.1:9200");
You don't technically have to do that, but I think it will be simpler in the long term. It keeps the Lucene backend as a separate hack that doesn't affect your whole application.
I also tried adding a fake entity with #Indexed(..., backend = "lucene")
I confirm you will need that fake entity mapped to the "lucene" backend, otherwise Hibernate Search will not create the "lucene" backend.
With Relational Databases, I could create a model in Java since I know How many columns a table has and their respective names.
But MongoDb doesn't work often with such static schema(If I'm not wrong). So, I want to create a model in Java that hold all the data and convert it into JSON and sent it to the response as a web service.
But I could do by returning just Document or DBObject objects. But It's converting ID as
"_id": {
"timestamp": 1505194179,
"machineIdentifier": 13503772,
"processIdentifier": 3816,
"counter": 1819499,
"date": 1505194179000,
"time": 1505194179000,
"timeSecond": 1505194179
}
Where I just need an ID value for further to make network calls to use this identifier.
So I want to know a best practice or strategy in order to achieve this. I use spring boot.
I'me very much new to Mongodb with Spring boot. So, bear me If my understanding is wrong.
Edit: In Spring boot, it's necessary to define entity class to access the data but I just want to know other way where I don't want a model to be predefined but it should be dynamic as the schema in mongodb.
I'm using Spring 4 JPQL / Criteria API to get results for aggregate query, i've created the special constructor in the entity class (with sum and key) and it works without any issues, however the return JSON does not return all the attributes from that class...any idea how I can control what the JSON return structure is? What's even weirder - when running in debug mode then in the following break point:
List<ActivityResponse> tActivityResponses = responseRepository.getTRXByMonths(months);
return tActivityResponses;
I am looking at the list and each member of the model class has the correct attributes, like month but month does not appear in the JSON.
Thanks.
Thanks everyone for the replies, i found the answer:
In the REST config class I had to specify config.exposeIdsFor(class name), and that showed all those missing fields.
Thanks.
This might seem like an odd question, but I am trying to get a handle on what the "best practice" is for converting an application that is set up to use something like Roo's or Grails' generation of controllers (which provides basic CRUD functionality) to something that returns a JSON response body instead for use in a JavaScript application.
The ambiguity of technology here is because I haven't actually started the project yet. I am still trying to decide which (Java-based) technology to use, and to see what kind of productivity tools I should learn/use in the process. It will be a web application, and it will use a database persistence layer. All other details are up in the air.
Perhaps the easiest way to accomplish my goal is to develop using some sort of AJAX plugin to start with, but most of the tutorials and descriptions out there say to start with a normal MVC architecture. Roo seems to make conversion of the controllers it generates to JSON-friendly return types difficult, and I'm not familiar enough with Groovy/Grails to know what it takes to do that.
This is mostly a learning experience for me, and I am open to any criticism or advice, but being a Q/A forum, I realize I need to incorporate an objective question of some sort. To fill that need, I ask:
What is the best way to set up an AJAX/RESTful interface for my entities in Roo and/or Grails?
I recently did exactly this with a Grails application and found it surprisingly easy to take the generated controllers and get them to output JSON or XML or the HTML from the view depending on the content negotiation.
The places in the Grails manual to look into are the section(s) on Content Negotiation and, if you need to deal with JSON or XML input, marshaling.
To get JSON and XML output, in the default list() method, changed it to this (I have a Session object, in this case...one of my domain classes):
def list() {
params.max = Math.min(params.max ? params.int('max') : 10, 100)
def response = [sessionInstanceList: Session.list(params), sessionInstanceTotal: Session.count()]
withFormat {
html response
json {render response as JSON}
xml {render response as XML}
}
}
Anywhere you are returning just an object by default, you will want to replace the returned value with the withFormat block.
You also may need to update your Config.groovy file where it deals with mime types. Here's what I have:
grails.mime.file.extensions = true // enables the parsing of file extensions from URLs into the request format
grails.mime.use.accept.header = true
grails.mime.types = [ html: ['text/html','application/xhtml+xml'],
xml: ['text/xml', 'application/xml'],
text: 'text/plain',
js: 'text/javascript',
rss: 'application/rss+xml',
atom: 'application/atom+xml',
css: 'text/css',
csv: 'text/csv',
all: '*/*',
json: ['application/json','text/json'],
form: 'application/x-www-form-urlencoded',
multipartForm: 'multipart/form-data'
]
As input (to an update() or save() action, for example) JSON and XML payloads will automatically be unmarshaled and will be bound just like a form input would be, but I've found that the unmarshaling process is a bit picky (especially with JSON).
I found that, in order for JSON to be handled correctly in the update() method, the class attribute had to be present and correct on the inbound JSON object. Since the library I was using in my client application didn't make this an easy issue to deal with, I switched to using XML instead.
Guys, Well I have done enough research still I can't find the solution to this.
In a nutshell, I'm simply passing url encoded form data to the Controller method and trying to convert it as a domain object which has Date and integers.
#RequestMapping(value = "/savePassport", method = RequestMethod.POST)
public #ResponseBody
AjaxResponse savePassport(#RequestBody StaffPassport passport, HttpServletResponse response) {
// Some operations.
}
The Staff Passport looks like this:
import java.sql.Date;
public class StaffPassport {
private int staffId;
private String passportNumber;
private String placeOfIssue;
private Date issueDate;
private Date expiryDate;
private String spouseName;
private String oldPassportRef;
private String visaInfo;
private String description;
//gets/sets
}
When I invoke the /savePassport, I get unsupported media exception. I guess it's related to casting.
I can't this working right. Of course I can catch individual form data using #RequestParam and manually do the casting but that's not the point of a framework isn't it?
Where am I going wrong? And you are right. I'm a beginner in Spring, but I love it.
Looks like you're using the wrong annotation. #RequestBody is for taking a request that has arbitrary content in its body,such as JSON, some application defined XML, comma separated variables.. whatever. And using a marshaller that you configure in the dispatcher servlet to turn it into objects.
If all you want to do is ask Spring to bind a plain old form post onto the backing object for you, the correct annotation to put on the method parameter is #ModelAttribute.
If you are posting a JSON Object with jQuery and you want Spring to be able to process it with #RequestBody, use JSON.stringify(....) in your data. Here an example:
var data = { "id": 3, "name": "test" }
$.post("processJsonData.html",JSON.stringify(data), function(data){
...
}
);
If you don't use the JSON.stringify() then you will submit the data as form data and Spring will tell you that you have an unsupported media type.
First of all be sure that you have
<mvc:annotation-driven />
in your Spring configuration file. This is mandatory for working with JSOn in SPring MVC.
Second, I recommend you to test wether request to the server has application/json content type. I belive Fiddler2 will help you to do so.
Third, but I'm not sure about it, Try to change Date items in your POJO from SQL type to regular java type.
UPDATE:
just looked at the Form and it seems like your "Accept" HTTP Header should be also application/json. Please test this issue with Fiddler2 as well.
I assume that you are posting JSON and want Spring to convert to StaffPassport. If you are getting an Unsupported media exception, it is because Spring could not figure out an appropriate way to perform the conversion.
For Spring to convert JSON, it needs Jackson -- make sure you have the Jackson jars in your project. If this is a Maven based project you can add the jackson-mapper-asl artifact ID to your pom.xml. This should give you the jackson-mapper and jackson-core jars.
Edit: I should mention that this applies to Spring 3 (I recently ran into this problem). I'm not sure what else is required for previous versions of Spring.
Check into HttpMessageConverter interface and its implementations. You could write your own implementation of it to convert it to the domain model you want. By the time the control gets to your method, you can access it as if your domain model object is passed.
Ok, I think I should refine my answer. I do not have direct experience of using it in a spring-mvc project but spring-integration. I am pretty sure the applicable media type (application/x-url-form-encoded) is already handled and converted to MultiMap by Spring framework; so, retrieve the values from that just like any other map with the key value being your form variable and populate your business model.
HTH.