I use spring-data-elasticsearch framework to get query result from elasticsearch server, the java code like this:
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery()).withSearchType(SearchType.COUNT)
.addAggregation(new MinBuilder("min_createDate").field("createDate"))
.build();
List<Entity> list = template.queryForList(searchQuery, Entity.class);
While how can I know the raw http query sent to elasticssearch server?
How can I enable the logging, I tried add log4j, but it seems the spring-data-elasticsearch doesn't log the query.
After digging through the spring data code i found this helpful little logger called "tracer" (name not very unique)
By setting the following in application.properties
logging.level.tracer=TRACE
It will print out a full curl statement for the request along with full JSON the response from Elasticsearch.
This one is quite old, but I'd still like to share the solution that worked for me. To log Spring Data Elasticsearch queries executed through the Repository, you need to enable DEBUG logging for the package org.springframework.data.elasticsearch.core.*, e.g. as follows:
logging:
level:
org:
springframework:
data:
elasticsearch:
core: DEBUG
After that, queries will appear in logs:
{
"from" : 0,
"size" : 1,
"query" : {
"bool" : {
"should" : [ {
"query_string" : {
"query" : "John Doe",
"fields" : [ "entityName" ],
"default_operator" : "and"
}
}, {
"query_string" : {
"query" : "John Doe",
"fields" : [ "alias" ],
"default_operator" : "and"
}
} ]
}
},
"post_filter" : {
"bool" : { }
}
}
One would expect an elegant solution similar to JPA, but it seems that it doesn't simply exist.
Tested with Spring Boot 1.4.0 and Spring Data Elasticsearch 1.7.3.
If you are using spring boot you can set the following in your application.properties:
logging.level.org.elasticsearch.index.search.slowlog.query=INFO
spring.data.elasticsearch.properties.index.search.slowlog.threshold.query.info=1ms
I don't have an answer for Spring Data Elasticsearch, but in ES itself you can bump up the default settings for slow query logging and see all the queries in the slow log. More details about slow log here.
As to how to change the thresholds, a command like this should be used:
PUT /_settings
{
"index.search.slowlog.threshold.query.info": "1ms"
}
1ms is kindof the smallest value you can set.
This works on Spring Boot 2.3.3.RELEASE
logging.level.org.springframework.data.elasticsearch.client.WIRE=trace
I encountered the same problem, In the ElasticsearchTemplate only a few method have log debug level, E.g:
public <T> Page<T> queryForPage(CriteriaQuery criteriaQuery, Class<T> clazz) {
QueryBuilder elasticsearchQuery = new CriteriaQueryProcessor().createQueryFromCriteria(criteriaQuery.getCriteria());
QueryBuilder elasticsearchFilter = new CriteriaFilterProcessor().createFilterFromCriteria(criteriaQuery.getCriteria());
SearchRequestBuilder searchRequestBuilder = prepareSearch(criteriaQuery, clazz);
if (elasticsearchQuery != null) {
searchRequestBuilder.setQuery(elasticsearchQuery);
} else {
searchRequestBuilder.setQuery(QueryBuilders.matchAllQuery());
}
if (criteriaQuery.getMinScore() > 0) {
searchRequestBuilder.setMinScore(criteriaQuery.getMinScore());
}
if (elasticsearchFilter != null)
searchRequestBuilder.setPostFilter(elasticsearchFilter);
if (logger.isDebugEnabled()) {
logger.debug("doSearch query:\n" + searchRequestBuilder.toString());
}
SearchResponse response = getSearchResponse(searchRequestBuilder
.execute());
return resultsMapper.mapResults(response, clazz, criteriaQuery.getPageable());
}
#2280258 is correct, and here comes the official doc:
https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/index.html#elasticsearch.clients.logging
<logger name="org.springframework.data.elasticsearch.client.WIRE" level="trace"/>
Here is the reason: in org.springframework.data.elasticsearch.client.ClientLogger, spring data elasticsearch creates a logger named "org.springframework.data.elasticsearch.client.WIRE"：
private static final Logger WIRE_LOGGER = LoggerFactory
.getLogger("org.springframework.data.elasticsearch.client.WIRE");
Just to add my two cents to #AndreiStefan: Now you can set 0ms instead of 1ms. It seems that some very fast queries can be captured using this method.
Simply do:
PUT /_settings
{
"index.search.slowlog.threshold.query.info": "0ms"
}
Related
I'm trying to create custom MongoDB queries in Java with Spring
A simple query, like this, works fine
#Query("{'status' : ?0}")
Page<Project> filterProjects(String status, Pageable pageable);
But when I try a more complicated query with $and and $or, I don't get back any results
#Query(value = "{ '$and' : [ { '$or' : [ { 'project_title': {$regex:?0,$options:'i'}}, { 'project_description': {$regex:?0,$options:'i'}}, { 'project_short_name': {$regex:?0,$options:'i'}}]}, { 'status' : ?1}, { 'assignee' : ?2} ]}")
Page<Project> filterProjects(String search, String status, String assignee, Pageable pageable);
The raw mongodb query version of the above works fine
db.project.find( {
$and: [
{ $or: [ { 'project_title': {$regex: <search> ,$options:'i'}}, { 'project_description': {$regex: <search>,$options:'i'}}, { 'project_short_name': {$regex:<search>,$options:'i'}}]},
{ 'status' : <status>},
{ 'assignee' : <assignee>}
]
} )
Is there something wrong with the query in #Query or are these operations not supported at all, in #Query?
have you tried out removing the single quotes ('') around $and/$or in the value of your #Query annotation ?
I am trying to make the below elasticsearch query to work with spring data. The intent is to return unique results for the field "serviceName". Just like a SELECT DISTINCT serviceName FROM table would do comparing to a SQL database.
{
"aggregations": {
"serviceNames": {
"terms": {
"field": "serviceName"
}
}
},
"size":0
}
I configured the field as a keyword and it made the query work perfectly in the index_name/_search api as per the response snippet below:
"aggregations": {
"serviceNames": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "service1",
"doc_count": 20
},
{
"key": "service2",
"doc_count": 8
},
{
"key": "service3",
"doc_count": 8
}
]
}
}
My problem is the same query doesn't work in Spring data when I try to run with a StringQuery I get the error below. I am guessing it uses a different api to run queries.
Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"parsing_exception","reason":"no [query] registered for [aggregations]","line":2,"col":19}],"type":"parsing_exception","reason":"no [query] registered for [aggregations]","line":2,"col":19} , message : null
I have tried using the SearchQuery type to achieve the same results, no duplicates and no object loading, but I had no luck. The below sinnipet shows how I tried doing it.
final TermsAggregationBuilder aggregation = AggregationBuilders
.terms("serviceName")
.field("serviceName")
.size(1);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices("index_name")
.withQuery(matchAllQuery())
.addAggregation(aggregation)
.withSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.withSourceFilter(new FetchSourceFilter(new String[] {"serviceName"}, new String[] {""}))
.withPageable(PageRequest.of(0, 10000))
.build();
Would someone know how to achieve no object loading and object property distinct aggregation on spring data?
I tried many things without success to print queries on spring data, but I could not, maybe because I am using the com.github.vanroy.springdata.jest.JestElasticsearchTemplate implementation.
I got the query parts with the below:
logger.info("query:" + searchQuery.getQuery());
logger.info("agregations:" + searchQuery.getAggregations());
logger.info("filter:" + searchQuery.getFilter());
logger.info("search type:" + searchQuery.getSearchType());
It prints:
query:{"match_all":{"boost":1.0}}
agregations:[{"serviceName":{"terms":{"field":"serviceName","size":1,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}]
filter:null
search type:DFS_QUERY_THEN_FETCH
I figured out, maybe can help someone. The aggregation don't come with the query results, but in a result for it self and is not mapped to any object. The Objects results that comes apparently are samples of the query elasticsearch did to run your aggregation (not sure, maybe).
I ended up by creating a method which can do a simulation of what would be on the SQL SELECT DISTINCT your_column FROM your_table, but I think this will work only on keyword fields, they have a limitation of 256 characters if I am not wrong. I explained some lines in comments.
Thanks #Val since I was only able to figure it out when debugged into Jest code and check the generated request and raw response.
public List<String> getDistinctField(String fieldName) {
List<String> result = new ArrayList<>();
try {
final String distinctAggregationName = "distinct_field"; //name the aggregation
final TermsAggregationBuilder aggregation = AggregationBuilders
.terms(distinctAggregationName)
.field(fieldName)
.size(10000);//limits the number of aggregation list, mine can be huge, adjust yours
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices("your_index")//maybe can be omitted
.addAggregation(aggregation)
.withSourceFilter(new FetchSourceFilter(new String[] { fieldName }, new String[] { "" }))//filter it to retrieve only the field we ar interested, probably we can take this out.
.withPageable(PageRequest.of(0, 1))//can't be zero, and I don't want to load 10 results every time it runs, will always return one object since I found no "size":0 in query builder
.build();
//had to use the JestResultsExtractor because com.github.vanroy.springdata.jest.JestElasticsearchTemplate don't have an implementation for ResultsExtractor, if you use Spring defaults, you can probably use it.
final JestResultsExtractor<SearchResult> extractor = new JestResultsExtractor<SearchResult>() {
#Override
public SearchResult extract(SearchResult searchResult) {
return searchResult;
}
};
final SearchResult searchResult = ((JestElasticsearchTemplate) elasticsearchOperations).query(searchQuery,
extractor);
final MetricAggregation aggregations = searchResult.getAggregations();
final TermsAggregation termsAggregation = aggregations.getTermsAggregation(distinctAggregationName);//this is where your aggregation results are, in "buckets".
result = termsAggregation.getBuckets().parallelStream().map(TermsAggregation.Entry::getKey)
.collect(Collectors.toList());
} catch (Exception e) {
// threat your error here.
e.printStackTrace();
}
return result;
}
I'm trying to perform the following query through ElasticSearch Java API
{
"query" : {
"fuzzy" : {
"dateOfBirth" : {
"value" : "1944-11-30",
"fuzziness" : "365d"
}
}
}
}
I doesn't understand how to specify the fuzzines value for 365 days in this kind of query.
You can use this:
FuzzyQueryBuilder queryBuilder = fuzzyQuery("dateOfBirth" ,"1944-11-30" ).fuzziness(Fuzziness.build("365d"))
Hope this helps
I have a template foo.mustache saved in {{ES_HOME}}/config/scripts.
POST to http://localhost:9200/forward/_search/template with the following message body returns a valid response:
{
"template": {
"file": "foo"
},
"params": {
"q": "a",
"hasfilters": false
}
}
I want to translate this to using the java API now that I've validated all the different components work. The documentation here describes how to do it in java:
SearchResponse sr = client.prepareSearch("forward")
.setTemplateName("foo")
.setTemplateType(ScriptService.ScriptType.FILE)
.setTemplateParams(template_params)
.get();
However, I would instead like to just send a plain string query (i.e. the contents of the message body from above) rather than build up the response using the java. Is there a way to do this? I know with normal queries, I can construct it like so:
SearchRequestBuilder response = client.prepareSearch("forward")
.setQuery("""JSON_QUERY_HERE""")
I believe the setQuery() method wraps the contents into a query object, which is not what I want for my template query. If this is not possible, I will just have to go with the documented way and convert my json params to Map<String, Object>
I ended up just translating my template_params to a Map<String, Object> as the documentation requires. I utilized groovy's JsonSlurper to convert the text to an object with a pretty simple method.
import groovy.json.JsonSlurper
public static Map<String,Object> convertJsonToTemplateParam(String s) {
Object result = new JsonSlurper().parseText(s);
//Manipulate your result if you need to do any additional work here.
//I.e. Programmatically determine value of hasfilters if filters != null
return (Map<String,Object>) result;
}
And you could pass in the following as a string to this method:
{
"q": "a",
"hasfilters": true
"filters":[
{
"filter_name" : "foo.untouched",
"filters" : [ "FOO", "BAR"]
},
{
"filter_name" : "hello.untouched",
"list" : [ "WORLD"]
}
]
}
I have a collection of users:
> db.users.find().pretty()
{
"_id" : ObjectId("544ab933e4b099c3cfb62e12"),
"token" : "8c9f8cf4-1689-48ab-bf53-ee071a377f60",
"categories" : [
DBRef("cue_categories", ObjectId("544ab933e4b099c3cfb62e10")),
DBRef("cue_categories", ObjectId("544ab933e4b099c3cfb62e11"))
]
}
I want to find all users who have (let's say) ObjectId("544ab933e4b099c3cfb62e10") category and remove it (because this category was deleted and I don't want users to refer to it anymore).
The valid query to do it in JSON format would be:
db.users.update({
categories:{
$in:[
DBRef("cue_categories", ObjectId("544ab933e4b099c3cfb62e10"))
]
}
},
{
$unset:{
"categories.$":true
}
})
Here's a Spring mongodb query:
Query query = new Query();
query.addCriteria(Criteria.where("categories.$id").in(categoryIds));
Update update = new Update();
update.unset("categories.$");
operations.updateMulti(query, update, User.class);
In order to make an appropriate DB reference I have to provide a list of category IDs, each category ID (in categoryIds) is an instance of org.bson.types.ObjectId.
The problem is that the result query turns out to be without a positional operator:
DEBUG o.s.data.mongodb.core.MongoTemplate - Calling update using
query: { "categories.$id" : { "$in" : [ { "$oid" :
"544ab933e4b099c3cfb62e10"}]}} and update: { "$unset" : { "categories"
: 1}} in collection: users
So the update part must be { "$unset" : { "categories.$" : 1}}
P.S.
I managed to get around by falling back to the plain Java driver use
DBObject query = new BasicDBObject("categories.$id", new BasicDBObject("$in", categoryIds));
DBObject update = new BasicDBObject("$unset", new BasicDBObject("categories.$", true));
operations.getCollection("users").updateMulti(query, update);
But my question still remains open!
P.S.S.
My case is very similar to Update Array Field Using Positional Operator ($) Does Not Work bug and looks like it was fixed for versions 1.4.1 and 1.5. That being said I use spring-data-mongodb version 1.5.1. And I'm confused. Does anybody have a clue?
You can not use positional $ operator with unset as per MongoDB documentation. It will set the value as Null. https://docs.mongodb.com/manual/reference/operator/update/positional/