Develop Reference

Query annotation not working with and / or operators

I'm trying to create custom MongoDB queries in Java with Spring
A simple query, like this, works fine
#Query("{'status' : ?0}")
Page<Project> filterProjects(String status, Pageable pageable);
But when I try a more complicated query with $and and $or, I don't get back any results
#Query(value = "{ '$and' : [ { '$or' : [ { 'project_title': {$regex:?0,$options:'i'}}, { 'project_description': {$regex:?0,$options:'i'}}, { 'project_short_name': {$regex:?0,$options:'i'}}]}, { 'status' : ?1}, { 'assignee' : ?2} ]}")
Page<Project> filterProjects(String search, String status, String assignee, Pageable pageable);
The raw mongodb query version of the above works fine
db.project.find( {
$and: [
{ $or: [ { 'project_title': {$regex: <search> ,$options:'i'}}, { 'project_description': {$regex: <search>,$options:'i'}}, { 'project_short_name': {$regex:<search>,$options:'i'}}]},
{ 'status' : <status>},
{ 'assignee' : <assignee>}
]
} )
Is there something wrong with the query in #Query or are these operations not supported at all, in #Query?

have you tried out removing the single quotes ('') around $and/$or in the value of your #Query annotation ?

Get the last inserted document in mongodb java

I need to get the last inserted client so i can create an secuencial id, i tried a #Query annotation with the follow path but it doesnt run the app.
public interface ClienteRepository extends MongoRepository<Cliente, String> {
#Query("[{ $sort: ({ _id: -1}).limit:(1)}]")
Cliente findLastCliente();
}```
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clienteRepository': Invocation of init method failed; nested exception is com.mongodb.util.JSONParseException:
[{ $sort: ({ _id: -1}).limit:(1)}]

You can do it in 3 ways
1: Directly use this. Spring will automatically create desired query for you
Cliente findTopByOrderByIdDesc();
2: You can use Pageable and get only 1 record.
#Query(sort = "{ _id : -1 }");
Page<Cliente> findByMethod(Pageable pageable);
// Inside service
Page fooPage = FooRepository.findByMethod('John', new PageRequest(0,1));
3: Custom Query
Query query = new Query();
query.limit(1);
query.with(new Sort(Sort.Direction.DESC, "_id"));
mongoOperation.find(query, Cliente.class);
I would prefer the 1st option
By the way issue you are getting is because this #Query("[{ $sort: ({ _id: -1}).limit:(1)}]") is not in correct JSON format which can be translate to Mongo Query.

Spring data aggregation query elasticsearch

I am trying to make the below elasticsearch query to work with spring data. The intent is to return unique results for the field "serviceName". Just like a SELECT DISTINCT serviceName FROM table would do comparing to a SQL database.
{
"aggregations": {
"serviceNames": {
"terms": {
"field": "serviceName"
}
}
},
"size":0
}
I configured the field as a keyword and it made the query work perfectly in the index_name/_search api as per the response snippet below:
"aggregations": {
"serviceNames": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "service1",
"doc_count": 20
},
{
"key": "service2",
"doc_count": 8
},
{
"key": "service3",
"doc_count": 8
}
]
}
}
My problem is the same query doesn't work in Spring data when I try to run with a StringQuery I get the error below. I am guessing it uses a different api to run queries.
Cannot execute jest action , response code : 400 , error : {"root_cause":[{"type":"parsing_exception","reason":"no [query] registered for [aggregations]","line":2,"col":19}],"type":"parsing_exception","reason":"no [query] registered for [aggregations]","line":2,"col":19} , message : null
I have tried using the SearchQuery type to achieve the same results, no duplicates and no object loading, but I had no luck. The below sinnipet shows how I tried doing it.
final TermsAggregationBuilder aggregation = AggregationBuilders
.terms("serviceName")
.field("serviceName")
.size(1);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices("index_name")
.withQuery(matchAllQuery())
.addAggregation(aggregation)
.withSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.withSourceFilter(new FetchSourceFilter(new String[] {"serviceName"}, new String[] {""}))
.withPageable(PageRequest.of(0, 10000))
.build();
Would someone know how to achieve no object loading and object property distinct aggregation on spring data?
I tried many things without success to print queries on spring data, but I could not, maybe because I am using the com.github.vanroy.springdata.jest.JestElasticsearchTemplate implementation.
I got the query parts with the below:
logger.info("query:" + searchQuery.getQuery());
logger.info("agregations:" + searchQuery.getAggregations());
logger.info("filter:" + searchQuery.getFilter());
logger.info("search type:" + searchQuery.getSearchType());
It prints:
query:{"match_all":{"boost":1.0}}
agregations:[{"serviceName":{"terms":{"field":"serviceName","size":1,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}]
filter:null
search type:DFS_QUERY_THEN_FETCH

I figured out, maybe can help someone. The aggregation don't come with the query results, but in a result for it self and is not mapped to any object. The Objects results that comes apparently are samples of the query elasticsearch did to run your aggregation (not sure, maybe).
I ended up by creating a method which can do a simulation of what would be on the SQL SELECT DISTINCT your_column FROM your_table, but I think this will work only on keyword fields, they have a limitation of 256 characters if I am not wrong. I explained some lines in comments.
Thanks #Val since I was only able to figure it out when debugged into Jest code and check the generated request and raw response.
public List<String> getDistinctField(String fieldName) {
List<String> result = new ArrayList<>();
try {
final String distinctAggregationName = "distinct_field"; //name the aggregation
final TermsAggregationBuilder aggregation = AggregationBuilders
.terms(distinctAggregationName)
.field(fieldName)
.size(10000);//limits the number of aggregation list, mine can be huge, adjust yours
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withIndices("your_index")//maybe can be omitted
.addAggregation(aggregation)
.withSourceFilter(new FetchSourceFilter(new String[] { fieldName }, new String[] { "" }))//filter it to retrieve only the field we ar interested, probably we can take this out.
.withPageable(PageRequest.of(0, 1))//can't be zero, and I don't want to load 10 results every time it runs, will always return one object since I found no "size":0 in query builder
.build();
//had to use the JestResultsExtractor because com.github.vanroy.springdata.jest.JestElasticsearchTemplate don't have an implementation for ResultsExtractor, if you use Spring defaults, you can probably use it.
final JestResultsExtractor<SearchResult> extractor = new JestResultsExtractor<SearchResult>() {
#Override
public SearchResult extract(SearchResult searchResult) {
return searchResult;
}
};
final SearchResult searchResult = ((JestElasticsearchTemplate) elasticsearchOperations).query(searchQuery,
extractor);
final MetricAggregation aggregations = searchResult.getAggregations();
final TermsAggregation termsAggregation = aggregations.getTermsAggregation(distinctAggregationName);//this is where your aggregation results are, in "buckets".
result = termsAggregation.getBuckets().parallelStream().map(TermsAggregation.Entry::getKey)
.collect(Collectors.toList());
} catch (Exception e) {
// threat your error here.
e.printStackTrace();
}
return result;
}

Get Required Field From Elasticsearch

I have the below fields in my elasticsearch
"_source": {
"#timestamp":
"cpu_stat_s": {
"temp_in_celsius":
"model_name":,
"cpu_MHz_String": ,
"cache_size_string":
},
"memory_stat_s": {
"total_memory": ,
"swap_total": ,
"swap_free": ,
"used": ,
"free": ,
"swap_used":
},
"process_stat_s": [
{
"process_name": ,
"mem_in_use":,
"process_pid":
}
]
I just want to get suppose cpu_stat_s and memory_stat_s field from the elasticsearch using java api ,I searched through the Query Filters and also did
String source="{\"query\":\"ether_stat_s\"}";
SearchResponse response = client.prepareSearch("cn_*")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(0).setSize(10).setExplain(true)
.setPostFilter(FilterBuilders.rangeFilter("#timestamp").from(TIMESTAMP_FROM).to(TIMESTAMP_TO)).**setSource(source).**execute().actionGet();
But I received all fields instead of the required fields .
I can do this by getting all fields and then use a for loop and from response get the source but it would have increase the CPU time incase of billions of records .I am new to elastic search and hence it would be of great help if someone could assist me

After Finally more research I found the answer
SearchResponse response = client.prepareSearch("cn_*")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFrom(0).setSize(10).setExplain(true)
.setPostFilter(FilterBuilders.rangeFilter("#timestamp").from(TIMESTAMP_FROM).to(TIMESTAMP_TO))
.setFetchSource(new String[]{"ether_stat_s"}, null)
.execute()
.actionGet();

elasticsearch - No query registered for [query]]

I am trying to send request to ES from my tests. I applied mapping and inserted documents to ES index named 'gccount_test' from the same test. I have a very simple query maintained in a file named member that I want to test.
{
"query" : {
"match_all" : {}
}
}
My test method is
public void testMemberQuery(){
final Charset CHARSET = StandardCharsets.UTF_8
//load query
byte[] bytes = Files.readAllBytes(Paths.get(MEMBER_QUERY_PATH))
String query = CHARSET.decode(ByteBuffer.wrap(bytes)).toString()
println "QUERY => ${query}"
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
searchSourceBuilder.query(query)
SearchRequestBuilder searchRequestBuilder = client.prepareSearch(INDEX_NAME)
//ClusterAdminClient adminClient = client.admin().cluster()
//searchRequestBuilder.setTypes(Constants.ESTYPE_MEMBER)
//println "CLUSTER => ${adminClient}"
searchRequestBuilder.setSearchType(SearchType.QUERY_THEN_FETCH);
searchRequestBuilder.internalBuilder(searchSourceBuilder)
SearchResponse searchResponse = searchRequestBuilder.execute().actionGet()
println "Search Response => ${searchResponse.toString()}"
//blah blah
}
Unfortunately, I get following error.
Failed to execute phase [query_fetch], total failure; shardFailures {[1][gccount][0]: SearchParseException[[gccount_test][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query_binary":"ewogICAgInF1ZXJ5IiA6IHsgCiAgICAgICAgICAibWF0Y2hfYWxsIiA6IHt9IAogICAgIH0KfQ=="}]]]; nested: QueryParsingException[[gccount_test] No query registered for [query]]; }
org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query_fetch], total failure; shardFailures {[1][gccount_test][0]: SearchParseException[[gccount_test][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query_binary":"ewogICAgInF1ZXJ5IiA6IHsgCiAgICAgICAgICAibWF0Y2hfYWxsIiA6IHt9IAogICAgIH0KfQ=="}]]]; nested: QueryParsingException[[gccount_test] No query registered for [query]]; }
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:261)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:214)
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:246)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:206)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:193)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
I am using elasticsearch 0.90.2 dependecy
[group: 'org.elasticsearch', name: 'elasticsearch', version: '0.90.2']
Same thing runs fine in real environment(snapshot below)
Is the problem with while loading query from file that caused it's malformation or what?

The exception basically means "There is no known query type called query". I'm guessing that your client library is automatically inserting the top-level query property, so your generated query actually looks like this:
{
"query" : {
"query" : {
"match_all" : {}
}
}
}
If your client can dump the JSON representation of the query, that can help a lot in debugging.
Try removing the query portion from your text file so that it is just the match_all query, see if that works for you.

your query string should be
String query = "{\"match_all\":{}}";
you can see from here
https://discuss.elastic.co/t/parsingexception-in-elastic-5-0-0/64626