I am trying to map a successful MongoDB Aggregation to morphia but I am not able to get a satisfying result. I fails everytime but I cannot figure out why. Maybe someone of you could help me to state the aggregation correctly in morphia. My MongoDB Query looks like the following:
db.user.aggregate([{$match: { roles: "MEMBER" }}, {$group:{_id: "$roles", sum:{$sum: "$payments.2039.amount"}}}])
Roles is an array and the aggregation works fine and outputs:
{ "_id" : [ "MEMBER" ], "sum" : 100 }
I tried to do this in morphia with this java code:
final Query<User> query = datastore.createQuery(User.class).field("roles").in(Lists.newArrayList(Role.MEMBER));
final Iterator<AggregatePayments> aggregatePayments = datastore
.createAggregation(User.class)
.match(query)
.group("$roles", grouping("sum", sum("payments." + currentSeason + ".amount")))
.out(AggregatePayments.class);
But, unfortunately, this fails with the following exception:
com.mongodb.MongoCommandException: Command failed with error 17276 (Location17276): 'Use of undefined variable: roles' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "Use of undefined variable: roles", "code" : 17276, "codeName" : "Location17276" }
My problem is now finding out why this works in MongoDB but not in morphia. When I try to leave out the "$" at roles variable for _id in morphia I get the following exception:
com.mongodb.MongoCommandException: Command failed with error 16996 (Location16996): 'insert for $out failed: { connectionId: 1419, err: "can't use an array for _id", code: 2, codeName: "BadValue", n: 0, ok: 1.0 }' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "insert for $out failed: { connectionId: 1419, err: \"can't use an array for _id\", code: 2, codeName: \"BadValue\", n: 0, ok: 1.0 }", "code" : 16996, "codeName" : "Location16996" }
Any help appreciated! Thank you very much!
Using MongoDB version 3.6 and current morphia build 1.4-SNAPSHOT.
EDIT:
What seems really strange to me is that above Query in MongoDB works but the query generated by morphia not. But the manually edited Query in MongoDB and the generated one by morphia seem to correspond closely to another. Does anyone see any mistakes? Generated query is as follows:
11040 [qtp104739310-39] DEBUG org.mongodb.driver.protocol.command - Sending command '{ "aggregate" : "user", "pipeline" : [{ "$match" : { "roles" : { "$in" : ["MEMBER"] } } }, { "$group" : { "_id" : "$roles", "sum" : { "$sum" : "$payments.2039.amount" } } }, { "$out" : "AggregatePayments" }], "cursor" : { }, "$db" : "sua", "$readPreference" : { "mode" : "primaryPreferred" } }' with request id 19 to database sua on connection [connectionId{localValue:2, serverValue:1419}] to server localhost:27017
and produces the exception:
com.mongodb.MongoCommandException: Command failed with error 16996 (Location16996): 'insert for $out failed: { connectionId: 1419, err: "can't use an array for _id", code: 2, codeName: "BadValue", n: 0, ok: 1.0 }' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "insert for $out failed: { connectionId: 1419, err: \"can't use an array for _id\", code: 2, codeName: \"BadValue\", n: 0, ok: 1.0 }", "code" : 16996, "codeName" : "Location16996" }
When manually editing the query in MongoDB I can also use $group:{_id: "$roles".... Seems quite strange to me...
I didn't recognize the difference between morphia's "out()" and "aggregate()" method. "out()" changes the collection in the MongoDB store which was not intended. Using "aggregate()" now solved all problems:
final Query<User> query = datastore.createQuery(User.class).field("roles").in(Lists.newArrayList(Role.MEMBER));
final Iterator<AggregatePayments> aggregatePayments = datastore
.createAggregation(User.class)
.match(query)
.group("roles", grouping("sum", sum("payments." + currentSeason + ".amount")))
.aggregate(AggregatePayments.class);
Works now as expected with morphia 1.4.0-SNAPSHOT (most recent build from git master) and MongoDB 3.6.
Related
I'm using Mongodb 4 with spring boot 2.1.0-M4
The follwing code works locally on my computer, but fails at dev environment:
#Transactional
public void registerNewUser(UserRegistrationForm registrationForm)
throws IllegalArgumentException {
validator.validate(registrationForm);
User user = new User();
user.setEmail(registrationForm.email);
user.setPassword(encoder.encode(registrationForm.password));
user.setEnabled(false);
user.setGroups(Sets.newHashSet(ClientRoles.USER));
User saved = userRepository.save(user);
registrationService.sendInvitation(user.getEmail());
}
With the following error:
com.mongodb.MongoCommandException: Command failed with error 40527 (Location40527): 'Unable to persist transaction state because the session transaction collection is missing. This indicates that the config.transactions collection has been manually deleted.' on server mongodb-1-servers-vm-0:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1540422989, "i" : 1 } }, "ok" : 0.0, "errmsg" : "Unable to persist transaction state because the session transaction collection is missing. This indicates that the config.transactions collection has been manually deleted.", "code" : 40527, "codeName" : "Location40527", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1540422994, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type" : "00" }, "keyId" : { "$numberLong" : "0" } } } } at
I have no idea what I should try debugging, or what to try, both databases are of the same version and have replica sets initialized. Any help is appreciated.
Actually, the error is pretty self explanatory. It seems that for some reason transactions collection from dev environment db was deleted. The fix is straightforward:
use config
db.createCollection("transactions",{})
I search through the half of the Internet but found either nothing or outdated solutions with db.eval(). Hence my question.
I have some code using mongo driver 3.5.0:
MongoDatabase db = mongoClient.getDatabase 'mydb'
println db.runCommand( new BasicDBObject( eval:'db.version()' ) )
Exception in thread "main" com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on mydb to execute command { eval: "db.version()", $readPreference: { mode: "secondaryPreferred" }, $db: "mydb" }' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "not authorized on mydb to execute command { eval: \"db.version()\", $readPreference: { mode: \"secondaryPreferred\" }, $db: \"mydb\" }", "code" : 13, "codeName" : "Unauthorized" }
at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:107)
I can run this command in the shell:
mongo "mongodb://admin:admin#somehost:27003/mydb?replicaSet=myset" -eval 'db.version()'
outputs
3.6.6
If I run the command from inside the shell:
db.runCommand( { 'eval':'db.version()' } )
I'm also getting
{
"operationTime" : Timestamp(1536931496, 11),
"ok" : 0,
"errmsg" : "not authorized on mydb to execute command { eval: \"db.version()\", $db: \"mydb\" }",
"code" : 13,
"codeName" : "Unauthorized"
}
The user I'm connecting with looks like:
{
"_id" : "mydb.admin",
"user" : "admin",
"db" : "mydb",
"roles" : [
{
"role" : "readWrite",
"db" : "mydb"
},
{
"role" : "dbAdmin",
"db" : "mydb"
},
{
"role" : "userAdmin",
"db" : "mydb"
},
{
"role" : "dbOwner",
"db" : "mydb"
}
]
}
What am I missing?
Do I need special privileges for eval?
As per mongodb documentation, you need a role that grants anyAction on AnyResource.
I am looking for a Mongo query for finding data in the following document.
{
"key1": [{
"subkey1": ["america.south.gas"],
"subkey2": ["9898989898"]
}],
"key2": [{
"subkey1": ["america"],
"subkey2": ["hsadjsahjsahdjsah879878u9"]
},
{
"subkey1": ["america.south.gas","america"],
"subkey2": ["hsadjsahjsahdjsah879878u9"]
},
{
"subkey1": ["america.south.#"],
"subkey2": ["sjadkjsahdkjsahdj989s89d8sa98d9sa"]
}]
}
I want only subkey2 only of above mentioned document as following output:
"subkey2": ["hsadjsahjsahdjsah879878u9"]
"subkey2": ["sjadkjsahdkjsahdj989s89d8sa98d9sa"]
Now I want to fetch data with following query:-
db.collectionName.find({$or:[
{"key2.subkey1": "america.south.gas"},
{"key1.subkey1": "america.south.#"}
]},
{"_id": 0, "key2.subkey2.$": 1}
);
But it is showing me this error:
{
"waitedMS" : NumberLong(0),
"ok" : 0,
"errmsg" : "Executor error during find command: BadValue: positional operator (key1.$) requires corresponding field in query specifier",
"code" : 96
}
Any idea how can I achieve this for getting specific field with multiple query field in Mongo find operation??
This error ...
Error: error: { "waitedMS" : NumberLong(0), "ok" : 0, "errmsg" : "Executor error during find command: BadValue: positional operator (key1.$) requires corresponding field in query specifier", "code" : 96 }
.. is caused by this projection:
"key2.subkey2.$": 1
Soecifically, the positional operator: $.
According to the comments above, you ...
want only subkey2 only of above mentioned document as following output:- "subkey2": ["hsadjsahjsahdjsah879878u9"] "subkey2": ["sjadkjsahdkjsahdj989s89d8sa98d9sa"]
The following command ...
db.collection.aggregate([
{$unwind:'$key2'},
// find the documents having key2.subkey1 = 'america.south.#' or key2.subkey1 = 'america.south.gas'
{$match:{'key2.subkey1':{$in:['america.south.#','america.south.gas']}}},
// limit the output to only those key2.subkey2 entries for which the associated key2.subkey1 match the previous $match clause
{$group:{_id:0,'subkey2':{$push:'$key2.subkey2'}}},
// only return subkey2
{$project: {_id: 0, 'subkey2':1}}
])
... will return:
{
"subkey2" : [
[
"hsadjsahjsahdjsah879878u9"
],
[
"sjadkjsahdkjsahdj989s89d8sa98d9sa"
]
]
}
#glytching, If i need output like that as below given then what need to change in query and also no duplicate output value.
{
"subkey2" : [ "hsadjsahjsahdjsah879878u9","sjadkjsahdkjsahdj989s89d8sa98d9sa"]
}
I am new to aggregations with Elasticsearch and am stuck with a very simple example taken from this link.
Basically I am trying to do the Java API version of this very simple working aggregation:
http://localhost:9200/cars/transactions/_search?search_type=count
{
"aggs" : {
"colors" : {
"terms" : {
"field" : "color"
}
}
}
}
and this is the Java version I am trying to build but it returns empty buckets:
SearchResponse sr = client
.prepareSearch("cars")
.setTypes("transactions")
.setSearchType(SearchType.COUNT)
.addAggregation(AggregationBuilders.terms("colors").field("color"))
.execute()
.actionGet();
while using the Elastic DSL I get a proper response with buckets grouped by colors, but with the Java version I get an empty bucket.
Update
It turns out the code is correct, the issue I had is related to using it in a test case; when used against a running cluster it works.
I suspect your issue is not with your Java version of the request, which is just fine. I tried it on some test data I have and got the expected result. The cluster is running Elasticsearch 1.7.5.
The Java code snippet I used :
final SearchResponse sr = client
.prepareSearch("index")
.setTypes("docType")
.setSearchType(SearchType.COUNT)
.addAggregation(AggregationBuilders.terms("aggName").field("fieldName"))
.execute()
.actionGet();
System.out.println(sr);
The result I got :
{
"took" : 130,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1927227,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"aggName" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ {
"key" : "key1",
"doc_count" : 757843
}, {
"key" : "key2",
"doc_count" : 620033
}, {
"key" : "key3",
"doc_count" : 549351
} ]
}
}
}
You are doing SearchType as Count which is not correct for aggregations. Can you please remove searchtype and construct the query as below
SearchResponse sr = client
.prepareSearch("cars")
.setTypes("transactions")
.addAggregation(AggregationBuilders.terms("colors").field("color"))
.execute()
.actionGet();
I have a string as:
'"startDate" : {"\\$gte" : new Date() }'
I actually want to query my mongodb database collection for all the collections with dates after the present date.Here is a sample document:
{
"_id" : ObjectId("51e2a857adc0c2fb535f6904"),
"dateInfo" : {
"dates" : {
"startDate" : ISODate("2013-07-13T04:00:00Z"),
"endDate" : ISODate("2013-07-19T20:00:00Z")
},
"named" : "name1",
"fieldX" : "field1",
"contact" : {
"numbs" : ["+44 121 127 127", "+44 568 789 256", "+44 687 5788 9875"]
}
},
"Locality" : "locality1",
"type" : "ewhet"
}
from mongo shell, I am able to query it as follows:
db.collectionName.find({"dateInfo.dates.startDate" : {"$gte" : new Date()}})
Now I want to do it from groovy. I am trying to do it as(using mongoDB java api):
DBCursor cursor = db.collectionName.find(new JsonSlurper().parseText('{"startDate" : {"\$gte" : new Date() }}'))
Above code gives error as:
Lexing failed on line: 1, column: 48, while reading 'new ', was trying to match the constant 'null'
Now the problem with JsonSlurper is that it requires double quotes in both key and value names.But in case I put a double quote in new Date(), it wont be evaluated by mongodb.
So what can be done here?
Have you tried:
db.collectionName.find( [ 'dateInfo.dates.startDate' : ['$gte' : new Date() ] ] )