data in mongo :
enter image description here
db.test2.aggregate([
{
"$project" : {
"contents" : 1,
"comments" : {
"$filter" : {
"input" : "$comments",
"as" : "item",
"cond" : {"$gt" : ['$$item.score', 2]}
},
},
"comments2" : {
"$filter" : {
"input" : "$comments2",
"as" : "item",
"cond" : {"$gt" : ["$$item.score", 5]}
}
}
}
},
{
"$project" : {
"content" : 1,
"commentsTotal" : {
"$reduce" : {
"input" : "$comments",
"initialValue" : 0,
"in" : {"$add" : ["$$value", "$$this.score"]}
}
},
"comments2Total" : {
"$reduce" : {
"input" : "$comments2",
"initialValue" : 0,
"in" : {"$add" : ["$$value", "$$this.score"]}
}
}
}
},
{$skip : 0},
{$limit: 3}
]);
<!-- language: lang-json-->
So you can see, this does the following :
1、filter the comments and comments2 which score is gt 5.
2、count total of the socre in comment array.
and i write the aggregation query in Spring like this:
AggregationExpression reduce = ArithmeticOperators.Add.valueOf("$$value").add("$$this.socre");
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.project().andExclude("_id")
.andInclude("content")
.and("comments").filter("item", ComparisonOperators.Gt.valueOf("item.score").greaterThanValue(3)).as("comments")
.and("comments2").filter("item", ComparisonOperators.Gt.valueOf("item.score").greaterThanValue(3)).as("comments2"),
Aggregation.project("comments", "comments2")
.and(ArrayOperators.Reduce.arrayOf("comments").withInitialValue("0").reduce(reduce)).as("commentsTotal")
);
when i run like up , it will throws exception :
java.lang.IllegalArgumentException: Invalid reference '$$value'!
You can try below aggregation by wrapping $filter inside the $reduce operation.
Something like below
AggregationExpression reduce1 = new AggregationExpression() {
#Override
public DBObject toDbObject(AggregationOperationContext aggregationOperationContext) {
DBObject filter = new BasicDBObject("$filter", new BasicDBObject("input", "$comments").append("as", "item").append("cond",
new BasicDBObject("$gt", Arrays.<Object>asList("$$item.score", 2))));
DBObject reduce = new BasicDBObject("input", filter).append("initialValue", 0).append("in", new BasicDBObject("$add", Arrays.asList("$$value", "$$this.socre")));
return new BasicDBObject("$reduce", reduce);
}
};
Aggregation aggregation = newAggregation(
Aggregation.project().andExclude("_id")
.andInclude("content")
.and(reduce1).as("commentsTotal")
);
This is an old question, but in case some one winds up here like me, here's how I was able to solve it.
You cannot access "$$this" and "$$value" variables directly like this in spring.
AggregationExpression reduce = ArithmeticOperators.Add.valueOf("$$value").add("$$this.socre");
To do this we have to use reduce variable enum, like this:
AggregationExpression reduce = ArithmeticOperators.Add.valueOf(ArrayOperators.Reduce.Variable.VALUE.getTarget()).add(ArrayOperators.Reduce.Variable.THIS.referringTo("score").getTarget());
Hope this helps!
I had to solve next task and hadn't find any solutions. So i hope my answer will help somebody.
User with roles (user have list of rights + list of roles, each role have own list of rights, needed to find full list of rights):
user structure
role structure
First, i lookup roles to roleDto (for example), then i collect rights from roles to 1 list:
ArrayOperators.Reduce reduce = ArrayOperators.Reduce.arrayOf("$roleDto.rights")
.withInitialValue(new ArrayList<>())
.reduce(ArrayOperators.ConcatArrays.arrayOf("$$value").concat("$$this"));
As result in reduce i have this 1 list of rights collected from roles.
After that i make:
SetOperators.SetUnion.arrayAsSet(reduce).union("$rights")
using previous result. Result type is AggregationExpression because AbstractAggregationExpression implements AggregationExpression.
So, finally i get smth like this (sorry for messy code):
private static AggregationExpression getAllRightsForUser() {
// concat rights from list of roles (each role have list of rights) - list of list to list
ArrayOperators.Reduce reduce = ArrayOperators.Reduce.arrayOf("$roleDto.rights")
.withInitialValue(new ArrayList<>())
.reduce(ArrayOperators.ConcatArrays.arrayOf("$$value").concat("$$this"));
// union result with user.rights
return SetOperators.SetUnion.arrayAsSet(reduce).union("$rights");
}
Result of this operation can be finally used somewhere like here ;) :
public static AggregationOperation addFieldOperation(AggregationExpression aggregationExpression, String fieldName) {
return aoc -> new Document("$addFields", new Document(fieldName, aggregationExpression.toDocument(aoc)));
}
I had the same issue, one of the solutions is to create a custom Reduce function, here's Union example:
public class SetUnionReduceExpression implements AggregationExpression {
#Override
public Document toDocument(AggregationOperationContext context) {
return new Document("$setUnion", ImmutableList.of("$$value", "$$this"));
}
}
Related
I want to check if a specific value of a key exists in the Realtime Database or not and perform some action based on it.
I have the following data:
"Orders" : {
"03403426747" : {
"17" : {
"State" : "(4) Canceled",
"address" : "yubt",
"date" : "Feb 28, 2022",
"discount" : "0",
"name" : "uk",
"phone" : "0311111111",
"time" : "15:33:58 PM",
"totalAmount" : "3778"
},
"18" : {
"State" : "(1) Approved",
"address" : "yubt",
"date" : "Feb 28, 2022",
"discount" : "120",
"name" : "uk",
"phone" : "03111111111",
"time" : "16:01:58 PM",
"totalAmount" : "7703"
}
}
}
I want to check If any order from these has has "State" value other than "(3) Completed" & "(4) Canceled".
if anyone's order has a value other than these, I want to remove that user from the list which contains users with pending orders.
and if at any time that user has new order or older order State changed I want to again add that user to the list.
I want to check If any order from these has has "State" value other than "(3) Completed" & "(4) Canceled".
There is no way you can query the Realtime Database using a negation. What you can do instead is to create a separate query for each "other" state and join the results on the client.
However, if you consider at some point in time to try using Cloud Firestore, then you should consider using not equal (!=) query or if it fits your needs the not-in query.
After a long time, my brain light burned and I came up with a solution.
Create a data class for OrderState
public class OrderState {
public static int ordersCount = 0;
public static boolean state = false;
public static void update(boolean state){
if (state){
ordersCount = ordersCount + 1;
OrderState.state = true;
}else
if (!state && ordersCount > 0){
ordersCount = ordersCount - 1;
if (ordersCount < 1) OrderState.state = false;
}
}
public static void reset(){
ordersCount = 0;
state = false;
}
}
On FirebaseRecyclerAdapter -> onBindViewHolder
//if order not canceled or completed. it will update OrderState,
//ordersCount + 1 and state to true
if (adminOrders.getState().equals(Prevalent.orderStateNew) ||
adminOrders.getState().equals(Prevalent.orderStateApproved) ||
adminOrders.getState().equals(Prevalent.orderStateShipped) ||
adminOrders.getState().equals(Prevalent.orderStateApproved)){
OrderState.update(true);
}
changeUserWithOrderState();
On changing state of order by admin
//if the order is not already cancelled or completed, reduce one order from OrderState as it will be readded automatically upon Recycler refresh.
if (!adminOrders.getState().equals(Prevalent.orderStateCanceled) &&
!adminOrders.getState().equals(Prevalent.orderStateCompleted)) OrderState.update(false);
Al last if the user does not has any order with states New, Approved, and Shipped
OrderState.orderCount = 0;
OrderState.state = false;
and upon updating the database it will set the state to false.
private void changeUserWithOrderState() {
DatabaseReference userWithOrder = FirebaseDatabase.getInstance().getReference()
.child(Prevalent.usersWithOrders)
.child(userPhoneKey);
HashMap<String, Object> map = new HashMap<>();
map.put(Prevalent.orderState, String.valueOf(OrderState.state));
userWithOrder.updateChildren(map).addOnCompleteListener(task -> {
//Changed state based upon OrderState.state value...
});
}
We have a collection of scrips :
{
"_id" : ObjectId("xxxxxxx"),
"scrip" : "3647"
}
{
"_id" : ObjectId("yyyyyy"),
"scrip" : "5647"
}
...
We are simply attempting to return the scrip numerals as an array of string using java driver 3.7
ArrayList<Document> scriplist = scrips.aggregate(Arrays.asList(
Aggregates.group(
Accumulators.push("scripids",
new Document("_id", "$id").
append("scripids", "$scripid"))
)
)).into(new ArrayList<>());
System.out.println(scriplist.toString());
Expected output is ['3647','5647'].
However,we get a 'Can't find a codec for class com.mongodb.client.model.BsonField.' exception.
How is this to be done?
The following query can get us the expected output:
db.scrips.distinct("scrip");
Output:
["3647","5647"]
Equivalent code in Java:
DistinctIterable<String> iterable = scrips.distinct("scrip", String.class);
List<String> scrips = new ArrayList<>();
Block<String> block = scrip -> scrips.add(scrip);
iterable.forEach(block);
The 'scrips' set would hold the distinct scrips.
Some other ways to do the same:
db.scrips.aggregate([
{
$group:{
"_id":"$scrip"
}
},
{
$group:{
"_id":null,
"scrips":{
$push:"$_id"
}
}
},
{
$project:{
"_id":0
}
}
])
Java code:
scrips.aggregate(
Arrays.asList(Aggregates.group("$scrip"), Aggregates.group(null, Accumulators.push("scrips", "$_id")),
Aggregates.project(Projections.exclude("_id"))));
db.scrips.aggregate([
{
$group:{
"_id":null,
"scrips":{
$addToSet:"$scrip"
}
}
},
{
$project:{
"_id":0
}
}
])
Java code:
scrips.aggregate(Arrays.asList(Aggregates.group(null, Accumulators.addToSet("scrips", "$_id")),
Aggregates.project(Projections.exclude("_id"))));
I had tried a few example codes on suggester feature of ElasticSearch on the net but I couldn't solve my problem against the autocomplete solution
my index:
client.prepareIndex("kodcucom", "article", "1")
.setSource(putJsonDocument("ElasticSearch: Java",
"ElasticSeach provides Java API, thus it executes all operations " +
"asynchronously by using client object..",
new Date(),
new String[]{"elasticsearch"},
"Hüseyin Akdoğan")).execute().actionGet();
and I used suggestbuilder to obtain the keyword then scan through the content "field", and here is where the null pointer exception occurs due to no result
CompletionSuggestionBuilder skillNameSuggest = new CompletionSuggestionBuilder("skillNameSuggest");
skillNameSuggest.text("lien");
skillNameSuggest.field("content");
SuggestRequestBuilder suggestRequestBuilder = client.prepareSuggest("kodcucom").addSuggestion(skillNameSuggest);
SuggestResponse suggestResponse = suggestRequestBuilder.execute().actionGet();
Iterator<? extends Suggest.Suggestion.Entry.Option> iterator =
suggestResponse.getSuggest().getSuggestion("skillNameSuggest").iterator().next().getOptions().iterator();
Am I missing some filters or input criteria in order to get result? Any result should ok such as autocomplete or record found.
EDIT 1:
This is where I got the NPE and I could see that none of any result return at suggestResponse from debug mode
Iterator<? extends Suggest.Suggestion.Entry.Option> iterator =
suggestResponse.getSuggest().getSuggestion("skillNameSuggest").iterator().next().getOptions().iterator();
EDIT 2:
I am using 2.1.1 version of ElasticSearch Java API
EDIT 3:
I tried in splitting up the iterator line into several code blocks, the NPE occur at the last line when converting a set of data into iterator, but there is not much helping
Suggest tempSuggest = suggestResponse.getSuggest();
Suggestion tempSuggestion = tempSuggest.getSuggestion("skillNameSuggest");
Iterator tempIterator = tempSuggestion.iterator();
I see that the codes:
SuggestRequestBuilder suggestRequestBuilder = client.prepareSuggest("kodcucom").addSuggestion(skillNameSuggest);
SuggestResponse suggestResponse = suggestRequestBuilder.execute().actionGet();
has already consists a empty array/dataset, am I using the suggest request builder incorrectly?
In order to use completion feature, you need to dedicate one field, which will be called completion and you have to specify a special mapping for it.
For example:
"mappings": {
"article": {
"properties": {
"content": {
"type": "string"
},
"completion_suggest": {
"type": "completion"}
}
}
}
The completion_suggest field is the field we will use for the autocomplete function in the above code sample. After this mapping defination, the data must be indexing as follow:
curl -XPOST localhost:9200/kodcucom/article/1 -d '{
"content": "elasticsearch",
"completion_suggest": {
"input": [ "es", "elastic", "elasticsearch" ],
"output": "ElasticSearch"
}
}'
Then Java API can be used as follows for get suggestions:
CompletionSuggestionBuilder skillNameSuggest = new CompletionSuggestionBuilder("complete");
skillNameSuggest.text("es");
skillNameSuggest.field("completion_suggest");
SearchResponse searchResponse = client.prepareSearch("kodcucom")
.setTypes("article")
.setQuery(QueryBuilders.matchAllQuery())
.addSuggestion(skillNameSuggest)
.execute().actionGet();
CompletionSuggestion compSuggestion = searchResponse.getSuggest().getSuggestion("complete");
List<CompletionSuggestion.Entry> entryList = compSuggestion.getEntries();
if(entryList != null) {
CompletionSuggestion.Entry entry = entryList.get(0);
List<CompletionSuggestion.Entry.Option> options =entry.getOptions();
if(options != null) {
CompletionSuggestion.Entry.Option option = options.get(0);
System.out.println(option.getText().string());
}
}
Following link provides you the details of how to create a suggester index. https://www.elastic.co/blog/you-complete-me
Now, I use asynchronous Suggestionbuilder Java API to generate suggestions based on terms.
SearchRequestBuilder suggestionsExtractor = elasticsearchService.suggestionsExtractor("yourIndexName", "yourIndexType//not necessary", "name_suggest", term);
System.out.println(suggestionsExtractor);
Map<String,Object> suggestionMap = new HashMap<>();
suggestionsExtractor.execute(new ActionListener<SearchResponse>() {
#Override
public void onResponse(SearchResponse searchResponse) {
if(searchResponse.status().equals(RestStatus.OK)) {
searchResponse.getSuggest().getSuggestion("productsearch").getEntries().forEach(e -> {
e.getOptions().forEach(s -> {
ArrayList<Object> contents = new ArrayList<>();
suggestionMap.put(s.getText().string(), s.getScore());
});
});
}
}
#Override
public void onFailure(Exception e) {
Helper.sendErrorResponse(routingContext,new JsonObject().put("details","internal server error"));
e.printStackTrace();
}
});
Following is how suggestionbuilder is created.
public SearchRequestBuilder suggestionsExtractor(String indexName, String typeName, String field, String term) {
CompletionSuggestionBuilder csb = SuggestBuilders.completionSuggestion(field).text(term);
SearchRequestBuilder suggestBuilder = client.prepareSearch()
.suggest(new SuggestBuilder().addSuggestion(indexName, csb));
return suggestBuilder;
}
I have a collection in mongodb - "text_failed" which has all the numbers on which I failed to send an SMS, the time they failed and some other information.
A document in this collection looks like this:
{
_id(ObjectId): xxxxxx2af8....
failTime(String): 2015-05-15 01:15:48
telNum(String): 95634xxxxx
//some other information
}
I need to fetch the top 500 numbers which failed the most in a month's duration. A number can occur any number of time during this month.(Eg: a number failed 143 times, other 46 etc.)
The problem I have is that during this duration the numbers failed crossed 7M. It's difficult to process this much information using the following code which doesn't use aggregation:
DBCollection collection = mongoDB.getCollection("text_failed");
BasicDBObject query = new BasicDBObject();
query.put("failTime", new BasicDBObject("$gt", "2015-05-15 00:00:00").append("$lt", "2015-06-15 00:00:00"));
BasicDBObject field = new BasicDBObject();
field.put("telNum", 1);
DBCursor cursor = collection.find(query, field);
HashMap<String, Integer> hm = new HashMap<String, Integer>();
//int count = 1;
System.out.println(cursor);
while(cursor.hasNext()) {
//System.out.println(count);
//count++;
DBObject object = cursor.next();
if(hm.containsKey(object.get("telNum").toString())) {
hm.put(object.get("telNum").toString(), hm.get(object.get("telNum").toString()) + 1);
}
else {
hm.put(object.get("telNum").toString(), 1);
}
}
This fetches 7M+ documents for me. I need only the top 500 numbers. The result should look something like this:
{
telNum: xxxxx54654 //the number which failed
count: 129 //number of times it failed
}
I used aggregation myself but didn't get the desired results. Can this be accomplished by aggregation? Or is there any other way more efficient in which I can do this?
You could try the following aggregation pipeline:
db.getCollection("text_failed").aggregate([
{
"$match": {
"failTime": { "$gt": "2015-05-01 00:00:00", "$lt": "2015-06-01 00:00:00" }
}
},
{
"$group": {
"_id": "$telNum",
"count": { "$sum": 1 }
}
},
{
"$sort": { "count": -1 }
},
{
"$limit": 500
}
])
I have 300,000 documents in this specific collection.
Each document is considered as one taxi trip.
Each document contains a TaxiStation number and a License number.
My goal is to figure out the number of trips per TaxiLicense per TaxiStation.
For example:
TaxiStation A License X had 5 trips.
TaxiStation A License Y had 9 trips. And so on.
How can I optimize my query? It is takes an upwards time of 30 minutes to complete!
List /*of*/ taxistationOfCollection, taxiLicenseOfTaxistation;
//Here I get all the distinct TaxiStation numbers in the collection
taxistationOfCollection = coll.distinct("TaxiStation");
BasicDBObject query, tripquery;
int tripcount;
//Now I have to loop through each Taxi Station
for(int i = 0; i<taxistationOfCollection.size(); i++)
{
query = new BasicDBObject("TaxiStation", taxistationOfCollection.get(i));
//Here, I make a list of each distinct Taxi License in the current Taxi station
taxiLicenseOfTaxistation = coll.distinct("TaxiLicense", query);
//Now I make a loop to process each Taxi License within the current Taxi station
for(int k = 0; k<taxiLicenseOfTaxistation.size();k++)
{
tripcount=0;
if(taxiLicenseOfTaxistation.get(k) !=null)
{
//I'm looking for each Taxi Station with this Taxi License
tripquery= new BasicDBObject("TaxiStation", taxistationOfCollection.get(i)).append("TaxiLicense", taxiLicenseOfTaxistation.get(k));
DBCursor cursor = coll.find(tripquery);
try {
while(cursor.hasNext()) {
//Increasing my counter everytime I find a match
tripcount++;
cursor.next();
}
} finally {
//Finally printing the results
System.out.println("Station: " + taxistationOfCollection.get(i) + " License:" + taxiLicenseOfTaxistation.get(k)
+ " Trips: " + tripcount);
}
}
}
}
Sample Document :
{
"_id" : ObjectId("53df46ed9b2ed78fb7ca4f23"),
"Version" : "2",
"Display" : [],
"Generated" : "2014-08-04,16:40:05",
"GetOff" : "2014-08-04,16:40:05",
"GetOffCellInfo" : "46001,43027,11237298",
"Undisplay" : [],
"TaxiStation" : "0000",
"GetOn" : "2014-08-04,16:40:03",
"GetOnCellInfo" : "46001,43027,11237298",
"TaxiLicense" : "000000",
"TUID" : "26921876-3bd5-432e-a014-df0fb26c0e6c",
"IMSI" : "460018571356892",
"MCU" : "CM8001MA121225V1",
"System_ID" : "000",
"MeterGetOffTime" : "",
"MeterGetOnTime" : "",
"Setup" : [],
"MeterSID" : "",
"MeterWaitTime" : "",
"OS" : "4.2",
"PackageVersion" : "201407300888",
"PublishVersion" : "201312060943",
"SWVersion" : "rel_touchbox_20101010",
"MeterMile" : 0,
"MeterCharged" : 0,
"GetOnLongitude" : 0,
"GetOnLatitude" : 0,
"GetOffLongitude" : 0,
"TripLength" : 2,
"GetOffLatitude" : 0,
"Clicks" : 0,
"updateTime" : "2014-08-04 16:40:10"
}
Aggregation is probably what you are looking for. With an aggregation operation your whole code runs on the database and can be performed in a few lines. Performance should also be a lot better since the database handles everything that needs to be done an can take full advantage of indexes and other stuff.
From what you postet this boils down to a simple $group operation. In the shell this would look like:
db.taxistationOfCollection.aggregate([
{$group:
{ _id:
{station: "$TaxiStation",
licence: "$TaxiLicense"},
count : {$sum : 1}
}
])
This will give you documents of the form
{_id : {station: stationid, licence: licence_number}, count: number_of_documents}
For Java it would look like this:
DBObject taxigroup = new BasicDBObject("$group",
new BasicDBObject("_id",
new BasicDBObject("station","$TaxiStation")
.append("Licence","$TaxiLicense"))
.append("count", new BasicDBObject("$sum",1)));
AggregationOutput aggout = taxistationOfCollection.aggregate(
Arrays.asList(taxigroup));
Please note that the code snippets are not tested.