Develop Reference

Java Stream GroupBy and Reduce

I have an Item class which contains a code, quantity and amount fields, and a list of items which may contain many items (with same code). I want to group the items by code and sum up their quantities and amounts.
I was able to achieve half of it using stream's groupingBy and reduce. The grouping by worked, but the reduce is reducing all of the grouped items into one single item repeated over the different codes (groupingBy key).
Shouldn't reduce here reduce the list of items for each code from the map? Why is it retuning the same combined item for all.
Below is a sample code.
import java.util.List;
import java.util.Arrays;
import java.util.stream.Collectors;
import java.util.Map;
class HelloWorld {
public static void main(String[] args) {
List<Item> itemList = Arrays.asList(
createItem("CODE1", 1, 12),
createItem("CODE2", 4, 22),
createItem("CODE3", 5, 50),
createItem("CODE4", 2, 11),
createItem("CODE4", 8, 20),
createItem("CODE2", 1, 42)
);
Map<String, Item> aggregatedItems = itemList
.stream()
.collect(Collectors.groupingBy(
Item::getCode,
Collectors.reducing(new Item(), (aggregatedItem, item) -> {
int aggregatedQuantity = aggregatedItem.getQuantity();
double aggregatedAmount = aggregatedItem.getAmount();
aggregatedItem.setQuantity(aggregatedQuantity + item.getQuantity());
aggregatedItem.setAmount(aggregatedAmount + item.getAmount());
return aggregatedItem;
})
));
System.out.println("Map total size: " + aggregatedItems.size()); // expected 4
System.out.println();
aggregatedItems.forEach((key, value) -> {
System.out.println("key: " + key);
System.out.println("value - quantity: " + value.getQuantity() + " - amount: " + value.getAmount());
System.out.println();
});
}
private static Item createItem(String code, int quantity, double amount) {
Item item = new Item();
item.setCode(code);
item.setQuantity(quantity);
item.setAmount(amount);
return item;
}
}
class Item {
private String code;
private int quantity;
private double amount;
public Item() {
quantity = 0;
amount = 0.0;
}
public String getCode() { return code; }
public int getQuantity() { return quantity; }
public double getAmount() { return amount; }
public void setCode(String code) { this.code = code; }
public void setQuantity(int quantity) { this.quantity = quantity; }
public void setAmount(double amount) { this.amount = amount; }
}
and below is the output.
Map total size: 4
key: CODE2
value - quantity: 21 - amount: 157.0
key: CODE1
value - quantity: 21 - amount: 157.0
key: CODE4
value - quantity: 21 - amount: 157.0
key: CODE3
value - quantity: 21 - amount: 157.0

You must not modify the input arguments to Collectors.reducing. new Item() is only executed once and all your reduction operations will share the same "aggregation instance". In other words: the map will contain the same value instance 4 times (you can easily check yourself with System.identityHashCode() or by comparing for reference-equality: aggregatedItems.get("CODE1") == aggregatedItems.get("CODE2")).
Instead, return a new result instance:
final Map<String, Item> aggregatedItems = itemList
.stream()
.collect(Collectors.groupingBy(
Item::getCode,
Collectors.reducing(new Item(), (item1, item2) -> {
final Item reduced = new Item();
reduced.setQuantity(item1.getQuantity() + item2.getQuantity());
reduced.setAmount(item1.getAmount() + item2.getAmount());
return reduced;
})
));
Output:
Map total size: 4
key: CODE2
value - quantity: 5 - amount: 64.0
key: CODE1
value - quantity: 1 - amount: 12.0
key: CODE4
value - quantity: 10 - amount: 31.0
key: CODE3
value - quantity: 5 - amount: 50.0

You are using reducing, which assumes that you won't mutate the accumulator passed in. reducing won't create new Items for you for each new group, and expects you to create new Items and return them in the lambda, like this:
// this works as expected
.collect(Collectors.groupingBy(
Item::getCode,
Collectors.reducing(new Item(), (item1, item2) -> createItem(
item1.getCode(),
item1.getQuantity() + item2.getQuantity(),
item1.getAmount() + item2.getAmount()
))
));
so it is very suitable if you are using immutable objects like numbers or strings.
Since you are not creating new Items in your code, reducing keeps on reusing that same instance, resulting in the behaviour you see.
If you want to mutate the objects, you can do mutable reduction in a thread safe way with Collector.of:
.collect(Collectors.groupingBy(
Item::getCode,
Collector.of(Item::new, (aggregatedItem, item) -> {
int aggregatedQuantity = aggregatedItem.getQuantity();
double aggregatedAmount = aggregatedItem.getAmount();
aggregatedItem.setQuantity(aggregatedQuantity + item.getQuantity());
aggregatedItem.setAmount(aggregatedAmount + item.getAmount());
}, (item1, item2) -> createItem(
item1.getCode(),
item1.getQuantity() + item2.getQuantity(),
item1.getAmount() + item2.getAmount()
))
));
Notice that you now pass the reference to Item's constructor, i.e. a way to create new Items when necessary, as opposed to just a single new Item(). In addition, you also provide a third argument, the combiner, that tells the collector how to create a new item from two existing ones, which will be used if this collector is used in a concurrent situation. (See here for more info about the combiner)
This contrast between Collector.of and Collectors.reducing is the same contrast between Stream.reduce and Stream.collect. Learn more here.

Mutable reduction vs Immutable reduction
In this case, Collectors.reducing() isn't the right tool because it meant for immutable reduction, i.e. for performing folding operation in which every reduction step results in creation of a new immutable object.
But instead of generating a new object at each reduction step, you're changing the state of the object provided as an identity.
As a consequence, you're getting an incorrect result because the identity object would be created only once per thread. This single instance of the Item is used for accumulation, and reference to it end up in every value of the map.
More elaborate information you can find in the Stream API documentation, specifically in these parts: Reduction and Mutable Reduction.
And here's a short quote explaining how Stream.reduce() works (the mechanism behind Collectors.reducing() is the same):
The accumulator function takes a partial result and the next element, and produces a new partial result.
Use mutable reduction
The problem can be fixed by generating a new instance of Item while accumulating values mapped to the same key with, but a more performant approach would be to use mutable reduction instead.
For that, you can implement a custom collector created via static method Collector.of():
Map<String, Item> aggregatedItems = itemList.stream()
.collect(Collectors.groupingBy(
Item::getCode,
Collector.of(
Item::new, // mutable container of the collector
Item::merge, // accumulator - defines how stream data should be accumulated
Item::merge // combiner - mergin the two containers while executing stream in parallel
)
));
For convenience, you can introduce method merge() responsible for accumulating properties of the two items. It would allow to avoid repeating the same logic in accumulator and combiner, and keep the collector implementation lean and well-readable.
public class Item {
private String code;
private int quantity;
private double amount;
// getters, constructor, etc.
public Item merge(Item other) {
this.quantity += other.quantity;
this.amount += other.amount;
return this;
}
}

Java functional programming for multiple functionality with single stream data

There is a List of object like:-
ID Employee IN_COUNT OUT_COUNT Date
1 ABC 5 7 2020-06-11
2 ABC 12 5 2020-06-12
3 ABC 9 6 2020-06-13
This is the an employee data for three date which I get from a query in List object.
Not I want total number of IN_COUNT and OUT_COUNT between three date. This can be achieved by doing first iterating stream for only IN_COUNT and calling sum() and then in second iteration, only OUT_COUNT data can be summed. But I don't want to iterate the list two times.
How is this possible in functional programming using stream or any other option.

What you are trying to do is called a 'fold' operation in functional programming. Java streams call this 'reduce' and 'sum', 'count', etc. are just specialized reduces/folds. You just have to provide a binary accumulation function. I'm assuming Java Bean style getters and setters and an all args constructor. We just ignore the other fields of the object in our accumulation:
List<MyObj> data = fetchData();
Date d = new Date();
MyObj res = data.stream()
.reduce((a, b) -> {
return new MyObj(0, a.getEmployee(),
a.getInCount() + b.getInCount(), // Accumulate IN_COUNT
a.getOutCount() + b.getOutCount(), // Accumulate OUT_COUNT
d);
})
.orElseThrow();
This is simplified and assumes that you only have one employee in the list, but you can use standard stream operations to partition and group your stream (groupBy).
If you don't want to or can't create a MyObj, you can use a different type as accumulator. I'll use Map.entry, because Java lacks a Pair/Tuple type:
Map.Entry<Integer, Integer> res = l.stream().reduce(
Map.entry(0, 0), // Identity
(sum, x) -> Map.entry(sum.getKey() + x.getInCount(), sum.getValue() + x.getOutCount()), // accumulate
(s1, s2) -> Map.entry(s1.getKey() + s2.getKey(), s1.getValue() + s2.getValue()) // combine
);
What's happening here? We now have a reduce function of Pair accum, MyObj next -> Pair. The 'identity' is our start value, the accumulator function adds the next MyObj to the current result and the last function is only used to combine intermediate results (e.g., if done in parallel).
Too complicated? We can split the steps of extracting interesting properties and accumulating them:
Map.Entry<Integer, Integer> res = l.stream()
.map(x -> Map.entry(x.getInCount(), x.getOutCount()))
.reduce((x, y) -> Map.entry(x.getKey() + y.getKey(), x.getValue() + y.getValue()))
.orElseGet(() -> Map.entry(0, 0));

You can use reduce to done this:
public class Counts{
private int inCount;
private int outCount;
//constructor, getters, setters
}
public static void main(String[] args){
List<Counts> list = new ArrayList<>();
list.add(new Counts(5, 7));
list.add(new Counts(12, 5));
list.add(new Counts(9, 6));
Counts total = list.stream().reduce(
//it's start point, like sum = 0
//you need this if you don't want to modify objects from list
new Counts(0,0),
(sum, e) -> {
sum.setInCount( sum.getInCount() + e.getInCount() );
sum.setOutCount( sum.getOutCount() + e.getOutCount() );
return sum;
}
);
System.out.println(total.getInCount() + " - " + total.getOutCount());
}

How can I manage items with overlapping ranges, where based on a value I get the matching items

Say I have the following items (unsorted):
A, with A.amount = 10
B, with B.amount = 100
C, with C.amount = 50
D, with D.amount = 50
Now for every unique amount boundary AB in items, find the items whose range include the value and calculate cumulative bracket. So:
AB=10 results in { A, B, C, D } -> cumulative bracket 210
AB=50 results in { B, C, D } -> cumulative bracket 200
AB=100 results in { B } -> cumulative bracket 100
It would be used like so:
for (int AB : collectAmountBoundaries(items)) {
Collection<Item> itemsInBracket = findItemsForAB(items, AB);
// execute logic, calculations etc with cumulative bracket value for AB
}
Now I can code all this using vanilla Java, by first manually transforming the collection of items into a map of AB→cumulativeBracketValue or something. However, since I'm working with ranges and overlap-logic I feel somehow a clean solution involving NavigableMap, Range logic or something clever should be possible (it feels like a common pattern). Or perhaps using streams to do a collect groupingBy?
I'm not seeing it right now. Any ideas on how to tackle this cleanly?

I think, doing a simple filter and then adding the filtered result to a List and amount to a total is sufficient.
static ListAndCumalativeAmount getCR(List<Item> items, double amount) {
ListAndCumalativeAmount result = new ListAndCumalativeAmount();
items.stream().filter(item -> item.amount >= amount).forEach((i) -> {
result.getItems().add(i.name);
result.add(i.amount);
});
return result;
}
static class ListAndCumalativeAmount {
private List<String> items = new ArrayList<>();
private Double amount = new Double(0.0);
public List<String> getItems() {
return items;
}
public void add(double value) {
amount = amount + value;
}
public Double getAmount() {
return amount;
}
}

This is a way to do it with streams and groupingBy:
Map<Integer, SimpleEntry<List<Item>, Double>> groupedByBracketBoundary = items.stream()
.collect(groupingBy(o -> o.getAmount())).entrySet().stream()
// map map-values to map-entries of original value and sum, keeping key the same
.collect(toMap(Entry::getKey, e -> new SimpleEntry<>(e.getValue(),
e.getValue().stream()
.map(o -> o.getAmount())
.reduce(0d, (amount1, amount2) -> amount1 + amount2))));
LinkedHashSet<Integer> sortedUniqueAmountBoundaries = internalList.stream()
.map(o -> o.getAmount())
.sorted()
.collect(Collectors.toCollection(LinkedHashSet::new));
for (int ab : sortedUniqueAmountBoundaries) {
List<Item> itemsInBracket = groupedByBracketBoundary.get(ab).getKey();
double cumulativeAmountForBracket = groupedByBracketBoundary.get(ab).getValue();
// execute logic, calculations etc with cumulative bracket value for AB
}
Somehow this feels succinct and verbose at the same time, it's rather dense. Isn't there a JDK api or 3rd party library that does this kind of thing?

Java 8 lambda sum, count and group by

Select sum(paidAmount), count(paidAmount), classificationName,
From tableA
Group by classificationName;
How can i do this in Java 8 using streams and collectors?
Java8:
lineItemList.stream()
.collect(Collectors.groupingBy(Bucket::getBucketName,
Collectors.reducing(BigDecimal.ZERO,
Bucket::getPaidAmount,
BigDecimal::add)))
This gives me sum and group by. But how can I also get count on the group name ?
Expectation is :
100, 2, classname1
50, 1, classname2
150, 3, classname3

Using an extended version of the Statistics class of this answer,
class Statistics {
int count;
BigDecimal sum;
Statistics(Bucket bucket) {
count = 1;
sum = bucket.getPaidAmount();
}
Statistics() {
count = 0;
sum = BigDecimal.ZERO;
}
void add(Bucket b) {
count++;
sum = sum.add(b.getPaidAmount());
}
Statistics merge(Statistics another) {
count += another.count;
sum = sum.add(another.sum);
return this;
}
}
you can use it in a Stream operation like
Map<String, Statistics> map = lineItemList.stream()
.collect(Collectors.groupingBy(Bucket::getBucketName,
Collector.of(Statistics::new, Statistics::add, Statistics::merge)));
this may have a small performance advantage, as it only creates one Statistics instance per group for a sequential evaluation. It even supports parallel evaluation, but you’d need a very large list with sufficiently large groups to get a benefit from parallel evaluation.
For a sequential evaluation, the operation is equivalent to
lineItemList.forEach(b ->
map.computeIfAbsent(b.getBucketName(), x -> new Statistics()).add(b));
whereas merging partial results after a parallel evaluation works closer to the example already given in the linked answer, i.e.
secondMap.forEach((key, value) -> firstMap.merge(key, value, Statistics::merge));

As you're using BigDecimal for the amounts (which is the correct approach, IMO), you can't make use of Collectors.summarizingDouble, which summarizes count, sum, average, min and max in one pass.
Alexis C. has already shown in his answer one way to do it with streams. Another way would be to write your own collector, as shown in Holger's answer.
Here I'll show another way. First let's create a container class with a helper method. Then, instead of using streams, I'll use common Map operations.
class Statistics {
int count;
BigDecimal sum;
Statistics(Bucket bucket) {
count = 1;
sum = bucket.getPaidAmount();
}
Statistics merge(Statistics another) {
count += another.count;
sum = sum.add(another.sum);
return this;
}
}
Now, you can make the grouping as follows:
Map<String, Statistics> result = new HashMap<>();
lineItemList.forEach(b ->
result.merge(b.getBucketName(), new Statistics(b), Statistics::merge));
This works by using the Map.merge method, whose docs say:
If the specified key is not already associated with a value or is associated with null, associates it with the given non-null value. Otherwise, replaces the associated value with the results of the given remapping function

You could reduce pairs where the keys would hold the sum and the values would hold the count:
Map<String, SimpleEntry<BigDecimal, Long>> map =
lineItemList.stream()
.collect(groupingBy(Bucket::getBucketName,
reducing(new SimpleEntry<>(BigDecimal.ZERO, 0L),
b -> new SimpleEntry<>(b.getPaidAmount(), 1L),
(v1, v2) -> new SimpleEntry<>(v1.getKey().add(v2.getKey()), v1.getValue() + v2.getValue()))));
although Collectors.toMap looks cleaner:
Map<String, SimpleEntry<BigDecimal, Long>> map =
lineItemList.stream()
.collect(toMap(Bucket::getBucketName,
b -> new SimpleEntry<>(b.getPaidAmount(), 1L),
(v1, v2) -> new SimpleEntry<>(v1.getKey().add(v2.getKey()), v1.getValue() + v2.getValue())));

Java8 Lambda Expressions Evaluation

List <Person> roster = new List<Person>();
Integer totalAgeReduce = roster
.stream()
.map(Person::getAge)
.reduce(
0,
(a, b) -> a + b);
Can anyone help me understand the above code snippet. My understanding is that the stream method will first iterate through the entire roster List and while it is iterating it will create a new List of the mapped objects with every person's age in it. Then it will finally call the reduce after the mapping is done (the reduce is only called at the end after mapping correct?). And in the reduce it starts of at 0, and in the first iteration of reduce on the newly mapped list a = 0 and b is equal to the first element in the List that was created from the mapping function. Then it will continue and add all the elements from the mapped list and return to you an integer with the sum of all the ages.

Each item in the stream will each be sent through all the steps one at a time. Here's some test code to help you see whats happening:
List<String> test = Arrays.asList("A","B");
System.out.println("END: " + test.stream()
.map(s -> {System.out.println("1 " + s); return s; })
.map(s -> {System.out.println("2 " + s); return s; })
.reduce("", (acc, s) -> {System.out.println("3 " + s); return acc + s; })
);
Output
1 A
2 A
3 A
1 B
2 B
3 B
END: AB

TL;DR
It sums all the ages from the Person's within the List.
stream() : Creates a stream from the Collection (List)
map() : Will make a mapping from the received object to another object (here from Person to Integer (getAge returns an Integer))
reduce(0,(a, b) -> a + b) : reduce is a reduction (it reduces all the objects received into one (here the action is to add them all together, a big addition). It takes the identity (first value to begin with) as first argument and the following lambda expression (BinaryOperator<Integer> or BiFunction<Integer, Integer, Integer>) presents the logic to apply for the reduction.
Example
List<Person> persons = Arrays.asList(new Person("John", 20),
new Person("Mike", 40),
new Person("Wayne", 30));
Integer totalAgeReduce = roster.stream()
.map(Person::getAge)
.reduce(0,(a, b) -> a + b);
System.out.println(totalAgeReduce); // 90

The thing is
(a, b) -> a + b);
is an accumulator, and if you look at it like a recursive function, it will be passing the result of the sum, for every element in the stream, as Andreas Point out is not a list, is a pipeline.
Just to point out lambda expressions is just passing an Argument which in fact is a function.

If you would use loops it would look like this:
List<Integer> ages = new ArrayList<>();
for (Person p : roster) {
ages.add(p.getAge());
}
int sum = 0;
for (Integer age : ages) {
sum += age;
}