Related
I have a stream of data as shown below and I wish to collect the data based on a condition.
Stream of data:
452857;0;L100;csO;20220411;20220411;EUR;000101435;+; ;F;1;EUR;000100000;+;
452857;0;L120;csO;20220411;20220411;EUR;000101435;+; ;F;1;EUR;000100000;+;
452857;0;L121;csO;20220411;20220411;EUR;000101435;+; ;F;1;EUR;000100000;+;
452857;0;L126;csO;20220411;20220411;EUR;000101435;+; ;F;1;EUR;000100000;+;
452857;0;L100;csO;20220411;20220411;EUR;000101435;+; ;F;1;EUR;000100000;+;
452857;0;L122;csO;20220411;20220411;EUR;000101435;+; ;F;1;EUR;000100000;+;
I wish to collect the data based on the index = 2 (L100,L121 ...) and store it in different lists of L120,L121,L122 etc using Java 8 streams. Any suggestions?
Note: splittedLine array below is my stream of data.
For instance: I have tried the following but I think there's a shorter way:
List<String> L100_ENTITY_NAMES = Arrays.asList("L100", "L120", "L121", "L122", "L126");
List<List<String>> list= L100_ENTITY_NAMES.stream()
.map(entity -> Arrays.stream(splittedLine)
.filter(line -> {
String[] values = line.split(String.valueOf(DELIMITER));
if(values.length > 0){
return entity.equals(values[2]);
}
else{
return false;
}
}).collect(Collectors.toList())).collect(Collectors.toList());
I'd rather change the order and also collect the data into a Map<String, List<String>> where the key would be the entity name.
Assuming splittedLine is the array of lines, I'd probably do something like this:
Set<String> L100_ENTITY_NAMES = Set.of("L100", ...);
String delimiter = String.valueOf(DELIMITER);
Map<String, List<String>> result =
Arrays.stream(splittedLine)
.map(line -> {
String[] values = line.split(delimiter );
if( values.length < 3) {
return null;
}
return new AbstractMap.SimpleEntry<>(values[2], line);
})
.filter(Objects::nonNull)
.filter(tempLine -> L100_ENTITY_NAMES.contains(tempLine.getEntityName()))
.collect(Collectors.groupingBy(Map.Entry::getKey,
Collectors.mapping(Map.Entry::getValue, Collectors.toList());
Note that this isn't necessarily shorter but has a couple of other advantages:
It's not O(n*m) but rather O(n * log(m)), so it should be faster for non-trivial stream sizes
You get an entity name for each list rather than having to rely on the indices in both lists
It's easier to understand because you use distinct steps:
split and map the line
filter null values, i.e. lines that aren't valid in the first place
filter lines that don't have any of the L100 entity names
collect the filtered lines by entity name so you can easily access the sub lists
I would convert the semicolon-delimited lines to objects as soon as possible, instead of keeping them around as a serialized bunch of data.
First, I would create a model modelling our data:
public record LBasedEntity(long id, int zero, String lcode, …) { }
Then, create a method to parse the line. This can be as well an external parsing library, for this looks like CSV with semicolon as delimiter.
private static LBasedEntity parse(String line) {
String[] parts = line.split(";");
if (parts.length < 3) {
return null;
}
long id = Long.parseLong(parts[0]);
int zero = Integer.parseInt(parts[1]);
String lcode = parts[2];
…
return new LBasedEntity(id, zero, lcode, …);
}
Then the mapping is trivial:
Map<String, List<LBasedEntity>> result = Arrays.stream(lines)
.map(line -> parse(line))
.filter(Objects::nonNull)
.filter(lBasedEntity -> L100_ENTITY_NAMES.contains(lBasedEntity.lcode()))
.collect(Collectors.groupingBy(LBasedEntity::lcode));
map(line -> parse(line)) parses the line into an LBasedEntity object (or whatever you call it);
filter(Objects::nonNull) filters out all null values produced by the parse method;
The next filter selects all entities of which the lcode property is contained in the L100_ENTITY_NAMES list (I would turn this into a Set, to speed things up);
Then a Map is with key-value pairs of L100_ENTITY_NAME → List<LBasedEntity>.
You're effectively asking for what languages like Scala provide on collections: groupBy. In Scala you could write:
splitLines.groupBy(_(2)) // Map[String, List[String]]
Of course, you want this in Java, and in my opinion, not using streams here makes sense due to Java's lack of a fold or groupBy function.
HashMap<String, ArrayList<String>> map = new HashMap<>();
for (String[] line : splitLines) {
if (line.length < 2) continue;
ArrayList<String> xs = map.getOrDefault(line[2], new ArrayList<>());
xs.addAll(Arrays.asList(line));
map.put(line[2], xs);
}
As you can see, it's very easy to understand, and actually shorter than the stream based solution.
I'm leveraging two key methods on a HashMap.
The first is getOrDefault; basically if the value associate with our key doesn't exist, we can provide a default. In our case, an empty ArrayList.
The second is put, which actually acts like a putOrReplace because it lets us override the previous value associated with the key.
I hope that was helpful. :)
you're asking for a shorter way to achieve the same, actually your code is good. I guess the only part that makes it look lengthy is the if/else check in the stream.
if (values.length > 0) {
return entity.equals(values[2]);
} else {
return false;
}
I would suggest introduce two tiny private methods to improve the readability, like this:
List<List<String>> list = L100_ENTITY_NAMES.stream()
.map(entity -> getLinesByEntity(splittedLine, entity)).collect(Collectors.toList());
private List<String> getLinesByEntity(String[] splittedLine, String entity) {
return Arrays.stream(splittedLine).filter(line -> isLineMatched(entity, line)).collect(Collectors.toList());
}
private boolean isLineMatched(String entity, String line) {
String[] values = line.split(DELIMITER);
return values.length > 0 && entity.equals(values[2]);
}
My question is: is there a better way to implement this task?
I have a list of orderable elements (in this example by age, the youngest first).
And I want to delete all elements that fulfill a condition (in this example red elements) but keep the first 2 of them.
Stream<ElementsVO> stream = allElements.stream();
Stream<ElementsVO> redStream = stream.filter(elem->elem.getColor()==RED).sorted((c1, c2) -> { return c1.getAge() - c2.getAge();
}).limit(2);
Stream<ElementsVO> nonRedStream=stream.filter(elem->elem.getColor()!=RED);
List<ElementsVO> resultList = Stream.concat(redStream,nonRedStream).sorted((c1, c2) -> { return c1.getAge() - c2.getAge();
}).collect(Collectors.toList());
Any idea to improve this? Any way to implement an accumulator function or something like that with streams?
You can technically do this with a stateful predicate:
Predicate<ElementsV0> statefulPredicate = new Predicate<ElementsV0>() {
private int reds = 0;
#Override public boolean test(ElementsV0 e) {
if (elem.getColor() == RED) {
reds++;
return reds < 2;
}
return true;
}
};
Then:
List<ElementsVO> resultList =
allElements.stream()
.sorted(comparingInt(ElementsV0::getAge))
.filter(statefulPredicate)
.collect(toList());
This might work, but it is a violation of the Stream API: the documentation for Stream.filter says that the predicate should be stateless, which in general allows the stream implementation to apply the filter in any order. For small input lists, streamed sequentially, this will almost certainly be the appearance order in the list, but it's not guaranteed.
Caveat emptor. Your current way works, although you could do the partitioning of the list more efficiently using Collectors.partitioningBy to avoid iterating it twice.
You can implement a custom collector that will maintain two separate collections of RED and non-RED element.
And since you need only two red elements having the greatest age to improve performance, you can introduce a partial sorting. I.e. collection of non-red element needs to maintain an order and always must be of size 2 at most, with that overhead of sorting will be far less significant in comparison to sorting of elements having the property of RED in order to pick only two of them.
In order to create a custom collector, you might make use of the static method Collector.of() which expects the following arguments:
Supplier Supplier<A> is meant to provide a mutable container which store elements of the stream. Because we need to separate elements by color into two groups as a container, we can use a map that will contain only 2 keys (true and false), denoting whether elements mapped to this key are red. In order to store red-elements and perform a partial sorting, we need a collection that is capable of maintaining the order. PriorityQueue is a good choice for that purpose. To store all other elements, I've used ArrayDeque, which doesn't maintain the order and as fast as ArrayList.
Accumulator BiConsumer<A,T> defines how to add elements into the mutable container provided by the supplier. For this task, the accumulator needs to guarantee that the queue, containing red-elements will not exceed the given size by rejecting values that are smaller than the lowest value previously added to the queue and by removing the lowest value if the size has reached the limit and a new value needs to be added. This functionality extracted into a separate method tryAdd()
Combiner BinaryOperator<A> combiner() establishes a rule on how to merge two containers obtained while executing stream in parallel. Here, combiner rely on the same logic that was described for accumulator.
Finisher Function<A,R> is meant to produce the final result by transforming the mutable container. In the code below, finisher dumps the contents of both queues into a stream, sorts them and collects into an immutable list.
Characteristics allow fine-tuning the collector by providing additional information on how it should function. Here a characteristic Collector.Characteristics.UNORDERED is being applied. Which indicates that the order in which partial results of the reduction produced in parallel is not significant, that can improve performance of this collector with parallel streams.
The code might look like this:
public static void main(String[] args) {
List<ElementsVO> allElements =
List.of(new ElementsVO(Color.RED, 25), new ElementsVO(Color.RED, 23), new ElementsVO(Color.RED, 27),
new ElementsVO(Color.BLACK, 19), new ElementsVO(Color.GREEN, 23), new ElementsVO(Color.GREEN, 29));
Comparator<ElementsVO> byAge = Comparator.comparing(ElementsVO::getAge);
List<ElementsVO> resultList = allElements.stream()
.collect(getNFiltered(byAge, element -> element.getColor() != Color.RED, 2));
resultList.forEach(System.out::println);
}
The method below is responsible for creating of a collector that partition the elements based on the given predicate and will sort them in accordance with the provided comparator.
public static <T> Collector<T, ?, List<T>> getNFiltered(Comparator<T> comparator,
Predicate<T> condition,
int limit) {
return Collector.of(
() -> Map.of(true, new PriorityQueue<>(comparator),
false, new ArrayDeque<>()),
(Map<Boolean, Queue<T>> isRed, T next) -> {
if (condition.test(next)) isRed.get(false).add(next);
else tryAdd(isRed.get(true), next, comparator, limit);
},
(Map<Boolean, Queue<T>> left, Map<Boolean, Queue<T>> right) -> {
left.get(false).addAll(right.get(false));
left.get(true).forEach(next -> tryAdd(left.get(true), next, comparator, limit));
return left;
},
(Map<Boolean, Queue<T>> isRed) -> isRed.values().stream()
.flatMap(Queue::stream).sorted(comparator).toList(),
Collector.Characteristics.UNORDERED
);
}
This method is responsible for adding the next red-element into the priority queue. It expects a comparator in order to be able to determine whether the next element should be added or discarded, and a value of the maximum size of the queue (2), to check if it was exceeded.
public static <T> void tryAdd(Queue<T> queue, T next, Comparator<T> comparator, int size) {
if (queue.size() == size && comparator.compare(queue.element(), next) < 0)
queue.remove(); // if the next element is greater than the smallest element in the queue and max size has been exceeded, the smallest element needs to be removed from the queue
if (queue.size() < size) queue.add(next);
}
Output
lementsVO{color=BLACK, age=19}
ElementsVO{color=GREEN, age=23}
ElementsVO{color=RED, age=25}
ElementsVO{color=RED, age=27}
ElementsVO{color=GREEN, age=29}
I wrote a generic Collector with a predicate and a limit of elements to add which match the predicate:
public class LimitedMatchCollector<T> implements Collector<T, List<T>, List<T>> {
private Predicate<T> filter;
private int limit;
public LimitedMatchCollector(Predicate<T> filter, int limit)
{
super();
this.filter = filter;
this.limit = limit;
}
private int count = 0;
#Override
public Supplier<List<T>> supplier() {
return () -> new ArrayList<T>();
}
#Override
public BiConsumer<List<T>, T> accumulator() {
return this::accumulator;
}
#Override
public BinaryOperator<List<T>> combiner() {
return this::combiner;
}
#Override
public Set<Characteristics> characteristics() {
return Stream.of(Characteristics.IDENTITY_FINISH)
.collect(Collectors.toCollection(HashSet::new));
}
public List<T> accumulator(List<T> list , T e) {
if (filter.test(e)) {
if (count >= limit) {
return list;
}
count++;
}
list.add(e);
return list;
}
public List<T> combiner(List<T> left , List<T> right) {
right.forEach( e -> {
if (filter.test(e)) {
if (count < limit) {
left.add(e);
count++;
}
}
});
return left;
}
#Override
public Function<List<T>, List<T>> finisher()
{
return Function.identity();
}
}
Usage:
List<ElementsVO> list = Arrays.asList(new ElementsVO("BLUE", 1)
,new ElementsVO("BLUE", 2) // made color a String
,new ElementsVO("RED", 3)
,new ElementsVO("RED", 4)
,new ElementsVO("GREEN", 5)
,new ElementsVO("RED", 6)
,new ElementsVO("YELLOW", 7)
);
System.out.println(list.stream().collect(new LimitedMatchCollector<ElementsVO>( (e) -> "RED".equals(e.getColor()),2)));
I am new to java 8.
me want to use java8 and want to convert below to java8.
List<Model> listModel;
for (Model model : listModel)
{
try
{
new UpDateData().bankData(model.getCust_id(), model.getBank_id(), model.getDate());
}
catch (Exception e) {
// TODO: handle exception
e.printStackTrace();
}
}
Model class::
public class Model
{
private int cust_id;
private int bank_id;
private String date;
//setter and getter
}
My question How me can apply java 8 features on above list, me want to iterate and call teh another function.
you could just use a forEach:
listModel.forEach(model -> {
try {
new UpDateData().bankData(model.getCust_id(), model.getBank_id(), model.getDate());
} catch(){
.... handle
}
})
The forEach is used in java 8 for iteration.You can loop over all elements using Iterable.forEach() method.
List<String> words = new ArrayList<>(Arrays.asList("the", "this",
"that", "there"));
alphabets.forEach(s -> { System.out.println(s) }); //using lamda
The List<> is the iteratable type, so you can directly use forEach on list.This is known as internal iteration as compared to old external iteration. This is automatically converted to external iteration by java. Curly braces are optional in lambda if there is only one statement else mandatory.
You can use filter on iterators as compared to old if blocks in loops.
alphabets.stream()
.filter(s -> s.startsWith("the"))
.forEach(System.out::println);
alphabets.stream()
.filter(s -> s.length() > 3)
.forEach(System.out::println);
The above syntax use method reference for printing the elements.
https://docs.oracle.com/javase/tutorial/java/javaOO/methodreferences.html
My list contains sets like [1,3,5][2,6,4] etc, all of the same size.
I tried doing this but it doesn't seem to work.
List<TreeSet<T>> block;
for(TreeSet<T> t : block){
block.stream().sorted((n,m)->n.compareTo(m)).collect(Collectors.toSet());
}
The end result I want is [1,2,3][4,5,6].
I could try to add all the elements in an ArrayList and sort that out then make a new List of TreeSet's. But is there is some kind of one liner?
UPDATE:
List<T> list=new ArrayList<T>();
for(TreeSet<T> t : block){
for(T t1 : t)
{
list.add(t1);
}
}
list=list.stream().sorted((n,m)->n.compareTo(m)).collect(Collectors.toList());
This works but could this be simplified?
#Eugene's answer is sweet, because Guava is sweet. But if you happen to not have Guava in your classpath, here's another way:
List<Set<Integer>> list = block.stream()
.flatMap(Set::stream)
.sorted()
.collect(partitioning(3));
First, I'm flatmapping all the sets into one stream, then I'm sorting all the elements and finally, I'm collecting the whole sorted stream to a list of sets. For this, I'm invoking a helper method that uses a custom collector:
private static <T> Collector<T, ?, List<Set<T>>> partitioning(int size) {
class Acc {
int count = 0;
List<Set<T>> list = new ArrayList<>();
void add(T elem) {
int index = count++ / size;
if (index == list.size()) list.add(new LinkedHashSet<>());
list.get(index).add(elem);
}
Acc merge(Acc another) {
another.list.stream().flatMap(Set::stream).forEach(this::add);
return this;
}
}
return Collector.of(Acc::new, Acc::add, Acc::merge, acc -> acc.list);
}
The method receives the size of each partition and uses the Acc local class as the mutable structure to be used by the collector. Inside the Acc class, I'm using a List that will contain LinkedHashSet instances, which will hold the elements of the stream.
The Acc class keeps the count of all the elements that have been already collected. In the add method, I calculate the index of the list and increment this count, and if there was no set in that position of the list, I append a new empty LinkedHashSet to it. Then, I add the element to the set.
As I'm calling sorted() on the stream to sort its elements before collecting, I need to use data structures that preserve insertion order. This is why I'm using ArrayList for the outer list and LinkedHashSet for the inner sets.
The merge method is to be used by parallel streams, to merge two previously accumulated Acc instances. I'm just adding all the elements of the received Acc instance to this Acc instance, by delegating to the add method.
Finally, I'm using Collector.of to create a collector based on the methods of the Acc class. The last argument is a finisher function, which just returns the Acc instance's list.
If you have guava on the classpath this is a breeze:
block
.stream()
.flatMap(Set::stream)
.collect(Collectors.toCollection(TreeSet::new));
Iterable<List<Integer>> result = Iterables.partition(sorted, 3);
Adding another answer since this would be bigger than a comment. It's really what the accepted answer has done, but with a "smarter" combiner that does not have to stream all the time again.
private static <T> Collector<T, ?, List<Set<T>>> partitioning(int size) {
class Acc {
int count = 0;
List<List<T>> list = new ArrayList<>();
void add(T elem) {
int index = count++ / size;
if (index == list.size()) {
list.add(new ArrayList<>());
}
list.get(index).add(elem);
}
Acc merge(Acc right) {
List<T> lastLeftList = list.get(list.size() - 1);
List<T> firstRightList = right.list.get(0);
int lastLeftSize = lastLeftList.size();
int firstRightSize = firstRightList.size();
// they have both the same size, simply addAll will work
if (lastLeftSize + firstRightSize == 2 * size) {
System.out.println("Perfect!");
list.addAll(right.list);
return this;
}
// last and first from each chunk are merged "perfectly"
if (lastLeftSize + firstRightSize == size) {
System.out.println("Almost perfect");
int x = 0;
while (x < firstRightSize) {
lastLeftList.add(firstRightList.remove(x));
--firstRightSize;
}
right.list.remove(0);
list.addAll(right.list);
return this;
}
right.list.stream().flatMap(List::stream).forEach(this::add);
return this;
}
public List<Set<T>> finisher() {
return list.stream().map(LinkedHashSet::new).collect(Collectors.toList());
}
}
return Collector.of(Acc::new, Acc::add, Acc::merge, Acc::finisher);
}
Is there any Utility function in Java to convert List<Object[]> to List<Object>
No, there's no method that does this directly for you. You can write a nested for loop or use the flatMap of the stream API as follows:
List<Object> flat = objArrs.stream()
.flatMap(Stream::of)
.collect(Collectors.toList());
You can easily write one yourself:
public static<T> List<T> append (List<T[]> input) {
List<T> res = new ArrayList<T>();
for(T[] subarr : input) {
if(subarr != null) {
int n = subarr.length;
for(int i = 0; i < n; i++) {
res.add(subarr[i]);
}
}
}
return res;
}
The function appends the different arrays and null arrays are ignored, null elements are however not. Thus if the input is [null,[a,b],[null,null],[c,null,d],null]. The output is [a,b,null,null,c,null,d].
No.
Why don't you just write the function yourself? It would probably be faster than asking this question and waiting for an answer.
In Java 8 you can do it with Streams :
List<Object[]> list = ...
List<Object> l = list.stream()
.flatMap(arr -> Stream.of(arr))
.collect(Collectors.toList());
As others have already said, there is no utility and creating one yourself wouldn't be hard, for example using old school for loops:
public List<Object> flatten( List<Object[]> source )
{
// if ( source == null ) return null; // which check you use it up to you
List<Object> result = new ArrayList<Object>();
if ( source == null ) return result; // Personally I like this check
for ( Object[] array: source )
{
if ( array == null ) continue; // skip nulls
for ( Object object: array )
{
result.add(object);
}
}
return result;
}