Develop Reference

Finite generated Stream in Java - how to create one?

In Java, one can easily generate an infinite stream with Stream.generate(supplier). However, I would need to generate a stream that will eventually finish.
Imagine, for example, I want a stream of all files in a directory. The number of files can be huge, therefore I can not gather all the data upfront and create a stream from them (via collection.stream()). I need to generate the sequence piece by piece. But the stream will obviously finish at some point, and terminal operators like (collect() or findAny()) need to work on it, so Stream.generate(supplier) is not suitable here.
Is there any reasonable easy way to do this in Java, without implementing the entire Stream interface on my own?
I can think of a simple hack - doing it with infinite Stream.generate(supplier), and providing null or throwing an exception when all the actual values are taken. But it would break the standard stream operators, I could use it only with my own operators that are aware of this behaviour.
CLARIFICATION
People in the comments are proposing me takeWhile() operator. This is not what I meant. How to phrase the question better... I am not asking how to filter (or limit) an existing stream, I am asking how to create (generate) the stream - dynamically, without loading all the elements upfront, but the stream would have a finite size (unknown in advance).
SOLUTION
The code I was looking for is
Iterator it = myCustomIteratorThatGeneratesTheSequence();
StreamSupport.stream(Spliterators.spliteratorUnknownSize(it, Spliterator.DISTINCT), false);
I just looked into java.nio.file.Files, how the list(path) method is implemented.

Is there any reasonable easy way to do this in Java, without implementing the entire Stream interface on my own?
A simple .limit() guarantees that it will terminate. But that's not always powerful enough.
After the Stream factory methods the simplest approach for creating customs stream sources without reimplementing the stream processing pipeline is subclassing java.util.Spliterators.AbstractSpliterator<T> and passing it to java.util.stream.StreamSupport.stream(Supplier<? extends Spliterator<T>>, int, boolean)
If you're intending to use parallel streams note that AbstractSpliterator only yields suboptimal splitting. If you have more control over your source fully implementing the Spliterator interface can better.
For example, the following snippet would create a Stream providing an infinite sequence 1,2,3...
in that particular example you could use IntStream.range()
But the stream will obviously finish at some point, and terminal operators like (collect() or findAny()) need to work on it.
short-circuiting operations like findAny() can actually finish on an infinite stream, as long as there is any element that matches.
Java 9 introduces Stream.iterate to generate finite streams for some simple cases.

Kotlin code to create Stream of JsonNode from InputStream
private fun InputStream.toJsonNodeStream(): Stream<JsonNode> {
return StreamSupport.stream(
Spliterators.spliteratorUnknownSize(this.toJsonNodeIterator(), Spliterator.ORDERED),
false
)
}
private fun InputStream.toJsonNodeIterator(): Iterator<JsonNode> {
val jsonParser = objectMapper.factory.createParser(this)
return object: Iterator<JsonNode> {
override fun hasNext(): Boolean {
var token = jsonParser.nextToken()
while (token != null) {
if (token == JsonToken.START_OBJECT) {
return true
}
token = jsonParser.nextToken()
}
return false
}
override fun next(): JsonNode {
return jsonParser.readValueAsTree()
}
}
}

Here is a stream which is custom and finite :
package org.tom.stream;
import java.util.*;
import java.util.function.*;
import java.util.stream.*;
public class GoldenStreams {
private static final String IDENTITY = "";
public static void main(String[] args) {
Stream<String> stream = java.util.stream.StreamSupport.stream(new Spliterator<String>() {
private static final int LIMIT = 25;
private int integer = Integer.MAX_VALUE;
{
integer = 0;
}
#Override
public int characteristics() {
return Spliterator.DISTINCT;
}
#Override
public long estimateSize() {
return LIMIT-integer;
}
#Override
public boolean tryAdvance(Consumer<? super String> arg0) {
arg0.accept(IDENTITY+integer++);
return integer < 25;
}
#Override
public Spliterator<String> trySplit() {
System.out.println("trySplit");
return null;
}}, false);
List<String> peeks = new ArrayList<String>();
List<String> reds = new ArrayList<String>();
stream.peek(data->{
peeks.add(data);
}).filter(data-> {
return Integer.parseInt(data)%2>0;
}).peek(data ->{
System.out.println("peekDeux:"+data);
}).reduce(IDENTITY,(accumulation,input)->{
reds.add(input);
String concat = accumulation + ( accumulation.isEmpty() ? IDENTITY : ":") + input;
System.out.println("reduce:"+concat);
return concat;
});
System.out.println("Peeks:"+peeks.toString());
System.out.println("Reduction:"+reds.toString());
}
}

While the author has discarded the takeWhile option, I find it adequate for certain use cases and worth an explanation.
The method takeWhile can be used on any stream and will terminate the stream when the predicate provided to the method returns false. The object which results in a false is not appended to the stream; only the objects which resulted in true are passed downstream.
So one method for generating a finite stream could be to use the Stream.generate method and return a value which signals the end of the stream by being evaluated to false by the predicate provided to takeWhile.
Here's an example, generating all the permutations of an array :
public static Stream<int[]> permutations(int[] original) {
int dim = original.length;
var permutation = original.clone();
int[] controller = new int[dim];
var low = new AtomicInteger(0);
var up = new AtomicInteger(1);
var permutationsStream = Stream.generate(() -> {
while (up.get() < dim) {
if (controller[up.get()] < up.get()) {
low.set(up.get() % 2 * controller[up.get()]);
var tmp = permutation[low.get()];
permutation[low.get()] = permutation[up.get()];
permutation[up.get()] = tmp;
controller[up.get()]++;
up.set(1);
return permutation.clone();
} else {
controller[up.get()] = 0;
up.incrementAndGet();
}
}
return null;
}).takeWhile(Objects::nonNull);
return Stream.concat(
Stream.ofNullable(original.clone()),
permutationsStream
);
}
In this example, I used the null value to signal the end of the stream.
The caller of the method won't receive the null value !
OP could use a similar strategy, and combine it with a visitor pattern.
If it's a flat directory, OP would be better off using Stream.iterate with the seed being the index of the file to yield and Stream.limit on the number of files (which can be known without browsing the directory).

Simple Logic or Regex Expresssion for a string

Hi I've a string like the following -
name,number,address(line1,city),status,contact(id,phone(number,type),email(id),type),closedate
I need to output the following -
name,number,address.line1,address.city,status,contact.id,contact.phone.number,contact.phone.type,contact.email.id,contact.type,closedate
Is it possible to do it using regex in java. Logic I have thought of is using string manipulation (with substring,recursion etc). Is there a simple way of achieving this? I would prefer a regular expression which works in java. Other suggestions are also welcome.
To give you a context
The string above is coming as query parameter, I have to find out what all columns I need to select based on that. so all these individual items in the output will have a respective column name in property file.
Thanks
Pal

public class Main {
public static void main(String[] args) {
;
String input ="name,number,address(line1,test(city)),status,contact(id,phone(number,type),email(id),type),closedate";
List<String> list = new ArrayList<String>(Arrays.asList(input.split(","))); // We need a list for the iterator (or ArrayIterator)
List<String> result = new Main().parse(list);
System.out.println(String.join(",", result));
}
private List<String> parse(List<String> inputString){
Iterator<String> it = inputString.iterator();
ArrayList<String> result = new ArrayList<>();
while(it.hasNext()){
String word = it.next();
if(! word.contains("(")){
result.add(word);
} else { // if we come across a "(", start the recursion and parse it till we find the matching ")"
result.addAll(buildDistributedString(it, word,""));
}
}
return result;
}
/*
* recursivly parse the string
* #param startword The first word of it (containing the new prefix, the ( and the first word of this prefic
* #param prefix Concatenation of previous prefixes in the recursion
*/
private List<String> buildDistributedString(Iterator<String> it, String startword,String prefix){
ArrayList<String> result = new ArrayList<>();
String[] splitted = startword.split("\\(");
prefix += splitted[0]+".";
if(splitted[1].contains(")")){ //if the '(' is immediately matches, return only this one item
result.add(prefix+splitted[1].substring(0,splitted[1].length()-1));
return result;
} else {
result.add(prefix+splitted[1]);
}
while(it.hasNext()){
String word = it.next();
if( word.contains("(")){ // go deeper in the recursion
List<String> stringList = buildDistributedString(it, word, prefix);
if(stringList.get(stringList.size()-1).contains(")")){
// if multiple ")"'s were found in the same word, go up multiple recursion levels
String lastString = stringList.remove(stringList.size()-1);
stringList.add(lastString.substring(0,lastString.length() -1));
result.addAll(stringList);
break;
}
result.addAll(stringList);
} else if(word.contains(")")) { // end this recursion level
result.add(prefix + word.substring(0,word.length()-1)); // ")" is always the last char
break;
} else {
result.add(prefix+word);
}
}
return result;
}
}
I wrote a quick parser for this. There probably are some improvements possible, but this should give you an idea. It was just meant to get a working version asap.

Since nested parentheses appear in your string, regular expressions can't do the job. The explanation why is complicated, requiring knowledge in context free grammars. See Can regular expressions be used to match nested patterns?
I've heard this kind of parsing can be done through callbacks, but I believe it doesn't exist in Java.
Parser generators like JavaCC would do the job, but that's a huge overkill for the task you are describing.
I recommend you to look into java.util.Scanner, and you recursively call the parse method whether you see a left paren.

Implementing a Metric Suite using ASTParser in Java

Background - Question below
I am at the start of implementing a metric suite in Java for Java however I am concerned that my approach is not appropriate.
Currently I am using the JDT's ASTParser for every file within a directory. This started off well and I was able to collect things around line count and average lines per method for each class. This was done via a MethodVisitor class which extends ASTVisitor and contains a method visit(MethodDeclaration node).
I am now trying to calculate Cyclomatic Complexity for every method. I have split out the method body and have a ComplexityVisitor which contains a visit(IfStatement node) and a visit(ReturnStatement node).
Using this structure I know that there is a if statement within the code but I am unsure on how to know how many levels of "if else"s there are. The only method I can find that is helpful is the node.getElseStatement() but this returns what is basically (or seems to me) a string and would therefore have to use regex to know the number of paths the statement could take.
So my question is:
Is there a way to deduce how many levels are in the "if - else if - else" statement when using eclipses ASTParser?
or
Should I be looking for a cleaner solution such as IJavaElement or parsing the code myself putting key words onto a list then looping back through them.
Some sample Code - very much in testing phase
public class Test {
private static List<ClassInfo> klasses = new ArrayList<ClassInfo>();
// Called for every file where str is what the file contains
public static void parse(String str) {
ASTParser parser = ASTParser.newParser(AST.JLS3);
parser.setSource(str.toCharArray());
parser.setKind(ASTParser.K_COMPILATION_UNIT);
final CompilationUnit cu = (CompilationUnit) parser.createAST(null);
ClassVisitor cv = new ClassVisitor();
cu.accept(cv);
MethodVisitor methodsVisitor = new MethodVisitor(cu);
cu.accept(methodsVisitor);
ClassInfo klass = new ClassInfo(cv.getClassName(),
cu.getLineNumber(cu.getLength() - 1),
methodsVisitor.getNumberOfMethods(),
methodsVisitor.getAverageLinesPerMethod(),
methodsVisitor.getMethods());
for(int i = 0; i < klass.methods.size(); i++){
parser.setSource(klass.methods.get(i).body.toCharArray());
CyclomaticComplexityVisitor ccv = new CyclomaticComplexityVisitor();
cu.accept(ccv);
}
klasses.add(klass);
}
-
public class MethodVisitor extends ASTVisitor {
private CompilationUnit cu;
private int numberOfMethods;
private int lineCount;
private List<MethodInfo> methods = new ArrayList<MethodInfo>();
public MethodVisitor(CompilationUnit cu){
this.cu = cu;
}
public boolean visit(MethodDeclaration node){
int startPos = cu.getLineNumber(node.getStartPosition());
int endPos = cu.getLineNumber(node.getStartPosition() + node.getLength());
lineCount += (endPos - startPos);
numberOfMethods++;
String methodBody = node.getBody().toString();
MethodInfo m = new MethodInfo(node.getName().getIdentifier(),
(endPos - startPos),
node.getReturnType2());
m.body = methodBody;
methods.add(m);
return true;
}
-
public class CyclomaticComplexityVisitor extends ASTVisitor {
private int complexityScore = 0;
private int edges = 0;
private int nodes = 0;
private int exitPoints = 1;
private boolean firstReturn = true;
public boolean visit(IfStatement node){
System.out.println("THERE WAS AN IF");
String statement = node.toString();
System.out.println(statement);
return true;
}
public boolean visit(ReturnStatement node){
if (firstReturn) {
firstReturn = false;
} else {
exitPoints++;
}
return true;
}
Cheers

I'm not sure if this will answer your question, but for calculating McCabe's Cyclomatic Complexity (McCC) metric, you don't need to care about if-else-if nesting levels. You simply need to count the number of "branching" instructions and add 1 in the end. See the definition in the User's Guide of our SourceMeter tool:
McCabe's Cyclomatic Complexity (McCC)
Method: complexity of the method expressed as the number of independent control flow paths in it. It represents a lower bound for the number of possible execution paths in the source code and at the same time it is an upper bound for the minimum number of test cases needed for achieving full branch test coverage. The value of the metric is calculated as the number of the following instructions plus 1: if, for, foreach, while, do-while, case label (which belongs to a switch instruction), catch, conditional statement (?:). Moreover,
logical “and” (&&) and logical “or” (||) expressions also add 1 to the value because their short-circuit evaluation can cause branching depending on the first operand. The following instructions are not included: else, switch, default label (which belongs to a switch instruction), try, finally.

a question on conversion logic

I am designing a unit converter using android.i m using logic which is not nice logic i think and i m giving some code here too.can anyone suggest me any better logic????
public void temperature(String value1,String value2)
{
if(value1.equals("Celsius") && value2.equals("fahrenheit") )
{
double i=Double.parseDouble(at1.getText().toString());
double value=i*33.8;
Double d=new Double(value);
at2.setText(d.toString());
}
else if(value1.equals("fahrenheit") && value2.equals("Celsius"))
{
double i=Double.parseDouble(at1.getText().toString());
double value=i*(-17.2222);
Double d=new Double(value);
at2.setText(d.toString());
}
There is like this many unit in every category like angle,computer.this is only a small example of temperature category.

Why not just
at2.setText(String.valueOf(Double.parseDouble(at1.getText().toString()) * 33.8);
Anyway, it's just to shorten your code, the logic stays the same.
P.S. Define some constants for values like 33.8 and -17.2222.

There's a lot of duplicated code in your example. Try to refactor it so you common actions are expressed only once.
Example:
public void temperature(String value1, String value2) {
double i = Double.parseDouble(at1.getText().toString());
double value;
if (value1.equals("fahrenheit") && value2.equals("celsius")) {
value = /* convert i */;
} else if (....) {
value = ...;
}
at2.setText(String.valueOf(value));
}
If your list of if statements becomes large, consider using a switch statement instead. You might want to change representation of the units from String to something else like integer constants or enums.

My suggestion would be to use the command pattern.
For temperature, I believe you can use Kelvin to convert to a common scale:
public class TemperatureConverter {
private final static String CELSIUS = "Celsius";
private final static String FARENHEIT = "Farenheit";
public double temperature(double temperature, String temp1Type, String temp2Type) {
Converter fromConverter, toConverter;
if (temp1Type.equals(CELSIUS)) {
fromConverter = new CelsiusConverter();
} else if (temp1Type.equals(FARENHEIT)) {
fromConverter = new FarenheitConverter();
} else {
fromConverter = new noopConverter();
}
if (temp2Type.equals(CELSIUS)) {
toConverter = new CelsiusConverter();
} else if (temp2Type.equals(FARENHEIT)) {
toConverter = new FarenheitConverter();
} else {
toConverter = new noopConverter();
}
return toConverter.fromKelvin(fromConverter.toKelvin(temperature));
}
}
Converter is an interface, and FarenheitConverter and CelsiusConverter are concrete classes that implement this interface. They contain the details to convert from Farenheit and Celsius (respectively) to Kelvin. They also contain the details of converting from Kelvin to their respective scales. This keeps the logic cleanly separated, and easily extended in the future.
Say in the future you need to convert from degrees blignox to degrees fizzbuzz. It's as simple as writing the BlignoxConverter and the FizzbuzzConverter, implementing the two methods, and plugging them in. Once they're written and integrated, you can convert from either of those to or from F and C as well.

Is there a Java equivalent to C#'s 'yield' keyword?

I know there is no direct equivalent in Java itself, but perhaps a third party?
It is really convenient. Currently I'd like to implement an iterator that yields all nodes in a tree, which is about five lines of code with yield.

The two options I know of is Aviad Ben Dov's infomancers-collections library from 2007 and Jim Blackler's YieldAdapter library from 2008 (which is also mentioned in the other answer).
Both will allow you to write code with yield return-like construct in Java, so both will satisfy your request. The notable differences between the two are:
Mechanics
Aviad's library is using bytecode manipulation while Jim's uses multithreading. Depending on your needs, each may have its own advantages and disadvantages. It's likely Aviad's solution is faster, while Jim's is more portable (for example, I don't think Aviad's library will work on Android).
Interface
Aviad's library has a cleaner interface - here's an example:
Iterable<Integer> it = new Yielder<Integer>() {
#Override protected void yieldNextCore() {
for (int i = 0; i < 10; i++) {
yieldReturn(i);
if (i == 5) yieldBreak();
}
}
};
While Jim's is way more complicated, requiring you to adept a generic Collector which has a collect(ResultHandler) method... ugh. However, you could use something like this wrapper around Jim's code by Zoom Information which greatly simplifies that:
Iterable<Integer> it = new Generator<Integer>() {
#Override protected void run() {
for (int i = 0; i < 10; i++) {
yield(i);
if (i == 5) return;
}
}
};
License
Aviad's solution is BSD.
Jim's solution is public domain, and so is its wrapper mentioned above.

Both of these approaches can be made a bit cleaner now Java has Lambdas. You can do something like
public Yielderable<Integer> oneToFive() {
return yield -> {
for (int i = 1; i < 10; i++) {
if (i == 6) yield.breaking();
yield.returning(i);
}
};
}
I explained a bit more here.

I know it's a very old question here, and there are two ways described above:
bytecode manipulation that's not that easy while porting;
thread-based yield that obviously has resource costs.
However, there is another, the third and probably the most natural, way of implementing the yield generator in Java that is the closest implementation to what C# 2.0+ compilers do for yield return/break generation: lombok-pg. It's fully based on a state machine, and requires tight cooperation with javac to manipulate the source code AST. Unfortunately, the lombok-pg support seems to be discontinued (no repository activity for more than a year or two), and the original Project Lombok unfortunately lacks the yield feature (it has better IDE like Eclipse, IntelliJ IDEA support, though).

I just published another (MIT-licensed) solution here, which launches the producer in a separate thread, and sets up a bounded queue between the producer and the consumer, allowing for buffering, flow control, and parallel pipelining between producer and consumer (so that the consumer can be working on consuming the previous item while the producer is working on producing the next item).
You can use this anonymous inner class form:
Iterable<T> iterable = new Producer<T>(queueSize) {
#Override
public void producer() {
produce(someT);
}
};
for example:
for (Integer item : new Producer<Integer>(/* queueSize = */ 5) {
#Override
public void producer() {
for (int i = 0; i < 20; i++) {
System.out.println("Producing " + i);
produce(i);
}
System.out.println("Producer exiting");
}
}) {
System.out.println(" Consuming " + item);
Thread.sleep(200);
}
Or you can use lambda notation to cut down on boilerplate:
for (Integer item : new Producer<Integer>(/* queueSize = */ 5, producer -> {
for (int i = 0; i < 20; i++) {
System.out.println("Producing " + i);
producer.produce(i);
}
System.out.println("Producer exiting");
})) {
System.out.println(" Consuming " + item);
Thread.sleep(200);
}

Stream.iterate(seed, seedOperator).limit(n).foreach(action) is not the same as yield operator, but it may be usefull to write your own generators this way:
import java.util.stream.Stream;
public class Test01 {
private static void myFoo(int someVar){
//do some work
System.out.println(someVar);
}
private static void myFoo2(){
//do some work
System.out.println("some work");
}
public static void main(String[] args) {
Stream.iterate(1, x -> x + 1).limit(15).forEach(Test01::myFoo); //var1
Stream.iterate(1, x -> x + 1).limit(10).forEach(item -> myFoo2()); //var2
}
}

I'd also suggest if you're already using RXJava in your project to use an Observable as a "yielder". It can be used in a similar fashion if you make your own Observable.
public class Example extends Observable<String> {
public static void main(String[] args) {
new Example().blockingSubscribe(System.out::println); // "a", "b", "c", "d"
}
#Override
protected void subscribeActual(Observer<? super String> observer) {
observer.onNext("a"); // yield
observer.onNext("b"); // yield
observer.onNext("c"); // yield
observer.onNext("d"); // yield
observer.onComplete(); // finish
}
}
Observables can be transformed into iterators so you can even use them in more traditional for loops. Also RXJava gives you really powerful tools, but if you only need something simple then maybe this would be an overkill.

// Java code for Stream.generate()
// to generate an infinite sequential
// unordered stream
import java.util.*;
import java.util.stream.Stream;
class GFG {
// Driver code
public static void main(String[] args) {
// using Stream.generate() method
// to generate 5 random Integer values
Stream.generate(new Random()::nextInt)
.limit(5).forEach(System.out::println);
}
}
From here.

I wrote a new library that has implemented generator for Java. It's simple, thread-free and fast.
Here is an example of generating endless fibonacci numbers:
public static Seq<Integer> fibonacci() {
return c -> {
int a = 1;
int b = 1;
c.accept(a);
c.accept(b);
while (true) {
c.accept(b = a + (a = b));
}
};
}
The Seq interface is just like Java Stream and Kotlin Sequence, but faster than all of them.
Here, let's print the first 7 elements of the fibonacci series
Seq<Integer> fib = fibonacci();
fib.take(7).printAll(","); // => 1,1,2,3,5,8,13
For the original problem, yielding all nodes of a tree? One line is enough.
Seq<Node> seq = Seq.ofTree(root, n -> Seq.of(n.left, n.right));