Recursive stream - java

I want to list all the files on my computer recursively using Java 8.
Java 8 provides a listFiles method that returns all the files and directories but without recursion. How can I use it to get a full recursive list of files (without using a mutating collection)?
I've tried the code below but it only goes one level deep:
static Function<Path, Stream<Path>> listFiles = p -> {
if (p.toFile().isDirectory()) {
try { return Files.list(p); }
catch (Exception e) { return Stream.empty(); }
} else {
return Stream.of(p);
}
};
public static void main(String[] args) throws IOException {
Path root = Paths.get("C:/temp/");
Files.list(root).flatMap(listFiles).forEach(System.out::println);
}
And using return Files.list(p).flatMap(listFiles); does not compile (not sure why)...
Note: I am not interested in solutions involving FileVisitors or external libraries.

A new API to generate a stream of Paths by walking the filesystem recursively is Files.walk.
If you really want to generate a stream recursively (not necessarily walking the file tree, but I'll continue using that as an example), it might be a bit more straightforward to accomplish the recursion using method references:
class RecursiveStream {
static Stream<Path> listFiles(Path path) {
if (Files.isDirectory(path)) {
try { return Files.list(path).flatMap(RecursiveStream::listFiles); }
catch (Exception e) { return Stream.empty(); }
} else {
return Stream.of(path);
}
}
public static void main(String[] args) {
listFiles(Paths.get(".")).forEach(System.out::println);
}
}
Method references turn out to be quite useful for adapting a named method that has the same "shape" (arguments and return type) as a functional interface to that functional interface. This also avoids the potential initialization circularity with storing a lambda in an instance or static variable and calling itself recursively.

It is apparently not possible to refer to a function within that function definition through a method reference but it works with a lambda.
So in the function, return Files.list(p).flatMap(listFiles); does not compile but return Files.list(p).flatMap(q -> listFiles.apply(q)); does.
This prints all the files in the given folder recursively:
static final Function<Path, Stream<Path>> listFiles = p -> {
if (p.toFile().isDirectory()) {
try { return Files.list(p).flatMap(q -> listFiles.apply(q)); }
catch (Exception e) { return Stream.empty(); }
} else {
return Stream.of(p);
}
};
public static void main(String[] args) throws IOException {
Path root = Paths.get("C:/temp/");
Files.list(root).flatMap(listFiles).forEach(System.out::println);
}
but as pointed out, this is unnecessary:
Files.walk(root).forEach(System.out::println);
does the same thing...

Related

Testing a method by overriding a private class variable as an initial step before refactoring

What is the best way of writing a unit test for a method, such as my setProperties (see below), that uses a private configuration variable (config). I tried but failed to override it using reflection and Makito, but without success. I realize that changing the design to make the code easier to test is best, but I want to created some unit tests before I refactor the code.
public class MainClass {
private final java.lang.String config = "app.properties";
public TestClass() {
try {
setProperties();
} catch (Exception e) {
e.printStackTrace();
}
}
public void setProperties() throws Exception {
try {
InputStream input = new BufferedInputStream(new FileInputStream(config));
..
..
} catch (Exception exception) {
throw exception;
}
}
}
Do refactor a tiny bit by extracting a method with a parameter that takes an input stream. Call this new method (probably package-protected) from the old one. Write tests against the new method. Then do more refactorings.
This is an indication of a broken design; don't hard-code things like this. Better yet, determine what the appropriate responsibility for this class is, and, in decreasing order of preference:
pass in an object with the configuration properties, strongly typed
pass in a Map with the configuration properties
pass in an InputStream for the properties file
As File objects are never available from a jar, you shouldn't ever make interfaces like this more specific than InputStream or Reader, so that you can always pass in streams from your jar classpath.
So you can use Properties class in Java for this. Please have a look at this code.
public class PropertyUtil {
private static Properties prop;
private static Logger logger = Logger.getLogger(PropertyUtil.class);
private PropertyUtil() {
}
public void setProperty() {
String filePath = System.getenv("JAVA_HOME") + "/lib" + "/my_file.properties";
prop = new Properties();
try (InputStream input = new FileInputStream(filePath)) {
prop.load(input);
} catch (IOException ex) {
logger.error("Error while reading property file " + ex);
}
}
public static String getProperty(String key) {
if (prop.containsKey(key)) {
return prop.getProperty(key);
} else {
return null;
}
}
public static <T> T getProperty(String key, Class<T> claz) {
if (claz.getName().equals(Integer.class.getName())) {
return claz.cast(Integer.parseInt(prop.getProperty(key)));
}
if (claz.getName().equals(Long.class.getName())) {
return claz.cast(Long.parseLong(prop.getProperty(key)));
}
if (claz.getName().equals(Boolean.class.getName())) {
return claz.cast(Boolean.parseBoolean(prop.getProperty(key)));
}
if (claz.getName().equals(Double.class.getName())) {
return claz.cast(Double.parseDouble(prop.getProperty(key)));
}
if (claz.getName().equals(String.class.getName())) {
return claz.cast(prop.getProperty(key));
}
return null;
}

Method wrapper for dealing with Exceptions?

I'm implementing an Iterator and in order to deal with the Exceptions I'm using the following pattern: The actual work is done in the private hasNextPriv() method whereas the hasNext() method deals with the Exceptions. The reason for doing it this way is because I don't want to litter hasNextPriv() with try-catch blocks.
#Override
public boolean hasNext()
{
try {
return hasNextPriv();
} catch (XMLStreamException e) {
e.printStackTrace();
try {
reader.close();
} catch (XMLStreamException e1) {
e1.printStackTrace();
}
}
return false;
}
Questions:
Is there a better way to do this?
What would be a good name for the private method hasNextPriv()?
Another way to handle exceptions would be to extract each part that throws exception in a small pure function that properly handles each exception. And then construct final result composing those functions.
Optional<Resource> open() {
try{
//...
return Optional.of(resource);
} catch {
//....
return Optional.empty();
}
}
Optional<Value> read(Resource resource) {
try{
//...
return Optional.of(resource.value);
} catch {
//....
return Optional.empty();
}
}
boolean hasNext() {
open().flatMap(this::read).isPresent();
}
There is no need to return Optional everywhere. Usually there is some dummy value like in Null Object Pattern
Another pattern is to wrap a function execution in object that produces either result or error value. In library javaslang it looks like
return Try.of(this::hasNextPriv)
.recover(x -> Match(x).of(
Case(instanceOf(Exception_1.class), /*handle exception*/),
Case(instanceOf(Exception_2.class), ...)))
.getOrElse(false);
Try object is similar to java 8 Optional but instead of holding present value or missing value Try contains value of either success or failure.
Regarding naming hasNextPriv in your case there is specific domain of data structure. Probably you could come up with more specific name like hasMoreNodes or notEmpty etc.

Basic layout/structure of a Java Applet

I have a very simple Java applet that just works.
import java.sql.*;
import org.apache.commons.lang3.*;
public class doQuery {
public static void main (String[] args) {
...
try {
...
} catch (Exception e) {
...
}
try {
...
try {
...
} catch (SQLException e) {
...
} finally {
...
}
} catch (SQLException e) {
...
}
}
}
It allows me to open a database, do some queries, and perform a series of outputs that is captured through stdio of a bash script, connections closed and then bash script emails the output.
However, I am looking to expand it, and I am stuck. I am programmer, just not a Java programmer. What I have come up with is something I hacked together. I want to add some functions, and more. I have tried to the function definitions in different places in the code, but it always generates compilation errors.
Can anyone provide some insight as to what I can change to enable me to add some functions? Generally programming say define the function before you attempt to use it, but I probably am not using the right keywords or something.
I can not figure out where to place a simple function like:
function display_number(number) {
return number + "";
}
in the source code that I can call and have it compile! :(
import java.sql.*;
import org.apache.commons.lang3.*;
public class doQuery {
public static void main (String[] args) {
...
try {
...
} catch (Exception e) {
...
}
try {
...
try {
...
} catch (SQLException e) {
...
} finally {
...
}
} catch (SQLException e) {
...
}
displayNumber(4); // Replace 4 with a variable to output or what not
}
private int displayNumber(int number) {
return number;
}
}
In Java you'll add your methods inside the scope of the class like so, when you want to use them call them inside of another function like in the example above. I'd recommend reading this link, it should give you a good understanding of how methods work, how to call them, etc.
If you just want a quick and easy function, here is a layout you can use.
public void display_number(int number) {
System.out.print(number);
}
Or if you prefer returning the number to use it somewhere else...
public int display_number() {
return number;
}
Your methods can be basically anywhere between the start { and the end } of your class, assuming you do not place it inside another method (such as the main method from your code). As far as anything else I recommend reading up on the subject, I am sure a google search will give you millions of examples of method calls.

How do I write a unit test to cover the case where an IOException is thrown?

I have the following class:
public class FileLoader {
private Map<Brand, String> termsOfUseText = new HashMap<Brand, String>();
public void load() {
for (Brand brand : Brand.values()) {
readAndStoreTermsOfUseForBrand(brand);
}
}
private void readAndStoreTermsOfUseForBrand(Brand brand) {
String resourceName = "termsOfUse/" + brand.name().toLowerCase() + ".txt";
InputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
try {
String content = IOUtils.toString(in);
termsOfUseText.put(brand, content);
} catch (IOException e) {
throw new IllegalStateException(String.format("Failed to find terms of use source file %s", resourceName),e);
}
}
public String getTextForBrand(Brand brand) {
return termsOfUseText.get(brand);
}
}
Brand is an enum, and I need all the valid .txt files to be on the classpath. How do I make the IOException occur, given that the Brand enum contains all the valid brands and therfore all the .txt files for them exist?
Suggestions around refactoring the current code are welcome if it makes it more testable!
Three options I see right off:
Use PowerMock to mock IOUtils.toString(). I consider PowerMock to be quite a last resort. I'd rather refactor the source to something a little more test-friendly.
Extract the IOUtils call to a protected method. Create a test-specific subclass of your class that overrides this method and throws the IOException.
Extract the InputStream creation to a protected method. Create a test-specific subclass to override the method and return a mock InputStream.
I would suggest a bit of refactoring. All your methods are void, this usually means they are not functional.
For example, you can extract this functionality:
private String readTermsOfUseForBrand(InputStream termsOfUserIs) {
try {
String content = IOUtils.toString(in);
return content;
} catch (IOException e) {
throw new IllegalStateException(String.format("Failed to find terms of use source file %s", resourceName), e);
}
return null;
}
So that we can assert on the String result in our tests.
Of course this is not functional code, as it reads from an Input Stream. And it does so with IOUtils.toString() method that cannot be mocked easily (well, there's PowerMock but as Ryan Stewart said it's the last resort).
To test IO exceptions you can create a failing input stream (tested with JDK7):
public class FailingInputStream extends InputStream {
#Override
public int read() throws IOException {
throw new IOException("Test generated exception");
}
}
And test like that:
#Test
public void testReadTermsOfUseForBrand() {
FileLoader instance = new FileLoader();
String result = instance.readTermsOfUseForBrand(new FailingInputStream());
assertNull(result);
}
Missing file will cause NullPointerException because getResourceAsStream will return null and you will have in==null. IOException in this case may actually be pretty rare. If it's critical for you to see it, I can only think of instrumenting this code to throw it if code is executed in test scope. But is it really that important?
I would use a mock to accomplish this.
Example (untested, just to give you some thought):
#Test(expected=IllegalStateException.class)
public void testThrowIOException() {
PowerMockito.mockStatic(IOUtils.class);
PowerMockito.when(IOUtils.toString()).thenThrow(
new IOException("fake IOException"));
FileLoader fileLoader = new FileLoader();
Whitebox.invokeMethod(fileLoader,
"readAndStoreTermsOfUseForBrand", new Brand(...));
// If IllegalStateException is not thrown then this test case fails (see "expected" above)
}
Code below is completely untested
To cause the IOException use:
FileInputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
in.mark(0);
//read some data
in.reset(); //IOException
To test the IOException case use:
void test
{
boolean success = false;
try
{
//code to force ioException
}
catch(IOException ioex)
{
success = true;
}
assertTrue(success);
}
In JUnit4
#Test(expected=IOException.class)
void test
{
//code to force ioException
}
Other JUnit
void test
{
try
{
//code to force IOException
fail("If this gets hit IO did not occur, fail test");
}
catch(IOException ioex)
{
//success!
}
}

Java exception handling in non sequential tasks (pattern/good practice)

There are some task that should't be done in parallel, (for example opening a file, reading, writing, and closing, there is an order on that...)
But... Some task are more like a shoping list, I mean they could have a desirable order but it's not a must..example in communication or loading independient drivers etc..
For that kind of tasks,
I would like to know a java best practice or pattern for manage exceptions..
The java simple way is:
getUFO {
try {
loadSoundDriver();
loadUsbDriver();
loadAlienDetectorDriver();
loadKeyboardDriver();
} catch (loadSoundDriverFailed) {
doSomethingA;
} catch (loadUsbDriverFailed) {
doSomethingB;
} catch (loadAlienDetectorDriverFailed) {
doSomethingC;
} catch (loadKeyboardDriverFailed) {
doSomethingD;
}
}
But what about having an exception in one of the actions but wanting to
try with the next ones??
I've thought this approach, but don't seem to be a good use for exceptions
I don't know if it works, doesn't matter, it's really awful!!
getUFO {
Exception ex=null;
try {
try{ loadSoundDriver();
}catch (Exception e) { ex=e; }
try{ loadUsbDriver();
}catch (Exception e) { ex=e; }
try{ loadAlienDetectorDriver();
}catch (Exception e) { ex=e; }
try{ loadKeyboardDriver()
}catch (Exception e) { ex=e; }
if(ex!=null)
{ throw ex;
}
} catch (loadSoundDriverFailed) {
doSomethingA;
} catch (loadUsbDriverFailed) {
doSomethingB;
} catch (loadAlienDetectorDriverFailed) {
doSomethingC;
} catch (loadKeyboardDriverFailed) {
doSomethingD;
}
}
seems not complicated to find a better practice for doing that.. I still didn't
thanks for any advice
Consider the execute around idiom.
Another option (which isn't really all that different, it just decouples them more) is to do each task in a separate thread.
Edit:
Here is the kind of thing I have in mind:
public interface LoadableDriver {
public String getName();
public void loadDriver() throws DriverException;
public void onError(Throwable e);
}
public class DriverLoader {
private Map<String, Exception> errors = new HashMap<String, Exception>();
public void load(LoadableDriver driver) {
try {
driver.loadDriver();
} catch (DriverException e) {
errors.put(driver.getName(), e);
driver.onError(e);
}
}
public Map<String, Exception> getErrors() { return errors; }
}
public class Main {
public void loadDrivers() {
DriverLoader loader = new DriverLoader();
loader.loadDriver(new LoadableDriver(){
public String getName() { return "SoundDriver"; }
public void loadDriver() { loadSoundDriver(); }
public void onError(Throwable e) { doSomethingA(); }
});
//etc. Or in the alternative make a real class that implements the interface for each driver.
Map<String, Exception> errors = loader.getErrors();
//react to any specific drivers that were not loaded and try again.
}
}
Edit: This is what a clean Java version would ultimately look like if you implemented the drivers as classes (which is what the Java OO paradigm would expect here IMHO). The Main.loadDrivers() method would change like this:
public void loadDrivers(LoadableDriver... drivers) {
DriverLoader loader = ...
for(LoadableDriver driver : drivers) {
loader.load(driver);
}
//retry code if you want.
Set<LoadableDriver> failures = loader.getErrors();
if(failures.size() > 0 && tries++ > MAX_TRIES) {
//log retrying and then:
loadDrivers(drivers.toArray(new LoadableDriver[0]));
}
}
Of course I no longer use a map because the objects would be self-sufficient (you could get rid of the getName() method as well, but probably should override toString()), so the errors are just returned in a set to retry. You could make the retry code even simpler if each driver was responsible for knowing how often it should it retry.
Java won't look as nice as a well done C++ template, but that is the Java language design choice - prefer simplicity over complex language features that can make code hard to maintain over time if not done properly.
Try this:
protected void loadDrivers() {
loadSoundDriver();
loadUsbDriver();
loadAlienDetectorDriver();
loadKeyboardDriver();
}
Then:
protected void loadSoundDriver() {
try {
// original code ...
}
catch( Exception e ) {
soundDriverFailed( e );
}
}
protected void soundDriverFailed( Exception e ) {
log( e );
}
This gives subclasses a chance to change the behaviour. For example, a subclass could implement loading each driver in a separate thread. The main class need not care about how the drivers are loaded, nor should any users of the main class.
IMO, for your case, if the exception is "ignorable" it's best if the "loadSoundDriver" method catches the exception and simply returns an error.
Then in the function that loads stuff, you can record all the errors and at the end of the sequence, decide what to do with them.
[edit]
Something like this:
// init
MyError soundErr = loadSoundDriver();
MyError otherErr = loadOtherDriver();
if(soundErr!=null || otherErr !=null){
// handle the error(s)
}
Just surround every single load operation with its own try / catch block.
try {
loadSoundDriver();
} catch (loadSoundDriverFailed) {
doSomethingA;
}
try {
loadUsbDriver();
} catch (loadUsbDriverFailed) {
doSomethingB;
}
// ...
So you can handle every exception by itself and continue processing the oder operations.

Categories

Resources