Develop Reference

Read two lines of a file at once in a flink streaming process

I want to process files with a flink stream in which two lines belong together. In the first line there is a header and in the second line a corresponding text.
The files are located on my local file system. I am using the readFile(fileInputFormat, path, watchType, interval, pathFilter, typeInfo) method with a custom FileInputFormat.
My streaming job class looks like this:
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Read> inputStream = env.readFile(new ReadInputFormatTest("path/to/monitored/folder"), "path/to/monitored/folder", FileProcessingMode.PROCESS_CONTINUOUSLY, 100);
inputStream.print();
env.execute("Flink Streaming Java API Skeleton");
and my ReadInputFormatTest like this:
public class ReadInputFormatTest extends FileInputFormat<Read> {
private transient FileSystem fileSystem;
private transient BufferedReader reader;
private final String inputPath;
private String headerLine;
private String readLine;
public ReadInputFormatTest(String inputPath) {
this.inputPath = inputPath;
}
#Override
public void open(FileInputSplit inputSplit) throws IOException {
FileSystem fileSystem = getFileSystem();
this.reader = new BufferedReader(new InputStreamReader(fileSystem.open(inputSplit.getPath())));
this.headerLine = reader.readLine();
this.readLine = reader.readLine();
}
private FileSystem getFileSystem() {
if (fileSystem == null) {
try {
fileSystem = FileSystem.get(new URI(inputPath));
} catch (URISyntaxException | IOException e) {
throw new RuntimeException(e);
}
}
return fileSystem;
}
#Override
public boolean reachedEnd() throws IOException {
return headerLine == null;
}
#Override
public Read nextRecord(Read r) throws IOException {
r.setHeader(headerLine);
r.setSequence(readLine);
headerLine = reader.readLine();
readLine = reader.readLine();
return r;
}
}
As expected, the headers and the text are stored together in one object. However, the file is read eight times. So the problem is the parallelization. Where and how can I specify that a file is processed only once, but several files in parallel?
Or do I have to change my custom FileInputFormat even further?

I would modify your source to emit the available filenames (instead of the actual file contents) and then add a new processor to read a name from the input stream and then emit pairs of lines. In other words, split the current source into a source followed by a processor. The processor can be made to run at any degree of parallelism and the source would be a single instance.

Invalidate Stream without Closing

This is a followup to anonymous file streams reusing descriptors
As per my previous question, I can't depend on code like this (happens to work in JDK8, for now):
RandomAccessFile r = new RandomAccessFile(...);
FileInputStream f_1 = new FileInputStream(r.getFD());
// some io, not shown
f_1 = null;
f_2 = new FileInputStream(r.getFD());
// some io, not shown
f_2 = null;
f_3 = new FileInputStream(r.getFD());
// some io, not shown
f_3 = null;
However, to prevent accidental errors and as a form of self-documentation, I would like to invalidate each file stream after I'm done using it - without closing the underlying file descriptor.
Each FileInputStream is meant to be independent, with positioning controlled by the RandomAccessFile. I share the same FileDescriptor to prevent any race conditions arising from opening the same path multiple times. When I'm done with one FileInputStream, I want to invalidate it so as to make it impossible to accidentally read from it while using the second FileInputStream (which would cause the second FileInputStream to skip data).
How can I do this?
notes:
the libraries I use require compatibiity with java.io.*
if you suggest a library (I prefer builtin java semantics if at all possible), it must be commonly available (packaged) for linux (the main target) and usable on windows (experimental target)
but, windows support isn't a absolutely required
Edit: in response to a comment, here is my workflow:
RandomAccessFile r = new RandomAccessFile(String path, "r");
int header_read;
int header_remaining = 4; // header length, initially
byte[] ba = new byte[header_remaining];
ByteBuffer bb = new ByteBuffer.allocate(header_remaining);
while ((header_read = r.read(ba, 0, header_remaining) > 0) {
header_remaining -= header_read;
bb.put(ba, 0, header_read);
}
byte[] header = bb.array();
// process header, not shown
// the RandomAccessFile above reads only a small amount, so buffering isn't required
r.seek(0);
FileInputStream f_1 = new FileInputStream(r.getFD());
Library1Result result1 = library1.Main.entry_point(f_1)
// process result1, not shown
// Library1 reads the InputStream in large chunks, so buffering isn't required
// invalidate f_1 (this question)
r.seek(0)
int read;
while ((read = r.read(byte[4096] buffer)) > 0 && library1.continue()) {
library2.process(buffer, read);
}
// the RandomAccessFile above is read in large chunks, so buffering isn't required
// in a previous edit the RandomAccessFile was used to create a FileInputStream. Obviously that's not required, so ignore
r.seek(0)
Reader r_1 = new BufferedReader(new InputStreamReader(new FileInputStream(r.getFD())));
Library3Result result3 = library3.Main.entry_point(r_2)
// process result3, not shown
// I'm not sure how Library3 uses the reader, so I'm providing buffering
// invalidate r_1 (this question) - bonus: frees the buffer
r.seek(0);
FileInputStream f_2 = new FileInputStream(r.getFD());
Library1Result result1 = library1.Main.entry_point(f_2)
// process result1 (reassigned), not shown
// Yes, I actually have to call 'library1.Main.entry_point' *again* - same comments apply as from before
// invalidate f_2 (this question)
//
// I've been told to be careful when opening multiple streams from the same
// descriptor if one is buffered. This is very vague. I assume because I only
// ever use any stream once and exclusively, this code is safe.
//

A pure Java solution might be to create a forwarding decorator that checks on each method call whether the stream is validated or not. For InputStream this decorator may look like this:
public final class CheckedInputStream extends InputStream {
final InputStream delegate;
boolean validated;
public CheckedInputStream(InputStream stream) throws FileNotFoundException {
delegate = stream;
validated = true;
}
public void invalidate() {
validated = false;
}
void checkValidated() {
if (!validated) {
throw new IllegalStateException("Stream is invalidated.");
}
}
#Override
public int read() throws IOException {
checkValidated();
return delegate.read();
}
#Override
public int read(byte b[]) throws IOException {
checkValidated();
return read(b, 0, b.length);
}
#Override
public int read(byte b[], int off, int len) throws IOException {
checkValidated();
return delegate.read(b, off, len);
}
#Override
public long skip(long n) throws IOException {
checkValidated();
return delegate.skip(n);
}
#Override
public int available() throws IOException {
checkValidated();
return delegate.available();
}
#Override
public void close() throws IOException {
checkValidated();
delegate.close();
}
#Override
public synchronized void mark(int readlimit) {
checkValidated();
delegate.mark(readlimit);
}
#Override
public synchronized void reset() throws IOException {
checkValidated();
delegate.reset();
}
#Override
public boolean markSupported() {
checkValidated();
return delegate.markSupported();
}
}
You can use it like:
CheckedInputStream f_1 = new CheckedInputStream(new FileInputStream(r.getFD()));
// some io, not shown
f_1.invalidate();
f_1.read(); // throws IllegalStateException

Under unix you could generally avoid such problems by dup'ing a file descriptor.
Since java does not not offer such a feature one option would be a native library which exposes that. jnr-posix does that for example. On the other hand jnr depends on a lot more jdk implementation properties than your original question.

Is it possible to create an URL pointing to an in-memory object?

I'm trying to extend my library for integrating Swing and JPA by making JPA config as automatic (and portable) as can be done, and it means programmatically adding <class> elements. (I know it can be done via Hibernate's AnnotationConfiguration or EclipseLInk's ServerSession, but - portability). I'd also like to avoid using Spring just for this single purpose.
I can create a persistence.xml on the fly, and fill it with <class> elements from specified packages (via the Reflections library). The problem starts when I try to feed this persistence.xml to a JPA provider. The only way I can think of is setting up a URLClassLoader, but I can't think of a way what wouldn't make me write the file to the disk somewhere first, for sole ability to obtain a valid URL. Setting up a socket for serving the file via an URL(localhost:xxxx) seems... I don't know, evil?
Does anyone have an idea how I could solve this problem? I know it sounds like a lot of work to avoid using one library, but I'd just like to know if it can be done.
EDIT (a try at being more clear):
Dynamically generated XML is kept in a String object. I don't know how to make it available to a persistence provider. Also, I want to avoid writing the file to disk.
For purpose of my problem, a persistence provider is just a class which scans the classpath for META-INF/persistence.xml. Some implementations can be made to accept dynamic creation of XML, but there is no common interface (especially for a crucial part of the file, the <class> tags).
My idea is to set up a custom ClassLoader - if you have any other I'd be grateful, I'm not set on this one.
The only easily extendable/configurable one I could find was a URLClassLoader. It works on URL objects, and I don't know if I can create one without actually writing XML to disk first.
That's how I'm setting things up, but it's working by writing the persistenceXmlFile = new File("META-INF/persistence.xml") to disk:
Thread.currentThread().setContextClassLoader(
new URLResourceClassLoader(
new URL[] { persistenceXmlFile.toURI().toURL() },
Thread.currentThread().getContextClassLoader()
)
);
URLResourceClassLoader is URLCLassLoader's subclass, which allows for looking up resources as well as classes, by overriding public Enumeration<URL> findResources(String name).

Maybe a bit late (after 4 years), but for others that are looking for a similar solution, you may be able to use the URL factory I created:
public class InMemoryURLFactory {
public static void main(String... args) throws Exception {
URL url = InMemoryURLFactory.getInstance().build("/this/is/a/test.txt", "This is a test!");
byte[] data = IOUtils.toByteArray(url.openConnection().getInputStream());
// Prints out: This is a test!
System.out.println(new String(data));
}
private final Map<URL, byte[]> contents = new WeakHashMap<>();
private final URLStreamHandler handler = new InMemoryStreamHandler();
private static InMemoryURLFactory instance = null;
public static synchronized InMemoryURLFactory getInstance() {
if(instance == null)
instance = new InMemoryURLFactory();
return instance;
}
private InMemoryURLFactory() {
}
public URL build(String path, String data) {
try {
return build(path, data.getBytes("UTF-8"));
} catch (UnsupportedEncodingException ex) {
throw new RuntimeException(ex);
}
}
public URL build(String path, byte[] data) {
try {
URL url = new URL("memory", "", -1, path, handler);
contents.put(url, data);
return url;
} catch (MalformedURLException ex) {
throw new RuntimeException(ex);
}
}
private class InMemoryStreamHandler extends URLStreamHandler {
#Override
protected URLConnection openConnection(URL u) throws IOException {
if(!u.getProtocol().equals("memory")) {
throw new IOException("Cannot handle protocol: " + u.getProtocol());
}
return new URLConnection(u) {
private byte[] data = null;
#Override
public void connect() throws IOException {
initDataIfNeeded();
checkDataAvailability();
// Protected field from superclass
connected = true;
}
#Override
public long getContentLengthLong() {
initDataIfNeeded();
if(data == null)
return 0;
return data.length;
}
#Override
public InputStream getInputStream() throws IOException {
initDataIfNeeded();
checkDataAvailability();
return new ByteArrayInputStream(data);
}
private void initDataIfNeeded() {
if(data == null)
data = contents.get(u);
}
private void checkDataAvailability() throws IOException {
if(data == null)
throw new IOException("In-memory data cannot be found for: " + u.getPath());
}
};
}
}
}

We can use the Jimfs google library for that.
First, we need to add the maven dependency to our project:
<dependency>
<groupId>com.google.jimfs</groupId>
<artifactId>jimfs</artifactId>
<version>1.2</version>
</dependency>
After that, we need to configure our filesystem behavior, and write our String content to the in-memory file, like this:
public static final String INPUT =
"\n"
+ "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
+ "<note>\n"
+ " <to>Tove</to>\n"
+ " <from>Jani</from>\n"
+ " <heading>Reminder</heading>\n"
+ " <body>Don't forget me this weekend!</body>\n"
+ "</note>";
#Test
void usingJIMFS() throws IOException {
try (var fs = Jimfs.newFileSystem(Configuration.unix())) {
var path = fs.getPath(UUID.randomUUID().toString());
Files.writeString(path, INPUT);
var url = path.toUri().toURL();
assertThat(url.getProtocol()).isEqualTo("jimfs");
assertThat(Resources.asCharSource(url, UTF_8).read()).isEqualTo(INPUT);
}
}
We can find more examples in the official repository.
If we look inside the jimfs source code we will find the implementation is similar to #NSV answer.

Dynamically skipping logging messages

I have two public methods that I'd like to trace. One of the methods calls the other repeatedly. What I'd like to do is trace only the method that was called from the outside.
Here's a simple class to demonstrate what I mean:
public class LoggingExample {
private static final Logger logger = LoggerFactory.getLogger(LoggingExample.class);
public static final String USER_ROOT = "/home/waisbrot";
/** could be called by fileExistsRobust *or* from outside */
public static boolean fileExists(String filename) {
logger.trace("Checking for file {}", filename);
File f = new File(filename);
return f.exists();
}
/** always gets called from outside */
public static boolean fileExistsRobust(String filename) {
logger.trace("Checking for any varient of {}", filename);
if (fileExists(filename))
return true;
for (String prefix : prefixes) { // this list is 100 items long
if (fileExists(prefix + filename));
return true;
}
return false;
}
}
Elsewhere in my code, I might call fileExists, in which case I want its logging message to get printed (assuming I'm tracing it). But if I call fileExistsRobost than I want that log message, but not fileExists.
I want to have both methods traced, but I'm getting buried in output when I call the second one. I was hoping Logback could be configured to understand what I want, but I'm not seeing anything useful in the documentation. I could flip a flag when I enter fileExistsRobust and then test for it in fileExists, but that's going to get ugly with more than one thread (since these are static methods) and it seems like it starts polluting the class with lots of logging infrastructure. I could use MDC to store the info, but that seems like an abuse of MDC.
Anyone run into this situation before? How'd you deal with it?

I assume that you are able to change the code. Then the simplest way in my opinion is avoiding the problem by introducing another internalFileExists(String filename) or overloading fileExists(String filename) with a logging toogle:
public static boolean fileExists(String filename, boolean doLog) {
if (doLog) logger.trace("Checking for file {}", filename);
File f = new File(filename);
return f.exists();
}
and let fileExistsRobust use the overloaded version with doLog = false, while the single argument version redirects to fileExists(filename, true).
That does not really address the problem, but mitigates it.

How do I write a unit test to cover the case where an IOException is thrown?

I have the following class:
public class FileLoader {
private Map<Brand, String> termsOfUseText = new HashMap<Brand, String>();
public void load() {
for (Brand brand : Brand.values()) {
readAndStoreTermsOfUseForBrand(brand);
}
}
private void readAndStoreTermsOfUseForBrand(Brand brand) {
String resourceName = "termsOfUse/" + brand.name().toLowerCase() + ".txt";
InputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
try {
String content = IOUtils.toString(in);
termsOfUseText.put(brand, content);
} catch (IOException e) {
throw new IllegalStateException(String.format("Failed to find terms of use source file %s", resourceName),e);
}
}
public String getTextForBrand(Brand brand) {
return termsOfUseText.get(brand);
}
}
Brand is an enum, and I need all the valid .txt files to be on the classpath. How do I make the IOException occur, given that the Brand enum contains all the valid brands and therfore all the .txt files for them exist?
Suggestions around refactoring the current code are welcome if it makes it more testable!

Three options I see right off:
Use PowerMock to mock IOUtils.toString(). I consider PowerMock to be quite a last resort. I'd rather refactor the source to something a little more test-friendly.
Extract the IOUtils call to a protected method. Create a test-specific subclass of your class that overrides this method and throws the IOException.
Extract the InputStream creation to a protected method. Create a test-specific subclass to override the method and return a mock InputStream.

I would suggest a bit of refactoring. All your methods are void, this usually means they are not functional.
For example, you can extract this functionality:
private String readTermsOfUseForBrand(InputStream termsOfUserIs) {
try {
String content = IOUtils.toString(in);
return content;
} catch (IOException e) {
throw new IllegalStateException(String.format("Failed to find terms of use source file %s", resourceName), e);
}
return null;
}
So that we can assert on the String result in our tests.
Of course this is not functional code, as it reads from an Input Stream. And it does so with IOUtils.toString() method that cannot be mocked easily (well, there's PowerMock but as Ryan Stewart said it's the last resort).
To test IO exceptions you can create a failing input stream (tested with JDK7):
public class FailingInputStream extends InputStream {
#Override
public int read() throws IOException {
throw new IOException("Test generated exception");
}
}
And test like that:
#Test
public void testReadTermsOfUseForBrand() {
FileLoader instance = new FileLoader();
String result = instance.readTermsOfUseForBrand(new FailingInputStream());
assertNull(result);
}

Missing file will cause NullPointerException because getResourceAsStream will return null and you will have in==null. IOException in this case may actually be pretty rare. If it's critical for you to see it, I can only think of instrumenting this code to throw it if code is executed in test scope. But is it really that important?

I would use a mock to accomplish this.
Example (untested, just to give you some thought):
#Test(expected=IllegalStateException.class)
public void testThrowIOException() {
PowerMockito.mockStatic(IOUtils.class);
PowerMockito.when(IOUtils.toString()).thenThrow(
new IOException("fake IOException"));
FileLoader fileLoader = new FileLoader();
Whitebox.invokeMethod(fileLoader,
"readAndStoreTermsOfUseForBrand", new Brand(...));
// If IllegalStateException is not thrown then this test case fails (see "expected" above)
}

Code below is completely untested
To cause the IOException use:
FileInputStream in = this.getClass().getClassLoader().getResourceAsStream(resourceName);
in.mark(0);
//read some data
in.reset(); //IOException
To test the IOException case use:
void test
{
boolean success = false;
try
{
//code to force ioException
}
catch(IOException ioex)
{
success = true;
}
assertTrue(success);
}
In JUnit4
#Test(expected=IOException.class)
void test
{
//code to force ioException
}
Other JUnit
void test
{
try
{
//code to force IOException
fail("If this gets hit IO did not occur, fail test");
}
catch(IOException ioex)
{
//success!
}
}