I am trying to use Spring batch and implement an aggregated reader (batch file, where multiple records should be treated as one record while writing). Here is the code snippet for my reader:
public class AggregatePeekableReader implements ItemReader<List<T>>, ItemStream {
private SingleItemPeekableItemReader<T> reader;
private boolean process(T currentRecord , InvoiceLineItemsHolder holder) throws UnexpectedInputException, ParseException, Exception {
next = peekNextInvoiceRecord();
// finish processing if we hit the end of file
if (currentRecord == null ) {
LOG.info("Exhausted ItemReader ( END OF FILE)");
holder.exhausted = true;
return false;
}
if ( currentRecord.hasSameInvoiceNumberAndVendorNumber(next)){
LOG.info("Found new line item to current invocie record");
holder.records.add(currentRecord);
currentRecord = null;
return true;
}else{
holder.records.add(currentRecord);
return false;
}
}
private T getNextInvoiceRecord () {
T record=null;
try {
record=reader.read();
} catch (UnexpectedInputException e) {
ALERT.error(LogMessageFormatter.format(Severity.HIGH,
BATCH_FILE_READ_EXCEPTION, e), e);
throw e;
} catch (ParseException e) {
ALERT.error(LogMessageFormatter.format(Severity.HIGH,
BATCH_FILE_READ_EXCEPTION, e), e);
throw e;
} catch (Exception e) {
ALERT.error(LogMessageFormatter.format(Severity.HIGH,
BATCH_FILE_READ_EXCEPTION, e), e);
}
return record;
}
private T peekNextInvoiceRecord() {
T next=null;
try {
next=reader.peek();
} catch (UnexpectedInputException e) {
ALERT.error(LogMessageFormatter.format(Severity.HIGH,
BATCH_FILE_READ_EXCEPTION, e), e);
throw e;
} catch (ParseException e) {
ALERT.error(LogMessageFormatter.format(Severity.HIGH,
BATCH_FILE_READ_EXCEPTION, e), e);
throw e;
} catch (Exception e) {
ALERT.error(LogMessageFormatter.format(Severity.HIGH,
BATCH_FILE_READ_EXCEPTION, e), e);
}
return next;
}
public void close () {
reader.close();
}
public SingleItemPeekableItemReader<T> getReader() {
return reader;
}
public void setReader(SingleItemPeekableItemReader<T> reader) {
this.reader = reader;
}
private class InvoiceLineItemsHolder {
List<T> records = new ArrayList<T>();
boolean exhausted = false;
}
#Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
//
reader.open(executionContext);
}
#Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
// TODO
}
#Override
public List<T> read() throws Exception, UnexpectedInputException, ParseException,
NonTransientResourceException {
CLASS holder = new SOMECLASS()
synchronized (this) {
while (process(getNextInvoiceRecord(), holder)) {
continue;
}
if (!holder.exhausted) {
return holder.records;
} else {
//When you hit the end of the file,close the reader.
close();
return null;
}
}
}
}
The above is a working example for implementing a peekable reader.This peeks the next line
(doesnt read it) and determines whether a logical end of line is reached (some times
multiple lines can make up a single transaction)
You need to implement ItemStream interface for reader. This will give a hint to Spring Batch, that your reader requires some actions to open/close a stream:
public class InvoiceLineItemAggregatePeekableReader extends AbstractItemStreamItemReader<List<SAPInvoicePaymentRecord>> {
#Override
public void close() {
...
}
}
Streams are closed whatever error occurred during step execution. For more examples check classes from Spring Batch itself (e.g. FlatFileItemReader).
I can't move the input file to to an Error folder because the Reader
is not closed
you could copy the file and either use File.deleteOnExit() on the old file for later deletion or delete the old file in an extra step, e.g. with a simple tasklet and a flow which calls the deleteTaskletStep only if the business step had an exception
Related
I have this bit of code which depends from a custom Exception thrown by a function inside findID() it throws a NoClientFound Exception that I made whenever this mentioned function returns a null (The client does not exist).
The IDE suggests that I shall apply that Exception into the code, but in this bit of code, where I need the ID to be null (unique IDs) I "can't catch that exception" since if I catch it, the function will not be executed as intended.
Question: How I can manage this?
Function with the Exception problem
public boolean add(Client c) {
StringBuilder sb = new StringBuilder();
boolean added = false;
try {
if (findID(c.getID()) == null) {
try (BufferedWriter bw = new BufferedWriter(
new FileWriter(fitxer, true));) {
//Add client to file
bw.write(sb.append(c.getID()).append(SEPARADOR).
append(c.getName()).toString());
bw.newLine();//New line
bw.flush(); //Push to file
added = true;
} catch (IOException e) {
Logger.getLogger(DaoClient.class.getName()).log(Level.SEVERE,
null, "Error appeding data to file" + e);
}
}
} catch (IOException ex) {
Logger.getLogger(DaoClient.class.getName()).log(Level.SEVERE, null,
"Error appeding data to file" + ex);
} finally {
}
return addded;
}
Exception Code
public class NoClientFound extends Exception {
private String msg;
public NoClientFound() {
super();
}
public NoClientFound(String msg) {
super(msg);
this.msg = msg;
}
#Override
public String toString() {
return msg;
}
You can catch that exception and handle it accordingly. When you catch NoClientFound exception that means findID(c.getID()) is null. So without handling that in the if block you can handle that within the catch block.
public boolean add(Client c) {
StringBuilder sb = new StringBuilder();
boolean added = false;
try {
// call the function
findID(c.getID());
} catch (NoClientFound ex) {
//handle the NoClientFound exception as you like here
BufferedWriter bw = new BufferedWriter(
new FileWriter(fitxer, true));
//Add client to file
bw.write(sb.append(c.getID()).append(SEPARADOR).
append(c.getName()).toString());
bw.newLine();//New line
bw.flush(); //Push to file
added = true;
}catch (IOException ex) {
Logger.getLogger(DaoClient.class.getName()).log(Level.SEVERE, null,
"Error appeding data to file" + ex);
}finally {
}
return addded;
}
I assume you already have a null check on findID(...)
if( c == null || findID(c.getID()) == null){
throw new NoClientFound("Client not found!");
}else{
//add your file writing operation
}
and Also in NoClientFound class extend it from RuntimeException, not the Exception.
public class NoClientFound extends RuntimeException {
...
}
Caller method:
public void caller(){
Client client = new Client();
client.setId(1);
...
try{
add(client);
}catch(NoClientFound ex){
//client not found then create one for ex...
}
catch(Exception ex){
//somthing else happend
log.error(ex.getmessge());
}
}
I have just read, that in Java the classes PrintStream and PrintWriter don't throw checked exceptions. Instead they are using a kind of an error flag which I can read invoking the method boolean checkError() (API link).
Now, I am asking myself how to find out the reason why the exception occurred. The information that there was an exception is sometimes maybe not enough, or?
Based on the source code, it looks like they discard the exception. All of the catch blocks look like this:
try {
...
}
catch (IOException x) {
trouble = true; // (x is ignored)
}
So the most straightforward solution is probably to not use PrintStream, if possible.
One workaround could be to extend PrintStream and wrap the output in another OutputStream which captures the exception before PrintStream catches (and discards) it. Something like this:
package mcve.util;
import java.io.*;
public class PrintStreamEx extends PrintStream {
public PrintStreamEx(OutputStream out) {
super(new HelperOutputStream(out));
}
/**
* #return the last IOException thrown by the output,
* or null if there isn't one
*/
public IOException getLastException() {
return ((HelperOutputStream) out).lastException;
}
#Override
protected void clearError() {
super.clearError();
((HelperOutputStream) out).setLastException(null);
}
private static class HelperOutputStream extends FilterOutputStream {
private IOException lastException;
private HelperOutputStream(OutputStream out) {
super(out);
}
private IOException setLastException(IOException e) {
return (lastException = e);
}
#Override
public void write(int b) throws IOException {
try {
super.write(b);
} catch (IOException e) {
throw setLastException(e);
}
}
#Override
public void write(byte[] b) throws IOException {
try {
super.write(b);
} catch (IOException e) {
throw setLastException(e);
}
}
#Override
public void write(byte[] b, int off, int len) throws IOException {
try {
super.write(b, off, len);
} catch (IOException e) {
throw setLastException(e);
}
}
#Override
public void flush() throws IOException {
try {
super.flush();
} catch (IOException e) {
throw setLastException(e);
}
}
#Override
public void close() throws IOException {
try {
super.close();
} catch (IOException e) {
throw setLastException(e);
}
}
}
}
I have a similar problem as asked here - How to disable Redis Caching at run time if redis connection failed. My application is using #Cacheable at the service layer for most of the database/static resources call.
Cache is backed by Couchbase and whenever application fails to connect Couchbase node application goes down. Which is what we are not expecting, we expect data should be served from the source system whenever connection failed.
We tried implementing CacheErrorHandler but it does not work as expected because we want to execute the actual method which is making a service call and return the response rather than logging the Cache fail, basically bypassing the cache and as soon as the Couchbase node is up or connection established get the data from cache.
Any idea how we can achieve it?
Thanks #Daniel Bickler for the suggestion, below is the implementation I written referring #John Blum answer.
CouchbaseCustomCacheManager:
import java.util.Map;
import org.springframework.cache.Cache;
import com.couchbase.client.spring.cache.CacheBuilder;
import com.couchbase.client.spring.cache.CouchbaseCacheManager;
public class CouchbaseCustomCacheManager extends CouchbaseCacheManager {
public CouchbaseCustomCacheManager(
final Map<String, CacheBuilder> initialCaches) {
super(initialCaches);
}
#Override
public Cache getCache(String name) {
return new CouchbaseCacheWrapper(super.getCache(name));
}
protected static class CouchbaseCacheWrapper implements Cache {
private final Cache delegate;
public CouchbaseCacheWrapper(Cache couchbaseCache) {
this.delegate = couchbaseCache;
}
#Override
public String getName() {
try {
return delegate.getName();
} catch (Exception e) {
return null;
}
}
#Override
public Object getNativeCache() {
try {
return delegate.getNativeCache();
} catch (Exception e) {
return null;
}
}
#Override
public ValueWrapper get(Object key) {
try {
return delegate.get(key);
} catch (Exception e) {
return null;
}
}
#Override
public <T> T get(Object key, Class<T> type) {
try {
return delegate.get(key, type);
} catch (Exception e) {
return null;
}
}
#Override
public void put(Object key, Object value) {
try {
delegate.put(key, value);
} catch (Exception e) {
try {
handleErrors(e);
} catch (Exception e1) {
}
}
}
#Override
public ValueWrapper putIfAbsent(Object key, Object value) {
try {
return delegate.putIfAbsent(key, value);
} catch (Exception e) {
return null;
}
}
#Override
public void evict(Object key) {
try {
delegate.evict(key);
} catch (Exception e) {
try {
handleErrors(e);
} catch (Exception e1) {
}
}
}
#Override
public void clear() {
try {
delegate.clear();
} catch (Exception e) {
try {
handleErrors(e);
} catch (Exception e1) {
}
}
}
protected <T> T handleErrors(Exception e) throws Exception {
if (e instanceof Exception) {
return null;
} else {
throw e;
}
}
}
}
And used it as:
#Bean
public CacheManager cacheManager() {
final Map<String, CacheBuilder> cache = new HashMap<>();
for (final String appCache : "127.0.0.1,127.0.0.2,127.0.0.3".split(",")) {
cache.put(appCache, CacheBuilder.newInstance(CouchbaseCluster.create().openBucket(
"default", "")));
}
return new CouchbaseCustomCacheManager(cache);
}
I have made this abstract class to automatically retry network calls if some exception is thrown.
I take care to not retry after InterruptedException &
UnknownHostException.
I retry 5 times. After each failure
I perform an exponential back off, starting from 300ms going upto
1500ms.
public abstract class AutoRetry {
private Object dataToReturn = null;
public Object getDataToReturn() {
return this.dataToReturn;
}
public AutoRetry() {
short retry = -1;
while (retry++ < StaticData.NETWORK_RETRY) {
try {
Thread.sleep(retry * StaticData.NETWORK_CALL_WAIT);
this.dataToReturn = doWork();
break;
} catch (InterruptedException | UnknownHostException e) {
e.printStackTrace();
this.dataToReturn = null;
return;
} catch (IOException e) {
e.printStackTrace();
}
}
}
protected abstract Object doWork() throws IOException;
}
I use it as follows :
final Object dataAfterWork = new AutoRetry() {
#Override
protected Object doWork() throws IOException {
return; //a network call which returns something
}
}.getDataToReturn();
So is this implementation good/correct ?
EDIT
moved to https://codereview.stackexchange.com/questions/87686
This looks pretty good, but I would split the running task from the retry. Also use generics, don't just throw Object about.
Use a Java 8 lambda and the return of the method:
public static <T> Optional<T> doWithRetry(final Supplier<T> t) {
for (int retry = 0; retry <= StaticData.NETWORK_RETRY; ++retry) {
try {
Thread.sleep(retry * StaticData.NETWORK_CALL_WAIT);
return Optional.of(t.get());
} catch (InterruptedException | UnknownHostException e) {
LOGGER.log(Level.SEVERE, "Call failed.", e);
return Optional.empty();
} catch (IOException e) {
LOGGER.log(Level.WARNING, "Call failed. Retry.", e);
}
}
LOGGER.log(Level.SEVERE, "Call failed. Retries exceeded.");
return Optional.empty();
}
Also, use a real logger, not printStackTrace...
Usage:
final String data = doWithRetry(() -> {
//do stuff
});
If your lambda needs to throw an exception, you'll need to define your own #FunctionalInterface:
#FunctionalInterface
interface StuffDoer<T> {
T doStuff() throws Exception;
}
And use that in the method signature, you'll need to handle generic Exception.
Pre-Java 8 usage:
final String data = doWithRetry(new StuffDoer<T>() {
#Override
public T get() throws Exception {
return null;
}
});
For one of my projects, I implement a Java 7 FileSystem over the Box API Java SDK (the new one).
However, for downloading files, when you want to have a stream to the content, it only provides methods taking OutputStream as an argument; specifically, I am using this one at the moment.
But this doesn't sit well with the JDK API; I need to be able to implement FileSystemProvider#newInputStream()... Therefore I elected to use Pipe{Input,Output}Stream.
Moreover, since the Box SDK API methods are synchronous (not that it matters here), I wrap them in a Future. My code is as follows (imports ommitted for brevity):
#ParametersAreNonnullByDefault
public final class BoxFileInputStream
extends InputStream
{
private final Future<Void> future;
private final PipedInputStream in;
public BoxFileInputStream(final ExecutorService executor,
final BoxFile file)
{
in = new PipedInputStream(16384);
future = executor.submit(new Callable<Void>()
{
#Override
public Void call()
throws IOException
{
try {
file.download(new PipedOutputStream(in));
return null;
} catch (BoxAPIException e) {
throw BoxIOException.wrap(e);
}
}
});
}
#Override
public int read()
throws IOException
{
try {
return in.read();
} catch (IOException e) {
future.cancel(true);
throw new BoxIOException("download failure", e);
}
}
#Override
public int read(final byte[] b)
throws IOException
{
try {
return in.read(b);
} catch (IOException e) {
future.cancel(true);
throw new BoxIOException("download failure", e);
}
}
#Override
public int read(final byte[] b, final int off, final int len)
throws IOException
{
try {
return in.read(b, off, len);
} catch (IOException e) {
future.cancel(true);
throw new BoxIOException("download failure", e);
}
}
#Override
public long skip(final long n)
throws IOException
{
try {
return in.skip(n);
} catch (IOException e) {
future.cancel(true);
throw new BoxIOException("download failure", e);
}
}
#Override
public int available()
throws IOException
{
try {
return in.available();
} catch (IOException e) {
future.cancel(true);
throw new BoxIOException("download failure", e);
}
}
#Override
public void close()
throws IOException
{
IOException streamException = null;
IOException futureException = null;
try {
in.close();
} catch (IOException e) {
streamException = e;
}
try {
future.get(5L, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
futureException = new BoxIOException("donwload interrupted", e);
} catch (ExecutionException e) {
futureException = new BoxIOException("download failure",
e.getCause());
} catch (CancellationException e) {
futureException = new BoxIOException("download cancelled", e);
} catch (TimeoutException e) {
futureException = new BoxIOException("download timeout", e);
}
if (futureException != null) {
if (streamException != null)
futureException.addSuppressed(streamException);
throw futureException;
}
if (streamException != null)
throw streamException;
}
#Override
public synchronized void mark(final int readlimit)
{
in.mark(readlimit);
}
#Override
public synchronized void reset()
throws IOException
{
try {
in.reset();
} catch (IOException e) {
future.cancel(true);
throw new BoxIOException("download failure", e);
}
}
#Override
public boolean markSupported()
{
return in.markSupported();
}
}
The code consistenly fails with the following stack trace (that is in int read(byte[]):
Exception in thread "main" com.github.fge.filesystem.box.exceptions.BoxIOException: download failure
at com.github.fge.filesystem.box.io.BoxFileInputStream.read(BoxFileInputStream.java:81)
at java.nio.file.Files.copy(Files.java:2735)
at java.nio.file.Files.copy(Files.java:2854)
at java.nio.file.CopyMoveHelper.copyToForeignTarget(CopyMoveHelper.java:126)
at java.nio.file.Files.copy(Files.java:1230)
at Main.main(Main.java:37)
[ IDEA specific stack trace elements follow -- irrelevant]
Caused by: java.io.IOException: Pipe broken
at java.io.PipedInputStream.read(PipedInputStream.java:322)
at java.io.PipedInputStream.read(PipedInputStream.java:378)
at java.io.InputStream.read(InputStream.java:101)
at com.github.fge.filesystem.box.io.BoxFileInputStream.read(BoxFileInputStream.java:78)
... 10 more
But when it fails, the download is already complete...
OK, the thing is, I can grab the file size and hack around it but I'd prefer not to if at all possible; how can I modify this code so as to avoid EPIPE?
The SDK also provides BoxAPIRequest and BoxAPIResponse classes that let you make manual requests for advanced use-cases. These classes still automatically handle authentication, errors, back-off, etc. but give you more granular control over the request.
In your case, you could do make a download request manually by doing:
// Note: this example assumes you already have a BoxAPIConnection.
URL url = new URL("files/" + file.getID() + "/content")
BoxAPIRequest request = new BoxAPIRequest(api, url, "GET");
BoxAPIResponse response = request.send();
InputStream bodyStream = response.getBody();
// Use the stream.
response.disconnect();
Well, I found the solution, although I am not very satisfied with it...
Since I can know the file size which I try to open an inputstream on, I just pick the size and decrease it by the amount of bytes read -- unless the size reaches 0, in this case all read methods return -1.