Develop Reference

Spring batch FlatFileItemWriter write as csv from Object

I am using Spring batch and have an ItemWriter as follows:
public class MyItemWriter implements ItemWriter<Fixing> {
private final FlatFileItemWriter<Fixing> writer;
private final FileSystemResource resource;
public MyItemWriter () {
this.writer = new FlatFileItemWriter<>();
this.resource = new FileSystemResource("target/output-teste.txt");
}
#Override
public void write(List<? extends Fixing> items) throws Exception {
this.writer.setResource(new FileSystemResource(resource.getFile()));
this.writer.setLineAggregator(new PassThroughLineAggregator<>());
this.writer.afterPropertiesSet();
this.writer.open(new ExecutionContext());
this.writer.write(items);
}
#AfterWrite
private void close() {
this.writer.close();
}
}
When I run my spring batch job, the items are written to file as:
Fixing{id='123456', source='TEST', startDate=null, endDate=null}
Fixing{id='1234567', source='TEST', startDate=null, endDate=null}
Fixing{id='1234568', source='TEST', startDate=null, endDate=null}
1/ How can I write just the data so that the values are comma separated and where it is null, it is not written. So the target file should look like this:
123456,TEST
1234567,TEST
1234568,TEST
2/ Secondly, I am having an issue where only when I exit spring boot application, I am able to see the file get created. What I would like is once it has processed all the items and written, the file to be available without closing the spring boot application.

There are multiple options to write the csv file. Regarding second question writer flush will solve the issue.
https://howtodoinjava.com/spring-batch/flatfileitemwriter-write-to-csv-file/
We prefer to use OpenCSV with spring batch as we are getting more speed and control on huge file example snippet is below
class DocumentWriter implements ItemWriter<BaseDTO>, Closeable {
private static final Logger LOG = LoggerFactory.getLogger(StatementWriter.class);
private ColumnPositionMappingStrategy<Statement> strategy ;
private static final String[] columns = new String[] { "csvcolumn1", "csvcolumn2", "csvcolumn3",
"csvcolumn4", "csvcolumn5", "csvcolumn6", "csvcolumn7"};
private BufferedWriter writer;
private StatefulBeanToCsv<Statement> beanToCsv;
public DocumentWriter() throws Exception {
strategy = new ColumnPositionMappingStrategy<Statement>();
strategy.setType(Statement.class);
strategy.setColumnMapping(columns);
filename = env.getProperty("globys.statement.cdf.path")+"-"+processCount+".dat";
File cdf = new File(filename);
if(cdf.exists()){
writer = Files.newBufferedWriter(Paths.get(filename), StandardCharsets.UTF_8,StandardOpenOption.APPEND);
}else{
writer = Files.newBufferedWriter(Paths.get(filename), StandardCharsets.UTF_8,StandardOpenOption.CREATE_NEW);
}
beanToCsv = new StatefulBeanToCsvBuilder<Statement>(writer).withQuotechar(CSVWriter.NO_QUOTE_CHARACTER)
.withMappingStrategy(strategy).withSeparator(',').build();
}
#Override
public void write(List<? extends BaseDTO> items) throws Exception {
List<Statement> settlementList = new ArrayList<Statement>();
for (int i = 0; i < items.size(); i++) {
BaseDTO baseDTO = items.get(i);
settlementList.addAll(baseDTO.getStatementList());
}
beanToCsv.write(settlementList);
writer.flush();
}
#PreDestroy
#Override
public void close() throws IOException {
writer.close();
}
}

Since you are using PassThroughLineAggregator which does item.toString() for writing the object, overriding the toString() function of classes extending Fixing.java should fix it.

1/ How can I write just the data so that the values are comma separated and where it is null, it is not written.
You need to provide a custom LineAggregator that filters out null fields.
2/ Secondly, I am having an issue where only when I exit spring boot application, I am able to see the file get created
This is probably because you are calling this.writer.open in the write method which is not correct. You need to make your item writer implement ItemStream and call this.writer.open and this this.writer.close respectively in ItemStream#open and ItemStream#close

Is there a way to get the Apache Commons FileAlterationMonitor to only alert once for a batch of incoming files?

I am monitoring several (about 15) paths for incoming files using the Apache Commons FileAlterationMonitor. These incoming files can come in batches of anywhere between 1 and 500 files at a time. I have everything set up and the application monitors the folders as expected, I have it set to poll the folders every minute. My issue is that, as expected, the listener that I have set up alerts for each incoming file when all I really need, and want, is to know when a new batch of files come in. So I would like to receive a single alert as opposed to up to 500 at a time.
Does anyone have any ideas for how to control the number of alerts or only pick up the first or last notification or something to that effect? I would like to stick with the FileAlterationMonitor if at all possible because it will be running for long periods and so far from what I can tell in testing is that it doesn't seem to put a heavy load on the system or slow the rest of the application down. But I am definitely open to other ideas if what I'm looking for isn't possible with the FileAlterationMonitor.
public class FileMonitor{
private final String newDirectory;
private FileAlterationMonitor monitor;
private final Alerts gui;
private final String provider;
public FileMonitor (String d, Alerts g, String pro) throws Exception{
newDirectory = d;
gui = g;
provider = pro;
}
public void startMonitor() throws Exception{
// Directory to monitor
final File directory = new File(newDirectory);
// create new observer
FileAlterationObserver fao = new FileAlterationObserver(directory);
// add listener to observer
fao.addListener(new FileAlterationListenerImpl(gui, provider));
// wait 1 minute between folder polls.
monitor = new FileAlterationMonitor(60000);
monitor.addObserver(fao);
monitor.start();
}
}
public class FileAlterationListenerImpl implements FileAlterationListener{
private final Alerts gui;
private final String provider;
private final LogFiles monitorLogs;
public FileAlterationListenerImpl(Alerts g, String pro){
gui = g;
provider = pro;
monitorLogs = new LogFiles();
}
#Override
public void onStart(final FileAlterationObserver observer){
System.out.println("The FileListener has started on: " + observer.getDirectory().getAbsolutePath());
}
#Override
public void onDirectoryCreate(File file) {
}
#Override
public void onDirectoryChange(File file) {
}
#Override
public void onDirectoryDelete(File file) {
}
#Override
public void onFileCreate(File file) {
try{
switch (provider){
case "Spectrum": gui.alertsAreaAppend("New/Updated schedules available for Spectrum zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for Spectrum zones!\r\n");
break;
case "DirecTV ZTA": gui.alertsAreaAppend("New/Updated schedules available for DirecTV ZTA zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for DirecTV ZTA zones!\r\n");
break;
case "DirecTV RSN": gui.alertsAreaAppend("New/Updated schedules available for DirecTV RSN zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for DirecTV RSN zones!\r\n");
break;
case "Suddenlink": gui.alertsAreaAppend("New/Updated schedules available for Suddenlink zones!\r\n");
monitorLogs.appendNewLogging("New/Updated schedules available for Suddenlink zones!\r\n");
break;
}
}catch (IOException e){}
}
#Override
public void onFileChange(File file) {
}
Above is the FileMonitor class and overridden FileAlterationListener I have so far.
Any suggestions would be greatly appreciated.

Here's a quick and crude implementation:
public class FileAlterationListenerAlterThrottler {
private static final int DEFAULT_THRESHOLD_MS = 5000;
private final int thresholdMs;
private final Map<String, Long> providerLastFileProcessedAt = new HashMap<>();
public FileAlterationListenerAlterThrottler() {
this(DEFAULT_THRESHOLD_MS);
}
public FileAlterationListenerAlterThrottler(int thresholdMs) {
this.thresholdMs = thresholdMs;
}
public synchronized boolean shouldAlertFor(String provider) {
long now = System.currentTimeMillis();
long last = providerLastFileProcessedAt.computeIfAbsent(provider, x -> 0l);
if (now - last < thresholdMs) {
return false;
}
providerLastFileProcessedAt.put(provider, now);
return true;
}
}
And a quicker and cruder driver:
public class Test {
public static void main(String[] args) throws Exception {
int myThreshold = 1000;
FileAlterationListenerAlterThrottler throttler = new FileAlterationListenerAlterThrottler(myThreshold);
for (int i = 0; i < 3; i++) {
doIt(throttler);
}
Thread.sleep(1500);
doIt(throttler);
}
private static void doIt(FileAlterationListenerAlterThrottler throttler) {
boolean shouldAlert = throttler.shouldAlertFor("Some Provider");
System.out.println("Time now: " + System.currentTimeMillis());
System.out.println("Should alert? " + shouldAlert);
System.out.println();
}
}
Yields:
Time now: 1553739126557
Should alert? true
Time now: 1553739126557
Should alert? false
Time now: 1553739126557
Should alert? false
Time now: 1553739128058
Should alert? true

Apache Camel: Get move path after file transfer

I am creating a file transfer route which is using move to set a dynamic path where the file is moved after successful file transfer. I have also setup a notifier to keep track of file transfer events.
As the move path is dynamic, I need to get the evaluated path where file was moved after the file transfer. How can this path inside the notifier?
public class MyFtpServiceBuilder extends RouteBuilder {
#Override
public void configure() throws Exception {
getContext()
.getManagementStrategy()
.addEventNotifier(new MyFtpServiceNotifier());
from("file:C:/tmp/inputfolder?move=archive/${date:now:yyyyMMdd}/${file:onlyname}")
.routeId("myRoute")
.to("file:C:/tmp/outputfolder")
}
}
public class MyFtpServiceNotifier extends EventNotifierSupport {
#Override
public void notify(EventObject event) throws Exception {
Exchange exchange = ((AbstractExchangeEvent) event).getExchange();
if (event instanceof ExchangeSentEvent) {
// Want to get here the path where file was moved
}
}
#Override
public boolean isEnabled(EventObject event) {
return event instanceof AbstractExchangeEvent;
}
}

You could try to add a header to the exchange, something like:
.setHeader("FILE_PROCESSED").simple("archive/${date:now:yyyyMMdd}/${file:onlyname}")

I'm not sure how to do it in a notifier, but you can always add a Processor and use the FileEndpoint.getMove().evaluate() method on the exchange to obtain the final moved file path.
For example:
#Override
public void process(final Exchange exchange) throws Exception {
File movedFile = null;
if (exchange.getFromEndpoint() instanceof FileEndpoint) {
FileEndpoint fileEndpoint = (FileEndpoint) exchange.getFromEndpoint();
String movePath = fileEndpoint.getMove().evaluate(exchange, String.class);
File inputDir = fileEndpoint.getFile();
movedFile = new File(inputDir, movePath);
}
}

java.util.logging assign logger to a specific package

I implemented two customized handler to log information on DB and an additional flat file (DBHandler and MyFileHandler). This new log handlers will be used by a single class on a specific package.
I attached the two new loggers to a specific package only.
The idea is to switch between this two handlers (file and database)for the classes contaiend on a specific package, but currently with the current configuration I could not do that, so either I am logging with both handler either there is no log at all.
I tried to set the log level for DB handler to off but it is still logging normally on DB.
below the configuration file is use logging.properties
############################################################
##### Global properties
############################################################
handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler, com.test.logging.DBHandler, com.test.logging.MyFileHandler
.level = INFO
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.pattern = %t/CLog%g.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
com.test.logging.MyFileHandler.level = ALL
com.test.logging.MyFileHandler.pattern = %t/custLog%g.log
com.test.logging.MyFileHandler.limit = 50000
com.test.logging.MyFileHandler.count = 1
com.test.logging.MyFileHandler.formatter = java.util.logging.SimpleFormatter
com.test.logging.DBHandler.level=OFF
com.test.ccb.mon.handlers=com.test.logging.DBHandler, com.test.logging.MyFileHandler
The class using the logger to track he inforamtion is below
package com.test.ccb.mon;
public class Utils {
public static final Logger logger = Logger.getLogger(Utils.class.getCanonicalName());
public void logging()
{
//processing
logger.info("message);
}
}
DBHandler class:
public class DBHandler extends Handler {
#Override
public void close() throws SecurityException {
}
#Override
public void flush() {
}
#Override
public void publish(LogRecord logRecord) {
if (isLoggable(logRecord))
{
try {
//SQL call to insert onDB
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
MyFileHandler class:
public class MyFileHandler extends FileHandler{
public MyileHandler() throws IOException, SecurityException {
super();
}
#Override
public void close() throws SecurityException {
super.close();
}
#Override
public void flush() {
super.flush();
}
#Override
public void publish(LogRecord record) {
super.publish(record);
}
}

The Handler class doesn't read any properties from the LogManager by default. You have to code that logic in all of your subclasses.
public class DBHandler extends Handler {
public DBHandler() {
LogManager m = LogManager.getLogManager();
String p = getClass().getName();
String v = m.getProperty(p + ".level");
try {
if (v != null) {
super.setLevel(Level.parse(v));
}
} catch (RuntimeException re) {
reportError(v, re, ErrorManager.OPEN_FAILURE);
}
//#todo create code to parse filter, formatter, encoding, etc.
}
#Override
public void close() throws SecurityException {
}
#Override
public void flush() {
}
#Override
public void publish(LogRecord logRecord) {
if (isLoggable(logRecord)) {
try {
//SQL call to insert onDB
} catch (Exception e) {
reportError("", e, ErrorManager.WRITE_FAILURE);
}
}
}
}

Reproducing your problem is not so easy for me. With handler classes similar to yours, changes to the configuration file have the expected effect. With the DBHandler.level=OFF setting, the database handler output is missing for me:
Aug 11, 2015 1:47:26 PM com.test.ccb.mon.Utils logging
DBHandler.publish - handler level: OFF; log record level: INFO
INFO: message
MyFileHandler - message
Logging handlers:
###java.util.logging.FileHandler-ALL
###java.util.logging.ConsoleHandler-ALL
###com.test.logging.DBHandler-OFF
###com.test.logging.MyFileHandler-ALL
Your debug code to print the logging handlers is now also included in the following main method to your Utils class. You could run this method yourself, to see whether this way of reading the configuration file works better for you:
public static void main(final String[] arguments) throws IOException
{
final String fileName = "logging.properties";
final InputStream propertiesStream = Utils.class.getResourceAsStream(fileName);
//final InputStream propertiesStream = new FileInputStream("path to file");
LogManager.getLogManager().readConfiguration(propertiesStream);
new Utils().logging();
System.out.println();
// No handlers for this logger directly, but four for its parent.
System.out.println("Logging handlers:");
for (final Handler handler : logger.getParent().getHandlers())
System.out.println("###" + handler.getClass().getName()
+ "-" + handler.getLevel());
}
A very simple version of your DBHandler class could look like this (please note the if (isLoggable(record)) check in the publish method):
package com.test.logging;
import java.util.logging.*;
/**
* Logging handler that stores logging in the database.
*/
public class DBHandler extends Handler {
#Override
public void publish(final LogRecord record) {
System.out.println("DBHandler.publish - handler level: " + getLevel()
+ "; log record level: " + record.getLevel());
if (isLoggable(record))
System.out.println(getClass().getSimpleName() + " - " + record.getMessage());
}
#Override
public void flush() {
// Empty.
}
#Override
public void close() throws SecurityException {
// Empty.
}
}

Vaadin Upload component how get fileName before submitUpload?

I try make implementation for comparing the files before they are uploaded.
If file whith name is exist in system ask about create new version or just override it.
Here is the problem, how to get file name?
I can't use receiveUpload(), because after this method file is remove from upload component ?

The problem is that once you start an upload using the Upload component, it can only be interrupted by calling the interruptUpload() method, and you cannot resume anytime later.
The interruption is permanent.
This means you cannot pause in the middle of the upload to see if you already have the file in your system. You have to upload the file all the way.
Considering this drawback, you can sill check in your system if you have the file, after the upload finishes. If you have the file, you can show a confirmation dialog in which you decide wether to keep the file or overwrite.
The following is an example in which I check in the "system" (I just keep a String list with the filenames) if the file has already been uploaded:
public class RestrictingUpload extends Upload implements Upload.SucceededListener, Upload.Receiver {
private List<String> uploadedFilenames;
private ByteArrayOutputStream latestUploadedOutputStream;
public RestrictingUpload() {
setCaption("Upload");
setButtonCaption("Upload file");
addSucceededListener(this);
setReceiver(this);
uploadedFilenames = new ArrayList<String>();
}
#Override
public OutputStream receiveUpload(String filename, String mimeType) {
latestUploadedOutputStream = new ByteArrayOutputStream();
return latestUploadedOutputStream;
}
#Override
public void uploadSucceeded(SucceededEvent event) {
if (fileExistsInSystem(event.getFilename())) {
confirmOverwrite(event.getFilename());
} else {
uploadedFilenames.add(event.getFilename());
}
}
private void confirmOverwrite(final String filename) {
ConfirmDialog confirmDialog = new ConfirmDialog();
String message = String.format("The file %s already exists in the system. Overwrite?", filename);
confirmDialog.show(getUI(), "Overwrite?", message, "Overwrite", "Cancel", new ConfirmDialog.Listener() {
#Override
public void onClose(ConfirmDialog dialog) {
if (dialog.isConfirmed()) {
copyFileToSystem(filename);
}
}
});
}
private void copyFileToSystem(String filename) {
try {
IOUtils.write(latestUploadedOutputStream.toByteArray(), new FileOutputStream(filename));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e2) {
e2.printStackTrace();
}
}
private boolean fileExistsInSystem(String filename) {
return uploadedFilenames.contains(filename);
}
}
Note that I have used 2 external libraries:
Apache Commons IO 2.4 (http://mvnrepository.com/artifact/commons-io/commons-io/2.4) for writing to streams
ConfirmDialog from Vaadin Directory (https://vaadin.com/directory#addon/confirmdialog)
You can get the code snippet for this class from Gist: https://gist.github.com/gabrielruiu/9960772 which you can paste into your UI and test it out.