I am using spring batch in my application. I am using multithreading using spring batch at step level .
Below is my configuration:
<step id="testStep">
<tasklet
task-executor="taskExecutor" throttle-limit="5">
<chunk reader="reader" writer="writer"
commit-interval="10">
</chunk>
</tasklet>
</step>
<bean id="writer" class="org.springframework.batch.item.file.FlatFileItemWriter"
scope="step">
<property name="resource" value="file:${output.file.location.directory}/<<<NEED TO PASS CURRENT EXECUTING THREAD NAME>>>" />
<property name="appendAllowed" value="true"></property>
<property name="lineAggregator" ref="aggregator" />
</bean>
So in resource property of flatfileitemWriter , i need to pass current executing thread name, so that each thread wil write in a separate unique file.
I can able to get the current executing thread name in writer class but dont know how to get the same in the spring confirguration file .
Any idea to do so ?
If you want you can use spEL; with this scripting language you can use standard java method and to retrive current thread name the syntax is #{T(java.lang.Thread).getCurrentThread().getName()}.
In a more clear way use a StepExecutionListener.beforeStep() injecting the current thread name into step execution context (eg. currentThreadName key) and retrieve from step xml config in the standard way
<property name="resource" value="file:${output.file.location.directory}/#{stepExecutionContext['currentThreadName']}" />
Related
Currently in my data context XML file, it's looking up values to substitute from a application.properties file:
<bean id="applicationProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" lazy-init="default">
<property name="location" value="classpath:application.properties" />
</bean>
<bean id="appleDataSource" class="org.apache.commons.dbcp2.BasicDataSource">
<property name="url" value="${apple.url}" />
</bean>
I'd like to change this from being looked up from the application.properties file, to being read out of a Properties/Map object.
This is because the configuration could come from more than one place (i.e. not just the classpath), and this logic is already implemented in my application.
I've seen MapPropertySource but I think while that can be done when the bean is configured in Java, I'm not sure this logic can be implemented when working with the XML file alone?
I am using Spring Integration RecursiveDirectoryScanner to scan a directory recursively to process the incoming file that will be placed under the configured directory (/home/test).
I am frequently getting the below error:
ERROR org.springframework.integration.handler.LoggingHandler - java.lang.IllegalArgumentException: java.nio.file.FileSystemException: /home/test: Too many open files
at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:89)
at org.springframework.integration.file.FileReadingMessageSource.scanInputDirectory(FileReadingMessageSource.java:387)
at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:361)
at org.springframework.integration.file.FileReadingMessageSource.doReceive(FileReadingMessageSource.java:90)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:134)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:224)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:245)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:58)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:190)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:186)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:353)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /home/test: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72)
at java.nio.file.Files.walk(Files.java:3574)
at org.springframework.integration.file.RecursiveDirectoryScanner.listFiles(RecursiveDirectoryScanner.java:73)
My Spring Integration flow is as below:
Configuration in XML
<task:executor id="pollerPool"
pool-size="${pollerThreadPoolSize}"
queue-capacity="${pollerThreadQueueCapacity}" rejection-policy="ABORT" />
<task:executor id="fileHandlerPool"
pool-size="${fileHandlerPoolSize}"
queue-capacity="${fileHandlerPoolThreadQueueCapacity}" rejection-policy="CALLER_RUNS" />
<bean id="iFilter" class="org.springframework.integration.file.filters.ChainFileListFilter">
<constructor-arg>
<list>
<bean id="lastModifiedFileListFilter" class="org.springframework.integration.file.filters.LastModifiedFileListFilter">
<property name="age" value="120" />
</bean>
<ref bean="acceptOnceFileListFilter"/>
<bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
<constructor-arg value="^.*\.(txt|csv|xls|xlsx|asc)$"/>
</bean>
</list>
</constructor-arg>
</bean>
<bean id="acceptOnceFileListFilter" name="acceptOnceFileListFilter" class="org.springframework.integration.file.filters.AcceptOnceFileListFilter" primary="true" />
<bean id="recursiveDirectoryScanner" class="org.springframework.integration.file.RecursiveDirectoryScanner">
<property name="filter" ref="iFilter" />
<property name="locker" ref="nioFileLocker" />
</bean>
<bean id="nioFileLocker" class="org.springframework.integration.file.locking.NioFileLocker" />
<int-file:inbound-channel-adapter
id="fileSource" channel="fileReceivedChannel" auto-startup="true"
directory="file:${polling.directory}"
scanner="recursiveDirectoryScanner" >
<int:poller task-executor="pollerPool"
fixed-rate="${pollerFixedRate}"
receive-timeout="${pollerReceiveTimeout}">
</int:poller>
</int-file:inbound-channel-adapter>
Dynamic parameters are as below:
polling.directory=/home/test
pollerThreadPoolSize=1
pollerThreadQueueCapacity=10
pollerFixedRate=5000
pollerReceiveTimeout=5000
fileHandlerPoolSize=2
fileHandlerPoolThreadQueueCapacity=100
EDIT:
I do unlock file in a service activator that comes in to picture when a file is picked. I get some information from file and unlock it.
#Autowired
NioFileLocker nioFileLocker;
protected void doTransform(Message<?> message) throws Exception {
MessageBuilder<File> payload = (MessageBuilder<File>) message.getPayload();
File inFile = payload.getPayload();
try {
nioFileLocker.unlock(inFile);
} catch (Exception e) {
LOGGER.error("file not unlock");
}
}
Is there any issue with the configuration ? How do I make sure this exception never appear again ?
Thank you in advance.
I would suggest to test your solution without NioFileLocker. Doesn't look like you are using it for unlocking files, but the lock(File fileToLock) really keeps some file marker in OS.
On the other hand the file locker doesn't work reliable on UNIX systems. It still allows an access to files. At least for reading.
For better exclusive file access I would recommend to use a FileSystemPersistentAcceptOnceFileListFilter with the external MetadataStore instead of in memory AcceptOnceFileListFilter. This way only one instance of your application will get access to the file and it won't be processed again at all.
Following is the code sample of a reader bean
<bean id="reader"
class="org.springframework.batch.item.database.JdbcCursorItemReader"
scope="step">
<property name="dataSource" ref="dataSource" />
<property name="sql"
value="#{#sqlStatements[jobParameters['key']]}" />
<property name="rowMapper" ref="#{jobParameters['key'] + 'Mapper'}" />
</bean>
There is <util:map id="sqlStatements"/>. I have a similar bean for writer. I want to have a dynamic switcher for processor. The interface ItemProcessor<I,O> needs I & O to be mentioned, either I must switch it in some way or create a custom ItemProcessor.
I tried the below code but it didn't work.
<batch:job id="springBatch">
<batch:step id="step1">
<batch:tasklet>
<batch:chunk reader="reader" processor="#{jobParameters['key'] + 'Processor'}"
writer="writer" commit-interval="1"></batch:chunk>
</batch:tasklet>
</batch:step>
</batch:job>
It gave a org.springframework.expression.spel.SpelEvaluationException for jobParameters, probably because scope=step was not defined for it.
Could someone please provide an alternative for this ?
Using a Classifier can simplify a lot resolution of this problem: create a custom ItemProcessor<Object, Object> and inject a Classifier<Class, ItemProcessor> (a PatternMatchingClassifier can fits) and in ItemProcessor.process() detect right processor using classifier.
Also look at ClassifierCompositeItemWriter.
Not sure if I follow your question completely but one simple solution can be creating a custom processor as delegator. This delegator will delegate processor code to relevant Item Processor class based on job parameters key or whatever logic you want.
I have three different .properties files in a Spring Batch project, and I'm trying to set which .properties file should be used as a JobParameter. I'd like to be able to run the job like so:
java CommandLineJobRunner context.xml jobName region=us
The region should specify which .properties file should be used. The problem is getting the context to recognize the JobParameter. I've tried the following to no avail:
<context:property-placeholder location="classpath:batch.#{jobParameters['region']}.properties"/>
And also:
<util:properties id="batchProperties" location="classpath:batch.#{jobParameters['region']}.properties"></util:properties>
I had heard that adding scope="step" could fix similar issues, but I tried adding that to both of the above solutions and still had exceptions.
I think I'm missing a fundamental idea of why I can't get this working, but I'm unable to figure out what that idea is.
If anyone has any suggestions on getting this working and/or explaining why my previous approaches failed, I'd appreciate it.
This is not the right way to proceed (it is impossible do what you are trying to do).
You have to think that jobParameters is available only when a job is running and only for its composing steps marked with scope="step" (and not <context:property-placeholder> nor <util:properties> has a step attribute).
A way for solving the problem is to load properties file in job's execution context before first step is running with a listener:
public class PropertyLoaderJobExecutionListener extends StepExecutionListenerSupport {
Properties countryProperties;
public void setCountryProperties(Properties pfb) {
this.countryProperties = pfb;
}
#Override
public void beforeStep(StepExecution stepExecution) {
super.beforeStep(stepExecution);
// Store property file content in jobExecutionContext with name "batchProperties"
stepExecution.getJobExecution().getExecutionContext().put("batchProperties", countryProperties);
}
}
in your job.xml
<bean id="loadPropertiesListener" class="PropertyLoaderJobExecutionListener" scope="step">
<property name="pfb">
<bean class="org.springframework.beans.factory.config.PropertiesFactoryBean">
<property name="location" value="classpath:batch.#{jobParameters['region']}.properties" />
</bean>
</property>
</bean>
and register this listener in your first step (you can't do that in your JobExectionListsner.beforeJob() because there isn't a scope="job" for now and late-binding of #{jobParameters['region']} value is not available).
To access your data with spEL use this syntax:
#{jobExecutionContext.get('batchProperties').getProperty('language')}
or a better syntax to access properties (IDK spEL so good, sorry).
Hope to be clear and can help to solve your problem.
EDIT (full code of my working job.xml):
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
http://www.springframework.org/schema/jdbc http://www.springframework.org/schema/jdbc/spring-util-3.1.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">
<job id="sourceJob" xmlns="http://www.springframework.org/schema/batch">
<step id="step1">
<tasklet ref="getRemoteFileTasklet" />
<listeners>
<listener ref="loadPropertiesListener" />
</listeners>
</step>
</job>
<bean id="loadPropertiesListener" class="PropertyLoaderJobExecutionListener" scope="step">
<property name="pfb">
<bean class="org.springframework.beans.factory.config.PropertiesFactoryBean">
<property name="location" value="classpath:batch.#{jobParameters['region']}.properties" />
</bean>
</property>
</bean>
<bean id="getRemoteFileTasklet" class="GetRemoteFileTasklet" />
I am using quartz to schedule a spring batch job.
The job reads a file from a folder( which has multiple files ) does some processing and copies it to another folder.
is it possible to create multiple instances of the job, that will run concurrenty,reading multiple files ?
My question is :
In spring batch, is it possible to spawn multiple instances of the same job? I am using quartz schedular ?
In Spring Batch it is possible to start several jobs, provided you have supplied different JobParameters for each jobLauncher.run() call. jobLauncher in your Spring configuration will spawn each job in a separate thread, if it is configured with appropriate task executor:
<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
<property name="taskExecutor" ref="taskExecutor" />
</bean>
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor"
p:corePoolSize="5"
p:maxPoolSize="30" />
It is possible with Quartz, using a MethodInvokingJobDetailFactoryBean, for instance:
<bean id="myjob"
class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
<property name="targetObject">
<ref bean="someBean" />
</property>
<property name="targetMethod" value="someMethod" />
<!-- The concurrent property is already true by default
<property name="concurrent" value="true" />
-->
</bean>
Citing the Spring documentation
By default, Quartz Jobs are stateless, resulting in the possibility of jobs interfering with each other. If you specify two triggers for the same JobDetail, it might be possible that before the first job has finished, the second one will start. If JobDetail classes implement the Stateful interface, this won't happen. The second job will not start before the first one has finished. To make jobs resulting from the MethodInvokingJobDetailFactoryBean non-concurrent, set the concurrent flag to false.