In Spring Integration we have a Setup that looks something like this:
--->
--->
(dispatcher) Messages --> Gateway ----> QueueChannel ---> MessageHandler (worker)
--->
--->
So we have one Dispatcher Thread that takes Messages from a MQTT-Broker and forwards them into the Queue. The Poller for the Queue is provided with a TaskExecuter, so the Consumer is multithreaded.
We managed to implement all the functionalities. So the just described setup is already implemented.
Now to guarantee no data loss we want to make two things:
1.:
We want our queue to persist the data, so when the Programm shuts down ungracefully, all the data in the queue will still be there.
This also worked for us, we are using MongoDB as a database because we read somewhere in your docs that this is the recommended way to do it.
2.:
The second thing we want to assure is that the worker threads are working transactional. So only if the worker threads return correctly the messages will permanently be deleted from the queue (and therefore the persistent MessageStore). If the program shuts down during the processing of a message (by the worker thread) the message will still be in the queue at the next startup.
Also if the worker, for example, thows an exception during the processing of the message, it will be put back into the queue.
Our implementation:
As explained before, the basic setup of the program is already implemented. We then extended the basic implementation with a message store implementation for the queue.
QueueChannel:
#Bean
public PollableChannel inputChannel(BasicMessageGroupStore mongoDbChannelMessageStore) {
return new QueueChannel(new MessageGroupQueue(mongoDbChannelMessageStore, "inputChannel"));
}
backed by a Messagestore:
#Bean
public BasicMessageGroupStore mongoDbChannelMessageStore(MongoDbFactory mongoDbFactory) {
MongoDbChannelMessageStore store = new MongoDbChannelMessageStore(mongoDbFactory);
store.setPriorityEnabled(true);
return store;
}
the matching Poller:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
PollerMetadata poll = Pollers.fixedDelay(10).get();
poll.setTaskExecutor(consumer);
return poll;
}
Executor:
private Executor consumer = Executors.newFixedThreadPool(5);
What we have tried?
As explained now we want to extend this implementation with a transactional functionality. We tried using the setTransactionSynchronizationFactory like explained here but it wasn't working (didn't get errors or anything but the behavior was still as it was before we added the TransactionSynchronizer):
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
PollerMetadata poll = Pollers.fixedDelay(10).get();
poll.setTaskExecutor(consumer);
BeanFactory factory = mock(BeanFactory.class);
ExpressionEvaluatingTransactionSynchronizationProcessor etsp = new ExpressionEvaluatingTransactionSynchronizationProcessor();
etsp.setBeanFactory(factory);
etsp.setAfterRollbackChannel(inputChannel());
etsp.setAfterRollbackExpression(new SpelExpressionParser().parseExpression("#bix"));
etsp.setAfterCommitChannel(inputChannel());
etsp.setAfterCommitExpression(new SpelExpressionParser().parseExpression("#bix"));
DefaultTransactionSynchronizationFactory dtsf = new DefaultTransactionSynchronizationFactory(etsp);
poll.setTransactionSynchronizationFactory(dtsf);
return poll;
}
What would be the best way to realize our requirements in spring integration?
EDIT:
As recommended in the answer I chose to do this with the JdbcChannelMessageStore. So I tried converting the XML Implementation described here (18.4.2) into Java. I wasn't quite sure on how to do it, this is what I have tried so far:
I created H2 database and run the script shown here on it.
Created JDBCChannelMessageStore Bean:
#Bean
public JdbcChannelMessageStore store() {
JdbcChannelMessageStore ms = new JdbcChannelMessageStore();
ms.setChannelMessageStoreQueryProvider(queryProvider());
ms.setUsingIdCache(true);
ms.setDataSource(dataSource);
return ms;
}
Created H2ChannelMessageStoreQueryProvider
#Bean
public ChannelMessageStoreQueryProvider queryProvider() {
return new H2ChannelMessageStoreQueryProvider();
}
Adapted the poller:
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() throws Exception {
PollerMetadata poll = Pollers.fixedDelay(10).get();
poll.setTaskExecutor(consumer);
poll.setAdviceChain(Collections.singletonList(transactionInterceptor()));
return poll;
}
Autowired my PlaatformTransactionManager:
#Autowired
PlatformTransactionManager transactionManager;
And created TransactionInterceptor from the TransactonManager:
#Bean
public TransactionInterceptor transactionInterceptor() {
return new TransactionInterceptorBuilder(true)
.transactionManager(transactionManager)
.isolation(Isolation.READ_COMMITTED)
.propagation(Propagation.REQUIRED)
.build();
}
If you need to have queue as transactional, you definitely should take a look into the transactional MessageStore. And only JDBC one is like that. Just because only JDBC support transactions. So, when we perform DELETE, it is OK only if TX is committed.
The MongoDB, nor any other NoSQL DataBases, support such a model, therefore you only can push back the failed messages to the DB on rollback using TransactionSynchronizationFactory.
UPDATE
#RunWith(SpringRunner.class)
#DirtiesContext
public class So47264688Tests {
private static final String MESSAGE_GROUP = "transactionalQueueChannel";
private static EmbeddedDatabase dataSource;
#BeforeClass
public static void init() {
dataSource = new EmbeddedDatabaseBuilder()
.setType(EmbeddedDatabaseType.H2)
.addScript("classpath:/org/springframework/integration/jdbc/schema-drop-h2.sql")
.addScript("classpath:/org/springframework/integration/jdbc/schema-h2.sql")
.build();
}
#AfterClass
public static void destroy() {
dataSource.shutdown();
}
#Autowired
private PollableChannel transactionalQueueChannel;
#Autowired
private JdbcChannelMessageStore jdbcChannelMessageStore;
#Autowired
private PollingConsumer serviceActivatorEndpoint;
#Autowired
private CountDownLatch exceptionLatch;
#Test
public void testTransactionalQueueChannel() throws InterruptedException {
GenericMessage<String> message = new GenericMessage<>("foo");
this.transactionalQueueChannel.send(message);
assertTrue(this.exceptionLatch.await(10, TimeUnit.SECONDS));
this.serviceActivatorEndpoint.stop();
assertEquals(1, this.jdbcChannelMessageStore.messageGroupSize(MESSAGE_GROUP));
Message<?> messageFromStore = this.jdbcChannelMessageStore.pollMessageFromGroup(MESSAGE_GROUP);
assertNotNull(messageFromStore);
assertEquals(message, messageFromStore);
}
#Configuration
#EnableIntegration
public static class ContextConfiguration {
#Bean
public PlatformTransactionManager transactionManager() {
return new DataSourceTransactionManager(dataSource);
}
#Bean
public ChannelMessageStoreQueryProvider queryProvider() {
return new H2ChannelMessageStoreQueryProvider();
}
#Bean
public JdbcChannelMessageStore jdbcChannelMessageStore() {
JdbcChannelMessageStore jdbcChannelMessageStore = new JdbcChannelMessageStore(dataSource);
jdbcChannelMessageStore.setChannelMessageStoreQueryProvider(queryProvider());
return jdbcChannelMessageStore;
}
#Bean
public PollableChannel transactionalQueueChannel() {
return new QueueChannel(new MessageGroupQueue(jdbcChannelMessageStore(), MESSAGE_GROUP));
}
#Bean
public TransactionInterceptor transactionInterceptor() {
return new TransactionInterceptorBuilder()
.transactionManager(transactionManager())
.isolation(Isolation.READ_COMMITTED)
.propagation(Propagation.REQUIRED)
.build();
}
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(5);
return threadPoolTaskExecutor;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return Pollers.fixedDelay(10)
.advice(transactionInterceptor())
.taskExecutor(threadPoolTaskExecutor())
.get();
}
#Bean
public CountDownLatch exceptionLatch() {
return new CountDownLatch(2);
}
#ServiceActivator(inputChannel = "transactionalQueueChannel")
public void handle(Message<?> message) {
System.out.println(message);
try {
throw new RuntimeException("Intentional for rollback");
}
finally {
exceptionLatch().countDown();
}
}
}
}
Thanks to Artem Bilan for your great support. I finally found the solution. It seemed like there was another bean with the name transactionManager and transactionInterceptor active. This resulted in the strange behavior, that my trans-manager was never initialized, instead the other transactionmanager (null) was used for the transactioninterceptor and the PollingConsumer. Thats why my Transactionmanager in PollingConsumer was null, and why my Transactions were never working.
The solution was to rename all my beans, for some beans I also used the annotation #Primary to tell spring to always use this speciffic bean when autowired.
I also downgraded two 4.3, just to make sure this wasn't an error related to Version 5. I haven't testet if it would work with V 5 yet, but I think it should work also.
Related
I want to poll 100 messages from DB every 120 seconds for which I have written following bean.
#Component
class AccountConfiguration {
#Autowired
#Qualifier("outChannel") // defined in spring xml configuration
private MessageChannel outChannel;
#Bean
#InboundChannelAdapter(value = "outChannel",
poller = #Poller(fixedDelay = "120", maxMessagesPerPoll = "100"))
public List<Account> getAccounts(DataSource dataSource) {
JdbcPollingChannelAdapter adapter = newJdbcPollingChannelAdapter(dataSource);
adapter.setRowMapper(new AccountMapper());
Message<Object> result = adapter.receive();
List<Account> list = (ArrayList) result.payload();
return list;
}
}
Above code retrieves rows from DB. But now I want to pass this list to a listener below
#Component
class AccountMessageListener {
public void onMessage(List<Account> list){
System.out.println("Message received");
}
}
Above listener I am trying to call as below
#Component
class AccountService{
#Autowired
#Qualifier("outChannel") // Autowiring the same channel here used above
private MessageChannel outChannel;
#Autowired
AccountMessageListener listener;
public void generateFile(String region){
IntegrationFlows.from("outChannel").handle(listener,"onMessage").get();
}
}
#SpringBootApplication
public class SpringBootExampleApplication
{
public static void main(String[] args)
{
ApplicationContext context = SpringApplication.run(SpringBootExampleApplication.class, args);
AccountService service = context.getBean(AccountService.class);
service.generateFile("ASIA");
}
}
// Below is from Spring xml
<int:channel id="outChannel"/>
<bean id="messageHandler" class="com.account.AccountMessageListener/>
My assumption is that when generateFile is invoked, outChannel will already have data which is passed to "outChannl" by bean getAccounts in AccountConfiguration class
But when generateFile is invoked, it seems outChannel does not have any data and so onMessage is not called.
My queries are - how can I pass data from JdbcPollingChannelAdapter -> outChannel-> onMessage of AccountMessageListener every 120 secs for 2 hours;
Also, is there a way to check number of messages in channel
Your #InboundChannelAdapter configuration is not correct. Since you deal with a JdbcPollingChannelAdapter, then exactly this one has to be a result of getAccounts() bean method:
#Bean
#InboundChannelAdapter(value = "outChannel",
poller = #Poller(fixedDelay = "120", maxMessagesPerPoll = "100"))
public JdbcPollingChannelAdapter getAccounts(DataSource dataSource) {
JdbcPollingChannelAdapter adapter = newJdbcPollingChannelAdapter(dataSource);
adapter.setRowMapper(new AccountMapper());
return adapter;
}
You just don't call MessageSource.receive() yourself. The framework knows what to do with an #InboundChannelAdapter and how to configure and poll the provided bean.
See more in docs: https://docs.spring.io/spring-integration/reference/html/configuration.html#annotations_on_beans
You don't need an XML config if you deal with Spring Boot: better to have everything configured via #Configuration or #Component.
Your usage of the Java DSL (an IntegrationFlow) is wrong. The IntegrationFlow has to be declared as a #Bean and you don't call configuration-related methods yourself:
#Bean
public IntegrationFlow generateFile(String region){
return IntegrationFlows.from("outChannel").handle(listener,"onMessage").get();
}
See docs about Java DSL: https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl
It is not is not clear what is that region since it is out of use.
The #Autowire for outChannel bean is useless: you just don't use that property in your code.
There might be some other flaws and questions: your current code requires too much clean up and reworks.
I have a Spring Boot application which consumes messages from queue (ActiveMQ) and writes them to the database (DB2) and I need it to be fully transactional. I got to a point where I understood that transaction manager (using spring-boot-starter-jta-atomikos) is a best solution for distributed transactions and I'm trying to correctly implement it.
JMS configuration class:
#EnableJms
#Configuration
public class MQConfig {
#Bean
public ConnectionFactory connectionFactory() {
RedeliveryPolicy rp = new RedeliveryPolicy();
rp.setMaximumRedeliveries(3);
rp.setRedeliveryDelay(1000L);
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory();
cf.setBrokerURL("tcp://localhost:61616");
cf.setRedeliveryPolicy(rp);
return cf;
}
#Bean
public JmsTemplate jmsTemplate() {
JmsTemplate template = new JmsTemplate(connectionFactory());
template.setConnectionFactory(connectionFactory());
return template;
}
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setCacheLevelName("CACHE_CONSUMER");
factory.setReceiveTimeout(1000L);
factory.setSessionTransacted(true);
return factory;
}
}
JMS listener class:
#Component
public class MQListener {
#Autowired
private ImportRecordsService importRecordsService;
#JmsListener(
containerFactory = "jmsListenerContainerFactory",
destination = "test.queue"
// concurrency = "4-10"
)
public void receiveMessage(TextMessage message) throws JMSException {
importRecordsService.createRecord();
}
}
Service class that writes to DB:
#Service
public class ImportRecordsService {
#Autowired
private ImportRecordsDAO dao;
#Transactional
public void createRecord() {
ImportRecord record = new ImportRecord();
record.setDateCreated(LocalDateTime.now());
record.setName("test-001");
dao.save(record);
}
}
If exception is thrown inside createRecord() after save, rollback works as it should. When an exception is thrown inside JMS listener in receiveMessage() after save, message is returned to queue but database record stays.
Any help greatly appreciated.
This should be as simple as adding your transactionManager to your DefaultJmsListenerContainerFactory.
In this case add the PlatformTransactionManager (should be an available Spring Bean) to your jmsListenerContainerFactory method signature and then call
factory.setTransactionManager(jtaTransactionManager).
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(ConnectionFactory connectionFactory, PlatformTransactionManager jtaTransactionManager) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setCacheLevelName("CACHE_CONSUMER");
factory.setReceiveTimeout(1000L);
factory.setTransactionManager(jtaTransactionManager);
factory.setSessionTransacted(true);
return factory;
}
Although you will need to note that setting the transactionManager will reset your cache level to CACHE_NONE.
I've got a problem using Spring Batch and can't seem to find a solution.
So, I've got a batch processing some items in chunks (size 10). I've also got a transaction manager used by this batch to persist the processed items after each chunk.
But... I would also like to persist some progress status for those items, in real time. So before processing an item I want to save a status saying this item is in progress.
And I can't seem to find a solution to achieve that. I tried the following solutions :
If I just annotate my status manager service with Transactional annotation the statuses are commited after the whole chunk processing.
If I add REQUIRES_NEW as propagation level to the annotation, it works... but the batch ends in some kind of deadlock (I read that it was common issue with REQUIRES_NEW).
So my last guess was to add a second transaction manager (on the same datasource) and use it on the status manager service... But I got the same result as solution #1 (which seem wierd to me as I expected the transaction from this manager to act independently of the chunk transaction).
Has anybody ever encountered this problem?
EDIT :
Here is my configuration, simplified on purpose :
Class DbConfiguration:
#Configuration
public class DbConfiguration {
#Bean
#Primary
public JpaTransactionManager transactionManager() throws Exception {
final JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory());
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Class JobConfiguration:
#Configuration
#Import(DbConfiguration.class)
#EnableTransactionManagement
public class JobConfiguration {
#Autowired
private EntityManagerFactory entityManagerFactory;
#Bean
public Job jobDefinition() {
return jobBuilderFactory
.get(JOB_NAME)
.start(step())
.build();
}
#Bean
public Step step() {
return stepBuilderFactory
.get(STEP_NAME)
.<Object, Object>chunk(COMMIT_INTERVAL)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
#Bean
public PlatformTransactionManager statusTransactionManager() {
final JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory);
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Class StatusManagerServiceImpl:
#Transactional("statusTransactionManager")
public class StatusManagerServiceImpl implements StatusManagerService {
...
}
A way to achieve this is to use Spring TransactionTemplate as follows:
#Service
public class StatusManagerServiceImpl implements StatusManagerService {
#Autowired
private PlatformTransactionManager transactionManager;
public separateTransactionMethod() {
TransactionTemplate transactionTemplate = new TransactionTemplate(transactionManager);
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
#Override
protected void doInTransactionWithoutResult(TransactionStatus status) {
// Do something here that will be committed right away.
}
});
}
}
You can alternatively use new TransactionCallback if you got a result to return. Then execute method would return that result.
I have implemented a basic asynchronous RPC call using spring boot 1.4 and rabbit mq.
My intention is to use this example as a basis of communication
among micro services.For example, Publisher.java and Subscriber.java could be two micro services talking to each other.
The code shown works fine, but I am curious to know if there are any better ways
of doing this?
My queries as follows:
For subscriber to listen to request queue using #RabbitListener annotation , I did not had to declare directExchange() and binding() beans in configuration. But for asyncRabbitTemplate to read response from reply queue, I had to declare directExchange() and binding() beans in configuration.
Is there any way I can avoid it, because I feel it is code duplication as I am declaring these beans twice.
In real world application, there would be many such calls between micro services.And as per my understanding , I would need to declare similar rpcReplyMessageListenerContainer() and asyncRabbitTemplate() for each request-reply call.Is that correct?
Code as follows.
Link to Github
Config.java
#Configuration("asyncRPCConfig")
#Profile("async_rpc")
#EnableScheduling
#EnableRabbit
#ComponentScan(basePackages = {"in.rabbitmq.async_rpc"})
public class Config {
#Value("${queue.reply}")
private String replyQueue;
#Value("${exchange.direct}")
private String directExchange;
#Value("${routingKey.reply}")
private String replyRoutingKey;
#Bean
public Publisher publisher() {
return new Publisher();
}
#Bean
public SimpleRabbitListenerContainerFactory simpleMessageListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
return factory;
}
#Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate template = new RabbitTemplate(connectionFactory);
template.setMessageConverter(jsonMessageConverter());
return template;
}
#Bean
public MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public Queue replyQueueRPC() {
return new Queue(replyQueue);
}
#Bean
public SimpleMessageListenerContainer rpcReplyMessageListenerContainer(ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer(connectionFactory);
simpleMessageListenerContainer.setQueues(replyQueueRPC());
simpleMessageListenerContainer.setReceiveTimeout(2000);
simpleMessageListenerContainer.setTaskExecutor(Executors.newCachedThreadPool());
return simpleMessageListenerContainer;
}
#Bean
public AsyncRabbitTemplate asyncRabbitTemplate(ConnectionFactory connectionFactory) {
return new AsyncRabbitTemplate(rabbitTemplate(connectionFactory),
rpcReplyMessageListenerContainer(connectionFactory),
directExchange + "/" + replyRoutingKey);
}
#Bean
public DirectExchange directExchange() {
return new DirectExchange(directExchange);
}
#Bean
public Binding binding() {
return BindingBuilder.bind(replyQueueRPC()).to(directExchange()).with(replyRoutingKey);
}
#Bean
public Subscriber subscriber() {
return new Subscriber();
}
}
Publisher.java
public class Publisher {
#Value("${routingKey.request}")
private String requestRoutingKey;
#Autowired
private DirectExchange directExchange;
private static SecureRandom SECURE_RANDOM;
static {
try {
SECURE_RANDOM = SecureRandom.getInstanceStrong();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
}
#Autowired
private AsyncRabbitTemplate asyncRabbitTemplate;
#Scheduled(fixedDelay = 100 * 1)
public void publishToDirectExchangeRPCStyle() {
Integer integer = SECURE_RANDOM.nextInt();
SampleRequestMessage sampleRequestMessage = new SampleRequestMessage(String.valueOf(integer));
System.out.println("Sending out message on direct directExchange:" + sampleRequestMessage);
AsyncRabbitTemplate.RabbitConverterFuture<SampleResponseMessage> sampleResponseMessageRabbitConverterFuture = asyncRabbitTemplate
.convertSendAndReceive(directExchange.getName(), requestRoutingKey, sampleRequestMessage);
sampleResponseMessageRabbitConverterFuture.addCallback(
sampleResponseMessage ->
System.out.println("Response for request message:" + sampleRequestMessage + " is:" + sampleResponseMessage)
, failure ->
System.out.println(failure.getMessage())
);
}
}
Subscriber.java
public class Subscriber {
#RabbitHandler
#RabbitListener(
bindings = {
#QueueBinding(value = #Queue("${queue.request}"),
key = "${routingKey.request}",
exchange = #Exchange(value = "${exchange.direct}", type = ExchangeTypes.DIRECT, durable = "true"))})
public SampleResponseMessage subscribeToRequestQueue(#Payload SampleRequestMessage sampleRequestMessage, Message message) {
System.out.println("Received message :" + message);
return new SampleResponseMessage(sampleRequestMessage.getMessage());
}
}
Your solution is fine.
It is not clear what you are asking...
I had to declare directExchange() and binding() beans in configuration.
Is there any way I can avoid it, because I feel it is code duplication as I am declaring these beans twice.
#QueueBinding is simply a convenience on #RabbitListener and an alternative to declaring the queue, exchange and binding as #Beans.
If you are using a common #Config class you can simply omit the bindings attribute on the listener and use queues = "${queue.reply}" to avoid the duplication.
I would need to declare similar rpcReplyMessageListenerContainer() and asyncRabbitTemplate() for each request-reply call.
Is that correct?
Yes; although with the upcoming 2.0 release, you can use a DirectReplyToMessageListenerContainer which avoids the need for a separate reply queue for each service; when you send a message.
See the documentation here and here.
Starting with version 2.0, the async template now supports Direct reply-to instead of a configured reply queue.
(Should read "as an alternative to " rather than "instead of").
So you can use the same template to talk to multiple services.
So I have a problem in Spring Batch 3.0.7.RELEASE and Spring 4.3.2.RELEASE where we want to use the prototype scope for the ItemProcessor when using concurrency.
See appBatchCreationProcessor() and BatchCreationStep(), I've tried to make the scope of appBatchCreationProcessor prototype, but it doesn't seem to have any effect, the same item processor is used across all 10 threads.
Is there a way around this? Or is this by design?
AppBatchConfiguration.java
#Configuration
#EnableBatchProcessing
#ComponentScan(basePackages = "our.org.base")
public class AppBatchConfiguration {
private final static SimpleLogger LOGGER = SimpleLogger.getInstance(AppBatchConfiguration.class);
private final static String OUTPUT_XML_FILE_PATH_PLACEHOLDER = null;
private final static String INPUT_XML_FILE_PATH_PLACEHOLDER = null;
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean(name = "cimAppXmlReader")
#StepScope
public <T> ItemStreamReader<T> appXmlReader(#Value("#{jobParameters[inputXmlFilePath]}")
String inputXmlFilePath) {
LOGGER.info("Job Parameter => App XML File Path :" + inputXmlFilePath);
StaxEventItemReader<T> reader = new StaxEventItemReader<T>();
reader.setResource(new FileSystemResource(inputXmlFilePath));
reader.setUnmarshaller(mecaUnMarshaller());
reader.setFragmentRootElementNames(getAppRootElementNames());
reader.setSaveState(false);
// Make the StaxEventItemReader thread-safe
SynchronizedItemStreamReader<T> synchronizedItemStreamReader = new SynchronizedItemStreamReader<T>();
synchronizedItemStreamReader.setDelegate(reader);
return synchronizedItemStreamReader;
}
#Bean
#StepScope
public ItemStreamReader<JAXBElement<AppIBTransactionHeaderType>> appXmlTransactionHeaderReader(#Value("#{jobParameters[inputXmlFilePath]}")
String inputXmlFilePath) {
LOGGER.info("Job Parameter => App XML File Path for Transaction Header :" + inputXmlFilePath);
StaxEventItemReader<JAXBElement<AppIBTransactionHeaderType>> reader = new StaxEventItemReader<>();
reader.setResource(new FileSystemResource(inputXmlFilePath));
reader.setUnmarshaller(mecaUnMarshaller());
String[] fragmentRootElementNames = new String[] {"AppIBTransactionHeader"};
reader.setFragmentRootElementNames(fragmentRootElementNames);
reader.setSaveState(false);
return reader;
}
#Bean
public Unmarshaller mecaUnMarshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setPackagesToScan(ObjectFactory.class.getPackage().getName());
return marshaller;
}
#Bean
public Marshaller uberMarshaller() {
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setClassesToBeBound(ServiceRequestType.class);
marshaller.setSupportJaxbElementClass(true);
return marshaller;
}
#Bean(destroyMethod="") // To stop multiple close calls, see: http://stackoverflow.com/a/23089536
#StepScope
public ResourceAwareItemWriterItemStream<JAXBElement<ServiceRequestType>> writer(#Value("#{jobParameters[outputXmlFilePath]}")
String outputXmlFilePath) {
SyncStaxEventItemWriter<JAXBElement<ServiceRequestType>> writer = new SyncStaxEventItemWriter<JAXBElement<ServiceRequestType>>();
writer.setResource(new FileSystemResource(outputXmlFilePath));
writer.setMarshaller(uberMarshaller());
writer.setSaveState(false);
HashMap<String, String> rootElementAttribs = new HashMap<String, String>();
rootElementAttribs.put("xmlns:ns1", "http://some.org/corporate/message/2010/1");
writer.setRootElementAttributes(rootElementAttribs);
writer.setRootTagName("ns1:SetOfServiceRequests");
return writer;
}
#Bean
#StepScope
public <T> ItemProcessor<T, JAXBElement<ServiceRequestType>> appNotificationProcessor() {
return new AppBatchNotificationItemProcessor<T>();
}
#Bean
#Scope(scopeName=ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public ItemProcessor<JAXBElement<AppIBTransactionHeaderType>, Boolean> appBatchCreationProcessor() {
return new AppBatchCreationItemProcessor();
}
public String[] getAppRootElementNames() {
//get list of App Transaction Element Names
return AppProcessorEnum.getValues();
}
#Bean
public Step AppStep() {
// INPUT_XML_FILE_PATH_PLACEHOLDER and OUTPUT_XML_FILE_PATH_PLACEHOLDER will be overridden
// by injected jobParameters using late binding (StepScope)
return stepBuilderFactory.get("AppStep")
.<Object, JAXBElement<ServiceRequestType>> chunk(10)
.reader(appXmlReader(INPUT_XML_FILE_PATH_PLACEHOLDER))
.processor(appNotificationProcessor())
.writer(writer(OUTPUT_XML_FILE_PATH_PLACEHOLDER))
.taskExecutor(concurrentTaskExecutor())
.throttleLimit(1)
.build();
}
#Bean
public Step BatchCreationStep() {
return stepBuilderFactory.get("BatchCreationStep")
.<JAXBElement<AppIBTransactionHeaderType>, Boolean>chunk(1)
.reader(appXmlTransactionHeaderReader(INPUT_XML_FILE_PATH_PLACEHOLDER))
.processor(appBatchCreationProcessor())
.taskExecutor(concurrentTaskExecutor())
.throttleLimit(10)
.build();
}
#Bean
public Job AppJob() {
return jobBuilderFactory.get("AppJob")
.incrementer(new RunIdIncrementer())
.listener(AppJobCompletionNotificationListener())
.flow(AppStep())
.next(BatchCreationStep())
.end()
.build();
}
#Bean
public JobCompletionNotificationListener AppJobCompletionNotificationListener() {
return new JobCompletionNotificationListener();
}
#Bean
public TaskExecutor concurrentTaskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(10);
return taskExecutor;
}
}
Yes, this is by design.
Think of a job and its steps with its reader, processor, and writer as a static structure that is created before the job gets executed. This means, that your appropriate createReader, createProcessor methods had been called and the bean instances had been created before the job is executed.
SCOPE_PROTOTYPE is evaluated during this phase and since your createProcessor method is called only once during this phase, there will be only one instance of it.
After the job is started, this structure stays "stable".
Now, Spring Batch tweaks that a little bit by providing a "stepscope" which defers bean creation to the start of the step. However, this will not help if you run your step with multiple threads. There is still only one instance of, let's say the processor in your example, and this instance is used for all threads.
What you would need is something like a "ThreadScope" but there isn't such a concept inside spring or spring-batch. You would need to implement your processor accordingly, for instance by using ThreadLocal members.
For instance, you could wrap your Processor in something like this:
public class ThreadLocalItemProcessor implements ItemProcessor {
private ThreadLocal<ItemProcessor> threadProcessor = ThreadLocal.withInitial(() -> new MyProcessor());
#Override
public Object process(Object item) throws Exception {
return threadProcessor.get().process(item);
}
}
Edit: Example with prototype method
If your Processor is instantiated as a SpringBean, it can also use Autowired for injection. Therefore, you could inject a prototype-factory (of course, the prototype-factory has to be instantiated as springbean) as follows:
#Configuration
public class PrototypeFactory {
#Bean
#Scope(Prototype)
public YourInterfaceOrClass createInstance() {
return new YourInterfaceOrClass();
}
}
public class ThreadLocalItemProcessor implements ItemProcessor {
#Autowired
private PrototypeFactory prototypeFactory;
private ThreadLocal<ItemProcessor> threadProcessor = ThreadLocal.withInitial(this::processorCreator);
#Override
public Object process(Object item) throws Exception {
return threadProcessor.get().process(item);
}
//ItemProcessor directly implemented as lambda
// this will only be called once per working thread
private Object process(Object input) {
// will produce a valid SpringBean instance
YourInterfaceOrClass inst = prototypeFactory.createInstance();
... process the input
}
}
The reason that the same of appBatchCreationProcessor() is used in all thread is because it is injected into a singleton BatchCreationStep(). Furthermore, the BatchCreationStep() is also injected into a singleton AppJob()
According to this documentation:
When you use singleton-scoped beans with dependencies on prototype
beans, be aware that dependencies are resolved at instantiation time.
Thus if you dependency-inject a prototype-scoped bean into a
singleton-scoped bean, a new prototype bean is instantiated and then
dependency-injected into the singleton bean. The prototype instance is
the sole instance that is ever supplied to the singleton-scoped bean.
However if you do really need to create a new appBatchCreationProcessor(), you can use method injection