I am facing some issues while writing integration tests for Spring Batch jobs. The main problem is that an exception is thrown whenever a transaction is started inside the batch job.
Well, first things first. Imagine this is the step of a simple job. A Tasklet for the sake of simplicity. Of course, it is used in a proper batch config (MyBatchConfig) which I also omit for brevity.
#Component
public class SimpleTask implements Tasklet {
private final MyRepository myRepository;
public SimpleTask(MyRepository myRepository) {
this.myRepository = myRepository;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
myRepository.deleteAll(); // or maybe saveAll() or some other #Transactional method
return RepeatStatus.FINISHED;
}
}
MyRepository is a very unspecial CrudRepository.
Now, to test that job I use the following test class.
#SpringBatchTest
#EnableAutoConfiguration
#SpringJUnitConfig(classes = {
H2DataSourceConfig.class, // <-- this is a configuration bean for an in-memory testing database
MyBatchConfig.class
})
public class MyBatchJobTest {
#Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
#Autowired
private JobRepositoryTestUtils jobRepositoryTestUtils;
#Autowired
private MyRepository myRepository;
#Test
public void testJob() throws Exception {
var testItems = List.of(
new MyTestItem(1),
new MyTestItem(2),
new MyTestItem(3)
);
myRepository.saveAll(testItems); // <--- works perfectly well
jobLauncherTestUtils.launchJob();
}
}
When it comes to the tasklet execution and more precisely to the deleteAll() method call this exception is fired:
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException: Already value [org.springframework.jdbc.datasource.ConnectionHolder#68f48807] for key [org.springframework.jdbc.datasource.DriverManagerDataSource#49a6f486] bound to thread [SimpleAsyncTaskExecutor-1]
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:448)
...
Do you have any ideas why this is happening?
As a workaround I currently mock the repository with #MockBean and back it with an ArrayList but this is not what the inventor intended, I guess.
Any advice?
Kind regards
Update 1.1 (includes solution)
The mentioned data source configuration class is
#Configuration
#EnableJpaRepositories(
basePackages = {"my.project.persistence.repository"},
entityManagerFactoryRef = "myTestEntityManagerFactory",
transactionManagerRef = "myTestTransactionManager"
)
#EnableTransactionManagement
public class H2DataSourceConfig {
#Bean
public DataSource myTestDataSource() {
var dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("org.h2.Driver");
dataSource.setUrl("jdbc:h2:mem:myDb;DB_CLOSE_DELAY=-1");
return dataSource;
}
#Bean
public LocalContainerEntityManagerFactoryBean myTestEntityManagerFactory() {
var emFactory = new LocalContainerEntityManagerFactoryBean();
var adapter = new HibernateJpaVendorAdapter();
adapter.setDatabasePlatform("org.hibernate.dialect.H2Dialect");
adapter.setGenerateDdl(true);
emFactory.setDataSource(myTestDataSource());
emFactory.setPackagesToScan("my.project.persistence.model");
emFactory.setJpaVendorAdapter(adapter);
return emFactory;
}
#Bean
public PlatformTransactionManager myTestTransactionManager() {
return new JpaTransactionManager(myTestEntityManagerFactory().getObject());
}
#Bean
public BatchConfigurer testBatchConfigurer() {
return new DefaultBatchConfigurer() {
#Override
public PlatformTransactionManager getTransactionManager() {
return myTestTransactionManager();
}
};
}
}
By default, when you declare a datasource in your application context, Spring Batch will use a DataSourceTransactionManager to drive step transactions, but this transaction manager knows nothing about your JPA context.
If you want to use another transaction manager, you need to override BatchConfigurer#getTransactionManager and return the transaction manager you want to use to drive step transactions. In your case, you are only declaring a transaction manager bean in the application context which is not enough. Here a quick example:
#Bean
public BatchConfigurer batchConfigurer() {
return new DefaultBatchConfigurer() {
#Override
public PlatformTransactionManager getTransactionManager() {
return new JpaTransactionManager(myTestEntityManagerFactory().getObject());
}
};
}
For more details, please refer to the reference documentation.
Related
I have a spring batch project wherein I read data from a datasource , process the data and write into another primary data source. I am extending CrudRepository for dao operations.
I am trying to configure multiple datasources for my springbatch + spring boot application below is the package structure :
myproject
---com
---batch
---config
---firstDsConfig.java
---secondDsConfig.java
---firstrepository
---firstCrudRepository.java
---secondRepository
---SecondCrudRepository.java
---firstEntity
---firstDBEntity.java
---secondEntity
---secondDBEntity.java
----main
---MyMainClass.java
Code for firstDsConfig.java:
#Configuration
#EnableJpaRepositories(
entityManagerFactoryRef = "firstEntityManagerFactory",
transactionManagerRef = "firstTransactionManager",
basePackages = "com.batch.firstrepository"
)
#EnableTransactionManagement
public class FirstDbConfig {
#Primary
#Bean(name = "firstEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean firstEntityManagerFactory(final EntityManagerFactoryBuilder builder,
final #Qualifier("firstDs") DataSource dataSource) {
return builder
.dataSource(dataSource)
.packages("com.batch.firstEntity")
.persistenceUnit("abc")
.build();
}
#Primary
#Bean(name = "firstTransactionManager")
public PlatformTransactionManager firstTransactionManager(#Qualifier("firstEntityManagerFactory")
EntityManagerFactory firstEntityManagerFactory) {
return new JpaTransactionManager(firstEntityManagerFactory);
}
#Primary
#Bean(name = "firstDs")
#ConfigurationProperties(prefix = "spring.datasource.first")
public DataSource firstDataSource() {
return DataSourceBuilder.create().build();
}
}
Code for secondDsConfig:
#Configuration
#EnableJpaRepositories(
entityManagerFactoryRef = "secondEntityManagerFactory",
transactionManagerRef = "secondTransactionManager",
basePackages = "com.batch.secondrepository"
)
#EnableTransactionManagement
public class FirstDbConfig {
#Primary
#Bean(name = "secondEntityManagerFactory")
public LocalContainerEntityManagerFactoryBean secondEntityManagerFactory(final EntityManagerFactoryBuilder builder,
final #Qualifier("secondDs") DataSource dataSource) {
return builder
.dataSource(dataSource)
.packages("com.batch.secondEntity")
.persistenceUnit("xyz")
.build();
}
#Primary
#Bean(name = "secondTransactionManager")
public PlatformTransactionManager secondTransactionManager(#Qualifier("secondEntityManagerFactory")
EntityManagerFactory firstEntityManagerFactory) {
return new JpaTransactionManager(secondEntityManagerFactory);
}
#Primary
#Bean(name = "secondDs")
#ConfigurationProperties(prefix = "spring.datasource.second")
public DataSource secondDataSource() {
return DataSourceBuilder.create().build();
}
}
Here is my main class
#EnableScheduling
#EnableBatchProcessing
#SpringBootApplication(scanBasePackages = { "com.batch" })
public class MyMainClass {
#Autowired
private JobLauncher jobLauncher;
#Autowired
private JobRepository jobRepository;
#Autowired
#Qualifier(firstDs)
private DataSource dataSource;
#Autowired
#Qualifier("myJob")
private Job job;
public static void main(String[] args) throws Exception {
SpringApplication.run(MyMainClass.class, args);
}
#EventListener(ApplicationReadyEvent.class)
private void start() throws Exception {
jobLauncher.run(job, new JobParameters());
}
#Bean(name="jobService")
public JobService jobService() throws Exception {
SimpleJobServiceFactoryBean factoryBean = new SimpleJobServiceFactoryBean();
factoryBean.setDataSource(dataSource);
factoryBean.setJobRepository(jobRepository);
factoryBean.setJobLocator(new MapJobRegistry());
factoryBean.setJobLauncher(jobLauncher);
factoryBean.afterPropertiesSet();
return factoryBean.getObject();
}
}
Here are the crud repo:
First curd repo
public interface FirstCrudRepository extends CrudRepository<FirstDbEntity, Integer> {
List<FirstDbEntity> findByOId(String oId);
}
Second curd repo
public interface SecondCrudRepository extends CrudRepository<SecondDbEntity, Integer> {
List<SecondDbEntity> findByPid(String pid);
}
When I run my application I see following error while saving record using FirstCrudRepository:
org.springframework.transaction.IllegalTransactionStateException: Pre-bound JDBC Connection found! JpaTransactionManager does not support running within DataSourceTransactionManager if told to manage the DataSource itself. It is recommended to use a single JpaTransactionManager for all transactions on a single DataSource, no matter whether JPA or JDBC access.
Note: I am able to fetch details successfully from SecondCrudRepository
By default, if you provide a DataSource, Spring Batch will use a DataSourceTransactionManager which knows nothing about your JPA configuration. You need to tell Spring Batch to use your JpaTransactionManager. This is explained in the:
reference documentation: https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html#javaConfig.
Javadoc of #EnableBatchProcessing: https://docs.spring.io/spring-batch/4.1.x/api/org/springframework/batch/core/configuration/annotation/EnableBatchProcessing.html
I suspect there are 2 different transaction managers present which would be causing issue. Annotate with #Transactional and specifying the transaction manager would help
Source:
Spring - Is it possible to use multiple transaction managers in the same application?
I've got a problem using Spring Batch and can't seem to find a solution.
So, I've got a batch processing some items in chunks (size 10). I've also got a transaction manager used by this batch to persist the processed items after each chunk.
But... I would also like to persist some progress status for those items, in real time. So before processing an item I want to save a status saying this item is in progress.
And I can't seem to find a solution to achieve that. I tried the following solutions :
If I just annotate my status manager service with Transactional annotation the statuses are commited after the whole chunk processing.
If I add REQUIRES_NEW as propagation level to the annotation, it works... but the batch ends in some kind of deadlock (I read that it was common issue with REQUIRES_NEW).
So my last guess was to add a second transaction manager (on the same datasource) and use it on the status manager service... But I got the same result as solution #1 (which seem wierd to me as I expected the transaction from this manager to act independently of the chunk transaction).
Has anybody ever encountered this problem?
EDIT :
Here is my configuration, simplified on purpose :
Class DbConfiguration:
#Configuration
public class DbConfiguration {
#Bean
#Primary
public JpaTransactionManager transactionManager() throws Exception {
final JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory());
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Class JobConfiguration:
#Configuration
#Import(DbConfiguration.class)
#EnableTransactionManagement
public class JobConfiguration {
#Autowired
private EntityManagerFactory entityManagerFactory;
#Bean
public Job jobDefinition() {
return jobBuilderFactory
.get(JOB_NAME)
.start(step())
.build();
}
#Bean
public Step step() {
return stepBuilderFactory
.get(STEP_NAME)
.<Object, Object>chunk(COMMIT_INTERVAL)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
#Bean
public PlatformTransactionManager statusTransactionManager() {
final JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory);
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Class StatusManagerServiceImpl:
#Transactional("statusTransactionManager")
public class StatusManagerServiceImpl implements StatusManagerService {
...
}
A way to achieve this is to use Spring TransactionTemplate as follows:
#Service
public class StatusManagerServiceImpl implements StatusManagerService {
#Autowired
private PlatformTransactionManager transactionManager;
public separateTransactionMethod() {
TransactionTemplate transactionTemplate = new TransactionTemplate(transactionManager);
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
#Override
protected void doInTransactionWithoutResult(TransactionStatus status) {
// Do something here that will be committed right away.
}
});
}
}
You can alternatively use new TransactionCallback if you got a result to return. Then execute method would return that result.
My Spring Batch job is started every 5 minutes - basically it reads a string, uses the string as a parameter in a sql query, and prints out the resulting sql result list. Mostly it seems to be running ok, but I notice sporadic errors in my logs every 5-10 runs
2017-05-05 11:13:26.101 INFO 9572 --- [nio-8081-exec-8] c.u.r.s.AgentCollectorServiceImpl : Could not open JPA E
ntityManager for transaction; nested exception is java.lang.IllegalStateException: Transaction already active
My job is started like from my AgentCollectorServiceImpl class
#Override
public void addReportIds(List<Integer> reportIds) {
try {
.toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (Exception e) {
log.info(e.getMessage());
}
}
My BatchConfig class looks like
#Configuration
#EnableBatchProcessing
#Import(AppConfig.class)
public class BatchConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private AppConfig appConfig;
#Bean
public Reader reader() {
return new Reader();
}
#Bean
public Processor processor() {
return new Processor();
}
#Bean
public Writer writer() {
return new Writer();
}
#Bean
public Job job() {
return jobBuilderFactory.get("job")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<String, String> chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
}
My AppConfig class looks like
#Configuration
#PropertySource("classpath:application.properties")
#ComponentScan
public class AppConfig {
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(organizationDataSource());
em.setPackagesToScan(new String[]{"com.organization.agentcollector.model"});
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalProperties());
return em;
}
Properties additionalProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.dialect", "com.organization.agentcollector.config.SQLServerDialectOverrider");
return properties;
}
#Bean
JpaTransactionManager transactionManager(final EntityManagerFactory emf) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getObject());
return transactionManager;
}
My Processor class looks like
public class Processor implements ItemProcessor<String, String> {
private final Logger log = LoggerFactory.getLogger(Processor.class);
#Autowired
EventReportsDAOImpl eventReportsDAOImpl;
#Override
public String process(String reportIdsJson) throws Exception {
String eventReportsJson = eventReportsDAOImpl.listEventReportsInJsonRequest(reportIdsJson);
//System.out.println(returnContent+"PROCESSOR");
return eventReportsJson;
}
}
My DAOImpl class looks like
#Component
#Transactional
public class EventReportsDAOImpl implements EventReportsDAO {
#PersistenceContext
private EntityManager em;
#Override
public EventReports getEventReports(Integer reportId) {
return null;
}
#Override
public String listEventReportsInJsonRequest(String reportIds) {
System.out.println("Event Report reportIds processing");
ArrayList<EventReports> erArr = new ArrayList<EventReports>();
String reportIdsList = reportIds.substring(1, reportIds.length() - 1);
//System.out.println(reportIdsList);
try {
StoredProcedureQuery q = em.createStoredProcedureQuery("sp_get_event_reports", "eventReportsResult");
q.registerStoredProcedureParameter("reportIds", String.class, ParameterMode.IN);
q.setParameter("reportIds", reportIdsList);
boolean isResultSet = q.execute();
erArr = (ArrayList<EventReports>) q.getResultList();
} catch (Exception e) {
System.out.println("No event reports found for list " + reportIdsList);
}
return erArr.toString();
}
I thought Spring would manage transactions automatically. The error seems to suggest that a transaction is not being properly closed?
One thing I tried was removing all #Transactional annotations from my code as I read that #EnableBatchProcessing already injects a Transaction Manager into each step - but when I did this, I saw the 'transaction already active' error much more frequently.
Any advice appreciated on how to fix this, thank you!
The #Transactional notation establishes a transactional scope that dictates when a transaction starts and ends, also called its boundary. If you operate outside of this boundary you'll receive errors.
First off, I found this bit of documentation the most helpful on Spring transactions: http://docs.spring.io/spring-framework/docs/4.2.x/spring-framework-reference/html/transaction.html specifically this section
Secondly, you may wish to enable trace level logs and potentially the SQL statements to help debug this. In order to do so I added the following to my application.properties:
spring.jpa.properties.hibernate.show_sql=false
spring.jpa.properties.hibernate.use_sql_comments=true
spring.jpa.properties.hibernate.format_sql=true
spring.jpa.properties.hibernate.type=trace
spring.jpa.show-sql=true
logging.level.org.hibernate=TRACE
There will be ALOT of output here, but you'll get a good idea of whats happening behind the scenes.
Third, and the most important part for me in learning how to use #Transactional is that every call to the DAO creates a new session -or- reuses the existing session if within the same transactional scope. Refer to the documentation above for examples of this.
At the outset, I have tried the options mentioned in various forums for the same stack trace I get. A few of them did not work while with others (like removing javax.persistence.transactiontype) I did not understand how and where to try it.
I am using Spring Boot data JPA (1.2 RC2) + Hibernate (with a custom persistence.xml). Here is my Application.java
#Configuration
#ComponentScan(<our package>)
#EnableAutoConfiguration(exclude = EmbeddedServletContainerAutoConfiguration.class)
#EnableTransactionManagement
#DependsOn("transactionManager")
#EnableJpaRepositories(transactionManagerRef = "transactionManager")
public class Application {
public static void main(String[] args) {
run(Application.class, args);
}
}
My RepositoryConfiguration (as we have custom persistence.xml - currently need to reuse it)
#Configuration
public class RepositotyConfiguration {
#Autowired
private DataSource dataSource;
#Value("${db.dialect}")
private String dialectClass;
#Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean entityManagerFactory(EntityManagerFactoryBuilder builder) {
LocalContainerEntityManagerFactoryBean entityManagerFactory = builder.dataSource(dataSource).
persistenceUnit("main").build();
entityManagerFactory.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
Properties additionalProperties = new Properties();
additionalProperties.put("hibernate.dialect", dialectClass);
entityManagerFactory.setJpaProperties(additionalProperties);
return entityManagerFactory;
}
#Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(LocalContainerEntityManagerFactoryBean entityManagerFactoryBean) {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(entityManagerFactoryBean.getObject());
return txManager;
}
}
The transactionManager here is created if I do not have a single Repository but the moment I add one, ny tests fail with this exception:
Caused by: java.lang.NullPointerException
at org.hibernate.engine.transaction.internal.jta.JtaStatusHelper.getStatus(JtaStatusHelper.java:76)
at org.hibernate.engine.transaction.internal.jta.JtaStatusHelper.isActive(JtaStatusHelper.java:118)
at org.hibernate.engine.transaction.internal.jta.CMTTransaction.join(CMTTransaction.java:149)
My test application context is:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = Application.class)
#WebAppConfiguration
#EnableAutoConfiguration
#TransactionConfiguration(transactionManager = "transactionManager")
#IntegrationTest("server.port:0")
#ActiveProfiles("test")
public abstract class TestApplicationContext {
#Autowired
private WebApplicationContext wac;
#Value("${local.server.port}")
private int port;
...
}
An example (not the actual) of the repository I try to add (where let's say Item is a model object)
public interface ItemRepository extends CrudReposity<Item, Long> {
Item findByCode(String code); // this seems to cause the problem, assume 'code' is field in Item
}
Any pointers will be of utmost help.
EDIT: It now fails only if I add extra method in ItemRepository say Item findByItemCode(String itemCode) where let's say itemCode is a field in Item model, but can't understand why?
Thanks,
Paddy
I want to use Spring Data JPA to do the ORM. I have the following declared repository interface:
public interface SegmentRepository extends JpaRepository<Segment, Integer> {
// query methods ...
}
Following is the Java Config class:
#Configuration
#EnableJpaRepositories("com.example.cap.repositories")
#EnableTransactionManagement
public class CAPRepositoryConfig {
#Bean
public DataSource dataSource() {
BasicDataSource ds = new BasicDataSource();
ds.setDriverClassName(org.postgresql.Driver.class.getName());
ds.setUsername("postgres");
ds.setPassword("password");
ds.setUrl("jdbc:postgresql://localhost:5432/postgres");
ds.setInitialSize(10);
return ds;
}
#Bean
public EntityManagerFactory entityManagerFactory() {
EclipseLinkJpaVendorAdapter vendorAdapter = new EclipseLinkJpaVendorAdapter();
vendorAdapter.setGenerateDdl(true);
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
Map<String, Object> jpaProperties = new HashMap<String, Object>();
jpaProperties.put("eclipselink.weaving", "false");
factory.setJpaVendorAdapter(vendorAdapter);
factory.setPackagesToScan("com.example.cap.repositories");
factory.setDataSource(dataSource());
factory.setJpaPropertyMap(jpaProperties);
factory.afterPropertiesSet();
return factory.getObject();
}
#Bean
public PlatformTransactionManager transactionManager() {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(entityManagerFactory());
return txManager;
}
}
And the Segment class is defined in com.example.cap.repositories as:
#Entity
public class Segment {
#Id
private int segmentID;
private int caseID;
private Timestamp segStartTime;
private Timestamp segEndTime;
//setter and getters
}
But when I run the JUnit test using auto injected bean SegmentRepository, I got null point exception for the bean repository:
#ContextConfiguration(classes=CAPRepositoryConfig.class)
public class CAPRepositoryTest {
#Autowired
private SegmentRepository repository;
#Test
public void testRepository() {
Segment seg = repository.findOne(123); //null pointer exception for repository
}
}
According to the Spring Data JPA documentation, the SegmentRepository bean repository should be auto injected as long as I specify #EnableJpaRepositories in the Java Config class. But why do I get null pointer exception for repository in the JUnit test class? Since SegmentRepository is an interface rather than a class, I cannot create the instance through Java Config class.
I think you forget SpringJUnit4ClassRunner which makes #Autowired in tests work:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes=CAPRepositoryConfig.class)
public class CAPRepositoryTest { ... }
When I work with test classes and I need to do an unit test, I prefer instantiate the class because although you have an interface you need to have to an implementation class too. In my case I do something like this:
#ContextConfiguration(classes=CAPRepositoryConfig.class)
public class CAPRepositoryTest {
private SegmentRepository repository;
#Before
public void testRepository() {
repository = new SegmentRepositoryImpl();
}
#Test
public void testRepository() {
Segment seg = repository.findOne(123);
}
}