Develop Reference

How to manage shorter transaction inside Spring Batch chunk?

I've got a problem using Spring Batch and can't seem to find a solution.
So, I've got a batch processing some items in chunks (size 10). I've also got a transaction manager used by this batch to persist the processed items after each chunk.
But... I would also like to persist some progress status for those items, in real time. So before processing an item I want to save a status saying this item is in progress.
And I can't seem to find a solution to achieve that. I tried the following solutions :
If I just annotate my status manager service with Transactional annotation the statuses are commited after the whole chunk processing.
If I add REQUIRES_NEW as propagation level to the annotation, it works... but the batch ends in some kind of deadlock (I read that it was common issue with REQUIRES_NEW).
So my last guess was to add a second transaction manager (on the same datasource) and use it on the status manager service... But I got the same result as solution #1 (which seem wierd to me as I expected the transaction from this manager to act independently of the chunk transaction).
Has anybody ever encountered this problem?
EDIT :
Here is my configuration, simplified on purpose :
Class DbConfiguration:
#Configuration
public class DbConfiguration {
#Bean
#Primary
public JpaTransactionManager transactionManager() throws Exception {
final JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory());
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Class JobConfiguration:
#Configuration
#Import(DbConfiguration.class)
#EnableTransactionManagement
public class JobConfiguration {
#Autowired
private EntityManagerFactory entityManagerFactory;
#Bean
public Job jobDefinition() {
return jobBuilderFactory
.get(JOB_NAME)
.start(step())
.build();
}
#Bean
public Step step() {
return stepBuilderFactory
.get(STEP_NAME)
.<Object, Object>chunk(COMMIT_INTERVAL)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
#Bean
public PlatformTransactionManager statusTransactionManager() {
final JpaTransactionManager jpaTransactionManager = new JpaTransactionManager();
jpaTransactionManager.setEntityManagerFactory(entityManagerFactory);
jpaTransactionManager.afterPropertiesSet();
return jpaTransactionManager;
}
}
Class StatusManagerServiceImpl:
#Transactional("statusTransactionManager")
public class StatusManagerServiceImpl implements StatusManagerService {
...
}

A way to achieve this is to use Spring TransactionTemplate as follows:
#Service
public class StatusManagerServiceImpl implements StatusManagerService {
#Autowired
private PlatformTransactionManager transactionManager;
public separateTransactionMethod() {
TransactionTemplate transactionTemplate = new TransactionTemplate(transactionManager);
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
#Override
protected void doInTransactionWithoutResult(TransactionStatus status) {
// Do something here that will be committed right away.
}
});
}
}
You can alternatively use new TransactionCallback if you got a result to return. Then execute method would return that result.

Spring Transactional doesn't persist an entity using EntityManager but does using Spring Data

Maybe it is kind of too common but still.
I have a small test project where I'm testing all the JPA stuff. Almost everywhere I'm using Spring Data and JPA repositories work just fine. But now I'm trying to make my service to save entities. The service looks something like this:
#Service
public class SomeServiceImpl implements SomeService {
#Autowired
private EntityManagerFactory entityManagerFactory;
public SomeServiceImpl(EntityManagerFactory entityManagerFactory) {
this.entityManagerFactory = entityManagerFactory;
}
#Override
#Transactional
public SomeEntity save(SomeEntity someEntity) {
EntityManager entityManager = entityManagerFactory.createEntityManager();
entityManager.persist(someEntity);
return someEntity;
}
The persistence config looks like this (I'm intentionally copying and pasting the whole config. Maybe it would help you to reproduce the error):
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories
#PropertySource({"classpath:conf/application.properties"})
public class PersistenceConfig {
#Autowired
private Environment environment;
#Bean
public DataSource dataSource() throws SQLException {
PoolDataSourceImpl dataSource = new PoolDataSourceImpl();
dataSource.setConnectionFactoryClassName(environment.getRequiredProperty("db.driverClassName"));
dataSource.setURL(environment.getRequiredProperty("db.url"));
dataSource.setUser(environment.getRequiredProperty("db.username"));
dataSource.setPassword(environment.getRequiredProperty("db.password"));
dataSource.setFastConnectionFailoverEnabled(
Boolean.valueOf(environment.getRequiredProperty("db.fast.connect.failover.enabled")));
dataSource.setValidateConnectionOnBorrow(true);
dataSource.setSQLForValidateConnection("SELECT SYSDATE FROM DUAL");
dataSource.setONSConfiguration(environment.getRequiredProperty("db.ons.config"));
dataSource.setInitialPoolSize(Integer.valueOf(environment.getRequiredProperty("db.initial.pool.size")));
dataSource.setMinPoolSize(Integer.valueOf(environment.getRequiredProperty("db.min.pool.size")));
dataSource.setMaxPoolSize(Integer.valueOf(environment.getRequiredProperty("db.max.pool.size")));
dataSource.setAbandonedConnectionTimeout(0);
dataSource.setInactiveConnectionTimeout(60 * 25);
dataSource.setTimeToLiveConnectionTimeout(0);
dataSource.setMaxConnectionReuseTime(60 * 30L);
return dataSource;
}
private Properties hibernateProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.dialect", environment.getRequiredProperty("hibernate.dialect"));
properties.setProperty("hibernate.show_sql", environment.getRequiredProperty("hibernate.show_sql"));
properties.setProperty("hibernate.format_sql", environment.getRequiredProperty("hibernate.format_sql"));
properties.setProperty("hibernate.hbm2ddl.auto", environment.getRequiredProperty("hibernate.hbm2ddl.auto"));
return properties;
}
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory(#Autowired DataSource dataSource) {
LocalContainerEntityManagerFactoryBean entityManagerFactory = new LocalContainerEntityManagerFactoryBean();
entityManagerFactory.setDataSource(dataSource);
entityManagerFactory.setPackagesToScan("com.dropbinc.learning.jpa.model");
JpaVendorAdapter jpaVendorAdapter = new HibernateJpaVendorAdapter();
entityManagerFactory.setJpaVendorAdapter(jpaVendorAdapter);
Map jpaProperties = new HashMap();
jpaProperties.put("javax.persistence.schema-generation.database.action", "drop-and-create");
entityManagerFactory.setJpaPropertyMap(jpaProperties);
entityManagerFactory.setJpaProperties(hibernateProperties());
return entityManagerFactory;
}
#Bean
public PlatformTransactionManager transactionManager(#Autowired EntityManagerFactory entityManagerFactory) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory);
return transactionManager;
}
#Bean
public PersistenceExceptionTranslationPostProcessor exceptionTranslation() {
return new PersistenceExceptionTranslationPostProcessor();
}
}
And one more where I'm planning to configurate the rest of the application (placed in the same package):
#Configuration
#ComponentScan(basePackages = "com.dropbinc.learning.jpa")
public class AppConfig {
}
I've tried to debug Spring but all that I wasn't able to detect a difference between transaction behaviour of JPA repositories and my service. I saw transaction was created and even commited. But in case of JPA repositories it got saved while in my service implementation it did generated ids but an entity didn't appeared in a database.
I'm running all the stuff in tests, autowiring the service by interface:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = { AppConfig.class, PersistenceConfig.class })
public class AllTheTests {
#Autowired
SomeService someService;
...
}
Thank you very much for any suggestion!
EDIT Adding entityManager.flush() call generates nested exception is javax.persistence.TransactionRequiredException: no transaction is in progress.

Spring Annotation Config with Multiple JPA Persistence Units Doesn't Persist

I started out with this guide, to migrate our xml config to an annotation config.
The current problem is, that my test a persist seems to not actually write the data (= no transaction commit). This results in the next check to fail. The environment currently has five persistence units and according entityManagerFactories and transactionManagers.
MyTest.java
#RunWith(SpringRunner.class)
#ContextConfiguration(classes = {PersistenceJPAConfig.class}, loader = AnnotationConfigContextLoader.class)
public class MyTest {
#Autowired
private MyDao testable;
#Transactional(transactionManager = "tm1") // tried with name= and without
#Test
public void crudTest() throws Exception {
// assert that the table is empty
List<MyDO> all = testable.getAll();
assertTrue(all.isEmpty());
// write one entity
MyDO anEntity = new MyDO();
testable.persist(anEntity);
// load all entities and assert that the details match
List<MyDO> allAfterInsert = testable.getAll();
// THIS FAILS
assertFalse("The database result should not be empty.", allAfterInsert.isEmpty());
}
}
PersistenceJPAConfig.java
#Configuration
#EnableTransactionManagement
#ComponentScan("my.package")
public class PersistenceJPAConfig {
#PersistenceContext(unitName = "myPersistenceUnit") // also tried
#Bean(name = "myEntityManager")
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource());
em.setPackagesToScan(new String[] {"my.package"});
em.setPersistenceUnitName("myPersistenceUnit");
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalProperties());
return em;
}
// ... the same for the other persistenceUnits with increasing names
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("org.h2.Driver");
dataSource.setUrl("jdbc:h2:mem:test;DB_CLOSE_DELAY=-1");
dataSource.setUsername("sa");
dataSource.setPassword("");
return dataSource;
}
#Bean(name = "tm1")
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getNativeEntityManagerFactory());
transactionManager.setPersistenceUnitName("myPersistenceUnit");
transactionManager.afterPropertiesSet();
return transactionManager;
}
// ... the same for the other persistenceUnits with increasing names
#Bean
public PersistenceExceptionTranslationPostProcessor exceptionTranslation() {
return new PersistenceExceptionTranslationPostProcessor();
}
#Bean
public PersistenceAnnotationBeanPostProcessor persistenceAnnotationBeanPostProcessor() {
return new PersistenceAnnotationBeanPostProcessor();
}
Properties additionalProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.hbm2ddl.auto", "update");
properties.setProperty("hibernate.dialect", "org.hibernate.dialect.H2Dialect");
return properties;
}
}
MyDao.java
public class MyDao {
#PersistenceContext(unitName = "myPersistenceUnit")
EntityManager em;
#Override
#Transactional(transactionManager = "tm1") // also tried with different variations like above
public Long persist(final MyDO entity) {
em.persist(entity); // tried to add an em.flush(), but that throws a TransactionRequiredException: no transaction is in progress
// handling transactions would throw IllegalStateException: Not allowed to create transaction on shared EntityManager - use Spring transactions or EJB CMT instead
return entity.getId();
}
}
The persistence units are located in the file META-INF/persistence.xml.
The tests worked with the xml configuration, but don't work with my current annotation config. Is there something I forgot? What more information could I provide?

I found a solution: Instead of entityManagerFactory().getNativeEntityManagerFactory() I used entityManagerFactory().getObject() and everything works as expected.
#Bean(name = "tm1")
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getObject());
transactionManager.setPersistenceUnitName("myPersistenceUnit");
transactionManager.afterPropertiesSet();
return transactionManager;
}
I am not sure why though.

Spring Batch Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException: Transaction already active

My Spring Batch job is started every 5 minutes - basically it reads a string, uses the string as a parameter in a sql query, and prints out the resulting sql result list. Mostly it seems to be running ok, but I notice sporadic errors in my logs every 5-10 runs
2017-05-05 11:13:26.101 INFO 9572 --- [nio-8081-exec-8] c.u.r.s.AgentCollectorServiceImpl : Could not open JPA E
ntityManager for transaction; nested exception is java.lang.IllegalStateException: Transaction already active
My job is started like from my AgentCollectorServiceImpl class
#Override
public void addReportIds(List<Integer> reportIds) {
try {
.toJobParameters();
jobLauncher.run(job, jobParameters);
} catch (Exception e) {
log.info(e.getMessage());
}
}
My BatchConfig class looks like
#Configuration
#EnableBatchProcessing
#Import(AppConfig.class)
public class BatchConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private AppConfig appConfig;
#Bean
public Reader reader() {
return new Reader();
}
#Bean
public Processor processor() {
return new Processor();
}
#Bean
public Writer writer() {
return new Writer();
}
#Bean
public Job job() {
return jobBuilderFactory.get("job")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<String, String> chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
}
My AppConfig class looks like
#Configuration
#PropertySource("classpath:application.properties")
#ComponentScan
public class AppConfig {
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(organizationDataSource());
em.setPackagesToScan(new String[]{"com.organization.agentcollector.model"});
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalProperties());
return em;
}
Properties additionalProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.dialect", "com.organization.agentcollector.config.SQLServerDialectOverrider");
return properties;
}
#Bean
JpaTransactionManager transactionManager(final EntityManagerFactory emf) {
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(entityManagerFactory().getObject());
return transactionManager;
}
My Processor class looks like
public class Processor implements ItemProcessor<String, String> {
private final Logger log = LoggerFactory.getLogger(Processor.class);
#Autowired
EventReportsDAOImpl eventReportsDAOImpl;
#Override
public String process(String reportIdsJson) throws Exception {
String eventReportsJson = eventReportsDAOImpl.listEventReportsInJsonRequest(reportIdsJson);
//System.out.println(returnContent+"PROCESSOR");
return eventReportsJson;
}
}
My DAOImpl class looks like
#Component
#Transactional
public class EventReportsDAOImpl implements EventReportsDAO {
#PersistenceContext
private EntityManager em;
#Override
public EventReports getEventReports(Integer reportId) {
return null;
}
#Override
public String listEventReportsInJsonRequest(String reportIds) {
System.out.println("Event Report reportIds processing");
ArrayList<EventReports> erArr = new ArrayList<EventReports>();
String reportIdsList = reportIds.substring(1, reportIds.length() - 1);
//System.out.println(reportIdsList);
try {
StoredProcedureQuery q = em.createStoredProcedureQuery("sp_get_event_reports", "eventReportsResult");
q.registerStoredProcedureParameter("reportIds", String.class, ParameterMode.IN);
q.setParameter("reportIds", reportIdsList);
boolean isResultSet = q.execute();
erArr = (ArrayList<EventReports>) q.getResultList();
} catch (Exception e) {
System.out.println("No event reports found for list " + reportIdsList);
}
return erArr.toString();
}
I thought Spring would manage transactions automatically. The error seems to suggest that a transaction is not being properly closed?
One thing I tried was removing all #Transactional annotations from my code as I read that #EnableBatchProcessing already injects a Transaction Manager into each step - but when I did this, I saw the 'transaction already active' error much more frequently.
Any advice appreciated on how to fix this, thank you!

The #Transactional notation establishes a transactional scope that dictates when a transaction starts and ends, also called its boundary. If you operate outside of this boundary you'll receive errors.
First off, I found this bit of documentation the most helpful on Spring transactions: http://docs.spring.io/spring-framework/docs/4.2.x/spring-framework-reference/html/transaction.html specifically this section
Secondly, you may wish to enable trace level logs and potentially the SQL statements to help debug this. In order to do so I added the following to my application.properties:
spring.jpa.properties.hibernate.show_sql=false
spring.jpa.properties.hibernate.use_sql_comments=true
spring.jpa.properties.hibernate.format_sql=true
spring.jpa.properties.hibernate.type=trace
spring.jpa.show-sql=true
logging.level.org.hibernate=TRACE
There will be ALOT of output here, but you'll get a good idea of whats happening behind the scenes.
Third, and the most important part for me in learning how to use #Transactional is that every call to the DAO creates a new session -or- reuses the existing session if within the same transactional scope. Refer to the documentation above for examples of this.

Spring boot data JPA getting NullPointerException related to transaction manager

At the outset, I have tried the options mentioned in various forums for the same stack trace I get. A few of them did not work while with others (like removing javax.persistence.transactiontype) I did not understand how and where to try it.
I am using Spring Boot data JPA (1.2 RC2) + Hibernate (with a custom persistence.xml). Here is my Application.java
#Configuration
#ComponentScan(<our package>)
#EnableAutoConfiguration(exclude = EmbeddedServletContainerAutoConfiguration.class)
#EnableTransactionManagement
#DependsOn("transactionManager")
#EnableJpaRepositories(transactionManagerRef = "transactionManager")
public class Application {
public static void main(String[] args) {
run(Application.class, args);
}
}
My RepositoryConfiguration (as we have custom persistence.xml - currently need to reuse it)
#Configuration
public class RepositotyConfiguration {
#Autowired
private DataSource dataSource;
#Value("${db.dialect}")
private String dialectClass;
#Bean(name = "entityManagerFactory")
public LocalContainerEntityManagerFactoryBean entityManagerFactory(EntityManagerFactoryBuilder builder) {
LocalContainerEntityManagerFactoryBean entityManagerFactory = builder.dataSource(dataSource).
persistenceUnit("main").build();
entityManagerFactory.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
Properties additionalProperties = new Properties();
additionalProperties.put("hibernate.dialect", dialectClass);
entityManagerFactory.setJpaProperties(additionalProperties);
return entityManagerFactory;
}
#Bean(name = "transactionManager")
public PlatformTransactionManager transactionManager(LocalContainerEntityManagerFactoryBean entityManagerFactoryBean) {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(entityManagerFactoryBean.getObject());
return txManager;
}
}
The transactionManager here is created if I do not have a single Repository but the moment I add one, ny tests fail with this exception:
Caused by: java.lang.NullPointerException
at org.hibernate.engine.transaction.internal.jta.JtaStatusHelper.getStatus(JtaStatusHelper.java:76)
at org.hibernate.engine.transaction.internal.jta.JtaStatusHelper.isActive(JtaStatusHelper.java:118)
at org.hibernate.engine.transaction.internal.jta.CMTTransaction.join(CMTTransaction.java:149)
My test application context is:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = Application.class)
#WebAppConfiguration
#EnableAutoConfiguration
#TransactionConfiguration(transactionManager = "transactionManager")
#IntegrationTest("server.port:0")
#ActiveProfiles("test")
public abstract class TestApplicationContext {
#Autowired
private WebApplicationContext wac;
#Value("${local.server.port}")
private int port;
...
}
An example (not the actual) of the repository I try to add (where let's say Item is a model object)
public interface ItemRepository extends CrudReposity<Item, Long> {
Item findByCode(String code); // this seems to cause the problem, assume 'code' is field in Item
}
Any pointers will be of utmost help.
EDIT: It now fails only if I add extra method in ItemRepository say Item findByItemCode(String itemCode) where let's say itemCode is a field in Item model, but can't understand why?
Thanks,
Paddy