Develop Reference

Why jackson can't cast long to long into my spring-batch?

I'm trying to read datas from database into my spring-batch application.
I have one oracle table with a lot of fields and i just want to get a list of id (primary key).
So i have created a simple job with 2 steps (reader and writer).
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Bean
public Job importPurchase(
JobBuilderFactory jobBuilderFactory, Step findProSellerAccountStep, Step importPurchaseStep) {
return jobBuilderFactory
.get("importJob")
.incrementer(new RunIdIncrementer())
.start(findProSellerAccountStep)
// .next(importPurchaseStep)
.build();
}
#Bean
#JobScope
public Step findProSellerAccountStep(
StepBuilderFactory stepBuilderFactory,
ItemReader<Long> accountItemReader,
ItemWriter<Long> accountItemWriter) {
return stepBuilderFactory
.get("findProSellersStep")
.<Long, Long>chunk(10)
.reader(accountItemReader)
.writer(accountItemWriter)
.build();
}
}
And i have also created an item reader :
#Component
#JobScope
#Scope(proxyMode = ScopedProxyMode.DEFAULT)
public class AccountItemReader extends RepositoryItemReader<Long> {
public AccountItemReader(AccountEntityRepository accountRepository) {
// Get all elements
this.setMethodName("findAllAccount");
HashMap<String, Direction> sorts = new HashMap<>();
sorts.put("userAccountId", Sort.Direction.ASC);
this.setSort(sorts);
this.setPageSize(10);
this.setRepository(accountRepository);
}
}
Finally i have created a repository :
#Repository
public interface AccountEntityRepository extends PagingAndSortingRepository<AccountEntity, Long> {
#Query(
value =
"""
SELECT userAccountId
FROM AccountEntity
WHERE myparam = 30 and anotherparam = 20""",
countQuery =
"""
SELECT COUNT(userAccountId)
FROM AccountEntity
WHERE myparam = 30 and anotherparam = 20""")
Page<Long> findAllActiveAccount(Pageable page);
}
But when i try to execute my program with Intellij IDEA, i have got a big error :
2022-12-01 14:35:30.655 ERROR [o.s.b.SpringApplication ] - Application run failed
java.lang.IllegalStateException: Failed to execute ApplicationRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:765)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
at com.rakuten.merchant_index.MerchantIndexApplication.main(MerchantIndexApplication.java:10)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.lang.IllegalArgumentException: Unable to deserialize the execution context
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao$ExecutionContextRowMapper.mapRow(JdbcExecutionContextDao.java:328)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao$ExecutionContextRowMapper.mapRow(JdbcExecutionContextDao.java:312)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:94)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:61)
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:723)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao.getExecutionContext(JdbcExecutionContextDao.java:114)
at org.springframework.batch.core.explore.support.SimpleJobExplorer.getJobExecutionDependencies(SimpleJobExplorer.java:239)
at org.springframework.batch.core.explore.support.SimpleJobExplorer.getLastJobExecution(SimpleJobExplorer.java:104)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy4/jdk.proxy4.$Proxy107.getLastJobExecution(Unknown Source)
at org.springframework.batch.core.JobParametersBuilder.getNextJobParameters(JobParametersBuilder.java:265)
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.getNextJobParameters(JobLauncherApplicationRunner.java:213)
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:198)
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173)
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160)
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155)
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:762)
... 10 common frames omitted
Caused by: com.fasterxml.jackson.databind.JsonMappingException: The class with [Ljava.lang.Object; and name of [Ljava.lang.Object; is not trusted. If you believe this class is safe to deserialize, you can add it to the base set of trusted classes at construction time or provide an explicit mapping using Jackson annotations or a custom ObjectMapper. If the serialization is only done by a trusted source, you can also enable default typing. (through reference chain: java.util.HashMap["accountIds"]->java.util.ArrayList[0])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:392)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:363)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:371)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
at com.fasterxml.jackson.databind.deser.std.StdDelegatingDeserializer.deserialize(StdDelegatingDeserializer.java:175)
at com.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer._deserialize(AsArrayTypeDeserializer.java:120)
at com.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromArray(AsArrayTypeDeserializer.java:53)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromAny(AsPropertyTypeDeserializer.java:211)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserializeWithType(UntypedObjectDeserializer.java:781)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:611)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:437)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:144)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:110)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserializeWithType(MapDeserializer.java:482)
at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3674)
at org.springframework.batch.core.repository.dao.Jackson2ExecutionContextStringSerializer.deserialize(Jackson2ExecutionContextStringSerializer.java:133)
at org.springframework.batch.core.repository.dao.Jackson2ExecutionContextStringSerializer.deserialize(Jackson2ExecutionContextStringSerializer.java:104)
at org.springframework.batch.core.repository.dao.JdbcExecutionContextDao$ExecutionContextRowMapper.mapRow(JdbcExecutionContextDao.java:325)
... 41 common frames omitted
Caused by: java.lang.IllegalArgumentException: The class with [Ljava.lang.Object; and name of [Ljava.lang.Object; is not trusted. If you believe this class is safe to deserialize, you can add it to the base set of trusted classes at construction time or provide an explicit mapping using Jackson annotations or a custom ObjectMapper. If the serialization is only done by a trusted source, you can also enable default typing.
at org.springframework.batch.core.repository.dao.Jackson2ExecutionContextStringSerializer$TrustedTypeIdResolver.typeFromId(Jackson2ExecutionContextStringSerializer.java:353)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:159)
at com.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer._deserialize(AsArrayTypeDeserializer.java:97)
at com.fasterxml.jackson.databind.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromArray(AsArrayTypeDeserializer.java:53)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromAny(AsPropertyTypeDeserializer.java:211)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserializeWithType(UntypedObjectDeserializer.java:781)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:357)
... 61 common frames omitted
I don't understand because there is no "HashMap["accountIds"]" into my actual java program.
So why jackson try to create this Map ? Is it an old cache ?

Finally, it was only an old code that was stored in the spring-batch context and that was reused each time the batch was restarted. I delete my job in database so i wouldn’t have to worry about it.

Spring Batch - Unable to save the records into Postgres database

I am using the same code: na] Caused by: java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist - Spring Batch and unable to save the records into DB.
I already have the table created
CREATE TABLE "BATCH".employee (
rowid_object int4 NOT NULL,
status varchar NULL
)
WITH (
OIDS=FALSE
) ;
-- Permissions
ALTER TABLE "BATCH".employee OWNER TO postgres;
LifeCycleStatusWriter.java
#Component
public class EmployeeWriter implements ItemWriter<Employee> {
#Autowired
#Qualifier(value="postgresDS")
private DataSource dataSourcePostgres;
private NamedParameterJdbcTemplate namedParamJdbcTemplate;
#PostConstruct
private void postConstruct() {
namedParamJdbcTemplate = new NamedParameterJdbcTemplate(this.dataSourcePostgres);
}
String sql = "INSERT INTO BATCH.employee(id, status) VALUES (:rowid_object, :status)";
#SuppressWarnings("unchecked")
#Override
public void write(List<? extends Employee> items) throws Exception {
List<Map<String, Object>> batchValues = new ArrayList<>(items.size());
for (LifeCycleStatus l : items) {
System.out.println("id : "+l.getRowIdObject());
System.out.println("status : "+l.getLifeCycleStatCd());
batchValues.add(new MapSqlParameterSource("id", l.getRowIdObject())
.addValue("status", l.getLifeCycleStatCd()).getValues());
}
int[] updateCounts = namedParamJdbcTemplate.batchUpdate(sql, batchValues.toArray(new Map[items.size()]));
System.out.println(updateCounts);
}
}
Writer
#Configuration
public class EmployeeBatchConfig {
private static final String SQL = "SQL HERE";
#Autowired
#Qualifier(value="oracleDS")
private DataSource dataSourceOracle;
#Autowired
#Qualifier(value="postgresDS")
private DataSource dataSourcePostgres;
#Bean(destroyMethod = "")
#StepScope
public JdbcCursorItemReader<Employee> EmployeeReader() throws Exception {
JdbcCursorItemReader<Employee> reader = new JdbcCursorItemReader<>();
reader.setDataSource(this.dataSourceOracle);
reader.setSql(SQL);
reader.setRowMapper(new EmployeeRowMapper());
reader.afterPropertiesSet();
return reader;
}
#Bean
public EmployeeWriter getEmployeeWriter() {
return new EmployeeWriter();
}
}
Error:
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [INSERT INTO program.BATCH.employee(rowid_object, status) VALUES (?, ?)]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "batch.employee" does not exist
Position: 13
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:235) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1443) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:633) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:647) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:936) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.batchUpdate(NamedParameterJdbcTemplate.java:366) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.batchUpdate(NamedParameterJdbcTemplate.java:354) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at com.mastercard.customer.data.management.writer.LifeCycleStatusWriter.write(LifeCycleStatusWriter.java:48) ~[classes/:na]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:410) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:319) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:147) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) [spring-aop-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) [spring-aop-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.2.1.RELEASE.jar:4.2.1.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at com.sun.proxy.$Proxy72.run(Unknown Source) [na:na]
at com.mastercard.customer.data.management.CustomerProfileStagingBatchApplication.run(CustomerProfileStagingBatchApplication.java:50) [classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784) [spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768) [spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) [spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) [spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at com.mastercard.customer.data.management.CustomerProfileStagingBatchApplication.main(CustomerProfileStagingBatchApplication.java:38) [classes/:na]
Caused by: org.postgresql.util.PSQLException: ERROR: relation "batch.employee" does not exist
Position: 13
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2578) ~[postgresql-42.2.11.jar:42.2.11]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2313) ~[postgresql-42.2.11.jar:42.2.11]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:331) ~[postgresql-42.2.11.jar:42.2.11]
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:832) ~[postgresql-42.2.11.jar:42.2.11]
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:874) ~[postgresql-42.2.11.jar:42.2.11]
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569) ~[postgresql-42.2.11.jar:42.2.11]
at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128) ~[HikariCP-3.4.2.jar:na]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) ~[HikariCP-3.4.2.jar:na]
at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$2(JdbcTemplate.java:950) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:617) ~[spring-jdbc-5.2.5.RELEASE.jar:5.2.5.RELEASE]
... 45 common frames omitted
config details
# ORACLE DATASOURCE - Primary
spring.datasource.url=jdbc:oracle:thin:#/****:1527/test
spring.datasource.username=***
spring.datasource.password=****
spring.datasource.driver-class-name=oracle.jdbc.OracleDriver
# PostgreSQL DB - "Secondary"
postgres.datasource.url=jdbc:postgresql://localhost:5432/test?currentSchema=BATCH
postgres.datasource.username=****
postgres.datasource.password=****
postgres.datasource.driver-class-name=org.postgresql.Driver
#By default, Spring runs all the job as soon as it has started its context.
spring.batch.job.enabled=false
# Chunk Size to save data
spring.chunk.size=200
spring.batch.initialize-schema=always
Database config
#Configuration
public class DatabaseConfig {
#Autowired
private Environment env;
#Bean(name = "oracleDS")
public DataSource batchDataSource() {
return DataSourceBuilder.create().url(env.getProperty("spring.datasource.url"))
.driverClassName(env.getProperty("spring.datasource.driver-class-name"))
.username(env.getProperty("spring.datasource.username"))
.password(env.getProperty("spring.datasource.password")).build();
}
// All metadata tables are present here
#Primary
#Bean(name = "postgresDS")
public DataSource mysqlBatchDataSource() {
return DataSourceBuilder.create().url(env.getProperty("postgres.datasource.url"))
.driverClassName(env.getProperty("postgres.datasource.driver-class-name"))
.username(env.getProperty("postgres.datasource.username"))
.password(env.getProperty("postgres.datasource.password")).build();
}
}

You created your tables's schema with double quotes ("BATCH".employee), and now the names are case sensitive. Ref
"BATCH".employee and BATCH.employee are two different names.
Try to use this
String sql = "INSERT INTO \"BATCH\".employee(rowid_object, status) VALUES (:rowid_object, :status)";

How to use HibernateCursorItemReader in spring batch

In a spring batch job, I have a step in which I want to read data by chunk from the database using Hibernate, process it and write it back to the database. Right now I'm having some trouble setting up the reader :
#EnableBatchProcessing
#Configuration
#ComponentScan(basePackages = {"com.x.y.z.database", "com.x.y.p.database"}, excludeFilters={
#ComponentScan.Filter(type= FilterType.ASSIGNABLE_TYPE, value=ConfigHelperMail.class)})
#EntityScan(basePackages = {"com.x.y.z.database", "com.x.y.p.database"})
#EnableJpaRepositories({"com.x.y.z.database", "com.x.y.p.database"})
public class BatchConfiguration {
#Autowired
public JobBuilderFactory jobs;
#Bean
public Job job(CustomJobListener listener,
#Qualifier("step1") Step step1,
#Qualifier("step2") Step step2,
#Qualifier("step3") Step step3) {
return jobs.get("SimpleJobName")
.incrementer(new RunIdIncrementer())
.preventRestart()
.listener(listener)
.start(step1)
.next(step2)
.next(step3)
.build();
}
}
In step1 writer I save the data to the database using Hibernate :
#Entity
#Table(name = "tmp_z_stepone")
public class StepOneEntity {
#Id
#Column(name = "reference")
private String reference;
/** Other properties and getters and setters **/
}
#Component
public class StepOneWriter implements ItemWriter<StepOneEntity> {
#Autowired
private StepOneService stepOneService;
#Override
public void write (List<? extends StepOneEntity> items) throws WriterException {
stepOneService.saveMany(items);
}
}
StepOneService is annotated with #Service and it uses a DAO interface that extends CrudRepository<StepOneEntity, String> and its saveMany method just uses the save method from CrudRepository<StepOneEntity, String>.
Step 2
#Component
public class StepOne {
#Autowired
private StepTwoReader reader;
#Autowired
private StepTwoProcessor processor;
#Autowired
private StepTwoWriter writer;
#Bean
#JobScope
#Qualifier("step2")
public Step step2() throws ReaderException {
return stepBuilderFactory.get("step2")
.<StepOneEntity, StepTwoEntity>chunk(10)
.reader(reader.read())
.processor(processor)
.writer(writer)
.build();
}
}
#Component
public class StepTwoReader {
public ItemReader<OutputControleFormat> read () throws ReaderException {
HibernateCursorItemReader itemReader = new HibernateCursorItemReader();
itemReader.setQueryString("from tmp_z_stepone");
itemReader.setUseStatelessSession(true);
return itemReader;
}
}
I think I'm missionf some session/ORM configuration but I don't know where, the error is below :
org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:147) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:96) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.step.item.ChunkMonitor.open(ChunkMonitor.java:114) ~[spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:96) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:310) ~[spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:197) ~[spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_92]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_92]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at com.sun.proxy.$Proxy110.execute(Unknown Source) [na:na]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:392) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_92]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_92]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at com.sun.proxy.$Proxy107.run(Unknown Source) [na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:211) [spring-boot-autoconfigure-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:227) [spring-boot-autoconfigure-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:123) [spring-boot-autoconfigure-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:117) [spring-boot-autoconfigure-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:732) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:716) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:703) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at com.cdn.oxc.signature.ApplicationConfig.main(ApplicationConfig.java:33) [classes/:na]
Caused by: java.lang.NullPointerException: null
at org.springframework.batch.item.database.HibernateItemReaderHelper.createQuery(HibernateItemReaderHelper.java:141) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.item.database.HibernateItemReaderHelper.getForwardOnlyCursor(HibernateItemReaderHelper.java:125) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.item.database.HibernateCursorItemReader.doOpen(HibernateCursorItemReader.java:185) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:144) ~[spring-batch-infrastructure-3.0.8.RELEASE.jar:3.0.8.RELEASE]
... 46 common frames omitted

HibernateCursorItemReader implements the InitializingBean interface, which means that Spring will call its afterPropertiesSet on container initialization if it is declared as a bean. However, in your case, the HibernateCursorItemReader is not declared as a bean in the application context, it is wrapped in a class (StepTwoReader) then created with reader.read().
If you declare the HibernateCursorItemReader as a bean with your current code in your read method, you should get an IllegalStateException with message: "A SessionFactory must be provided". You actually need to configure a SessionFactory for your Hibernate reader.
Another option is to use the HibernateCursorItemReaderBuilder which will ensure the reader is correctly configured.

#StepScope
#Slf4j
#Component
public class CursorItemReader extends HibernateCursorItemReader<ItemEntity> {
public CursorItemReader(EntityManagerFactory entityManagerFactory,
#Value("#{stepExecution}")StepExecution stepExecution) {
this.setName("CursorItemReader");
this.setSessionFactory(entityManagerFactory.createEntityManager().unwrap(org.hibernate.Session.class).getSessionFactory());
this.setQueryString("from ItemEntity");
this.setUseStatelessSession(true);
this.setFetchSize(5);
}
#Override
public ItemEntity read() throws Exception {
ItemEntity item = this.doRead();
return item;
}
}
You can also refer-
https://docs.spring.io/spring-batch/docs/current/reference/html/readersAndWriters.html#HibernateCursorItemReader

Spring boot repository does not save to the DB if called from scheduled job

I have a spring boot application in which I need to schedule a job to read files from a specific directory and store the data into the DB.
I used Spring batch for handling the files part as the number of files is very large.
The application has a component named PraserStarer which has a method named startParsing. This method is annotated with #scheduled annotation.
#scheduled(fixedDelay = 60 * 1000)
public startParsing(){
// start spring batch job
}
I have a repository interface NewsRepositry injected into the writer of the spring batch first step.
The application has a simple controller to manually call the startParsing method. When calling the startParsing method from the controller, everything works fine. The spring batch job starts normally, read the files, write the data into the DB, and archive the files.
When the method startParsing is invoked from the scheduling framework, the spring batch job starts normally, and read the files but nothing is stored the DB.
I suspect the problem here is that there are two different contexts, one for the scheduling part and another for the rest of the application.
For some reason, there is no transaction manager in the scheduling context which causes nothing to go to the DB.
1- Is my suspicion correct?
2- If yes, how can I force the transaction manager to be loaded to the other context?
EDIT
The code for the parser starter class is below
#Component
public class ParserStarter {
#Autowired
JobLauncher jobLauncher;
#Value("${app.data_directory}")
private String dataDir;
#Autowired
private ParserJobListener jobListener;
#Autowired
private JobBuilderFactory jobBuilderFactory;
public Resource[] getResources() throws IOException {
// return array of file resource to be processed
}
// #Scheduled(fixedDelay = 60 * 1000)
public void startParsing() throws Exception {
String jobName = System.currentTimeMillis() + " New Parser Job";
JobParameters jobParameters = new JobParametersBuilder().addString("source", jobName).toJobParameters();
jobLauncher.run(getParsingJob(), jobParameters);
}
#Bean(name="getParsingJob")
private Job getParsingJob() throws IOException {
jobListener.setResources(getResources());
Step processingStep = jobListener.processingStep();
Step archivingStep = jobListener.archivingStep();
Job job = jobBuilderFactory.get("Store News").incrementer(new RunIdIncrementer())
.listener(jobListener).start(processingStep).next(archivingStep).build();
return job;
}
}
The code for the job listener is below
#Component
public class ParserJobListener extends JobExecutionListenerSupport {
#Autowired
private StepBuilderFactory stepBuilderFactory;
private Resource[] resources;
#Value("${app.archive_directory}")
private String archiveDirectory;
#Autowired
private Writer writer;
public MultiResourceItemReader<DataRecord> multiResourceItemReader() {
MultiResourceItemReader<DataRecord> resourceItemReader = new MultiResourceItemReader<DataRecord>();
resourceItemReader.setResources(resources);
resourceItemReader.setDelegate(new Reader());
return resourceItemReader;
}
public Step archivingStep() {
FileArchivingTask archivingTask = new FileArchivingTask(resources, archiveDirectory);
return stepBuilderFactory.get("Archiving step").tasklet(archivingTask).build();
}
public Step processingStep() {
return stepBuilderFactory.get("Process news file").<DataRecord, DataRecord>chunk(1000)
.reader(multiResourceItemReader()).writer(writer).build();
}
#Override
public void afterJob(JobExecution jobExecution) {
if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
System.out.println("Job finished")
}
}
public void setResources(Resource[] resources) {
this.resources = resources;
}
}
What is remaining is the writer and it is below
#Component
public class Writer implements ItemWriter<DataRecord>{
#Autowired
private DataRepository dataRepo;
#Override
public void write(List<? extends DataRecord> items) throws Exception {
dataRepo.saveAll(items);
}
}
Edit 2
I have changed the writer's write method to save and flush each item indiviually as following
#Transactional
public void write(List<? extends GdeltRecord> items) throws Exception {
for (GdeltRecord gdeltRecord : items) {
dataRepo.saveAndFlush(gdeltRecord);
}
// dataRepo.saveAll(items);
}
This time the application throws a TransactionRequiredException: no transaction is in progress exception.
Here is the full stack trace of the exception
Caused by: javax.persistence.TransactionRequiredException: no transaction is in progress
at org.hibernate.internal.SessionImpl.checkTransactionNeeded(SessionImpl.java:3552) ~[hibernate-core-5.3.7.Final.jar:5.3.7.Final]
at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1444) ~[hibernate-core-5.3.7.Final.jar:5.3.7.Final]
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1440) ~[hibernate-core-5.3.7.Final.jar:5.3.7.Final]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:350) ~[spring-orm-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at com.sun.proxy.$Proxy87.flush(Unknown Source) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:308) ~[spring-orm-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at com.sun.proxy.$Proxy87.flush(Unknown Source) ~[na:na]
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.flush(SimpleJpaRepository.java:533) ~[spring-data-jpa-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.saveAndFlush(SimpleJpaRepository.java:504) ~[spring-data-jpa-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:359) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:644) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:608) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.lambda$invoke$3(RepositoryFactorySupport.java:595) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:595) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59) ~[spring-data-commons-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.1.4.RELEASE.jar:5.1.4.RELEASE]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]
... 66 common frames omitted

I tried the approach described in this question (JpaItemWriter: no transaction is in progress) and it worked for me.
I defined a JpaTransactionManager bean and used it with step configuration.
#Bean
#Primary
public JpaTransactionManager jpaTransactionManager() {
final JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setDataSource(dataSource);
return transactionManager;
}
and in the step configuration
#Autowired
JpaTransactionManager trxm;
public Step processingStep(Resource[] resources) throws IOException {
return stepBuilderFactory.get("Process CSV File")
.transactionManager(trxm)
.<DataRecord, DataRecord>chunk(1000)
.reader(multiResourceItemReader()).writer(writer).build();
}

It's difficult to analyze what exactly is happening without seeing the full code. However, based on the documentation, spring uses chunk oriented processing. This is what it says:
Once the number of items read equals the commit interval, the entire chunk is written out via the ItemWriter, and then the transaction is committed.
This might be why you are not seeing any db writes immediately.
Regarding the transaction manager, you can define it like this (explained here):
#Bean
public Job sampleJob(JobRepository jobRepository, Step sampleStep) {
return this.jobBuilderFactory.get("sampleJob")
.repository(jobRepository)
.start(sampleStep)
.build();
}
/**
* Note the TransactionManager is typically autowired in and not needed to be explicitly
* configured
*/
#Bean
public Step sampleStep(PlatformTransactionManager transactionManager) {
return this.stepBuilderFactory.get("sampleStep")
.transactionManager(transactionManager)
.<String, String>chunk(10)
.reader(itemReader())
.writer(itemWriter())
.build();
}

We have to explicitly mention JpaTransactionManager instead of default spring batch transactions.
#Configuration
#EnableBatchProcessing
public class MyJob extends DefaultBatchConfigurer {
#Autowired
private DataSource dataSource;
#Bean
#Primary
public JpaTransactionManager jpaTransactionManager() {
final JpaTransactionManager tm = new JpaTransactionManager();
tm.setDataSource(dataSource);
return tm;
}
}

Updating a LOB/BLOB value on a HSQLDB with Hibernate produces a data exception

Having a strange behaviour with my update code:
Entity:
#Entity
public class Data {
#Id
#GeneratedValue
private long id;
#Lob
private byte[] data;
//Getters and Setters...
}
Spring Data Repository:
#Repository
public interface DataRepository extends JpaRepository<Data, Long> {}
Test Configuration
#Configuration
#EnableTransactionManagement
#EnableJpaRepositories(basePackages={"de.ehscheidt.bugs.data"})
public class TestConfiguration {
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setPersistenceUnitName("data");
em.setDataSource(dataSource());
em.setPackagesToScan(new String[] {"de.ehscheidt.bugs.data"});
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalProperties());
return em;
}
#Bean
public DataSource dataSource(){
EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
return builder
.setType(EmbeddedDatabaseType.HSQL)
.setName("database")
.build();
}
#Bean
public PlatformTransactionManager transactionManager(EntityManagerFactory emf){
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(emf);
return transactionManager;
}
#Bean
public PersistenceExceptionTranslationPostProcessor exceptionTranslation(){
return new PersistenceExceptionTranslationPostProcessor();
}
private Properties additionalProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.hbm2ddl.auto", "update");
properties.setProperty("hibernate.dialect", "org.hibernate.dialect.HSQLDialect");
properties.setProperty("hibernate.show_sql", "true");
return properties;
}
}
Test:
#RunWith(SpringRunner.class)
#ContextConfiguration(classes={TestConfiguration.class})
#Transactional
#SpringBootTest
public class DataTest {
#Autowired
private DataRepository repository;
#Test
public void fail() {
Data d = new Data();
d.setData(new byte[255]);
d = repository.save(d);
repository.flush();
d.setData(new byte[256]);
repository.save(d);
repository.flush();
}
#Test
public void fail2() {
Data d = new Data();
d.setData(new byte[1025]);
d = repository.save(d);
repository.flush();
d.setData(new byte[1026]);
repository.save(d);
repository.flush(); // <- fails here
}
#Test
public void ok() {
Data d = new Data();
d.setData(new byte[254]);
d = repository.save(d);
repository.flush();
d.setData(new byte[255]);
repository.save(d);
repository.flush();
}
}
The fail method is throwing:
org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.DataException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:263)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:225)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:527)
at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:153)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:135)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy45.flush(Unknown Source)
at de.ehscheidt.bugs.data.DataTest.fail(DataTest.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:73)
at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:83)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
Caused by: org.hibernate.exception.DataException: could not execute statement
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:52)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3217)
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3090)
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3491)
at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:145)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:600)
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:474)
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1437)
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1423)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:304)
at com.sun.proxy.$Proxy42.flush(Unknown Source)
at org.springframework.data.jpa.repository.support.SimpleJpaRepository.flush(SimpleJpaRepository.java:534)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:377)
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:629)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:593)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
... 41 more
Caused by: java.sql.SQLDataException: data exception: string data, right truncation
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:175)
... 75 more
Caused by: org.hsqldb.HsqlException: data exception: string data, right truncation
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.error.Error.error(Unknown Source)
at org.hsqldb.types.BlobType.convertToType(Unknown Source)
at org.hsqldb.StatementDML.getUpdatedData(Unknown Source)
at org.hsqldb.StatementDML.executeUpdateStatement(Unknown Source)
at org.hsqldb.StatementDML.getResult(Unknown Source)
at org.hsqldb.StatementDMQL.execute(Unknown Source)
at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
at org.hsqldb.Session.execute(Unknown Source)
... 78 more
Important: The exception is thrown at the second flush which results in an UPDATE.
Used Libs:
spring-boot-starter-data-jpa: 2.0.2.RELEASE
hibernate-core: 5.2.17.Final
hsqldb: 2.4.0
I guess it has to do something with hibernate or/and hsqldb. Interesting is that hibernate generates the table as 'data blob(255)'.
So, INSERT with more than 255 Bytes is ok, but UPDATE with more than it is not?
What am I missing?

I think this is to do with using byte[] as the datatype. It seems that once you have created a record with 255 (or 1025 or in fact any length of) characters that's all that you have available to use. My guess is that HSQLDB treats byte[] LOBs as a primitive type and stores them inline in a variable length record.
If you use Java streams instead then HSQLDB will treat it as a proper variable length LOB. Spring Content uses streams underneath the covers so you wont hit this limitation.
It is super easy to modify your app to use it too:
Add the Spring Content dependencies to your pom.
pom.xml
<dependency
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-jpa-boot-starter</artifactId>
<version>0.0.11</version>
</dependency>
<!-- and if you want the REST endpoints too -->
<dependency>
<groupId>com.github.paulcwarren</groupId>
<artifactId>spring-content-rest-boot-starter</artifactId>
<version>0.0.11</version>
</dependency>
Update your entity so that you can associate content.
data.java
#Entity
public class Data {
#Id
#GeneratedValue
private long id;
#ContentId
private String contentId;
#ContentLength
private long contentLength = 0L;
// if you have rest endpoints
#MimeType
private String mimeType = "text/plain";
...
Define a ContentStore interface:
DataContentStore.java
#StoreRestResource(path="dataContent")
public interface DataContentStore extends ContentStore<Data, String> {
}
And that's it. At runtime Spring Content will manage the database schema for you. It will inject a JPA implementation of DataContentStore that uses java streams rather than byte[]. If you added the rest dependency it will also add a controller offering REST endpoints at '/dataContent'.
Over and above being able to insert larger LOBs over time, the other advantages of using Spring Content are:
you don't have to write any of the code to do this yourself
very large file handling (because it uses streams, not byte arrays)
video streaming is support
full text search is possible (Spring Content Solr)
renditions in other formats are possible
finally, you can easily switch your application to different
storage with just one dependency change
There is a getting started guides that describes almost exactly what you are trying to do here and here. The reference guide for Spring Content JPA is here.
HTH