Does placement of beans make a different when loading them into a scoped context? Is this a bug or a timing of instantiation issue?
If I include the #StepScope and #Bean directly in the BatchConfiguration class, everything works seamlessly with StepScope. However, if I define another class, say "BatchProcessProcessor" as included below, and mark a method within that other class as a Bean with StepScope, it does not resolve properly. The actual symptom in spring batch is StepScope not triggering and the beans being loaded as Singletons.
Something about providing the #Bean and #StepScope from another class that is loaded via constructor injection in the BatchConfiguration does not resolve properly.
Format described above, included below:
Main batch configuration class
#Slf4j
#Configuration
#EnableAutoConfiguration
#EnableBatchProcessing
public class BatchConfiguration extends DefaultBatchConfigurer {
private BatchProcessProcessor processor;
#Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autwired
public BatchConfiguration(BatchProcessProcessor processor){
this.processor = processor;
}
#Bean
#StepScope
public ListItemReader<String> reader() {
List<String> stringList = new ArrayList<>();
stringList.add("test");
stringList.add("another test");
log.info("LOGGING A BUNCH OF STUFF THIS IS UNIQUE" + String.valueOf(System.currentTimeMillis()));
return new ListItemReader<>(stringList);
}
#Bean
#StepScope
public CustomWriter writer() {
return new CustomWriter();
}
#Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.<String, String> chunk(10)
.reader(reader())
.processor(processor.processor())
.writer(writer()).build();
}
}
Processor container class
#Component
public class BatchProcessProcessor {
private MyService service;
#Autowired
BatchProcessProcessor(MyService service){
this.service= service;
}
/**
* Generate processor utilized for processing
* #return StringProcessor for testing
*/
#Bean
#StepScope
public DeploymentProcesser processor() {
return new DeploymentProcessor(service);
}
}
Actual Processor
#Slf4j
#Component
public class DeploymentProcesser implements ItemProcessor<Deployment, Model> {
private MyService service;
#Autowired
public DeploymentProcesser(MyService service){
this.service= service;
}
#Override
public Model process(final Deployment deployment) {
log.info(String.format("Processing %s details", deployment.getId()));
Model model = new Model();
model.setId(deployment.getId());
return model;
}
}
As far as I understand, when the BatchConfiguration loads it should inject the BatchProcessProcessor and load the bean with stepscope, but that doesn't seem to work.
As I said before, just copy-pasting the #Bean/#StepScope directly into the BatchConfiguration and returning the same DeploymentProcessor works perfectly and StepScope resolves.
Is this a lifecycle issue?
It does not make sense to declare a bean in a class annotated with #Component:
#Component
public class BatchProcessProcessor {
private MyService service;
#Autowired // This is correct, you can autowire collaborators
public DeploymentProcesser(MyService service){
this.service= service;
}
#Bean // THIS IS NOT CORRECT
#StepScope
public DeploymentProcesser processor() {
return new DeploymentProcessor(service);
}
}
You should rather do it in a configuration class annotated with #Configuration. That's why it works when you do it in BatchConfiguration.
Related
I'm doing the java examples from the book Spring Batch In Action chapter 1.
In this example, a tasket unzips a zip file. The tasklet receives the zip file path as a job parameter.
I implemented a test method that runs the job and passes the parameters.
#StepScope
#Component
public class DecompressTasklet implements Tasklet {
private static final Logger LOGGER = LogManager.getLogger(DecompressTasklet.class);
#Value("#{jobParameters['inputResource']}")
private Resource inputResource;
#Value("#{jobParameters['targetDirectory']}")
private String targetDirectory;
#Value("#{jobParameters['targetFile']}")
private String targetFile;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
//code here
}
}
#Configuration
public class DescompressStep {
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private DecompressTasklet decompressTasklet;
#Bean
public Step stepDescompress() {
return stepBuilderFactory
.get(DescompressStep.class.getSimpleName())
.tasklet(decompressTasklet)
.build();
}
}
#EnableBatchProcessing
#Configuration
public class ImportProductsJob {
#Autowired
private DescompressStep descompressStep;
#Autowired
private ReadWriteProductStep readWriteProductStep;
#Bean
public Job job(JobBuilderFactory jobBuilderFactory) {
return jobBuilderFactory
.get("importProductsJob")
.start(descompressStep.stepDescompress())
.next(readWriteProductStep.stepReaderWriter())
.incrementer(new RunIdIncrementer())
.build();
}
}
Below is the test code that runs the job
#RunWith(SpringRunner.class)
#SpringBootTest
#SpringBatchTest
#AutoConfigureTestDatabase
public class ImportProductsIntegrationTest {
#Autowired
private JobRepositoryTestUtils jobRepositoryTestUtils;
#Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
#After
public void cleanUp() {
jobRepositoryTestUtils.removeJobExecutions();
}
#Test
public void importProducts() throws Exception {
jobLauncherTestUtils.launchJob(defaultJobParameters());
}
private JobParameters defaultJobParameters() {
JobParametersBuilder paramsBuilder = new JobParametersBuilder();
paramsBuilder.addString("inputResource", "classpath:input/products.zip");
paramsBuilder.addString("targetDirectory", "./target/importproductsbatch/");
paramsBuilder.addString("targetFile", "products.txt");
paramsBuilder.addLong("timestamp", System.currentTimeMillis());
return paramsBuilder.toJobParameters();
}
}
The products.zip file is in src/main/resources/input
The problem is that when running the test the error occurs
java.lang.NullPointerException: null
at com.springbatch.inaction.ch01.DecompressTasklet.execute(DecompressTasklet.java:62) ~[classes/:na]
I verified that the inputResource property is null. Why does this error occur?
In your job definition, you have:
#Bean
public Job job(JobBuilderFactory jobBuilderFactory) {
return jobBuilderFactory
.get("importProductsJob")
.start(descompressStep.stepDescompress())
.next(readWriteProductStep.stepReaderWriter())
.incrementer(new RunIdIncrementer())
.build();
}
The way you are passing steps to start and next methods is incorrect (I don't even see how this would compile). What you can do is import step configuration classes and inject both steps in your job definition. Something like:
#EnableBatchProcessing
#Configuration
#Import({DescompressStep.class, ReadWriteProductStep.class})
public class ImportProductsJob {
#Bean
public Job job(JobBuilderFactory jobBuilderFactory,
Step stepDescompress, Step stepReaderWriter) {
return jobBuilderFactory
.get("importProductsJob")
.start(stepDescompress)
.next(stepReaderWriter)
.incrementer(new RunIdIncrementer())
.build();
}
}
I am trying to load data from SQL server, apply some transformations and put it into CSV using the spring batch scheduler. All works fine when everything is in the same class.
This is my code:
package com.abc.tools.bootbatch;
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
private static final String qry = "select top 20 colA, colB, colC from ABC";
private Resource outputResource = new FileSystemResource("output/outputData.csv");
#Bean
public DataSource dataSource() {
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driver_class);
dataSource.setUrl("db_url");
dataSource.setUsername(usr);
dataSource.setPassword(pwd);
return dataSource;
}
#Bean
ItemReader<Trade> reader() {
JdbcCursorItemReader<Trade> databaseReader = new JdbcCursorItemReader<>();
databaseReader.setDataSource(dataSource);
databaseReader.setSql(qry);
databaseReader.setRowMapper(new BeanPropertyRowMapper<>(Trade.class));
return databaseReader;
}
#Bean
public TradeProcessor processor() {
return new TradeProcessor();
}
#Bean
public FlatFileItemWriter<Trade> writer()
{
//Create writer instance
FlatFileItemWriter<Trade> writer = new FlatFileItemWriter<>();
//Set output file location
writer.setResource(outputResource);
//All job repetitions should "append" to same output file
writer.setAppendAllowed(true);
//Name field values sequence based on object properties
writer.setLineAggregator(new DelimitedLineAggregator<Trade>() {
{
setDelimiter(",");
setFieldExtractor(new BeanWrapperFieldExtractor<Trade>() {
{
setNames(new String[] { "colA", "colB", "colC" });
}
});
}
});
return writer;
}
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").<Trade, Trade> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer())
.build();
}
#Bean
public Job exportUserJob() {
return jobBuilderFactory.get("exportUserJob")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
}
When I seperate the processing, loading and data reading in different classes, it works fine using autowire, unless I use batch job. On using the batch job it gives error in instantiating the database.
So I removed the autowire and tried to do something like this:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public DBConfig dbConfig;
public DataConnection dataconnection=new DataConnection();
DataReader reader=new DataReader();
TradeProcessor processor=new TradeProcessor();
FlatFileWriter flatFileWriter=new FlatFileWriter();
DataSource ds=dataconnection.getDataSource(dbConfig);
#Bean
public Step step1() {
return stepBuilderFactory.get("step1").<Trade, Trade> chunk(10)
.reader(reader.reader(ds))
.processor(processor.processor())
.writer(flatFileWriter.writer())
.build();
}
#Bean
public Job exportUserJob() {
return jobBuilderFactory.get("exportUserJob")
.incrementer(new RunIdIncrementer())
.flow(step1())
.end()
.build();
}
}
This gives Failed to initialize BatchConfiguration
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'batchConfiguration'
I think I am missing something to aggregate it all. I am new to Spring, any help is appreciated
In your first example, you are autowiring a datasource and declaring a datasource bean in the same class which is incorrect. In the second example, instead of autowiring DBConfig, you can import it with #Import(DBConfig.class) and autowire the datasource in your job configuration as needed. Here is a typical configuration:
#Configuration
public class DBConfig {
#Bean
public DataSource dataSource() {
final DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(driver_class);
dataSource.setUrl("db_url");
dataSource.setUsername(usr);
dataSource.setPassword(pwd);
return dataSource;
}
}
#Configuration
#EnableBatchProcessing
#Import(DBConfig.class)
public class BatchConfiguration {
#Bean
ItemReader<Trade> reader(DataSource datasource) {
// use datasource to configure the reader
}
}
Since you use Spring Boot, you can remove the DBConfig class, configure the datasource as needed in your application.properties file and the datasource will be automatically injected in your BatchConfiguration.
I've developed Spring Boot-Batch code and sonarlint/Sonar-Qube giving me below error. I went through the link, but solution is not understood as per this link: https://rules.sonarsource.com/java/tag/spring/RSPEC-3305.
Inject this field value directly into "studentStepOne", the only method that uses it.
Code:
#Configuration
#PropertySource("classpath:application.properties")
public class StudentJob {
#Value( "${spring.chunk.size}")
private String chunkSize;
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private JdbcCursorItemReader<Student> StudentReader;
#Autowired
private ItemProcessor<Student, Student> StudentProcessor;
#Autowired
private StudentWriter StudentWriter;
#Bean
public StudentStepExecuListner StudentStepExecuListner() {
return new StudentStepExecuListner();
}
#Bean("readStudentJob")
#Primary
public Job readStudentJob() {
return jobBuilderFactory.get("readStudentJob")
.incrementer(new RunIdIncrementer())
.start(StudentStepOne())
.build();
}
#Bean
public Step StudentStepOne() {
return stepBuilderFactory.get("StudentStepOne")
.<Student, Student>chunk(Integer.parseInt(chunkSize))
.reader(StudentReader)
.processor(StudentProcessor)
.writer(StudentWriter)
.listener(StudentStepExecuListner())
.build();
}
}
Based on the link, it seems self-explanatory:
That means using parameter injection instead of field injection for
dependencies that are only used in a single #Bean method.
So for your job, instantiate it like this:
#Bean("readStudentJob")
#Primary
public Job readStudentJob(Step StudentStepOne) {
return jobBuilderFactory.get("readStudentJob")
.incrementer(new RunIdIncrementer())
.start(StudentStepOne)
.build();
}
Unrelated, but you should follow java conventions. Methods should use camelcase. StudentStepOne() should be studentStepOne()
In a #Configuration class the spring will read the class trying reference with all existent beans.
So, if you wanna only reference one bean in your method, without share between methods, you don't need wired in a class attribute. Just receive in your method constructor.
#Bean
public Step StudentStepOne(StepBuilderFactory stepBuilderFactory) {
return stepBuilderFactory.get("StudentStepOne")
.<Student, Student>chunk(Integer.parseInt(chunkSize))
.reader(StudentReader)
.processor(StudentProcessor)
.writer(StudentWriter)
.listener(StudentStepExecuListner())
.build();
}
The spring will set the bean automatically.
Below Answer will help you.
private StepBuilderFactory stepBuilderFactory;
#Autowired
public void setStepBuilderFactory(StepBuilderFactory stepBuilderFactory) {
this.stepBuilderFactory= stepBuilderFactory;
}
Instead of implementing field Injection, You can go through with setter injection. Field Injection is not recommended.
when i am using #autowire to inject my dependencies in Configuration
class its giving me as null please refer the code below .
#Configuration
public class DataSourceConfig {
#Autowired
AppService appService;
#Bean
public BeanDefinitionRegistryPostProcessor beanPostProcessor() {
return new BeanDefinitionRegistryPostProcessor() {
public void postProcessBeanFactory(ConfigurableListableBeanFactory arg0) throws BeansException {
}
public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry beanRegistry) throws BeansException {
createBeans(beanRegistry);
}
};
}
private void createBeans(BeanDefinitionRegistry beanRegistry,DataSourceConfigService ds) {
appService.getDbDetails();
appService is null here if i will call it using this way
BeanDefinitionRegistryPostProcessor beanPostProcessor(AppService
appService) then in AppServiceImpl class AppDao dependency will be null
}
}
//// Service
#Service
public class AppServiceImpl implements AppService{
#Autowired
AppDao ds;
#Override
public List<A> getDatabaseConfiguration() {
return ds.getDbDetails(); // here ds is null
}
}
//dao
#Repository
public class AppDaoImpl implements AppDao {
#Qualifier("nameParamJdbcTemplate")
#Autowired
public NamedParameterJdbcTemplate nameParamJdbcTemplate;
#Override
public List<A> getDbDetails() {
return nameParamJdbcTemplate.query(SELECT_QUERY, new DataSourceMapper()); // nameParamJdbcTemplate is null
}
// datasource config
#Configuration
public class DataSourceBuilderConfig {
#Bean(name = "dbSource")
#ConfigurationProperties(prefix = "datasource")
#Primary
public DataSource dataSource1() {
return DataSourceBuilder.create().build();
}
#Bean(name = "nameParamJdbcTemplate")
#DependsOn("dbSource")
#Autowired
public NamedParameterJdbcTemplate jdbcTemplate1(#Qualifier("dbSource") DataSource dbSource) {
return new NamedParameterJdbcTemplate(dbSource);
}
}
What i want is when ever my beanPostProcessor()
is executed i want all my dependent beans should be instantiated ie
#Autowired
AppService appService;
#Autowired
AppDao ds;
#Qualifier("nameParamJdbcTemplate")
#Autowired
public NamedParameterJdbcTemplate nameParamJdbcTemplate;
I am new to spring so any help or working examples would be great. Thanks
It is null because this #Configuration class also defines a BeanDefinitionRegistryPostProcessor that forces the context to create that bean very early on.
Because you are using field injection, the context has to resolve AppService bean but it can't yet because the post-processor have to be applied first.
Your configuration looks very complex so you may want to simplify it a bit:
Separate low-level infrastructure configuration from main configuration
Always define such post processor as public static method so that the context can invoke the #Bean method without having to construct the class first.
I am practising on spring-social and it seems that the userConnectionRepository is not properly autowired in the following code when I do a "Run as Junit Test" in Eclipse. I get a Null pointer exception on the usersConnectionRepository when creating a new FacebookOffLine although breakpoints put in the #Bean java creation code shows that they seem to be properly created. Thanks in advance,
public class FacebookOffline {
private Facebook fb;
#Autowired
private UsersConnectionRepository usersConnectionRepository;
public FacebookOffline(User user) {
super();
ConnectionRepository cr = usersConnectionRepository.createConnectionRepository(user.getId());
fb = cr.getPrimaryConnection(Facebook.class).getApi();
}
}
Here is the test code :
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {
org.springframework.social.quickstart.config.MainConfig.class,
org.springframework.social.quickstart.config.SocialConfig.class })
public class FacebookOfflineTest {
#Test
public void test1() {
FacebookOffline essai = new FacebookOffline(new User("yves"));
And the Spring configuration classes adapted from Keith Donald Quick Start Sample :
#Configuration
#ComponentScan(basePackages = "org.springframework.social.quickstart", excludeFilters = { #Filter(Configuration.class) })
#PropertySource("classpath:org/springframework/social/quickstart/config/application.properties")
public class MainConfig {
#Bean
public DataSource datasource() {
DriverManagerDataSource toReturn = new DriverManagerDataSource("jdbc:mysql://localhost:3306/spring_social");
toReturn.setDriverClassName("com.mysql.jdbc.Driver");
toReturn.setUsername("spring");
toReturn.setPassword("spring");
return toReturn;
}
}
#Configuration
public class SocialConfig {
#Inject
private Environment environment;
#Inject
private DataSource dataSource;
#Bean
public ConnectionFactoryLocator connectionFactoryLocator() {
ConnectionFactoryRegistry registry = new ConnectionFactoryRegistry();
registry.addConnectionFactory(new FacebookConnectionFactory(environment
.getProperty("facebook.clientId"), environment
.getProperty("facebook.clientSecret")));
return registry;
}
#Bean
public UsersConnectionRepository usersConnectionRepository() {
JdbcUsersConnectionRepository repository = new JdbcUsersConnectionRepository(
dataSource, connectionFactoryLocator(), Encryptors.noOpText());
return repository;
}
}
Actually there are 2 problems here.
Spring cannot autowire beans it doesn't control (i.e. created with new)
Dependencies aren't available in the constructor (an object instance is needed before it can be injected)
The first one can be mitigated by letting spring manage an instance of FacebookOffline (or if you need multiple instances make the bean request or session scoped).
The second is a bit harder but can probaly solved by using a method annotated with #PostConstruct (or by implementing InitializingBean from spring).
You did
FacebookOffline essai = new FacebookOffline(new User("yves"));
That means, Spring isn't managing this essai instance and thus spring can't autowire any variables in the essai.
You'll have to create bean of FacebookOffline in SocialConfig.
Then you can have
/* ... */
public class FacebookOfflineTest {
#Autowired
ApplicationContext context;
#Test
public void test1() {
FacebookOffline essai = context.getBean(FacebookOffline.class);
OR
/* ... */
public class FacebookOfflineTest {
#Autowired
FacebookOffline essai;
#Test
public void test1() {
// You can use essai now
Also, you'll need to update FacebookOffline as Dependencies ain't available in constructor.
public class FacebookOffline {
private Facebook fb;
#Autowired
private UsersConnectionRepository usersConnectionRepository;
public FacebookOffline(User user) {
super();
}
#PostConstruct
void loadFacebook() {
ConnectionRepository cr = usersConnectionRepository.createConnectionRepository(user.getId());
fb = cr.getPrimaryConnection(Facebook.class).getApi();
}
}
Spring can't autowire fields on an instance you create via new since it doesn't know about it. Declare a bean of type FacebookOffline instead.