I coded a custom implementation of ListItemReader, triying to follow the example in the spring batch's github. Anyway, in my case, I need read a variable from the jobContext, this variable is a path where I have to read the files that contains. I can't use the constructor because the constructors executes before the beforeStep event and I don't have these var at this moment.
Anyway this will work first execution, but if the list never goes again to null I can't execute again the initialize method.
If I tried add an else in the !list.isEmpty() condition that set my list to null. I enter in an infinite loop.
There are other methods to solve this? Maybe I am overcomplicating this.
public class ListItemReader<Path> implements ItemReader<Path>, StepExecutionListener {
private List<Path> list;
private org.springframework.batch.core.JobExecution jobExecution;
public ListItemReader() {
this.list = new ArrayList<>();
}
public void initialize(){
//Here I made an listdirectories of a path and add all the files to the list
String pathString = jobExecution.getExecutionContext().getString(Constants.CONTEXT_PATH);
Path path = Paths.get(pathString );
...
items.add(Paths.get(..path..));
}
#Nullable
#Override
public T read() {
if(list == null) initialize();
if (!list.isEmpty()) {
return list.remove(0);
}
return null;
}
#Override
public ExitStatus afterStep(StepExecution se) {
return ExitStatus.COMPLETED;
}
#Override
public void beforeStep(StepExecution se) {
jobExecution = se.getJobExecution();
}
}
I can't use the constructor because the constructors executes before the beforeStep event and I don't have these var at this moment.
Actually, you can delay your bean constructor by using #JobScope and #StepScope. Additionally, you can use #Value and Spring SpEL to inject your jobParameters.
In your case, you might want to rewrite your code, for e.g:
#Configuration
#RequiredArgsConstructor
public class YourJobConfig {
private final StepBuilderFactory stepBuilder;
#Bean("yourStep")
public Step doWhatever() {
return stepBuilder
.get("yourStepName")
.<Path, Path>chunk(50) // <-- replace your chunk size here
.reader(yourItemReader(null, 0)) // <-- place your default values here
.process(yourItemProcessor())
.writer(yourItemWriter())
}
#Bean
public ItemReader<Path> yourItemReader(
#Value("#{jobParameters['attribute1']}") String attribute1,
#Value("#{jobParameters['attribute2']}") long attribute2
) {
return new ListItemReader(attribute1, attribute2) // <-- this is your ListItemReader
}
#Bean
public ItemProcessor<Path, Path> yourItemProcessor(){
return new YourItemProcessor<>();
}
#Bean
public ItemWriter<Path> yourItemWriter(){
return new YourItemWriter<>();
}
}
Before starting your job, you can add some jobParameters:
public JobParameters getJobParams() {
JobParametersBuilder params = new JobParametersBuilder();
params.addString("attribute1", "something_cool");
params.addLong("attribute2", 123456);
return params.toJobParameters();
}
and add it to your job:
Job yourJob = getYourJob();
jobParameters jobParams = getJobParams();
JobExecution execution = jobLauncher.run(job, jobParams);
References
The Delegate Pattern and Registering with the Step
How to get access to job parameters from ItemReader
Related
I have an old code base that I need to refactor using Java 8, so I have an interface, which tells whether my current site supports the platform.
public interface PlatformSupportHandler {
public abstract boolean isPaltformSupported(String platform);
}
and I have multiple classes implementing it and each class supports a different platform.
A few of the implementing classes are:
#Component("bsafePlatformSupportHandler")
public class BsafePlatoformSupportHandler implements PlatformSupportHandler {
String[] supportedPlatform = {"iPad", "Android", "iPhone"};
Set<String> supportedPlatformSet = new HashSet<>(Arrays.asList(supportedPlatform));
#Override
public boolean isPaltformSupported(String platform) {
return supportedPlatformSet.contains(platform);
}
}
Another implementation:
#Component("discountPlatformSupportHandler")
public class DiscountPlatoformSupportHandler implements PlatformSupportHandler{
String[] supportedPlatform = {"Android", "iPhone"};
Set<String> supportedPlatformSet = new HashSet<>(Arrays.asList(supportedPlatform));
#Override
public boolean isPaltformSupported(String platform) {
return supportedPlatformSet.contains(platform);
}
}
At runtime in my filter, I get the required bean which I want:
platformSupportHandler = (PlatformSupportHandler) ApplicationContextUtil
.getBean(subProductType + Constants.PLATFORM_SUPPORT_HANDLER_APPEND);
and call isPlatformSupported to get whether my current site supports the following platform or not.
I am new to Java 8, so is there any way I can refactor this code without creating multiple classes? As the interface only contains one method, can I somehow use lambda to refactor it?
If you want to stick to the current design, you could do something like this:
public class MyGeneralPurposeSupportHandler implements PlatformSupportHandler {
private final Set<String> supportedPlatforms;
public MyGeneralPurposeSupportHandler(Set<String> supportedPlatforms) {
this.supportedPlatforms = supportedPlatforms;
}
public boolean isPlatformSupported(String platform) {
return supportedPlatforms.contains(platform);
}
}
// now in configuration:
#Configuration
class MySpringConfig {
#Bean
#Qualifier("discountPlatformSupportHandler")
public PlatformSupportHandler discountPlatformSupportHandler() {
return new MyGeneralPurposeSupportHandler(new HashSefOf({"Android", "iPhone"})); // yeah its not a java syntax, but you get the idea
}
#Bean
#Qualifier("bsafePlatformSupportHandler")
public PlatformSupportHandler bsafePlatformSupportHandler() {
return new MyGeneralPurposeSupportHandler(new HashSefOf({"Android", "iPhone", "iPad"}));
}
}
This method has an advantage of not creating class per type (discount, bsafe, etc), so this answers the question.
Going step further, what happens if there no type that was requested, currently it will fail because the bean does not exist in the application context - not a really good approach.
So you could create a map of type to the set of supported platforms, maintain the map in the configuration or something an let spring do its magic.
You'll end up with something like this:
public class SupportHandler {
private final Map<String, Set<String>> platformTypeToSuportedPlatforms;
public SupportHandler(Map<String, Set<String>> map) {
this.platformTypeToSupportedPlatforms = map;
}
public boolean isPaltformSupported(String type) {
Set<String> supportedPlatforms = platformTypeToSupportedPlatforms.get(type);
if(supportedPlatforms == null) {
return false; // or maybe throw an exception, the point is that you don't deal with spring here which is good since spring shouldn't interfere with your business code
}
return supportedPlatforms.contains(type);
}
}
#Configuration
public class MyConfiguration {
// Configuration conf is supposed to be your own way to read configurations in the project - so you'll have to implement it somehow
#Bean
public SupportHandler supportHandler(Configuration conf) {
return new SupportHandler(conf.getRequiredMap());
}
}
Now if you follow this approach, adding a new supported types becomes codeless at all, you only add a configuration, by far its the best method I can offer.
Both methods however lack the java 8 features though ;)
You can use the following in your config class where you can create beans:
#Configuration
public class AppConfiguration {
#Bean(name = "discountPlatformSupportHandler")
public PlatformSupportHandler discountPlatformSupportHandler() {
String[] supportedPlatforms = {"Android", "iPhone"};
return getPlatformSupportHandler(supportedPlatforms);
}
#Bean(name = "bsafePlatformSupportHandler")
public PlatformSupportHandler bsafePlatformSupportHandler() {
String[] supportedPlatforms = {"iPad", "Android", "iPhone"};
return getPlatformSupportHandler(supportedPlatforms);
}
private PlatformSupportHandler getPlatformSupportHandler(String[] supportedPlatforms) {
return platform -> Arrays.asList(supportedPlatforms).contains(platform);
}
}
Also, when you want to use the bean, it is again very easy:
#Component
class PlatformSupport {
// map of bean name vs bean, automatically created by Spring for you
private final Map<String, PlatformSupportHandler> platformSupportHandlers;
#Autowired // Constructor injection
public PlatformSupport(Map<String, PlatformSupportHandler> platformSupportHandlers) {
this.platformSupportHandlers = platformSupportHandlers;
}
public void method1(String subProductType) {
PlatformSupportHandler platformSupportHandler = platformSupportHandlers.get(subProductType + Constants.PLATFORM_SUPPORT_HANDLER_APPEND);
}
}
As it was written in Mark Bramnik's answer you can move this to configuration.
Suppose that it would be in yaml in that way:
platforms:
bsafePlatformSupportHandler: ["iPad", "Android", "iPhone"]
discountPlatformSupportHandler: ["Android", "iPhone"]
Then you can create config class to read this:
#Configuration
#EnableConfigurationProperties
#ConfigurationProperties
public class Config {
private Map<String, List<String>> platforms = new HashMap<String, List<String>>();
// getters and setters
You can than create handler with checking code.
Or place it in your filter like below:
#Autowired
private Config config;
...
public boolean isPlatformSupported(String subProductType, String platform) {
String key = subProductType + Constants.PLATFORM_SUPPORT_HANDLER_APPEND;
return config.getPlatforms()
.getOrDefault(key, Collections.emptyList())
.contains(platform);
}
I am using java websockets in play framework 2.6 and having hard time figuring out Guice DI. Below I am injecting DbService (which perform some db operations) but I am getting DbService as null and throwing NPE at line
User user = dbService.findByName(inEvent.getUsername());
dbService is null. I am not sure why its not injecting DbService. Although this work when I don't use it via akka actors. I do bind DbService to its implementation.
Actor
public class TestActor extends AbstractActor {
#Inject
private DbService dbService;
private ActorRef out;
public static Props props(final ActorRef out) {
return Props.create(TestActor.class, out);
}
public TestActor(ActorRef out) {
this.out = out;
}
#Override
public Receive createReceive() {
return receiveBuilder()
.match(InEvent.class, inEvent -> {
System.out.println(inEvent.getUsername());
User user = dbService.findByName(inEvent.getUsername());
System.out.println(user.getFirstName());
out.tell("userName is ", self());
}
)
.build();
}
}
Controller
public class Application extends Controller {
public WebSocket TestWebSocket() {
return WebSocket.json(InEvent.class).acceptOrResult(request -> {
final Token token = verifyToken(request());
if (token == null) {
return CompletableFuture.completedFuture(F.Either.Left(forbidden()));
}
return CompletableFuture.completedFuture(
F.Either.Right(ActorFlow.actorRef(out -> TestActor.props(out), actorSystem, materializer)));
});
}
}
Your actor will be constructed based on the parameters used in Props.create().
In your example:
Props.create(Class<TestActor>, ActorRef);
Will match the constructor in the TestActor.class with one argument of type ActorRef, and that constructor will be calle with the arguments that you provide.
You can change your constructor signature and the props method like this:
...
public static Prop props(ActorRef out, DBServide db) {
return Props.create(TestActor.class, out, db);
}
...
public TestActor(ActorRef out, DBService db) {
this.out = out;
this.dbService = db;
}
...
With this, the constructor TestActor(ActorRef, DBService) will be used and you will get the instance of DBService used when you called the static method props.
I have a complex job flow where I have 3 separate jobs built into a JobStep, and then I call that JobStep from a Job. There will be four of these JobSteps that will run in parallel from the calling job.
I need to pass a string in to them as a parameter.
Somewhat simplified code:
My main looks like this:
public static void main(String[] args) {
SpringApplication.run(SomeApplication.class, args);
}
One of the JobSteps looks like
#Bean
public JobStep jobStep1(<snip>){
<snip for clarity>
JobStep jobStep = new JobStep() ;
jobStep.setJob(jobs.get(jobName)
.incrementer(new RunIdIncrementer()).listener(listener)
.start(Flow1)
.next(Flow2)
.next(Flow3)
.end().build());
jobStep.setJobRepository(jobRepository);
jobStep.setJobLauncher(jobLauncher);
return jobStep;
}
The top job that runs the rest looks like
#Bean
public Job parentJob(<snip>) {
Flow childJobFlow = new FlowBuilder<SimpleFlow>("childJob").start(job1).build();
Flow childJobFlow2 = new FlowBuilder<SimpleFlow>("childJob2").start(job2).build();
FlowBuilder<SimpleFlow> builder = new FlowBuilder<SimpleFlow>("jobFlow");
Flow jobFLow = builder.split(new SimpleAsyncTaskExecutor()).add(childJobFlow,childJobFlow2).build();
return jobs.get("parentJob")
.incrementer(new RunIdIncrementer()).listener(listener)
.start(jobFLow)
.end().build();
}
I need each JobStep to get a different string.
I was able to accomplish as Nghia Do suggested in his comment by using Partitioner. With partitioner I was able to push a string on to the context and then in a #Before Step retrieve it.
In my ItemReader I have:
#BeforeStep
public void beforeStep(StepExecution stepExecution) throws Exception {
this.stepExecution = stepExecution.getExecutionContext();
this.keyString = stepExecution.getString("keyString");
}
The Paritioner
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> partitionMap = new HashMap<String, ExecutionContext>();
List<String> codes = getCodes();
for (String code : codes)
{
ExecutionContext context = new ExecutionContext();
context.put("keyString", code);
partitionMap.put(code, context);
}
return partitionMap;
}
getCodes is just a placeholder function right now that returns a list of strings for testing. Eventually it will be replaces with something more useful.
private List<String> getCodes() {
ArrayList<String> result = new ArrayList<String>();
result.add("One");
result.add("Two");
result.add("Three");
result.add("Four");
result.add("Five");
result.add("Six");
result.add("Seven");
return result;
}
Then to get the steps I had make a master step that called my existing steps:
#Bean
public Step masterStep(#Value("#{proccessFilesStep}") Step readFilesStep) {
return stepBuilders.get("masterStep")
.partitioner(readFilesStep)
.partitioner("proccessFilesStep", partitioner())
.taskExecutor(taskExecutor())
.build();
}
And stepBuilders is:
#Autowired
private StepBuilderFactory stepBuilders;
Had to combine like 20 different examples on the net to get all the peices so I am putting them all in this answer for the next person that needs it.
I am trying to configure Spring Batch Steps for partitioning. The nice sample found here shows a partition about "id range", but I don't know where to start for a "data page" range.
In my sequential step, I have :
reader : a RepositoryItemReader using a PagingAndSortingRepository
processor : a data converter
writer : a RepositoryItemWriter using a CrudRepository
chunck : 5
listener : a StepListener
return stepBuilderFactory.get("stepApplicationForm")
.<OldApplicationForm, NewApplicationForm>chunk(5)
.reader(reader).processor(processor).writer(writer)
.listener(listener).build();
As I have understood, for partitionning, I have to create a partitioner, then I have a "parent" step that tells to use the partitioner with the child step, then the "child" step with a reader aware of the "pagination" parameters.
For the TaskExecutor, I think that the ThreadPoolTaskExecutor will fit.
What is the good way to implement/configure a paritioning based on data "pages" ? And what are the threading caveeats I should check ?
Thanks :)
Each partition has its own item reader and item writer instances. Your partition implementation will find min max values of a data load. Using your own logic you can create min and max values in the execution context. While querying the data base you can make use of these to handle specific slice of the data so that no concurrency issues takes place.
#Bean
public Step myMasterStep() {
return stepBuilderFactory.get("myMasterStep")
.partitioner("mySlaveWorker", myPartitioner())
.partitionHandler(myPartitionHandler()).build();
}
#Bean
public Step mySlaveWorker() {
return stepBuilderFactory
.get("mySlaveWorker")
.<OldApplicationForm, NewApplicationForm> chunk(5)
.faultTolerant()
.listener(MyStepListener())
.skip(DataAccessException.class)
.skip(FatalStepExecutionException.class)
.skip(Exception.class)
.skipLimit(75)
.noRollback(DataAccessException.class)
.noRollback(FatalStepExecutionException.class)
.noRollback(Exception.class)
.reader(myDataItemReader())
.writer(myDataItemWriter()).build();
}
#Bean
#StepScope
public MyDataItemReader myDataItemReader(
#Value("#{stepExecutionContext[minId]}") Long minId,
#Value("#{stepExecutionContext[maxId]}") Long maxId) {
MyDataItemReader myDataItemReader = new MyDataItemReader();
myDataItemReader.setPageSize(100);
myDataItemReader.setMinId(minId);
myDataItemReader.setMaxId(maxId);
return myDataItemReader;
}
#Bean
#StepScope
public MyDataItemWriter myDataItemWriter() {
return new MyDataItemWriter();
}
#Bean
#StepScope
public MyPartitioner myPartitioner() {
MyPartitioner myPartitioner = new MyPartitioner();
myPartitioner.setDataSource(dataSource);
return myPartitioner;
}
public class MyStepListener implements SkipListener<OldApplicationForm, NewApplicationForm> {
private static final Logger LOGGER = LoggerFactory.getLogger(MyStepListener.class);
public void onSkipInProcess(OldApplicationForm item, Throwable t) {
LOGGER.error("onSkipInProcess" + t.getMessage());
}
public void onSkipInRead(Throwable t) {
LOGGER.error("onSkipInRead " + t.getMessage());
}
public void onSkipInWrite(NewApplicationForm item, Throwable t) {
//logs
LOGGER.error("In MyStepListener --> onSkipInProcess" + t.getMessage());
}
}
I wonder how to pass an instance variable externally in Quartz?
Below is pseudo code I would like to write. How can I pass externalInstance into this Job?
public class SimpleJob implements Job {
#Override
public void execute(JobExecutionContext context)
throws JobExecutionException {
float avg = externalInstance.calculateAvg();
}
}
you can put your instance in the schedulerContext.When you are going to schedule the job ,just before that you can do below:
getScheduler().getContext().put("externalInstance", externalInstance);
Your job class would be like below:
public class SimpleJob implements Job {
#Override
public void execute(JobExecutionContext context)
throws JobExecutionException {
SchedulerContext schedulerContext = null;
try {
schedulerContext = context.getScheduler().getContext();
} catch (SchedulerException e1) {
e1.printStackTrace();
}
ExternalInstance externalInstance =
(ExternalInstance) schedulerContext.get("externalInstance");
float avg = externalInstance.calculateAvg();
}
}
If you are using Spring ,you can actually using spring's support to inject the whole applicationContext like answered in the Link
While scheduling the job using a trigger, you would have defined JobDataMap that is added to the JobDetail. That JobDetail object will be present in the JobExecutionContext passed to the execute() method in your Job. So, you should figure out a way to pass your externalInstance through the JobDataMap. HTH.
Add the object to the JobDataMap:
JobDetail job = JobBuilder.newJob(MyJobClass.class)
.withIdentity("MyIdentity",
"MyGroup")
.build();
job.getJobDataMap()
.put("MyObject",
myObject);
Access the data from the JobDataMap:
var myObject = (MyObjectClass) context.getJobDetail()
.getJobDataMap()
.get("carrier");
Solve this problem by creating one interface with one HashMap putting required information there.
Implement this interface in your Quartz Job class this information will be accessible.
In IFace
Map<JobKey,Object> map = new HashMap<>();
In Job
map.get(context.getJobDetail().getKey()) => will give you Object
Quartz has a simple way to grep params from JobDataMap using setters
I am using Quartz 2.3 and I simply used setter to fetch passed instance objects
For example I created this class
public class Data implements Serializable {
#JsonProperty("text")
private String text;
#JsonCreator
public Data(#JsonProperty("b") String text) {this.text = text;}
public String getText() {return text;}
}
Then I created an instance of this class and put it inside the JobDataMap
JobDataMap jobDataMap = new JobDataMap();
jobDataMap.put("data", new Data(1, "One!"));
JobDetail job = newJob(HelloJob.class)
.withIdentity("myJob", "group")
.withDescription("bla bla bla")
.usingJobData(jobDataMap) // <!!!
.build();
And my job class looks like this
public class HelloJob implements Job {
Data data;
public HelloJob() {}
public void execute(JobExecutionContext context)
throws JobExecutionException
{
String text = data.getText();
System.out.println(text);
}
public void setData(Data data) {this.data = data;}
}
Note: it is mandatory that the field and the setter matched the key
This code will print One! when you schedule the job.
That's it, clean and efficient
This is the responsibility of the JobFactory. The default PropertySettingJobFactory implementation will invoke any bean-setter methods, based on properties found in the schdeuler context, the trigger, and the job detail. So as long as you have implemnted an appropriate setContext() setter method you should be able to do any of the following:
scheduler.getContext().put("context", context);
Or
Trigger trigger = TriggerBuilder.newTrigger()
...
.usingJobData("context", context)
.build()
Or
JobDetail job = JobBuilder.newJob(SimpleJob.class)
...
.usingJobData("context", context)
.build()
Or if that isn't enough you can provide your own JobFactory class which instantiates the Job objects however you please.