I have a complex job flow where I have 3 separate jobs built into a JobStep, and then I call that JobStep from a Job. There will be four of these JobSteps that will run in parallel from the calling job.
I need to pass a string in to them as a parameter.
Somewhat simplified code:
My main looks like this:
public static void main(String[] args) {
SpringApplication.run(SomeApplication.class, args);
}
One of the JobSteps looks like
#Bean
public JobStep jobStep1(<snip>){
<snip for clarity>
JobStep jobStep = new JobStep() ;
jobStep.setJob(jobs.get(jobName)
.incrementer(new RunIdIncrementer()).listener(listener)
.start(Flow1)
.next(Flow2)
.next(Flow3)
.end().build());
jobStep.setJobRepository(jobRepository);
jobStep.setJobLauncher(jobLauncher);
return jobStep;
}
The top job that runs the rest looks like
#Bean
public Job parentJob(<snip>) {
Flow childJobFlow = new FlowBuilder<SimpleFlow>("childJob").start(job1).build();
Flow childJobFlow2 = new FlowBuilder<SimpleFlow>("childJob2").start(job2).build();
FlowBuilder<SimpleFlow> builder = new FlowBuilder<SimpleFlow>("jobFlow");
Flow jobFLow = builder.split(new SimpleAsyncTaskExecutor()).add(childJobFlow,childJobFlow2).build();
return jobs.get("parentJob")
.incrementer(new RunIdIncrementer()).listener(listener)
.start(jobFLow)
.end().build();
}
I need each JobStep to get a different string.
I was able to accomplish as Nghia Do suggested in his comment by using Partitioner. With partitioner I was able to push a string on to the context and then in a #Before Step retrieve it.
In my ItemReader I have:
#BeforeStep
public void beforeStep(StepExecution stepExecution) throws Exception {
this.stepExecution = stepExecution.getExecutionContext();
this.keyString = stepExecution.getString("keyString");
}
The Paritioner
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> partitionMap = new HashMap<String, ExecutionContext>();
List<String> codes = getCodes();
for (String code : codes)
{
ExecutionContext context = new ExecutionContext();
context.put("keyString", code);
partitionMap.put(code, context);
}
return partitionMap;
}
getCodes is just a placeholder function right now that returns a list of strings for testing. Eventually it will be replaces with something more useful.
private List<String> getCodes() {
ArrayList<String> result = new ArrayList<String>();
result.add("One");
result.add("Two");
result.add("Three");
result.add("Four");
result.add("Five");
result.add("Six");
result.add("Seven");
return result;
}
Then to get the steps I had make a master step that called my existing steps:
#Bean
public Step masterStep(#Value("#{proccessFilesStep}") Step readFilesStep) {
return stepBuilders.get("masterStep")
.partitioner(readFilesStep)
.partitioner("proccessFilesStep", partitioner())
.taskExecutor(taskExecutor())
.build();
}
And stepBuilders is:
#Autowired
private StepBuilderFactory stepBuilders;
Had to combine like 20 different examples on the net to get all the peices so I am putting them all in this answer for the next person that needs it.
Related
I've configured a route to extract some data from exchanges and aggregate them; here is simple summary:
#Component
#RequiredArgsConstructor
public class FingerprintHistoryRouteBuilder extends RouteBuilder {
private final FingerprintHistoryService fingerprintHistoryService;
#Override
public void configure() throws Exception {
from("seda:httpFingerprint")
.aggregate( (AggregationStrategy) (oldExchange, newExchange) -> {
final FingerprintHistory newFingerprint = extract(newExchange);
if (oldExchange == null) {
List<FingerprintHistory> fingerprintHistories = new ArrayList<>();
fingerprintHistories.add(newFingerprint);
newExchange.getMessage().setBody(fingerprintHistories);
return newExchange;
}
final Message oldMessage = oldExchange.getMessage();
final List<FingerprintHistory> fingerprintHistories = (List<FingerprintHistory>) oldMessage.getBody(List.class);
fingerprintHistories.add(newFingerprint);
return oldExchange;
})
.constant(true)
.completionSize(aggregateCount)
.completionInterval(aggregateDuration.toMillis())
.to("direct:processFingerprint")
.end();
from("direct:processFingerprint")
.process(exchange -> {
List<FingerprintHistory> fingerprintHistories = exchange.getMessage().getBody(List.class);
fingerprintHistoryService.saveAll(fingerprintHistories);
});
strong text
}
}
The problem is aggregation completion never works for example this is a sample of my test:
#SpringBootTest
class FingerprintHistoryRouteBuilderTest {
#Autowired
ProducerTemplate producerTemplate;
#Autowired
FingerprintHistoryRouteBuilder fingerprintHistoryRouteBuilder;
#Autowired
CamelContext camelContext;
#MockBean
FingerprintHistoryService historyService;
#Test
void api_whenAggregate() {
UserSearchActivity activity = ActivityFactory.buildSampleSearchActivity("127.0.0.1", "salam", "finger");
Exchange exchange = buildExchange();
exchange.getMessage().setBody(activity);
ReflelctionTestUtils.setField(fingerprintHistoryRouteBuilder, "aggregateCount", 1);
ReflectionTestUtils.setFiled(fingerprintHistoryRouteBuilder, "aggregateDuration", Duration.ofNanos(1));
producerTemplate.send(FingerprintHistoryRouteBuilder.FINGERPRINT_HISTORY_ENDPOINT, exchange);
Mockito.verify(historyService).saveAll(Mockito.any());
}
Exchange buildExchange() {
DefaultExchange defaultExchange = new DefaultExchange(camelContext);
defaultExchange.setMessage(new DefaultMessage(camelContext));
return defaultExchange;
}
}
with the following result:
Wanted but not invoked: fingerprintHistoryService bean.saveAll(
);
I build this simplified example, and the test passes, so it looks like your usage of aggregate is probably correct.
Have you considered that your Mockito.verify() call is happening before the exchange finishes routing? You could test this by removing the verify call and adding a .log() statement to the FINGERPRINT_PROCESS_AGGREGATION route. If you see the log output during execution, you know the exchange is being routed as you expect. If this is the case, then your verify() call needs to be able to wait for the exchange to finish routing. I don't use mockito much, but it looks like you can do this:
Mockito.verify(historyService, timeout(10000)).saveAll(Mockito.any());
I coded a custom implementation of ListItemReader, triying to follow the example in the spring batch's github. Anyway, in my case, I need read a variable from the jobContext, this variable is a path where I have to read the files that contains. I can't use the constructor because the constructors executes before the beforeStep event and I don't have these var at this moment.
Anyway this will work first execution, but if the list never goes again to null I can't execute again the initialize method.
If I tried add an else in the !list.isEmpty() condition that set my list to null. I enter in an infinite loop.
There are other methods to solve this? Maybe I am overcomplicating this.
public class ListItemReader<Path> implements ItemReader<Path>, StepExecutionListener {
private List<Path> list;
private org.springframework.batch.core.JobExecution jobExecution;
public ListItemReader() {
this.list = new ArrayList<>();
}
public void initialize(){
//Here I made an listdirectories of a path and add all the files to the list
String pathString = jobExecution.getExecutionContext().getString(Constants.CONTEXT_PATH);
Path path = Paths.get(pathString );
...
items.add(Paths.get(..path..));
}
#Nullable
#Override
public T read() {
if(list == null) initialize();
if (!list.isEmpty()) {
return list.remove(0);
}
return null;
}
#Override
public ExitStatus afterStep(StepExecution se) {
return ExitStatus.COMPLETED;
}
#Override
public void beforeStep(StepExecution se) {
jobExecution = se.getJobExecution();
}
}
I can't use the constructor because the constructors executes before the beforeStep event and I don't have these var at this moment.
Actually, you can delay your bean constructor by using #JobScope and #StepScope. Additionally, you can use #Value and Spring SpEL to inject your jobParameters.
In your case, you might want to rewrite your code, for e.g:
#Configuration
#RequiredArgsConstructor
public class YourJobConfig {
private final StepBuilderFactory stepBuilder;
#Bean("yourStep")
public Step doWhatever() {
return stepBuilder
.get("yourStepName")
.<Path, Path>chunk(50) // <-- replace your chunk size here
.reader(yourItemReader(null, 0)) // <-- place your default values here
.process(yourItemProcessor())
.writer(yourItemWriter())
}
#Bean
public ItemReader<Path> yourItemReader(
#Value("#{jobParameters['attribute1']}") String attribute1,
#Value("#{jobParameters['attribute2']}") long attribute2
) {
return new ListItemReader(attribute1, attribute2) // <-- this is your ListItemReader
}
#Bean
public ItemProcessor<Path, Path> yourItemProcessor(){
return new YourItemProcessor<>();
}
#Bean
public ItemWriter<Path> yourItemWriter(){
return new YourItemWriter<>();
}
}
Before starting your job, you can add some jobParameters:
public JobParameters getJobParams() {
JobParametersBuilder params = new JobParametersBuilder();
params.addString("attribute1", "something_cool");
params.addLong("attribute2", 123456);
return params.toJobParameters();
}
and add it to your job:
Job yourJob = getYourJob();
jobParameters jobParams = getJobParams();
JobExecution execution = jobLauncher.run(job, jobParams);
References
The Delegate Pattern and Registering with the Step
How to get access to job parameters from ItemReader
I am a newbie to Java unit testing and need help writing a unit test that just needs to handle 2 paths of an "if" statement where the method has no return value.
public class Consumer extends KafkaConsumer {
private final MessageService service;
public Consumer(MessageService service) {
this.service = service;
}
#Override
protected void processRecords(ConsumerRecords<Long, String> records) {
for (ConsumerRecord<Long, String> records) {
if (record.value().startsWith("{")) {
OldFormatMessageObject mo = new OldFormatMessageObject(record.value());
service.processMessage(mo);
} else if (record.value().startsWith("metadata")) {
NewFormatMessageObject mo = new NewFormatMessageObject(record.value());
service.processMessage(mo);
}
}
}
What to test?
Strictly speaking, there are actually 3 paths in branching logic to test:
if startsWith("{") then MessageService#processMessage(OldFormatMessageObject) is called.
if startsWith("metadata") then MessageService#processMessage(NewFormatMessageObject) is called.
otherwise nothing happens.
It would be nice if "iteration" logic is unit tested as well (for loop).
How to test?
We have to verify, that correct method of MessageService is called.
For this, we can use MessageService mock, which will be injected to Consumer via constructor.
Using mockito [1], it might look like this:
...
private static final String TEST_TOPIC = "test.topic";
#Mock
private MessageService messageServiceMock;
// Tested instance
private Consumer consumer;
#BeforeEach
public void beforeEach() {
consumer = new Consumer(messageServiceMock);
}
#Test
public void newMessageFormatIsHandled() {
// Given
final String expectedValue = "my-expected-value";
// When
consumer.processRecords(testSingleConsumerRecords(expectedValue));
// Then
verify(messageServiceMock, times(1)).processMessage(new NewFormatMessageObject(expectedValue));
// OR
verify(messageServiceMock, times(1)).processMessage(isA(NewFormatMessageObject.class));
}
...
private static ConsumerRecords<Long, String> testSingleConsumerRecords(String expectedValue) {
final Map<TopicPartition, List<ConsumerRecord<Long, String>>> records = new LinkedHashMap<>();
final ConsumerRecord<Long, String> record = new ConsumerRecord<>(TEST_TOPIC, 1, 0, 100L, expectedValue);
records.put(new TopicPartition(TEST_TOPIC, 0), Collections.singletonList(record));
return new ConsumerRecords<>(records);
}
Note, that if MessageService has interface, you don't even need any mock framework, and provide "mocked" instance to your unit test cases, but mock framework makes tests quite more readable.
[1] https://site.mockito.org
I have an old code base that I need to refactor using Java 8, so I have an interface, which tells whether my current site supports the platform.
public interface PlatformSupportHandler {
public abstract boolean isPaltformSupported(String platform);
}
and I have multiple classes implementing it and each class supports a different platform.
A few of the implementing classes are:
#Component("bsafePlatformSupportHandler")
public class BsafePlatoformSupportHandler implements PlatformSupportHandler {
String[] supportedPlatform = {"iPad", "Android", "iPhone"};
Set<String> supportedPlatformSet = new HashSet<>(Arrays.asList(supportedPlatform));
#Override
public boolean isPaltformSupported(String platform) {
return supportedPlatformSet.contains(platform);
}
}
Another implementation:
#Component("discountPlatformSupportHandler")
public class DiscountPlatoformSupportHandler implements PlatformSupportHandler{
String[] supportedPlatform = {"Android", "iPhone"};
Set<String> supportedPlatformSet = new HashSet<>(Arrays.asList(supportedPlatform));
#Override
public boolean isPaltformSupported(String platform) {
return supportedPlatformSet.contains(platform);
}
}
At runtime in my filter, I get the required bean which I want:
platformSupportHandler = (PlatformSupportHandler) ApplicationContextUtil
.getBean(subProductType + Constants.PLATFORM_SUPPORT_HANDLER_APPEND);
and call isPlatformSupported to get whether my current site supports the following platform or not.
I am new to Java 8, so is there any way I can refactor this code without creating multiple classes? As the interface only contains one method, can I somehow use lambda to refactor it?
If you want to stick to the current design, you could do something like this:
public class MyGeneralPurposeSupportHandler implements PlatformSupportHandler {
private final Set<String> supportedPlatforms;
public MyGeneralPurposeSupportHandler(Set<String> supportedPlatforms) {
this.supportedPlatforms = supportedPlatforms;
}
public boolean isPlatformSupported(String platform) {
return supportedPlatforms.contains(platform);
}
}
// now in configuration:
#Configuration
class MySpringConfig {
#Bean
#Qualifier("discountPlatformSupportHandler")
public PlatformSupportHandler discountPlatformSupportHandler() {
return new MyGeneralPurposeSupportHandler(new HashSefOf({"Android", "iPhone"})); // yeah its not a java syntax, but you get the idea
}
#Bean
#Qualifier("bsafePlatformSupportHandler")
public PlatformSupportHandler bsafePlatformSupportHandler() {
return new MyGeneralPurposeSupportHandler(new HashSefOf({"Android", "iPhone", "iPad"}));
}
}
This method has an advantage of not creating class per type (discount, bsafe, etc), so this answers the question.
Going step further, what happens if there no type that was requested, currently it will fail because the bean does not exist in the application context - not a really good approach.
So you could create a map of type to the set of supported platforms, maintain the map in the configuration or something an let spring do its magic.
You'll end up with something like this:
public class SupportHandler {
private final Map<String, Set<String>> platformTypeToSuportedPlatforms;
public SupportHandler(Map<String, Set<String>> map) {
this.platformTypeToSupportedPlatforms = map;
}
public boolean isPaltformSupported(String type) {
Set<String> supportedPlatforms = platformTypeToSupportedPlatforms.get(type);
if(supportedPlatforms == null) {
return false; // or maybe throw an exception, the point is that you don't deal with spring here which is good since spring shouldn't interfere with your business code
}
return supportedPlatforms.contains(type);
}
}
#Configuration
public class MyConfiguration {
// Configuration conf is supposed to be your own way to read configurations in the project - so you'll have to implement it somehow
#Bean
public SupportHandler supportHandler(Configuration conf) {
return new SupportHandler(conf.getRequiredMap());
}
}
Now if you follow this approach, adding a new supported types becomes codeless at all, you only add a configuration, by far its the best method I can offer.
Both methods however lack the java 8 features though ;)
You can use the following in your config class where you can create beans:
#Configuration
public class AppConfiguration {
#Bean(name = "discountPlatformSupportHandler")
public PlatformSupportHandler discountPlatformSupportHandler() {
String[] supportedPlatforms = {"Android", "iPhone"};
return getPlatformSupportHandler(supportedPlatforms);
}
#Bean(name = "bsafePlatformSupportHandler")
public PlatformSupportHandler bsafePlatformSupportHandler() {
String[] supportedPlatforms = {"iPad", "Android", "iPhone"};
return getPlatformSupportHandler(supportedPlatforms);
}
private PlatformSupportHandler getPlatformSupportHandler(String[] supportedPlatforms) {
return platform -> Arrays.asList(supportedPlatforms).contains(platform);
}
}
Also, when you want to use the bean, it is again very easy:
#Component
class PlatformSupport {
// map of bean name vs bean, automatically created by Spring for you
private final Map<String, PlatformSupportHandler> platformSupportHandlers;
#Autowired // Constructor injection
public PlatformSupport(Map<String, PlatformSupportHandler> platformSupportHandlers) {
this.platformSupportHandlers = platformSupportHandlers;
}
public void method1(String subProductType) {
PlatformSupportHandler platformSupportHandler = platformSupportHandlers.get(subProductType + Constants.PLATFORM_SUPPORT_HANDLER_APPEND);
}
}
As it was written in Mark Bramnik's answer you can move this to configuration.
Suppose that it would be in yaml in that way:
platforms:
bsafePlatformSupportHandler: ["iPad", "Android", "iPhone"]
discountPlatformSupportHandler: ["Android", "iPhone"]
Then you can create config class to read this:
#Configuration
#EnableConfigurationProperties
#ConfigurationProperties
public class Config {
private Map<String, List<String>> platforms = new HashMap<String, List<String>>();
// getters and setters
You can than create handler with checking code.
Or place it in your filter like below:
#Autowired
private Config config;
...
public boolean isPlatformSupported(String subProductType, String platform) {
String key = subProductType + Constants.PLATFORM_SUPPORT_HANDLER_APPEND;
return config.getPlatforms()
.getOrDefault(key, Collections.emptyList())
.contains(platform);
}
I wonder how to pass an instance variable externally in Quartz?
Below is pseudo code I would like to write. How can I pass externalInstance into this Job?
public class SimpleJob implements Job {
#Override
public void execute(JobExecutionContext context)
throws JobExecutionException {
float avg = externalInstance.calculateAvg();
}
}
you can put your instance in the schedulerContext.When you are going to schedule the job ,just before that you can do below:
getScheduler().getContext().put("externalInstance", externalInstance);
Your job class would be like below:
public class SimpleJob implements Job {
#Override
public void execute(JobExecutionContext context)
throws JobExecutionException {
SchedulerContext schedulerContext = null;
try {
schedulerContext = context.getScheduler().getContext();
} catch (SchedulerException e1) {
e1.printStackTrace();
}
ExternalInstance externalInstance =
(ExternalInstance) schedulerContext.get("externalInstance");
float avg = externalInstance.calculateAvg();
}
}
If you are using Spring ,you can actually using spring's support to inject the whole applicationContext like answered in the Link
While scheduling the job using a trigger, you would have defined JobDataMap that is added to the JobDetail. That JobDetail object will be present in the JobExecutionContext passed to the execute() method in your Job. So, you should figure out a way to pass your externalInstance through the JobDataMap. HTH.
Add the object to the JobDataMap:
JobDetail job = JobBuilder.newJob(MyJobClass.class)
.withIdentity("MyIdentity",
"MyGroup")
.build();
job.getJobDataMap()
.put("MyObject",
myObject);
Access the data from the JobDataMap:
var myObject = (MyObjectClass) context.getJobDetail()
.getJobDataMap()
.get("carrier");
Solve this problem by creating one interface with one HashMap putting required information there.
Implement this interface in your Quartz Job class this information will be accessible.
In IFace
Map<JobKey,Object> map = new HashMap<>();
In Job
map.get(context.getJobDetail().getKey()) => will give you Object
Quartz has a simple way to grep params from JobDataMap using setters
I am using Quartz 2.3 and I simply used setter to fetch passed instance objects
For example I created this class
public class Data implements Serializable {
#JsonProperty("text")
private String text;
#JsonCreator
public Data(#JsonProperty("b") String text) {this.text = text;}
public String getText() {return text;}
}
Then I created an instance of this class and put it inside the JobDataMap
JobDataMap jobDataMap = new JobDataMap();
jobDataMap.put("data", new Data(1, "One!"));
JobDetail job = newJob(HelloJob.class)
.withIdentity("myJob", "group")
.withDescription("bla bla bla")
.usingJobData(jobDataMap) // <!!!
.build();
And my job class looks like this
public class HelloJob implements Job {
Data data;
public HelloJob() {}
public void execute(JobExecutionContext context)
throws JobExecutionException
{
String text = data.getText();
System.out.println(text);
}
public void setData(Data data) {this.data = data;}
}
Note: it is mandatory that the field and the setter matched the key
This code will print One! when you schedule the job.
That's it, clean and efficient
This is the responsibility of the JobFactory. The default PropertySettingJobFactory implementation will invoke any bean-setter methods, based on properties found in the schdeuler context, the trigger, and the job detail. So as long as you have implemnted an appropriate setContext() setter method you should be able to do any of the following:
scheduler.getContext().put("context", context);
Or
Trigger trigger = TriggerBuilder.newTrigger()
...
.usingJobData("context", context)
.build()
Or
JobDetail job = JobBuilder.newJob(SimpleJob.class)
...
.usingJobData("context", context)
.build()
Or if that isn't enough you can provide your own JobFactory class which instantiates the Job objects however you please.