Develop Reference

Spring Batch with multi - step Spring Cloud Task (PartitionHandler) for Remote Partition

Latest Update (with an image to hope simplify the problem) (thanks for feedback from #Mahmoud)
Relate issue reports for other reference (after this original post created, it seem someone filed issues for Spring Cloud on similar issue, so also update there too):
https://github.com/spring-cloud/spring-cloud-task/issues/793 relate to approach #1
https://github.com/spring-cloud/spring-cloud-task/issues/792 relate to approach #2
Also find a workaround resolution for that issue and update on that github issue, will update this once it is confirmed good by developer
https://github.com/spring-cloud/spring-cloud-task/issues/793#issuecomment-894617929
I am developing an application involved multi-steps using spring batch job but hit some roadblock. Did try to research doc and different attempts, but no success. So thought to check if community can shed light
Spring batch job 1 (received job parameter for setting for step 1/setting for step 2)
Step 1 -> remote partition (partitionhandler (cpu/memory for step 1 + grid) + partitioner) with setting from step1 (job configuration or step configuration)
Step 2 -> remote partition (partitionhandler (cpu/memory for step 2 + grid) + partitioner) with setting from step2 (job configuration or step configuration, and diff from step 1)
The reason we want is to have different step with different k8s setting (like cpu/memory/grid)
Attempts:
Create two partition handler (partitionHandlerReader + partitionHandlerProcessor) and their corresponding launcher (LauncherReader + LauncherProcessor)
Complete Project can be found in
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_1_two_partitionhandlers
The main class of configuration is try to simplify into one class
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/attempt_1_two_partitionhandlers/src/main/java/com/example/batchprocessing/BatchConfiguration.java
Use one PartitionerHandler + one TaskLauncher but with #StepScope for late binding for dynamic change base on step and job setup
Complete Project can be found in
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope
The main class of configuration is try to simplify into one class
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/attempt_2_partitionhandler_with_stepscope/src/main/java/com/example/batchprocessing/BatchConfiguration.java
Both Result Following (full trace at above git repo):
During job trigger, it will error (it seem pass initial start up, but error during execution)
Because below will only occur when there are multiple PartitionHandler or when that Bean is at #StepScope or #JobScope
java.lang.NullPointerException: null
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:347) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
Full Log
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.4.6)
2021-08-06 11:24:29.242 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : Starting BatchProcessingApplication v0.0.1-SNAPSHOT using Java 11.0.7 on localhost.localdomain with PID 90294 (/home/danilko/IdeaProjects/partition/target/batchprocessing-0.0.1-SNAPSHOT.jar started by danilko in /home/danilko/IdeaProjects/partition)
2021-08-06 11:24:29.244 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : The following profiles are active: controller
2021-08-06 11:24:29.790 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2021-08-06 11:24:29.794 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2021-08-06 11:24:29.797 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2021-08-06 11:24:29.833 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.947 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.947 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.959 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$83e6c2be] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.968 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration' of type [org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration$$EnhancerBySpringCGLIB$$cc3cccc1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:30.093 INFO 90294 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-08-06 11:24:30.160 INFO 90294 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-08-06 11:24:30.724 INFO 90294 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: MYSQL
2021-08-06 11:24:30.736 INFO 90294 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'application.errorChannel' has 1 subscriber(s).
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean '_org.springframework.integration.errorLogger'
2021-08-06 11:24:30.974 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : Started BatchProcessingApplication in 2.024 seconds (JVM running for 2.366)
2021-08-06 11:24:30.975 INFO 90294 --- [ main] o.s.b.a.b.JobLauncherApplicationRunner : Running default command line with: []
2021-08-06 11:24:31.010 INFO 90294 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=partitionedJob-1538890488]] launched with the following parameters: [{}]
Set readerGridSize == 1
2021-08-06 11:24:31.020 INFO 90294 --- [ main] o.s.c.t.b.l.TaskBatchExecutionListener : The job execution id 22 was run within the task execution 54
2021-08-06 11:24:31.046 INFO 90294 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [partitionReaderStep]
2021-08-06 11:24:31.101 ERROR 90294 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step partitionReaderStep in job partitionedJob-1538890488
java.lang.NullPointerException: null
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:347) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.7.jar!/:5.3.7]
at com.sun.proxy.$Proxy65.handle(Unknown Source) ~[na:na]
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:106) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-5.3.7.jar!/:5.3.7]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.7.jar!/:5.3.7]
at com.sun.proxy.$Proxy51.run(Unknown Source) ~[na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:799) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:789) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:346) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) ~[spring-boot-2.4.6.jar!/:2.4.6]
at com.example.batchprocessing.BatchProcessingApplication.main(BatchProcessingApplication.java:10) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Study/Reference:
Most tutorial I found online only involved one partition step.
https://dataflow.spring.io/docs/feature-guides/batch/partitioning/
Thanks for info/helps in advance

Is above even possible setup?
yes, nothing prevents you from having two partitioned steps in a single Spring Batch job.
Is it possible to use JobScope/StepScope to pass info to the partitionhandler
yes, it is possible for the partition handler to be declared as a job/step scoped bean if it needs the late-binding feature to be configured.
Updated on 08/14/2021 by #DanilKo
The original answer is correct in high - level. However, to actually achieve the partition handeler to be step scoped, a code modification is required
Below is the analyze + my proposed workaround/fix (maybe eventually code maintainer will have better way to make it work, but so far below fix is working for me)
Issue being continued to discuss at:
https://github.com/spring-cloud/spring-cloud-task/issues/793 (multiple partitioner handler discussion)
https://github.com/spring-cloud/spring-cloud-task/issues/792
(which this fix is based up to use partitionerhandler at step scope to configure different worker steps + resources + max worker)
Root cause analyze (hypothesis)
The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task) or there are two partitionHandler, that setup is no longer triggered, as #EnableTask seem not able to locate one partitionhandler during creation.
https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-batch/src/main/java/org/springframework/cloud/task/batch/partition/DeployerPartitionHandler.java # 269
Resulted created DeployerHandler faced a null with taskExecution when trying to launch (as it is never setup)
https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-batch/src/main/java/org/springframework/cloud/task/batch/partition/DeployerPartitionHandler.java # 347
Workaround Resolution
Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
It seem to work, but still not clear if there is other side effect (so far during test not found any)
Full code can be found in https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope_workaround_resolution
In the partitionHandler method
#Bean
#StepScope
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
JobExplorer jobExplorer,
#Value("#{stepExecution}") StepExecution stepExecution) throws Exception {
...
// After the declaration of partitionhandler
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerStep")
, taskRepository);
// Issue https://github.com/spring-cloud/spring-cloud-task/issues/793
// Perform the setting of execution as this partitioner now not created at task level so #beforetask is no longer vaild
// The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
// But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task), that setup is no longer triggered
// Resulted created DeployerHandler faced a null
// Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
// From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
// It seem to work, but still not clear if there is other side effect (so far during test not found any)
long executionId = taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId());
System.out.println("Current execution job to task execution id " + executionId);
TaskExecution taskExecution = taskExplorer.getTaskExecution(taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId()));
System.out.println("Current execution job to task execution is not null: " + (taskExecution != null));
partitionHandler.beforeTask(taskExecution);
...
// rest of code continue
(note it utilize stepExecution context to find out the current trigger step name and therefore assign different worker step)
Worker name in this case is coming from pre-defined job execution, but may able to come from jobparameter or another place too)
That job context is populated with job listner
Job is configured with job listener
#Bean(name = "partitionedJob")
#Profile("!worker")
public Job partitionedJob()throws Exception {
Random random = new Random();
return jobBuilderFactory.get("partitionedJob" + random.nextInt())
.start(partitionReaderStep())
.listener(jobExecutionListener())
.next(partitionProcessorStep())
.build();
}
In job listener populated it
#Bean
public JobExecutionListener jobExecutionListener() {
JobExecutionListener listener = new JobExecutionListener(){
#Override
public void beforeJob(JobExecution jobExecution)
{
jobExecution.getExecutionContext().putString("readerCPURequest", "1");
jobExecution.getExecutionContext().putString("readerCPULimit", "2");
jobExecution.getExecutionContext().putString("readerWorkerGridSize", "1");
// For now using same image for reader/processor, but if it work, can split them
jobExecution.getExecutionContext().putString("readerWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("readerWorkerStep", "workerStepReader");
jobExecution.getExecutionContext().putString("processorCPURequest", "3");
jobExecution.getExecutionContext().putString("processorCPULimit", "4");
jobExecution.getExecutionContext().putString("processorWorkerGridSize", "2");
// For now using same image for reader/processor, but if it work, will split them
jobExecution.getExecutionContext().putString("processorWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("processorWorkerStep", "workerStepProcessor");
System.out.println("Set readerGridSize == " + jobExecution.getExecutionContext().getString("readerGridSize", "IT IS NULL WHICH IS INCORRECT"));
}
#Override
public void afterJob(JobExecution jobExecution) {
}
};
return listener;
}
Full code (can also be found in my code github after the workaround fix is being applied): https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/main/src/main/java/com/example/batchprocessing/BatchConfiguration.java
package com.example.batchprocessing;
import io.fabric8.kubernetes.api.model.DeletionPropagation;
import io.fabric8.kubernetes.api.model.batch.JobList;
import io.fabric8.kubernetes.api.model.batch.JobSpec;
import io.fabric8.kubernetes.api.model.batch.JobStatus;
import io.fabric8.kubernetes.client.KubernetesClient;
import org.springframework.batch.core.*;
import org.springframework.batch.core.configuration.JobRegistry;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.explore.JobExplorer;
import org.springframework.batch.core.partition.PartitionHandler;
import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.cloud.deployer.resource.docker.DockerResource;
import org.springframework.cloud.deployer.resource.support.DelegatingResourceLoader;
import org.springframework.cloud.deployer.spi.kubernetes.*;
import org.springframework.cloud.deployer.spi.task.TaskLauncher;
import org.springframework.cloud.task.batch.partition.*;
import org.springframework.cloud.task.configuration.EnableTask;
import org.springframework.cloud.task.repository.TaskExecution;
import org.springframework.cloud.task.repository.TaskExplorer;
import org.springframework.cloud.task.repository.TaskRepository;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.env.Environment;
import org.springframework.core.env.SystemEnvironmentPropertySource;
import org.springframework.core.io.Resource;
import org.springframework.core.task.TaskExecutor;
import org.springframework.core.task.TaskRejectedException;
import org.springframework.util.StringUtils;
import java.util.*;
#Configuration
#EnableBatchProcessing
#EnableTask
public class BatchConfiguration {
private static int BACK_OFF_LIMIT = 6;
// Set the kuberentes job name
private String taskName_prefix="partitionedbatchjob";
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public JobExplorer jobExplorer;
#Autowired
public JobRepository jobRepository;
#Autowired
public TaskExecutor taskExecutor;
#Autowired
public TaskRepository taskRepository;
#Autowired
public TaskExplorer taskExplorer;
#Autowired
private ConfigurableApplicationContext context;
#Autowired
private DelegatingResourceLoader resourceLoader;
#Autowired
private Environment environment;
#Bean
#StepScope
public Partitioner partitioner( #Value("#{stepExecution}") StepExecution stepExecution) {
return new Partitioner() {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> partitions = new HashMap<>(gridSize);
int targetGridSize = 0;
String step = "";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep"))
{
step = "reader";
}
else
{
step = "processor";
}
targetGridSize = Integer.parseInt(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerGridSize"));
for (int i = 0; i < targetGridSize; i++) {
ExecutionContext context1 = new ExecutionContext();
context1.put("partitionNumber", i);
partitions.put("partition" + i, context1);
}
return partitions;
}
};
}
#Bean
public KubernetesClient kuberentesClient()
{
KubernetesDeployerProperties kubernetesDeployerProperties = new KubernetesDeployerProperties();
return KubernetesClientFactory.getKubernetesClient(kubernetesDeployerProperties);
}
#Bean
#StepScope
public TaskLauncher taskLauncher( #Value("#{stepExecution}") StepExecution stepExecution)
{
KubernetesDeployerProperties kubernetesDeployerProperties = new KubernetesDeployerProperties();
kubernetesDeployerProperties.setNamespace("default");
kubernetesDeployerProperties.setCreateJob(true);
// Database setup to reference configmap for database info
List<KubernetesDeployerProperties.ConfigMapKeyRef> configMapKeyRefList = new ArrayList<KubernetesDeployerProperties.ConfigMapKeyRef>();
KubernetesDeployerProperties.ConfigMapKeyRef configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_URL");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_URL");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_USERNAME");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_USERNAME");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_PASSWORD");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_PASSWORD");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_DRIVERCLASSNAME");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_DRIVERCLASSNAME");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_PROFILES_ACTIVE");
configMapKeyRef.setEnvVarName("SPRING_PROFILES_ACTIVE");
configMapKeyRefList.add(configMapKeyRef);
kubernetesDeployerProperties.setConfigMapKeyRefs(configMapKeyRefList);
// Set request resource
KubernetesDeployerProperties.RequestsResources request = new KubernetesDeployerProperties.RequestsResources();
KubernetesDeployerProperties.LimitsResources limit = new KubernetesDeployerProperties.LimitsResources();
String step = "";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep"))
{
step="reader";
}
else
{
step="processor";
}
request.setCpu(stepExecution.getJobExecution().getExecutionContext().getString(step + "CPURequest"));
request.setMemory("2000Mi");
limit.setCpu(stepExecution.getJobExecution().getExecutionContext().getString(step +"CPULimit"));
limit.setMemory("3000Mi");
kubernetesDeployerProperties.setRequests(request);
kubernetesDeployerProperties.setLimits(limit);
// as build on local image, so need to use local
kubernetesDeployerProperties.setImagePullPolicy(ImagePullPolicy.IfNotPresent);
// Set task launcher properties to not repeat and not restart
KubernetesTaskLauncherProperties kubernetesTaskLauncherProperties = new KubernetesTaskLauncherProperties();
// https://kubernetes.io/docs/concepts/workloads/controllers/job/
// Set to never to create new pod on restart
kubernetesTaskLauncherProperties.setBackoffLimit(BACK_OFF_LIMIT);
kubernetesTaskLauncherProperties.setRestartPolicy(RestartPolicy.Never);
KubernetesTaskLauncher kubernetesTaskLauncher = new KubernetesTaskLauncher(kubernetesDeployerProperties,
kubernetesTaskLauncherProperties, kuberentesClient());
return kubernetesTaskLauncher;
}
#Bean(name = "partitionedJob")
#Profile("!worker")
public Job partitionedJob()throws Exception {
Random random = new Random();
return jobBuilderFactory.get("partitionedJob" + random.nextInt())
.start(partitionReaderStep())
.listener(jobExecutionListener())
.next(partitionProcessorStep())
.build();
}
#Bean(name = "partitionReaderStep")
public Step partitionReaderStep() throws Exception {
return stepBuilderFactory.get("partitionReaderStep")
.partitioner(workerStepReader().getName(), partitioner( null))
.step(workerStepReader())
.partitionHandler(partitionHandler(
taskLauncher( null),
jobExplorer, null))
.build();
}
#Bean(name = "partitionProcessorStep")
public Step partitionProcessorStep() throws Exception {
return stepBuilderFactory.get("partitionProcessorStep")
.partitioner(workerStepProcessor().getName(), partitioner( null))
.step(workerStepProcessor())
.partitionHandler(partitionHandler(
taskLauncher( null),
jobExplorer, null))
.build();
}
#Bean
#StepScope
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
JobExplorer jobExplorer,
#Value("#{stepExecution}") StepExecution stepExecution) throws Exception {
String step ="processor";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep")) {
step = "reader";
}
// Use local build image
DockerResource resource = new DockerResource(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerImage"));
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerStep")
, taskRepository);
// Issue https://github.com/spring-cloud/spring-cloud-task/issues/793
// Perform the setting of execution as this partitioner now not created at task level so #beforetask is no longer vaild
// The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
// But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task), that setup is no longer triggered
// Resulted created DeployerHandler faced a null
// Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
// From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
// It seem to work, but still not clear if there is other side effect (so far during test not found any)
long executionId = taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId());
System.out.println("Current execution job to task execution id " + executionId);
TaskExecution taskExecution = taskExplorer.getTaskExecution(taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId()));
System.out.println("Current execution job to task execution is not null: " + (taskExecution != null));
partitionHandler.beforeTask(taskExecution);
List<String> commandLineArgs = new ArrayList<>(3);
commandLineArgs.add("--spring.profiles.active=worker");
commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
commandLineArgs.add("--spring.batch.initializer.enabled=false");
partitionHandler
.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
partitionHandler.setEnvironmentVariablesProvider(new NoOpEnvironmentVariablesProvider());
partitionHandler.setMaxWorkers(Integer.parseInt(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerGridSize")));
partitionHandler.setApplicationName(taskName_prefix + step);
return partitionHandler;
}
#Bean
public JobExecutionListener jobExecutionListener() {
JobExecutionListener listener = new JobExecutionListener(){
#Override
public void beforeJob(JobExecution jobExecution)
{
jobExecution.getExecutionContext().putString("readerCPURequest", "1");
jobExecution.getExecutionContext().putString("readerCPULimit", "2");
jobExecution.getExecutionContext().putString("readerWorkerGridSize", "1");
// For now using same image for reader/processor, but if it work, can split them
jobExecution.getExecutionContext().putString("readerWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("readerWorkerStep", "workerStepReader");
jobExecution.getExecutionContext().putString("processorCPURequest", "3");
jobExecution.getExecutionContext().putString("processorCPULimit", "4");
jobExecution.getExecutionContext().putString("processorWorkerGridSize", "2");
// For now using same image for reader/processor, but if it work, will split them
jobExecution.getExecutionContext().putString("processorWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("processorWorkerStep", "workerStepProcessor");
System.out.println("Set readerGridSize == " + jobExecution.getExecutionContext().getString("readerGridSize", "IT IS NULL WHICH IS INCORRECT"));
}
#Override
public void afterJob(JobExecution jobExecution) {
}
};
return listener;
}
#Bean
#Profile("worker")
public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) {
return new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository);
}
#Bean(name = "workerStepReader")
public Step workerStepReader() {
return this.stepBuilderFactory.get("workerStepReader")
.tasklet(workerTaskletReader(null))
.build();
}
#Bean(name = "workerStepProcessor")
public Step workerStepProcessor() {
return this.stepBuilderFactory.get("workerStepProcessor")
.tasklet(workerTaskletProcessor(null))
.build();
}
#Bean
#StepScope
public Tasklet workerTaskletReader(
final #Value("#{stepExecution}") StepExecution stepExecution) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
Integer partitionNumber = stepExecution.getExecutionContext().getInt("partitionNumber");
System.out.println("This workerTaskletReader ran partition: " + partitionNumber);
return RepeatStatus.FINISHED;
}
};
}
#Bean
#StepScope
public Tasklet workerTaskletProcessor(
final #Value("#{stepExecution}") StepExecution stepExecution) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
Integer partitionNumber = stepExecution.getExecutionContext().getInt("partitionNumber");
System.out.println("This workerTaskletProcessor ran partition: " + partitionNumber);
return RepeatStatus.FINISHED;
}
};
}
}

Consider defining a bean of type 'com.example.amazonsync.Service.IAmazonUtilService' in your configuration

I am getting null pointer exception in my service class. I have autowired myservice class named IAmazonUtilService. But it am facing null pointervexception.
Also i have written PropertiesUtil to read data from application.properties which is also not working. Please help me on this.
2020-03-27 18:39:20.172 INFO 17536 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
java.lang.NullPointerException
at com.example.amazonsync.AmazonSync.ImportAmazonDataService.getOrdersFromAmazonStore(ImportAmazonDataService.java:36)
ImportAmazonDataService.java:36
at com.example.amazonsync.SyncData.OrderSync.execute(OrderSync.java:17)
OrderSync.java:17
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
JobRunShell.java:202
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
SimpleThreadPool.java:573
2020-03-27 18:39:20.254 INFO 17536 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2020-03-27 18:39:20.263 INFO 17536 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-03-27 18:39:20.330 INFO 17536 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-03-27 18:39:20.841 ERROR 17536 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Field IAmazonUtilService in com.example.amazonsync.AmazonSync.ImportAmazonDataService required a bean of type 'com.example.amazonsync.Service.IAmazonUtilService' that could not be found.
ImportAmazonDataService.java
public class ImportAmazonDataService {
#Autowired
private IAmazonUtilService IAmazonUtilService;
public ArrayList<String> getChannelLoc() {
ArrayList<String> channelLoc = new ArrayList<String>();
channelLoc.add("US");
channelLoc.add("CA");
channelLoc.add("MX");
return channelLoc;
}
public void getProductsFromAmazonStore(JobExecutionContext context) throws SQLException, ClassNotFoundException {
try {
final Long taskID = (Long) context.getJobDetail().getJobDataMap().get("taskId");
IAmazonUtilService.getChannelConfig("US");
} catch (Exception e) {
e.printStackTrace();
}
}
public void getOrdersFromAmazonStore(JobExecutionContext context) throws SQLException, ClassNotFoundException {
final long taskID = (long) context.getJobDetail().getJobDataMap().get("taskId");
IAmazonUtilService.getChannelConfig("CA");
}
}
Git link : https://github.com/Ezhilarasu1330/SpringBootQuartzSchedular.git

Change extends by implements !!
from:
public class AmazonUtilService extends ImportAmazonDataService {}
to:
public class AmazonUtilService implements ImportAmazonDataService {}

your ImportAmazonDataService class should be annotated as #Service in order spring will know to inject the #Autowired IAmazonUtilService
also refactor
#Autowired
private IAmazonUtilService IAmazonUtilService;
to
#Autowired
private IAmazonUtilService iAmazonUtilService;

Can't run SpringBoot Application

I am new to Spring and i have to make an "HelloWorld" Application. After several attempts, i can't solve my problem.
package com.example.demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class CiaoMondoApplication {
public static void main(String[] args) {
SpringApplication.run(CiaoMondoApplication.class, args);
}
} // that's the Application
package com.example.demo;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;
#RestController
public class CiaoMondoController {
#RequestMapping("/")
public String index() {
return "HelloWorld";
}
} // and that's the controller
Below the output, when i try to run the Application with Eclipse:
2019-06-11 11:09:33.089 INFO 9572 --- [ main] com.example.demo.CiaoMondoApplication : Starting CiaoMondoApplication on Asus-Mattia with PID 9572 (C:\Users\matti\eclipse-workspace\ciao-mondo\target\classes started by matti in C:\Users\matti\eclipse-workspace\ciao-mondo)
2019-06-11 11:09:33.091 INFO 9572 --- [ main] com.example.demo.CiaoMondoApplication : No active profile set, falling back to default profiles: default
2019-06-11 11:09:33.627 INFO 9572 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
2019-06-11 11:09:33.644 INFO 9572 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 11ms. Found 0 repository interfaces.
2019-06-11 11:09:33.895 INFO 9572 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$bb774f16] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-06-11 11:09:34.107 INFO 9572 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2019-06-11 11:09:34.127 INFO 9572 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2019-06-11 11:09:34.127 INFO 9572 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.19]
2019-06-11 11:09:34.219 INFO 9572 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-06-11 11:09:34.219 INFO 9572 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1094 ms
2019-06-11 11:09:34.271 WARN 9572 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception; nested exception is org.springframework.boot.autoconfigure.jdbc.DataSourceProperties$DataSourceBeanCreationException: Failed to determine a suitable driver class
2019-06-11 11:09:34.274 INFO 9572 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2019-06-11 11:09:34.284 INFO 9572 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-06-11 11:09:34.289 ERROR 9572 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class
Action:
Consider the following:
If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

You have a data start dependency but didn't add the driver for H2 or you want to use another database then you have to add at least: url, username, password
If you don't want to use a database remove the starter from your maven or gradle project file.

You might have mentioned below dependency
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa'
So, Spring will look for datasource url in application.properties, so remove that or if you you want to work with jdbc add the properties in application.properties
spring.datasource.url = <jdbc-url>
spring.datasource.username = <db-username>
spring.datasource.password = <db-password>
spring.datasource.driverClassName = <db-driver>

If you don't use any datasource in your application you can put following annotation on your CiaoMondoApplication class
#EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class})

/*
* Hi Simon,
*
* Couple of questions, have you used the Spring Initializr to build the project or * have you done if from scratch?
*
* https://start.spring.io/
*
* Is a good place to start:
*
* Here is my basic app (to call a sendgrid api) but that does not matter.
*
*/
// Rest Application
package com.XXXXXXX.restapi;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class RestApiApplication{
public static void main(String[] args)
{
SpringApplication.run(RestApiApplication.class, args);
}
}
and that is all that is in my restapi.java
I then have a controller
sendgridcontroller.java
import java.io.IOException;
#RestController
#RequestMapping(sendGridController.BASE_URL)
public class sendGridController
{
public static final String BASE_URL = "/api/v1/feedbacks";
private final sendGridAPI sendGridAPIService;
public sendGridController(sendGridAPIImpl sendGridAPIService)
{
this.sendGridAPIService = sendGridAPIService;
}
#CrossOrigin
#PostMapping("/sendfeedback")
public ResponseEntity<Boolean> sendFeedback(#RequestParam(name="fullname")String fullName,
#RequestParam(name="subject")String subject,
#RequestParam(name="email") String emailAddr,
#RequestParam(name="message")String message )
{
Boolean bReturn = true;
System.out.println("fullName ="+fullName);
System.out.println("subject ="+subject);
System.out.println("email ="+emailAddr);
System.out.println("message ="+message);
try
{
sendGridAPIService.sendFeedbackEmail(fullName, subject, emailAddr, message);
}
catch(Exception e)
{
return new ResponseEntity<>(false,HttpStatus.BAD_GATEWAY);
}
return new ResponseEntity<>(bReturn, HttpStatus.OK);
}
}
This is in its own package
Next i created a services package with two files
an Interface sendGridApi.java
package com.ea_optimised.restapi.services;
import org.springframework.stereotype.Service;
import java.io.IOException;
public interface sendGridAPI
{
void sendFeedbackEmail(String fullNameme, String subject, String emailAddr, String message) throws IOException;
}
// .... and Finally, I have an implementation of the interface
package com.XXXXX.restapi.services;
import com.sendgrid.*;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.net.UnknownHostException;
import java.rmi.ServerException;
#Service
public class sendGridAPIImpl implements sendGridAPI
{
// API_KEY
private static final String SENDGRID_API_KEY = "My key went here";
#Override
public void sendFeedbackEmail(String fullNameme, String subject, String emailAddr, String message) throws IOException
{
Email from = new Email(emailAddr);
Email to = new Email("info#ea-optimised.co.uk");
// Email to = new Email("test#example.com");
Content content = new Content("text/plain", message);
Mail mail = new Mail(from, subject, to, content);
SendGrid sg = new SendGrid(SENDGRID_API_KEY);
Request request = new Request();
try
{
request.setMethod(Method.POST);
request.setEndpoint("mail/send");
request.setBody(mail.build());
Response response = sg.api(request);
System.out.println("****\nSTATUS="+response.getStatusCode());
System.out.println("\n***Body :\n\n"+response.getBody());
System.out.println("***Headers:\n\n"+response.getHeaders());
}
catch ( UnknownHostException e )
{
System.out.println("***Unknown Host Exception:\n\n");
throw e;
}
catch (Exception ex)
{
System.out.println( "SendGrid Failed "+ex.getMessage());
try
{
throw ex;
}
catch (IOException e)
{
e.printStackTrace();
throw e;
}
}
}
}
Hope this helps
Finally are you using Gradle or Maven?

Spring boot Async + multithreading shuts down without completing all the tasks

Spring boot shuts down without completing all the task.
I want to collect the data from the MYSQL database and then export this data as csv file.. However Spring boot shuts down after some time..
I tried following the example Here and Here but it seems like
CompletableFuture.allOf(tasks.toArray(new CompletableFuture[tasks.size()])).join();
doesnot wait untill all the tasks is finished.
NOTE: I have over 150 task for which i need to export data to the CSV file.
Code:
MainApplication.java
#SpringBootApplication
#EnableAsync
public class MainApplication implements CommandLineRunner {
#Autowired
private TaskRunner taskrunner;
public static void main(String[] args) throws Exception{
SpringApplication application = new SpringApplication(MainApplication.class);
application.setBannerMode(Banner.Mode.OFF);
application.run(args);
}
public void run(String... args) throws Exception {
taskrunner.executeTasks();
exit(0);
}
}
TaskRunner.java
#Service
public class TaskRunner {
#Autowired
public DataCollector dataCollector;
public void executeTask() throws Exception {
final List<String> parameters = dataCollector.getParameters();
List<CompletableFuture> tasks = new ArrayList<CompletableFuture>();
for (String name : parameters) {
try{
tasks.add(CompletableFuture.runAsync(() -> dataCollector.ExportDataToCsv(name)));
}catch(){
ex.printStackTrace();
System.out.println("Export failed for Param: "+name);
}
}
CompletableFuture.allOf(tasks.toArray(new CompletableFuture[tasks.size()])).join();
System.out.println("All Task Finished");
}
}
DataCollector.java
public class DataCollector{
#Autowired
public DataRepository dataRepository;
#Async("ThreadPoolTaskExecutor")
public CompletableFuture<String> ExportDataToCsv(String tableName){
// Code To export data to csv
}
}
DataRepository.java
#Repository
public class DataRepository {
#Qualifier("jdbcExportService")
#Autowired
public JdbcTemplate jdbcTemplate;
public SqlRowSet getParamData(String param){
String Statement = "select * FROM " + param;
return jdbcTemplate.queryForRowSet(Statement);
}
}
DatabaseConfiguration.java
#Configuration
public class DatabaseConfiguration {
#Bean(name="db")
#ConfigurationProperties(prefix = "spring.db")
public DataSource createExportDataSource(){
return DataSourceBuilder.create().build();
}
#Bean(name = "jdbcExportService")
#Autowired
public JdbcTemplate createJdbcTemplateExportService(#Qualifier("db") DataSource exportServiceDS){
return new JdbcTemplate(exportServiceDS);
}
}
Output:
2019-06-25 14:39:56.868 INFO 13163 --- [ Thread-2] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'ThreadPoolTaskExecutor'
2019-06-25 14:39:56.868 INFO 13163 --- [ Thread-2] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2019-06-25 14:39:56.997 INFO 13163 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2019-06-25 14:39:57.011 DEBUG 13163 --- [ Thread-2] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Before shutdown stats (total=10, active=5, idle=5, waiting=0)
2019-06-25 14:39:57.371 DEBUG 13163 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#2c373ded: (connection evicted)
2019-06-25 14:39:57.991 DEBUG 13163 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#4856fa0b: (connection evicted)
2019-06-25 14:39:57.994 DEBUG 13163 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#375c83e: (connection evicted)
2019-06-25 14:39:57.995 DEBUG 13163 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#417d7b5b: (connection evicted)
2019-06-25 14:39:57.996 DEBUG 13163 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.mysql.cj.jdbc.ConnectionImpl#77ebb3e9: (connection evicted)
2019-06-25 14:39:58.094 DEBUG 13163 --- [ Thread-2] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - After shutdown stats (total=0, active=0, idle=0, waiting=0)
2019-06-25 14:39:58.095 INFO 13163 --- [ Thread-2] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2019-06-25 14:39:58.651 WARN 13163 --- [ export-thread1] com.zaxxer.hikari.pool.ProxyConnection : HikariPool-1 - Connection com.mysql.cj.jdbc.ConnectionImpl#4c18c2a2 marked as broken because of SQLSTATE(08003), ErrorCode(0)

my guess after reading up a bit and i'm in no way a pro at this. So this could very well be wrong.
CompletableFuture#join waits until you can extract the resulting value from that CompletableFuture. CompletableFuture#get throws a checked exception and is interruptible while CompletableFuture#join is non-interruptible. But both do the same thing, extract the value from the completable future, and block if needed until they do.
What i think you are looking for since you want to print something when they are done is to use the CompletableFuture#thenAccept that will perform "something" when all are finished.
CompletableFuture<Void> allFutures = CompletableFuture.allOf(tasks.toArray(new CompletableFuture[tasks.size()]));
allFutures.thenAccept(Void -> System.out.println("All Task Finished"));
My guess is that it sets up all async tasks and then passes it all and then exits the application.
references:
java-8-completablefuture-in-action
completablefuture-join-vs-get

Hibernate trying to set field value to object

Trying to use Hibernate from scratch for first time.
I have entity class:
package centaurus.domain;
import javax.persistence.*;
#Entity
#Table(name="users")
public class Player {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name="USER_ID")
private int id;
#Column(name="email")
private String email;
public Player(){};
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
public void setId(int id) {
this.id = id;
}
public int getId() {
return id;
}
}
DAOimpl:
package centaurus.service;
import centaurus.domain.Player;
import restx.factory.Component;
import org.hibernate.HibernateException;
import org.hibernate.Session;
import org.hibernate.Transaction;
import org.hibernate.cfg.Configuration;
import org.hibernate.SessionFactory;
#Component
public class PlayerDAOimpl implements PlayerDAO{
private static SessionFactory factory;
public PlayerDAOimpl() {
try{
factory = new Configuration().
configure().addAnnotatedClass(Player.class).
buildSessionFactory();
//
}catch (Throwable ex) {
System.err.println("Failed to create sessionFactory object." + ex);
throw new ExceptionInInitializerError(ex);
}
}
public Player savePlayer(Player player){
Session session = factory.openSession();
Transaction tx = null;
Integer playerID = null;
try{
tx = session.beginTransaction();
playerID = (Integer) session.save(player);
tx.commit();
}catch (HibernateException e) {
if (tx!=null) tx.rollback();
e.printStackTrace();
}finally {
session.close();
}
return player;
}
public Player getPlayer(Integer playerId){
Session session = factory.openSession();
try{
Player player = (Player)session.get(Player.class, playerId);
return player;
}catch (HibernateException e) {
}finally {
session.close();
}
return null;
}
}
to demo the error i made it add a player on startupscript.
and error:
-- RESTX >> LOAD ON REQUEST << >> DEV MODE << >> AUTO COMPILE <<
-- for admin console,
-- VISIT http://127.0.0.1:9091/api/#/ui/
--
2016-05-31 21:42:48,901 [main ] [ ] INFO restx.Apps - can't enable Apidocs doclet: make sure tools.jar is in your classpath
2016-05-31 21:42:49,978 [pool-1-thread-1 ] [ ] INFO restx.classloader.CompilationManager - compilation finished: 12 sources compiled in 1.015 s
2016-05-31 21:42:50,061 [main ] [ ] INFO restx.Apps - can't enable Apidocs doclet: make sure tools.jar is in your classpath
2016-05-31 21:42:50,078 [main ] [ ] INFO restx.classloader.CompilationManager - watching for changes in [src/main/java, src/main/resources]; current location is /home/arthur/elorhia/api/.
2016-05-31 21:42:50,145 [main ] [ ] INFO org.hibernate.Version - HHH000412: Hibernate Core {5.1.0.Final}
2016-05-31 21:42:50,146 [main ] [ ] INFO org.hibernate.cfg.Environment - HHH000206: hibernate.properties not found
2016-05-31 21:42:50,147 [main ] [ ] INFO org.hibernate.cfg.Environment - HHH000021: Bytecode provider name : javassist
2016-05-31 21:42:50,802 [main ] [ ] INFO o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
2016-05-31 21:42:50,851 [main ] [ ] WARN org.hibernate.orm.connections - HHH10001002: Using Hibernate built-in connection pool (not for production use!)
2016-05-31 21:42:50,856 [main ] [ ] INFO org.hibernate.orm.connections - HHH10001005: using driver [com.mysql.jdbc.Driver] at URL [jdbc:mysql://localhost/andromeda]
2016-05-31 21:42:50,857 [main ] [ ] INFO org.hibernate.orm.connections - HHH10001001: Connection properties: {user=api, password=****}
2016-05-31 21:42:50,858 [main ] [ ] INFO org.hibernate.orm.connections - HHH10001003: Autocommit mode: false
2016-05-31 21:42:50,860 [main ] [ ] INFO o.h.e.j.c.i.DriverManagerConnectionProviderImpl - HHH000115: Hibernate connection pool size: 1 (min=1)
2016-05-31 21:42:51,090 [main ] [ ] INFO org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.MySQLDialect
2016-05-31 21:42:51,321 [main ] [ ] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 5.0.1.Final
START SCRIPT!
org.hibernate.property.access.spi.PropertyAccessException: Error accessing field [private java.lang.String centaurus.domain.Player.email] by reflection for persistent property [centaurus.domain.Player#email] : centaurus.domain.Player#507b79f7
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:43)
at org.hibernate.property.access.spi.GetterFieldImpl.getForInsert(GetterFieldImpl.java:58)
at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValuesToInsert(AbstractEntityTuplizer.java:521)
at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValuesToInsert(PojoEntityTuplizer.java:228)
at org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValuesToInsert(AbstractEntityPersister.java:4701)
at org.hibernate.event.internal.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:254)
at org.hibernate.event.internal.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:182)
at org.hibernate.event.internal.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:113)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:192)
at org.hibernate.event.internal.DefaultSaveEventListener.saveWithGeneratedOrRequestedId(DefaultSaveEventListener.java:38)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:177)
at org.hibernate.event.internal.DefaultSaveEventListener.performSaveOrUpdate(DefaultSaveEventListener.java:32)
at org.hibernate.event.internal.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:73)
at org.hibernate.internal.SessionImpl.fireSave(SessionImpl.java:682)
at org.hibernate.internal.SessionImpl.save(SessionImpl.java:674)
at org.hibernate.internal.SessionImpl.save(SessionImpl.java:669)
at centaurus.service.PlayerDAOimpl.savePlayer(PlayerDAOimpl.java:39)
at centaurus.Dbmaintain.start(Dbmaintain.java:26)
at restx.factory.Factory.start(Factory.java:846)
at restx.RestxMainRouterFactory.build(RestxMainRouterFactory.java:450)
at restx.RestxMainRouterFactory.newInstance(RestxMainRouterFactory.java:70)
at restx.servlet.RestxMainRouterServlet.init(RestxMainRouterServlet.java:74)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:519)
at org.eclipse.jetty.servlet.ServletHolder.doStart(ServletHolder.java:331)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:747)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:265)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:706)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:492)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerCollection.doStart(HandlerCollection.java:229)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:95)
at org.eclipse.jetty.server.Server.doStart(Server.java:277)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at restx.server.JettyWebServer.start(JettyWebServer.java:109)
at restx.server.JettyWebServer.startAndAwait(JettyWebServer.java:114)
at centaurus.AppServer.main(AppServer.java:30)
Caused by: java.lang.IllegalArgumentException: Can not set java.lang.String field centaurus.domain.Player.email to centaurus.domain.Player
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167)
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171)
at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58)
at sun.reflect.UnsafeObjectFieldAccessorImpl.get(UnsafeObjectFieldAccessorImpl.java:36)
at java.lang.reflect.Field.get(Field.java:393)
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:39)
... 40 more
2016-05-31 21:42:51,673 [main ] [ ] INFO restx.monitor.MetricsConfiguration - registering Metrics JVM metrics
2016-05-31 21:44:41,646 [pool-2-thread-1 ] [ ] INFO restx.classloader.CompilationManager - compilation finished: 1 sources compiled in 86.94 ms
it looks like its trying to set the email field to the player object.
I cant work out why!, what have I done wrong?
EDIT: as requested i have added the code creating the player and calling the dao
package centaurus;
import centaurus.domain.Player;
import centaurus.service.PlayerDAO;
import centaurus.service.PlayerDAOimpl;
import restx.factory.AutoStartable;
import restx.factory.Component;
import javax.inject.Named;
#Component
public class Dbmaintain implements AutoStartable{
private PlayerDAO playerDAO;
public Dbmaintain(#Named("PlayerDAOimpl") PlayerDAO playerDAO) {
this.playerDAO = playerDAO;
}
public void start(){
System.out.println("START SCRIPT!");
//test
Player p = new Player();
p.setEmail("test");
playerDAO.savePlayer(p);
try {
// URL configurationUrl = new File("dbmaintain.properties").toURI().toURL();
// MainFactory mainFactory = new MainFactory(configurationUrl);
// DbMaintainer dbMaintainer = mainFactory.createDbMaintainer();
// dbMaintainer.updateDatabase(false);
} catch(Exception e) {
}
}
}
edit added git link if people wanted other bits:
https://github.com/ArthurGibbs/Centaurus-

How are you setting/creating your Player?
Caused by: java.lang.IllegalArgumentException: Can not set java.lang.String field centaurus.domain.Player.email to centaurus.domain.Player
This says, that there is an attempt to store object of class centaurus.domain.Player into a String field. For futher explanation, attach che actual code that is modifying/creating player perfore passing it to DAO

I have used your exact class Player to execute this piece of code and got no error at all. A row is inserted in table USERS correctly.
public class TestPlayer {
public static void main(String[] args) {
EntityManagerFactory emf = Persistence.createEntityManagerFactory("TestDB");
EntityManager em = emf.createEntityManager();
Session session = (Session) em.getDelegate();
Transaction tx = session.beginTransaction();
try {
Player p = new Player();
p.setEmail("email");
session.save(p);
tx.commit();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
em.close();
emf.close();
}
}
}
The problem is not in the entity class. Maybe some changes that you made do not refresh correctly on deploy.