Develop Reference

java read data from json and save it in ms sql database - error argument "src" is null

I want to read data from json file and save it to ms sql data base
This is my application.java class. The error is in this file on the line before for loop
#SpringBootApplication
public class Covid19Application {
public static void main(String[] args) {
SpringApplication.run(Covid19Application.class, args);
}
#Bean
CommandLineRunner runner(SaveDataFromJsonToDBService saveDataFromJsonToDBService) {
return args -> {
ObjectMapper mapper = new ObjectMapper();
TypeReference<List<CovidStatistics>> typeReference = new TypeReference<List<CovidStatistics>>(){};
InputStream inputStream = TypeReference.class.getResourceAsStream("/CovidStatistics.json");
try {
List<CovidStatistics> covidStatistics = mapper.readValue(inputStream,typeReference);
for (CovidStatistics cs : covidStatistics) {
saveDataFromJsonToDBService.save(cs);
}
//saveDataFromJsonToDBService.save(covidStatistics);
System.out.println("Users Saved!");
} catch (IOException e){
System.out.println("Unable to save users: " + e.getMessage());
}
};
}
}
And this is the error. I dont know where is the problem, I try to google it, but I dont found a solution
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:774) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:755) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.4.jar:2.7.4]
at com.example.Covid19.Covid19Application.main(Covid19Application.java:18) ~[classes/:na]
Caused by: java.lang.IllegalArgumentException: argument "src" is null
at com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4737) ~[jackson-databind-2.11.4.jar:2.11.4]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3512) ~[jackson-databind-2.11.4.jar:2.11.4]
at com.example.Covid19.Covid19Application.lambda$runner$0(Covid19Application.java:28) ~[classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:771) ~[spring-boot-2.7.4.jar:2.7.4]
... 5 common frames omitted
2022-10-06 20:42:51.780 INFO 30652 --- [ main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2022-10-06 20:42:51.782 INFO 30652 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2022-10-06 20:42:51.788 INFO 30652 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
Process finished with exit code 1

As you want to read from the classpath root you must use, It seems the inputstream is null so its not finding the file.
InputStream inputStream = TypeReference.class.getClassLoader().getResourceAsStream("/CovidStatistics.json");
TypeReference.class.getResourceAsStream will look in the directory where the class file is.

First wrap your InputStream into a try with resources. Next print all errors with catching exception. Also make sure that the file you are referring to can be found.
CommandLineRunner runner(SaveDataFromJsonToDBService saveDataFromJsonToDBService) {
return args -> {
ObjectMapper mapper = new ObjectMapper();
TypeReference<List<CovidStatistics>> typeReference = new TypeReference<List<CovidStatistics>>(){};
try(InputStream inputStream = TypeReference.class.getResourceAsStream("/CovidStatistics.json")) {
List<CovidStatistics> covidStatistics = mapper.readValue(inputStream,typeReference);
for (CovidStatistics cs : covidStatistics) {
saveDataFromJsonToDBService.save(cs);
}
//saveDataFromJsonToDBService.save(covidStatistics);
System.out.println("Users Saved!");
} catch (Exception e){
System.out.println("Unable to save users: " + e.getMessage());
}
};
}```

Spring Batch with multi - step Spring Cloud Task (PartitionHandler) for Remote Partition

Latest Update (with an image to hope simplify the problem) (thanks for feedback from #Mahmoud)
Relate issue reports for other reference (after this original post created, it seem someone filed issues for Spring Cloud on similar issue, so also update there too):
https://github.com/spring-cloud/spring-cloud-task/issues/793 relate to approach #1
https://github.com/spring-cloud/spring-cloud-task/issues/792 relate to approach #2
Also find a workaround resolution for that issue and update on that github issue, will update this once it is confirmed good by developer
https://github.com/spring-cloud/spring-cloud-task/issues/793#issuecomment-894617929
I am developing an application involved multi-steps using spring batch job but hit some roadblock. Did try to research doc and different attempts, but no success. So thought to check if community can shed light
Spring batch job 1 (received job parameter for setting for step 1/setting for step 2)
Step 1 -> remote partition (partitionhandler (cpu/memory for step 1 + grid) + partitioner) with setting from step1 (job configuration or step configuration)
Step 2 -> remote partition (partitionhandler (cpu/memory for step 2 + grid) + partitioner) with setting from step2 (job configuration or step configuration, and diff from step 1)
The reason we want is to have different step with different k8s setting (like cpu/memory/grid)
Attempts:
Create two partition handler (partitionHandlerReader + partitionHandlerProcessor) and their corresponding launcher (LauncherReader + LauncherProcessor)
Complete Project can be found in
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_1_two_partitionhandlers
The main class of configuration is try to simplify into one class
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/attempt_1_two_partitionhandlers/src/main/java/com/example/batchprocessing/BatchConfiguration.java
Use one PartitionerHandler + one TaskLauncher but with #StepScope for late binding for dynamic change base on step and job setup
Complete Project can be found in
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope
The main class of configuration is try to simplify into one class
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/attempt_2_partitionhandler_with_stepscope/src/main/java/com/example/batchprocessing/BatchConfiguration.java
Both Result Following (full trace at above git repo):
During job trigger, it will error (it seem pass initial start up, but error during execution)
Because below will only occur when there are multiple PartitionHandler or when that Bean is at #StepScope or #JobScope
java.lang.NullPointerException: null
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:347) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
Full Log
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.4.6)
2021-08-06 11:24:29.242 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : Starting BatchProcessingApplication v0.0.1-SNAPSHOT using Java 11.0.7 on localhost.localdomain with PID 90294 (/home/danilko/IdeaProjects/partition/target/batchprocessing-0.0.1-SNAPSHOT.jar started by danilko in /home/danilko/IdeaProjects/partition)
2021-08-06 11:24:29.244 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : The following profiles are active: controller
2021-08-06 11:24:29.790 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2021-08-06 11:24:29.794 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2021-08-06 11:24:29.797 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2021-08-06 11:24:29.833 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.947 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.947 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.959 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$83e6c2be] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.968 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration' of type [org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration$$EnhancerBySpringCGLIB$$cc3cccc1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:30.093 INFO 90294 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-08-06 11:24:30.160 INFO 90294 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-08-06 11:24:30.724 INFO 90294 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: MYSQL
2021-08-06 11:24:30.736 INFO 90294 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'application.errorChannel' has 1 subscriber(s).
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean '_org.springframework.integration.errorLogger'
2021-08-06 11:24:30.974 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : Started BatchProcessingApplication in 2.024 seconds (JVM running for 2.366)
2021-08-06 11:24:30.975 INFO 90294 --- [ main] o.s.b.a.b.JobLauncherApplicationRunner : Running default command line with: []
2021-08-06 11:24:31.010 INFO 90294 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=partitionedJob-1538890488]] launched with the following parameters: [{}]
Set readerGridSize == 1
2021-08-06 11:24:31.020 INFO 90294 --- [ main] o.s.c.t.b.l.TaskBatchExecutionListener : The job execution id 22 was run within the task execution 54
2021-08-06 11:24:31.046 INFO 90294 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [partitionReaderStep]
2021-08-06 11:24:31.101 ERROR 90294 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step partitionReaderStep in job partitionedJob-1538890488
java.lang.NullPointerException: null
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:347) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.7.jar!/:5.3.7]
at com.sun.proxy.$Proxy65.handle(Unknown Source) ~[na:na]
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:106) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-5.3.7.jar!/:5.3.7]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.7.jar!/:5.3.7]
at com.sun.proxy.$Proxy51.run(Unknown Source) ~[na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:799) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:789) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:346) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) ~[spring-boot-2.4.6.jar!/:2.4.6]
at com.example.batchprocessing.BatchProcessingApplication.main(BatchProcessingApplication.java:10) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Study/Reference:
Most tutorial I found online only involved one partition step.
https://dataflow.spring.io/docs/feature-guides/batch/partitioning/
Thanks for info/helps in advance

Is above even possible setup?
yes, nothing prevents you from having two partitioned steps in a single Spring Batch job.
Is it possible to use JobScope/StepScope to pass info to the partitionhandler
yes, it is possible for the partition handler to be declared as a job/step scoped bean if it needs the late-binding feature to be configured.
Updated on 08/14/2021 by #DanilKo
The original answer is correct in high - level. However, to actually achieve the partition handeler to be step scoped, a code modification is required
Below is the analyze + my proposed workaround/fix (maybe eventually code maintainer will have better way to make it work, but so far below fix is working for me)
Issue being continued to discuss at:
https://github.com/spring-cloud/spring-cloud-task/issues/793 (multiple partitioner handler discussion)
https://github.com/spring-cloud/spring-cloud-task/issues/792
(which this fix is based up to use partitionerhandler at step scope to configure different worker steps + resources + max worker)
Root cause analyze (hypothesis)
The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task) or there are two partitionHandler, that setup is no longer triggered, as #EnableTask seem not able to locate one partitionhandler during creation.
https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-batch/src/main/java/org/springframework/cloud/task/batch/partition/DeployerPartitionHandler.java # 269
Resulted created DeployerHandler faced a null with taskExecution when trying to launch (as it is never setup)
https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-batch/src/main/java/org/springframework/cloud/task/batch/partition/DeployerPartitionHandler.java # 347
Workaround Resolution
Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
It seem to work, but still not clear if there is other side effect (so far during test not found any)
Full code can be found in https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope_workaround_resolution
In the partitionHandler method
#Bean
#StepScope
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
JobExplorer jobExplorer,
#Value("#{stepExecution}") StepExecution stepExecution) throws Exception {
...
// After the declaration of partitionhandler
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerStep")
, taskRepository);
// Issue https://github.com/spring-cloud/spring-cloud-task/issues/793
// Perform the setting of execution as this partitioner now not created at task level so #beforetask is no longer vaild
// The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
// But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task), that setup is no longer triggered
// Resulted created DeployerHandler faced a null
// Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
// From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
// It seem to work, but still not clear if there is other side effect (so far during test not found any)
long executionId = taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId());
System.out.println("Current execution job to task execution id " + executionId);
TaskExecution taskExecution = taskExplorer.getTaskExecution(taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId()));
System.out.println("Current execution job to task execution is not null: " + (taskExecution != null));
partitionHandler.beforeTask(taskExecution);
...
// rest of code continue
(note it utilize stepExecution context to find out the current trigger step name and therefore assign different worker step)
Worker name in this case is coming from pre-defined job execution, but may able to come from jobparameter or another place too)
That job context is populated with job listner
Job is configured with job listener
#Bean(name = "partitionedJob")
#Profile("!worker")
public Job partitionedJob()throws Exception {
Random random = new Random();
return jobBuilderFactory.get("partitionedJob" + random.nextInt())
.start(partitionReaderStep())
.listener(jobExecutionListener())
.next(partitionProcessorStep())
.build();
}
In job listener populated it
#Bean
public JobExecutionListener jobExecutionListener() {
JobExecutionListener listener = new JobExecutionListener(){
#Override
public void beforeJob(JobExecution jobExecution)
{
jobExecution.getExecutionContext().putString("readerCPURequest", "1");
jobExecution.getExecutionContext().putString("readerCPULimit", "2");
jobExecution.getExecutionContext().putString("readerWorkerGridSize", "1");
// For now using same image for reader/processor, but if it work, can split them
jobExecution.getExecutionContext().putString("readerWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("readerWorkerStep", "workerStepReader");
jobExecution.getExecutionContext().putString("processorCPURequest", "3");
jobExecution.getExecutionContext().putString("processorCPULimit", "4");
jobExecution.getExecutionContext().putString("processorWorkerGridSize", "2");
// For now using same image for reader/processor, but if it work, will split them
jobExecution.getExecutionContext().putString("processorWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("processorWorkerStep", "workerStepProcessor");
System.out.println("Set readerGridSize == " + jobExecution.getExecutionContext().getString("readerGridSize", "IT IS NULL WHICH IS INCORRECT"));
}
#Override
public void afterJob(JobExecution jobExecution) {
}
};
return listener;
}
Full code (can also be found in my code github after the workaround fix is being applied): https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/main/src/main/java/com/example/batchprocessing/BatchConfiguration.java
package com.example.batchprocessing;
import io.fabric8.kubernetes.api.model.DeletionPropagation;
import io.fabric8.kubernetes.api.model.batch.JobList;
import io.fabric8.kubernetes.api.model.batch.JobSpec;
import io.fabric8.kubernetes.api.model.batch.JobStatus;
import io.fabric8.kubernetes.client.KubernetesClient;
import org.springframework.batch.core.*;
import org.springframework.batch.core.configuration.JobRegistry;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.explore.JobExplorer;
import org.springframework.batch.core.partition.PartitionHandler;
import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.cloud.deployer.resource.docker.DockerResource;
import org.springframework.cloud.deployer.resource.support.DelegatingResourceLoader;
import org.springframework.cloud.deployer.spi.kubernetes.*;
import org.springframework.cloud.deployer.spi.task.TaskLauncher;
import org.springframework.cloud.task.batch.partition.*;
import org.springframework.cloud.task.configuration.EnableTask;
import org.springframework.cloud.task.repository.TaskExecution;
import org.springframework.cloud.task.repository.TaskExplorer;
import org.springframework.cloud.task.repository.TaskRepository;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.env.Environment;
import org.springframework.core.env.SystemEnvironmentPropertySource;
import org.springframework.core.io.Resource;
import org.springframework.core.task.TaskExecutor;
import org.springframework.core.task.TaskRejectedException;
import org.springframework.util.StringUtils;
import java.util.*;
#Configuration
#EnableBatchProcessing
#EnableTask
public class BatchConfiguration {
private static int BACK_OFF_LIMIT = 6;
// Set the kuberentes job name
private String taskName_prefix="partitionedbatchjob";
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public JobExplorer jobExplorer;
#Autowired
public JobRepository jobRepository;
#Autowired
public TaskExecutor taskExecutor;
#Autowired
public TaskRepository taskRepository;
#Autowired
public TaskExplorer taskExplorer;
#Autowired
private ConfigurableApplicationContext context;
#Autowired
private DelegatingResourceLoader resourceLoader;
#Autowired
private Environment environment;
#Bean
#StepScope
public Partitioner partitioner( #Value("#{stepExecution}") StepExecution stepExecution) {
return new Partitioner() {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> partitions = new HashMap<>(gridSize);
int targetGridSize = 0;
String step = "";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep"))
{
step = "reader";
}
else
{
step = "processor";
}
targetGridSize = Integer.parseInt(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerGridSize"));
for (int i = 0; i < targetGridSize; i++) {
ExecutionContext context1 = new ExecutionContext();
context1.put("partitionNumber", i);
partitions.put("partition" + i, context1);
}
return partitions;
}
};
}
#Bean
public KubernetesClient kuberentesClient()
{
KubernetesDeployerProperties kubernetesDeployerProperties = new KubernetesDeployerProperties();
return KubernetesClientFactory.getKubernetesClient(kubernetesDeployerProperties);
}
#Bean
#StepScope
public TaskLauncher taskLauncher( #Value("#{stepExecution}") StepExecution stepExecution)
{
KubernetesDeployerProperties kubernetesDeployerProperties = new KubernetesDeployerProperties();
kubernetesDeployerProperties.setNamespace("default");
kubernetesDeployerProperties.setCreateJob(true);
// Database setup to reference configmap for database info
List<KubernetesDeployerProperties.ConfigMapKeyRef> configMapKeyRefList = new ArrayList<KubernetesDeployerProperties.ConfigMapKeyRef>();
KubernetesDeployerProperties.ConfigMapKeyRef configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_URL");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_URL");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_USERNAME");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_USERNAME");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_PASSWORD");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_PASSWORD");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_DRIVERCLASSNAME");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_DRIVERCLASSNAME");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_PROFILES_ACTIVE");
configMapKeyRef.setEnvVarName("SPRING_PROFILES_ACTIVE");
configMapKeyRefList.add(configMapKeyRef);
kubernetesDeployerProperties.setConfigMapKeyRefs(configMapKeyRefList);
// Set request resource
KubernetesDeployerProperties.RequestsResources request = new KubernetesDeployerProperties.RequestsResources();
KubernetesDeployerProperties.LimitsResources limit = new KubernetesDeployerProperties.LimitsResources();
String step = "";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep"))
{
step="reader";
}
else
{
step="processor";
}
request.setCpu(stepExecution.getJobExecution().getExecutionContext().getString(step + "CPURequest"));
request.setMemory("2000Mi");
limit.setCpu(stepExecution.getJobExecution().getExecutionContext().getString(step +"CPULimit"));
limit.setMemory("3000Mi");
kubernetesDeployerProperties.setRequests(request);
kubernetesDeployerProperties.setLimits(limit);
// as build on local image, so need to use local
kubernetesDeployerProperties.setImagePullPolicy(ImagePullPolicy.IfNotPresent);
// Set task launcher properties to not repeat and not restart
KubernetesTaskLauncherProperties kubernetesTaskLauncherProperties = new KubernetesTaskLauncherProperties();
// https://kubernetes.io/docs/concepts/workloads/controllers/job/
// Set to never to create new pod on restart
kubernetesTaskLauncherProperties.setBackoffLimit(BACK_OFF_LIMIT);
kubernetesTaskLauncherProperties.setRestartPolicy(RestartPolicy.Never);
KubernetesTaskLauncher kubernetesTaskLauncher = new KubernetesTaskLauncher(kubernetesDeployerProperties,
kubernetesTaskLauncherProperties, kuberentesClient());
return kubernetesTaskLauncher;
}
#Bean(name = "partitionedJob")
#Profile("!worker")
public Job partitionedJob()throws Exception {
Random random = new Random();
return jobBuilderFactory.get("partitionedJob" + random.nextInt())
.start(partitionReaderStep())
.listener(jobExecutionListener())
.next(partitionProcessorStep())
.build();
}
#Bean(name = "partitionReaderStep")
public Step partitionReaderStep() throws Exception {
return stepBuilderFactory.get("partitionReaderStep")
.partitioner(workerStepReader().getName(), partitioner( null))
.step(workerStepReader())
.partitionHandler(partitionHandler(
taskLauncher( null),
jobExplorer, null))
.build();
}
#Bean(name = "partitionProcessorStep")
public Step partitionProcessorStep() throws Exception {
return stepBuilderFactory.get("partitionProcessorStep")
.partitioner(workerStepProcessor().getName(), partitioner( null))
.step(workerStepProcessor())
.partitionHandler(partitionHandler(
taskLauncher( null),
jobExplorer, null))
.build();
}
#Bean
#StepScope
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
JobExplorer jobExplorer,
#Value("#{stepExecution}") StepExecution stepExecution) throws Exception {
String step ="processor";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep")) {
step = "reader";
}
// Use local build image
DockerResource resource = new DockerResource(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerImage"));
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerStep")
, taskRepository);
// Issue https://github.com/spring-cloud/spring-cloud-task/issues/793
// Perform the setting of execution as this partitioner now not created at task level so #beforetask is no longer vaild
// The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
// But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task), that setup is no longer triggered
// Resulted created DeployerHandler faced a null
// Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
// From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
// It seem to work, but still not clear if there is other side effect (so far during test not found any)
long executionId = taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId());
System.out.println("Current execution job to task execution id " + executionId);
TaskExecution taskExecution = taskExplorer.getTaskExecution(taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId()));
System.out.println("Current execution job to task execution is not null: " + (taskExecution != null));
partitionHandler.beforeTask(taskExecution);
List<String> commandLineArgs = new ArrayList<>(3);
commandLineArgs.add("--spring.profiles.active=worker");
commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
commandLineArgs.add("--spring.batch.initializer.enabled=false");
partitionHandler
.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
partitionHandler.setEnvironmentVariablesProvider(new NoOpEnvironmentVariablesProvider());
partitionHandler.setMaxWorkers(Integer.parseInt(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerGridSize")));
partitionHandler.setApplicationName(taskName_prefix + step);
return partitionHandler;
}
#Bean
public JobExecutionListener jobExecutionListener() {
JobExecutionListener listener = new JobExecutionListener(){
#Override
public void beforeJob(JobExecution jobExecution)
{
jobExecution.getExecutionContext().putString("readerCPURequest", "1");
jobExecution.getExecutionContext().putString("readerCPULimit", "2");
jobExecution.getExecutionContext().putString("readerWorkerGridSize", "1");
// For now using same image for reader/processor, but if it work, can split them
jobExecution.getExecutionContext().putString("readerWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("readerWorkerStep", "workerStepReader");
jobExecution.getExecutionContext().putString("processorCPURequest", "3");
jobExecution.getExecutionContext().putString("processorCPULimit", "4");
jobExecution.getExecutionContext().putString("processorWorkerGridSize", "2");
// For now using same image for reader/processor, but if it work, will split them
jobExecution.getExecutionContext().putString("processorWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("processorWorkerStep", "workerStepProcessor");
System.out.println("Set readerGridSize == " + jobExecution.getExecutionContext().getString("readerGridSize", "IT IS NULL WHICH IS INCORRECT"));
}
#Override
public void afterJob(JobExecution jobExecution) {
}
};
return listener;
}
#Bean
#Profile("worker")
public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) {
return new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository);
}
#Bean(name = "workerStepReader")
public Step workerStepReader() {
return this.stepBuilderFactory.get("workerStepReader")
.tasklet(workerTaskletReader(null))
.build();
}
#Bean(name = "workerStepProcessor")
public Step workerStepProcessor() {
return this.stepBuilderFactory.get("workerStepProcessor")
.tasklet(workerTaskletProcessor(null))
.build();
}
#Bean
#StepScope
public Tasklet workerTaskletReader(
final #Value("#{stepExecution}") StepExecution stepExecution) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
Integer partitionNumber = stepExecution.getExecutionContext().getInt("partitionNumber");
System.out.println("This workerTaskletReader ran partition: " + partitionNumber);
return RepeatStatus.FINISHED;
}
};
}
#Bean
#StepScope
public Tasklet workerTaskletProcessor(
final #Value("#{stepExecution}") StepExecution stepExecution) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
Integer partitionNumber = stepExecution.getExecutionContext().getInt("partitionNumber");
System.out.println("This workerTaskletProcessor ran partition: " + partitionNumber);
return RepeatStatus.FINISHED;
}
};
}
}

Consider defining a bean of type 'com.example.amazonsync.Service.IAmazonUtilService' in your configuration

I am getting null pointer exception in my service class. I have autowired myservice class named IAmazonUtilService. But it am facing null pointervexception.
Also i have written PropertiesUtil to read data from application.properties which is also not working. Please help me on this.
2020-03-27 18:39:20.172 INFO 17536 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
java.lang.NullPointerException
at com.example.amazonsync.AmazonSync.ImportAmazonDataService.getOrdersFromAmazonStore(ImportAmazonDataService.java:36)
ImportAmazonDataService.java:36
at com.example.amazonsync.SyncData.OrderSync.execute(OrderSync.java:17)
OrderSync.java:17
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
JobRunShell.java:202
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
SimpleThreadPool.java:573
2020-03-27 18:39:20.254 INFO 17536 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2020-03-27 18:39:20.263 INFO 17536 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-03-27 18:39:20.330 INFO 17536 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-03-27 18:39:20.841 ERROR 17536 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Field IAmazonUtilService in com.example.amazonsync.AmazonSync.ImportAmazonDataService required a bean of type 'com.example.amazonsync.Service.IAmazonUtilService' that could not be found.
ImportAmazonDataService.java
public class ImportAmazonDataService {
#Autowired
private IAmazonUtilService IAmazonUtilService;
public ArrayList<String> getChannelLoc() {
ArrayList<String> channelLoc = new ArrayList<String>();
channelLoc.add("US");
channelLoc.add("CA");
channelLoc.add("MX");
return channelLoc;
}
public void getProductsFromAmazonStore(JobExecutionContext context) throws SQLException, ClassNotFoundException {
try {
final Long taskID = (Long) context.getJobDetail().getJobDataMap().get("taskId");
IAmazonUtilService.getChannelConfig("US");
} catch (Exception e) {
e.printStackTrace();
}
}
public void getOrdersFromAmazonStore(JobExecutionContext context) throws SQLException, ClassNotFoundException {
final long taskID = (long) context.getJobDetail().getJobDataMap().get("taskId");
IAmazonUtilService.getChannelConfig("CA");
}
}
Git link : https://github.com/Ezhilarasu1330/SpringBootQuartzSchedular.git

Change extends by implements !!
from:
public class AmazonUtilService extends ImportAmazonDataService {}
to:
public class AmazonUtilService implements ImportAmazonDataService {}

your ImportAmazonDataService class should be annotated as #Service in order spring will know to inject the #Autowired IAmazonUtilService
also refactor
#Autowired
private IAmazonUtilService IAmazonUtilService;
to
#Autowired
private IAmazonUtilService iAmazonUtilService;

Can't run SpringBoot Application

I am new to Spring and i have to make an "HelloWorld" Application. After several attempts, i can't solve my problem.
package com.example.demo;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class CiaoMondoApplication {
public static void main(String[] args) {
SpringApplication.run(CiaoMondoApplication.class, args);
}
} // that's the Application
package com.example.demo;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;
#RestController
public class CiaoMondoController {
#RequestMapping("/")
public String index() {
return "HelloWorld";
}
} // and that's the controller
Below the output, when i try to run the Application with Eclipse:
2019-06-11 11:09:33.089 INFO 9572 --- [ main] com.example.demo.CiaoMondoApplication : Starting CiaoMondoApplication on Asus-Mattia with PID 9572 (C:\Users\matti\eclipse-workspace\ciao-mondo\target\classes started by matti in C:\Users\matti\eclipse-workspace\ciao-mondo)
2019-06-11 11:09:33.091 INFO 9572 --- [ main] com.example.demo.CiaoMondoApplication : No active profile set, falling back to default profiles: default
2019-06-11 11:09:33.627 INFO 9572 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
2019-06-11 11:09:33.644 INFO 9572 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 11ms. Found 0 repository interfaces.
2019-06-11 11:09:33.895 INFO 9572 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$bb774f16] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-06-11 11:09:34.107 INFO 9572 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2019-06-11 11:09:34.127 INFO 9572 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2019-06-11 11:09:34.127 INFO 9572 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.19]
2019-06-11 11:09:34.219 INFO 9572 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-06-11 11:09:34.219 INFO 9572 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1094 ms
2019-06-11 11:09:34.271 WARN 9572 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception; nested exception is org.springframework.boot.autoconfigure.jdbc.DataSourceProperties$DataSourceBeanCreationException: Failed to determine a suitable driver class
2019-06-11 11:09:34.274 INFO 9572 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2019-06-11 11:09:34.284 INFO 9572 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-06-11 11:09:34.289 ERROR 9572 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class
Action:
Consider the following:
If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

You have a data start dependency but didn't add the driver for H2 or you want to use another database then you have to add at least: url, username, password
If you don't want to use a database remove the starter from your maven or gradle project file.

You might have mentioned below dependency
compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-jpa'
So, Spring will look for datasource url in application.properties, so remove that or if you you want to work with jdbc add the properties in application.properties
spring.datasource.url = <jdbc-url>
spring.datasource.username = <db-username>
spring.datasource.password = <db-password>
spring.datasource.driverClassName = <db-driver>

If you don't use any datasource in your application you can put following annotation on your CiaoMondoApplication class
#EnableAutoConfiguration(exclude = {DataSourceAutoConfiguration.class})

/*
* Hi Simon,
*
* Couple of questions, have you used the Spring Initializr to build the project or * have you done if from scratch?
*
* https://start.spring.io/
*
* Is a good place to start:
*
* Here is my basic app (to call a sendgrid api) but that does not matter.
*
*/
// Rest Application
package com.XXXXXXX.restapi;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class RestApiApplication{
public static void main(String[] args)
{
SpringApplication.run(RestApiApplication.class, args);
}
}
and that is all that is in my restapi.java
I then have a controller
sendgridcontroller.java
import java.io.IOException;
#RestController
#RequestMapping(sendGridController.BASE_URL)
public class sendGridController
{
public static final String BASE_URL = "/api/v1/feedbacks";
private final sendGridAPI sendGridAPIService;
public sendGridController(sendGridAPIImpl sendGridAPIService)
{
this.sendGridAPIService = sendGridAPIService;
}
#CrossOrigin
#PostMapping("/sendfeedback")
public ResponseEntity<Boolean> sendFeedback(#RequestParam(name="fullname")String fullName,
#RequestParam(name="subject")String subject,
#RequestParam(name="email") String emailAddr,
#RequestParam(name="message")String message )
{
Boolean bReturn = true;
System.out.println("fullName ="+fullName);
System.out.println("subject ="+subject);
System.out.println("email ="+emailAddr);
System.out.println("message ="+message);
try
{
sendGridAPIService.sendFeedbackEmail(fullName, subject, emailAddr, message);
}
catch(Exception e)
{
return new ResponseEntity<>(false,HttpStatus.BAD_GATEWAY);
}
return new ResponseEntity<>(bReturn, HttpStatus.OK);
}
}
This is in its own package
Next i created a services package with two files
an Interface sendGridApi.java
package com.ea_optimised.restapi.services;
import org.springframework.stereotype.Service;
import java.io.IOException;
public interface sendGridAPI
{
void sendFeedbackEmail(String fullNameme, String subject, String emailAddr, String message) throws IOException;
}
// .... and Finally, I have an implementation of the interface
package com.XXXXX.restapi.services;
import com.sendgrid.*;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.net.UnknownHostException;
import java.rmi.ServerException;
#Service
public class sendGridAPIImpl implements sendGridAPI
{
// API_KEY
private static final String SENDGRID_API_KEY = "My key went here";
#Override
public void sendFeedbackEmail(String fullNameme, String subject, String emailAddr, String message) throws IOException
{
Email from = new Email(emailAddr);
Email to = new Email("info#ea-optimised.co.uk");
// Email to = new Email("test#example.com");
Content content = new Content("text/plain", message);
Mail mail = new Mail(from, subject, to, content);
SendGrid sg = new SendGrid(SENDGRID_API_KEY);
Request request = new Request();
try
{
request.setMethod(Method.POST);
request.setEndpoint("mail/send");
request.setBody(mail.build());
Response response = sg.api(request);
System.out.println("****\nSTATUS="+response.getStatusCode());
System.out.println("\n***Body :\n\n"+response.getBody());
System.out.println("***Headers:\n\n"+response.getHeaders());
}
catch ( UnknownHostException e )
{
System.out.println("***Unknown Host Exception:\n\n");
throw e;
}
catch (Exception ex)
{
System.out.println( "SendGrid Failed "+ex.getMessage());
try
{
throw ex;
}
catch (IOException e)
{
e.printStackTrace();
throw e;
}
}
}
}
Hope this helps
Finally are you using Gradle or Maven?

Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

I'm new to kafka. i'm trying to call kafka consumer through my spring boot and kafka-producer through my terminal.
The code worked fine previously but recently i keep getting this the log form consumer side
2019-04-22 11:56:39.415 INFO 10253 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version : 2.0.1
2019-04-22 11:56:39.416 INFO 10253 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : fa14705e51bd2ce5
2019-04-22 11:56:39.424 INFO 10253 --- [ main] org.apache.kafka.clients.Metadata : Cluster ID: iR8AmeB9RC-eqS7rrFyYGw
2019-04-22 11:56:39.425 INFO 10253 --- [ main] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-2, groupId=com.stellapps.rtc] Discovered group coordinator akash-Lenovo-ideapad-330-15IKB:9092 (id: 2147483647 rack: null)
2019-04-22 11:56:39.426 INFO 10253 --- [ main] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-2, groupId=com.stellapps.rtc] Revoking previously assigned partitions []
2019-04-22 11:56:39.427 INFO 10253 --- [ main] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-2, groupId=com.stellapps.rtc] (Re-)joining group
2019-04-22 11:56:42.164 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:56:45.183 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:56:48.201 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:56:51.218 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:56:54.235 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:56:57.256 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:57:00.276 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:57:03.294 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:57:06.315 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
2019-04-22 11:57:09.334 INFO 10253 --- [m.stellapps.rtc] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=com.stellapps.rtc] Attempt to heartbeat failed since group is rebalancing
and this from producer side
2019-04-22 11:57:47,755] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-04-22 12:00:38,119] INFO [GroupCoordinator 0]: Member consumer-1-c8b9d4fd-d2e3-45a3-8f9a-2e825a9a87bd in group com.stellapps.rtc has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-04-22 12:00:38,123] INFO [GroupCoordinator 0]: Stabilized group com.stellapps.rtc generation 11 (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator)
[2019-04-22 12:00:48,124] INFO [GroupCoordinator 0]: Member consumer-2-8f068bb6-d78c-458f-9775-d6b13ca54b57 in group com.stellapps.rtc has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-04-22 12:00:48,125] INFO [GroupCoordinator 0]: Preparing to rebalance group com.stellapps.rtc in state PreparingRebalance with old generation 11 (__consumer_offsets-30) (reason: removing member consumer-2-8f068bb6-d78c-458f-9775-d6b13ca54b57 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
[2019-04-22 12:00:48,125] INFO [GroupCoordinator 0]: Group com.stellapps.rtc with generation 12 is now empty (__consumer_offsets-30) (kafka.coordinator.group.GroupCoordinator)
[2019-04-22 12:07:47,755] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
the consumer does not accept any input from the producer. I tried changing the poll value but no vain , the message keeps coming or the consumer simply does not accept the input form producer.
here is my Spring boot code.
import com.stellapps.rtc.reset.RTCInterpreter;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.common.serialization.LongDeserializer;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import com.stellapps.rtc.reset.json.DataParser;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
import javax.annotation.PostConstruct;
#Component
public class DataConsumer {
#Autowired
private RTCInterpreter interpret;
private Consumer<Long, String> createConsumer(String topic) {
final String BOOTSTRAP_SERVERS = "localhost:9092";
final Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "com.stellapps.rtc"); // group Id of the consumer group (if a consumer
// group exists).
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, LongDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
// Create the consumer using props.
final Consumer<Long, String> consumer = new KafkaConsumer<>(props);
// Subscribe to the topic.
consumer.subscribe(Collections.singletonList(topic));
return consumer;
}
public void runConsumer(String topic) throws InterruptedException {
try {
final Consumer<Long, String> consumer = createConsumer(topic);
final int giveUp = 1000;
int noRecordsCount = 0;
while (true) {
final ConsumerRecords<Long, String> consumerRecords = consumer.poll(Duration.ofSeconds(1));
if (consumerRecords.count() == 0) {
noRecordsCount++;
if (noRecordsCount > giveUp)
break;
else
continue;
}
consumerRecords.forEach(record -> {
// System.out.printf(" %s\n", record.value().getClass().getName());
interpret.call(record.value());
});
consumer.commitAsync();
}
consumer.close();
System.out.println("Kafka is closed");
} catch (Exception e) {
e.printStackTrace();
}
}
#PostConstruct
public void init() {
try {
Runnable r = new Runnable() {
#Override
public void run() {
System.out.println(Thread.currentThread().getName());
try {
runConsumer("consume");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
Thread run = new Thread(r);
run.join();
run.start();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
please help me.

the problem was that i was calling Consumer from Application.java as well , hence i had 2 instance of Kafka consumer but i limited my group size as 1.
The producer problem was due to properties i set up for the producer.
sorry for the trouble.