Develop Reference

Spring Batch with multi - step Spring Cloud Task (PartitionHandler) for Remote Partition

Latest Update (with an image to hope simplify the problem) (thanks for feedback from #Mahmoud)
Relate issue reports for other reference (after this original post created, it seem someone filed issues for Spring Cloud on similar issue, so also update there too):
https://github.com/spring-cloud/spring-cloud-task/issues/793 relate to approach #1
https://github.com/spring-cloud/spring-cloud-task/issues/792 relate to approach #2
Also find a workaround resolution for that issue and update on that github issue, will update this once it is confirmed good by developer
https://github.com/spring-cloud/spring-cloud-task/issues/793#issuecomment-894617929
I am developing an application involved multi-steps using spring batch job but hit some roadblock. Did try to research doc and different attempts, but no success. So thought to check if community can shed light
Spring batch job 1 (received job parameter for setting for step 1/setting for step 2)
Step 1 -> remote partition (partitionhandler (cpu/memory for step 1 + grid) + partitioner) with setting from step1 (job configuration or step configuration)
Step 2 -> remote partition (partitionhandler (cpu/memory for step 2 + grid) + partitioner) with setting from step2 (job configuration or step configuration, and diff from step 1)
The reason we want is to have different step with different k8s setting (like cpu/memory/grid)
Attempts:
Create two partition handler (partitionHandlerReader + partitionHandlerProcessor) and their corresponding launcher (LauncherReader + LauncherProcessor)
Complete Project can be found in
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_1_two_partitionhandlers
The main class of configuration is try to simplify into one class
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/attempt_1_two_partitionhandlers/src/main/java/com/example/batchprocessing/BatchConfiguration.java
Use one PartitionerHandler + one TaskLauncher but with #StepScope for late binding for dynamic change base on step and job setup
Complete Project can be found in
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope
The main class of configuration is try to simplify into one class
https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/attempt_2_partitionhandler_with_stepscope/src/main/java/com/example/batchprocessing/BatchConfiguration.java
Both Result Following (full trace at above git repo):
During job trigger, it will error (it seem pass initial start up, but error during execution)
Because below will only occur when there are multiple PartitionHandler or when that Bean is at #StepScope or #JobScope
java.lang.NullPointerException: null
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:347) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
Full Log
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.4.6)
2021-08-06 11:24:29.242 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : Starting BatchProcessingApplication v0.0.1-SNAPSHOT using Java 11.0.7 on localhost.localdomain with PID 90294 (/home/danilko/IdeaProjects/partition/target/batchprocessing-0.0.1-SNAPSHOT.jar started by danilko in /home/danilko/IdeaProjects/partition)
2021-08-06 11:24:29.244 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : The following profiles are active: controller
2021-08-06 11:24:29.790 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2021-08-06 11:24:29.794 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2021-08-06 11:24:29.797 INFO 90294 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2021-08-06 11:24:29.833 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.947 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.947 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.959 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$83e6c2be] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:29.968 INFO 90294 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration' of type [org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration$$EnhancerBySpringCGLIB$$cc3cccc1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-08-06 11:24:30.093 INFO 90294 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-08-06 11:24:30.160 INFO 90294 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2021-08-06 11:24:30.724 INFO 90294 --- [ main] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: MYSQL
2021-08-06 11:24:30.736 INFO 90294 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'application.errorChannel' has 1 subscriber(s).
2021-08-06 11:24:30.897 INFO 90294 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started bean '_org.springframework.integration.errorLogger'
2021-08-06 11:24:30.974 INFO 90294 --- [ main] c.e.b.BatchProcessingApplication : Started BatchProcessingApplication in 2.024 seconds (JVM running for 2.366)
2021-08-06 11:24:30.975 INFO 90294 --- [ main] o.s.b.a.b.JobLauncherApplicationRunner : Running default command line with: []
2021-08-06 11:24:31.010 INFO 90294 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=partitionedJob-1538890488]] launched with the following parameters: [{}]
Set readerGridSize == 1
2021-08-06 11:24:31.020 INFO 90294 --- [ main] o.s.c.t.b.l.TaskBatchExecutionListener : The job execution id 22 was run within the task execution 54
2021-08-06 11:24:31.046 INFO 90294 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [partitionReaderStep]
2021-08-06 11:24:31.101 ERROR 90294 --- [ main] o.s.batch.core.step.AbstractStep : Encountered an error executing step partitionReaderStep in job partitionedJob-1538890488
java.lang.NullPointerException: null
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorker(DeployerPartitionHandler.java:347) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.launchWorkers(DeployerPartitionHandler.java:313) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at org.springframework.cloud.task.batch.partition.DeployerPartitionHandler.handle(DeployerPartitionHandler.java:302) ~[spring-cloud-task-batch-2.3.1-SNAPSHOT.jar!/:2.3.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.7.jar!/:5.3.7]
at com.sun.proxy.$Proxy65.handle(Unknown Source) ~[na:na]
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:106) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:152) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:413) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:136) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:320) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:149) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) ~[spring-core-5.3.7.jar!/:5.3.7]
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:140) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.7.jar!/:5.3.7]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.7.jar!/:5.3.7]
at com.sun.proxy.$Proxy51.run(Unknown Source) ~[na:na]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:799) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:789) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:346) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) ~[spring-boot-2.4.6.jar!/:2.4.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) ~[spring-boot-2.4.6.jar!/:2.4.6]
at com.example.batchprocessing.BatchProcessingApplication.main(BatchProcessingApplication.java:10) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[batchprocessing-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Study/Reference:
Most tutorial I found online only involved one partition step.
https://dataflow.spring.io/docs/feature-guides/batch/partitioning/
Thanks for info/helps in advance

Is above even possible setup?
yes, nothing prevents you from having two partitioned steps in a single Spring Batch job.
Is it possible to use JobScope/StepScope to pass info to the partitionhandler
yes, it is possible for the partition handler to be declared as a job/step scoped bean if it needs the late-binding feature to be configured.
Updated on 08/14/2021 by #DanilKo
The original answer is correct in high - level. However, to actually achieve the partition handeler to be step scoped, a code modification is required
Below is the analyze + my proposed workaround/fix (maybe eventually code maintainer will have better way to make it work, but so far below fix is working for me)
Issue being continued to discuss at:
https://github.com/spring-cloud/spring-cloud-task/issues/793 (multiple partitioner handler discussion)
https://github.com/spring-cloud/spring-cloud-task/issues/792
(which this fix is based up to use partitionerhandler at step scope to configure different worker steps + resources + max worker)
Root cause analyze (hypothesis)
The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task) or there are two partitionHandler, that setup is no longer triggered, as #EnableTask seem not able to locate one partitionhandler during creation.
https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-batch/src/main/java/org/springframework/cloud/task/batch/partition/DeployerPartitionHandler.java # 269
Resulted created DeployerHandler faced a null with taskExecution when trying to launch (as it is never setup)
https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-batch/src/main/java/org/springframework/cloud/task/batch/partition/DeployerPartitionHandler.java # 347
Workaround Resolution
Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
It seem to work, but still not clear if there is other side effect (so far during test not found any)
Full code can be found in https://github.com/danilko/spring-batch-remote-k8s-paritition-example/tree/attempt_2_partitionhandler_with_stepscope_workaround_resolution
In the partitionHandler method
#Bean
#StepScope
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
JobExplorer jobExplorer,
#Value("#{stepExecution}") StepExecution stepExecution) throws Exception {
...
// After the declaration of partitionhandler
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerStep")
, taskRepository);
// Issue https://github.com/spring-cloud/spring-cloud-task/issues/793
// Perform the setting of execution as this partitioner now not created at task level so #beforetask is no longer vaild
// The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
// But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task), that setup is no longer triggered
// Resulted created DeployerHandler faced a null
// Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
// From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
// It seem to work, but still not clear if there is other side effect (so far during test not found any)
long executionId = taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId());
System.out.println("Current execution job to task execution id " + executionId);
TaskExecution taskExecution = taskExplorer.getTaskExecution(taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId()));
System.out.println("Current execution job to task execution is not null: " + (taskExecution != null));
partitionHandler.beforeTask(taskExecution);
...
// rest of code continue
(note it utilize stepExecution context to find out the current trigger step name and therefore assign different worker step)
Worker name in this case is coming from pre-defined job execution, but may able to come from jobparameter or another place too)
That job context is populated with job listner
Job is configured with job listener
#Bean(name = "partitionedJob")
#Profile("!worker")
public Job partitionedJob()throws Exception {
Random random = new Random();
return jobBuilderFactory.get("partitionedJob" + random.nextInt())
.start(partitionReaderStep())
.listener(jobExecutionListener())
.next(partitionProcessorStep())
.build();
}
In job listener populated it
#Bean
public JobExecutionListener jobExecutionListener() {
JobExecutionListener listener = new JobExecutionListener(){
#Override
public void beforeJob(JobExecution jobExecution)
{
jobExecution.getExecutionContext().putString("readerCPURequest", "1");
jobExecution.getExecutionContext().putString("readerCPULimit", "2");
jobExecution.getExecutionContext().putString("readerWorkerGridSize", "1");
// For now using same image for reader/processor, but if it work, can split them
jobExecution.getExecutionContext().putString("readerWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("readerWorkerStep", "workerStepReader");
jobExecution.getExecutionContext().putString("processorCPURequest", "3");
jobExecution.getExecutionContext().putString("processorCPULimit", "4");
jobExecution.getExecutionContext().putString("processorWorkerGridSize", "2");
// For now using same image for reader/processor, but if it work, will split them
jobExecution.getExecutionContext().putString("processorWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("processorWorkerStep", "workerStepProcessor");
System.out.println("Set readerGridSize == " + jobExecution.getExecutionContext().getString("readerGridSize", "IT IS NULL WHICH IS INCORRECT"));
}
#Override
public void afterJob(JobExecution jobExecution) {
}
};
return listener;
}
Full code (can also be found in my code github after the workaround fix is being applied): https://github.com/danilko/spring-batch-remote-k8s-paritition-example/blob/main/src/main/java/com/example/batchprocessing/BatchConfiguration.java
package com.example.batchprocessing;
import io.fabric8.kubernetes.api.model.DeletionPropagation;
import io.fabric8.kubernetes.api.model.batch.JobList;
import io.fabric8.kubernetes.api.model.batch.JobSpec;
import io.fabric8.kubernetes.api.model.batch.JobStatus;
import io.fabric8.kubernetes.client.KubernetesClient;
import org.springframework.batch.core.*;
import org.springframework.batch.core.configuration.JobRegistry;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.explore.JobExplorer;
import org.springframework.batch.core.partition.PartitionHandler;
import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.cloud.deployer.resource.docker.DockerResource;
import org.springframework.cloud.deployer.resource.support.DelegatingResourceLoader;
import org.springframework.cloud.deployer.spi.kubernetes.*;
import org.springframework.cloud.deployer.spi.task.TaskLauncher;
import org.springframework.cloud.task.batch.partition.*;
import org.springframework.cloud.task.configuration.EnableTask;
import org.springframework.cloud.task.repository.TaskExecution;
import org.springframework.cloud.task.repository.TaskExplorer;
import org.springframework.cloud.task.repository.TaskRepository;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.env.Environment;
import org.springframework.core.env.SystemEnvironmentPropertySource;
import org.springframework.core.io.Resource;
import org.springframework.core.task.TaskExecutor;
import org.springframework.core.task.TaskRejectedException;
import org.springframework.util.StringUtils;
import java.util.*;
#Configuration
#EnableBatchProcessing
#EnableTask
public class BatchConfiguration {
private static int BACK_OFF_LIMIT = 6;
// Set the kuberentes job name
private String taskName_prefix="partitionedbatchjob";
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public JobExplorer jobExplorer;
#Autowired
public JobRepository jobRepository;
#Autowired
public TaskExecutor taskExecutor;
#Autowired
public TaskRepository taskRepository;
#Autowired
public TaskExplorer taskExplorer;
#Autowired
private ConfigurableApplicationContext context;
#Autowired
private DelegatingResourceLoader resourceLoader;
#Autowired
private Environment environment;
#Bean
#StepScope
public Partitioner partitioner( #Value("#{stepExecution}") StepExecution stepExecution) {
return new Partitioner() {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> partitions = new HashMap<>(gridSize);
int targetGridSize = 0;
String step = "";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep"))
{
step = "reader";
}
else
{
step = "processor";
}
targetGridSize = Integer.parseInt(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerGridSize"));
for (int i = 0; i < targetGridSize; i++) {
ExecutionContext context1 = new ExecutionContext();
context1.put("partitionNumber", i);
partitions.put("partition" + i, context1);
}
return partitions;
}
};
}
#Bean
public KubernetesClient kuberentesClient()
{
KubernetesDeployerProperties kubernetesDeployerProperties = new KubernetesDeployerProperties();
return KubernetesClientFactory.getKubernetesClient(kubernetesDeployerProperties);
}
#Bean
#StepScope
public TaskLauncher taskLauncher( #Value("#{stepExecution}") StepExecution stepExecution)
{
KubernetesDeployerProperties kubernetesDeployerProperties = new KubernetesDeployerProperties();
kubernetesDeployerProperties.setNamespace("default");
kubernetesDeployerProperties.setCreateJob(true);
// Database setup to reference configmap for database info
List<KubernetesDeployerProperties.ConfigMapKeyRef> configMapKeyRefList = new ArrayList<KubernetesDeployerProperties.ConfigMapKeyRef>();
KubernetesDeployerProperties.ConfigMapKeyRef configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_URL");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_URL");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_USERNAME");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_USERNAME");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_PASSWORD");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_PASSWORD");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_DATASOURCE_DRIVERCLASSNAME");
configMapKeyRef.setEnvVarName("SPRING_DATASOURCE_DRIVERCLASSNAME");
configMapKeyRefList.add(configMapKeyRef);
configMapKeyRef = new KubernetesDeployerProperties.ConfigMapKeyRef();
configMapKeyRef.setConfigMapName("mariadb");
configMapKeyRef.setDataKey("SPRING_PROFILES_ACTIVE");
configMapKeyRef.setEnvVarName("SPRING_PROFILES_ACTIVE");
configMapKeyRefList.add(configMapKeyRef);
kubernetesDeployerProperties.setConfigMapKeyRefs(configMapKeyRefList);
// Set request resource
KubernetesDeployerProperties.RequestsResources request = new KubernetesDeployerProperties.RequestsResources();
KubernetesDeployerProperties.LimitsResources limit = new KubernetesDeployerProperties.LimitsResources();
String step = "";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep"))
{
step="reader";
}
else
{
step="processor";
}
request.setCpu(stepExecution.getJobExecution().getExecutionContext().getString(step + "CPURequest"));
request.setMemory("2000Mi");
limit.setCpu(stepExecution.getJobExecution().getExecutionContext().getString(step +"CPULimit"));
limit.setMemory("3000Mi");
kubernetesDeployerProperties.setRequests(request);
kubernetesDeployerProperties.setLimits(limit);
// as build on local image, so need to use local
kubernetesDeployerProperties.setImagePullPolicy(ImagePullPolicy.IfNotPresent);
// Set task launcher properties to not repeat and not restart
KubernetesTaskLauncherProperties kubernetesTaskLauncherProperties = new KubernetesTaskLauncherProperties();
// https://kubernetes.io/docs/concepts/workloads/controllers/job/
// Set to never to create new pod on restart
kubernetesTaskLauncherProperties.setBackoffLimit(BACK_OFF_LIMIT);
kubernetesTaskLauncherProperties.setRestartPolicy(RestartPolicy.Never);
KubernetesTaskLauncher kubernetesTaskLauncher = new KubernetesTaskLauncher(kubernetesDeployerProperties,
kubernetesTaskLauncherProperties, kuberentesClient());
return kubernetesTaskLauncher;
}
#Bean(name = "partitionedJob")
#Profile("!worker")
public Job partitionedJob()throws Exception {
Random random = new Random();
return jobBuilderFactory.get("partitionedJob" + random.nextInt())
.start(partitionReaderStep())
.listener(jobExecutionListener())
.next(partitionProcessorStep())
.build();
}
#Bean(name = "partitionReaderStep")
public Step partitionReaderStep() throws Exception {
return stepBuilderFactory.get("partitionReaderStep")
.partitioner(workerStepReader().getName(), partitioner( null))
.step(workerStepReader())
.partitionHandler(partitionHandler(
taskLauncher( null),
jobExplorer, null))
.build();
}
#Bean(name = "partitionProcessorStep")
public Step partitionProcessorStep() throws Exception {
return stepBuilderFactory.get("partitionProcessorStep")
.partitioner(workerStepProcessor().getName(), partitioner( null))
.step(workerStepProcessor())
.partitionHandler(partitionHandler(
taskLauncher( null),
jobExplorer, null))
.build();
}
#Bean
#StepScope
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
JobExplorer jobExplorer,
#Value("#{stepExecution}") StepExecution stepExecution) throws Exception {
String step ="processor";
if(stepExecution.getStepName().equalsIgnoreCase("partitionReaderStep")) {
step = "reader";
}
// Use local build image
DockerResource resource = new DockerResource(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerImage"));
DeployerPartitionHandler partitionHandler =
new DeployerPartitionHandler(taskLauncher, jobExplorer, resource,
stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerStep")
, taskRepository);
// Issue https://github.com/spring-cloud/spring-cloud-task/issues/793
// Perform the setting of execution as this partitioner now not created at task level so #beforetask is no longer vaild
// The problem is DeployerPartitionHandler utilize annoation #BeforeTask to force task to pass in TaskExecution object as part of Task setup
// But as this partionerHandler is now at #StepScope (instead of directly at #Bean level with #Enable Task), that setup is no longer triggered
// Resulted created DeployerHandler faced a null
// Below is essentially a workaround to use the current job execution id to retrieve the associated task execution id
// From there, got that task execution and passed to deploy handler to fulfill its need of taskExecution reference
// It seem to work, but still not clear if there is other side effect (so far during test not found any)
long executionId = taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId());
System.out.println("Current execution job to task execution id " + executionId);
TaskExecution taskExecution = taskExplorer.getTaskExecution(taskExplorer.getTaskExecutionIdByJobExecutionId(stepExecution.getJobExecutionId()));
System.out.println("Current execution job to task execution is not null: " + (taskExecution != null));
partitionHandler.beforeTask(taskExecution);
List<String> commandLineArgs = new ArrayList<>(3);
commandLineArgs.add("--spring.profiles.active=worker");
commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
commandLineArgs.add("--spring.batch.initializer.enabled=false");
partitionHandler
.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
partitionHandler.setEnvironmentVariablesProvider(new NoOpEnvironmentVariablesProvider());
partitionHandler.setMaxWorkers(Integer.parseInt(stepExecution.getJobExecution().getExecutionContext().getString(step + "WorkerGridSize")));
partitionHandler.setApplicationName(taskName_prefix + step);
return partitionHandler;
}
#Bean
public JobExecutionListener jobExecutionListener() {
JobExecutionListener listener = new JobExecutionListener(){
#Override
public void beforeJob(JobExecution jobExecution)
{
jobExecution.getExecutionContext().putString("readerCPURequest", "1");
jobExecution.getExecutionContext().putString("readerCPULimit", "2");
jobExecution.getExecutionContext().putString("readerWorkerGridSize", "1");
// For now using same image for reader/processor, but if it work, can split them
jobExecution.getExecutionContext().putString("readerWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("readerWorkerStep", "workerStepReader");
jobExecution.getExecutionContext().putString("processorCPURequest", "3");
jobExecution.getExecutionContext().putString("processorCPULimit", "4");
jobExecution.getExecutionContext().putString("processorWorkerGridSize", "2");
// For now using same image for reader/processor, but if it work, will split them
jobExecution.getExecutionContext().putString("processorWorkerImage", "worker:latest");
jobExecution.getExecutionContext().putString("processorWorkerStep", "workerStepProcessor");
System.out.println("Set readerGridSize == " + jobExecution.getExecutionContext().getString("readerGridSize", "IT IS NULL WHICH IS INCORRECT"));
}
#Override
public void afterJob(JobExecution jobExecution) {
}
};
return listener;
}
#Bean
#Profile("worker")
public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) {
return new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository);
}
#Bean(name = "workerStepReader")
public Step workerStepReader() {
return this.stepBuilderFactory.get("workerStepReader")
.tasklet(workerTaskletReader(null))
.build();
}
#Bean(name = "workerStepProcessor")
public Step workerStepProcessor() {
return this.stepBuilderFactory.get("workerStepProcessor")
.tasklet(workerTaskletProcessor(null))
.build();
}
#Bean
#StepScope
public Tasklet workerTaskletReader(
final #Value("#{stepExecution}") StepExecution stepExecution) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
Integer partitionNumber = stepExecution.getExecutionContext().getInt("partitionNumber");
System.out.println("This workerTaskletReader ran partition: " + partitionNumber);
return RepeatStatus.FINISHED;
}
};
}
#Bean
#StepScope
public Tasklet workerTaskletProcessor(
final #Value("#{stepExecution}") StepExecution stepExecution) {
return new Tasklet() {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
Integer partitionNumber = stepExecution.getExecutionContext().getInt("partitionNumber");
System.out.println("This workerTaskletProcessor ran partition: " + partitionNumber);
return RepeatStatus.FINISHED;
}
};
}
}

Error when trying to use kafka-client 2.6

I'm learning Kafka and when I was trying to implement a simple producer that will only send "Hello World"
package com.github.joe.kafka.tutorial1;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class ProducerDemo {
public static void main(String[] args) {
Properties properties = new Properties();
String bootstrapServers = "127.0.0.1:9092";
String topic = "first_topic";
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);
ProducerRecord<String,String> record = new ProducerRecord<String, String>(topic,"Hello World");
producer.send(record);
producer.flush();
producer.close();
}
}
The problem is when I'm using kafka-client v2.0.0 it works, but it fails when I'm using kafka-client 2.6.0 and I get the following error:
[kafka-producer-network-thread | producer-1] ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/JsonNode
at org.apache.kafka.common.requests.ApiVersionsRequest$Builder.<clinit>(ApiVersionsRequest.java:36)
at org.apache.kafka.clients.NetworkClient.handleConnections(NetworkClient.java:910)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:555)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 6 more
[main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.

as it turns out , for Kafka-client 2.6.0 there's an additional dependency:
Jackson Databind
this was not required for version 2.0.0

The above suggestion worked for me. I added the below dependency which resolved the problem.
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.8</version>
</dependency>

Spring Kafka. Not started EmbeddedKafkaBroker

I am coding Kafka Broker and Consumer to catch messages from the application. When trying to get messages from Consumer, an error occurs
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:216)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:531)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:444)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1267)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1231)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
at org.springframework.kafka.test.utils.KafkaTestUtils.getRecords(KafkaTestUtils.java:303)
at org.springframework.kafka.test.utils.KafkaTestUtils.getRecords(KafkaTestUtils.java:280)
On the application side (Producer), there is also a connection error
2020-03-25 12:29:33.689 WARN 25786 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1, transactionalId=tx0] Connection to node -1 (<here broker hostname>:9092) could not be established. Broker may not be available.
My project has the following dependencies:
compile "org.springframework.kafka:spring-kafka-test:2.4.4.RELEASE"
compile "org.springframework.kafka:spring-kafka:2.4.4.RELEASE"
Code of My Kafka Broker
public class KafkaServer {
private static final String BROKERPORT = "9092";
private static final String BROKERHOST = "localhost";
public static final String TOPIC1 = "fss-fsstransdata";
public static final String TOPIC2 = "fss-fsstransscores";
public static final String TOPIC3 = "fss-fsstranstimings";
public static final String TOPIC4 = "fss-fssdevicedata";
#Getter
private Consumer<String, String> consumer;
private EmbeddedKafkaBroker embeddedKafkaBroker;
public void run() {
String[] topics = {TOPIC1, TOPIC2, TOPIC3, TOPIC4};
this.embeddedKafkaBroker = new EmbeddedKafkaBroker(
1,
false,
1,
topics
).kafkaPorts(BROKERPORT);
Map<String, Object> configs = new HashMap<>(KafkaTestUtils.consumerProps("consumer", "false", this.embeddedKafkaBroker));
this.consumer = new DefaultKafkaConsumerFactory<>(configs, new StringDeserializer(), new StringDeserializer()).createConsumer();
this.consumer.subscribe(Arrays.asList(topics));
}
}
Please help to deal with the situation. I am not good at kafka architecture and how it can be implemented on Spring.

The EmbeddedKafkaBroker is designed to be used from a Spring application context or by a JUnit4 #Rule or #ClassRule or by a JUnit5 Condition.
To use it outside those environments, you must call afterPropertiesSet() to initialize it and destroy() to shut it down.

If you are using spring then you need to annotate your bean with #EmbeddedKafka and then use #Autowire on EmbeddedKafkaBroker
Example embeded kafka annotation configuration:
#EmbeddedKafka(
partitions = 1,
controlledShutdown = false,
brokerProperties = {// place your proerties here
})
What I would do is to create a spring bean KafkaServerConfig and place all my logic for configuration and bean construction inside.
PS: it should be noted that EmbeddedKafkaBroker is intended for unit tests.

InvalidServerSideConfigurationException when creating cache using XML

I'm new with terracotta. I want to create a clustered server cache but found some difficulties with configuration files.
Here is my tc-config-terracotta.xml file (with which I launch terracotta server)
<?xml version="1.0" encoding="UTF-8"?>
<tc-config xmlns="http://www.terracotta.org/config"
xmlns:ohr="http://www.terracotta.org/config/offheap-resource">
<servers>
<server host="localhost" name="clustered">
<logs>/path/log/terracotta/server-logs</logs>
</server>
</servers>
<plugins>
<config>
<ohr:offheap-resources>
<ohr:resource name="primary-server-resource" unit="MB">128
</ohr:resource>
<ohr:resource name="secondary-server-resource" unit="MB">96
</ohr:resource>
</ohr:offheap-resources>
</config>
</plugins>
</tc-config>
I used the ehcache-clustered-3.3.1-kit to launch the server.
$myPrompt/some/dir/with/ehcache/clustered/server/bin>./start-tc-server.sh -f /path/to/conf/tc-config-terracotta.xml
No problem for the server to start
2017-06-01 11:29:14,052 INFO - New logging session started.
2017-06-01 11:29:14,066 INFO - Terracotta 5.2.2, as of 2017-03-29 at 15:26:20 PDT (Revision 397a456cfe4b8188dfe8b017a5c14346f79c2fcf from UNKNOWN)
2017-06-01 11:29:14,067 INFO - PID is 6114
2017-06-01 11:29:14,697 INFO - Successfully loaded base configuration from file at '/path/to/conf/tc-config-terracotta.xml'.
2017-06-01 11:29:14,757 INFO - Available Max Runtime Memory: 1822MB
2017-06-01 11:29:14,836 INFO - Log file: '/path/log/terracotta/server-logs/terracotta-server.log'.
2017-06-01 11:29:15,112 INFO - Becoming State[ ACTIVE-COORDINATOR ]
2017-06-01 11:29:15,129 INFO - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9510 successfully, and is now ready for work.
Here is the ehcache-terracotta.xml configuration file
<ehcache:config xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
xmlns:terracotta='http://www.ehcache.org/v3/clustered'
xmlns:ehcache='http://www.ehcache.org/v3'
xsi:schemaLocation="http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.3.xsd
http://www.ehcache.org/v3/clustered http://www.ehcache.org/schema/ehcache-clustered-ext-3.3.xsd">
<ehcache:service>
<terracotta:cluster>
<terracotta:connection url="terracotta://localhost:9510/clustered" />
<terracotta:server-side-config
auto-create="true">
<terracotta:default-resource from="primary-server-resource" />
</terracotta:server-side-config>
</terracotta:cluster>
</ehcache:service>
<ehcache:cache alias="myTest">
<ehcache:key-type>java.lang.String</ehcache:key-type>
<ehcache:value-type>java.lang.String</ehcache:value-type>
<ehcache:resources>
<terracotta:clustered-dedicated unit="MB">10
</terracotta:clustered-dedicated>
</ehcache:resources>
<terracotta:clustered-store consistency="strong" />
</ehcache:cache>
</ehcache:config>
I have a class to test the conf:
import java.net.URL;
import org.ehcache.Cache;
import org.ehcache.CacheManager;
import org.ehcache.config.Configuration;
import org.ehcache.config.builders.CacheManagerBuilder;
import org.ehcache.xml.XmlConfiguration;
public class TestTerracottaCacheManager
{
private static TestTerracottaCacheManager cacheManager = null;
private CacheManager cm;
private Cache<Object, Object> cache;
private static final String DEFAULT_CACHE_NAME = "myTest";
private String cacheName;
public static TestTerracottaCacheManager getInstance()
{
if (cacheManager == null)
{
cacheManager = new TestTerracottaCacheManager();
}
return cacheManager;
}
private TestTerracottaCacheManager()
{
// 1. Create a cache manager
final URL url =
TestTerracottaCacheManager.class.getResource("/ehcache-terracotta.xml");
System.out.println(url);
Configuration xmlConfig = new XmlConfiguration(url);
cm = CacheManagerBuilder.newCacheManager(xmlConfig);
cm.init();
intializeCache();
}
private void intializeCache()
{
// 2. Get a cache called "cache1", declared in ehcache.xml
cache = cm.getCache(cacheName == null ? DEFAULT_CACHE_NAME : cacheName,
Object.class, Object.class);
if (cache == null)
{
throw new NullPointerException();
}
}
public void put(Object key, Object value)
{
cache.put(key, value);
}
public Object get(String key)
{
// 5. Print out the element
Object ele = cache.get(key);
return ele;
}
public boolean isKeyInCache(Object key)
{
return cache.containsKey(key);
}
public void closeCache()
{
// 7. shut down the cache manager
cm.close();
}
public static void main(String[] args)
{
TestTerracottaCacheManager testCache = TestTerracottaCacheManager.getInstance();
testCache.put("titi", "1");
System.out.println(testCache.get("titi"));
testCache.closeCache();
}
public String getCacheName()
{
return cacheName;
}
public void setCacheName(String cacheName)
{
this.cacheName = cacheName;
}
}
I've got an exception. Here it's the stack trace:
14:18:38.978 [main] ERROR org.ehcache.core.EhcacheManager - Initialize failed.
Exception in thread "main" org.ehcache.StateTransitionException: Unable to validate cluster tier manager for id clustered
at org.ehcache.core.StatusTransitioner$Transition.failed(StatusTransitioner.java:235)
at org.ehcache.core.EhcacheManager.init(EhcacheManager.java:587)
at fr.test.cache.TestTerracottaCacheManager.<init>(TestTerracottaCacheManager.java:41)
at fr.test.cache.TestTerracottaCacheManager.getInstance(TestTerracottaCacheManager.java:28)
at fr.test.cache.TestTerracottaCacheManager.main(TestTerracottaCacheManager.java:81)
Caused by: org.ehcache.clustered.client.internal.ClusterTierManagerValidationException: Unable to validate cluster tier manager for id clusteredENS
at org.ehcache.clustered.client.internal.ClusterTierManagerClientEntityFactory.retrieve(ClusterTierManagerClientEntityFactory.java:196)
at org.ehcache.clustered.client.internal.service.DefaultClusteringService.autoCreateEntity(DefaultClusteringService.java:215)
at org.ehcache.clustered.client.internal.service.DefaultClusteringService.start(DefaultClusteringService.java:148)
at org.ehcache.core.internal.service.ServiceLocator.startAllServices(ServiceLocator.java:118)
at org.ehcache.core.EhcacheManager.init(EhcacheManager.java:559)
... 3 more
Caused by: org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException: Default resource not aligned. Client: primary-server-resource Server: null
at org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException.withClientStackTrace(InvalidServerSideConfigurationException.java:43)
at org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException.withClientStackTrace(InvalidServerSideConfigurationException.java:22)
at org.ehcache.clustered.common.internal.messages.ResponseCodec.decode(ResponseCodec.java:197)
at org.ehcache.clustered.common.internal.messages.EhcacheCodec.decodeResponse(EhcacheCodec.java:110)
at org.ehcache.clustered.common.internal.messages.EhcacheCodec.decodeResponse(EhcacheCodec.java:37)
at com.tc.object.EntityClientEndpointImpl$InvocationBuilderImpl$1.getWithTimeout(EntityClientEndpointImpl.java:193)
at com.tc.object.EntityClientEndpointImpl$InvocationBuilderImpl$1.getWithTimeout(EntityClientEndpointImpl.java:175)
at org.ehcache.clustered.client.internal.SimpleClusterTierManagerClientEntity.waitFor(SimpleClusterTierManagerClientEntity.java:184)
at org.ehcache.clustered.client.internal.SimpleClusterTierManagerClientEntity.invokeInternal(SimpleClusterTierManagerClientEntity.java:148)
at org.ehcache.clustered.client.internal.SimpleClusterTierManagerClientEntity.validate(SimpleClusterTierManagerClientEntity.java:120)
at org.ehcache.clustered.client.internal.ClusterTierManagerClientEntityFactory.retrieve(ClusterTierManagerClientEntityFactory.java:190)
... 7 more
Caused by: org.ehcache.clustered.common.internal.exceptions.InvalidServerSideConfigurationException: Default resource not aligned. Client: primary-server-resource Server: null
at org.ehcache.clustered.server.EhcacheStateServiceImpl.checkConfigurationCompatibility(EhcacheStateServiceImpl.java:207)
at org.ehcache.clustered.server.EhcacheStateServiceImpl.validate(EhcacheStateServiceImpl.java:194)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.validate(ClusterTierManagerActiveEntity.java:253)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.invokeLifeCycleOperation(ClusterTierManagerActiveEntity.java:203)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.invoke(ClusterTierManagerActiveEntity.java:147)
at org.ehcache.clustered.server.ClusterTierManagerActiveEntity.invoke(ClusterTierManagerActiveEntity.java:57)
at com.tc.objectserver.entity.ManagedEntityImpl.performAction(ManagedEntityImpl.java:741)
at com.tc.objectserver.entity.ManagedEntityImpl.invoke(ManagedEntityImpl.java:488)
at com.tc.objectserver.entity.ManagedEntityImpl.lambda$processInvokeRequest$2(ManagedEntityImpl.java:319)
at com.tc.objectserver.entity.ManagedEntityImpl$SchedulingRunnable.run(ManagedEntityImpl.java:1048)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.invoke(RequestProcessor.java:170)
at com.tc.objectserver.entity.RequestProcessor$EntityRequest.run(RequestProcessor.java:161)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:27)
at com.tc.objectserver.entity.RequestProcessorHandler.handleEvent(RequestProcessorHandler.java:23)
at com.tc.async.impl.StageQueueImpl$HandledContext.runWithHandler(StageQueueImpl.java:502)
at com.tc.async.impl.StageImpl$WorkerThread.run(StageImpl.java:192)
I think it's a problem in the XML files, but I'm not sure. Someone can help please?
Thanks

What the exception tells you is that the configuration of the clustered bits of your cache manager and cache differ between what the cluster knows and what the client ask.
The most likely explanation is that you ran your client code once with a different config, realised there was an issue or just wanted to change something. And then tried to run the client without destroying the cahche manager on the cluster or restarting the server.
You simply need to restart your server, to lose all clustered state since you want a different setup.

I've tried to reproduce your issue in my IDE, copying / pasting your 3 files.
I found an error in intializeCache():
cache = cm.getCache(cacheName == null ? DEFAULT_CACHE_NAME : cacheName,
Object.class, Object.class);
triggered a :
Exception in thread "main" java.lang.IllegalArgumentException: Cache 'myTest' type is <java.lang.String, java.lang.String>, but you retrieved it with <java.lang.Object, java.lang.Object>
at org.ehcache.core.EhcacheManager.getCache(EhcacheManager.java:162)
at MyXmlClient.intializeCache(MyXmlClient.java:48)
So please make sure that your xml configuration matches your Java code : you used <String, String> in XML, use <String, String> in your java code :
cache = cm.getCache(cacheName == null ? DEFAULT_CACHE_NAME : cacheName,
String.class, String.class);
Everything else worked fine !
INFO --- [8148202b7ba8914] customer.logger.tsa : Connection successfully established to server at 127.0.0.1:9510
INFO --- [ main] org.ehcache.core.EhcacheManager : Cache 'myTest' created in EhcacheManager.
1
INFO --- [ main] org.ehcache.core.EhcacheManager : Cache 'myTest' removed from EhcacheManager.
INFO --- [ main] o.e.c.c.i.s.DefaultClusteringService : Closing connection to cluster terracotta://localhost:9510
The error you gave is coming form a mismatch between the terracotta server offheap resource use in your client and the terracotta server offheap configuration ; make sure they match ! (copying / pasting your example they did !)

#AnthonyDahanne I am using ehcache-clustered-3.8.1-kit to launch the server. I have ehcache.xml my spring boot application is automatically picking my ehache.xml so I am not explicitly writing cacheManager.
<ehcache:config
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
xmlns:terracotta='http://www.ehcache.org/v3/clustered'
xmlns:ehcache='http://www.ehcache.org/v3'
xsi:schemaLocation="http://www.ehcache.org/v3 http://www.ehcache.org/schema/ehcache-core-3.8.xsd
http://www.ehcache.org/v3/clustered http://www.ehcache.org/schema/ehcache-clustered-ext-3.8.xsd">
<ehcache:service>
<terracotta:cluster>
<terracotta:connection url="terracotta://localhost:9410/clustered"/>
<terracotta:server-side-config auto-create="true">
<!--<terracotta:default-resource from="default-resource"/>-->
<terracotta:shared-pool name="shared-pool-expense" unit="MB">100</terracotta:shared-pool>
</terracotta:server-side-config>
</terracotta:cluster>
</ehcache:service>
<ehcache:cache alias="areaOfCircleCache">
<ehcache:key-type>java.lang.String</ehcache:key-type>
<ehcache:value-type>com.db.entity.LogMessage</ehcache:value-type>
<ehcache:resources>
<!-- <ehcache:heap unit="entries">100</ehcache:heap>
<ehcache:offheap unit="MB">10</ehcache:offheap>-->
<terracotta:clustered-dedicated unit="MB">10</terracotta:clustered-dedicated>
</ehcache:resources>
</ehcache:cache>
</ehcache:config>

Error when execute MavenCli in the loop (maven-embedder)?

What is the problem when I execute the maven command in the loop ? The goal is to update the version of pom.xml of the list of bundles. The first iteration, maven execute correctly (update pom.xml), but it makes error for all item after.
for (String bundlePath: bundlesToUpdate)
{
MavenCli cli = new MavenCli();
String[] arguments = {
"-Dtycho.mode=maven",
"org.eclipse.tycho:tycho-versions-plugin:set-version",
"-DgenerateBackupPoms=false",
"-DnewVersion=" + version};
int result = cli.doMain(arguments, bundlePath, System.out, System.err);
}
Same error with the code:
`MavenCli cli = new MavenCli();
for (String bundlePath: bundlesToUpdate)
{
String[] arguments = {
"-Dtycho.mode=maven",
"org.eclipse.tycho:tycho-versions-plugin:set-version",
"-DgenerateBackupPoms=false",
"-DnewVersion=" + version};
int result = cli.doMain(arguments, bundlePath, System.out, System.err);
}`
First time, it's ok:
[main] INFO org.eclipse.tycho.versions.manipulation.PomManipulator - pom.xml//project/version: 2.2.6-SNAPSHOT => 2.2.7-SNAPSHOT
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - ------------------------------------------------------------------------
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - Reactor Summary:
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger -
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - XXXX project ....................... SUCCESS [ 0.216 s]
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - com.sungard.valdi.bus.fixbroker.client.bnp ........ SKIPPED
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - XXX project Feature ...................... SKIPPED
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - ------------------------------------------------------------------------
[main] INFO org.apache.maven.cli.event.ExecutionEventLogger - BUILD SUCCESS
After the errors are:
[main] ERROR org.apache.maven.cli.MavenCli - Error executing Maven.
[main] ERROR org.apache.maven.cli.MavenCli - java.util.NoSuchElementException
role: org.apache.maven.eventspy.internal.EventSpyDispatcher
roleHint:
[main] ERROR org.apache.maven.cli.MavenCli - Caused by: null
[main] ERROR org.apache.maven.cli.MavenCli - Error executing Maven.
[main] ERROR org.apache.maven.cli.MavenCli - java.util.NoSuchElementException
role: org.apache.maven.eventspy.internal.EventSpyDispatcher
roleHint:

The solution I found is to use Maven Invoker and it works fine for the same functionality:
public class MavenInvoker {
public static void main(String[] args) throws IOException, NoHeadException, GitAPIException
{
MavenInvoker toto = new MavenInvoker();
toto.updateVersionMavenInvoker("2.2.8-SNAPSHOT", "TECHNICAL\\WEB" );
}
private InvocationRequest request = new DefaultInvocationRequest();
private DefaultInvoker invoker = new DefaultInvoker();
public InvocationResult updateVersionMavenInvoker(String newVersion, String folderPath)
{
InvocationResult result = null;
request.setPomFile( new File(folderPath+"\\pom.xml" ) );
String version = "-DnewVersion="+newVersion;
request.setGoals( Arrays.asList("-Dtycho.mode=maven",
"org.eclipse.tycho:tycho-versions-plugin:set-version",
"-DgenerateBackupPoms=false",
version) );
try {
result = invoker.execute( request );
} catch (MavenInvocationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return result;
}
}

This works for me inside a custom maven plugin (using Maven 3.5.0):
ClassRealm classRealm = (ClassRealm) Thread.currentThread().getContextClassLoader();
MavenCli cli = new MavenCli(classRealm.getWorld());
cli.doMain( ... );
The plexus Launcher sets the context class loader to its ClassRealm, which has access to the "global" ClassWorld.
Not sure how stable that solution is, but so far looking good.
Used imports:
import org.codehaus.plexus.classworlds.ClassWorld;
import org.codehaus.plexus.classworlds.realm.ClassRealm;
import org.apache.maven.cli.MavenCli;

See the email thread for a more detail explanation: https://dev.eclipse.org/mhonarc/lists/sisu-users/msg00063.html
It seems the correct way is to give MainCli a ClassWorld instance on construction so it can maintain a proper state through multiple calls.
Example:
final ClassWorld classWorld = new ClassWorld("plexus.core", getClass().getClassLoader());
MavenCli cli = new MavenCli(classWorld);
String[] arguments = {
"-Dtycho.mode=maven",
"org.eclipse.tycho:tycho-versions-plugin:set-version",
"-DgenerateBackupPoms=false",
"-DnewVersion=" + version};
int result = cli.doMain(arguments, bundlePath, System.out, System.err);