Develop Reference

Spring Cloud Stream Kafka consumer app, AdminClient is called for no reason

Small question regarding Spring Cloud Stream Kafka please.
I am having a very simple and straightforward consumer.
It is consuming only, does not produce messages, the topic is already there, I do not need to create topic.
#SpringBootApplication
public class StreamReactiveConsumerApplication implements CommandLineRunner {
private static final Logger log = LoggerFactory.getLogger(StreamReactiveConsumerApplication.class);
public static void main(String... args) {
new SpringApplicationBuilder(StreamReactiveConsumerApplication.class).web(WebApplicationType.NONE).run(args);
}
#Override
public void run(String... args) throws Exception {
log.warn("Remember about calling <.subscribe()> at the end of your Consumer<Flux> bean!");
log.warn("Remember about finishing the span manually before calling subscribe!");
}
#Bean
Consumer<Flux<Message<String>>> channel(Tracer tracer, ObservationRegistry observationRegistry) {
return flux -> flux.doOnNext(msg -> log.info("<ACCEPTANCE_TEST> <TRACE:{}> Hello from consumer",
tracer.currentSpan().context().traceId())).subscribe();
}
}
Yet on application startup, I am seeing interactions from the admin client please see logs below:
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.s.i.c.PublishSubscribeChannel : Channel 'stream-reactive-consumer.errorChannel' has 1 subscriber(s).
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.s.i.endpoint.EventDrivenConsumer : started bean '_org.springframework.integration.errorLogger'
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.s.c.s.binder.DefaultBinderFactory : Creating binder: kafka
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.s.c.s.binder.DefaultBinderFactory : Constructing binder child context for kafka
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.s.c.s.binder.DefaultBinderFactory : Caching the binder: kafka
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.a.k.c.admin.AdminClientConfig : AdminClientConfig values:
[bunch of values...]
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.a.kafka.common.utils.AppInfoParser : Kafka version: 3.3.1
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: e23c59d00e687ff5
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1675650324403
As you can see, the app is calling the admin client:
2023-02-06 10:25:24 [stream-reactive-consumer,,] o.a.k.c.admin.AdminClientConfig : AdminClientConfig values:
May I ask why this reliance on the admin for a consumer only app please?
Again, I am not producing anything, not creating any topic.
Is there a way to consume without this admin client, disable this admin client please?
Thank you

As suggested by #Gary Russell, the bug is fixed and seeing Admin client on startup for a consumer only app is indeed not expected.
In my case, it is due to a mix of couple of things:
1 - How can I configure a Spring Cloud Stream (Kafka) application to autocreate the topics in Confluent Cloud?
spring.cloud.stream.kafka.binder.autoCreateTopics property is set to true by default, this will "trigger the Admin Client"
I did set it to false on my side, but because of this issue: https://github.com/spring-cloud/spring-cloud-stream/issues/2644 when running on native, the property was not being taken into account.
Making sure the property is false, and setting the breakpoint as Gary suggested is the correct solution (upvoted what he said)

How to avoid default Quartz Scheduler in Spring Boot application?

I'm setting up a Quartz Scheduler in Spring Boot. I want the scheduled Jobs to be able to inject Beans from the Application Context. For that I created my own Quartz Job Factory:
public final class MyQuartzJobFactory extends SpringBeanJobFactory implements ApplicationContextAware {
private transient AutowireCapableBeanFactory beanFactory;
#Override
public void setApplicationContext(final ApplicationContext context) throws BeansException {
beanFactory = context.getAutowireCapableBeanFactory();
}
#Override
protected Object createJobInstance(final TriggerFiredBundle bundle) throws Exception {
final Object job = super.createJobInstance(bundle);
beanFactory.autowireBean(job);
return job;
}
}
For keeping a reference to the Scheduler, I have a SchedulerManager Bean that initializes the Scheduler:
public void initScheduler() {
try (InputStream propsInputStream = Thread.currentThread().getContextClassLoader()
.getResourceAsStream("my-quartz.properties")) {
log.info("Creating Quartz scheduler...");
Properties myQuartzProperties = new Properties();
myQuartzProperties.load(propsInputStream);
SchedulerFactoryBean schedulerFactory = new SchedulerFactoryBean();
schedulerFactory.setQuartzProperties(myQuartzProperties);
schedulerFactory.afterPropertiesSet();
schedulerFactory.setJobFactory(quartzJobFactory);
this.scheduler = schedulerFactory.getScheduler();
this.schedulingConfigMap = new HashMap<>();
log.info("Quartz scheduler created.");
} catch (Exception e) {
log.error("Unable to create Quartz scheduler", e);
throw new RuntimeException("Scheduler extension initialization error", e);
}
}
I do the initialization of the Scheduler once the context has been initialized:
public class MyLifecycleListener implements ServletContextListener {
private final SchedulerManager schedulerManager;
#Override
public void contextInitialized(ServletContextEvent sce) {
schedulerManager.initScheduler();
}
}
The quartz configuration file is as follows:
org.quartz.scheduler.instanceName=MyQuartzScheduler
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=12
org.quartz.threadPool.threadPriority=5
org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread=true
org.quartz.jobStore.class=org.quartz.simpl.RAMJobStore
org.quartz.jobStore.misfireThreshold=60000
When starting the application I see my Quartz Scheduler is initialized, but the problem is that I see a second Quartz Scheduler initialized (maybe a Spring default one?).
Mine: 'MyQuartzScheduler' with 12 threads
2022-08-23 14:13:03.239 INFO 24788 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.2 created.
2022-08-23 14:13:03.240 INFO 24788 --- [ main] org.quartz.simpl.RAMJobStore : RAMJobStore initialized.
2022-08-23 14:13:03.241 INFO 24788 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.3.2) 'MyQuartzScheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 12 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
2022-08-23 14:13:03.241 INFO 24788 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'MyQuartzScheduler' initialized from an externally provided properties instance.
Some milliseconds later, another one called 'quartzScheduler' with 10 threads.
2022-08-23 14:13:03.713 INFO 24788 --- [ main] org.quartz.core.QuartzScheduler : Quartz Scheduler v.2.3.2 created.
2022-08-23 14:13:03.713 INFO 24788 --- [ main] org.quartz.simpl.RAMJobStore : RAMJobStore initialized.
2022-08-23 14:13:03.713 INFO 24788 --- [ main] org.quartz.core.QuartzScheduler : Scheduler meta-data: Quartz Scheduler (v2.3.2) 'quartzScheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
2022-08-23 14:13:03.713 INFO 24788 --- [ main] org.quartz.impl.StdSchedulerFactory : Quartz scheduler 'quartzScheduler' initialized from an externally provided properties instance.
How can I disable this second one?
Sample project can be found here: https://github.com/dajoropo/spring-quartz-demo/tree/stackoverflow_question_73459083

The easiest way is to use the Spring configured scheduler instead of doing it yourself. Move the quartz.properties to the spring.quartz namespace in the application.properties.
spring.quartz.scheduler-name=MyQuartzScheduler
spring.quartz.properties.org.quartz.threadPool.threadCount=12
spring.quartz.properties.org.quartz.threadPool.threadPriority=5
spring.quartz.properties.org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread=true
spring.quartz.properties.org.quartz.jobStore.misfireThreshold=60000
To configure your own JobFactory write a SchedulerFactoryBeanCustomizer to configure it on the SchedulerFactoryBean.
#Component
class MySchedulerFactoryBeanCustomizer implements SchedulerFactoryBeanCustomizer {
void customize(SchedulerFactoryBean schedulerFactoryBean) {
schedulerFactoryBean.setJobFactory(new MyQuartzJobFactory());
}
}
Now ditch the quartz.properties, MyLifecycleListener and the SchedulerManager and let Spring handle all that for you.

Spring boot how to use #PostConstruct correctly

Spring boot 2.5.4 I used #PostConstruct for the very first time in my service class. As following:-
#Slf4j
#Service
#AllArgsConstructor
public class FileMonitorService {
private final AppProperties appProperties;
private final WatchService watchService;
private final RestTemplate restTemplate;
#PostConstruct
#Async
public void startMonitoring() {
FileUtils.setAppProperties(appProperties);
FileUtils.setRestTemplate(restTemplate);
FileUtils.readFilesForDirectory();
log.info("START_MONITORING");
try {
WatchKey key;
while ((key = watchService.take()) != null) {
for (WatchEvent<?> event : key.pollEvents()) {
log.info("Event kind: {}; File affected: {}", event.kind(), event.context());
if((event.kind() == StandardWatchEventKinds.ENTRY_CREATE ||
event.kind() == StandardWatchEventKinds.ENTRY_MODIFY) &&
event.context().toString().contains(".xml")){
try {
restTemplateRequest(event.context().toString()+" processing");
FileUtils.readXml(Path.of(FileUtils.getFileAbsolutePath(appProperties.getDataIn()),
event.context().toString()));
}catch (Exception e){
log.error("startMonitoring Exception: "+e.getMessage());
}
}
}
key.reset();
}
} catch (InterruptedException e) {
log.warn("startMonitoring: interrupted exception for monitoring service: "+e.getMessage());
}
}
}
This method is called as soon as app launched. That is my requirements to process all file as soon as the app starts. I have controller as following:-
#RestController
#RequestMapping("/xml")
public class FileController {
#Autowired
FileMonitorService fileMonitorService;
#SneakyThrows
#GetMapping("/restart")
public String restartFileMonitoring(){
fileMonitorService.startMonitoring();
return "File monitoring restarted started successfully";
}
}
My app starts on port 8080 and no exception at all. But when I get call this end point localhost:8080/xml/restart
It is not reachable. If I comment out the #PostConstruct then I can call the end point. I am confused how to use this annotation properly. What is wrong in my code?
Update info:-
:: Spring Boot :: (v2.5.4)
2021-09-14 18:23:21.521 INFO 71192 --- [ main] c.f.i.task.BatchProcessorApplication : Starting BatchProcessorApplication using Java 14.0.2 on dev with PID 71192 (/home/dev/Desktop/batch-processor/batch-processor/target/classes started by dev in /home/dev/Desktop/batch-processor/batch-processor)
2021-09-14 18:23:21.523 INFO 71192 --- [ main] c.f.i.task.BatchProcessorApplication : No active profile set, falling back to default profiles: default
2021-09-14 18:23:22.485 INFO 71192 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-09-14 18:23:22.495 INFO 71192 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-09-14 18:23:22.495 INFO 71192 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.52]
2021-09-14 18:23:22.564 INFO 71192 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-09-14 18:23:22.564 INFO 71192 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 988 ms
File to monitor: /home/dev/Desktop/batch-processor/batch-processor/data/in
2021-09-14 18:23:22.647 INFO 71192 --- [ main] c.f.i.task.config.FileMonitorConfig : MONITORING_DIR: /home/dev/Desktop/batch-processor/batch-processor/data/in/
2021-09-14 18:23:22.667 INFO 71192 --- [ main] c.f.i.task.service.FileMonitorService : START_MONITORING
That is the log when I run the app. After debugging I found that while ((key = watchService.take()) != null) { call never returns until I copy some XML file as this app process xml files. Then I copy any xml file in the monitoring dir. I was expecting that #Async it will run in back ground thread in async mode. How to monitory this dir in background thread? So the caller of this method won't be blocked.

PostContstruct semantics
The PostConstruct annotation is part of JSR 330 (Dependency Injection) and is not a Spring custom annotation.
The annotation specification dictates that the annotated method MUST run before the service being injected into context or translated into a service.
Spring supports the PostConstruct lifecycle hook allowing to perform extra post-initialization actions once a bean has been initialized, i.e., it had all its dependencies injected.
Async semantics
The Async annotation on the other hand is a Spring specific annotation allowing to mark a method or a type as being a candidate for asynchronous execution.
Alternative
In a case where you are interested into starting a background process as long as you application starts, you should better use the application lifecycle events and more specifically the ApplicationReadyEvent to spin your monitoring activity:
#Slf4j
#Service
#AllArgsConstructor
public class FileMonitorService {
private final AppProperties appProperties;
private final WatchService watchService;
private final RestTemplate restTemplate;
#EventListener(ApplicationReadyEvent.class)
#Async
public void startMonitoring() {
// ...
}
}
And don't forget to add the #EnableAsync annotation on your Spring Boot configuration type to activate the asynchronous processing feature.

For your case, you don't need to use #PostConstruct and that is why its working when removing the #PostConstruct
to simplify, #PostConstruct is considered as a class empty constructor but it make sure all the Beans are loaded before being called

How to implement ApplicationRunner before springboot application starts?

By reading Spring-boot docs I have understood that I can create a class implementing ApplicationRunner or CommandLineRunner in order to execute code before application starts.
From docs:
An ApplicationReadyEvent is sent after any application and command-line runners have been called. It indicates that the application is ready to service requests.
However, I have following class:
#Component
public class MyClass implements ApplicationRunner {
#Override
public void run(ApplicationArguments args) throws Exception {
Thread.sleep(10000);
}
}
And instead of waiting those 10 seconds to start, it says application started in 3seconds:
2018-11-19 08:51:21.906 INFO 24872 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2018-11-19 08:51:21.909 INFO 24872 --- [ main] com.mycompany.Application : Started Application in 3.565 seconds (JVM running for 4.016)
I assume my class is running in another thread. But I have the feeling that this does not guarantee that my code has finished executing before any possible incoming request.
Is this the correct approach? Am I missing something?

Running a Spring Boot app with Quartz

I have this Spring Boot app. using Quartz (a richly featured, open source job scheduling library that can be integrated within virtually any Java application ) to execute a Job the 10th day of the month
#EnableScheduling
#SpringBootApplication
public class IberiaUtilsApplication implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication app = new SpringApplication(IberiaUtilsApplication.class);
app.run(args);
}
#Override
public void run(String... args) throws Exception {
..
}
}
and inside the config package:
#Configuration
public class JobConfig {
#Bean
public JobDetail sampleJobDetail() {
return JobBuilder.newJob(MonthlyIberiaMetrics.class).withIdentity("iberiaJob")
.usingJobData("iberiaMetrics", "fleet").storeDurably().build();
}
#Bean
public Trigger sampleJobTrigger() {
return newTrigger()
.forJob(sampleJobDetail())
.withIdentity("iberiaTrigger")
.withSchedule(cronSchedule("0 0 10 10 * ?"))
.build();
}
}
I run the app. from the command line, using mvn spring-boot:run but it seems that the Quartz is not initialised:
...
017-10-31 15:11 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2017-10-31 15:11 [main] WARN c.z.hikari.util.DriverDataSource - Registered driver with driverClassName=oracle.jdbc.driver.OracleDriver was not found, trying direct instantiation.
2017-10-31 15:12 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2017-10-31 15:12 [main] INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
2017-10-31 15:12 [main] INFO o.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2017-10-31 15:12 [main] INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.3.0 created.
2017-10-31 15:12 [main] INFO org.quartz.simpl.RAMJobStore - RAMJobStore initialized.
2017-10-31 15:12 [main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.3.0) 'quartzScheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
2017-10-31 15:12 [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'quartzScheduler' initialized from an externally provided properties instance.
2017-10-31 15:12 [main] INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.3.0
2017-10-31 15:12 [main] INFO org.quartz.core.QuartzScheduler - JobFactory set to: org.springframework.boot.autoconfigure.quartz.AutowireCapableBeanJobFactory#64ea8964
2017-10-31 15:12 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2017-10-31 15:12 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Bean with name 'dataSource' has been autodetected for JMX exposure
2017-10-31 15:12 [main] INFO o.s.j.e.a.AnnotationMBeanExporter - Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2017-10-31 15:12 [main] INFO o.s.c.s.DefaultLifecycleProcessor - Starting beans in phase 2147483647
2017-10-31 15:12 [main] INFO o.s.s.quartz.SchedulerFactoryBean - Starting Quartz Scheduler now
2017-10-31 15:12 [main] INFO org.quartz.core.QuartzScheduler - Scheduler quartzScheduler_$_NON_CLUSTERED started.
2017-10-31 15:12 [Thread-3] INFO o.s.c.a.AnnotationConfigApplicationContext - Closing org.springframework.context.annotation.AnnotationConfigApplicationContext#589ebb: startup date [Tue Oct 31 15:11:56 CET 2017]; root of context hierarchy
2017-10-31 15:12 [Thread-3] INFO o.s.c.s.DefaultLifecycleProcessor - Stopping beans in phase 2147483647
2017-10-31 15:12 [Thread-3] INFO org.quartz.core.QuartzScheduler - Scheduler quartzScheduler_$_NON_CLUSTERED paused.
2017-10-31 15:12 [Thread-3] INFO o.s.s.quartz.SchedulerFactoryBean - Shutting down Quartz Scheduler
2017-10-31 15:12 [Thread-3] INFO org.quartz.core.QuartzScheduler - Scheduler quartzScheduler_$_NON_CLUSTERED shutting down.
2017-10-31 15:12 [Thread-3] INFO org.quartz.core.QuartzScheduler - Scheduler quartzScheduler_$_NON_CLUSTERED paused.
2017-10-31 15:12 [Thread-3] INFO org.quartz.core.QuartzScheduler - Scheduler quartzScheduler_$_NON_CLUSTERED shutdown complete.
2017-10-31 15:12 [Thread-3] INFO o.s.j.e.a.AnnotationMBeanExporter - Unregistering JMX-exposed beans on shutdown
2017-10-31 15:12 [Thread-3] INFO o.s.j.e.a.AnnotationMBeanExporter - Unregistering JMX-exposed beans
2017-10-31 15:12 [Thread-3] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
2017-10-31 15:12 [Thread-3] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
MacBook-Pro-de-lopes:iberiaUtils lopes$
I also tried changing the expression to 0 * * * * ? (every minute) with the same result
and also created this other class with the same result:
public class ScheduledTasks1 {
private static final SimpleDateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");
#Scheduled(cron = "0 * * * * ?")
public void reportCurrentTime() {
System.out.println("The time is now {}" +
dateFormat.format(new Date()));
}
}
for your information, I have another similar application without Quartz properties and its working fine:
#SpringBootApplication
public class IberiaReservationsApplication {
public static void main(String[] args) {
SpringApplication.run(IberiaReservationsApplication.class, args);
}
#Bean
public JobDetail sampleJobDetail() {
return JobBuilder.newJob(CheckDBJobExecution.class).withIdentity("sampleJob")
.usingJobData("name", "World").storeDurably().build();
}
#Bean
public Trigger sampleJobTrigger() {
return newTrigger()
.forJob(sampleJobDetail())
.withIdentity("sampleTrigger")
.withSchedule(cronSchedule("0 0/2 8-17 * * ?"))
.build();
}
}

Have you considered using Springs own scheduler, it's easy to configure and I've always found it to work well.
https://spring.io/guides/gs/scheduling-tasks/

You are missing various configuration for Quartz here like
Quartz properties
SchedulerFactoryBean
Follow below examples for complete implementation:
https://chynten.wordpress.com/2016/06/17/quartz-with-databse-and-spring-4/
http://www.baeldung.com/spring-quartz-schedule