Develop Reference

Possible to add a delay in spring integration DSL?

I have the following adapter:
#Component
class MyCustomFlow : IntegrationFlowAdapter() {
fun singleThreadTaskExecutor(): TaskExecutor {
val executor = ThreadPoolTaskExecutor()
executor.maxPoolSize = 1
executor.initialize()
return executor
}
#Filter
fun filter(data: SomeData): Boolean = ...
#Transformer
fun transform(customer: Data): Message<SomeData> {
....
}
#ServiceActivator
fun handle(data: Data): SomeData {
....
}
#Bean(name = [PollerMetadata.DEFAULT_POLLER])
fun poller(): PollerSpec? {
return Pollers.fixedRate(500)
}
#Bean
override fun buildFlow(): IntegrationFlowDefinition<*> {
return from(MessageChannels.queue("updateCustomersLocation"))
.channel(MessageChannels.executor(singleThreadTaskExecutor()))
.split()
.filter(this)
.transform(this)
.handle(this)
.channel("customerLocationFetched")
}
}
And if I understand the Java DSL, the handle is:
handle → ServiceActivator
Given a ServiceActivator, outside of the IntegrationFlowAdapter, I have an option to define a poller. Within that poller I could add a delay before the next messages is processed.
#ServiceActivator(
poller = [Poller(fixedDelay = "3000", maxMessagesPerPoll = "1", fixedRate = "3000")]
)
Would it be possible, within the adapter, to add a delay for the ServiceActivator (the method named handle, similar to the logic I get if I add another channel.
#Bean
override fun buildFlow(): IntegrationFlowDefinition<*> {
return from(MessageChannels.queue("updateCustomersLocation"))
.channel(MessageChannels.executor(singleThreadTaskExecutor()))
.split()
.filter(this)
.transform(this)
.channel("myNewChannel")
//.handle(this)
//.channel("customerLocationFetched")
}
And then from outside the adapter I could just define a new ServiceActivator:
#ServiceActivator(
inputChannel = "myNewChannel"
poller = [Poller(fixedDelay = "3000", maxMessagesPerPoll = "1", fixedRate = "3000")]
)
The reason I want a delay between the messages in the handle/service activator is that the method sends requests and I wanna control the rate the request are sent.

First of all the buildFlow() method must not be marked with #Bean : the IntegrationFlowAdapter does the trick to register everything.
You can add poller to the endpoint spec. See a second arg of that handle(). Only the point that input channel for this polling endpoint must be pollable - the QueueChannel exists out-of-the-box and you can place it just before your handle().

Using spring integration v5.5.14 lots of queued task are increasing

after upgrading to spring boot version v2.7.1 we are seeing that there are lots of queued task, we never had seen such queued task increasing in the last version we were using v2.2.2.
Our team has tried to check the things in v2.7.1 but couldn't found anything in this version.
Can anyone please review the code and let us know what we are missing or have written wrong that is causing the issue. We are using spring integration to pull emails from client server and for that we have add a taskexecutor to have concurrent polling.
Versions that we use:
Spring Boot = 2.7.1
Spring Integration = 5.5.14
Earlier we were using:
Spring Boot = 2.2.2 release
Spring Integration = 5.2.3 release
I've attached the code below.
Configuration class for Imap Integration
#Configuration
#EnableIntegration
public class ImapIntegrationConfig {
private final ApplicationContext applicationContext;
#Autowired
public ImapIntegrationConfig(ApplicationContext applicationContext) {
this.applicationContext = applicationContext;
}
#Bean("mailTaskExecutor")
public ThreadPoolTaskExecutor mailTaskExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(1000);
taskExecutor.setCorePoolSize(100);
taskExecutor.setTaskDecorator(new SecurityAwareTaskDecorator(applicationContext));
taskExecutor.setWaitForTasksToCompleteOnShutdown(true);
taskExecutor.setAwaitTerminationSeconds(Integer.MAX_VALUE);
return taskExecutor;
}
#Bean("imapMailChannel")
public ExecutorChannelSpec imapMailChannel() {
return MessageChannels.executor(mailTaskExecutor());
}
#Bean
public HeaderMapper<MimeMessage> mailHeaderMapper() {
return new DefaultMailHeaderMapper();
}
}
ImapListener Class to register the flow
public void registerImapFlow(ImapSetting imapSetting) {
ImapMailReceiver mailReceiver = createImapMailReceiver(imapSetting);
// create the flow for an email process
//#formatter:off
StandardIntegrationFlow flow = IntegrationFlows
.from(Mail.imapInboundAdapter(mailReceiver),
consumer -> consumer.autoStartup(true)
.poller(Pollers.fixedDelay(Duration.ofSeconds(5), Duration.ofMinutes(2))
.taskExecutor(taskExecutor)
.errorHandler(t -> logger.error("Error while polling emails for address " + imapSetting.getUsername(), t))
.maxMessagesPerPoll(10)))
.enrichHeaders(Map.of(CONCERN_CODE, imapSetting.getConcernCode(), IMAP_CONFIG_ID, imapSetting.getImapSettingId()))
.channel(imapMailChannel).get();
//#formatter:on
// give the bean a unique name to avoid clashes with multiple imap settings
String flowId = concernIdentifier.getConcernIdentifier() + "-" + imapSetting.getImapSettingId();
IntegrationFlowContext.IntegrationFlowRegistration existingFlow = integrationFlowContext.getRegistrationById(flowId);
if (existingFlow != null) {
// destroy the previous beans
existingFlow.destroy();
}
// register the new flow
integrationFlowContext.registration(flow).id(flowId).useFlowIdAsPrefix().register();
}
Process message method
#ServiceActivator(inputChannel = "imapMailChannel")
public void processMessage(Message<?> message) throws InvalidMessageException {
String concern = (String) message.getHeaders().get(CONCERN_CODE);
if (isEmpty(concern)) {
logger.error("Received null concern!");
}
Long imapConfigId = (Long) message.getHeaders().get(IMAP_CONFIG_ID);
String logMessage = null;
String messageId = null;
try {
Object payload = message.getPayload();
if (payload instanceof MimeMultipart) {
//.......................//
}
else if (payload instanceof String) {
//......................//
}
catch (Exception e) {
logger.error("Error while processing " + logMessage, e);
if (concern != null) {
metricUtil.emailFailed(concern);
}
throw new MaxxtonException("CCM-MessageID: Exception in processMessage() method", e, MessageErrorCode.UNABLE_TO_PROCESS_EMAIL);
}
metricUtil.emailProcessed(concern);
}
ImapMailReceiver method
private ImapMailReceiver createImapMailReceiver(ImapSetting imapSettings) {
String url = String.format(imapSettings.getImapUrl(), URLEncoder.encode(imapSettings.getUsername(), UTF_8), URLEncoder.encode(imapSettings.getPassword(), UTF_8));
ImapMailReceiver receiver = new ImapMailReceiver(url);
receiver.setSimpleContent(true);
Properties mailProperties = new Properties();
mailProperties.put("mail.debug", "false");
mailProperties.put("mail.imap.connectionpoolsize", "5");
mailProperties.put("mail.imap.fetchsize", 4194304);
mailProperties.put("mail.imap.connectiontimeout", 15000);
mailProperties.put("mail.imap.timeout", 30000);
mailProperties.put("mail.imaps.connectionpoolsize", "5");
mailProperties.put("mail.imaps.fetchsize", 4194304);
mailProperties.put("mail.imaps.connectiontimeout", 15000);
mailProperties.put("mail.imaps.timeout", 30000);
receiver.setJavaMailProperties(mailProperties);
receiver.setSearchTermStrategy(this::notSeenTerm);
receiver.setAutoCloseFolder(false);
receiver.setShouldDeleteMessages(false);
receiver.setShouldMarkMessagesAsRead(true);
receiver.setHeaderMapper(mailHeaderMapper);
receiver.setEmbeddedPartsAsBytes(false);
return receiver;
}
Added a screenshot taken from Grafana of active and queued task when we have upgraded to SP v2.7.1 and SI v5.5.14

At a glance it all looks OK. Unless you really don't close that folder manually elsewhere since you use receiver.setAutoCloseFolder(false);
There is no reason in that .taskExecutor(taskExecutor) since you use MessageChannels.executor(mailTaskExecutor()) immediately after producing message from the Mail.imapInboundAdapter().
I remember that in Gitter I suggested you to check how it works with the spring.task.scheduling.pool.size=10 placed into the application.properties. This is the only obvious difference between the mentioned versions: https://docs.spring.io/spring-boot/docs/current/reference/html/messaging.html#messaging.spring-integration.
Your screenshot doesn't prove that the problem is exactly with Spring Integration. Perhaps tasks are queued somehow by the tool which exports metrics to Graphana. I believe you have upgraded not just Spring Integration in your project...

Polled Consumer With Functional Programming & Stream bridge

I'm using spring cloud stream with kafka broker for microservice inter-communication. As part of which, stream bridge will be used to send the message, which is fine.
But during consumption of said message, the message need not be immediately consumed, rather when a condition is satisfied, then only, it should be consumed.
From the documentation, I understand that I need to use Polled Consumers(do correct me if I'm mistaken) for this.
This is what I've tried from what I've understood of the documentation.
application.properties
spring.cloud.stream.pollable-source = consumeResponse
spring.cloud.stream.function.definition = consumeResponse
#stream bridge
spring.cloud.stream.bindings.outputchannel1.destination = REQUEST_TOPIC
spring.cloud.stream.bindings.outputchannel1.binder= kafka1
#polled Consumer
spring.cloud.stream.bindings.consumeResponse-in-0.binder= kafka1
spring.cloud.stream.bindings.consumeResponse-in-0.destination = REQUEST_TOPIC
spring.cloud.stream.bindings.consumeResponse-in-0.group = consumer_cloud_stream1
spring.cloud.stream.binders.kafka1.type=kafka
MainApplication.java
#Bean
public CommandLineRunner commandLineRunner(ApplicationContext ctx) {
return args -> {
//produce message
for (int i = 0; i < 5; i++) {
streamBridge.send("outputchannel1", "msg"+i);
System.out.println("Request :: " + "msg"+i);
}
};
}
#Bean
public Consumer<String> consumeResponse() {
return (response) -> {
//consume message
System.out.println("Response :: " + response);
};
}
#Bean
public ApplicationRunner poller(PollableMessageSource destIn, MessageChannel destOut) {
return args -> {
while (someCondition) { //some condition that checks whether or not to consume the message
try {
//condition satisfied, so forward message to consumer
if (!destIn.poll(m -> {
String newPayload = ((String) m.getPayload());
destOut.send(new GenericMessage<>(newPayload));
})) {
Thread.sleep(1000);
}
} catch (Exception e) {
e.printStackTrace();
}
}
};
}
But this throws the following exception:-
Parameter 1 of method poller in com.MainApplication required a single bean, but 2 were found:
- nullChannel: defined in null
- errorChannel: defined in null
I'd appreciate it if someone could help me out here or point me towards a working example for the same.
Spring boot version: 2.6.4,
Spring cloud version: 2021.0.1

Why don't you just inject the StreamBridge into your runner instead of the message channel?
By default, stream bridge output channels are created on-demand (first send).

Periodic work not reach running state when unit testing WorkManager

I created a PeriodicWorkRequest from my SyncDatabaseWorker, which like below:
class SyncDatabaseWorker(ctx: Context, params: WorkerParameters) : RxWorker(ctx, params) {
private val dataManager: DataManager = App.getDataManager()
override fun createWork(): Single<Result> {
return Single.create { emitter ->
dataManager.loadStoresFromServer()
.subscribe(object : SingleObserver<List<Store>> {
override fun onSubscribe(d: Disposable) {
}
override fun onSuccess(storeList: List<Store>) {
if (!storeList.isEmpty()) {
emitter.onSuccess(Result.success())
} else {
emitter.onSuccess(Result.retry())
}
}
override fun onError(e: Throwable) {
emitter.onSuccess(Result.failure())
}
})
}
}
companion object {
fun prepareSyncDBWorker(): PeriodicWorkRequest {
val constraints = Constraints.Builder()
.setRequiredNetworkType(NetworkType.CONNECTED)
.build()
val myWorkBuilder = PeriodicWorkRequest.Builder(SyncDatabaseWorker::class.java, 7, TimeUnit.DAYS)
.setBackoffCriteria(BackoffPolicy.EXPONENTIAL, 1, TimeUnit.DAYS) // Backoff retry after 1 day
.setConstraints(constraints)
return myWorkBuilder.build()
}
}
}
Then I write unit test base on Google's guide like this:
#RunWith(AndroidJUnit4::class)
class SyncDatabaseWorkerTest {
private lateinit var context: Context
#Before
fun setup() {
context = InstrumentationRegistry.getInstrumentation().targetContext
val config = Configuration.Builder()
.setMinimumLoggingLevel(Log.DEBUG)
.setExecutor(SynchronousExecutor())
.build()
// Initialize WorkManager for instrumentation tests.
WorkManagerTestInitHelper.initializeTestWorkManager(context, config)
}
#Test
#Throws(Exception::class)
fun testPeriodicWork_WithConstrains() {
// Create request
val request = SyncDatabaseWorker.prepareSyncDBWorker()
val workManager = WorkManager.getInstance(context)
val testDriver = WorkManagerTestInitHelper.getTestDriver(context)
// Enqueue and wait for result.
workManager.enqueue(request).result.get()
// Check work request is enqueued
var workInfo = workManager.getWorkInfoById(request.id).get()
assertThat(workInfo.state, `is`(WorkInfo.State.ENQUEUED))
// Tells the testing framework the period delay & all constrains is met
testDriver!!.setPeriodDelayMet(request.id)
testDriver.setAllConstraintsMet(request.id)
// Check work request is running
workInfo = workManager.getWorkInfoById(request.id).get()
assertThat(workInfo.state, `is`(WorkInfo.State.RUNNING))
}
}
It's always in state ENQUEUED, even when period delay & all constrains is met.
Expected: is <RUNNING>
but: was <ENQUEUED>
When I debugged the test & find out that the createWork(): Single<Result> method is also triggered, but why the state is not RUNNING?
May be I'm wrong about the approach, but the documents about unit testing WorkManager is very few now, and I don't know the right way to do it.

Since you're using a synchronous executor, you will never actually see your work in the RUNNING state - it should have already executed. I suspect your work is actually being marked for retry and therefore enters the ENQUEUED state again. You should be able to verify this by either setting breakpoints or looking at your logs.

Given you are executing a PeriodicWorkRequest with a SynchronousExecutor, you will never see the WorkRequest in RUNNING state. It will be done executing before you can assert that it was in RUNNING.
After you return a Result.success() or a Result.failure() in your doWork(), the WorkRequest goes back to ENQUEUED for the next period (given it's a periodic request).

Spring sync vs async rest controller

I try to see a difference between Spring synchronous REST Controller vs async version of same controller.
Each controller do the same thing : take a RequestBody and save it in a Mongo database.
#RestController
#RequestMapping ("/api/1/ticks")
public class TickController {
#Autowired
private TickManager tickManager;
#RequestMapping (method = RequestMethod.POST)
public ResponseEntity save(#RequestBody List<Tick> ticks) {
tickManager.save(ticks);
return new ResponseEntity(HttpStatus.OK);
}
#RequestMapping (value = "/async", method = RequestMethod.POST)
public #ResponseBody Callable<ResponseEntity> saveAsync(#RequestBody List<Tick> ticks) {
return () -> {
tickManager.save(ticks);
return new ResponseEntity(HttpStatus.OK);
};
}
}
The tickManager has only a dependency on a tickRepository and just do call to sub-layer.
The tickRepository is based on Spring Data Mongodb:
#Repository
public interface TickRepository extends MongoRepository<Tick, String> {}
I use Gatling to test those controllers.
This is my scenario:
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class TicksSaveSyncSimulation extends Simulation {
val rampUpTimeSecs = 20
val testTimeSecs = 5
val noOfUsers = 1000
val minWaitMs = 1000 milliseconds
val maxWaitMs = 3000 milliseconds
val baseURL = "http://localhost:9080"
val requestName = "ticks-save-sync-request"
val scenarioName = "ticks-save-sync-scenario"
val URI = "/api/1/ticks"
val httpConf = http.baseURL(baseURL)
val http_headers = Map(
"Accept-Encoding" -> "gzip,deflate",
"Content-Type" -> "application/json;charset=UTF-8",
"Keep-Alive" -> "115"
)
val scn = scenario(scenarioName)
.repeat(100) {
exec(
http(requestName)
.post(URI)
.headers(http_headers)
.body(StringBody(
"""[{
| "type": "temperature",
| "datas": {}
|}]""".stripMargin))
.check(status.is(200))
)
}
setUp(scn.inject(rampUsers(1000) over (1 seconds))).protocols(httpConf)
}
I tried several situations and the sync version always handle 2 times more request by second than the async version.
When I increase the number of users the two versions crash.
I tried to override the taskExecutor for the async version with no more success:
#Configuration
public class TaskExecutorConfig implements AsyncConfigurer {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(1000);
taskExecutor.setThreadNamePrefix("LULExecutor-");
taskExecutor.initialize();
return taskExecutor;
}
#Override
public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
return new SimpleAsyncUncaughtExceptionHandler();
}
}
I thought see a difference in favor of the async implementation.
What am I doing wrong?

Your test looks to be flawed. It doesn't make any sense being non blocking at one end of the pipeline (here, your controllers), and being blocking at the other end (tickManager.save really looks like a blocking call). You're just paying the extra cost of jumping into a ThreadPoolTaskExecutor.
Then, generally speaking, you won't gain anything from a non blocking architecture when all your tasks are very fast, like a tick. You can expect gains when some tasks take some longer time, so you don't want to waste resources (threads, CPU cycles) just waiting for those to complete, and you want to use them to perform other tasks in the meanwhile.
Regarding your Too many open files exception, you probably haven't properly tuned your OS for load testing, check relevant documentation. There's also a good chance that you're running your app and Gatling (and possibly your database too) on the same host, which is bad as they'll compete for resources.