I'm trying to migrate from Vert.x to Quarkus and in Vert.x when I write message consumers like Kafka/AMQP etc. I have to scale the number of verticals to maximize performance across multiple cores i.e. Vertical Scaling - is this possible in Quarkus? I see a similar question here but it wasn't answered.
For example, with Kafka I might create a consumer inside a vertical and then scale that vertical say 10 times (that is specify the number of instances in the deployment to be 10) after doing performance testing to determine that's the optimal number. My understanding is that by default, 1 vertical = 1 event loop and does not scale across multiple cores.
I know that it's possible to use Vert.x verticals in Quarkus but is there another way to scale things like the number of Kafka consumers across multiple core?
I see that this type of scalability is configurable for things like Quarkus HTTP but I can't find anything about message consumers.
Here's the Vert.x Verticle approach that overall I'm very happy with, but I wish there were better documentation on how to do this.
UPDATE - Field injection doesn't work with this example but constructor injection does work.
Lets say I want to inject this
#ApplicationScoped
public class CoffeeRepositoryService {
public CoffeeRepositoryService() {
System.out.println("Injection succeeded!");
}
}
Here's my Verticle
package org.acme;
import io.smallrye.mutiny.Uni;
import io.smallrye.mutiny.vertx.core.AbstractVerticle;
import io.vertx.core.impl.logging.Logger;
import io.vertx.core.impl.logging.LoggerFactory;
import io.vertx.mutiny.core.eventbus.EventBus;
import io.vertx.mutiny.rabbitmq.RabbitMQClient;
import io.vertx.mutiny.rabbitmq.RabbitMQConsumer;
import io.vertx.rabbitmq.QueueOptions;
import io.vertx.rabbitmq.RabbitMQOptions;
public class RQVerticle extends AbstractVerticle {
private final Logger LOGGER = LoggerFactory.getLogger(org.acme.RQVerticle.class);
//This doesn't work - returns null
#Inject
CoffeeRepositoryService coffeeRepositoryService;
RQVerticle() {} // dummy constructor needed
#Inject // constructor injection - this does work
RQVerticle(CoffeeRepositoryService coffeeRepositoryService) {
//Here coffeeRepositoryService is injected properly
}
#Override
public Uni<Void> asyncStart() {
LOGGER.info(
"Creating RabbitMQ Connection after Quarkus successful initialization");
RabbitMQOptions config = new RabbitMQOptions();
config.setUri("amqp://localhost:5672");
RabbitMQClient client = RabbitMQClient.create(vertx, config);
Uni<Void> clientResp = client.start();
clientResp.subscribe()
.with(asyncResult -> {
LOGGER.info("RabbitMQ successfully connected!");
});
return clientResp;
}
}
Main Class - injection doesn't work like this
package org.acme;
import io.quarkus.runtime.Quarkus;
import io.quarkus.runtime.QuarkusApplication;
import io.quarkus.runtime.annotations.QuarkusMain;
import io.vertx.core.DeploymentOptions;
import io.vertx.mutiny.core.Vertx;
#QuarkusMain
public class Main {
public static void main(String... args) {
Quarkus.run(MyApp.class, args);
}
public static class MyApp implements QuarkusApplication {
#Override
public int run(String... args) throws Exception {
var vertx = Vertx.vertx();
System.out.println("Deployment Starting");
DeploymentOptions options = new DeploymentOptions()
.setInstances(2);
vertx.deployVerticleAndAwait(RQVerticle::new, options);
System.out.println("Deployment completed");
Quarkus.waitForExit();
return 0;
}
}
}
Main Class with working injection but cannot deploy more than one instance
package org.acme;
import io.quarkus.runtime.StartupEvent;
import io.vertx.mutiny.core.Vertx;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.event.Observes;
import org.jboss.logging.Logger;
#ApplicationScoped
public class MainVerticles {
private static final Logger LOGGER = Logger.getLogger(MainVerticles.class);
public void init(#Observes StartupEvent e, Vertx vertx, RQVerticle verticle) {
public void init(#Observes StartupEvent e, Vertx vertx, RQVerticle verticle) {
DeploymentOptions options = new DeploymentOptions()
.setInstances(2);
vertx.deployVerticle(verticle,options).await().indefinitely();
}
}
Std Out - first main class looks good
2021-09-15 15:48:12,052 INFO [org.acm.RQVerticle] (vert.x-eventloop-thread-2) Creating RabbitMQ Connection after Quarkus successful initialization
2021-09-15 15:48:12,053 INFO [org.acm.RQVerticle] (vert.x-eventloop-thread-3) Creating RabbitMQ Connection after Quarkus successful initialization
Std Out - second main class
2021-09-22 15:48:11,986 ERROR [io.qua.run.Application] (Quarkus Main
Thread) Failed to start application (with profile dev):
java.lang.IllegalArgumentException: Can't specify > 1 instances for
already created verticle
Related
I have a Micronaut declarative HTTP client written using #client annotation. I want to call this while starting micronaut app before creating the ApplicationContext itslef.
HttpClient : SampleHttpClient.java
import io.micronaut.http.HttpResponse;
import io.micronaut.http.MediaType;
import io.micronaut.http.annotation.Get;
import io.micronaut.http.annotation.Header;
import io.micronaut.http.client.annotation.Client;
#Client("http://127.0.0.1:8200")
#Header(name = "X-Vault-Token", value = "hvs.CEGT7cKyMA8wsDbgKZqxC34q")
public interface SampleHttpClient {
#Get(value = "/v1/kv/data/KMS", produces = MediaType.APPLICATION_JSON)
HttpResponse<String> getVaultSecret();
}
Application.java (Main class)
import io.micronaut.context.ApplicationContext;
import io.micronaut.runtime.Micronaut;
public class Application {
public static void main(String[] args) {
// Following code works perfect. I am creating context here. But I dont want to do this
SampleHttpClient client = Micronaut.run(Application.class, args).
getBeansOfType(SampleHttpClient.class).stream().findFirst().get();
System.out.println("Response Body ="+client.getVaultSecret().body());
// How do we get the instance of SampleHttpClient without using Micronaut's dependency
injection process???
}
}
I want to call this while starting micronaut app before creating the
ApplicationContext itslef.
Micronaut doesn't provide a mechanism to support that. You could write your own thing that instantiates the beans, but that is a large undertaking. You would be writing your own bean container.
I have developed a #JMSListener that gets the destination from Java properties and works just fine.
But now I would need to be able to change the "destination" of the queue on runtime without having to reset the whole application, and even if I modify the Properties on runtime, the queue "destination" does not change.
Here is how We are implementing the #JMSListener:
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.jms.annotation.JmsListener;
import org.springframework.jms.support.converter.MessageConverter;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.interceptor.TransactionAspectSupport;
#Component("b2b.CCRReceiver")
#Slf4j
public class CCRReceiver {
//SOME_VARIABLES
#Transactional
#JmsListener(destination = "${tibco.configuration.queues.upsert}", containerFactory = "jmsFactory", concurrency = "${jms.concurrency}")
public void receiveMessage(Message message) {
//DO_SOME_STUFF
}
}
As you can see, I get the destination from a Value Expression the first time and it works fine, but then I don't know how to access the JMSListener and change it's destination.
Can this be done? Is there any way to change the destination?
Or I will have to implement this JMS Listener in an other way that allows me to do this?
This should work:
Give the listener an id property
Auto wire the JmsListenerEndpointRegistry (or otherwise get a reference to it)
registry.getListenerContainer("myListener").stop();
registry.getListenerContainer("myListener").shutdown();
((AbstractMessageListenerContainer) registry.getListenerContainer("myListener"))
.setDestinationName("newOne")
registry.getListenerContainer("myListener").initialize();
registry.getListenerContainer("myListener").start();
I solve this problem work with a component Listener Thread. Using TaskExecutor and ApplicationContext to manage. You can create at runtime. I'm still working on it. I'll try Gary Russell's suggestion too.
Sorry about english. Feel free to correct.
applicationContext.getBean(ExampleListenerJMS.class);
...
taskExecutor.execute(exampleListenerJMS);
The class listener "implements Runnable, MessageListener" with a implementation getting custom connection managers (activemq servers different).
#Component
#Scope("application")
public class ExampleListenerJMS implements Runnable, MessageListener {
private EspecificManagerJMS jms = new EspecificManagerJMS();
#Override
public void run() {
customAndChekingActions();
}
protected void customAndChekingActions() {
...
try {
Destination destination = jms.getSession().createQueue(queue);
MessageConsumer consumer = jms.getSession().createConsumer(destination);
consumer.setMessageListener(this);
...
} catch (JMSException e) {
e.printStackTrace();
...
}
}
#Override
public void onMessage(Message message) {
...
}
I hope it helped you
Well, i felt myself really lost with Vertx structure due to everything is a lambda expression.
i followed this tutorial exactly in order to well structure my application,
unfortunately it doesn't register any router i have no idea why. pleasefind bellow what i did
serviceEndPoint same with the above tutorial
import io.vertx.core.Vertx;
import io.vertx.ext.web.Router;
public interface ServiceEndPoint {
String mountPoint();
Router router(Vertx vertx);
}
and here is the subscriptionService
import com.poc.poc.repositories.SubscriptionRepository;
import com.poc.poc.services.ServiceEndPoint;
import io.vertx.core.Vertx;
import io.vertx.ext.web.Router;
public class SubscriptionService implements ServiceEndPoint {
private SubscriptionRepository subscriptionRepository;
public SubscriptionService() {
subscriptionRepository = new SubscriptionRepository();
}
#Override
public String mountPoint() {
return "/test";
}
#Override
public Router router(Vertx vertx) {
Router router = Router.router(vertx);
router.get("/test").handler(rc -> rc.response().end(subscriptionRepository.getSubscriptionInfo(rc, vertx).toString()));
return router;
}
}
And finally here is the server vertical
import com.poc.poc.services.ServiceEndPoint;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.Future;
import java.util.ServiceLoader;
import java.util.stream.StreamSupport;
import io.vertx.ext.web.Router;
public class ServerVertical extends AbstractVerticle {
#Override
public void start(final Future<Void> startFuture) throws Exception {
ServiceLoader<ServiceEndPoint> loader = ServiceLoader.load(ServiceEndPoint.class);
Router main = StreamSupport.stream(loader.spliterator(), false)
.collect(() -> Router.router(vertx), //the main router
(r, s) -> r.mountSubRouter(s.mountPoint(), s.router(vertx)),
(r1, r2) -> {
});
vertx.createHttpServer().requestHandler(main::accept).listen(8080, res -> {
if (res.succeeded()) {
startFuture.complete();
} else {
startFuture.fail(res.cause());
}
});
}
}
Please be noted once i run the application, im getting those warnings
Jun 12, 2018 6:16:45 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2486 ms, time limit is 2000
Jun 12, 2018 6:16:46 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3485 ms, time limit is 2000
by the way Router main = StreamSupport.stream(loader.spliterator(), false) size is 0.
any help ?
First, not everything in Vert.x is a lambda expression. That's just quite a weird tutorial you've found. As you can see, it uses java.util.ServiceLoader which is not a Vert.x class. Nor I'm familiar with anyone else recommending to use this class with Vert.x applications.
What it tries to do is to load your classes dynamically. What you probably miss is putting the correct file in META-INF directory, as described here: https://docs.oracle.com/javase/tutorial/ext/basics/spi.html#register-service-providers
Anyway, that's not the way I would recommend to use VertX. Instead, go with the regular VertX tutorial, which is excellent: https://vertx.io/docs/vertx-web/java/#_handling_requests_and_calling_the_next_handler
Once you have created you service interface (com.poc.poc.services.ServiceEndPoint) declaration and the concrete implementation (SubscriptionService), you should add the service provider binding.
As per the ServiceLocator documentation, the binding should be inserted under a file named after you interface FQN, i.e. under META-INF/services/com.poc.poc.services.ServiceEndPoint (The whole directory structure goes under the project / module resources directory).
The file will contain the actual interface implementation:
com.poc.poc.services.SubscriptionService
I am trying to attach a subscriber to an event in Esper but I would like to use .epl file for that. I've been browsing repositories and I have seen examples of doing that by using annotation interfaces. I was trying to do it the same way they do it in CoinTrader, but I can't seem to get it to work. Yet, if I set the subscriber in Java, it works.
This is my project structure for reference
This is my .epl file:
module queries;
import events.*;
import configDemo.*;
import annotations.*;
create schema MyTickEvent as TickEvent;
#Name('allEvents')
#Description('test')
#Subscriber(className='configDemo.TickSubscriber')
select * from TickEvent;
#Name('tickEvent')
#Description('Get a tick event every 3 seconds')
select currentPrice from TickEvent;
This is my config file:
<?xml version="1.0" encoding="UTF-8"?>
<esper-configuration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.espertech.com/schema/esper"
xsi:noNamespaceSchemaLocation="esper-configuration-6-0.xsd">
<event-type-auto-name package-name="events"/>
<auto-import import-name="annotations.*"/>
<auto-import import-name="events.*"/>
<auto-import import-name="configDemo.*"/>
This is my Subscriber interface:
package annotations;
public #interface Subscriber {
String className();
}
This is my event class:
package configDemo;
import events.TickEvent;
public class TickSubscriber {
public void update(TickEvent tick) {
System.out.println("Event registered by subscriber - Tick is: " +
tick.getCurrentPrice());
}
}
And my main file is this:
package configDemo;
import java.io.IOException;
import java.util.concurrent.CountDownLatch;
import com.espertech.esper.client.EPStatement;
import com.espertech.esper.client.deploy.DeploymentException;
import com.espertech.esper.client.deploy.DeploymentOptions;
import com.espertech.esper.client.deploy.Module;
import com.espertech.esper.client.deploy.ParseException;
public class Main {
public static EngineHelper engineHelper;
public static Thread engineThread;
public static boolean continuousSimulation = true;
public static void main(String[] args) throws DeploymentException, InterruptedException, IOException, ParseException {
engineHelper = new EngineHelper();
DeploymentOptions options = new DeploymentOptions();
options.setIsolatedServiceProvider("validation"); // we isolate any statements
options.setValidateOnly(true); // validate leaving no started statements
options.setFailFast(false); // do not fail on first error
Module queries = engineHelper.getDeployAdmin().read("queries.epl");
engineHelper.getDeployAdmin().deploy(queries, null);
CountDownLatch latch = new CountDownLatch(1);
EPStatement epl = engineHelper.getAdmin().getStatement("allEvents");
//epl.setSubscriber(new TickSubscriber());
engineThread = new Thread(new EngineThread(latch, continuousSimulation, engineHelper.getRuntime()));
engineThread.start();
}
}
As you can see the setSubscriber line is commented out. When I run it as is, I expected that the subscriber will be recognized and registered and yet it isn't. I only get the tick events flowing in the console. If I decomment the line and I run it, I get a notification after each tick that the subscriber received the event and it all works fine.
What am I doing wrong? How can I set a subscriber within the .epl file?
Assigning a subscriber is done by the application and is not something that the engine does for you. The app code would need to loop thru the statements, get the annotations "stmt.getAnnotations" and inspect these and assign the subscriber.
I have been inventing a way how to work around the problem of adding consumers to a jetty endpoint (it does not allow multiple consumers). The way we do it in our company is to build our own router and a broadcasting endpoint which consumes from jetty and routes requests to underlying "subscriptions". Only one of them will eventually process the request. It kind of works but it's not completely ok, since recently when updating to latest Camel we have found our custom built component to leak memory and in general I consider using built-in functionality over custom hacks.
I started investigating the Camel REST API and found it very nice and pretty much replacing our home-grown component apart from one thing - you cannot re-configure it at runtime - you have to stop the context basically for this to work. Below I include my unit test with a happy path and the path that fails. Frankly I think is a bug, but if there is a legitimate way to achieve what I want, I'd like to hear sound advice:
package com.anydoby.camel;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.commons.io.IOUtils;
import org.junit.Before;
import org.junit.Test;
/**
* Test tries to add/remove routes at runtime.
*/
public class RoutesTest {
private DefaultCamelContext ctx;
#Before
public void pre() throws Exception {
ctx = new DefaultCamelContext();
new RouteBuilder(ctx) {
#Override
public void configure() throws Exception {
restConfiguration("jetty").host("localhost").port(8080);
rest("/")
.get("/issues/{isin}").route().id("issues")
.process(e -> e.getOut().setBody("Here's your issue " + e.getIn().getHeader("isin"))).endRest()
.get("/listings").route().id("listings").process(e -> e.getOut().setBody("some listings"));
}
}.addRoutesToCamelContext(ctx);
ctx.start();
}
#Test
public void test() throws IOException {
{
InputStream stream = new URL("http://localhost:8080/issues/35").openStream();
assertEquals("Here's your issue 35", IOUtils.toString(stream));
}
{
InputStream stream = new URL("http://localhost:8080/listings").openStream();
assertEquals("some listings", IOUtils.toString(stream));
}
}
#Test
public void disableRoute() throws Exception {
ctx.stopRoute("issues");
ctx.removeRoute("issues");
try (InputStream stream = new URL("http://localhost:8080/issues/35").openStream()) {
fail();
} catch (Exception e) {
}
new RouteBuilder(ctx) {
#Override
public void configure() throws Exception {
rest().get("/issues/{isin}/{sedol}").route().id("issues")
.process(e -> e.getOut()
.setBody("Here's your issue " + e.getIn().getHeader("isin") + ":" + e.getIn().getHeader("sedol")))
.endRest();
}
}.addRoutesToCamelContext(ctx);
{
InputStream stream = new URL("http://localhost:8080/issues/35/65").openStream();
assertEquals("Here's your issue 35:65", IOUtils.toString(stream));
}
}
}
The disableRoute() test fails since I cannot add another consumer to an existing endpoint.
So my question is - "is there a way to add a new URL mapping to a restful camel-jetty endpoint"? If you do it during first configuration it works fine, but when later you want to reconfigure one of the routes the error is:
org.apache.camel.FailedToStartRouteException: Failed to start route because of Multiple consumers for the same endpoint is not allowed: jetty:http://localhost:8080/issues/%7Bisin%7D/%7Bsedol%7D?httpMethodRestrict=GET