I'm trying to start a Camel route on GAE and am running into one brick wall after the next. First I tried a route that looked like this:
from("direct:start")
.process(new Processor() {
#Override
public void process(Exchange exchange) {
logger.info("I made it!");
}
})
.to("direct:end");
But that doesn't do anything. Being new to both Camel and GAE, I suspect that's because the direct component doesn't act as a consumer for the beginning of a route. So then I tried kicking off a timer:
from("timer://runOnce?repeatCount=1")
.process(new Processor() {
#Override
public void process(Exchange exchange) {
logger.info("I made it!");
}
})
.to("direct:end");
Only to get a nasty error from GAE:
Error: access denied (java.lang.RuntimePermission modifyThreadGroup)
And it turns out you can't create new Thread instances on GAE, and that's exactly what the Timer component does.
So then I tried kicking the route off with a bean:
public class DummyBean {
public void kickoffRoute() { return; }
}
// Inside the method that creates and starts the Camel route
SimpleRegistry reg = new SimpleRegistry();
DummyBean bean = new DummyBean();
reg.put("dummy", bean);
CamelContext camel = new DefaultCamelContext(reg);
camel.disableJMX();
// Inside my RouteBuilder
from("bean:dummy")
.process(new Processor() {
#Override
public void process(Exchange exchange) {
logger.info("I made it!");
}
})
.to("direct:end");
I ran this, only to get the same "access denied" error as from before with the Timer.
All I'm trying to do is kick off a route from inside GAE so I can start getting familiar with both technologies (Camel & GAE). But for the life of me, I can't figure out how to do this - every Camel component seems to create threads, which are illegal on GAE! So I ask:
How do you even kickoff a route (an initial producer - Timer, or otherwise) on GAE? I see there is a Camel-GAE component, and I'm certainly willing to read up on it and learn how to use it, so that I could have routes starting with, say, a ghttp:///startRoute endpoint, but I'm really just starting out here and am hoping for an easier component/endpoint to work with.
If both Timer and Bean components create threads, and this is disallowed on GAE, I have a sick feeling in my stomach that most/all Camel components are disallowed on GAE. Is this the case?!?! If so, how do you build useful/meaningful routes on GAE? Or is there some GAE "trick" to getting Camel to run and create threads? For instance, I know GAE backends do not suffer the same threading restrictions as frontend instances, etc.
Thanks in advance!
Camel is built to run on Java environments using Java SE 6/7 with whatever that implies in file system access, thread creation, possibilities to open ports etc.
Google App Engine has, as you realised, several restrictions. That does not make Camel unusable at all. You can use a lot of the routing features, producers endpoints (.to(..)), transformations, etc etc.
In fact, in Java EE Application Servers, there are similar restrictions, but that does not prevent usage of camel.
If you really are up for it, and need to do everything camel inside GAE, it's perhaps possible to make jack in Camels pluggable threading models into GAE.
Disclaimer: I have never used GAE tasks, but there is some documentations here
https://developers.google.com/appengine/docs/java/taskqueue/
http://camel.apache.org/threading-model.html
You can also try to start the events by whatever feature you have in GAE and just use a producer template to kick start a Camel route.
Related
Spring boot Apache Camel-Java DSL app reads messages from Kafka topic.
#Component
public class KafkaTopicService extends RouteBilder {
public void configure(){
from("kafka:myTopic?brokers=localhost:9092")
.log("Message received from Kafka: ${body}")}
}
If I stop Kafka I get org.apache.kafka.common.errors.DisconnectException
I looked into onException(...class).handled(true) but Im not sure how to implement handling of the exception in my code. Can someone give me few implementation examples? What options are available? For example logging the message or reattempting to read message?
Documentation also mentions Quarkus. Do I need Quarkus to use onException()?
You can do something like (have not tried running it so please take care of any typos)
#Component
public class KafkaTopicService extends RouteBilder {
public void configure(){
onException(org.apache.kafka.common.errors.DisconnectException.class)
.log("Error connecting kafka");
from("kafka:myTopic?brokers=localhost:9092&bridgeErrorHandler=true")
.log("Message received from Kafka: ${body}")}
}
Please note that I have added bridgeErrorHandler=true. Normally exception handling happens after from. In most of the case using bridgeErrorHandler we can use onException function for those.
Also note that I have defined onException outside your route, so the exception handling logic which you add would be global and applicable to all routes wherever you encounter DisconnectException
Problem
Very short: I want to create Spring Shell, but as a web application.
I want to create a web-application (preferably using Spring Boot), where the frontend (ReactJS) looks like a terminal (or shell), and the backend processes inputted commands. Look at https://codepen.io/AndrewBarfield/pen/qEqWMq. I want to build a full web app for something that looks like that.
I want to build a framework, so that I can develop backend commands without knowing anything about the frontend/web application structure. I basically want to instantiate a "Terminal" object, where I give some kind of input-stream and output-stream. This way I can program this Terminal based on my given interfaces and structure, without the need of setting up all kind of front-end stuff.
A good summary of the question would be: how to send all keyboard inputs to the backend, and how to send all output to the frontend?
The reason I want to create a web application, is because I want it to be available online.
What I tried
I think the way of reaching this is using websockets. I have created a small web application using this (https://developer.okta.com/blog/2018/09/25/spring-webflux-websockets-react) tutorial, without the security part. The websocket part is almost suitable, I just cannot get an "input" and "output" stream-like object.
#Controller
public class WebSocketController {
private SimpMessagingTemplate simpMessagingTemplate;
#Autowired
public WebSocketController(SimpMessagingTemplate simpMessagingTemplate) {
this.simpMessagingTemplate = simpMessagingTemplate;
}
#MessageMapping("/queue")
#SendToUser("/topic/greetings")
public Greeting greeting(HelloMessage message, #Header(name = "simpSessionId") String sessionId) throws Exception {
System.out.println(sessionId);
// Do some command parsing or whatever.
String output = "You inputted:" + HtmlUtils.htmlEscape(message.getName());
return new Greeting(output);
}
private MessageHeaders createHeaders(String sessionId) {
SimpMessageHeaderAccessor headerAccessor = SimpMessageHeaderAccessor.create(SimpMessageType.MESSAGE);
headerAccessor.setSessionId(sessionId);
return headerAccessor.getMessageHeaders();
}
Now with this code, you can parse a command. However, it doesn't keep any "state". I don't know how it works with states and websockets.
I saw you had this Spring Sessions + WebSockets (https://docs.spring.io/spring-session/docs/current/reference/html5/guides/boot-websocket.html), but this is not really what I want.
I can send a message from the backend to the frontend by using this code:
simpMessagingTemplate.convertAndSendToUser(sessionId, "/topic/greetings", "hey", createHeaders(sessionId));
However, I want my terminal to be able to wait for input commands from the user. Seems like a stretch, but does anybody know how to achieve this?
What I sort of want
I basically want other people to program to this interface:
public interface ITerminal {
void setInputStream(Object someKindOfWrapperForTheInput);
void setOutputStream(Object someWrapperOfSimpMessagingTemplate);
void start();
}
When somebody opens the web application, they get a dedicated terminal object (so a single connection per user). Whever somebody enters a command in the frontend application, I want it to be received by the terminal object, processed, and response outputted to the frontend.
Reasons for doing this
I really like creating command-line applications, and I don't like building frontend stuff. I work as a software engineer for a company where we build a web application, where I mostly program backend stuff. All the frontend part is done by other people (lucky for me!). However, I like doing some projects at home, and this seemed cool.
If you have any thoughts or ideas on how to approach this, just give an answer! I am interested in the solution, using the SpringBoot framework is not a requirement. I ask this question using Spring Boot and ReactJS, because I have already built applications with that. A lot has been figured out already, and I think this probably exists as well.
The only requirement is that I can achieve this with Java on a tomcat-server. The rest is optional :)
Unclear?
I tried my best to make my story clear, but I am not sure if my purpose of what I want to achieve is clear. However, I don't know how to formulate it in such a way you understand. If you have any suggestions or questions, dont hesitate to comment!
If the only thing you want is a Live Spring shell that shows up in the browser it's fairly simple, all you need is to expose a standard WebSocket via the WebSocketConfigurer, then add a WebSocketHandler that executes the command and then returns the resulting String as a TextMessage.
Firstly the Socket configuration that allows clients to connect to the 'cli' endpoint
#Configuration
#EnableWebSocket
public class WebSocketConfiguration implements WebSocketConfigurer {
#Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(cliHandler(), "/cli").setAllowedOrigins("*");
}
#Bean
public CLIWebSocketHandler cliHandler() {
return new CLIWebSocketHandler();
}
}
Then the WebSocketHandler that executes the command. I recommend that for every #ShellMethod you specify the return type as String, don't use logging or System writes as they won't be returned during the evaluation.
#Component
public class CLIWebSocketHandler extends TextWebSocketHandler {
#Autowired
private Shell shell;
#Override
protected void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
String result = shell.evaluate(() -> message.getPayload()).toString();
session.sendMessage(new TextMessage(result));
}
}
You can use an extension like Simple WebSocket Client to test it, by going to ws://localhost:port/cli
This is the most basic solution, adding features like security should be easy after this. Notice that I don't use STOMP, because you probably want to isolate users. But it can work alongside STOMP based endpoints, so you can have pub-sub functionality for other parts of the project.
From the question I sense that answer you'd like is something that involved Input and OutputStreams. You could possibly look into redirecting the output of Spring Shell to a different stream then have them forwarded to the sessions but it's probably much more complicated and has other trade-offs. It's simpler to just return a String as the result, it looks better in print outs anyway.
I am trying to get familiar with EIP and Apache Camel and I have a use case that I am not quite sure how can be expressed or implemented using Camel
Use case:
Imagine you have designed an integration solution that takes files from an ftp, does some processing and uploads it to a queue. You chose Apache Camel to implement this solution and your route in Java DSL looks something like this:
from("ftp://user#hostname/directoryname")
.process(new Processor() {
public void process(Exchange exchange) throws Exception
{
//my fantastic prosessing goes here
}
}).to("jms:queue:queueName");
the route could be way more complex than this, but it doesn't matter here. Imagine your solution is such a spectacular success that there's a plan to implement a service where anyone could add his ftp server and get the files processed and uploaded to the queue. So what you want is
(Flexibility) Being able to dynamically add/remove servers from your app
(Scaling) Being able to handle potentially large number of such servers
Let's forget about #2 and focus on Flexibility part.
So the question is, I suppose:
How to dynamically (at runtime) add/remove endpoints to/from Apache Camel route?
What I considered so far:
First, I admit I am not that familiar with Integration Patterns, but just scanning the catalogue, the only thing that kind of could fit the bill is the Content Enricher. It can take a message and just go off to somewhere else and bring sth else. So I was thinking if someone adds an ftp server, the connection details could be encapsulated in the message and a Content Enricher could then connect to that ftp server and fetch files and push it further through the route.... so it would effectively be a Content Enricher capable of connecting to multiple ftp servers.... That kind of sound wrong. First I don't think this is the intention behind that pattern and secondly since there's ftp Component in Camel, I should somehow be able to use it in that scenario
The second approach would be to break the route into two like using the vm component, like this:
from("ftp://user#hostname/directoryname").to("vm:internalQ");
from("vm:internalQ")
.process(new Processor() {
public void process(Exchange exchange) throws Exception
{
//my fantastic prosessing goes here
}
}).to("jms:queue:queueName");
so now, I can create many routes with ftp endpoints that write to that internal queue so it can be picked up. Adding route dynamically to CamelContext seems to be possible (Add camel route at runtime in Java). Is that the way to go? Or am I just trying to use Camel in a way that it was not designed to?
You can dynamically add routes to your CamelContext:
MyRouteBuilder trb = new MyRouteBuilder(servletEndpoint, mockEndpoint);
camelContext.addRoutes(trb);
And MyRouteBuilder:
MyRouteBuilder(Endpoint servletEndpointStart, MockEndpoint mockEndpointEnd, String allowedParameters){
this._servletEndpoint = servletEndpointStart;
this._mockEndpoint = mockEndpointEnd;
}
#Override
public void configure() throws Exception {
from(this._servletEndpoint)
.id(TESTING_ROUTE_NAME)
.process(new Processor(){ // some processor })
.to(_mockEndpoint);
}
You can also modify the route, but you will need to restart it, in order to work properly, checkout how it is done in:
org.apache.camel.model.RouteDefinition.adviceWith(ModelCamelContext, RouteBuilder)
I will clarify my question.
I have a task to integrate two systems: a frontend serving html and backend which gives data to frontend.
Backend have a very large REST api so I have to use multiple routes.
I planned to use single camel context and wrap all routes into it.
<camelContext xmlns="http://activemq.apache.org/camel/schema/spring">
<from uri="direct:data"/>
<to uri="ahc:http://localhost/data"/>
<!--And so on. More than 70 routes-->
</camelContext>
Then, I planned to invoke the route using #Produce annotation on service method as adviced in Hiding middleware article
public interface Service {
String data();
}
public class MyBean {
#Produce(uri = "direct:data")
protected Service producer;
public void doSomething() {
// lets send a message
String response = producer.data();
}
}
As I understand information taken from here and here I'll end up with additional 70 thread in my app (one for each route). I fear that it can cause a serious performance hit and while the backend api will grow the thread number will grow with it. Is it correct? How can I avoid this if it's true? As I understand, I can't employ ExecutorService thread pool in this case.
Thanks in advance for any answer.
No you will not end up with a thread per route. The threading module is often tied to the threading model of the consumer (eg the route input).
For example a route that uses a timer component will use a scheduled thread pool (1 thread). And the JMS component will use 1 or more threads, depending on if you set concurrentConsumers=N, etc.
The direct component is like a direct method invocation and it uses the caller thread, so there is 0 new threads for that threading model.
If all your 70 routes uses the AHC in the < to > then you may want to re-use the same endpoint, so you reuse the thread pool of the AHC library. Or alternative to configure a shared pool to be used for all AHC endpoints.
And btw this question was also posted on the Camel user forum / mailinglist: http://camel.465427.n5.nabble.com/Can-multiple-camel-routes-cause-a-very-large-number-of-threads-tp5736620.html
You'll have to excuse me if I'm describing this incorrectly, but essentially I'm trying to get a service-like class to be instantiated just once at server start and to sort of "exist" in the background until it is killed off at server stop. At least from what I can tell, this is not exactly the same as a typical servlet (though I may be wrong about this). What's even more important is that I need to also be able to access this service/object later down the line.
As an example, in another project I've worked on, we used the Spring Framework to accomplish something similar. Essentially, we used the configuration XML file along with the built-in annotations to let Spring know to instantiate instances of some of our services. Later down the line, we used the annotation #Autowired to sort of "grab" the object reference of this pre-instantiated service/object.
So, though it may seem against some of the major concepts of Java itself, I'm just trying to figure out how to reinvent this wheel here. I guess sometimes I feel like these big app frameworks do too much "black-box magic" behind the scenes that I'd really like to be able to fine-tune.
Thanks for any help and/or suggestions!
Oh and I'm trying to run this all from JBoss 6
Here's one way to do it. Add a servlet context listener to your web.xml, e.g.:
<listener>
<listener-class>com.example.BackgroundServletContextListener</listener-class>
</listener>
Then create that class to manage your background service. In this example I use a single-threaded ScheduledExecutorService to schedule it to run every 5 minutes:
public class BackgroundServletContextListener implements ServletContextListener {
private ScheduledExecutorService executor;
private BackgroundService service;
public void contextInitialized(ServletContextEvent sce) {
service = new BackgroundService();
// setup single thread to run background service every 5 minutes
executor = Executors.newSingleThreadScheduledExecutor();
executor.scheduleAtFixedRate(service, 0, 5, TimeUnit.MINUTES);
// make the background service available to the servlet context
sce.getServletContext().setAttribute("service", service);
}
public void contextDestroyed(ServletContextEvent sce) {
executor.shutdown();
}
}
public class BackgroundService implements Runnable {
public void run() {
// do your background processing here
}
}
If you need to access the BackgroundService from web requests, you can access it through the ServletContext. E.g.:
ServletContext context = request.getSession().getServletContext();
BackgroundService service = (BackgroundService) context.getAttribute("service");
Have you considered using an EJB 3.1 Session bean? These can be deployed in a war file, and can be annotated with #Singleton and #Startup.
A number of annotations available with EJB 3.1 are designed to bring Spring goodies into the Java EE framework. It may be the re-invention you're considering has been done for you.
If you must roll your own, you can create a servlet and configure it start up when the application does using load-on-startup. I built a system like that a few years ago. We then used the new(ish) java.util.concurrent stuff like ExecutorService to have it process work from other servlets.
More information about what you're trying to do, and why the existing ways of doing things is insufficient, would be helpful.
You can use messaging for that. Just send message to the queue, and let the message listener do the processing asynchronously in the background.
You can use JMS for the implementation, and ActiveMQ for the message broker.
Spring has JMSTemplate, JMSGateWaySupport API to make JMS Implementation simple
http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/jms.html