I am trying to get familiar with EIP and Apache Camel and I have a use case that I am not quite sure how can be expressed or implemented using Camel
Use case:
Imagine you have designed an integration solution that takes files from an ftp, does some processing and uploads it to a queue. You chose Apache Camel to implement this solution and your route in Java DSL looks something like this:
from("ftp://user#hostname/directoryname")
.process(new Processor() {
public void process(Exchange exchange) throws Exception
{
//my fantastic prosessing goes here
}
}).to("jms:queue:queueName");
the route could be way more complex than this, but it doesn't matter here. Imagine your solution is such a spectacular success that there's a plan to implement a service where anyone could add his ftp server and get the files processed and uploaded to the queue. So what you want is
(Flexibility) Being able to dynamically add/remove servers from your app
(Scaling) Being able to handle potentially large number of such servers
Let's forget about #2 and focus on Flexibility part.
So the question is, I suppose:
How to dynamically (at runtime) add/remove endpoints to/from Apache Camel route?
What I considered so far:
First, I admit I am not that familiar with Integration Patterns, but just scanning the catalogue, the only thing that kind of could fit the bill is the Content Enricher. It can take a message and just go off to somewhere else and bring sth else. So I was thinking if someone adds an ftp server, the connection details could be encapsulated in the message and a Content Enricher could then connect to that ftp server and fetch files and push it further through the route.... so it would effectively be a Content Enricher capable of connecting to multiple ftp servers.... That kind of sound wrong. First I don't think this is the intention behind that pattern and secondly since there's ftp Component in Camel, I should somehow be able to use it in that scenario
The second approach would be to break the route into two like using the vm component, like this:
from("ftp://user#hostname/directoryname").to("vm:internalQ");
from("vm:internalQ")
.process(new Processor() {
public void process(Exchange exchange) throws Exception
{
//my fantastic prosessing goes here
}
}).to("jms:queue:queueName");
so now, I can create many routes with ftp endpoints that write to that internal queue so it can be picked up. Adding route dynamically to CamelContext seems to be possible (Add camel route at runtime in Java). Is that the way to go? Or am I just trying to use Camel in a way that it was not designed to?
You can dynamically add routes to your CamelContext:
MyRouteBuilder trb = new MyRouteBuilder(servletEndpoint, mockEndpoint);
camelContext.addRoutes(trb);
And MyRouteBuilder:
MyRouteBuilder(Endpoint servletEndpointStart, MockEndpoint mockEndpointEnd, String allowedParameters){
this._servletEndpoint = servletEndpointStart;
this._mockEndpoint = mockEndpointEnd;
}
#Override
public void configure() throws Exception {
from(this._servletEndpoint)
.id(TESTING_ROUTE_NAME)
.process(new Processor(){ // some processor })
.to(_mockEndpoint);
}
You can also modify the route, but you will need to restart it, in order to work properly, checkout how it is done in:
org.apache.camel.model.RouteDefinition.adviceWith(ModelCamelContext, RouteBuilder)
Related
Problem
Very short: I want to create Spring Shell, but as a web application.
I want to create a web-application (preferably using Spring Boot), where the frontend (ReactJS) looks like a terminal (or shell), and the backend processes inputted commands. Look at https://codepen.io/AndrewBarfield/pen/qEqWMq. I want to build a full web app for something that looks like that.
I want to build a framework, so that I can develop backend commands without knowing anything about the frontend/web application structure. I basically want to instantiate a "Terminal" object, where I give some kind of input-stream and output-stream. This way I can program this Terminal based on my given interfaces and structure, without the need of setting up all kind of front-end stuff.
A good summary of the question would be: how to send all keyboard inputs to the backend, and how to send all output to the frontend?
The reason I want to create a web application, is because I want it to be available online.
What I tried
I think the way of reaching this is using websockets. I have created a small web application using this (https://developer.okta.com/blog/2018/09/25/spring-webflux-websockets-react) tutorial, without the security part. The websocket part is almost suitable, I just cannot get an "input" and "output" stream-like object.
#Controller
public class WebSocketController {
private SimpMessagingTemplate simpMessagingTemplate;
#Autowired
public WebSocketController(SimpMessagingTemplate simpMessagingTemplate) {
this.simpMessagingTemplate = simpMessagingTemplate;
}
#MessageMapping("/queue")
#SendToUser("/topic/greetings")
public Greeting greeting(HelloMessage message, #Header(name = "simpSessionId") String sessionId) throws Exception {
System.out.println(sessionId);
// Do some command parsing or whatever.
String output = "You inputted:" + HtmlUtils.htmlEscape(message.getName());
return new Greeting(output);
}
private MessageHeaders createHeaders(String sessionId) {
SimpMessageHeaderAccessor headerAccessor = SimpMessageHeaderAccessor.create(SimpMessageType.MESSAGE);
headerAccessor.setSessionId(sessionId);
return headerAccessor.getMessageHeaders();
}
Now with this code, you can parse a command. However, it doesn't keep any "state". I don't know how it works with states and websockets.
I saw you had this Spring Sessions + WebSockets (https://docs.spring.io/spring-session/docs/current/reference/html5/guides/boot-websocket.html), but this is not really what I want.
I can send a message from the backend to the frontend by using this code:
simpMessagingTemplate.convertAndSendToUser(sessionId, "/topic/greetings", "hey", createHeaders(sessionId));
However, I want my terminal to be able to wait for input commands from the user. Seems like a stretch, but does anybody know how to achieve this?
What I sort of want
I basically want other people to program to this interface:
public interface ITerminal {
void setInputStream(Object someKindOfWrapperForTheInput);
void setOutputStream(Object someWrapperOfSimpMessagingTemplate);
void start();
}
When somebody opens the web application, they get a dedicated terminal object (so a single connection per user). Whever somebody enters a command in the frontend application, I want it to be received by the terminal object, processed, and response outputted to the frontend.
Reasons for doing this
I really like creating command-line applications, and I don't like building frontend stuff. I work as a software engineer for a company where we build a web application, where I mostly program backend stuff. All the frontend part is done by other people (lucky for me!). However, I like doing some projects at home, and this seemed cool.
If you have any thoughts or ideas on how to approach this, just give an answer! I am interested in the solution, using the SpringBoot framework is not a requirement. I ask this question using Spring Boot and ReactJS, because I have already built applications with that. A lot has been figured out already, and I think this probably exists as well.
The only requirement is that I can achieve this with Java on a tomcat-server. The rest is optional :)
Unclear?
I tried my best to make my story clear, but I am not sure if my purpose of what I want to achieve is clear. However, I don't know how to formulate it in such a way you understand. If you have any suggestions or questions, dont hesitate to comment!
If the only thing you want is a Live Spring shell that shows up in the browser it's fairly simple, all you need is to expose a standard WebSocket via the WebSocketConfigurer, then add a WebSocketHandler that executes the command and then returns the resulting String as a TextMessage.
Firstly the Socket configuration that allows clients to connect to the 'cli' endpoint
#Configuration
#EnableWebSocket
public class WebSocketConfiguration implements WebSocketConfigurer {
#Override
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(cliHandler(), "/cli").setAllowedOrigins("*");
}
#Bean
public CLIWebSocketHandler cliHandler() {
return new CLIWebSocketHandler();
}
}
Then the WebSocketHandler that executes the command. I recommend that for every #ShellMethod you specify the return type as String, don't use logging or System writes as they won't be returned during the evaluation.
#Component
public class CLIWebSocketHandler extends TextWebSocketHandler {
#Autowired
private Shell shell;
#Override
protected void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
String result = shell.evaluate(() -> message.getPayload()).toString();
session.sendMessage(new TextMessage(result));
}
}
You can use an extension like Simple WebSocket Client to test it, by going to ws://localhost:port/cli
This is the most basic solution, adding features like security should be easy after this. Notice that I don't use STOMP, because you probably want to isolate users. But it can work alongside STOMP based endpoints, so you can have pub-sub functionality for other parts of the project.
From the question I sense that answer you'd like is something that involved Input and OutputStreams. You could possibly look into redirecting the output of Spring Shell to a different stream then have them forwarded to the sessions but it's probably much more complicated and has other trade-offs. It's simpler to just return a String as the result, it looks better in print outs anyway.
Hope you have some spare minutes for my question. For the last couple of days I was reading about Camel and managed to setup everything up and running. Now, I have bumped on a tricky part :). Basically, I define a route using Java in runtime and put the route configuration in DB. Routes are working and messages are flowing from one side to another. But, when an exception occurs I would like to know where the exception has occurred (between which route endpoints), store the current exchange body (in-flight message that is useful for further processing) in the DB, update the message by the user and then retry the route execution from the point of exception (where it failed). Lets say that all route steps are idempotent.
My solution:
Make a Interceptor for the route
Granulate the route in as much as possible parts (each new step is a new route)
Between each step update the DB with the current and future step and current exchange body
If exception occurs, store the message in DB, update it using an editor
Send a message to a next route point (taken from DB) using ProducerTemplate
What do you think about this ? Is it doable or Camel cannot support me with this approach ?
Thank you for patience and your time.
Hope I was clear enough.
You can use Camel's tracer component. A detailed example meeting your needs is already available on camel's site : http://camel.apache.org/tracer-example.html
You should use onException() clause to tackle this. For eg :
public void configure() throws Exception{
//This is a global onException definition and will work for all routes defined in this calss
onException().process(new Processor(){
public void process(Exchang arg0){
Exception e = arg0.getProperty(Exchange.EXCEPTION_CAUGHT,Exception.class);
//get message and other properties that you are interested in
db.saveOrUpdate(/*Pass your custom object here*/);
}
}).handled(true);
from("direct:route1")
//some processing
.to("jms:route1");
from("direct:route2")
//some processing
.to("http://route2");
}
You might need to consult exact details at apache camel site, since i just wrote this code here.
I have an application I coded which I am refactoring to make better use of Spring Integration. The application processes the contents of files.
The problem (as I see it) is that my current implementation passes Files instead of Messages, i.e. Spring Integration Messages.
In order to avoid further rolling my own code, which I then have to maintain later, I'm wondering if there is a recommended structure for constructing Messages in Spring Integration. What I wonder is if there is some recommended combination of channel with something like MessageBuilder that I should use.
Process/Code (eventually)
I don't yet have the code to configure it but I would like to end up with the following components/processes:
Receive a file, remove header and footer of the file, take each line and convert it into a Message<String> (This it seems will actually be a Splitter) which I send on to...
Channel/Endpoint sends message to Router
Router detects format String in Payload and routes to the appropriate channel similar to Order Router here...
Selected channel then builds appropriate type of Message, specifically typed messages. For example I have the following builder to build a Message...
public class ShippedBoxMessageBuilder implements CustomMessageBuilder {
#Override
public Message buildMessage(String input) {
ShippedBox shippedBox = (ShippedBox) ShippedBoxFactory.manufactureShippedFile(input);
return MessageBuilder.withPayload(shippedBox).build();
}
...
Message is routed by type to the appropriate processing channel
My intended solution does seem like I've complicated it. However, I've purposefully separated two tasks 1) Breaking a file into many lines of Messages<String> and 2) Converting Messages<String> into Messages<someType>. Because of that I think I need an additional router/Message builder for the second task.
Actually, there is MessageBuilder support in the Spring Integration.
The general purpose of such Frameworks is to help back-end developers to decouple their domain code from messaging infrastructure. Finally, to work with Spring Integration you need to follow the POJO and Method Invocation principles.
You write your own services, transformers and domain models. Then you just use some out of the box compoenents (e.g. <int-file:inbound-channel-adapter>) and just refer from there to your POJOs, but not vise versa.
I recommend you to read Spring Integration in Action book to have more pictures on the matter.
Can you explain the reason to get deal with Spring Integration components directly?
UPDATE
1) Breaking a file into many lines of Messages
The <splitter> is for you. You should write some POJO which returns List<String> - the lines from your file without header and footer. How to read lines from File isn't a task of Spring Integration. Especially, if the "line" is something logical, not the real file line.
2) Converting Messages into Messages
One more time: there is no reason to build Message object. It's just enough to build new payload in some transformer (again POJO) and framework wrap to its Message to send.
Payload Type Router speaks for itself: it checks a payload type, but not Message type.
Of course, payload can be Message too, and even any header can be as well.
Anyway your Builder snapshot shows exactly a creation of plain Spring Integration Message in the end. And as I said: it will be enough just to transform one payload to another and return it from some POJO, which you will use as a transformer reference.
Please excuse stupidity as this is my first Camel application
To respond to a web request, I am sourcing the content from two different sources.
I am, therefore, making a multicast request to two methods and parallelizing it.
The response is an marshalled JSON object (using camel-jackson)
All works fine.
public class RestToBeanRouter extends RouteBuilder{
#Override
public void configure() throws Exception {
from("cxfrs://bean://rsServer")
.multicast()
.parallelProcessing()
.aggregationStrategy(new CoreSearchResponseAggregator())
.beanRef("searchRestServiceImpl", "firstMethod")
.beanRef("searchRestServiceImpl", "secondMethod")
.end()
.marshal().json(JsonLibrary.Jackson)
.to("log://camelLogger?level=DEBUG");
}
Question :
The Multicast routing expects a to in the DSL. Currently, I am mapping this to a log endpoint. Is this fine?
Since I am not using the to and the last exchange of the Aggregator strategy is the one which is returned to the user, should my endpoint be configured to something else - like a null or something? (Ah, the stupidity kicks in)
For the benefit of SO visitors, copying the solution given in the Camel mailing list here :
by Robert Simmons Jr. MSc. - Lead Java Architect # EA
Author of: Hardcore Java (2003) and Maintainable Java (2012)
The aggregated exchange is the one that gets returned and how the
aggregated exchange is created depends on the aggregation strategy you use.
When a route stops either by calling stop or merely not routing anymore,
the exchange on the last part of the route could be considered a reply. In
most cases it will reply back to the caller (unless you set a reply-to
destination in a JMS based route or some other cases). In your case if all
you want to do is return the enriched exchange then you dont need any to()
call. Just stop after the marshal.
I'm trying to start a Camel route on GAE and am running into one brick wall after the next. First I tried a route that looked like this:
from("direct:start")
.process(new Processor() {
#Override
public void process(Exchange exchange) {
logger.info("I made it!");
}
})
.to("direct:end");
But that doesn't do anything. Being new to both Camel and GAE, I suspect that's because the direct component doesn't act as a consumer for the beginning of a route. So then I tried kicking off a timer:
from("timer://runOnce?repeatCount=1")
.process(new Processor() {
#Override
public void process(Exchange exchange) {
logger.info("I made it!");
}
})
.to("direct:end");
Only to get a nasty error from GAE:
Error: access denied (java.lang.RuntimePermission modifyThreadGroup)
And it turns out you can't create new Thread instances on GAE, and that's exactly what the Timer component does.
So then I tried kicking the route off with a bean:
public class DummyBean {
public void kickoffRoute() { return; }
}
// Inside the method that creates and starts the Camel route
SimpleRegistry reg = new SimpleRegistry();
DummyBean bean = new DummyBean();
reg.put("dummy", bean);
CamelContext camel = new DefaultCamelContext(reg);
camel.disableJMX();
// Inside my RouteBuilder
from("bean:dummy")
.process(new Processor() {
#Override
public void process(Exchange exchange) {
logger.info("I made it!");
}
})
.to("direct:end");
I ran this, only to get the same "access denied" error as from before with the Timer.
All I'm trying to do is kick off a route from inside GAE so I can start getting familiar with both technologies (Camel & GAE). But for the life of me, I can't figure out how to do this - every Camel component seems to create threads, which are illegal on GAE! So I ask:
How do you even kickoff a route (an initial producer - Timer, or otherwise) on GAE? I see there is a Camel-GAE component, and I'm certainly willing to read up on it and learn how to use it, so that I could have routes starting with, say, a ghttp:///startRoute endpoint, but I'm really just starting out here and am hoping for an easier component/endpoint to work with.
If both Timer and Bean components create threads, and this is disallowed on GAE, I have a sick feeling in my stomach that most/all Camel components are disallowed on GAE. Is this the case?!?! If so, how do you build useful/meaningful routes on GAE? Or is there some GAE "trick" to getting Camel to run and create threads? For instance, I know GAE backends do not suffer the same threading restrictions as frontend instances, etc.
Thanks in advance!
Camel is built to run on Java environments using Java SE 6/7 with whatever that implies in file system access, thread creation, possibilities to open ports etc.
Google App Engine has, as you realised, several restrictions. That does not make Camel unusable at all. You can use a lot of the routing features, producers endpoints (.to(..)), transformations, etc etc.
In fact, in Java EE Application Servers, there are similar restrictions, but that does not prevent usage of camel.
If you really are up for it, and need to do everything camel inside GAE, it's perhaps possible to make jack in Camels pluggable threading models into GAE.
Disclaimer: I have never used GAE tasks, but there is some documentations here
https://developers.google.com/appengine/docs/java/taskqueue/
http://camel.apache.org/threading-model.html
You can also try to start the events by whatever feature you have in GAE and just use a producer template to kick start a Camel route.