I have a general question about Apache Camel. I wasn't able to find whether the aggregator is transacted. If it is transacted, how the transactions are implemented and how fast the aggregation is?
Sending the messages into the aggregator can run in a transaction.
You would need a persistent store with the aggregator to let the outgoing messages act as a transaction. See the documentation about persistence
http://camel.apache.org/aggregator2
For example there is a JDBC based and HawtDB (file based) persistent support out of the box. Its pluggable as you can also build your custom.
Camel in Action book chapter 8 and 9 convers this in much more details.
Related
I am working on design where the requirement is as follows
The application would be built using spring boot
As part of the requirement we would have to build some orchestration where we would end up calling multiple SOAP and REST services and we would perform aggregation, transformation etc.
When an exception/error occurs during the orchestration, we need to capture and persist the state and retry it from the same point where the error occurred.
For example, Lets say there is a Java method in which we have multiple calls to external components say 1 soap call and 2 rest calls i.e.
String doOrchestration(InputObject obj){
Line 1: Object obj = soap call[ input is XML format];
Line 2: Object obj1 = rest1 call [the input is obj in JSON format]
Line 3: Object obj3 = rest2 call[the input is obj1 in JOSN format]
//do something
return str
}
The error occurred on Line 2. We need to capture/persist the state and continue the execution from the Line 2 later.
Is there any workflow library, orchestration patterns and/or framework that we can integrate with spring/Spring-boot
Thanks,
well, one tecnology that i use to handle this kind of problems is apache camel, whith camel you can set routes,for instance in your requirement, you can build the main route who will orchest other routes such as yours endpoints, if one endpoint has an exception you can call the route who will persist that and call again the route of the endpoint, or something like that.
http://camel.apache.org/getting-started.html
edit: camel has a lot of potencial transforming messages such as json, xml, objects.
i have a very simple project on github with spring boot and camel if you want to check it.
https://github.com/chill-hub/spring_boot_camel_base_archetype
The Apache Camel version spring like is spring-integration at
https://spring.io/projects/spring-integration
Apache Camel and Spring Integration implements enterprise integration pattern called EIP. Please look at https://www.enterpriseintegrationpatterns.com/ first.
It is basically design patterns for workflow and orchestration !
You have an integration (starter) with spring boot made by spring named spring-boot-starter-integration.
You should probably look at Java Spring Config dsl reference at https://github.com/spring-projects/spring-integration-java-dsl/wiki/spring-integration-java-dsl-reference
which will show you a nice overview, with the more modern approach (xml, and annotation starts to disapear).
With what you need to do you will probably need https://docs.spring.io/spring-integration/reference/html/ws.html which is the web extension for spring integration.
Be careful it is a very elegant solution but with lot of abstraction and can be difficult at the beginning. Keep in mind EIP defines building block and vocabulary to build workflows, Apache Camel and Spring Integration are frameworks that implements these building blocks and offer a threading model abstraction (also queue persistant abstraction), and both have several extensions to adapt easily these building blocks to real word (web, ldap, databases, ...)
Both frameworks are good with pros and cons, but you will probably prefer spring integration in a spring ecosysteme team, same conventions than other spring modules !
Again theses framework are really elegant but abstraction has always a cost!
Disclaimer : I was a trainer for spring integration course - Official
I would like to migrate a multi-threaded application in JSE to Spring Integration but I have to clarify some points before. First of all, the application will have the following Spring integration components:
JMS to Transformer to router to TCPOut
TcpIn (to router) to Transformer to JMS
In this context, I have to load all the TCP connections dynamically from a configuration file. I saw a couple of example of this here in StackOverflow (based in the FTP sample). These samples could be enough for the first part but I am looking for how to do that in Spring Boot and what is the best (and elegant) way to create this type of configuration.
Finally, I have to access to each different context (this is maybe the most important) from a type of Swing monitor to start/stop manually this TCP connections. Is this possible? What do you suggest me to do?
All my current components are java based configuration (not DSL).
See my answers to this question and its follow-up for examples of how to dynamically create application contexts using Java Configuration.
Also, take a look at the new feature in the Java DSL for dynamically registering/removing integration flows with the context. The 1.2 version of the DSL, containing this feature, will be released shortly.
You can stop/start endpoints using JMX or a control bus, or programmatically.
we plan to use Kafka as a central component in our data warehouse given that the producer is able to handle transactions (in short: rollbacks and commits).
When googling Kafka + Transactions I find a lot of theoretical thoughts about the possibility of how Kafka could handle transactions but at the moment I do not see any function in the java API that supports commits and rollbacks for the producer.
Has anybody made some experiences with transactions and Kafka and can give me an hint?
I think what you are looking for is basically called transactional messaging in Kafka where producers are capable of creating session (aka transactional session) and send messages within the sessions. Hence it can choose to either commit / abort the transaction.
[Source]: Please read the wiki for details
Actually, from the last version 0.11.0.0 transactions are supported. See Guarantee unique global transaction for Kafka Producers
No; Kafka does not support transactions.
You can get certainty that a message has been produced to a partition, but once produced you are not able to rollback that message.
Since version 0.11.0 Apache Kafka supports transactions: https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
My questions is as follows:
I have a service, which queries the DB and retrieves some records. Then it updates an external system using that information and in the end in updates back the DB.
Using spring trascations and weblogic jta transaction manager i was able with sample code below not to loose any messages in case
No records are retrived. (these are mandatory for the External System)
External System Error
Failed to update the DB
So in all aboce cases the JMS Listener puts the errror message back to the queue.
My question, is there any better way, using Spring with all its goodies, to manage that? The sample code below throws explicilty a RuntimeException which i dont think is a good design...
Please for your comments.
EDIT:
The queue is being polled by the submissionListener MDP that its configuration is shown below. After the message is consumed it invokes the registerDocument() of the service. (another Spring bean). That service invokes 2 times the DAO and the external system.
Check out Spring's documentation on JmsTemplate and Message Driven POJOs for patterns in the core Spring framework.
Spring Integration models higher-level abstractions related to message-orientated patterns.
Are there any good examples of using AMQP client in CMT/JTA environment? I know that AMQP supports transactional messaging, but I don't know how it aligns with JTA.
So far I have managed to setup Apache Qpid java broker and connect to it from a Java application using latest JBossTS (Narayana 5.0.0.M1) as standalone JTA. Basic tests showed that messages successfully appear in a queue when I commit TX and are properly ignored when I rollback. But I'm not sure whether this proves that I can safely use Qpid client in CMT.
There is a linkedin AMQP Group where it's said:
"There started a sub spec of AMQP 1.0 concerning distributed tx which was heavily influenced by Microsoft. Totally incompatible with JTA."
That might be reason why you (and me too) haven't found any example.
source: https://www.linkedin.com/groups/AMQP-JTA-1369617.S.5995685814774882304?trk=groups_most_popular-0-b-ttl&goback=%2Egmp_1369617