I have a Camel application and I'm trying to do some aggregation based on some outputs from different responses (REST Web services).
This is what I have so far (the Camel routes):
#Component
public final class AggregationRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
rest("/aggregation")
.get()
.to("direct:retrieve");
from("direct:retrieve")
.multicast(/*new BodyAggregationStrategy(), true*/)
.to("direct:foo")
.to("direct:foo1");
from("direct:foo")
.to("seda:http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?sector=conus")
.to("direct:aggregate");
from("direct:foo1")
.to("seda:http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?sector=conus")
.to("direct:aggregate");
from("direct:aggregate")
.aggregate(header("id"), new BodyAggregationStrategy())
.log(LoggingLevel.WARN, simple("${body}").getText());
}
}
...the "aggregation strategy"
public final class BodyAggregationStrategy implements AggregationStrategy {
#Override
public Exchange aggregate(final Exchange oldExchange, final Exchange newExchange) {
if (null == oldExchange) {
return newExchange;
}
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newExchange.getIn().getBody(String.class);
oldExchange.getIn().setBody(oldBody + "+" + newBody);
return oldExchange;
}
}
...eventually, the Web services are going to be different, but I'm just trying now to see if I can solve this basic trouble first.
I defined a REST endpoint, when GET /aggregation is hit (on my side), I want to consult two or more REST Web services and aggregate the response from those; then "answer" back.
Any clues?
The multicast EIP has built-in aggregator, so configure the aggregation strategy on this pattern, instead of using a separate aggregator. This ensures the messages are mutlicasted and aggregate as part of the same unit of work and the result can be visible and send back to the calling REST client.
Related
I'm trying to use request/response pattern for Spring Boot using AMQP and Spring-web. I have client service that has #RestController and AsyncRabbit configuration with Direct Exchange, Routing key etc. and server that has simple listener for request queue.
Client (something like rest gateway controller):
#RestController
public class ClientController {
#GetMapping("/test1")
public String getRequest() {
ListenableFuture<String> listenableFuture = asyncRabbitTemplate.convertAndReceiveAsType(
directExchange.getName(),
routingKey,
testDto,
new ParameterizedTypeReference<>() {}
);
return listenableFuture.get(); // Here I receive response from server services
}
#GetMapping("/test2")
public String getRequest2() {
ListenableFuture<String> listenableFuture = asyncRabbitTemplate.convertAndReceiveAsType(
/* Same properties but I use another DTO object for request */
);
return listenableFuture.get()
}
Server:
#RabbitListener(queues = "#{queue.name}", concurrency = "10")
#Component
public class Consumer {
#RabbitHandler
public String receive(TestDto testDto) {
...
}
#RabbitHandler
public String receive2(AnotherTestDto anotherTestDto) {
...
}
}
How should I implement Rabbit listener to process each REST request?
I found only two ways to do that:
Using #RabbitHandler as in the example above. But for each request method (GET, POST, etc.) I need unique DTO class to send message and process it in correct handler even if "request body" is almost same (number of request methods = number of DTO class to send). I'm not sure that is right way.
Leave one Consumer and call desired method to process that depends on message body (trivial if-else):
...
#RabbitListener(queues = "#{queue.name}")
public String receive(MessageDto messageDto) {
if (messageDto.requestType == "get.method1") {
return serverService.processThat(messageDto);
} else if (messageDto.requestType == "post.method2") {
return serverService.processAnother(messageDto);
} else if ...
...
}
...
But add new if-else branch every time is not very convenient so I really out of ideas.
You may consider to use different queues for different request types. All of them are going to be bound to the same direct exchange, but with their respective routing key.
What you would need on the consumer side is just to add a new #RabbitListener for respective queue. And bind that queue to the exchange with its routing key.
That's actually a beauty of the AMQP protol by itself: the producer always publish to the same exchange with respective routing key. The consumer registers its interest for routing keys and binds a queue. The rest of routing logic is done on the AMQP broker.
See more info in docs: https://www.rabbitmq.com/tutorials/tutorial-four-spring-amqp.html
Introduction
I would like to be able to have two different spring profiles, and depending on the profile to change to a hardcoded address for our feign builders.
Currently was have the following:
return builder.target(cls, "http://" + serviceName);
But I would actually like to do the following and over-ride the address:
return builder.target(cls, "http://our-server:8009/" + serviceName);
Why
Sometimes we don't want to run all the services within our development environment. Additionally, some of the services are only available through a zuul gateway sometimes.
So we run the same code in different situations and conditions.
Technical Details
We have the following code that we use for building our Feign Clients.
We had been using the #FeignClient annotation in the past, but lately we decided to start building our feignClients manually.
Example below:
#FeignClient(name = "ab-document-store", configuration = MultiPartSupportConfiguration.class, fallback = DocumentStoreFallback.class)
We call the feignRegistrar class with the following command:
return registerFeignClient(DocumentStoreClient.class, true);
#RequiredArgsConstructor
//#Component
#Slf4j
public class FeignRegistrar {
#Autowired
private Decoder decoder;
#Autowired
private Encoder encoder;
#Autowired
private Client client;
#Autowired
private Contract feignContract;
#Autowired
private ObjectFactory<HttpMessageConverters> messageConverters;
#Autowired
private List<RequestInterceptor> interceptors;
public <T> T register(Class<T> cls, String serviceName, boolean isDocumentStore) {
if(isDocumentStore){
encoder = new MultipartFormEncoder(new SpringEncoder(messageConverters));
}
//Client trustSSLSockets = new Client.Default(getSSLSocketFactory(), new NoopHostnameVerifier());
Feign.Builder builder = Feign.builder()
.client(client)
.encoder(encoder)
.decoder(decoder)
.contract(feignContract)
.logger(new Slf4Logger())
.logLevel(Logger.Level.HEADERS);
builder.requestInterceptor(new RequestInterceptor() {
#Override
public void apply(RequestTemplate template) {
template.header("X-Service-Name", serviceName);
}
});
for(RequestInterceptor interceptor : interceptors) {
builder.requestInterceptor(interceptor);
}
log.debug("Registering {} - as feign proxy ", serviceName);
return builder.target(cls, "http://" + serviceName);
}
public static class Slf4Logger extends Logger {
#Override
protected void log(String configKey, String format, Object... args) {
log.info("{} - {}", configKey, args);
}
}
}
Spring Cloud Property Over-ride
We have also been using property files such as application-ENV.property with entries such as the following:
ab-document-store.ribbon.NIWSServerListClassName:com.netflix.loadbalancer.ConfigurationBasedServerList
ab-document-store.ribbon.listOfServers: localhost:8025
Unfortunately, listOfServers is not enough for us. We would like to be able to assign a directory/path as well. Something like:
ab-document-store.ribbon.listOfServers: localhost:8025/ab-document-store
Otherworkaround
I have thought about sneaking in a header into all requests such as X-SERVICE-NAME using a feign interceptor. Then we could point all services to an address (e.g. localhost:9001) , and forward/proxy those requests to localhost:9001/X-SERVICE-NAME.
However, I would prefer a much easier solution such as:
ab-document-store.ribbon.listOfServers: localhost:8025/ab-document-store
But this doesn't work :(
Introduction
I found a solution for this using a proxy that detects a header.
So, I have a feign interceptor on the java-side that attaches a header x-service-name to every feign-request.
I also have a NodeJS proxy, that analyzes requests, finds x-service-name, and re-writes the requests to become: x-service-name/originalRequestPath.
This allows me to have all the microservices behind a zuul gateway but also access them using a eureka-over-ride.
Java-Feign-Interceptor
Feign.Builder builder = Feign.builder()
.client(client)
.encoder(usedEncoder)
.decoder(decoder)
.contract(feignContract)
.logger(new Slf4Logger())
.logLevel(Logger.Level.HEADERS);
builder.requestInterceptor(new RequestInterceptor() {
#Override
public void apply(RequestTemplate template) {
template.header("X-Service-Name", serviceName);
}
});
NodeJS proxy
In the example, my zuul gateway ( or another proxy ) is on localhost:9001.
I'm listening on localhost:1200 .
let enableProxyForJava = process.env.ENABLE_PROXY_FOR_JAVA;
if (enableProxyForJava != undefined && enableProxyForJava.toLowerCase() === 'true') {
var httpProxyJava = require('http-proxy');
var proxyJava = httpProxyJava.createProxy();
gutil.log( gutil.colors.green('Enabling Proxy for Java. Set your Eureka overrides to localhost:1200.') );
require('http').createServer(function(req, res) {
console.log("req.headers['x-service-name'] = " + req.headers['x-service-name']);
console.log("Before req.url:"+ req.url);
if( req.headers['x-service-name'] != undefined){
let change = req.headers['x-service-name'] +req.url;
console.log("After req.url:"+ change);
req.url = change;
}
proxyJava.web(req, res, {
target: 'http://localhost:9001/'
});
}).listen(1200);
}
Property file inside Java Application that has feign clients
mbak-microservice1.ribbon.NIWSServerListClassName:com.netflix.loadbalancer.ConfigurationBasedServerList
mbak-microservice1.ribbon.listOfServers: localhost:1200
mbak-microservice2.ribbon.NIWSServerListClassName:com.netflix.loadbalancer.ConfigurationBasedServerList
mbak-microservice2.ribbon.listOfServers: localhost:1200
mbak-document-store.ribbon.NIWSServerListClassName:com.netflix.loadbalancer.ConfigurationBasedServerList
mbak-document-store.ribbon.listOfServers: localhost:1200
I am going to do send my DATA toRabbitMq producer(message sender) and get responsible data from RabbitMq consumer(message receiver). producer part is working fine .now my problem is how to implement consumer part (receiver part) in side the Spring boot API. .Below is My spring boot API and i written ProducerAndConsumer one class.
ProducerAndConsumer.class
#Component
public class ProducerAndConsumer {
#Autowired
private RabbitTemplate rabbitTemplate;
//MessageProducer part (send part)
public boolean sendMessage(String message) {
rabbitTemplate.convertAndSend(RobbitMqConfig.ROUTING_KEY, message);
System.out.println("Is listener returned ::: ==========="+rabbitTemplate.isReturnListener());
return rabbitTemplate.isReturnListener();
}
//Consumer part (receiver part)
#RabbitListener(queues = RobbitMqConfig.QUEUE_NAME1)
public void receiveMessage ( final Message message){
System.out.println("Received message====Receiver=====" + message.getPayload());
}
}
API part
#PostMapping(value = {"/sendFilesName"})
public ResponseEntity<?> sendFilesName(#RequestBody SendFileNameRequest sendFileNameRequest, HttpServletRequest request) throws ParseException {
System.out.println("FileNameArray="+sendFileNameRequest.getFileNameArray());
if(sendFileNameRequest.getFileNameArray().size()!=0) {
List<String> message = sendFileNameRequest.getFileNameArray();
**//see here i send my message array data**
if(producerAndConsumer.sendMessage(message.toString())){
**//here i want implement my receiver part how to?**
return ResponseEntity.ok(new ApiResponse(true, "fileName List sent successfully", "",true));
}else {
return ResponseEntity.ok(new ApiResponse(false, "fileName List sent Fails", "",true));
}
}else {
return ResponseEntity.ok(new ApiResponse(false, "fileName List not present ", "",true));
}
}
The routing algorithm behind a direct exchange is simple - a message goes to the queues whose binding key exactly matches the routing key of the message.
spring amqp
Note: Check the routing key and queues binded using rabbitmq admin console to figure out whats going on or share the rabbitmq configuration.
I have a route that should process request from CXF endpoint and return results as JSON:
public class MyRoute extends RouteBuilder
{
// ... Autowired:
// msgRequestProcessor - converts json {string,string} to String of "\n" delimited json string: {string}\n{string}
// RangeProcessor, SingleProcessor - create mongodb Criteria object from json string
// msgTypeMapper - adds corresponding header "msg.type"
#Override
public void configure()
{
from("direct:list")
.process(msgRequestProcessor)
.split(body())
.bean(msgTypeMapper.class)
.choice()
.when(header("msg.type").isEqualTo("single"))
.log("Go to direct:single")
.to("direct:single")
.otherwise()
.log("Go to direct:range")
.to("direct:daterange")
.end()
.to("direct:aggregate");
from("direct:range")
.process(new RangeProcessor());
from("direct:single")
.process(new SingleProcessor());
from("direct:aggregate")
.aggregate(new MyAgg()).header("msg.collection").completionSize(2)
.log("RETVAL: ${body}")
.marshal().json(JsonLibrary.Gson).end();
}
public static final class MyAgg implements AggregationStrategy {
#Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange)
{
if (oldExchange == null) {
return newExchange;
}
Criteria oldCriteria = oldExchange.getIn().getBody(Criteria.class);
Criteria newCriteria = newExchange.getIn().getBody(Criteria.class);
Criteria criteria = new Criteria();
criteria.andOperator(oldCriteria, newCriteria);
oldExchange.getIn().setBody(criteria.getCriteriaObject().toString());
return oldExchange;
}
}
}
Everything works fine, I see correct aggregation results and aggregation completion in the log
but CXF endpoint always returns output of msgRequestProcessor (before split):
{"string"}
{"string"}
while I expect to see Criteria object converted to string (that I can see in the logs).
Any help would be much appreciated! Thanks.
Note first that your indentation is misleading, the end() is really the end of the choice(), and not of the split(); I was puzzled a while by this (as #Ralf was maybe.)
Now, the aggregation works, but its result is discarded because indeed the result of the split is the input message.
For a request/reply usage of the splitter (in-out), you really have to declare the aggregation strategy along with the split() as explained here (same misleading indentation).
In the official documentation you mention, the situation is the other way around (in-only): the result of the splitter is discarded, and the result of the aggregation is routed downstream.
Im new to Rest web services and say Ive created this web service using Netbeans
#Path("browse")
#Stateless
public class ArticleBrowseResource {
#EJB
private ArticleSearcherLocal ejbRef;
#GET
#Produces(MediaType.APPLICATION_XML)
public List<Article> browse(#DefaultValue("") #QueryParam("username") String username,#QueryParam("sd") String sd) {
// convert sd string to date
List<Article> articles = ejbRef.search(username, date);
return articles;
}
}
where Article is an entity which is anotated with #XmlRootElement
Now how am I supossed to retreive this list of articles in my client which for simplicity lets just say it is a java standard application? In SOAP web services I know that these objects are automatically generated but not in Rest.
This is the client class generated for this service by Netbeans
public class ArticleBrowseClient {
private WebResource webResource;
private Client client;
private static final String BASE_URI = "http://localhost:8080/cityblog/rest";
public ArticleBrowseClient() {
com.sun.jersey.api.client.config.ClientConfig config = new com.sun.jersey.api.client.config.DefaultClientConfig();
client = Client.create(config);
webResource = client.resource(BASE_URI).path("browse");
}
public <T> T browse(Class<T> responseType, String username, String sd) throws UniformInterfaceException {
WebResource resource = webResource;
if (username != null) {
resource = resource.queryParam("username", username);
}
if (sd != null) {
resource = resource.queryParam("sd", sd);
}
return resource.accept(javax.ws.rs.core.MediaType.APPLICATION_XML).get(responseType);
}
public void close() {
client.destroy();
}
}
What is the best and simplest way to resolve this issue?
Any help is appreciated and
thx in advance
Please try fewer code generation and more understanding of what you are actually doing. On the server, you generate a XML message with help of JAXB. On the client side, you can consume this XML with a programming language and library you like. Just use tools like curl to see what is going actually over "the wire". Your generated client site looks fully reasonable. You just need your Article class from the server side on the client side. The generated code uses Jersey which can read XML messages per JAXB per default. So just drop your server side Article class in your client side classpath and use it. But please also have a look at the wire level protocol to understand the portability of your REST API.