I have a requirement where the complete flow of different async requests is related with one trackingId across different services which communicate asynchronously with the help of messages, And I need to have a single traceId for all requests related to one flow or trackingId.
And I am facing issues with getting/setting TraceContext from/into Tracer, I wrote below code but it creates a new traceId for each request.
TraceContext traceContext = (TraceContext) contextMap.get(trackingId);
if (traceContext == null)
{
tracing.tracer().startScopedSpan(trackingId);
traceContext = tracing.currentTraceContext().get();
} else
{
tracing.tracer().startScopedSpanWithParent(trackingId, traceContext);
}
contextMap.put(trackingId, traceContext);
I'm not 100% sure what do you want to achieve here, as far as I can understand, what you described is the default behavior, please see the docs.
The tracing context is propagated out of the box, so all the related spans have the same traceId, please see the docs how you can set this up for messaging, there is also a sample project for messaging.
Related
My question is related to finding a best practice to include data persistence inside an integration flow while returning the Message object so that it can be further processed by the flow.
Let's consider the following flow:
#Bean
IntegrationFlow myFlow() {
return flowDefinition ->
flowDefinition
.filter(filterUnwantedMessages)
.transform(messageTransformer)
.wireTap(flow -> flow.trigger(messagePayloadPersister)) <--- here is the interesting part
.handle(terminalHandler);
}
The wide majority of cases, instead of the wireTap I have seen in some projects, a Transformer is used to persist data, which I do not particulary like, as
the name implies transformation of a message, and persistence is something else.
My wish is to find out alternatives to the wireTap, and a colleague of mine proposed using #ServiceActivator:
#Bean
IntegrationFlow myFlow() {
return flowDefinition ->
flowDefinition
.filter(filterUnwantedMessages)
.transform(messageTransformer)
.handle(messagePayloadPersister)
.handle(terminalHandler);
}
#Component
class MesssagePayloadPersister {
#ServiceActivator <--- interesting, but..
public Message handle(Message<?> msg) {
//persist the payload somewhere..
return message;
}
}
I like the flow, it looks clean now, but also I am not 100% happy with the solution, as I am mixing DSL with Spring.
Note: org.springframework.messaging.MessageHandler is not good because the handle method returns void so it is a terminal part to the flow. I need a method that returns Message object.
Is there any way to do this?
Need to understand what you are going to do with that persisted data in the future.
And what information from the message you are going to store (or the whole message at all).
See this parts of documentation - may be something will give you some ideas:
https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/system-management.html#message-store
https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/system-management.html#metadata-store
https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/message-transformation.html#claim-check
https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/core.html#persistent-queuechannel-configuration
https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/jdbc.html#jdbc-outbound-channel-adapter
With the last one you may need to consider to use a publishSubscribeChannel() of the Java DSL to be able to store in the DB and have a second subscriber to continue the flow.
I am receiving a MonoError caused by an IllegalStateException with the cause 'The underlying HTTP client completed without emitting a response.'.
NetworkList is a class contains lists of a network topology and has a setList which is called from within the method convertJsonToPojo.
client is a WebClient (spring-webflux) in Spring 5 (5.0.4.RELEASE) using reactor-netty (0.7.5.RELEASE).
The following snippet shows how I am using WebClient to make asynchronous calls to type = BOARDS, CARDS, PORTS, ROUTERS, etc where the network name is passed in as projectName and the topology type is passed in as type.
NetworkList netList = new NetworkList();
client.get().uri("/{project}/{type}",projectName,type).retrieve().bodyToMono(String.class).subscribe(json -> this.convertJsonToPojo(Type.BOARD, json, netList));
The rest service I am hitting against is a singular service which returns json representing the components asked for.
This seems similar to Reactive WebClient Not Emitting A Response but a response within points to the following reactor-netty issue Issue 138. That issue indicates fixed as of 0.6.6.RELEASE.
So my question is am I using WebClient wrong unknowingly or should I submit this as an unfixed bug? I am fairly new to Spring 5 and the Reactive library so do not want to naively submit a bug.
Thank you for your time in advance!
PS - I have this working with RestTemplate and CompletableFuture so no need to offer that suggestion as an alternative. My tasking is to begin the migration to a reactive architecture. Thank you!
UPDATE - I believe I understand now what is going on but would like a confirmation from someone. The rest service I am hitting against is a traditional one and not a Reactive Stream. Thus, there are no 'stage' updates being shared during the call. .subscribe() is expecting a reactive stream and thus never received the complete stage. replacing .subscribe() with .block() forced a completion and the code works now. So my learning from this is that when dealing with traditional rest services a block is required. Can someone confirm that please? Thank you!
I've been looking at using RabbitMQ for cross-service messaging. I've been able to configure our Exchanges / Queues / DLX etc. using Spring annotations. Example (simple) queue listener:
#RabbitListener(queues = RabbitMessageType.QueueNames.SMS_NOTIFICATIONS)
public void receive1(Message message) throws Exception {
RabbitMessageDto messageDto = OBJECT_MAPPER.readValue(message.getBody(), RabbitMessageDto.class);
SmsNotificationDto payload = OBJECT_MAPPER.readValue(messageDto.getPayload(), SmsNotificationDto.class);
log.info(payload.getMessage());
}
I'm using spring-cloud-sleuth to generate correlationIds / traceIds, which are preserved when using HTTP requests to talk to other services, enabling us to trace the given ID throughout the logs of our various microservices.
While I can get the current traceId and insert that into my DTO:
#Autowired
private Tracer tracer;
private RabbitMessageDto createRabbitMessageWithPayload(String messageType,
String messageVersion,
Object payload) {
return new RabbitMessageDto.Builder()
.withTraceId(tracer.getCurrentSpan().getTraceId())
.withDtoName(messageType)
.withDtoVersion(messageVersion)
.withPayload(payload)
.build();
}
I cannot find a way to set the traceId in the receiving method.
Googling keeps bringing me to spring-cloud-stream and spring-cloud-stream-starter-rabbit; documentation seems to indicate that it's possible automatically insert / set traceIds, but I'm not familiar with spring-cloud-stream at all, and don't find the documentation particularly helpful.
So, I would love answers to the following:
Using SpanAdjuster or Tracer etc; can I set the traceId based on the value in my DTO?
Using spring-cloud-stream, can I automagically insert / retrieve the traceId, and where would I start?
So, incase someone comes across this looking to set the sleuth traceId context, we came up with the following solution:
#Autowired Tracer tracer;
private void someMethod(long traceId) {
Span span = Span.builder()
.traceId(traceId)
.spanId(new Random().nextLong())
.build();
tracer.continueSpan(span);
// do work
tracer.closeSpan(span);
}
It should be noted that all the documentation says that a span should be closed once you've finished with it. The do work section above should be wrapped with a try / catch / finally block to ensure this is closed.
Any methods called with the span still open will inherit the traceId.
EDIT
I should also say, it seems the better solution would be to replace the Spring AMQP library with spring-cloud-stream; from what I can tell, this should automatically include the traceId in rabbit messages (correlationId) and set it at the other end. I haven't had the opportunity to test this, however.
Hope you have some spare minutes for my question. For the last couple of days I was reading about Camel and managed to setup everything up and running. Now, I have bumped on a tricky part :). Basically, I define a route using Java in runtime and put the route configuration in DB. Routes are working and messages are flowing from one side to another. But, when an exception occurs I would like to know where the exception has occurred (between which route endpoints), store the current exchange body (in-flight message that is useful for further processing) in the DB, update the message by the user and then retry the route execution from the point of exception (where it failed). Lets say that all route steps are idempotent.
My solution:
Make a Interceptor for the route
Granulate the route in as much as possible parts (each new step is a new route)
Between each step update the DB with the current and future step and current exchange body
If exception occurs, store the message in DB, update it using an editor
Send a message to a next route point (taken from DB) using ProducerTemplate
What do you think about this ? Is it doable or Camel cannot support me with this approach ?
Thank you for patience and your time.
Hope I was clear enough.
You can use Camel's tracer component. A detailed example meeting your needs is already available on camel's site : http://camel.apache.org/tracer-example.html
You should use onException() clause to tackle this. For eg :
public void configure() throws Exception{
//This is a global onException definition and will work for all routes defined in this calss
onException().process(new Processor(){
public void process(Exchang arg0){
Exception e = arg0.getProperty(Exchange.EXCEPTION_CAUGHT,Exception.class);
//get message and other properties that you are interested in
db.saveOrUpdate(/*Pass your custom object here*/);
}
}).handled(true);
from("direct:route1")
//some processing
.to("jms:route1");
from("direct:route2")
//some processing
.to("http://route2");
}
You might need to consult exact details at apache camel site, since i just wrote this code here.
I have an application I coded which I am refactoring to make better use of Spring Integration. The application processes the contents of files.
The problem (as I see it) is that my current implementation passes Files instead of Messages, i.e. Spring Integration Messages.
In order to avoid further rolling my own code, which I then have to maintain later, I'm wondering if there is a recommended structure for constructing Messages in Spring Integration. What I wonder is if there is some recommended combination of channel with something like MessageBuilder that I should use.
Process/Code (eventually)
I don't yet have the code to configure it but I would like to end up with the following components/processes:
Receive a file, remove header and footer of the file, take each line and convert it into a Message<String> (This it seems will actually be a Splitter) which I send on to...
Channel/Endpoint sends message to Router
Router detects format String in Payload and routes to the appropriate channel similar to Order Router here...
Selected channel then builds appropriate type of Message, specifically typed messages. For example I have the following builder to build a Message...
public class ShippedBoxMessageBuilder implements CustomMessageBuilder {
#Override
public Message buildMessage(String input) {
ShippedBox shippedBox = (ShippedBox) ShippedBoxFactory.manufactureShippedFile(input);
return MessageBuilder.withPayload(shippedBox).build();
}
...
Message is routed by type to the appropriate processing channel
My intended solution does seem like I've complicated it. However, I've purposefully separated two tasks 1) Breaking a file into many lines of Messages<String> and 2) Converting Messages<String> into Messages<someType>. Because of that I think I need an additional router/Message builder for the second task.
Actually, there is MessageBuilder support in the Spring Integration.
The general purpose of such Frameworks is to help back-end developers to decouple their domain code from messaging infrastructure. Finally, to work with Spring Integration you need to follow the POJO and Method Invocation principles.
You write your own services, transformers and domain models. Then you just use some out of the box compoenents (e.g. <int-file:inbound-channel-adapter>) and just refer from there to your POJOs, but not vise versa.
I recommend you to read Spring Integration in Action book to have more pictures on the matter.
Can you explain the reason to get deal with Spring Integration components directly?
UPDATE
1) Breaking a file into many lines of Messages
The <splitter> is for you. You should write some POJO which returns List<String> - the lines from your file without header and footer. How to read lines from File isn't a task of Spring Integration. Especially, if the "line" is something logical, not the real file line.
2) Converting Messages into Messages
One more time: there is no reason to build Message object. It's just enough to build new payload in some transformer (again POJO) and framework wrap to its Message to send.
Payload Type Router speaks for itself: it checks a payload type, but not Message type.
Of course, payload can be Message too, and even any header can be as well.
Anyway your Builder snapshot shows exactly a creation of plain Spring Integration Message in the end. And as I said: it will be enough just to transform one payload to another and return it from some POJO, which you will use as a transformer reference.