I am adding some headers in my-transformer:
public Message<?> transform(final Message<?> message) {
List<Item> items = doStuff(message);
final MessageBuilder<?> messageBuilder = MessageBuilder
.withPayload(message.getPayload())
.copyHeadersIfAbsent(message.getHeaders());
for (final Item item : items) {
messageBuilder.setHeader(item.getHeaderName(), item.getValue());
}
return messageBuilder.build();
}
And I wrote an integration test to confirm that my header is present on the output channel:
public static class HeaderTest extends TransformerTest {
#Test
public void test() throws Exception {
channels.input().send(new GenericMessage<>(TransformerTest.EXAMPLE_PAYLOAD));
final Message<?> out = this.collector.forChannel(this.channels.output()).poll(10, TimeUnit.SECONDS);
assertThat(out, HeaderMatcher.hasHeader("header-test", notNullValue()));
}
}
But, when I created a stream like:
http --port=1234 | my-transformer | log --expression=toString()
and sent the same EXAMPLE_PAYLOAD I received the following message in the logs log: GenericMessage [payload=..., headers={kafka_offset=0, id=f0a0727c-9351-274c-58b3-edee9ccbf6ce, kafka_receivedPartitionId=0, contentType=text/plain;charset=UTF-8, kafka_receivedTopic=myTopic.my-transformer, timestamp=1485171448947}].
Why isn't my header-test in the message headers?
-- EDIT --
So if I understood correctly I am supposed to do something like:
public class MyTransformer implements Transformer {
private final EmbeddedHeadersMessageConverter converter = new EmbeddedHeadersMessageConverter();
#Override
public Message<?> transform(final Message<?> message) {
List<Item> items = doStuff(message);
final MessageBuilder<byte[]> messageBuilder = MessageBuilder
.withPayload(((String) message.getPayload()).getBytes())
.copyHeadersIfAbsent(message.getHeaders());
final int itemsSize = items.size();
final String[] headerNames = new String[itemsSize];
for (int i = 0; i < itemsSize; i++) {
final Item item = items.get(i);
messageBuilder.setHeader(item.getHeaderName(), item.getValue());
headerNames[i] = item.getHeaderName();
}
final Message<byte[]> msg = messageBuilder.build();
final byte[] rawMessageWithEmbeddedHeaders;
try {
rawMessageWithEmbeddedHeaders = converter.embedHeaders(new MessageValues(msg), headerNames);
} catch (final Exception e) {
throw new HeaderEmbeddingException(String.format("Cannot embed headers from '%s' into message: %s", items, msg), e);
}
return new GenericMessage<>(rawMessageWithEmbeddedHeaders);
}
}
with spring.cloud.stream.bindings.output.producer.headerMode=raw set in application.properties and then convert the message payload on the receiving side? Or can I somehow make the receiving side automatically convert the message payload?
You don't say whether you are using Spring XD or Spring Cloud DataFlow, but the solution is similar in each case.
Since kafka has no native support for headers, we have to embed them in the message payload. Since we don't want to transport unnecessary headers, you have to opt-in for the headers you want transported by setting the header names in servers.yml for Spring XD or application.yml (or .properties) for a Spring Cloud Stream app.
EDIT
Unfortunately, there is no support for patterns. One option would be to use the EmbeddedHeadersMessageConverter yourself, and set the kafka mode to raw (on your transformer's output destination). Raw mode means the binder won't embed headers.
That way, the next app (without mode raw) should be able to decode the headers as if they had been encoded by the binder in your transformer. Javadocs here.
You are limited to 255 headers.
Related
I'm trying to send the UDP request and receive the response. Spring Integration has the appropriate instruments for such kind of task: UnicastSendingMessageHandler and UnicastReceivingChannelAdapter. I configured it in the following way
#Bean
public MessageChannel requestChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "requestChannel")
public UnicastSendingMessageHandler unicastSendingMessageHandler() {
UnicastSendingMessageHandler unicastSendingMessageHandler = new UnicastSendingMessageHandler("239.255.255.250", 1982);
return unicastSendingMessageHandler;
}
#Bean
public UnicastReceivingChannelAdapter unicastReceivingChannelAdapter() {
UnicastReceivingChannelAdapter unicastReceivingChannelAdapter = new UnicastReceivingChannelAdapter(8080);
unicastReceivingChannelAdapter.setOutputChannelName("nullChannel");
return unicastReceivingChannelAdapter;
}
How I send a message (I'm using sendDiscoveryMessage() wherever I want):
#Service
public class DiscoveryService {
private static final String DISCOVERY_MESSAGE = "M-SEARCH * HTTP/1.1\r\n"
+ "HOST: 239.255.255.250:1982\r\n"
+ "MAN: \"ssdp:discover\"\r\n"
+ "ST: wifi_bulb";
private final MessageChannel requestChannel;
public DiscoveryService(final MessageChannel requestChannel) {
this.requestChannel = requestChannel;
}
public void sendDiscoveryMessage() {
requestChannel.send(new GenericMessage<>(DISCOVERY_MESSAGE));
}
}
At this point, I can check the packets via WireShark and ensure that Datagram was sent and the appropriate response was sent too.
The only question is how to receive this response. As far as I understand reading the documentation, I need the method annotated with #ServiceActivator. But I don't understand where (which channel) I should receive the response (in order to correctly specify #ServiceActivator(inputChannel="")). Also, I'm not sure about #ServiceActivator(inputChannel = "requestChannel") I put for UnicastSendingMessageHandler bean.
I tried to create the following method(assuming that the response will come to the same channel):
#ServiceActivator(inputChannel = "requestChannel")
public void receiveResponse(Message<String> response) {
System.out.println(response);
}
but it actually intercepts my own request message (seems logical to me, because I send the request to requestChannel).
So I don't understand how many channels I need (maybe I need 1 for request and 1 for response) and how to create #ServiceActivator to catch the response.
unicastReceivingChannelAdapter.setOutputChannelName("nullChannel");
You are sending the result to nullChannel which is like /dev/null on Unix; you are discarding it.
Use #ServiceActivator(inputChannel = "replyChannel") and
unicastReceivingChannelAdapter.setOutputChannelName("replyChannel");
I have a producer in PHP and a consumer in Java that will communicate via RabbitMQ. They are going to be working with three different message types. If the producer was also a Java application, I could just serialize the objects as raw, and then in the consumer do:
Consumer consumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
Object deserializedBody = SerializationUtils.deserialize(body);
if (deserializedBody instanceof TypeOne) {
TypeOne typeOne = (TypeOne) deserializedBody;
// process with corresponding code
} else if (deserializedBody instanceof TypeTwo) {
TypeTwo typeTwo = (TypeTwo) deserializedBody;
// process with corresponding code
} else if (deserializedBody instanceof TypeThree) {
TypeThree typeThree = (TypeThre) deserializedBody;
// process with corresponding code
} else {
// throw exception
}
}
};
But since my producer is in PHP, I'll have to serialize the message as JSON strings.
How can I then distinguish between the three message types?
Messages in AMQP have attributes, and you can define one for your own use specifying the type of message. But most of the time people choose to use "content-type" and "content-encoding".
I have the following codes:
#Service
#EnableBinding(Sink.class)
public class AMQPService {
ObjectMapper mapper = new ObjectMapper();
#Autowired
private BinderAwareChannelResolver binderAwareChannelResolver;
#StreamListener(Sink.INPUT)
public void processMessage(#Payload Map<String, Object> inboundMessage, #Headers Map<String, Object> headers) throws JsonParseException, JsonMappingException, IOException {
headers.entrySet().forEach(e -> System.out.println(e.getKey() + '=' + e.getValue()));
String output = mapper.writeValueAsString(inboundMessage);
AMQPOutboundMessage outMessage = new AMQPOutboundMessage();
outMessage.setText(output);
if (headers.containsKey("expected_destination")) {
MessageChannel messageChannel = binderAwareChannelResolver.resolveDestination(headers.get("expected_destination").toString());
messageChannel.send(MessageBuilder.withPayload(outMessage).setHeader("contentType", "application/json;charset=UTF-8").build());
}
}
}
It just gets amqp message from RabbitMQ and then just according to the "expected_destination" header to sends the message to the destination.
I've set spring.cloud.stream.bindings.output.content-type=application/json;charset=UTF-8, but I saw the content-type of the message is application/x-java-object;type=xxx.AMQPOutboundMessage and base64 encoded message body.
But when I use #Autowired to get messageChannel, it seems everything is fine.
So, may I know how to set the content-type in this case please?
The setting spring.cloud.stream.bindings.output.content-type=application/json;charset=UTF-8 will only apply to the channel named output, not to all output channels.
In this version, you will need to have a configuration for each expected output, i.e. the possible values of expected_destination. We are considering supporting default settings in the future, i.e. https://github.com/spring-cloud/spring-cloud-stream/issues/446
I wrote a simple message flow with request and reply. I have to use two independent queues so i declare AmqpOutboundAdapter to send a message and AmqpInboundAdapter to receive a reply.
#Bean
#FindADUsers
public AmqpOutboundEndpoint newFindADUsersOutboundAdapter() {
return Amqp.outboundAdapter(amqpTemplate())
.routingKeyExpression("headers[" + ADUsersFindConfig.ROUTING_KEY_HEADER + "]")
.exchangeName(getExchange())
.headerMapper(amqpHeaderMapper())
.get();
}
#Bean
public AmqpInboundChannelAdapter newFindADUsersResponseInboundChannelAdapter(
ADUsersFindResponseConfig config) {
return Amqp.inboundAdapter(rabbitConnectionFactory(), findADUsersResponseQueue)
.headerMapper(amqpHeaderMapper())
.outputChannel(config.newADUsersFindResponseOutputChannel())
.get();
}
It should work with #MessagingGateway:
#MessagingGateway
public interface ADUsersFindService {
String FIND_AD_USERS_CHANNEL = "adUsersFindChannel";
String FIND_AD_USERS_REPLY_OUTPUT_CHANNEL = "adUsersFindReplyOutputChannel";
String FIND_AD_USERS_REPLY_CHANNEL = "adUsersFindReplyChannel";
String CORRELATION_ID_REQUEST_HEADER = "correlation_id";
String ROUTING_KEY_HEADER = "replyRoutingKey";
String OBJECT_TYPE_HEADER = "object.type";
#Gateway(requestChannel = FIND_AD_USERS_CHANNEL, replyChannel = FIND_AD_USERS_REPLY_CHANNEL)
ADResponse find(ADRequest adRequest, #Header(ROUTING_KEY_HEADER) String routingKey, #Header(OBJECT_TYPE_HEADER) String objectType);
}
And the ADUsersFindResponseConfig class looks like:
#Configuration
#Import(JsonConfig.class)
public class ADUsersFindResponseConfig {
#Autowired
public NullChannel nullChannel;
#Autowired
private JsonObjectMapper<?, ?> mapper;
/**
* #return The output channel for the flow
*/
#Bean(name = ADUsersFindService.FIND_AD_USERS_REPLY_OUTPUT_CHANNEL)
public MessageChannel newADUsersFindResponseOutputChannel() {
return MessageChannels.direct().get();
}
/**
* #return The output channel for gateway
*/
#Bean(name = ADUsersFindService.FIND_AD_USERS_REPLY_CHANNEL)
public MessageChannel newADUsersFindResponseChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow findADUsersResponseFlow() {
return IntegrationFlows
.from(newADUsersFindResponseOutputChannel())
.transform(new JsonToObjectTransformer(ADResponse.class, mapper))
.channel(newADUsersFindResponseChannel())
.get();
}
}
Sending message works properly, but i have a problem with receiving message. I am expecting that received message will be passed to channel called FIND_AD_USERS_REPLY_OUTPUT_CHANNEL, then the message will be deserialized to ADResponse object using findADUsersResponseFlow , and next ADResponse object will be passed to gateway replyChannel - FIND_AD_USERS_REPLY_CHANNEL. Finally, 'find' method return this object. Unfortunately when org.springframework.integration.handler.BridgeHandler receive a message, i got exception:
org.springframework.messaging.MessagingException: ; nested exception is org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available
Message log looks like:
11:51:35.697 [SimpleAsyncTaskExecutor-1] INFO New message - GenericMessage [payload={...somepayload...}, headers={correlation_id=7cbd958e-4b09-4e4c-ba8e-5ba574f3309a, replyRoutingKey=findADUsersResponse.ad, amqp_consumerQueue=findADUsersResponseQueue, history=newFindADUsersResponseInboundChannelAdapter,adUsersFindReplyOutputChannel,adUsersFindReplyChannel,infoLog,infoLoggerChain.channel#0,infoLoggerChain.channel#1, id=37a4735d-6983-d1ad-e0a1-b37dc17e48ef, amqp_consumerTag=amq.ctag-8Qs5YEun1jXYRf85Hu1URA, object.type=USER, timestamp=1469094695697}]
So i'm pretty sure that message was passed to adUsersFindReplyChannel. Also (if it's important) both request message and reply message have 'replyTo' header set to null. What am I doing wrong?
The replyChannel header is a live object and can't be serialized over AMQP.
You can use an outbound gateway instead of the pair of adapters and the framework will take care of the headers.
If you must use adapters for some reason, you need to do 2 things:
Use the header channel registry to convert the channel object to a String which is registered with the registry.
Make sure that the header mapper is configured to send/receive the replyChannel header and that your receiving system returns the header in the reply.
I implemented the Message interface to include some headers for use with a HeaderValueRouter on server side.
Within one VM this works (tested using a filter between two endpoints).
But if I send the message through HttpOutboundGatway my fields get stripped (not included in the HttpRequest). And therefor the routing information is lost on server side.
Am I not supposed to manipulate headers?
public class TaskMessage implements Message<String> {
private MessageHeaders headers;
private String payload;
public TaskMessage(String taskId, String boxId, String payload) {
super();
this.taskId = taskId;
this.boxId = boxId;
this.payload = payload;
StringMessage sm = new StringMessage(payload);
Set<String> keySet = sm.getHeaders().keySet();
HashMap<String, Object> map = new HashMap<String, Object>();
for (String key : keySet) {
map.put(key, sm.getHeaders().get(key));
}
map.put("taskId", taskId);
map.put("boxId", boxId);
headers = new MessageHeaders(map);
}
#Override
public MessageHeaders getHeaders() {
return headers;
}
#Override
public String getPayload() {
return payload;
}
}
EDIT:
The version is 1.0.3
The part of my configuration is:
<si:inbound-channel-adapter ref="jdbcInputAdapter" method="fetchData" channel="msgChannel">
<si:poller max-messages-per-poll="1">
<si:interval-trigger interval="5000" />
</si:poller>
</si:inbound-channel-adapter>
<http:outbound-gateway id="httpChannelAdapter" auto-startup="true" request-timeout="1000" request-channel="msgChannel" reply-channel="replyChannel" default-url="http://localhost:8080/taskserver/gateway"/>
The version you are using does not support (custom) header serialization. The solution would be to craft a request that contains all the information needed and pass it along as the payload. The new REST based http support in version 2.0.x does support header mapping and also exposes extension points for converting messages (including headers).
As a side note, it is quite uncommon to have to implement a custom Message, so instead of doing that I'd create a message using MessageBuilder
MessageBuilder.withPayload("foo").setHeader("taskId", "someTaskId").build();
In general not all headers can be transferred with all protocols, so if you want to use a distributed system it is usually more flexible to pack all information you need to send into the payload.