I have a simple question about messaging. I have two applications that communicate via RabbitMQ. The first application regularly sends messages to say "New data has arrived". The second application subscribes to these messages and looks up the information in a common database if it is interested. So the questions is: Should the messages be structured as an alert and an ID so that the second application can do a look up to retrieve the data or should the message contain the complete data set so that a look up is not required? The message content is around 1KBytes. I guess to some extent this is a stylistic question, but I am curious to know if there is an accepted "Best Practice"
Related
Imagine there is chatting software. A java microservice listens to agents chatting with customers. The microservice dumps some pertinent data items in an SQS queue. A java bot reads from the queue, formats some things into the appropriate json and fires this off to a RESTful endpoint outside the network. Alternatively, in simulation of that bot, I can post manually properly formatted json into postman. At the last line of appcode before it is sent off in an httpRequestEntity to get a synchronous response before asynchronous processing it can be seen that the json message is properly formed. This is seen in logging and traffic sniffing. The same is true if using the postman test harness method. Some seconds later the package that got sent out of the network arrives at where it was sent.
WITH AN EXTRA JSON ELEMENT!
That reads as if it is metadata from the chatting software!
But remember this happens if the message originates in the chatting software or if I am using the fake cut in line method of mocking the message starting at a later point in the sequence.
I am so stumped. Every step of the way in every way I know to monitor data along its way the package looks right. Then at some point it doesn't. I don't want to bring this up with the people at the other end, receiving the message. Because it sounds like something that couldn't possible happen.
Please any comments even long shots would be much appreciated. I will watch this thread and respond to any questions it generates. Thank you so much.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'd like to understand what exactly is the difference between the two Enterprise Integration Patterns
Content-Based Router
Message Router
The definition at camel documentation is suggestive that Content-Based router is a special case of Message Router. They why list them separately?
Actually "Message Router" is one of the "Basic Messaging Concepts". List of such basic messaging concepts is:
Channel - Messaging applications transmit data through a Message Channel, a virtual pipe that connects a sender to a receiver.
Message - A Message is an atomic packet of data that can be transmitted on a channel.
Multi-step delivery - Set of actions often need to be performed on the message after it is sent by its original sender but before it is received by its final receiver.
Routing - In a large enterprise with numerous applications and channels to connect them, a message may have to go through several channels to reach its final destination. The route a message must follow may be so complex that the original sender does not know what channel will get the message to the final receiver. Instead, the original sender sends the message to a Message Router.
Transformation - Various applications may not agree on the format for the same conceptual data; the sender formats the message one way, yet the receiver expects it to be formatted another way.
Endpoint - An application does not have some built-in capability to interface with a messaging system. Rather, it must contain a layer of code that knows both how the application works and how the messaging system works, bridging the two so that they work together.
"Content Based Router" is one of the "Message Routers" and there are a lot of different other Message Routers available like "Message Filter", "Splitter", "Aggregator", "Recipient list" etc.
I suggest reading a book that used by camel so all such points will be more clear:
https://www.amazon.com/o/asin/0321200683/ref=nosim/enterpriseint-20
As far as I understand the patterns Message Router only applies when the input and output is a queue or topic of a messaging system.
Content-based Router is not limited to messaging. I think you can say that inside a Message Router you have a Content-based router that decides which way to go.
In camel the decision is made using the choice() element.
If my assumption is correct then the documentation of the Message-Router at camel is wrong as it does not reflect the queues. I will check with the camel dev list and correct the wiki page if we agree on this.
I am developing an sms gateway using jsmpp library.
This is my scenario.
I bind to the SMSC(telecoms company) on 2 connections and receive messages on 2 shortcodes, say 30002 and 30003 , assigned to me by the SMSC.
Then, third party binders(companies involved in bulk sms push) who normally should bind directly to the SMSC bind to my gateway via SMPP connections to my SMPP server(which serves as an SMSC of sorts to them)
The third party binder sends messages to the SMPP server and I capture this messages, queue them and send them to the main SMSC(telecoms company).
The telecoms company then responds with delivery reports and messages from the bulk message reports and other data which I forward to the third party binders.
This scenario has worked well for when I have only 1 third party binder.
But now we want to handle a more general case where we have many third party binders.
Since I have only 2 short codes available on which the SMSC forwards messages to me, how do I know which of the third party binders owns the SMSC response?
I have the unsavoury and inefficient option of forwarding the responses to all connected third parties.
The only other option apart from the above is to get a different short code from the SMSC for each third party binder, which is not cost efficient for my scenario.
First of all, you can pretty easily forward all DLRs to the correct third parties. In order to do so, you will have to store some extra data in a database of your choice (if you only have 1 application node and low traffic, even SQLite will suffice; otherwise look towards some NoSQL solution):
Every time you receive a submit_sm_resp packet from telecom SMSC, start storing a pair (key-value, actually) message_id -> third_party_id.
Every time you receive a deliver_sm packet from SMSC, retrieve the message_id from the packet, then look for this message_id in your DB to find which third party to forward it to.
Consider removing the message_id -> third_party_id pair after you have successfully forwarded the final DLR for that message_id.
The situation is a lot worse with the MO messages. The only possible (but nonetheless very bad) way of using a single shortcode for multiple clients (third parties) is having a fragile combination of TTLs and timeouts on you data. It also only makes sense when every MO that you expect is a reply to a previous MT message.
Separate incoming MTs that need reply from the receiver using TLV or extra SMPP links. Let us call those MTs Questions.
Handle Questions something like this:
Handle MO messages (Answers) like this:
UPD. Text elaboration
In order to know the correct third party for MO message (not delivery report) you need some additional info about who the message is addressed to. This info might originate from either the third party or the subscriber (one who has initiated the MO message in the first place). My diagrams formally describe the case where the third party gives you this information.
A third party sends you an MT message that they expect to be Answered by the recipient.
You make a note in the database, that if the recipient sends you an MO message in the next N minutes, then this Answer should be forwarded to the aforementioned third party.
I apologise that my answer is not specific to jsmpp but I have little knowledge of this library, however, I also believe it should be fairly easy to implement an algorithm once you understood it regardless of your instrument.
I'm developing a 'WS oriented' application basing on Spring/CXF/Oracle DB. Now, I stuck with some architectural consideration about right approach to organize message processing (already stored in db).
Briefly, process looks as follows:
(A) Get the message from client -> Validate -> Store -> Send reposponse
(B) Process -> Update data
I consider two general approaches for part B of the process:
1) Use JMS queue
Just after validation and storing incoming message details in DB publish a message to the JSM queue. On the other side define cosumer which will retrieve the message and do the processing
2) Fetch data to be processed
Manually fetch data from with db and process it.
Additional facts:
The processing won't be compute-intensive, so for new I dont think that work distribution will be needed (all in single JVM).
All data in single db schema
So, I'm interested what are key factors to choose JMS in such case?
JMS would be a better approach. In positive scenario, approach #2 works as well. But JMS would provide you some in-built capability, specially for failed case. Though internally JMS would be using a DB-based persistent storage; it would provide a better interface to communicate that data.
For example, you could configure an error queue to track all the messages, whose processing failed.
It would also provide you scalable architecture, where some other app (in future) could starts consuming your message and process.
Reliable: Due to asynchronous messaging, all the pieces don’t need to be up for the application to function as a whole.
Flexible : Think of scenario, in which you might want to process certain type of data before all other (prioritization). JMS would provide more better approach than tweaking logic in a program.
I have a chat program implemented in Java. The client can send lots of different types of information to the server (i.e, Joins the server and sends username, password; requests a private chat with another user on the server, disconnects from the server, etc).
I'm looking for the correct way to have the server/client differentiate between 'text' messages that are just meant to be chat text messages sent from one client to the others, and 'command' messages (disconnect, request private chat, request file transfer, etc) that are meant for the server or the client.
I see two options:
Use serialized objects, and determine what they are on the receiving end by doing an 'instanceof'
Send the data as a byte array, reserving the first N bytes of the array to specify the 'type' of the incoming data.
What is the 'correct' way to do this? How to real protocols (oscar, irc) handle this situation?
I've googled around on this topic and only found examples/discussions centering on simple java chat applications. None that go into detail about protocol design (which I ultimately intend to practice).
Thanks to any help...
Second approach is much better, because serialization is a complex mechanism, that can be easily used in a wrong way (for example you may bind yourself to internal content of a concrete serialized class). Plus your protocol will be bound to JVM mechanism.
Using some "protocol header" for message differentiation is a common way in network protocols (FTP, HTTP, etc). It is even better when it is in a text form (people will be able to read it).
You typically have a little message header identifying the type of content in all messages, including standard text/chat messages.
Either of your two suggestions are fine. (In your second approach, you probably want to reserve some bytes for the length of the array as well.)