I'm doing a microservice that produces messages for an ActiveMQ broker.
My posible messages are;
1) Logs for my application.
2) The business messages I need.
Later I'll develop a microservice that consumes those messages, and I thought that it could be better to have two different queues at ActiveMQ.
My question is, should I use 2 queues, or should I use 1 queue with a flag to differenciate messages?
When we talk about microservices, it's about segregation of responsibilities and loosly coupled architecture which could be extensible lateron.
If you'll identify message based on flag
It will be harcoded even when messages are not related
Highly coupled architecture
Queue maintenance and scaling would be affected later on
and so on ..
I would recommend using different queues for different types of messages which serves unique purpose.
Related
We use Tibco EMS as our messaging system and have used apache camel to write our application. In our application, messages are written to a queue. A component, with concurrentConsumers set to 8, reads from the queue, processes the message, then writes to another queue. Another component, again with concurrentConsumers set to 8, then reads from this new queue and so on. Up until now, maintaining message order has not been important, but a new requirement means that it now is. Looking at the camel documentation, it is suggested that jmsxgroupid is used to maintain ordering. Unfortunately, this functionality is not available with Tibco EMS. Are there any other ways of maintaining ordering in camel in a multithreaded application? I have looked at sticky load balancing, but this seems to be applicable to end point load balancing only.
Thanks
Bruce
In the enterprise integration world, we generally use the Resequencer design pattern to solve such kind of problems that you need to ensure ordering in the messages.
Apache Camel covers a broad extent of the Enterprise Integration Patterns including Resequencer at its core and it has out-of-the-box imprementations for those patterns. So what you are looking for should be this:
http://camel.apache.org/resequencer.html
In your specific case, all you need to do would be add a custom message header like myMessageNo, which has a sequential number that specifies the ordering, to outgoing messages to TIBCO EMS. Then, at consumer side, use the resequencer EIP to restore the ordering of incoming messages from TIBCO EMS.
As you can see, however, it's not as easy as just putting the resequencer EIP to your Camel routes. (Any asynchronous solutions are always hard to build correctly.) For the resequencer, you need to consider when sad paths happen, e.g. when some messages get lost and never reach. To make sure your routes work fine even with those exceptional cases, you need to choose from two options: maximum batch size or timeout. Depending on the condition chosen, the resequenser will flush messages when the batch reaches the maximum size or it timeouts waiting for a missing message.
Best Architecture for implementing a WebService that takes requests from one side, save and enhance that and then call another service with new parameters.
is there any special Design Pattern for this?
There's not a lot to go on, but from what you've said it sounds like a job for "pipes and filters"!
To get a more precise answer, you might want to ask yourself some more detailed questions:
If you need to do any validation or transformation of the incoming message? Will you want to handle all requests the same way, or are there different types? Are the external services likely to change, and if so, will they do this frequently? What do you want to do if the final web service call fails (should you rollback the database record?)? How do you want to report failures/responses - do you need to report these back? Do you need a mechanism to track the progress of a particular request?
Since you are looking for a design pattern, I think you might want to compare the pros and cons of using microservices orchestration vs choreography in the context of your project.
If you do not need an immediate response to the calling system I would suggest to you to use event-driven approach if that's feasible. So instead of REST services, you will have a message broker and your services will be subscribed for certain events. This will hide your consumers behind the message broker which will make your system less coupled.
This can be implemented via Spring Cloud Stream, where you will have a Sink (microservice producing events, transformer - microservice that makes intermediate transformations possible and a source - microservice that receives a final result for further processing).
Another possible case could be Camel. It has basically all the integration patterns built in, so it should not be a problem to implement the solution either based on REST APIs or events.
For my master's degree final project I decided to design a drone delivery system. The main purpose is to learn to design complex systems.
The basic use case is this:
User goes to merchant online shop, selects the products, selects the delivery method as "Drone delivery" and selects his delivery location.
Merchant website, makes an API call to our drone delivery system (DDS) application to register the new delivery order.(The order will contain all information that we need: parcel pick up location, and destination location...)
The DDS application based on drones positions, and based on an algorithm will calculate and mark which drone can deliver this order in the shortest time.
The selected drone when is free will deliver the order.
So far so good. My questions are related to the software architecture of this system. I have some general questions and some specific questions.
General questions:
How do you design a system like this in order to be scalable? I mean: The system may be used by may merchants, if they hit my API in the same time with 100 orders, the system must be able to handle it.
What are some good design principles or patterns when designing an system like this?
Specific questions:
So far i have came up with this architecture:
System Components:
Java(Spring) application
Rest web servce
web interface managing dorens and parces
bussines logic and algorithms for routing drones
producer/consumer for RabbitMQ
Mysql Server
RabbitMq
System flow:
Merchant hits REST API to register the order
The Java Application saves the order to Mysql database.
After saving the order to the database, an Producer puts the order in a queue in RabbitMQ
An Consumer consumes the RabbitMQ order queue. It takes each order and calculates based on an algorithm the drone that offers the best time for the delivery. Each drone has a separate queue in RabbitMQ. After finding the best drone, the consumer inserts the order in the drone queue in RabbitMQ. The consumer also interrogates the mysql database during this process.
Whenever a drone is free, it will communicate with the system to ask for the next order. The system will look in the drone RabbitMQ queue and will take from there the next order.
My questions are related to the consumer and producer:
Is OK that the consumer to have logic in it, in my example it will have the algorithm that will determine the best drone, to do this it needs to talk to mysql also, for retrieving drone positions? Is this a good practice? If not how can i do different?
Is best practice for the consumer to stay in the application? Right now consumers are running in the same server as the web service and the code is not separated from web service code. I am thinking maybe in the future you may need to move the consumers in a separate server? How do you think the consumers so they can easily be separated from the application?
I think that the producer must stay in the application, i mean is coupled with the web service app. Is that OK?
Sorry for the long post, and for my poor English.
Thank you very much :)
Yes, the consumer should have logic in it. This is a standard EIP routing pattern.
If you properly separate your business logic layers from your data access layers (your queue access is a data access layer), then it probably isn't a problem to have them all share a common project. You ultimately probably want to separate your business logic/domain model from the web service and the router/consumer, but those are much more deployment and packaging concerns.
As long as you keep your web service code out of your business logic (and vice versa) you will probably be ok, you will just have to deploy the whole thing multiple times, and only expose the endpoints that are relevant for any given deployment. You ultimately might be happier though if you separate your layers via libraries, as it will actually enforce not mixing the concerns.
And yes, the producer must be deployed with the web service, just make sure you are aware that as a Data Access Layer, that it's in a separate package/class. It will make your testing much easier.
I recently watched a nice presentation about how RabbitMQ works and it kinda intrigued on how whole AMQP implentation works.
I was considering using it for a project but I would like some answers about the following questions:
1) Is it possible to have a broker and a producer of a message at the same place? I do understand that RabbitMQ allows the use of Virtual Hosts so something like this could be possible right?
2) Can RabbitMQ transmit it's messages over two diferrent subnets? I know it can trasmit over lan or wan, but how easy it to do this over two subnets? (One answer here would actually be to have them bridged).
3) Regarding question 1, how hard would it be to fail over the broker functionality to another place in case the original broker goes down?
4) I do understand that RabbitMQ actually provides different types of message transmissions. One of those is the fanout type which is more or less similar to a broadcast action. Would it be possible though to have something that is the inversed type of that. Meaning that you have multiple producers with multiple queues that all transmit to a single consumer?
1) It doesn't matter where the consumers/producers are, as long as they can reach (access IP:port) the broker. Virtual hosts have nothing to do with that.
2) More or less same as answer to first one, RabbitMQ us using the network and has no knowledge about what kind of networks is it in; also doesn't need to know or care.
3)Failover is easy, look for rabbitmq cluster and high availability. For the clients you'd have to take care on your own (so how to reconnect etc).
4) yes, broadcast is possible, you should have a look at tutorials and what kind of exchanges are there. EDIT As zapl pointed out in the comments, you can also do the inverse of broadcast.
Currently we are working on a project and we are in design and architecture phase of the project following are main points of projects.
There are switches which are generating real time data
We have two components in to be made in Java/Java EE, call it CompA and CompB
CompA apply some process based on input record from switch without contacting to any databases, CompA does not have DB access.
CompB takes process record of CompA and apply processing also, this involves business database
CompA and CompB have multiple instances in system for scalability and fault tolerance.
Record is text record having multiple fields
Record is transactional , On record is considered processed if it is process from both CompA and CompB , other wise it will be roll backed and resend again
Now the problem is what is best way for communication between CompA adn Comp B
One way is
1. CompA--------> CompB
2. CompA-------->Messaging Server(JMS)------> CompB
Requirment: There will be more than one CompA and CompB is the system and if any component fails it load will be shared by other peers e.g if CompA fails its load will be shared by other CompA instances in the system. For that we are going for second option with JMS so that CompA is not tightly bound with CompB. But as new Component (Messaging Server)is introduced this may cause performance degradation as the record processing is transactional the system is real time.
Your suggestions and expert advices will be highly appriciated
JMS is the way to go - http://docs.oracle.com/javaee/6/tutorial/doc/bnceh.html
It is very reliable, you can do things like set message expirations and enforce priorities and it is perfect for your model which is basically a "Multiple producer/multiple consumer" over the network.
JMS supports transactions and it is built for reliability - by far it is the most reliable mechanism available. Performance-wise, you should talk about "scalability" more than "raw performances". Provided your hardware can cope, JMS will.
Wikipedia has a very good list of available JMS implementations:
http://en.wikipedia.org/wiki/Java_Message_Service#Provider_implementations
I have used Apache ActiveMQ, Open Message Queue and OpenJMS and, even if I have no experience of deployment of JMS servers on a clustered environment, I agree ActiveMQ is the most reliable solution I used.
I would suggest to use JMS with spring integration. check example
In my case we have used ActiveMq with spring integration so that we were able to handle loadbalancing and fail over fetures easily.