I'm creating a kafka stream to replicate information from one application to another, the destination api has some maintenance windows when i don't have to send data or i can cause issues on it.
I have an api that gives me when there is a maintenance period this is not an issue, what i would like to know is how to disable the stream for a given period of time and start it again once the maintenance window is over.
I'm writing my code in Java
you could manage kafka streams state like starting/stopping in a way that required to your use case. for that you need to have collection with your kafka streams in memory, and during maintenance stop them (either all or part of them) using
kafkaStreams.close() and kafkaStreams.cleanUp() on each required stream. when maintenance completed, start them by using kafkaStreams.start().
Listening of mainetance could be done in a multiple ways, for example:
by scheduling (e.g. using quartz library). if you have multiple instances of the app, scheduler should be triggered on each node.
by some kafka topic like maintenance_operations (e.g. message with status MAINTENANCE_STARTED or MAINTENANCE_COMPLETED). your app always will listen this topic and start / stop required streams based on event. if you have multiple instances of the app, each node should have a unique consumer group for maintenance_operations topic.
Related
I'm trying to black-box test a Spring Boot application which is using Spring Cloud Stream Kafka. The expected results (in the DB) may differ based on the message processing order. How could I reliably tell if one message was processed and I can send in the next? One important factor is that one message from the test can generate multiple events (messages) within the application.
I did the following methods:
wait fixed amount of time: usually works, but if someone's PC is hot and throttling, it can become flaky, and to be honest this is just ugly
create an aspect to count the method invocations, serve it through a controller, query it multiple times, send the next message when we're "settled": timing of querying matters, unreliable
periodically check Kafka consumer lag, either from code or by querying actuator, with multiple samples: this is mixture of the above two, sometimes slower than the first but more reliable
Is there any official way of doing this?
Configure the container to emit ListenerContainerIdleEvents.
See https://docs.spring.io/spring-kafka/docs/current/reference/html/#idle-containers
I'm trying to figure out the best way to trigger all actors of a certain type in a sharded cluster, based on a time schedule (e.g. at 8am, 9am, etc - in cron-like fashion).
My plan was to have a single "timer" actor in the cluster that would send out a broadcast message to other cluster's actors on schedule. However, I'm not sure if this is workable and optimal. Akka Scheduler doesn't provide cron-like configuration. akka-quartz-scheduler doesn't seem to be fitted for Akka cluster.
Is it possible at all to trigger schedule-based actions from inside sharded Akka cluster, perhaps using some other framework's capabilities such as Spring scheduling? Or it's better to deploy a scheduling service outside of sharded Akka cluster, and use it to send periodic triggering events to Akka cluster?
Also, is it possible to broadcast a message to all actors of certain type in sharded Akka cluster?
Is it possible at all to trigger schedule-based actions from inside sharded Akka cluster?
In general yes. But it depends on the precision of the scheduled events and the type of the schedule. Let's assume a simple timetable schedule, e.g. everyday at 08:00, with no interest in high precision. It is possible to create an actor with TimerScheduler, lets name this actor TimerSchedulerActor. Given a specific time schedule e.g. everyday at 08:00 TimerSchedulerActor calculates the next time that the alarm needs to go off e.g. 01.09.2020 08:00, then TimerSchedulerActor compute the duration that it needs to wait from the current time java.lang.System.currentTimeMillis to 01.09.2020 08:00. When the TimerScheduler goes off TimerSchedulerActorsends the message and calculates the next alarm timestamp for this schedule.
If one TimerSchedulerActor is responsible for all messages it should be made sure that there is only TimerSchedulerActor running (Singleton actor) as multiple TimerSchedulerActors would send multiple messages for each scheduled event. You could also split have different TimerSchedulerActors to notify different groups of Actors or to be responsible for different events.
Or it's better to deploy a scheduling service outside of sharded Akka cluster, and use
it to send periodic events to Akka cluster?
It could be easier to maintain, easier to deploy and easier debug a scheduling service inside the Akka Cluster. However, the answer depends on the author skills and experience. Someone familiar with a scheduling system outside Akka, (e.g. cron) can perhaps find a faster solution by thinking outside Akka. I would advocate for a solution inside Akka Cluster as it offers more flexibility. For example, if facing a request to change cron (or another external system) from inside the java cluster the complexity would increase.
Also, is it possible to broadcast a message to all actors of certain type in sharded Akka cluster?
It depends on what the actor type is. Cluster Receptionist can be used to look up specific actors. If you are in control of the message and can add the desired actor type inside the messages and actors a good strategy could be to send the message to all potential actors of this type and let them act on the message based on their type. Actors of the correct type can act on the message and actors of another type could just ignore it.
You may either:
Expose an HTTP endpoint that will trigger some tasks, and let airflow or simply a cron job to hit the endpoint
Use quartz, either akka-quartz-scheduler or raw quartz. I think it fits akka cluster without a problem. What do you think is the blocker? A viable practice may be to run a cluster singleton actor inside which you run schedule-based actions based on quartz.
We are building an java application which will use embedded Neo4j for graph traversal. Below are the reasons why we want to use embedded version instead of centralized server
This app is not a data owner. Data will be ingested on it through other app. Keeping data locally will help us in doing quick calculation and hence it will improve our api sla.
Since data foot print is small we don't want to maintain centralized server which will incur additional cost and maintenance.
No need for additional cache
Now this architecture bring two challenges. First How to update data in all instance of embedded Neo4j application at same time. Second how to make sure that all instance are in sync i.e using same version of data.
We thought of using Kafka to solve first problem. Idea is to have kafka listener with different groupid(to ensure all get updates) in all instance . Whenever there is update, event will be posted in kafka. All instance will listen for event and will perform the update operation.
However we still don't have any solid design to solve second problem. For various reason one of the instance can miss the event (it's consumer is down). One of the way is to keep checking latest version by calling api of data owner app. If version is behind replay the events.But this brings additional complexity of maintaining the event logs of all updates. Do you guys think if it can be done in a better and simpler way?
Kafka consumers are extremely consistent and reliable once you have them configured properly, so there shouldn't be any reason for them to miss messages, unless there's an infrastructure problem, in which case any solution you architect will have problems. If the Kafka cluster is healthy (e.g. at least one of the copies of the data is available, and at least quorum zookeepers are up and running), then your consumers should receive every single message from the topics they're subscribed to. The consumer will handle the retries/reconnecting itself, as long as your timeout/retry configurations are sane. The default configs in the latest kafka versions are adequate 99% of the time.
Separately, you can add a separate thread, for example, that is constantly checking what the latest offset is per topic/partitions, and compare it to what the consumer has last received, and maybe issue an alert/warning if there is a discrepancy. In my experience, and with Kafka's reliability, it should be unnecessary, but it can give you peace of mind, and shouldn't be too difficult to add.
I can use the Java MQ Api to put and get messages.
I can also disable gets and put on a queue.
During a migration project, we'll have an App running in parallell. Old and New. Old and New will have their own separate queues. I regulary have messages from a client going to Old. Occasionally want the msgs to flow to New instead.
wondering if MQ supports a gate/switch concept. where via API I can point a queue to go only to New, or only to Old, for a short time.
Trying to avoid going to message based routing via WMB since I dont have to do that today. THe parallel mode is only for a few months.
You do not mention the version of MQ or whether there are message affinities or dependence on preserving the MQMD.MsgID. These are critical in devising a solution to this problem. I'll try to describe enough options so that at least one will be viable whatever version you are at.
Pub/Sub
The easiest thing to do is to have the messages arrive on an alias over a topic. Any message that arrives is published immediately on that topic. Then it is a simple matter to generate administrative subscriptions to direct messages to the queues on which the apps needing the messages are listening. This is entirely a configuration change and requires no external components, processes or code. It is available from v7.1 of MQ and higher, which is to say any of the currently supported versions of MQ.
The down side is that IBM MQ will change the MQMD.MsgID from the time the message is received on the topic to the time it is published on the application's input queue. This breaks the app's ability to use the MQMD.MsgID of the incoming message as a correlation ID when replying. If the requesting app pre-loads the correlation ID or doesn't rely on a correlation ID, this is not an issue.
Aliasing
But for apps where this is an issue, it gets a bit harder. You can alias over a queue and have inbound messages land on the alias. When you need to switch from one queue to another, you change the alias. There are a couple issues with this. The first is that it is never possible to deliver the message stream to more than one of the applications. In a parallel processing test it is often desirable to do exactly that and then compare summary or detail reports.
The second problem is more operational in nature. It isn't possible to change the alias while it is open. If the messages arrive over a RCVR, RQSTR or `CLUSRCVR channel, no problem. Stop the channe, switch the alias and restart the channel. In a series of MQSC script commands this can be done faster than it can be typed. However, if the applications putting the messages are connected in bindings mode or via client directly to the alias, they must all be stopped in order to change the alias.
That said, aliasing works on all versions of MQ out of the box.
Physical copy
One solution that's been around for quite some time is to use the Q program (SupportPac MA01) to direct the messages. In this scenario, the queue on which messages land is a local queue. The Q program is either triggered or set to constantly listen on the queue. When a message arrives, Q then copies it to one or both of the destination queues.
Switching the behavior if Q is triggered involves pre-defining 2 or 3 processes where each defines a different behavior - move new messages to QUEUEA, to QUEUEB or to both. Changing the queue's PROCESS attribute to point to a different process results in an instantaneous change of the behavior.
Alternatively, if Q is configured to listen on the queue forever then changing the behavior involves use of three different scripts to execute it where one causes messages to be copied to QUEUEA, another to QUEUEB and another to both queues. Changing the behavior involves killing the script and starting a different one.
The Q program works with all versions of MQ, regardless of whether it is triggered or scripted.
Downsides to this approach include the obvious - more moving parts. You have to trigger the queue or else make a transactional program act like a daemon. Not hard but if you are betting the business on it then perhaps some monitoring is in order to make sure the input queue doesn't start building.
Recommendation
Of all these methods, I really like the Pub/Sub version. It is extremely reliable, has the least moving parts, and if anything breaks it's under IBM support. When you need to change something, you can do that with minimal impact to the running applications. If at all possible, use that.
I've been asked to design and implement a system for receiving a high volume of automated sensor data from a large number of devices. This data will be produced at regular intervals and sent to the server as xml in an http post. The devices will keep resending the same data if they don't receive a specific acknowledgment from the server. Some potentially heavy duty processing of this data will need to occur before it's inserted to a number of tables in the main database via a transaction, and additionally some data points will need to be enqueued to be re-directed to other external urls.
I'm planning on using a Java application server (leaning towards GlassFish) with a servlet to receive the incoming data. I'd like to implement some kind of queuing mechanism to store the data temporarily so that the response back to the sensor isn't dependent on all the intermediate processing. Separate independent queues are also a requirement for the data re-direction piece. After doing some research the two main options seem to be:
1) Install a database on the app server and use tables for the various queues. The queues would be processed by a Java application, either running in the app server or standalone as it's own service.
2) Use a database backed JMS solution to implement the queuing.
I'm not that familiar with JMS but from what I've read it seems to be the better solution in this case. The primary requirement is that no sensor data ever be lost or dropped from the queue before being processed and that it be processed more or less sequentially. We'd also like to make it easy to halt the processing of some of the queues at certain times but still have them accumulate data and for these messages to never automatically expire.
With strategy 1 it's obvious to me how to meet these requirements but it may be less robust and scalable, and more complex to develop than strategy 2, since I'll need to write my own multi-threaded code to handle the various independent queues. I'm wondering what the potential pitfalls could be in using JMS queues for this purpose since I've never worked with them before.
Data integrity is a big issue so I need to make sure JMS can guarantee no data loss in the event of a server reboot, power outage, or if the queue gets very large for some reason. For instance could a problem completing transactions to the main database for a period of time potentially cause the JVM to run out of memory, crash, and lose all accumulated data? (This would be the nightmare scenario).
Also, I was wondering if there would be any way to pause the JMS queue processing via an app server admin tool or to easily see what's in the queue (I would be enqueuing an object which would be the message xml plus some other data, including timestamp received, etc.) I've read a few posts on here that deal with related issues but wanted to get some direct feedback. Basically I'd like to know of instances (if any) where JMS is not an appropriate queuing solution and if this is one of those cases. Any advice is greatly appreciated.
Kaleb's answer talks about the benefits of JMS quite eloquently, but since you're asking about pitfalls, here's what I can think of.
Not all JMS implementations are equal. In theory you can use whatever implementation suits your needs, but unless you're prepared to do some serious load testing and failure condition testing, you can't know that a particular implementation isn't going to fail under your particular use case.
Most JMS use a transactional datastore like a relational database as their back end. That means that rather than writing directly to whatever datastore you're familiar with, you have to rely on the JMS implementation's extra layer between you and that stored messages.
While swapping JMS implementations to find the one that perfectly fits your needs may seem like a simple endeavor because of the homogeneous JMS API, the critical features for failure handling, JMS server monitoring, and all the other cool stuff that exists above and beyond messaging is going to be a hassle to deal with if you do change your implementation.
That said, I think you'd be crazy to write to the DB yourself instead of going with JMS. On the first point, ActiveMQ is a venerable JMS server used in many enterprise environments. On the second point, the fact is you'd just end up writing that extra layer yourself in order to implement messaging, and your code won't have the benefit of thousands of eyes (or a set of paid developers who's sole job it is to respond to customers and make sure the JMS implementation is solid). On the third point, well the same ends up being true of your backend datastore. Use JMS, you'll save yourself trouble in the long run.
If you want to go the JMS route, a standalone JMS-compatible message broker (separate from your app server) would be a good choice. Message brokers range from free open-source (like ActiveMQ at http://activemq.apache.org/ or OpenMQ at https://mq.dev.java.net/), to large-scale commercial solutions (IBM's WebSphere MQ at http://www-01.ibm.com/software/integration/wmq/ is one of the largest).
Message brokers offer guaranteed delivery (provided the server's up and listening), and you can do quite a bit to ensure that the system is fail-safe including integrated backup broker servers and instant power backup. Broker queues can eventually run out of room if your app server isn't picking up the messages, but you can assign huge queue depth (100's of GB) and have the server send alerts if the messages aren't getting processed and the queue reaches a certain percentage.
Your Java app would then run on a different server entirely, and would connect to the broker and pull messages off of the queue as fast as possible. If the app server crashes or stops picking up messages for any other reason, the broker would just keep all messages in that queue until the app server begins picking them up again.
You will be wanting to implement a poison message queue in your implementation - this is the place that messages unable to be processed after some number of retries will arrive.
You will probably need to write some code that can examine the messages in that queue and re-send them to the appropriate destination after fixing whatever is causing them to fail.
If sequence of message processing is important, a message ending up in the poison queue could mean all processing is halted until that message is corrected.
As far as fault tolerance goes, you can have multiple instances of the consuming services subscribe to the same queue or topic, providing an ability to continue processing even if one or more instances goes down.
Finally, have a watchdog process that pings the various consumers on your message queue, and if one doesn't respond, have it send a message that results in a new instance being started. In this way, your message processing environment can be somewhat self regulating.