I need to limit messages received on websocket channel per second for netty server.
Could'n find any ideas how to do that.
Any ideas would be appreciated
Thank you
You need to add simple ChannelInboundHandlerAdapter handler to your pipeline and add the simple counter to channelRead(ChannelHandlerContext ctx, Object msg) method. I would recommend you to use some of CodaHale Metrics Class for that purpose.
Pseudo code:
private final QuotaLimitChecker limitChecker;
public MessageDecoder() {
this.limitChecker = new QuotaLimitChecker();
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
if (limitChecker.quotaReached(100)) { //assume limit is 100 req per sec
return;
}
}
Where QuotaLimitChecker is a class that increments counter and checks if the limit is reached.
public class QuotaLimitChecker {
private final static Logger log = LogManager.getLogger(QuotaLimitChecker.class);
private final int userQuotaLimit;
//here is specific implementation of Meter for your needs
private final InstanceLoadMeter quotaMeter;
public QuotaLimitChecker(int userQuotaLimit) {
this.userQuotaLimit = userQuotaLimit;
this.quotaMeter = new InstanceLoadMeter();
}
public boolean quotaReached() {
if (quotaMeter.getOneMinuteRate() > userQuotaLimit) {
log.debug("User has exceeded message quota limit.");
return true;
}
quotaMeter.mark();
return false;
}
}
Here is my implementation of QuotaLimitChecker that uses the simplified version Meter class of CodaHale Metrics library.
Related
Do we have custom serialization capability for EntryProcessor or ExecutorService ?. Hazelcast document is not specifying anything in this regard. There are no samples given in the document related to custom serialization of EntryProcessor. We are looking for a Portable serialization of the EntryProcessor.
public class SampleEntryProcessor implements EntryProcessor<SampleDataKey, SampleDataValue , SampleDataValue >,Portable {
/**
*
*/
private static final long serialVersionUID = 1L;
private SampleDataValue sampleDataValue ;
public SampleDataValue process(Map.Entry<SampleDataKey, SampleDataValue > entry) {
//Sample logic here
return null;
}
#Override
public int getFactoryId() {
return 1;
}
#Override
public int getClassId() {
return 1;
}
#Override
public void writePortable(PortableWriter writer) throws IOException {
writer.writePortable("i", sampleDataValue );
}
#Override
public void readPortable(PortableReader reader) throws IOException {
sampleDataValue = reader.readPortable("i");
}
}
UPDATE : When i try to call processor am getting error as follows.
Exception in thread "main" java.lang.ClassCastException: com.hazelcast.internal.serialization.impl.portable.DeserializedPortableGenericRecord cannot be cast to com.hazelcast.map.EntryProcessor
at com.hazelcast.client.impl.protocol.task.map.MapExecuteOnKeyMessageTask.prepareOperation(MapExecuteOnKeyMessageTask.java:42)
at com.hazelcast.client.impl.protocol.task.AbstractPartitionMessageTask.processInternal(AbstractPartitionMessageTask.java:45)
Yes, you can use different serialization mechanisms to serialize entry processors, provided that they are correctly configured on the sender and receiver sides. So, after making sure that the Portable factory for your class is registered on the members and on the instance you are sending the entry processor from (for example, your client), it should work.
I want to detect if two events happen in a defined timeframe based on two events that have the same identifier. For example a DoorEvent looks like this:
<doorevent>
<door>
<id>1</id>
<status>open</status>
</door>
<timestamp>12345679</timestamp>
</doorevent>
<doorevent>
<door>
<id>1</id>
<status>close</status>
</door>
<timestamp>23456790</timestamp>
</doorevent>
My DoorEvent java class in the example below has the same structure.
I want to detect that door with id 1 closes within 5 minutes of opening. I try to use the Apache flink CEP library for this purpose. The incoming stream contains all open and close messages from lets say 20 doors.
Pattern<String, ?> pattern = Pattern.<String>begin("door_open").where(
new SimpleCondition<String>() {
private static final long serialVersionUID = 1L;
public boolean filter(String doorevent) {
DoorEvent event = new DoorEvent().parseInstance(doorevent, DataType.XML);
if (event.getDoor().getStatus().equals("open")){
// save state of door as open
return true;
}
return false;
}
}
)
.followedByAny("door_close").where(
new SimpleCondition<String>() {
private static final long serialVersionUID = 1L;
public boolean filter(String doorevent) throws JsonParseException, JsonMappingException, IOException {
DoorEvent event = new DoorEvent().parseInstance(doorevent, DataType.XML);
if (event.getDoor().getStatus().equals("close")){
// check if close is of previously opened door
return true;
}
return false;
}
}
)
.within(Time.minutes(5));
How do I save the state of door 1 as open in the door_open so that in the door_close step I know that door 1 is the one being closed and it is not some other door?
If you have Flink 1.3.0 and above its really straightforard what you want to do
Your pattern would look something like this:
Pattern.<DoorEvent>begin("first")
.where(new SimpleCondition<DoorEvent>() {
private static final long serialVersionUID = 1390448281048961616L;
#Override
public boolean filter(DoorEvent event) throws Exception {
return event.getDoor().getStatus().equals("open");
}
})
.followedBy("second")
.where(new IterativeCondition<DoorEvent>() {
private static final long serialVersionUID = -9216505110246259082L;
#Override
public boolean filter(DoorEvent secondEvent, Context<DoorEvent> ctx) throws Exception {
if (!secondEvent.getDoor().getStatus().equals("close")) {
return false;
}
for (DoorEvent firstEvent : ctx.getEventsForPattern("first")) {
if (secondEvent.getDoor().getEventID().equals(firstEvent.getDoor().getEventId())) {
return true;
}
}
return false;
}
})
.within(Time.minutes(5));
So basically you can use IterativeConditions and get the context for the first patterns which are matched and iterate over that list while comparing for the one you need and proceed as you want.
IterativeConditions are expensive and should be handled accordingly
For more information on conditions check here at Flink - Conditions
I am working in a method (using spring) that will manage a lot of data and information, consulting to the database and generate some files.
I am trying to avoid a timeout exception, so, I decided I should use the #Async annotation.
Not quite sure if it works as I think or not, but I also realized that I will need the method who calls Async to wait until it is finished...so, could be the same problem, couldn't it?
Is there any way I can have a sort of listener that will read the Async information that is being processed at my bean without have to wait for all the Async process to finish??
Right now is somehow like this
private Long myFIrstMethod(){
// DO A LOT OF THINGS AND CALL TO MY ASYNC METHOD
// evaluate if the Async method will have something or not... and based on it make the return
if (myOtherMethod()){
return soemvalue;
}else{
return someOtherValue
}
#Async Future<Boolean> myOtherMethod() {
//do something
new AsyncResult<Boolean>(true); //or false...
}
}
So, I was thinking, I might get a timeout exception on myFirstMethod is there any way to handle long time processing methods and avoiding this exception?
Thanks.
You could use a Timeout
http://sourceforge.net/p/tus/code/HEAD/tree/tjacobs/io/TimeOut.java
Set your timeout length to the length you want to wait. In the meantime, should your method return in a timely manner, you can cancel the TimeOut.
package tjacobs.io;
public class TimeOut implements Runnable {
private long mWaitTime;
private boolean mRunning = true;
private Thread mMyThread;
private TimeOutCmd mTimeOutCmd;
public static final int DEFAULT_URL_WAIT_TIME = 30 * 1000; // 30 Seconds
public static final int NO_TIMEOUT = -1;
public static final int DEFAULT_WAIT_TIME = NO_TIMEOUT;
public static interface TimeOutCmd {
public void timeOut();
}
public TimeOut(TimeOutCmd cmd) {
this(cmd, DEFAULT_WAIT_TIME);
}
public TimeOut(TimeOutCmd cmd, int timeToWait) {
mWaitTime = timeToWait;
mTimeOutCmd = cmd;
}
public void stop() {
mRunning = false;
mTimeOutCmd.timeOut();
if (mMyThread != null) mMyThread.interrupt();
}
/**
* reset the TimeOut
*
*/
public void tick() {
if (mMyThread != null)
mMyThread.interrupt();
}
public void run () {
mMyThread = Thread.currentThread();
while (true) {
try {
Thread.sleep(mWaitTime);
stop();
}
catch (InterruptedException ex) {
if (!mRunning) {
return;
}
}
}
}
}
I am using Apache Curator library for doing leadership election on the Zookeeper. I have my application code deployed in various machines and I need to execute my code from one machine only so that's why I am doing leadership election on the zookeeper so that I can check if I am the leader, then execute this code.
Below is my LeaderElectionExecutor class which makes sure I am having one Curator instance per application
public class LeaderElectionExecutor {
private ZookeeperClient zookClient;
private static final String LEADER_NODE = "/testleader";
private static class Holder {
static final LeaderElectionExecutor INSTANCE = new LeaderElectionExecutor();
}
public static LeaderElectionExecutor getInstance() {
return Holder.INSTANCE;
}
private LeaderElectionExecutor() {
try {
String hostname = Utils.getHostName();
String nodes = "host1:2181,host2:2181;
zookClient = new ZookeeperClient(nodes, LEADER_NODE, hostname);
zookClient.start();
// added sleep specifically for the leader to get selected
// since I cannot call isLeader method immediately after starting the latch
TimeUnit.MINUTES.sleep(1);
} catch (Exception ex) {
// logging error
System.exit(1);
}
}
public ZookeeperClient getZookClient() {
return zookClient;
}
}
And below is my ZookeeperClient code -
// can this class be improved in any ways?
public class ZookeeperClient {
private CuratorFramework client;
private String latchPath;
private String id;
private LeaderLatch leaderLatch;
public ZookeeperClient(String connString, String latchPath, String id) {
client = CuratorFrameworkFactory.newClient(connString, new ExponentialBackoffRetry(1000, Integer.MAX_VALUE));
this.id = id;
this.latchPath = latchPath;
}
public void start() throws Exception {
client.start();
leaderLatch = new LeaderLatch(client, latchPath, id);
leaderLatch.start();
}
public boolean isLeader() {
return leaderLatch.hasLeadership();
}
public Participant currentLeader() throws Exception {
return leaderLatch.getLeader();
}
public void close() throws IOException {
leaderLatch.close();
client.close();
}
public CuratorFramework getClient() {
return client;
}
public String getLatchPath() {
return latchPath;
}
public String getId() {
return id;
}
public LeaderLatch getLeaderLatch() {
return leaderLatch;
}
}
Now in my application, I am using the code like this -
public void method01() {
ZookeeperClient zookClient = LeaderElectionExecutor.getInstance().getZookClient();
if (zookClient.isLeader()) {
// do something
}
}
public void method02() {
ZookeeperClient zookClient = LeaderElectionExecutor.getInstance().getZookClient();
if (zookClient.isLeader()) {
// do something
}
}
Problem Statement:-
In the Curator library - Calling isLeader() immediately after starting the latch will not work. It takes time for the leader to get selected. And because of this reason only, I have added a sleep of 1 minute in my LeaderElectionExecutor code which works fine but I guess is not the right way to do this.
Is there any better way of doing this? Keeping this in mind, I need a way to check whether I am the leader then execute this piece of code. I cannot do everything in a single method so I need to call isLeader method from different classes and methods to check if I am the leader then execute this piece of code only.
I am using Zookeeper 3.4.5 and Curator 1.7.1 version.
Once I solved a problem very similar to yours. This is how I did it.
First, I had my objects managed by Spring. So, I had a LeaderLatch that was injectable through the container. One of the components that used the LeaderLatch was a LeadershipWatcher, an implementation of Runnable interface that would dispatch the leadership event to other components. These last components were implementations of an interface that I named LeadershipObserver. The implementation of the LeadershipWatcher was mostly like the following code:
#Component
public class LeadershipWatcher implements Runnable {
private final LeaderLatch leaderLatch;
private final Collection<LeadershipObserver> leadershipObservers;
/* constructor with #Inject */
#Override
public void run() {
try {
leaderLatch.await();
for (LeadershipObserver observer : leadershipObservers) {
observer.granted();
}
} catch (InterruptedException e) {
for (LeadershipObserver observer : leadershipObservers) {
observer.interrupted();
}
}
}
}
As this is just a sketch-up, I recommend you to enhance this code, maybe applying the command pattern for calling the observers, or even submitting the observers to thread pools, if their job are blocking or long-running CPU intensive tasks.
I've not worked with zookeeper or curator before, so take my answer with a grain of salt.
Set a flag.
Boolean isLeaderSelected = false;
At the beginning of the Latch, set the flag to false.
When the leader has been selected, set the flag to true.
In the isLeader() function:
isLeader(){
while(!isLeaderSelected){} //waits until leader is selected
//do the rest of the function
}
This is also a relatively hacky workaround, but it should allow the isLeader method to execute as soon as it can. In the case that they are in different classes, a getter should be able to provide isLeaderSelected.
leaderLatch = new LeaderLatch(curatorClient, zkPath, String.valueOf(new Random().nextInt()));
leaderLatch.start();
Participant participant;
while(true) {
participant = leaderLatch.getLeader();
// Leader election happens asynchronously after calling start, this is a hack to wait until election happens
if (!(participant.getId().isEmpty() || participant.getId().equalsIgnoreCase(""))) {
break;
}
}
if(leaderLatch.hasLeadership()) {
...
}
Note that getLeader returns a dummy participant with id "" until it elects a leader.
Here's to reviving an old question...
This is similar to the answer srav gave, but I would caution against using that code because it utilizes a busy-wait and can cause certain callbacks that are issued in-thread to never be called, possibly blocking forever. Furthermore, it could retry forever if there are real issues.
This was my solution, which utilizes the CuratorClient's retry policy to attempt waiting on leadership election if necessary.
RetryPolicy retryPolicy = _client.getZookeeperClient().getRetryPolicy();
RetrySleeper awaitLeadership = _leaderLatch::await;
final long start = System.currentTimeMillis();
int count = 0;
do {
try {
// curator will return a dummy leader in the case when a leader has
// not yet actually been elected. This dummy leader will have isLeader
// set to false, so we need to check that we got a true leader
if (_leaderLatch.getLeader().isLeader()) {
return;
}
} catch (KeeperException.NoNodeException e) {
// this is the case when the leader node has not yet been created
// by any client - this is fine because we are still waiting for
// the algorithm to start up so we ignore the error
}
} while (retryPolicy.allowRetry(count++, System.currentTimeMillis() - start, awaitLeadership));
// we have exhausted the retry policy and still have not elected a leader
throw new IOException("No leader was elected within the specified retry policy!");
Though taking a look at your CuratorFramework initialization I'd caution against using Integer.MAX_VALUE when specifying the retry policy...
I hope this helps!
I have a data channel which can transfer some data over it, the channel is a wireless system implemented by myself with a low reliability 90% and very low bandwidth due to physical limitation.
In order to overcome this, I'm planning to wrap the whole data channel with a system which should use some data correct method, and send request for resend when when the data is corrupted (corruption will be checked with a checksum).
Whenever one of the wrapper receives a bad data it will send a resend request, and hold place in memory for the unknown data, in a stack.The stack will grow quickly when the reliability drops down, since each side will start send resend request to each other since it had not received the last resend request. Even when the reliability returns to normal, it will try to resend all the resend requests until the stack goes empty.
This will affect the bandwidth since most of the request won't be data, but resend requests.Moreover this system will run on a microcontroller with a very limited RAM, just a few bytes, which may cause a stack overflow in rare cases.
Any suggestions?
Here is a Java model which describes the data channel:
public interface DataChannel {
abstract void send(String s);
abstract void setOnDataListener(OnDataListener l);
interface OnDataListener {
abstract void onData(String s);
}
}
Here is an abstract class for a DataChannel which simplifies the implementation later on
public abstract class AbstractReliableChannel implements DataChannel,OnDataListener {
protected DataChannel mUnReliable;
private OnDataListener mUnDataListener;
public AbstractReliableChannel(DataChannel unReliableChannel){
mUnReliable = unReliableChannel;
}
#Override
public abstract void send(String s);
#Override
final public void setOnDataListener(OnDataListener l) {
mUnDataListener = l;
}
/*
* Should be called by the implimanting class
*/
final protected void notifiyListenerThatDataReceived(String s){
mUnDataListener.onData(s);
}
/**
* should be called by the implanting class
*/
final protected void sendOverUnreliableChannel(String s){
mUnReliable.send(s);
}
}
Here is an implementation of an UnReliable Channel
public class UnReliableChannel extends AbstractReliableChannel {
public ReliableChannel(DataChannel unReliableChannel) {
super(unReliableChannel);
}
#Override
public void send(String s) {
if( new Random().nextInt(10) % 5 == 0 )
s = ModifyStringRandomly(s);
sendOverUnreliableChannel(s);
}
#Override
public void onData(String s) {
if( new Random().nextInt(10) % 5 == 0 )
s = ModifyStringRandomly(s);
notifiyListenerThatDataReceived(s);
}
}
Here is a reliable channel implementation which i described erlier
public class ReliableChannel extends AbstractReliableChannel implements Runnable {
public static String DATA = "D";
public static String RESEND = "R";
public static String OK = "O";
private Thread mThread;
public ReliableChannel(DataChannel unReliableChannel) {
super(unReliableChannel);
mThread = new Thread(this);
mThread.start();
}
private Stack<String> mSend;
#Override
public void send(String s) {
mSend.add(s);
}
#Override
public void onData(String s) {
if(isDataValid(s)){
if(s.equals(RESEND)){
String toResend = mSend.pop();
mSend.push(toResend);
mThread.notify();
} else if (s.equals(OK) ) {
mSend.pop();
mThread.notify();
} else if(s.startsWith(DATA)){
notifiyListenerThatDataReceived(s.substring(1));
mSend.push(OK);
}
} else {
mSend.add(RESEND);
mThread.notify();
}
}
private void sendOverUnreliableChannelWithCheckSum(String s){
// ADD checkSUM
sendOverUnreliableChannel(RESEND);
}
#Override
public void run() {
while(true){
while(mSend.isEmpty())
;
sendOverUnreliableChannelWithCheckSum(mSend.pop());
mThread.wait();
}
}
private boolean isDataValid(String s){
// SHOULD BE SOME CHECKSUM IMPLEMINTATION
return true;
}
}
The problem comes from your inefficient protocol design, rather than programming. To get a reliable link in lossy channel, simply use tcp connection or define a protocol similar to tcp. Basically, number each of your data packet at Tx. At Rx, when you receive a bad packet, just throw it away to save memory. You check the integrity of your packets by checking whether all packets have continuous numbers. By maintaining a proper sliding window, you will reach both good effective bandwidth and affordable memory usage.