Develop Reference

track count of item published to event hub and store in execution context

I have an interface in a common maven project, there are two publish methods both returning boolean:
public interface IPublisher<T, U> {
boolean publish(Context ctx, T model);
boolean publish(Context ctx, List<? extends ModelHolder> model);
}
I have one implementation of this interface:
public class Publisher implements IPublisher<PriceHolder, JSONObject> {
private final DataPublisher dataPublisher;
public Publisher(final DataPublisher dataPublisher) {
this.dataPublisher = dataPublisher;
}
#Override
public boolean publish(Context context, PriceHolder priceHolder) {
if (isInvalid(priceHolder)) {
return false;
}
try {
dataPublisher.publish(data);
} catch (ConnectionException | DataBuilderException ex) {
String message = "someMessage";
LOGGER.error(message, ex);
} catch (TimeoutException e) {
LOGGER.error(message, e);
}
return true;
}
#Override
public boolean publish(Context ctx, List<? extends ModelHolder> list) {
if (list == null) {
return false;
}
for (ModelHolder item : list) {
publish(ctx, (PriceHolder) item);
}
return true;
}
private boolean isInvalid(PriceHolder priceHolder) {
}
}
Currently, this implementation does not not record how many have been published which is what I need to pass to the client who is calling publish().
The client instantiates a Spring Bean of the Publisher class:
#Bean
public IPublisher publisher(DataItemPublisher dataItemPublisher) {
return new Publisher(dataItemPublisher);
}
The client app is using spring batch, it publishes like this in workflow config class:
#Bean
#Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
public ItemWriter<PriceHolder> publish(#Value("#{jobExecutionContext['MY_CONTEXT']}") Context ctx) {
return items -> {
publisher.publish(ctx, items);
};
}
where publisher is IPublisher instance. The items are processed in chunks and at the end of processing I need a summary count of how many published but I am unsure how to achieve this since publish() returns boolean. I want to get total publish count then I can hold it in executionContext. so that I can use this count for reporting in subsequent step of my workflow. e.g. received vs published count. How can I achieve this?

Currently, this implementation does not not record how many have been published which is what I need to pass to the client who is calling publish().
Since this implementation does not provide a way for clients to know how many items were published, and since you said that The client app is using spring batch, you won't be able to get that information and store it the execution context of Spring Batch. You need to fix you interface and make it return a richer type than boolean.

Strategy for multiple subscribers and flow of data using RxJava 2

I cannot decide how to implement this task correctly using RxJava2.
The problem is following. I am recording audio using AuidoRecord.
Currently I have implemented the custom Flowable class like that
private class StreamAudioRecordRunnable extends Flowable<short[]> implements Runnable {
private int mShortBufferSize;
private List<Subscriber<? super short[]>> mSubscribers = new ArrayList<>();
private short[] mAudioShortBuffer;
private void removeAllNullableSubscribers() {
mSubscribers.removeAll(Collections.singleton(null));
}
private void notifyAllSubscribers(short[] audioBuffer) {
removeAllNullableSubscribers();
for (Subscriber<? super short[]> subscriber : mSubscribers) {
subscriber.onNext(audioBuffer);
}
}
#Override
protected void subscribeActual(Subscriber<? super short[]> newSubscriber) {
mSubscribers.add(newSubscriber);
}
private void notifyAllSubscribersAboutError(Throwable error) {
for (Subscriber<? super short[]> subscriber : mSubscribers) {
subscriber.onError(error);
}
}
#Override
public void run() {
// Init stuff
while (mIsRecording.get()) {
int ret;
ret = mAudioRecord.read(mAudioShortBuffer, 0, mShortBufferSize);
notifyAllSubscribers(mAudioShortBuffer);
}
mAudioRecord.release();
}
}
As you can see I am manually adding subscribers to the list. Then when I get new buffer all subscribers are notified.
I am guessing that this is not the most performant way to do this.
What I need
As far as this flowable running in a service. It should run until the service is alive, even if there are no subscribers.
Subscribers are not constant, they may subscribe and then unsubscribe, but the Flowable/Observable should still be running.
As the data emitted by the Flowable is the stream, subscribers should not be notified about already emitted items, they should only get current streaming data. Fire and forget.
The Flowable should run even all subscribers are gone.
Please suggest the right strategy to implement this.
I would be grateful for any help.

Something like
public class StreamAudioRecordRunnable {
private int mShortBufferSize;
private short[] mAudioShortBuffer;
private ConnectedFlowable<short[]> audioFlowable();
public StreamAudioRecordRunnable() {
audioFlowable = Flowable.create(new ObservableOnSubscribe<short[]>() {
#Override
public void subscribe(FlowableEmitter<short[]> emitter) throws Exception {
try {
while (mIsRecording.get()) {
int ret;
ret = mAudioRecord.read(mAudioShortBuffer, 0, mShortBufferSize);
emitter.onNext(mAudioShortBuffer);
}
emitter.onComplete();
mAudioRecord.release();
} catch (Exception e) {
emitter.onError(e);
mAudioRecord.release();
}
}
}).subscribeOn(Schedulers.io()).publish();
}
public Flowable<short[]> getFlowable() {
return audioFlowable.hide();
}
#Override
public void start() {
audioObservable.connect();
}
}
would be my preference.

Understanding supervisor strategy in Akka

I'm reading Akka documentation and now I'm at the section about UntypedActors. I decided to try some examples:
Here are my actors:
Parent
private static class MyUntypedActor extends UntypedActor{
public void onReceive(Object message) throws Exception {
System.out.println("Recieved: " + message);
}
#Override
public void preStart(){
getContext().actorOf(AnotherUntypedActor.props()).tell("Process it", getSelf());
}
public static Props props(){
return Props.create(MyUntypedActor.class);
}
}
Child
private static class AnotherUntypedActor extends UntypedActor{
public static Props props(){
return Props.create(AnotherUntypedActor.class);
}
public void onReceive(Object message) throws Exception {
System.out.println("My: " + message);
throw new RuntimeException("Crashed: " + getSelf());
}
}
main:
public static void main(String[] args) throws TimeoutException {
ActorSystem system = ActorSystem.create();
Inbox inbox = Inbox.create(system);
ActorRef actorRef = system.actorOf(MyUntypedActor.props());
inbox.send(actorRef, "Message");
}
so, my child actors experienced failure and I thought the it should have notified the parent somehow.
But what I recieved was this:
Recieved: Message
My: Process it
[ERROR] [07/14/2016 19:05:13.726] [default-akka.actor.default-dispatcher-4] [akka://default/user/$a/$a] Crashed: Actor[akka://default/user/$a/$a#-950392568]
What does the supervision do actually? Child actor had faulted and what? I got just an error message in a log. What does supervisorStrategy mean? It's by default set to
OneForOneStrategy(-1,Duration.Inf,true)

By using supervisor strategy, you decide what should be done with supervised actor if it fails. You have to override supervisionStrategy() method within your parent actor and define strategy. I.e. (not sure if it's correct since I use Scala for Akka)
#Override
public SupervisorStrategy supervisorStrategy() {
return strategy;
}
private static SupervisorStrategy strategy =
new OneForOneStrategy(10, Duration.create("1 minute"),
t -> {
if (t instanceof SomeException) {
return restart();
} else {
return stop();
}
});
In this case, if SomeException occurs, actor will be restarted. Otherwise, it will be stopped. You can choose one of four strategies.
Read documentation
Tip: Create specific exceptions!

How to use LeaderElection recipe efficiently using Curator for Zookeeper?

I am using Apache Curator library for doing leadership election on the Zookeeper. I have my application code deployed in various machines and I need to execute my code from one machine only so that's why I am doing leadership election on the zookeeper so that I can check if I am the leader, then execute this code.
Below is my LeaderElectionExecutor class which makes sure I am having one Curator instance per application
public class LeaderElectionExecutor {
private ZookeeperClient zookClient;
private static final String LEADER_NODE = "/testleader";
private static class Holder {
static final LeaderElectionExecutor INSTANCE = new LeaderElectionExecutor();
}
public static LeaderElectionExecutor getInstance() {
return Holder.INSTANCE;
}
private LeaderElectionExecutor() {
try {
String hostname = Utils.getHostName();
String nodes = "host1:2181,host2:2181;
zookClient = new ZookeeperClient(nodes, LEADER_NODE, hostname);
zookClient.start();
// added sleep specifically for the leader to get selected
// since I cannot call isLeader method immediately after starting the latch
TimeUnit.MINUTES.sleep(1);
} catch (Exception ex) {
// logging error
System.exit(1);
}
}
public ZookeeperClient getZookClient() {
return zookClient;
}
}
And below is my ZookeeperClient code -
// can this class be improved in any ways?
public class ZookeeperClient {
private CuratorFramework client;
private String latchPath;
private String id;
private LeaderLatch leaderLatch;
public ZookeeperClient(String connString, String latchPath, String id) {
client = CuratorFrameworkFactory.newClient(connString, new ExponentialBackoffRetry(1000, Integer.MAX_VALUE));
this.id = id;
this.latchPath = latchPath;
}
public void start() throws Exception {
client.start();
leaderLatch = new LeaderLatch(client, latchPath, id);
leaderLatch.start();
}
public boolean isLeader() {
return leaderLatch.hasLeadership();
}
public Participant currentLeader() throws Exception {
return leaderLatch.getLeader();
}
public void close() throws IOException {
leaderLatch.close();
client.close();
}
public CuratorFramework getClient() {
return client;
}
public String getLatchPath() {
return latchPath;
}
public String getId() {
return id;
}
public LeaderLatch getLeaderLatch() {
return leaderLatch;
}
}
Now in my application, I am using the code like this -
public void method01() {
ZookeeperClient zookClient = LeaderElectionExecutor.getInstance().getZookClient();
if (zookClient.isLeader()) {
// do something
}
}
public void method02() {
ZookeeperClient zookClient = LeaderElectionExecutor.getInstance().getZookClient();
if (zookClient.isLeader()) {
// do something
}
}
Problem Statement:-
In the Curator library - Calling isLeader() immediately after starting the latch will not work. It takes time for the leader to get selected. And because of this reason only, I have added a sleep of 1 minute in my LeaderElectionExecutor code which works fine but I guess is not the right way to do this.
Is there any better way of doing this? Keeping this in mind, I need a way to check whether I am the leader then execute this piece of code. I cannot do everything in a single method so I need to call isLeader method from different classes and methods to check if I am the leader then execute this piece of code only.
I am using Zookeeper 3.4.5 and Curator 1.7.1 version.

Once I solved a problem very similar to yours. This is how I did it.
First, I had my objects managed by Spring. So, I had a LeaderLatch that was injectable through the container. One of the components that used the LeaderLatch was a LeadershipWatcher, an implementation of Runnable interface that would dispatch the leadership event to other components. These last components were implementations of an interface that I named LeadershipObserver. The implementation of the LeadershipWatcher was mostly like the following code:
#Component
public class LeadershipWatcher implements Runnable {
private final LeaderLatch leaderLatch;
private final Collection<LeadershipObserver> leadershipObservers;
/* constructor with #Inject */
#Override
public void run() {
try {
leaderLatch.await();
for (LeadershipObserver observer : leadershipObservers) {
observer.granted();
}
} catch (InterruptedException e) {
for (LeadershipObserver observer : leadershipObservers) {
observer.interrupted();
}
}
}
}
As this is just a sketch-up, I recommend you to enhance this code, maybe applying the command pattern for calling the observers, or even submitting the observers to thread pools, if their job are blocking or long-running CPU intensive tasks.

I've not worked with zookeeper or curator before, so take my answer with a grain of salt.
Set a flag.
Boolean isLeaderSelected = false;
At the beginning of the Latch, set the flag to false.
When the leader has been selected, set the flag to true.
In the isLeader() function:
isLeader(){
while(!isLeaderSelected){} //waits until leader is selected
//do the rest of the function
}
This is also a relatively hacky workaround, but it should allow the isLeader method to execute as soon as it can. In the case that they are in different classes, a getter should be able to provide isLeaderSelected.

leaderLatch = new LeaderLatch(curatorClient, zkPath, String.valueOf(new Random().nextInt()));
leaderLatch.start();
Participant participant;
while(true) {
participant = leaderLatch.getLeader();
// Leader election happens asynchronously after calling start, this is a hack to wait until election happens
if (!(participant.getId().isEmpty() || participant.getId().equalsIgnoreCase(""))) {
break;
}
}
if(leaderLatch.hasLeadership()) {
...
}
Note that getLeader returns a dummy participant with id "" until it elects a leader.

Here's to reviving an old question...
This is similar to the answer srav gave, but I would caution against using that code because it utilizes a busy-wait and can cause certain callbacks that are issued in-thread to never be called, possibly blocking forever. Furthermore, it could retry forever if there are real issues.
This was my solution, which utilizes the CuratorClient's retry policy to attempt waiting on leadership election if necessary.
RetryPolicy retryPolicy = _client.getZookeeperClient().getRetryPolicy();
RetrySleeper awaitLeadership = _leaderLatch::await;
final long start = System.currentTimeMillis();
int count = 0;
do {
try {
// curator will return a dummy leader in the case when a leader has
// not yet actually been elected. This dummy leader will have isLeader
// set to false, so we need to check that we got a true leader
if (_leaderLatch.getLeader().isLeader()) {
return;
}
} catch (KeeperException.NoNodeException e) {
// this is the case when the leader node has not yet been created
// by any client - this is fine because we are still waiting for
// the algorithm to start up so we ignore the error
}
} while (retryPolicy.allowRetry(count++, System.currentTimeMillis() - start, awaitLeadership));
// we have exhausted the retry policy and still have not elected a leader
throw new IOException("No leader was elected within the specified retry policy!");
Though taking a look at your CuratorFramework initialization I'd caution against using Integer.MAX_VALUE when specifying the retry policy...
I hope this helps!

Reliable data transfer over an UnReliable Channel

I have a data channel which can transfer some data over it, the channel is a wireless system implemented by myself with a low reliability 90% and very low bandwidth due to physical limitation.
In order to overcome this, I'm planning to wrap the whole data channel with a system which should use some data correct method, and send request for resend when when the data is corrupted (corruption will be checked with a checksum).
Whenever one of the wrapper receives a bad data it will send a resend request, and hold place in memory for the unknown data, in a stack.The stack will grow quickly when the reliability drops down, since each side will start send resend request to each other since it had not received the last resend request. Even when the reliability returns to normal, it will try to resend all the resend requests until the stack goes empty.
This will affect the bandwidth since most of the request won't be data, but resend requests.Moreover this system will run on a microcontroller with a very limited RAM, just a few bytes, which may cause a stack overflow in rare cases.
Any suggestions?
Here is a Java model which describes the data channel:
public interface DataChannel {
abstract void send(String s);
abstract void setOnDataListener(OnDataListener l);
interface OnDataListener {
abstract void onData(String s);
}
}
Here is an abstract class for a DataChannel which simplifies the implementation later on
public abstract class AbstractReliableChannel implements DataChannel,OnDataListener {
protected DataChannel mUnReliable;
private OnDataListener mUnDataListener;
public AbstractReliableChannel(DataChannel unReliableChannel){
mUnReliable = unReliableChannel;
}
#Override
public abstract void send(String s);
#Override
final public void setOnDataListener(OnDataListener l) {
mUnDataListener = l;
}
/*
* Should be called by the implimanting class
*/
final protected void notifiyListenerThatDataReceived(String s){
mUnDataListener.onData(s);
}
/**
* should be called by the implanting class
*/
final protected void sendOverUnreliableChannel(String s){
mUnReliable.send(s);
}
}
Here is an implementation of an UnReliable Channel
public class UnReliableChannel extends AbstractReliableChannel {
public ReliableChannel(DataChannel unReliableChannel) {
super(unReliableChannel);
}
#Override
public void send(String s) {
if( new Random().nextInt(10) % 5 == 0 )
s = ModifyStringRandomly(s);
sendOverUnreliableChannel(s);
}
#Override
public void onData(String s) {
if( new Random().nextInt(10) % 5 == 0 )
s = ModifyStringRandomly(s);
notifiyListenerThatDataReceived(s);
}
}
Here is a reliable channel implementation which i described erlier
public class ReliableChannel extends AbstractReliableChannel implements Runnable {
public static String DATA = "D";
public static String RESEND = "R";
public static String OK = "O";
private Thread mThread;
public ReliableChannel(DataChannel unReliableChannel) {
super(unReliableChannel);
mThread = new Thread(this);
mThread.start();
}
private Stack<String> mSend;
#Override
public void send(String s) {
mSend.add(s);
}
#Override
public void onData(String s) {
if(isDataValid(s)){
if(s.equals(RESEND)){
String toResend = mSend.pop();
mSend.push(toResend);
mThread.notify();
} else if (s.equals(OK) ) {
mSend.pop();
mThread.notify();
} else if(s.startsWith(DATA)){
notifiyListenerThatDataReceived(s.substring(1));
mSend.push(OK);
}
} else {
mSend.add(RESEND);
mThread.notify();
}
}
private void sendOverUnreliableChannelWithCheckSum(String s){
// ADD checkSUM
sendOverUnreliableChannel(RESEND);
}
#Override
public void run() {
while(true){
while(mSend.isEmpty())
;
sendOverUnreliableChannelWithCheckSum(mSend.pop());
mThread.wait();
}
}
private boolean isDataValid(String s){
// SHOULD BE SOME CHECKSUM IMPLEMINTATION
return true;
}
}

The problem comes from your inefficient protocol design, rather than programming. To get a reliable link in lossy channel, simply use tcp connection or define a protocol similar to tcp. Basically, number each of your data packet at Tx. At Rx, when you receive a bad packet, just throw it away to save memory. You check the integrity of your packets by checking whether all packets have continuous numbers. By maintaining a proper sliding window, you will reach both good effective bandwidth and affordable memory usage.