Develop Reference

ZMQ missing events being propagated in jeromq scala

I am new to ZeroMQ and seem to be losing messages in a loop in my begin() method.
I'm wondering if I am missing a piece where I am not queuing messages or something?
When I cause an event on my publisher, that sends two messages to my subscriber with a small gap in between, I seem not to be getting the second message that is relayed. What am I missing?
class ZMQSubscriber[T <: Transaction, B <: Block](
socket: InetSocketAddress,
hashTxListener: Option[HashDigest => Future[Unit]],
hashBlockListener: Option[HashDigest => Future[Unit]],
rawTxListener: Option[Transaction => Future[Unit]],
rawBlockListener: Option[Block => Future[Unit]]) {
private val logger = BitcoinSLogger.logger
def begin()(implicit ec: ExecutionContext) = {
val context = ZMQ.context(1)
// First, connect our subscriber socket
val subscriber = context.socket(ZMQ.SUB)
val uri = socket.getHostString + ":" + socket.getPort
//subscribe to the appropriate feed
hashTxListener.map { _ =>
subscriber.subscribe(HashTx.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to the transaction hashes from zmq")
}
rawTxListener.map { _ =>
subscriber.subscribe(RawTx.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to raw transactions from zmq")
}
hashBlockListener.map { _ =>
subscriber.subscribe(HashBlock.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to the hashblock stream from zmq")
}
rawBlockListener.map { _ =>
subscriber.subscribe(RawBlock.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to raw block")
}
subscriber.connect(uri)
subscriber.setRcvHWM(0)
logger.info("Connection to zmq client successful")
while (true) {
val notificationTypeStr = subscriber.recvStr(ZMQ.DONTWAIT)
val body = subscriber.recv(ZMQ.DONTWAIT)
Future(processMsg(notificationTypeStr, body))
}
}
private def processMsg(topic: String, body: Seq[Byte])(implicit ec: ExecutionContext): Future[Unit] = Future {
val notification = ZMQNotification.fromString(topic)
val res: Option[Future[Unit]] = notification.flatMap {
case HashTx =>
hashTxListener.map { f =>
val hash = Future(DoubleSha256Digest.fromBytes(body))
hash.flatMap(f(_))
}
case RawTx =>
rawTxListener.map { f =>
val tx = Future(Transaction.fromBytes(body))
tx.flatMap(f(_))
}
case HashBlock =>
hashBlockListener.map { f =>
val hash = Future(DoubleSha256Digest.fromBytes(body))
hash.flatMap(f(_))
}
case RawBlock =>
rawBlockListener.map { f =>
val block = Future(Block.fromBytes(body))
block.flatMap(f(_))
}
}
}
}

So this seems to have been solved by using a ZMsg.recvMsg() in the while-loop instead of
val notificationTypeStr = subscriber.recvStr(ZMQ.DONTWAIT)
val body = subscriber.recv(ZMQ.DONTWAIT)
I'm not sure why this works, but it does. So here is what my begin method looks like now
while (run) {
val zmsg = ZMsg.recvMsg(subscriber)
val notificationTypeStr = zmsg.pop().getString(ZMQ.CHARSET)
val body = zmsg.pop().getData
Future(processMsg(notificationTypeStr, body))
}
Future.successful(Unit)
}

What am I missing?
How the blocking v/s non-blocking modus operandi work :
The trick is in the (non-)blocking mode of the respective call to the .recv() method.
A second call to the subscriber.recv( ZMQ.DONTWAIT )-method thus returns immediately, so your second part, ( the body ) may and will legally contain nothing, even though your promise stated a pair of messages was indeed dispached from the publisher-side ( a pair of .send() method calls - one may also object, there are chances the sender was actually sending just one message, in a multi-part fashion - MCVE-code is not specific on this part ).
So, once you have moved your code from non-blocking mode ( in the O/P ) into a principally blocking-mode ( which locked / sync-ed the further flow of the code with the external event of an arrival of any plausibly formatted message, not returning earlier ), in:
val zmsg = ZMsg.recvMsg(subscriber) // which BLOCKS-till-a-1st-zmsg-arrived
both the further processed .pop()-ed parts just unload the components ( ref. the remark on actual ZMsg multi-part structure actually sent by the published-side, presented above )
Safety next :unlimited alloc-s v/s a mandatory blocking / dropping messages ?
the code surprised me on several points. Besides a rather very "late" call to the .connect()-method, compared to all the previous socket-archetype detailed settings ( that normally get arranged "after" a request to setup a connection ). While this may work fine, as intended, yet it exposes even tighter ( smaller ) time-window for the .Context()-instance to setup and (re-)negotiate all the relevant connection-details so as to become RTO.
One particular line attracted my attention: subscriber.setRcvHWM( 0 ) this is a version-archetype dependent trick. Yet, the value of zero causes an application to become vulnerable and I would not advise doing so in any production-grade application.

Why does my JGit checkout return null ref?

I've the following code:
fun checkoutBranch(path: Path, name: String) {
Git.open(path.toFile()).use { git ->
val branchExists = git
.branchList()
.setListMode(ListBranchCommand.ListMode.ALL)
.call()
.filterNot { it.name.startsWith("refs/remotes/") }
.map { it.name }
.any { it.endsWith(name) }
val ref = git
.checkout()
.setCreateBranch(!branchExists)
.setName(name)
.setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK)
.call()
}
}
When I call it with name = master, everything works as expected. A subsequent call with name = test causes a new branch to be created, but ref is null. Looking at CheckoutCommand#L285, it seems that ref.name = refs/heads/master for master, but for test, ref.name = refs/tags/test, and ref is then set to null.
Ref ref = repo.findRef(name);
if (ref != null && !ref.getName().startsWith(Constants.R_HEADS))
ref = null;
What is happening here? Is this the expected behavior for a new branch? By going into the repo, I can see that it is in detached HEAD state, perhaps causing this issue.

Thanks to #ElpieKay for the clue. The problem was caused by the presence of a tag named test. Apparently, JGit prefers tags over branches when looking for references.
I solved the issue by explicitly specifying the branch name as refs/heads/test (in setName).

OptimisticLockException While deleting EBeans in Play 2 Framework

I am trying to delete an object from a list of an other object. But I'm stuck with a strange Exception. I have tryed multiple thing but here is my final test :
public static Result unpair(String tokenString)
{
Token token = Token.getToken(tokenString);
if(token == null) return ok(toJson(Error.AUTH_ERROR));
String epc = form(UnpairRequest.class).bindFromRequest().get().epc;
//Here the line throwing an Exception
CommitItem.find().where()
.eq("commit.user", token.user)
.eq("epc",epc)
.findUnique().delete();
return ok(toJson(Response.OK));
}
It raise this exception :
OptimisticLockException: Data has changed. updated [0] rows sql[delete from commit_item where epc=? and evat_price=? and ivat_price=? and vat=? and product_product_code=? and commit_id=?] bind[null]
It seems that the Lock is not released after the Select statement but I have no Idea why...
I have also try to delete my object this way :
PairingCommit commit = PairingCommit.find().where()
.eq("user", token.user).findUnique();
CommitItem item = CommitItem.find().where()
.eq("commit", commit)
.eq("epc", epc);
commit.items.remove(item);
commit.update();
But it doesn't delete my item from the database.
I don't really understand what is going wrong with my code.

Exception using QueryEngine inside a Transaction

I'm using the neo4j 1.9.M01 version with the java-rest-binding 1.8.M07, and I have a problem with this code that aims to get a node from a neo4j database with the property "URL" that is "ARREL", using the Query language via rest. The problems seems to happens only inside a transaction, throwing an exception, but otherwise works well :
RestGraphDatabase graphDb = new RestGraphDatabase("http://localhost:7474/db/data");
RestCypherQueryEngine queryEngine = new RestCypherQueryEngine(graphDb.getRestAPI());
Node nodearrel = null;
Transaction tx0 = gds.beginTx();
try{
final String queryStringarrel = ("START n=node(*) WHERE n.URL =~{URL} RETURN n");
QueryResult<Map<String, Object>> retornar = queryEngine.query(queryStringarrel, MapUtil.map("URL","ARREL"));
for (Map<String,Object> row : retornar)
{
nodearrel = (Node)row.get("n");
System.out.println("Arrel: "+nodearrel.getProperty("URL")+" id : "+nodearrel.getId());
}
tx0.success();
}
(...)
But an exception happens: *exception tx0: Error reading as JSON ''
* every execution at the line that returns the QueryResult object.
I also have tried to do it with the ExecutionEngine (between a transaction):
ExecutionEngine engine = new ExecutionEngine( graphDb );
String ARREL = "ARREL";
ExecutionResult result = engine.execute("START n=node(*) WHERE n.URL =~{"+ARREL+"} RETURN n");
Iterator<Node> n_column = result.columnAs("n");
Node arrelat = (Node) n_column.next();
for ( Node node : IteratorUtil.asIterable( n_column ) )
(...)
But it also fails at the *n_column.next()* returning a null object that throws an exception.
The problem is that I need to use the transactions to optimize the queries due if not it take too much time processing all the queries that I need to do. Should I try to join several operations to the query, to avoid using the transactions?

try to add single quotes at:
START n=node(*) WHERE n.URL =~ '{URL}' RETURN n

Can you update your java-rest-binding to the latest version (1.8) ? In between we had a version that automatically applied REST-batch-operations to places with transaction semantics.
So the transactions you see are not real transactions but just recording your operations to be executed as batch-rest-operations on tx.success/finish
Execute the queries within the transaction, but only access the results after the tx is finished. Then your results will be there.
This is for instance useful to send many cypher queries in one go to the server and have the results available all in one go afterwards.
And yes #ulkas use parameters but not like that:
START n=node(*) WHERE n.URL =~ {URL} RETURN n
params: { "URL" : "http://your.url" }
No quotes neccessary when using params, just like SQL prepared statements.

Out of heap space with hibernate - what's the problem?

I'm running into the following (common) error after I added a new DB table, hibernate class, and other classes to access the hibernate class:
java.lang.OutOfMemoryError: Java heap space
Here's the relevant code:
From .jsp:
<%
com.companyconnector.model.HomepageBean homepage = new com.companyconnector.model.HomepageBean();
%>
From HomepageBean:
public class HomepageBean {
...
private ReviewBean review1;
private ReviewBean review2;
private ReviewBean review3;
public HomepageBean () {
...
GetSurveyResults gsr = new GetSurveyResults();
List<ReviewBean> rbs = gsr.getRecentReviews();
review1 = rbs.get(0);
review2 = rbs.get(1);
review3 = rbs.get(2);
}
From GetSurveyResults:
public List<ReviewBean> getRecentReviews() {
List<OpenResponse> ors = DatabaseBean.getRecentReviews();
List<ReviewBean> rbs = new ArrayList<ReviewBean>();
for(int x = 0; ors.size() > x; x =+ 2) {
String employer = "";
rbs.add(new ReviewBean(ors.get(x).getUid(), employer, ors.get(x).getResponse(), ors.get(x+1).getResponse()));
}
return rbs;
}
and lastly, from DatabaseBean:
public static List<OpenResponse> getRecentReviews() {
SessionFactory session = HibernateUtil.getSessionFactory();
Session sess = session.openSession();
Transaction tx = sess.beginTransaction();
List results = sess.createQuery(
"from OpenResponse where (uid = 46) or (uid = 50) or (uid = 51)"
).list();
tx.commit();
sess.flush();
sess.close();
return results;
}
Sorry for all the code and such a long message, but I'm getting over a million instances of ReviewBean (I used jProfiler to find this). Am I doing something wrong in the for loop in GetSurveyResults? Any other problems?
I'm happy to provide more code if necessary.
Thanks for the help.
Joe

Using JProfiler to find which objects occupy the memory is a good first step. Now that you know that needlessly many instances are created, a logical next analysis step is to run your application in debug mode, and step through the code that allocates the ReviewBeans. If you do that, the bug should be obvious. (I am pretty sure I spotted it, but I'd rather teach you how to find such bugs on your own. It's a skill that is indispensable for any good programmer).

Also you probably want to close session/commit transaction if the finally block to make sure it's always invoked event if your method throws exception. Standard pattern for working with resources in java (simplified pseudo code):
Session s = null;
try {
s = openSession();
// do something useful
}
finally {
if (s != null) s.close();
}