Develop Reference

Getting HystrixRuntimeException: Function timed-out and fallback failed

I am using hystrix 1.5.3 and running this piece of code in my local machine.
#HystrixCommand (groupKey = "BookService", commandKey = "BookService.BookDetail", commandProperties = {
#HystrixProperty (name = EXECUTION_ISOLATION_STRATEGY, value = "THREAD"),
#HystrixProperty (name = CIRCUIT_BREAKER_ENABLED, value = "true"),
#HystrixProperty (name = EXECUTION_TIMEOUT_ENABLED, value = "true"),
#HystrixProperty (name = EXECUTION_ISOLATION_THREAD_TIMEOUT_IN_MILLISECONDS, value = "1500")}, threadPoolProperties = {
#HystrixProperty (name = CORE_SIZE, value = "60"),
#HystrixProperty (name = MAX_QUEUE_SIZE, value = "60"),
#HystrixProperty (name = QUEUE_SIZE_REJECTION_THRESHOLD, value = "60"),
#HystrixProperty (name = KEEP_ALIVE_TIME_MINUTES, value = "1")
})
public String getBookDetail(String bookId)
{
log.info("Getting details");
...
}
On our servers it works fine however I get this runtime exception on my local machine. My local server just waits for the timeout duration and in the end throws this HystrixRuntimeException. Also, I don't have any fallback defined but it should not be needed for my case. The expectation is it should work normally like in our production server.
com.netflix.hystrix.exception.HystrixRuntimeException: BookService.BookDetail timed-out and fallback failed.
at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:793) ~[hystrix-core-1.5.3.jar:1.5.3]
at com.netflix.hystrix.AbstractCommand$21.call(AbstractCommand.java:768) ~[hystrix-core-1.5.3.jar:1.5.3]
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77) ~[rxjava-1.0.12.jar:1.0.12]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70) ~[rxjava-1.0.12.jar:1.0.12]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70) ~[rxjava-1.0.12.jar:1.0.12]
at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1448) ~[hystrix-core-1.5.3.jar:1.5.3]
at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1373) ~[hystrix-core-1.5.3.jar:1.5.3]
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70) ~[rxjava-1.0.12.jar:1.0.12]
I checked the function is not being executed as the log is not getting printed.
When I change the timeout by modifying EXECUTION_ISOLATION_THREAD_TIMEOUT_IN_MILLISECONDS similar behaviour happens just after the new timeout duration.
When I remove the #HystrixCommand annotation it just works fine so it means there is some issue with Hystrix only. The hystrix properties seem is the one which is defined in the annotation which seem fine. Could this be because hystrix is not configured properly? Any help would be appreciated.

Your function is not being executed in 1500ms.
Change EXECUTION_TIMEOUT_ENABLED to false, hystrix will not timeout your method.

ZMQ missing events being propagated in jeromq scala

I am new to ZeroMQ and seem to be losing messages in a loop in my begin() method.
I'm wondering if I am missing a piece where I am not queuing messages or something?
When I cause an event on my publisher, that sends two messages to my subscriber with a small gap in between, I seem not to be getting the second message that is relayed. What am I missing?
class ZMQSubscriber[T <: Transaction, B <: Block](
socket: InetSocketAddress,
hashTxListener: Option[HashDigest => Future[Unit]],
hashBlockListener: Option[HashDigest => Future[Unit]],
rawTxListener: Option[Transaction => Future[Unit]],
rawBlockListener: Option[Block => Future[Unit]]) {
private val logger = BitcoinSLogger.logger
def begin()(implicit ec: ExecutionContext) = {
val context = ZMQ.context(1)
// First, connect our subscriber socket
val subscriber = context.socket(ZMQ.SUB)
val uri = socket.getHostString + ":" + socket.getPort
//subscribe to the appropriate feed
hashTxListener.map { _ =>
subscriber.subscribe(HashTx.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to the transaction hashes from zmq")
}
rawTxListener.map { _ =>
subscriber.subscribe(RawTx.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to raw transactions from zmq")
}
hashBlockListener.map { _ =>
subscriber.subscribe(HashBlock.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to the hashblock stream from zmq")
}
rawBlockListener.map { _ =>
subscriber.subscribe(RawBlock.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to raw block")
}
subscriber.connect(uri)
subscriber.setRcvHWM(0)
logger.info("Connection to zmq client successful")
while (true) {
val notificationTypeStr = subscriber.recvStr(ZMQ.DONTWAIT)
val body = subscriber.recv(ZMQ.DONTWAIT)
Future(processMsg(notificationTypeStr, body))
}
}
private def processMsg(topic: String, body: Seq[Byte])(implicit ec: ExecutionContext): Future[Unit] = Future {
val notification = ZMQNotification.fromString(topic)
val res: Option[Future[Unit]] = notification.flatMap {
case HashTx =>
hashTxListener.map { f =>
val hash = Future(DoubleSha256Digest.fromBytes(body))
hash.flatMap(f(_))
}
case RawTx =>
rawTxListener.map { f =>
val tx = Future(Transaction.fromBytes(body))
tx.flatMap(f(_))
}
case HashBlock =>
hashBlockListener.map { f =>
val hash = Future(DoubleSha256Digest.fromBytes(body))
hash.flatMap(f(_))
}
case RawBlock =>
rawBlockListener.map { f =>
val block = Future(Block.fromBytes(body))
block.flatMap(f(_))
}
}
}
}

So this seems to have been solved by using a ZMsg.recvMsg() in the while-loop instead of
val notificationTypeStr = subscriber.recvStr(ZMQ.DONTWAIT)
val body = subscriber.recv(ZMQ.DONTWAIT)
I'm not sure why this works, but it does. So here is what my begin method looks like now
while (run) {
val zmsg = ZMsg.recvMsg(subscriber)
val notificationTypeStr = zmsg.pop().getString(ZMQ.CHARSET)
val body = zmsg.pop().getData
Future(processMsg(notificationTypeStr, body))
}
Future.successful(Unit)
}

What am I missing?
How the blocking v/s non-blocking modus operandi work :
The trick is in the (non-)blocking mode of the respective call to the .recv() method.
A second call to the subscriber.recv( ZMQ.DONTWAIT )-method thus returns immediately, so your second part, ( the body ) may and will legally contain nothing, even though your promise stated a pair of messages was indeed dispached from the publisher-side ( a pair of .send() method calls - one may also object, there are chances the sender was actually sending just one message, in a multi-part fashion - MCVE-code is not specific on this part ).
So, once you have moved your code from non-blocking mode ( in the O/P ) into a principally blocking-mode ( which locked / sync-ed the further flow of the code with the external event of an arrival of any plausibly formatted message, not returning earlier ), in:
val zmsg = ZMsg.recvMsg(subscriber) // which BLOCKS-till-a-1st-zmsg-arrived
both the further processed .pop()-ed parts just unload the components ( ref. the remark on actual ZMsg multi-part structure actually sent by the published-side, presented above )
Safety next :unlimited alloc-s v/s a mandatory blocking / dropping messages ?
the code surprised me on several points. Besides a rather very "late" call to the .connect()-method, compared to all the previous socket-archetype detailed settings ( that normally get arranged "after" a request to setup a connection ). While this may work fine, as intended, yet it exposes even tighter ( smaller ) time-window for the .Context()-instance to setup and (re-)negotiate all the relevant connection-details so as to become RTO.
One particular line attracted my attention: subscriber.setRcvHWM( 0 ) this is a version-archetype dependent trick. Yet, the value of zero causes an application to become vulnerable and I would not advise doing so in any production-grade application.

Running a Java-based Spark Job on spark-jobserver

I need to run an aggregation Spark job using spark-jobserver using low-latency contexts. I have this Scala runner to run a job on using a Java method from a Java class.
object AggregationRunner extends SparkJob {
def main(args: Array[String]) {
val ctx = new SparkContext("local[4]", "spark-jobs")
val config = ConfigFactory.parseString("")
val results = runJob(ctx, config)
}
override def validate(sc: SparkContext, config: Config): SparkJobValidation = {
SparkJobValid;
}
override def runJob(sc: SparkContext, config: Config): Any = {
val context = new JavaSparkContext(sc)
val aggJob = new ServerAggregationJob()
val id = config.getString("input.string").split(" ")(0)
val field = config.getString("input.string").split(" ")(1)
return aggJob.aggregate(context, id, field)
}
}
However, I get the following error. I tried taking out the content returned in the Java method and am now just returning a test string, but it still doesn't work:
{
"status": "ERROR",
"result": {
"message": "Ask timed out on [Actor[akka://JobServer/user/context-supervisor/single-context#1243999360]] after [10000 ms]",
"errorClass": "akka.pattern.AskTimeoutException",
"stack": ["akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)", "akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)", "scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)", "scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)", "akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)", "akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)", "akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)", "akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)", "java.lang.Thread.run(Thread.java:745)"]
}
}
I am not too sure why there is a timeout since I am only returning a string.
EDIT
So I figured out that the issue was occurring because I was using a Spark context that was created before updating a JAR. However, now that I try to use JavaSparkContext inside the Spark job, it returns to the error shown above.
What would be a permanent way to get rid of the error.
Also, would the fact that I am running a heavy Spark job on a local docker container be a plausible reason for the timeout.

For resolving ask time out issue, please add/change below properties in jobserver configuration file.
spray.can.server {
idle-timeout = 210 s
request-timeout = 200 s
}
for more information take a look at this https://github.com/spark-jobserver/spark-jobserver/blob/d1843cbca8e0d07f238cc664709e73bbeea05f2c/doc/troubleshooting.md

Now do you create transaction in Neo4j 2.0?

Now do you create transaction in Neo4j 2.0 ? I tried dozens of ways and none of them worked.
Basically problem is that second and subsequent transactions are never successful. Perhaps I don't begin transactions properly. I don't know. I tried all possible combinations that I see in your unit-tests and also in ExecutionEngine.
Here is how I create transaction:
private def withTransaction[T](f: => T): T = {
// FIXME: Sometimes it returns PlaceboTransaction which causes TONS of issues
val tx = db.beginTx
try {
val result = f
tx.success()
result
} catch {
case e: Throwable =>
// If I don't check this I'll get NullPointerException in TopLevelTransaction.markAsRollbackOnly()
if (!tx.isInstanceOf[PlaceboTransaction])
tx.failure()
throw e
} finally {
// If I don't check this I'll get NullPointerException in TopLevelTransaction.markAsRollbackOnly()
if (!tx.isInstanceOf[PlaceboTransaction])
tx.close()
}
}
It never works. Attempts to fetch any data/properties of Node cause following exception
Exception in thread "main" org.neo4j.graphdb.NotInTransactionException
at org.neo4j.kernel.ThreadToStatementContextBridge.transaction(ThreadToStatementContextBridge.java:58)
at org.neo4j.kernel.ThreadToStatementContextBridge.statement(ThreadToStatementContextBridge.java:49)
at org.neo4j.kernel.impl.core.NodeProxy.hasLabel(NodeProxy.java:551)
at GraphDBManager$$anonfun$findUsers$1$$anonfun$apply$1.apply(GraphDBManager.scala:72)
at GraphDBManager$$anonfun$findUsers$1$$anonfun$apply$1.apply(GraphDBManager.scala:72)
at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
at GraphDBManager$$anonfun$findUsers$1.apply(GraphDBManager.scala:72)
at GraphDBManager$$anonfun$findUsers$1.apply(GraphDBManager.scala:72)
at GraphDBManager$.withTransaction(GraphDBManager.scala:38)
at GraphDBManager$.findUsers(GraphDBManager.scala:71)
at Test$.main(Test.scala:12)
at Test.main(Test.scala)
I created sample project here.
Any help is greatly appreciated. Thanks.

This was a client code bug, Pull Request to the project in question here: https://github.com/cppexpert/neo4j_2_bad_transactions/pull/1

Well.. After hours of debugging I figured it out. I hope it's going to be fixed in final release.
Here is how problem function looks like
def findUsers: List[ObjectId] = {
val query = engine.execute(s"MATCH (n:$label) RETURN n")
val it = query.columnAs[Node]("n")
withTransaction {
val lst = it.toList
val ret = for (node <- lst; if node.hasLabel(label)) yield new ObjectId(node.getProperty("id").asInstanceOf[String])
ret
}
}
Turned out ExecutionEngine.execute leaves transaction open that causes beginTx() in withTransaction return PlaceboTransaction instead of real transaction object. On the other hand I can't get rid of my transaction wrapper because NodeProxy surprisingly gets transaction object differently
Exception in thread "main" org.neo4j.graphdb.NotInTransactionException
at org.neo4j.kernel.ThreadToStatementContextBridge.transaction(ThreadToStatementContextBridge.java:58)
at org.neo4j.kernel.ThreadToStatementContextBridge.statement(ThreadToStatementContextBridge.java:49)
at org.neo4j.kernel.impl.core.NodeProxy.hasLabel(NodeProxy.java:551)
where it comes from
private KernelTransaction transaction()
{
checkIfShutdown();
KernelTransaction transaction = txManager.getKernelTransaction();
if ( transaction == null )
{
throw new NotInTransactionException();
}
return transaction;
}
What's the difference between transaction from getKernelTransaction and object from TLS map I don't know.
Therefore fixed version of my function would be
def findUsers: List[ObjectId] = {
val query = engine.execute(s"MATCH (n:$label) RETURN n")
val it = query.columnAs[Node]("n")
val lst = it.toList
query.close()
withTransaction {
val ret = for (node <- lst; if node.hasLabel(label)) yield new ObjectId(node.getProperty("id").asInstanceOf[String])
ret
}
}
Which in my opinion not only ugly from design prospective but also give inconsistent data when I iterate through nodes in second transaction.

Test WebSocket in PlayFramework

I have a WebSocket in my Play application and I want to write a test for it, but I couldn't find any example on how to write such a test. I found a discussion in the play-framework Google group but there has been no activity recently.
So, are there any ideas on how to test WebSocket's in a Java test?

You can retrieve underlying Iteratee,Enumerator and test them directly. This way you don't need to use a browser. You need akka-testkit though, to cope with asynchronous nature of iteratees.
A Scala example:
object WebSocket extends Controller {
def websocket = WebSocket.async[JsValue] { request =>
Future.successful(Iteratee.ignore[JsValue] -> Enumerator.apply[JsValue](Json.obj("type" -> "error")))
}
}
class WebSocketSpec extends PlaySpecification {
"WebSocket" should {
"respond with error packet" in new WithApplication {
val request = FakeRequest()
var message: JsValue = null
val iteratee = Iteratee.foreach[JsValue](chunk => message = chunk)(Akka.system.dispatcher)
Controller.websocket().f(request)(Enumerator.empty[JsValue],iteratee)
TestKit.awaitCond(message == Json.obj("type" -> "error"), 1 second)
}
}
}

I test WebSockets code using Firefox:
https://github.com/schleichardt/stackoverflow-answers/commit/13d5876791ef409e092e4a097f54247d851e17dc#L8R14
For Java it works similar replacing 'HTMLUNIT' with 'FIREFOX': http://www.playframework.com/documentation/2.1.x/JavaFunctionalTest

Chrome provides a plugin to test websocket service.
Edit
So using the plugin (as shown in picture below) you can provide websocket url and the request data and send message to service. And message log shows the message sent from client and also service response.

Assume that you have a websocket library that returns the Future[Itearatee[JsValue, Unit], Enumerator[JsValue]] your controller uses
trait WSLib {
def connect: Future[Itearatee[JsValue, Unit], Enumerator[JsValue]]
}
And you wanna test this library.
Here is a context you can use:
trait WebSocketContext extends WithApplication {
val aSecond = FiniteDuration(1, TimeUnit.SECONDS)
case class Incoming(iteratee: Iteratee[JsValue, Unit]) {
def feed(message: JsValue) = {
iteratee.feed(Input.El(message))
}
def end(wait: Long = 100) = {
Thread.sleep(wait) //wait until all previous fed messages are handled
iteratee.feed(Input.EOF)
}
}
case class OutGoing(enum: Enumerator[JsValue]) {
val messages = enum(Iteratee.fold(List[JsValue]()) {
(l, jsValue) => jsValue :: l
}).flatMap(_.run)
def get: List[JsValue] = {
Await.result(messages, aSecond)
}
}
def wrapConnection(connection: => Future[Iteratee[JsValue, Unit], Enumerator[JsValue]]): (Incoming, OutGoing) = {
val (iteratee, enumerator) = Await.result(conn, aSecond)
(Incoming(iteratee), OutGoing(enumerator))
}
}
Then your tests can be written as
"return all subscribers when asked for info" in new WebSocketContext {
val (incoming, outgoing) = wrapConnection(myWSLib.connect)
incoming.feed(JsObject("message" => "hello"))
incoming.end() //this closes the connection
val responseMessages = outgoing.get //you only call this "get" after the connection is closed
responseMessages.size must equalTo(1)
responseMessages must contain(JsObject("reply" => "Hey"))
}
Incoming represent the messages coming from the client side, while the outgoing represents the messages sent from the server. To write test, you first feed in the incoming messages from incoming and then close the connection by calling incoming.end, then you get the complete list of outgoing messages from the outgoing.get method.