Let's say I have a function
writeToFileOrStdout(fname: String = Nil) = { ... }
If the user passes a string value for fname, then I'd like to open a file with that name and write to it; otherwise, I'd like to print to stdout. I could always just write an if statement to take care of this, but how would I write a case statement on fname and open the correct corresponding outputStream?
val outStream = fname match {
case Nil => ???
case _ => new java.io.FileOutputStream(new java.io.File(fname))
}
outStream.write( ... )
Thanks!
Why not rewrite the function as:
def writeToFileOrStdout(fname: Option[String] = None) = {
val outStream = fname match{
case Some(name) => new java.io.FileOutputStream(new java.io.File(name))
case None => System.out
}
...
}
It's always a good idea to use Option for an optional input as opposed to using null. That's basically what it's there for. In good scala code, you will not see explicit references to null.
In fact, your code doesn't even compile for me. Nil is used to represent an empty list, not a null or non supplied String.
To augment cmbaxter's response...
Mapping a String with a possible null value to Option[String] is trivial: Option(stringValue) will return None where stringValue is null, and Some(stringValue) where non-null.
Thus, you can either:
writeToFileOrStdout(Option(stringValue)), or
If you're stuck on String (and possibly a null value) as the parameter to writeToFileOrStdout, then internally use Option(fname) and match to what it returns::
def writeToFileOrStdout(fname: String = null) = {
val outStream = Option(fname) match{
case Some(name) => new java.io.FileOutputStream(new java.io.File(name))
case None => System.out
}
...
}
To further augment cmbaxter's response, you might consider writing this:
def filenameToOutputStream(name: String) =
new java.io.FileOutputStream(new java.io.File(name))
def writeToFileOrStdout(fname: Option[String] = None) = {
val outStream = fname map filenameToOutputStream getOrElse System.out
...
}
As the post Idiomatic Scala: Your Options Do Not Match suggests, this might be more idiomatic Scala.
Related
How do I implement type safety to the values being aggregated within the agg function? My goal is to have type checking before runtime. $"event.zeroToSixty", for example won't be checked at compile time so I'd like to implement something that does.
SomeMetadataExtracted case class contains all types inside event column FYI
val currentDay = jobParameters.date.format(DateUtils.PartitionDateFormat)
val plusCar =
MPHEventWrapper
.loadMPHTable[SomeMetadataExtracted](
plusTable,
Seq(currentDay))
plusCar
.groupByKey(row =>
( row.date,
row.locale,
row.carCode))
.agg(
count(when($"event.zeroToSixty" === Within5Seconds, 1 ) ).as[Long], <= need type checking here
count(when( $"event.carLaunched" =!= Unknown, 1 ) ).as[Long], <= need type checking here
count(when($"event.successfulLaunch" === true, 1)).as[Long])) <= need type checking here
.map({
case (
(date, locale, carType),
total_quick_cars,
total_known_launches,
total_successful_launches,
) =>
carSpeedAggregate(
date,
carType,
locale,
total_quick_cars,
total_known_launches,
total_successful_launches,
)
})
Updated Code (Thanks Artem!!!) - new problem, code is super memory intensive
plusCar
.groupByKey(row => (row.date,
row.locale,
row.carCode,
))
.mapGroups {
case ((date: String, locale:String, carCode: String), events: Iterator[EventWithCommonDimensions[EventCombiner[SomeMetadataExtracted, ANStep]]]) => {
val newEvent = events.toTraversable
val zeroToSixty = newEvent.count(e => { e.event.exists(_. e.zeroToSixty.getOrElse("UNKNOWN") =!= FollowUpUnknown )})
val carLaunched = newEvent.count(e => { e.event.exists(_.carLaunched.get === Within5Seconds )})
val successfulLaunch = newEvent.count(e => {e.event.exists(_.successfulLaunch == isTrue) })
carSpeedAggregate(
date,
locale,
deviceCode,
taskName.get,
total_quick_hangups.toLong,
total_followup_calls.toLong,
total_phone_calls.toLong,
total_call_attempts.toLong
)}
}
agg is a function for untyped operations. Instead you can use, for example, combination of groupByKey and mapGroups.
# Suppose SomeMetadataExtracted has following fields
case class SomeMetadataExtracted(
date: Timestamp,
locale: String,
carCode: String,
zeroToSixty: String,
carLaunched: String,
successfulLaunch: Boolean
)
plusCar
.as[SomeMetadataExtracted] //you have to make following import to do like this: import spark_session.implicits._
.groupByKey((event: SomeMetadataExtracted) =>
( event.date,
event.locale,
event.carCode))
.mapGroups{
case ((date, locale, carCode), events: Iterator[SomeMetadataExtracted]) =>
carSpeedAggregate(
date,
locale,
carCode,
events.count(e => e.zeroToSixty == Within5Seconds),
events.count(e => e.carLaunched != Unknown),
events.count(e => e.successfulLaunch)
)}
Goal:
I am trying to parse the postman_echo collection json and persist the result into a new json copy on disk, resulting the same file as the original one.
I prefer built-in data structure from the language, but using json library should be good too. Not sure Antlr4 is a better way.
follow-up question
is it possible to allow any valid nested json in the body of post request?
update:
https://github.com/chakpongchung/postman-parser
In the end we come up with this satisfactory solution.
An alternative to what zoran mentioned is to create a case class if the structure is not too dynamic (with Play JSON). This would make it easier to compare the result.
case class MyObject(
queryString: List[KeyValue],
method: String,
url: String,
httpVersion: String
) ... and so on
object MyObject {
implicit val format: Format[MyObject] = Json.format
}
case class KeyValue(name: String, value: String)
object KeyValue {
implicit val format: Format[KeyValue] = Json.format
}
Then, you just need to do:
object JsonParser extends App {
val postman_collections = "./scala_input.json"
val jsonifiedString = scala.io.Source.fromFile(postman_collections).mkString
val myJsonData = Try(Json.parse(jsonifiedString)).map(_.as[MyObject])
myJsonData match {
case Success(myValue) => // compare your case class here
case Failure(err) => println("none")
}
}
I'm not sure if I understand your question well, but if you are trying to iterate over json string, you might try something like this:
import play.api.libs.json.{JsObject, JsValue, Json}
import scala.util.{Failure, Success, Try}
object JsonParser extends App {
val postman_coolections = "./resources/scala_input.json"
val jsonifiedString = scala.io.Source.fromFile(postman_coolections).mkString
val json: JsValue = Try(Json.parse(jsonifiedString)) match {
case Success(js) => js
case Failure(ex) => throw new Exception("Couldn't parse json", ex)
}
json.asInstanceOf[JsObject].fields.foreach{
case (key: String, value: JsValue)=>
println(s"Key:$key value:${value.toString}")
writeFile(s"$key.json", Json.prettyPrint(value))
}
//writing the whole postman input as a single file
writeFile("postmanInputFormatted.json", Json.prettyPrint(json))
writeFile("postmanInput.json", Json.stringify(json))
// To access individual property one option is to use this approach
val lookedValue = json \ "postData" \ "params" \ 1 \ "hello" \ "test"
lookedValue match {
case JsDefined(value) => println(s"Test value is $value")
case JsUndefined() => println("Didn't find test value")
}
// or
val lookedValueAlt = (json \ "postData" \ "params" \ 1 \ "hello" \ "test").getOrElse(throw SomeException)
There are multiple problems in your parser and most of them are that you are trying to use default parser to handle Json object as string. For example, in Request you are handling header as Seq[String] when it's actually Seq of (key, value) pairs. For this particular case, you should change it to something like this:
case class Request(
method: String,
header: Seq[HeaderItem], // header: []
url: Option[Url] = None,
description: String = ""
)
object Request {
implicit val format: Format[Request] = Json.using[Json.WithDefaultValues].format
case class HeaderItem(key: String, value: String)
object HeaderItem {
implicit val format: Format[HeaderItem] = Json.format
}
You can convert header to Seq[String] if you want, but you will have to write custom Read for that.
In the above case you also have cases when description is missing, so you want to handle that with default values.
You have such problems to handle in few other places as well, e.g. "response".
Another problem that I've noticed is a way how you handle "type" property from Json string. Type is reserved keyword, and you can handle it by wrapping it in ``, e.g.
case class Script(
`type`: String,
exec: Seq[String]
)
A satisfactory solution is posted in the github link above in the question.
I am new to ZeroMQ and seem to be losing messages in a loop in my begin() method.
I'm wondering if I am missing a piece where I am not queuing messages or something?
When I cause an event on my publisher, that sends two messages to my subscriber with a small gap in between, I seem not to be getting the second message that is relayed. What am I missing?
class ZMQSubscriber[T <: Transaction, B <: Block](
socket: InetSocketAddress,
hashTxListener: Option[HashDigest => Future[Unit]],
hashBlockListener: Option[HashDigest => Future[Unit]],
rawTxListener: Option[Transaction => Future[Unit]],
rawBlockListener: Option[Block => Future[Unit]]) {
private val logger = BitcoinSLogger.logger
def begin()(implicit ec: ExecutionContext) = {
val context = ZMQ.context(1)
// First, connect our subscriber socket
val subscriber = context.socket(ZMQ.SUB)
val uri = socket.getHostString + ":" + socket.getPort
//subscribe to the appropriate feed
hashTxListener.map { _ =>
subscriber.subscribe(HashTx.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to the transaction hashes from zmq")
}
rawTxListener.map { _ =>
subscriber.subscribe(RawTx.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to raw transactions from zmq")
}
hashBlockListener.map { _ =>
subscriber.subscribe(HashBlock.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to the hashblock stream from zmq")
}
rawBlockListener.map { _ =>
subscriber.subscribe(RawBlock.topic.getBytes(ZMQ.CHARSET))
logger.debug("subscribed to raw block")
}
subscriber.connect(uri)
subscriber.setRcvHWM(0)
logger.info("Connection to zmq client successful")
while (true) {
val notificationTypeStr = subscriber.recvStr(ZMQ.DONTWAIT)
val body = subscriber.recv(ZMQ.DONTWAIT)
Future(processMsg(notificationTypeStr, body))
}
}
private def processMsg(topic: String, body: Seq[Byte])(implicit ec: ExecutionContext): Future[Unit] = Future {
val notification = ZMQNotification.fromString(topic)
val res: Option[Future[Unit]] = notification.flatMap {
case HashTx =>
hashTxListener.map { f =>
val hash = Future(DoubleSha256Digest.fromBytes(body))
hash.flatMap(f(_))
}
case RawTx =>
rawTxListener.map { f =>
val tx = Future(Transaction.fromBytes(body))
tx.flatMap(f(_))
}
case HashBlock =>
hashBlockListener.map { f =>
val hash = Future(DoubleSha256Digest.fromBytes(body))
hash.flatMap(f(_))
}
case RawBlock =>
rawBlockListener.map { f =>
val block = Future(Block.fromBytes(body))
block.flatMap(f(_))
}
}
}
}
So this seems to have been solved by using a ZMsg.recvMsg() in the while-loop instead of
val notificationTypeStr = subscriber.recvStr(ZMQ.DONTWAIT)
val body = subscriber.recv(ZMQ.DONTWAIT)
I'm not sure why this works, but it does. So here is what my begin method looks like now
while (run) {
val zmsg = ZMsg.recvMsg(subscriber)
val notificationTypeStr = zmsg.pop().getString(ZMQ.CHARSET)
val body = zmsg.pop().getData
Future(processMsg(notificationTypeStr, body))
}
Future.successful(Unit)
}
What am I missing?
How the blocking v/s non-blocking modus operandi work :
The trick is in the (non-)blocking mode of the respective call to the .recv() method.
A second call to the subscriber.recv( ZMQ.DONTWAIT )-method thus returns immediately, so your second part, ( the body ) may and will legally contain nothing, even though your promise stated a pair of messages was indeed dispached from the publisher-side ( a pair of .send() method calls - one may also object, there are chances the sender was actually sending just one message, in a multi-part fashion - MCVE-code is not specific on this part ).
So, once you have moved your code from non-blocking mode ( in the O/P ) into a principally blocking-mode ( which locked / sync-ed the further flow of the code with the external event of an arrival of any plausibly formatted message, not returning earlier ), in:
val zmsg = ZMsg.recvMsg(subscriber) // which BLOCKS-till-a-1st-zmsg-arrived
both the further processed .pop()-ed parts just unload the components ( ref. the remark on actual ZMsg multi-part structure actually sent by the published-side, presented above )
Safety next :unlimited alloc-s v/s a mandatory blocking / dropping messages ?
the code surprised me on several points. Besides a rather very "late" call to the .connect()-method, compared to all the previous socket-archetype detailed settings ( that normally get arranged "after" a request to setup a connection ). While this may work fine, as intended, yet it exposes even tighter ( smaller ) time-window for the .Context()-instance to setup and (re-)negotiate all the relevant connection-details so as to become RTO.
One particular line attracted my attention: subscriber.setRcvHWM( 0 ) this is a version-archetype dependent trick. Yet, the value of zero causes an application to become vulnerable and I would not advise doing so in any production-grade application.
I want to prohibit resolving of a.b. I want to substitute param from another config. Like this:
val d = ConfigFactory.load(ConfigFactory.parseString(
"""
|param = x
|a.b = ${param}
""".stripMargin))
val a = ConfigFactory.parseString("param = 1")
val result = a.withFallback(d).resolve()
In this case param gets value 1, but a.b remains x
I've tried to set ConfigResolveOptions.defaults().setAllowUnresolved(true) when loading config d, but that doesn't work.
How can I overcome this?
The problem is that Config.load is resolving the substitution immediately. If you take that out it resolves like you want it to:
val p = ConfigFactory.parseString(
"""
|param = x
|a.b = ${param}
""".stripMargin)
val a = ConfigFactory.parseString("param = 1")
val result = a.withFallback(p).resolve()
println(result.getString("a.b"))
This prints 1.
You don't need to use Config.load unless you want to use reference.conf, etc. If you do want to use Config.load then you should do it after you have composed all the configs together using withFallback.
For example, this also prints 1:
val p = ConfigFactory.parseString(
"""
|param = x
|a.b = ${param}
""".stripMargin)
val d = ConfigFactory.load(p)
val a = ConfigFactory.parseString("param = 1")
val result = a.withFallback(p)
val loaded = ConfigFactory.load(result)
println(loaded.getString("a.b"))
Or, say you have an application.conf with include that you want to use with ConfigFactory.load() (per your comment).
If application.conf looks like
include "foo"
and foo.conf looks like
a.b = ${param}
then this prints 1 also:
val a = ConfigFactory.parseString("param = 1")
val app = ConfigFactory.load("application", ConfigParseOptions.defaults,
ConfigResolveOptions.defaults.setAllowUnresolved(true))
val result = a.withFallback(app).resolve
println(result.getString("a.b"))
In general, if you want A to override B to override C then you should use A.withFallback(B).withFallback(C).
I struggled a bit with the same thing: trying to use "fallbacks" to override values, when it was designed for layering/merging configs
Assuming I understand your use case, I recommend instead using file includes.
In my application.conf I have the default value
a.b = "placeholder"
And at the bottom I have the following include
# Local overrides - for development use
include "local.conf"
And finally in local.conf
param = 1
a.b = ${param}
The end result is that a.b will be overridden with 1
Found a workaround for my problem:
So If I have config file application.conf which uses include to include config files which contain substitution syntax and files which contain declaration of the config values which are going to be substituted.
val a = ConfigFactory.parseString(s"""param = 1""")
val z = ConfigFactory.parseResources("application.conf") //this doesn't resolve substitutions
val result = a.withFallback(z).resolve().withFallback(ConfigFactory.load("application.conf"))
Can't seem to create a functional way to insert a user from Java for Devise. Currently there are these fields:
"_id",
"access_level",
"confirmation_sent_at",
"confirmation_token",
"confirmed_at",
"email",
"encrypted_password",
"sign_in_count"
I am able to insert a document that counts as a user. The problem is that when I go to:
http://www.mysite.com:3000/users/confirmation?confirmation_token=TOKENHERE
I get a message saying that it's invalid.
EDIT 1:
When I resend confirmation instructions for this user (WHICH GENERATES A NEW TOKEN), the user can be logged into. This confirms my doubts about the token being the problem. How can I port Devise's token generator to Java?
EDIT 2:
When I register on site, it says I should check for a confirmation link. However, if I go into the Mongo shell, manually take out the confirmation token and paste it to site.com/users/confirmation?confirmation_token= then it doesn't work! However, if I actually use the confirmation link I was sent, it works. How can I make a VALID token, all from Java. Please help!
For this quoestion you should refer to this stackoverflow answer and to the Rails API of protect_from_forgery.
The short answer is to disable forgery protection in your controller, but this makes your application vulnerable to CSRF attacks:
skip_before_action :verify_authenticity_token
The better way would be to authenticate with a JSON or XML request as these requests are not protected by CSRF protection. You can find a solution for devise here.
Edit
Monkey patch devise to save unencoded confirmation token. In your config/initializers/devise.rb
module Devise
module Models
module Confirmable
def generate_confirmation_token
raw, enc = Devise.token_generator.generate(self.class, :confirmation_token)
#raw_confirmation_token = raw
self.my_unencoded_column = raw # Patch
self.confirmation_token = enc
self.confirmation_sent_at = Time.now.utc
end
end
end
end
In case anyone else finds themselves trying to get a java or scala app to coexist with a rails app, I hacked up the following. Its in scala but uses java apis so should be easy to read. As far as I can tell it replicates Devise's behavior, and if I hit the confirmation link in the rails app with the raw token rails/devise generates the same encoded string.
import java.security.spec.KeySpec
import javax.crypto.SecretKey
import javax.crypto.SecretKeyFactory
import javax.crypto.spec.PBEKeySpec
import javax.crypto.spec.SecretKeySpec
import javax.crypto.Mac
import javax.xml.bind.DatatypeConverter
import java.util.Base64
// copy functionality from Rails Devise
object TokenGenerator {
// sample values 9exithzwZ8P9meqdVs3K => 54364224169895883e87c8412be5874039b470e26e762cb3ddc37c0bdcf014f5
// 5zNMi6egbyPoDUy2t3NY => 75bd5d53aa36d3fc61ac186b4c6e2be8353e6b39536d3cf846719284e05474ca
private val deviseSecret = sys.env("DEVISE_SECRET")
private val factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1")
val encoder = Base64.getUrlEncoder()
case class TokenInfo(raw: String, encoded: String)
def createConfirmationToken: TokenInfo = {
// copy behavior from rails world. Don't know why it does this
val replacements = Map('l' -> "s", 'I' -> "x", 'O' -> "y", '0' -> "z")
// make a raw key of 20 chars, doesn't seem to matter what they are, just need url valid set
val bytes = new Array[Byte](16)
scala.util.Random.nextBytes(bytes)
val raw = encoder.encodeToString(bytes).take(20).foldLeft(""){(acc, x) => acc ++ replacements.get(x).getOrElse(x.toString)}
TokenInfo(raw, digestForConfirmationToken(raw))
}
private def generateKey(salt: String): Array[Byte] = {
val iter = 65536
val keySize = 512
val spec = new PBEKeySpec(deviseSecret.toCharArray, salt.getBytes("UTF-8"), iter, keySize)
val sk = factory.generateSecret(spec)
val skspec = new SecretKeySpec(sk.getEncoded, "AES")
skspec.getEncoded
}
def sha256HexDigest(s: String, key: Array[Byte]): String = {
val mac = Mac.getInstance("HmacSHA256")
val keySpec = new SecretKeySpec(key, "RAW")
mac.init(keySpec)
val result: Array[Byte] = mac.doFinal(s.getBytes())
DatatypeConverter.printHexBinary(result).toLowerCase
}
private def getDigest(raw: String, salt: String) = sha256HexDigest(raw, generateKey(salt))
// devise uses salt "Devise #{column}", in this case its confirmation_token
def digestForConfirmationToken(raw: String) = getDigest(raw, "Devise confirmation_token")
}