Develop Reference

Karate DSL: Convert String to Json and insert value? [duplicate]

I Have a file json named production_2.json
[
{
"v":{
"id":"rep_01564526",
"name":"tuttoverde.pgmx",
"type":"PRODUCTION_STARTED",
"ute":"CDL",
"ver":"1.0",
"status":"EXE"
},
"ts":"2020-11-19T08:00:00Z"
},
{
"v":{
"id":"rep_01564526",
"name":"tuttoverde.pgmx",
"type":"PRODUCTION_ENDED",
"ute":"CDL",
"ver":"1.0",
"status":"EXE"
},
"ts":"2020-11-19T17:00:00Z"
}
]
And have the folling Karate code, that:
Read the file production_2.json
and for each element of the array send a topic
I * def sendtopics =
"""
function(i){
var topic = "data." + machineID + ".Production";
var payload = productions[i];
karate.log('topic: ', topic )
karate.log('payload: ', payload )
return mqtt.sendMessage(payload, topic);
}
"""
* def productions = read('this:productions_json/production_2.json')
* def totalProductionEvents = productions.length
* def isTopicWasSent = karate.repeat(totalProductionEvents, sendtopics)
* match each isTopicWasSent == true
The function
mqtt.sendMessage(payload, topic);
is a function in java, that have the following segnature
public Boolean sendMessage(String payload, String topic) {
System.out.println("Publishing message: ");
System.out.println("payload " + payload);
System.out.println("topic " + topic);
return true;
}
the problem is that the value of the "payload" inside the javascript function is correct and is printed correctly, while inside the "sendMessage" function the value of the payload is formatted incorrectly.
For example here is what it prints inside karate.log('payload: ', payload )
payload: {
"v": {
"id": "rep_01564526",
"name": "tuttoverde.pgmx",
"type": "PRODUCTION_STARTED",
"ute": "CDL",
"ver": "1.0",
"status": "EXE"
},
"ts": "2021-01-08T08:00:00Z"
}
And Here instead what is printed on the function "sendMessage" of the java class
payload {v={id=rep_01564526, name=tuttoverde.pgmx, type=PRODUCTION_STARTED, ute=CDL, ver=1.0, status=EXE, ts=2021-01-08T08:00:00Z}
I don't understand why the payload is formatted incorrectly (= instead of : ) and is it not a string. I also tried using the following solution and it doesn't work for me
* def sendtopics =
"""
function(i){
var topic = "data." + machineID + ".Production";
var payload = productions[i];
var payload2 = JSON.stringify(payload);
return mqtt.sendMessage(payload2, topic);
}
"""
How do I convert an object inside javascript to a string so I can pass it to java?

You are doing some really advanced stuff in Karate. I strongly suggest you start looking at the new version (close to release) and you can find details here: https://github.com/intuit/karate/wiki/1.0-upgrade-guide
The reason is because the async and Java interop has some breaking changes - and there are some new API-s you can call on the karate object in JS to do format conversions:
var temp = karate.fromString(payload);
And karate.log() should work better and not give you that odd formatting you are complaining about. With the current version you can try karate.toJson() if that gives you the conversion you expect.
Given your advanced usage, I recommend you start using the new version and provide feedback on anything that may be still needed.

how to parse postman-collection?

Goal:
I am trying to parse the postman_echo collection json and persist the result into a new json copy on disk, resulting the same file as the original one.
I prefer built-in data structure from the language, but using json library should be good too. Not sure Antlr4 is a better way.
follow-up question
is it possible to allow any valid nested json in the body of post request?
update:
https://github.com/chakpongchung/postman-parser
In the end we come up with this satisfactory solution.

An alternative to what zoran mentioned is to create a case class if the structure is not too dynamic (with Play JSON). This would make it easier to compare the result.
case class MyObject(
queryString: List[KeyValue],
method: String,
url: String,
httpVersion: String
) ... and so on
object MyObject {
implicit val format: Format[MyObject] = Json.format
}
case class KeyValue(name: String, value: String)
object KeyValue {
implicit val format: Format[KeyValue] = Json.format
}
Then, you just need to do:
object JsonParser extends App {
val postman_collections = "./scala_input.json"
val jsonifiedString = scala.io.Source.fromFile(postman_collections).mkString
val myJsonData = Try(Json.parse(jsonifiedString)).map(_.as[MyObject])
myJsonData match {
case Success(myValue) => // compare your case class here
case Failure(err) => println("none")
}
}

I'm not sure if I understand your question well, but if you are trying to iterate over json string, you might try something like this:
import play.api.libs.json.{JsObject, JsValue, Json}
import scala.util.{Failure, Success, Try}
object JsonParser extends App {
val postman_coolections = "./resources/scala_input.json"
val jsonifiedString = scala.io.Source.fromFile(postman_coolections).mkString
val json: JsValue = Try(Json.parse(jsonifiedString)) match {
case Success(js) => js
case Failure(ex) => throw new Exception("Couldn't parse json", ex)
}
json.asInstanceOf[JsObject].fields.foreach{
case (key: String, value: JsValue)=>
println(s"Key:$key value:${value.toString}")
writeFile(s"$key.json", Json.prettyPrint(value))
}
//writing the whole postman input as a single file
writeFile("postmanInputFormatted.json", Json.prettyPrint(json))
writeFile("postmanInput.json", Json.stringify(json))
// To access individual property one option is to use this approach
val lookedValue = json \ "postData" \ "params" \ 1 \ "hello" \ "test"
lookedValue match {
case JsDefined(value) => println(s"Test value is $value")
case JsUndefined() => println("Didn't find test value")
}
// or
val lookedValueAlt = (json \ "postData" \ "params" \ 1 \ "hello" \ "test").getOrElse(throw SomeException)

There are multiple problems in your parser and most of them are that you are trying to use default parser to handle Json object as string. For example, in Request you are handling header as Seq[String] when it's actually Seq of (key, value) pairs. For this particular case, you should change it to something like this:
case class Request(
method: String,
header: Seq[HeaderItem], // header: []
url: Option[Url] = None,
description: String = ""
)
object Request {
implicit val format: Format[Request] = Json.using[Json.WithDefaultValues].format
case class HeaderItem(key: String, value: String)
object HeaderItem {
implicit val format: Format[HeaderItem] = Json.format
}
You can convert header to Seq[String] if you want, but you will have to write custom Read for that.
In the above case you also have cases when description is missing, so you want to handle that with default values.
You have such problems to handle in few other places as well, e.g. "response".
Another problem that I've noticed is a way how you handle "type" property from Json string. Type is reserved keyword, and you can handle it by wrapping it in ``, e.g.
case class Script(
`type`: String,
exec: Seq[String]
)

A satisfactory solution is posted in the github link above in the question.

Retain keys with null values while writing JSON in spark

I am trying to write a JSON file using spark. There are some keys that have null as value. These show up just fine in the DataSet, but when I write the file, the keys get dropped. How do I ensure they are retained?
code to write the file:
ddp.coalesce(20).write().mode("overwrite").json("hdfs://localhost:9000/user/dedupe_employee");
part of JSON data from source:
"event_header": {
"accept_language": null,
"app_id": "App_ID",
"app_name": null,
"client_ip_address": "IP",
"event_id": "ID",
"event_timestamp": null,
"offering_id": "Offering",
"server_ip_address": "IP",
"server_timestamp": 1492565987565,
"topic_name": "Topic",
"version": "1.0"
}
Output:
"event_header": {
"app_id": "App_ID",
"client_ip_address": "IP",
"event_id": "ID",
"offering_id": "Offering",
"server_ip_address": "IP",
"server_timestamp": 1492565987565,
"topic_name": "Topic",
"version": "1.0"
}
In the above example keys accept_language, app_name and event_timestamp have been dropped.

Apparently, spark does not provide any option to handle nulls. So following custom solution should work.
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
import com.fasterxml.jackson.databind.ObjectMapper
case class EventHeader(accept_language:String,app_id:String,app_name:String,client_ip_address:String,event_id: String,event_timestamp:String,offering_id:String,server_ip_address:String,server_timestamp:Long,topic_name:String,version:String)
val ds = Seq(EventHeader(null,"App_ID",null,"IP","ID",null,"Offering","IP",1492565987565L,"Topic","1.0")).toDS()
val ds1 = ds.mapPartitions(records => {
val mapper = new ObjectMapper with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
records.map(mapper.writeValueAsString(_))
})
ds1.coalesce(1).write.text("hdfs://localhost:9000/user/dedupe_employee")
This will produce output as :
{"accept_language":null,"app_id":"App_ID","app_name":null,"client_ip_address":"IP","event_id":"ID","event_timestamp":null,"offering_id":"Offering","server_ip_address":"IP","server_timestamp":1492565987565,"topic_name":"Topic","version":"1.0"}

If you are on Spark 3, you can add
spark.sql.jsonGenerator.ignoreNullFields false

ignoreNullFields is an option to set when you want DataFrame converted to json file since Spark 3.
If you need Spark 2 (specifically PySpark 2.4.6), you can try converting DataFrame to rdd with Python dict format. And then call pyspark.rdd.saveTextFile to output json file to hdfs. The following example may help.
cols = ddp.columns
ddp_ = ddp.rdd
ddp_ = ddp_.map(lambda row: dict([(c, row[c]) for c in cols])
ddp_ = ddp.repartition(1).saveAsTextFile(your_hdfs_file_path)
This should produce output file like,
{"accept_language": None, "app_id":"123", ...}
{"accept_language": None, "app_id":"456", ...}
What's more, if you want to replace Python None with JSON null, you will need to dump every dict into json.
ddp_ = ddp_.map(lambda row: json.dumps(row, ensure.ascii=False))

Since Spark 3, and if you are using the class DataFrameWriter
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html#json-java.lang.String-
(same applies for pyspark)
https://spark.apache.org/docs/3.0.0-preview/api/python/_modules/pyspark/sql/readwriter.html
its json method has an option ignoreNullFields=None
where None means True.
So just set this option to false.
ddp.coalesce(20).write().mode("overwrite").option("ignoreNullFields", "false").json("hdfs://localhost:9000/user/dedupe_employee")

To retain null values converting to JSON please set this config option.
spark = (
SparkSession.builder.master("local[1]")
.config("spark.sql.jsonGenerator.ignoreNullFields", "false")
).getOrCreate()

How to extract data from a JSON array to variables and pass variables to a PHP function

I have a Java Program that sends a HTTP POST request to a PHP file. I need the PHP script to extract the JSON data to some variables and call a PHP function with those variables (parameters). Please find the PHP code below.
<?php
if ($_SERVER['REQUEST_METHOD'] == 'POST')
{
$data = json_decode(file_get_contents("php://input"), true);
var_export($data);
}
else
{
var_export($_SERVER['REQUEST_METHOD']);
}
?>
The JSON Object created in Java
JSONObject json = new JSONObject();
json.put("name", "Dash");
json.put("num", new Integer(100));
json.put("balance", new Double(1000.21));
Please help me understand how to extract the JSON array data to variables And how to make the call.

Once you've run json_decode(), $data is just a "normal" php array with "normal" php values in it.
So, e.g.
/*
JSONObject json = new JSONObject();
json.put("name", "Dash");
json.put("num", new Integer(100));
json.put("balance", new Double(1000.21));
=>
*/
// $input = file_get_contents("php://input");
$input = '{"name":"Dash","num":100,"balance":1000.21}';
$data = json_decode($input, true);
$response = array(
'name_rev' => strrev($data['name']),
'num_mod_17' => $data['num'] % 17,
'balance_mul_2' => $data['balance'] * 2
);
echo json_encode($response, JSON_PRETTY_PRINT); // you might want to get rid off JSON_PRETTY_PRINT in production code
prints
{
"name_rev": "hsaD",
"num_mod_17": 15,
"balance_mul_2": 2000.42
}
two more tips:
You should test whether $data contains all the elements you expect, before accessing them, see http://docs.php.net/isset , http://docs.php.net/filter et al
a java Double() called balance. A java double is a 64-bit IEEE 754 floating point. And you might hit at one point (if not the range than) the precision limit, see What is the best data type to use for money in java app?

Plain string template query for elasticsearch through java API?

I have a template foo.mustache saved in {{ES_HOME}}/config/scripts.
POST to http://localhost:9200/forward/_search/template with the following message body returns a valid response:
{
"template": {
"file": "foo"
},
"params": {
"q": "a",
"hasfilters": false
}
}
I want to translate this to using the java API now that I've validated all the different components work. The documentation here describes how to do it in java:
SearchResponse sr = client.prepareSearch("forward")
.setTemplateName("foo")
.setTemplateType(ScriptService.ScriptType.FILE)
.setTemplateParams(template_params)
.get();
However, I would instead like to just send a plain string query (i.e. the contents of the message body from above) rather than build up the response using the java. Is there a way to do this? I know with normal queries, I can construct it like so:
SearchRequestBuilder response = client.prepareSearch("forward")
.setQuery("""JSON_QUERY_HERE""")
I believe the setQuery() method wraps the contents into a query object, which is not what I want for my template query. If this is not possible, I will just have to go with the documented way and convert my json params to Map<String, Object>

I ended up just translating my template_params to a Map<String, Object> as the documentation requires. I utilized groovy's JsonSlurper to convert the text to an object with a pretty simple method.
import groovy.json.JsonSlurper
public static Map<String,Object> convertJsonToTemplateParam(String s) {
Object result = new JsonSlurper().parseText(s);
//Manipulate your result if you need to do any additional work here.
//I.e. Programmatically determine value of hasfilters if filters != null
return (Map<String,Object>) result;
}
And you could pass in the following as a string to this method:
{
"q": "a",
"hasfilters": true
"filters":[
{
"filter_name" : "foo.untouched",
"filters" : [ "FOO", "BAR"]
},
{
"filter_name" : "hello.untouched",
"list" : [ "WORLD"]
}
]
}