I have dataframe , wanted to convert into JSON ARRAY Please find the example below
Dataframe
+------------+--------------------+----------+----------------+------------------+--------------
| Name| id|request_id|create_timestamp|deadline_timestamp|
+------------+--------------------+----------+----------------+------------------+--------------
| Freeform|59bbe3ad-f487-44| htvjiwmfe| 1589155200000| 1591272659556
| D23|59bbe3ad-f487-44| htvjiwmfe| 1589155200000| 1591272659556
| Stores|59bbe3ad-f487-44| htvjiwmfe| 1589155200000| 1591272659556
|VacationClub|59bbe3ad-f487-44| htvjiwmfe| 1589155200000| 1591272659556
Wanted in Json Like below:
[
{
"testname":"xyz",
"systemResponse":[
{
"name":"FGH",
"id":"59bbe3ad-f487-44",
"request_id":1590791280,
"create_timestamp":1590799280
},
{
"name":"FGH",
"id":"59bbe3ad-f487-44",
"request_id":1590791280,
"create_timestamp":1590799280,
}
]
}
]
You can define 2 beans
Create Array from the 1st DF as Array of inner Beans
Define a parent bean with testname and requestDetailArray as Array
Please also find code inline comments
object DataToJsonArray {
def main(args: Array[String]): Unit = {
val spark = Constant.getSparkSess
import spark.implicits._
//Load you dataframe
val requestDetailArray = List(
("Freeform", "59bbe3ad-f487-44", "htvjiwmfe", "1589155200000", "1591272659556"),
("D23", "59bbe3ad-f487-44", "htvjiwmfe", "1589155200000", "1591272659556"),
("Stores", "59bbe3ad-f487-44", "htvjiwmfe", "1589155200000", "1591272659556"),
("VacationClub", "59bbe3ad-f487-44", "htvjiwmfe", "1589155200000", "1591272659556")
).toDF
//Map your Dataframe to RequestDetails bean
.map(row => RequestDetails(row.getString(0), row.getString(1), row.getString(2), row.getString(3), row.getString(4)))
//Collect it as Array
.collect()
//Create another data frme with List[BaseClass] and set the (testname,Array[RequestDetails])
List(BaseClass("xyz", requestDetailArray)).toDF()
.write
//Output your Dataframe as JSON
.json("/json/output/path")
}
}
case class RequestDetails(Name: String, id: String, request_id: String, create_timestamp: String, deadline_timestamp: String)
case class BaseClass(testname: String = "xyz", systemResponse: Array[RequestDetails])
Check below code.
import org.apache.spark.sql.functions._
df.withColumn("systemResponse",
array(
struct("id","request_id","create_timestamp","deadline_timestamp").as("data")
)
)
.select("systemResponse")
.toJSON
.select(col("value").as("json_data"))
.show(false)
+-----------------------------------------------------------------------------------------------------------------------------------------------+
|json_data |
+-----------------------------------------------------------------------------------------------------------------------------------------------+
|{"systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
|{"systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
|{"systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
|{"systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
+-----------------------------------------------------------------------------------------------------------------------------------------------+
Updated
scala> :paste
// Entering paste mode (ctrl-D to finish)
df.withColumn("systemResponse",
array(
struct("id","request_id","create_timestamp","deadline_timestamp").as("data")
)
)
.withColumn("testname",lit("xyz"))
.select("testname","systemResponse")
.toJSON
.select(col("value").as("json_data"))
.show(false)
// Exiting paste mode, now interpreting.
+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
|json_data |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
|{"testname":"xyz","systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
|{"testname":"xyz","systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
|{"testname":"xyz","systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
|{"testname":"xyz","systemResponse":[{"id":"59bbe3ad-f487-44","request_id":"htvjiwmfe","create_timestamp":"1589155200000","deadline_timestamp":"1591272659556"}]}|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
I have a yaml file that I want to read its contents in scala , so I parse it using io.circe.yaml to json
var js = yaml.parser.parse(ymlText)
var json=js.valueOr(null)
var jsonstring=json.toString
val json2 = parse(jsonstring)
the yamltext is like this:
ALL:
Category1:
Subcategory11 : 1.5
Subcategory12 : 0
Subcategory13 : 0
Subcategory14 : 0.5
Category2:
Subcategory21 : 1.5
Subcategory22 : 0.3
Subcategory23 : 0
Subcategory24 : 0
what I want is to filter the subcategories that has Zero values, I've used this code:
val elements = (json2 \\"ALL" ).children.map(x=>(x.values))
var subCategories=elements.map{case(a,b)=>(b)}
var cats=elements.map{case(a,b)=>(b.asInstanceOf[Map[String,Double]])}
cats.map(x=>x.filter{case(a,b)=>b>0.0})
But the last line gives me this error:
scala.math.BigInt cannot be cast to java.lang.Double
I'm not sure why you do toString + parse and which parse is used but you probably don't need it. Also you didn't describe your expected result so here are a few guesses of what you might need:
import java.io._
import io.circe._
import io.circe.yaml._
import io.circe.parser._
def test(): Unit = {
// test data instead of a file
val ymlText =
"""
|ALL:
| Category1:
| Subcategory11 : 1.5
| Subcategory12 : 0
| Subcategory13 : 0
| Subcategory14 : 0.5
| Category2:
| Subcategory21 : 1.5
| Subcategory22 : 0.3
| Subcategory23 : 0
| Subcategory24 : 0
""".stripMargin
var js = yaml.parser.parse(new StringReader(ymlText))
var json: Json = js.right.get
val categories = (json \\ "ALL").flatMap(j => j.asObject.get.values.toList)
val subs = categories.flatMap(j => j.asObject.get.toList)
val elements: List[(String, Double)] = subs.map { case (k, v) => (k, v.asNumber.get.toDouble) }
.filter {
case (k, v) => v > 0.0
}
println(s"elements: $elements")
val allCategories = (json \\ "ALL").flatMap(j => j.asObject.get.toList).toMap
val filteredTree: Map[String, Map[String, Double]] = allCategories
.mapValues(catJson => catJson.asObject.get.toList.map { case (subName, subJson) => (subName, subJson.asNumber.get.toDouble) }
.filter { case (subName, subValue) => subValue > 0.0 }
.toMap)
println(s"filteredTree : $filteredTree")
}
And the output for that is:
elements: List((Subcategory11,1.5), (Subcategory14,0.5), (Subcategory21,1.5), (Subcategory22,0.3))
filteredTree : Map(Category1 -> Map(Subcategory11 -> 1.5, Subcategory14 -> 0.5), Category2 -> Map(Subcategory21 -> 1.5, Subcategory22 -> 0.3))
Hope one of those version is what you needed.
I am running the command below with is passing two argument to my script file(This below command is being constructed from java):
sh /home/accure/Desktop/scripts/cronhandler.sh "/home/accure/Desktop/scripts/pipeline.sh 028d8ccb-4c46-4e02-a9b0-3c97a383daaf" "* * * * */5"
Where "/home/accure/Desktop/scripts/pipeline.sh 028d8ccb-4c46-4e02-a9b0-3c97a383daaf" is the first argument and "* * * * */5" is the second argument which is basically a cron expression.
cronhandler.sh is my script file which contain this code:
if [ "$#" -eq 2 ]
then
echo "Crontab with create & update functionality"
command=$1
cron_exp=$2
echo "cron_exp=$cron_exp"
echo "command=$command"
cron_exp=`echo $cron_exp | sed 's/"//g' `
command=`echo $command | sed 's/"//g' `
echo "cron_exp=$cron_exp"
echo "command=$command"
if [[ "$cron_exp" != " " && "$command" != " " ]]
then
crontab -l | grep -q "$command"
if [ $? -eq 0 ]
then
crontab -l | grep -v "$command" | crontab -
echo "CRON entry deleted successfully.."
else
echo "CRON entry deletion failed."
fi
crontab -l | grep -q "$command" || (crontab -l 2>/dev/null; echo "$cron_exp $command") | crontab -
if [ $? -eq 0 ]
then
echo "CRON entry added successfully.."
else
echo "CRON entry addition failed."
fi
fi
fi
The error I'm getting while running the script is below. Actually it's listing out my files and folders from current working directory instead of adding the cron expression to cron tab.
Crontab with create & update functionality cron_exp=* * * * */5
command=/home/accure/Desktop/scripts/pipeline.sh
028d8ccb-4c46-4e02-a9b0-3c97a383daaf cron_exp=14.5 LinkedList vs
ArrayList in Java-QWMyhFUtFHo.mp4
[57.64911004342139,10.407439861446619]
accure-facebook-parser-1.0-SNAPSHOT-jar-with-dependencies.jar
apache-tomcat-7.0.41 Core Java With OCJP_SCJP - Collections Part-11 _
Map _ Hashmap _ linked Hashmap-pSGvbJ7GJ68.mp4.part Desktop Documents
Downloads epel-release-6-8.noarch.rpm How HashMap works in Java With
Animation!! whats new in java8 tutorial-c3RVW3KGIIE.mp4 ingester Java
interview - How Hashmap works -YR7Vp7HcAgs.mp4 jce_policy-8.zip
mongodata Music netbeans-7.4 NetBeansProjects Pictures Public quasar
robomongo-0.8.4-x86_64 sa softs solr-6.2.1 solr-6.2.1.tgz Templates
testdata UnlimitedJCEPolicyJDK8 Videos Vinod 14.5 LinkedList vs
ArrayList in Java-QWMyhFUtFHo.mp4
[57.64911004342139,10.407439861446619]
accure-facebook-parser-1.0-SNAPSHOT-jar-with-dependencies.jar
apache-tomcat-7.0.41 Core Java With OCJP_SCJP - Collections Part-11 _
Map _ Hashmap _ linked Hashmap-pSGvbJ7GJ68.mp4.part Desktop Documents
Downloads epel-release-6-8.noarch.rpm How HashMap works in Java With
Animation!! whats new in java8 tutorial-c3RVW3KGIIE.mp4 ingester Java
interview - How Hashmap works -YR7Vp7HcAgs.mp4 jce_policy-8.zip
mongodata Music netbeans-7.4 NetBeansProjects Pictures Public quasar
robomongo-0.8.4-x86_64 sa softs solr-6.2.1 solr-6.2.1.tgz Templates
testdata UnlimitedJCEPolicyJDK8 Videos Vinod 14.5 LinkedList vs
ArrayList in Java-QWMyhFUtFHo.mp4
[57.64911004342139,10.407439861446619]
accure-facebook-parser-1.0-SNAPSHOT-jar-with-dependencies.jar
apache-tomcat-7.0.41 Core Java With OCJP_SCJP - Collections Part-11 _
Map _ Hashmap _ linked Hashmap-pSGvbJ7GJ68.mp4.part Desktop Documents
Downloads epel-release-6-8.noarch.rpm How HashMap works in Java With
Animation!! whats new in java8 tutorial-c3RVW3KGIIE.mp4 ingester Java
interview - How Hashmap works -YR7Vp7HcAgs.mp4 jce_policy-8.zip
mongodata Music netbeans-7.4 NetBeansProjects Pictures Public quasar
robomongo-0.8.4-x86_64 sa softs solr-6.2.1 solr-6.2.1.tgz Templates
testdata UnlimitedJCEPolicyJDK8 Videos Vinod 14.5 LinkedList vs
ArrayList in Java-QWMyhFUtFHo.mp4
[57.64911004342139,10.407439861446619]
accure-facebook-parser-1.0-SNAPSHOT-jar-with-dependencies.jar
apache-tomcat-7.0.41 Core Java With OCJP_SCJP - Collections Part-11 _
Map _ Hashmap _ linked Hashmap-pSGvbJ7GJ68.mp4.part Desktop Documents
Downloads epel-release-6-8.noarch.rpm How HashMap works in Java With
Animation!! whats new in java8 tutorial-c3RVW3KGIIE.mp4 ingester Java
interview - How Hashmap works -YR7Vp7HcAgs.mp4 jce_policy-8.zip
mongodata Music netbeans-7.4 NetBeansProjects Pictures Public quasar
robomongo-0.8.4-x86_64 sa softs solr-6.2.1 solr-6.2.1.tgz Templates
testdata UnlimitedJCEPolicyJDK8 Videos Vinod */5
command=/home/accure/Desktop/scripts/pipeline.sh
028d8ccb-4c46-4e02-a9b0-3c97a383daaf CRON entry deletion failed.
"-":2: bad minute errors in crontab file, can't install. CRON entry
addition failed.
Note: Rather being added to crontab, * in cron expression is listing out my files and folder from current working directory.
Java code from where i am calling the command line is given below :
String pipelineFilePath="sh /home/accure/Desktop/scripts/pipeline.sh 028d8ccb-4c46-4e02-a9b0-3c97a383daaf";
String cronExp="* * * * */5";
pipelineFilePath = "\"" + pipelineFilePath + "\"";
cronExp = "\"" + cronExp + "\"";
command = "sh " + /home/accure/Desktop/scripts/cronhandler.sh + " " + pipelineFilePath + " " + cronExp;
runScript(command);
public void runScript(String script) throws InterruptedException {
final String cmd = script;
Thread runScript = new Thread(new Runnable() {
public void run() {
try {
DefaultExecutor executor = new DefaultExecutor();
CommandLine commandLine = CommandLine.parse(cmd);
executor.setExitValue(1);
int exitValue = executor.execute(commandLine);
} catch (Exception ex) {
}
}
});
runScript.start();
Thread.sleep(10000);
}
Try to put single quotes (') instead of double quotes (") around your arguments:
sh /home/accure/Desktop/scripts/cronhandler.sh '/home/accure/Desktop/scripts/pipeline.sh 028d8ccb-4c46-4e02-a9b0-3c97a383daaf' '* * * * */5'
It will prevent your shell's filename expansion.
I developed IDE using eclipse EMF/RCP in that I Developed one editor using xtext,rcp and emf
Following is my grammar for that Section
// automatically generated by Xtext
grammar com.xyz.pmide.RoutingLineINI with org.eclipse.xtext.common.Terminals
import "platform:/resource/com.xyz.pmide.routingline.xtext.model/model/pmrouting.ecore"
import "http://www.eclipse.org/emf/2002/Ecore" as ecore
IniRoutingLineProject:
{IniRoutingLineProject}
(iniRoutingConfig=IniRoutingConfiguration)?
;
IniRoutingConfiguration:
{IniRoutingConfiguration}
(iniRoutingSectConfig=IniRoutingLogicSection)
(iniLogMessageSectionConfig=IniLogMessageSet)?
(iniPreDefGroupSectionConfig=IniPreDefGroupVariableSet)?
(iniRegWSSectionConfig=IniRegisterWSCommandSet)?
(iniOtherRoutingSectionConfig=IniRoutingPropertySet)
(iniGlobalRoutingSectionConfig=IniGlobalRoutingSection)?
(iniAlarmSignalSectionConfig=IniAlarmSignalSection)?
(iniOptionPopupSectionConfig=IniOptionPopupSection)
;
IniRoutingLogicSection:
{IniRoutingLogicSection}
'[Routing]'('\r\n')*
// ('Warning_Rate='warningRate=ANS1TERM)?('\r\n')*
// ('Busy_Rate='busyRate=ANS1TERM)?('\r\n')*
// ('MaxRecursionConcurence=' maxRecursionOccurence=ANS1TERM)?('\r\n')*
// ('WS_Speed=' wsSpeed=ANS1TERM)?('\r\n')*
// ('ReactivationTime=' reactivationTime=ANS1TERM)?('\r\n')*
// ('WaitTime_WS_Reuse=' waitTimeWSReuse=ANS1TERM)?('\r\n')*
(routingLines+=IniRoutingLine)*('\r\n')*
;
terminal ALPHA :('a'..'z'|'A'..'Z');
terminal NUM:('0'..'9')+;
terminal SYMBOL : ('.'|','|'('|')'|'{'|'}'|'<'|'>'|'+'|'-'|'*'|'%'|'!'|'"'|':'|'|'|'?'|'#'|'_'|' '|'=');
terminal SYMBOL1 : ('['|']')*;
terminal SYMBOL2 : ('/'|';')* ;
terminal ANS : (ALPHA|NUM|SYMBOL)* SYMBOL2;
terminal SL_COMMENT : '//-' !('\n'|'\r')* ('\r'? '\n')?;
terminal ML_COMMENT : '//*' -> '*//';
ANSTERM : ANS;
ANSS1TERM : (ANS|SYMBOL1)+;
GRPCOMMENT_STR_END : SYMBOL2("===========================================================================================================================================================================") ;
GRPCOMMENT :
(GRPCOMMENT_STR_END)('\r\n')*
SYMBOL2(ANSS1TERM)('\r\n')*
(GRPCOMMENT_STR_END)('\r\n')*
;
LINECOMMENT :
// (LINECOMMENT_STR_END)('\r\n')*
// (ANSS1TERM|LINECOMMENT)('\r\n')*
// (LINECOMMENT_STR_END)?('\r\n')*;
(('/*'('\r\n')*
(ANSS1TERM)('\r\n')*
(LINECOMMENT)?('\r\n')*
'*/'?('\r\n')*)|GENCOMMENT)
;
IniTimeStampSampleSectPropSet:
{IniTimeStampSampleSectPropSet}
'[TimeStamp_Sample]'('\r\n')*
(inLabKeyValue+=IniUnitPositionsPair)*('\r\n')*
;
IniPositionConnect:
{IniPositionConnect}
// (srcPosition=ANS1TERM)(destPosition=ANS1TERM)?('\r\n')*
(src_dest_Pos=ANSS1TERM)('\r\n')*
;
IniListSampleSection:
{IniListSampleSection}
'[LISTSAMPLE]'('\r\n')*
(iniOnType+=IniListSample_OnType)*('\r\n')*
;
IniListSample_OnType:
{IniListSample_OnType}
(sampleType=('OnType'|ANSS1TERM))(popUpNo=ANSS1TERM)?('\r\n')*
;
IniGeneralRoutingSectPropSet:
{IniGeneralRoutingSectPropSet}
'[General]'('\r\n')*
// (version=('version='|ANSTERM))?('\r\n')*
(keySet+=IniNewKey_NL)*('\r\n')*
;
IniChildWSTriggerSectionSet:
{IniChildWSTriggerSectionSet}
(iniTriggerSectionConfig+=IniTriggerSectionConfiguration)*('\r\n')*
;
IniTriggerSectionConfiguration:
{IniTriggerSectionConfiguration}
SYMBOL1(sectName=ANSTERM)SYMBOL1('\r\n')*
(triggerUnit=('TriggerUnit='|ANSTERM))?('\r\n')*
(createPoint=('CreatePoint='|ANSTERM))?('\r\n')*
;
Following is my part validation or u can say check rules for my sections in editor
#Check
def validate_TriggerSectionConfiguration(IniTriggerSectionConfiguration iniTriggerSectionConfiguration){
val value = iniTriggerSectionConfiguration.sectName;
if (value.startsWith("[") && !value.endsWith("]")) {
error("Missing \"]\"",iniTriggerSectionConfiguration, PmroutingPackage.Literals.INI_TRIGGER_SECTION_CONFIGURATION__SECT_NAME)
}
}
Following is my part of editor
If I change above name by capital S
If I will go to red error mark this error message I am getting
I am trying to find solution I didn’t find. My problem is in checkRule
If I haven’t write any validation for that error why xtext giving such
error. This Problem I got when my name starts with this Capital
letters error A, G, L, M, O, P, R, S, T, W. Is this my code issue or
xtext has some bugs or I am missing something?
I'm replicating this Java function into Clojure.
Config createConfig(Map<String, String> options) {
Config conf = new Config();
String foo = options.get("foo");
if (foo != null) { conf.setFoo(foo); }
String bar = options.get("bar");
if (bar != null) { conf.setBar(bar); }
// many other configs
return conf;
}
I came up with this,
(defn create-config [options]
(let [conf (Config.)]
(when-let [a (:foo options)] (.setFoo a))
(when-let [a (:bar options)] (.setBar a))
conf))
Is there a better way to do this?
How about this?
(defn create-config [{:keys (foo bar)}]
(let [config (Config.)]
(when foo (.setFoo config foo))
(when bar (.setBar config bar))))
Probably better would be to create a generic macro that can set any number of fields on an object based on a Clojure map. This could be prettier, but it works:
(defmacro set-fields
[options obj]
(let [o (gensym)
m (gensym)]
`(let [~m ~options
~o ~obj]
~#(map (fn [[field value]]
(let [setter (symbol (str "set" (clojure.string/capitalize (name field))))]
`(. ~o (~setter ~value))))
options)
~o)))
;; call like:
(set-fields {:daemon true :name "my-thread"} (Thread.))
;; translates to:
;; (let [G__1275 {:daemon true, :name "my-thread"}
;; G__1274 (Thread.)]
;; (. G__1274 (setDaemon true))
;; (. G__1274 (setName "my-thread"))
;; G__1274)
I'm not checking for null as this only looks at exactly what is passed in the options.
maybe checkout some->>
fillertext fillertext