i have a json like this:
{"code":200,"msg":"111",data:[{"name":"va","list":[{"code":"11"}]},{"name":"va","list":[{"code":"11"}]}]}
my class like this:
#Serializable
class ServerResponse<T> {
#SerialName(value = "code")
var code = 0
#SerialName(value = "msg")
var msg: String? = null
#SerialName(value = "data")
var data: T? = null
private set
}
#Serializable
data class PluginInfo(
#SerialName(value = "name")
val name: String,
#SerialName(value = "list")
val list: MutableList<PluginConfig>
)
#Serializable
data class PluginConfig(
#SerialName(value = "code")
val code: String
)
when i use JSON.decodeFromString:
val response =
json.decodeFromString<ServerResponse<MutableList<PluginInfo>>>(pluginInfoJson)
this throw exception:
kotlinx.serialization.json.internal.JsonDecodingException: Unexpected JSON token at offset 271: Expected quotation mark '"', but had '1' instead at path: $.data[0].
Using Spark 2.11, I've the following Dataset (read from Cassandra table):
+------------+----------------------------------------------------------+
|id |attributes |
+------------+----------------------------------------------------------+
|YH8B135U123|[{"id":1,"name":"function","score":10.0,"snippets":1}] |
+------------+----------------------------------------------------------+
This is the printSchema():
root
|-- id: string (nullable = true)
|-- attributes: string (nullable = true)
The attributes column is an array of JSON objects. I'm trying to explode it into Dataset but keep failing. I was trying to define schema as follow:
StructType type = new StructType()
.add("id", new IntegerType(), false)
.add("name", new StringType(), false)
.add("score", new FloatType(), false)
.add("snippets", new IntegerType(), false );
ArrayType schema = new ArrayType(type, false);
And provide it to from_json as follow:
df = df.withColumn("val", functions.from_json(df.col("attributes"), schema));
This fails with MatchError:
Exception in thread "main" scala.MatchError: org.apache.spark.sql.types.IntegerType#43756cb (of class org.apache.spark.sql.types.IntegerType)
What's the correct way to do that?
You can specify the schema this way :
val schema = ArrayType(
StructType(Array(
StructField("id", IntegerType, false),
StructField("name", StringType, false),
StructField("score", FloatType, false),
StructField("snippets", IntegerType, false)
)),
false
)
val df1 = df.withColumn("val", from_json(col("attributes"), schema))
df1.show(false)
//+-----------+------------------------------------------------------+------------------------+
//|id |attributes |val |
//+-----------+------------------------------------------------------+------------------------+
//|YH8B135U123|[{"id":1,"name":"function","score":10.0,"snippets":1}]|[[1, function, 10.0, 1]]|
//+-----------+------------------------------------------------------+------------------------+
Or for Java:
import static org.apache.spark.sql.types.DataTypes.*;
StructType schema = createArrayType(createStructType(Arrays.asList(
createStructField("id", IntegerType, false),
createStructField("name", StringType, false),
createStructField("score", FloatType, false),
createStructField("snippets", StringType, false)
)), false);
You can define the schema as a literal string instead:
val df2 = df.withColumn(
"val",
from_json(
df.col("attributes"),
lit("array<struct<id: int, name: string, score: float, snippets: int>>")
)
)
df2.show(false)
+-----------+------------------------------------------------------+------------------------+
|id |attributes |val |
+-----------+------------------------------------------------------+------------------------+
|YH8B135U123|[{"id":1,"name":"function","score":10.0,"snippets":1}]|[[1, function, 10.0, 1]]|
+-----------+------------------------------------------------------+------------------------+
If you prefer to use a schema:
val spark_struct = new StructType()
.add("id", IntegerType, false)
.add("name", StringType, false)
.add("score", FloatType, false)
.add("snippets", IntegerType, false)
val schema = new ArrayType(spark_struct, false)
val df2 = df.withColumn(
"val",
from_json(
df.col("attributes"),
schema
)
)
Two problems with your original code were: (1) you used the reserved keyword type as a variable name, and (2) you don't need to use new in add.
So I'm trying to serialize/deserialize RSS feeds using Jackson XML in Kotlin. However, currently it's not doing either properly. When I instantiate the RSS object manually and print out the xml, it seems like it's ignoring the localName and namespace parameters in #JacksonXmlProperty. Here is my code:
#JacksonXmlRootElement(localName="rss")
class RSSFeed(
#JacksonXmlProperty(localName="channel")
var channel: Channel
) {
constructor() : this(Channel())
}
class Channel(
#JacksonXmlProperty(localName="title")
var title: String? = null,
#JacksonXmlCData
#JacksonXmlProperty(localName="description")
var description: String? = null,
#JacksonXmlProperty(localName="image")
var image: RSSImage? = null,
#JacksonXmlProperty(localName = "explicit", namespace="itunes")
var isExplicit: String? = "no",
#JacksonXmlProperty(localName = "pubDate")
var publishDate: String? = null,
#JacksonXmlProperty(localName = "type", namespace="itunes")
var episodeType: String? = null,
#JacksonXmlProperty(localName = "link", namespace="atom")
var atomLink: AtomLink? = null,
#JacksonXmlProperty(localName = "new-feed-url", namespace="itunes")
var newFeedUrl: String? = null
)
Then I just do:
println(XmlMapper().writeValueAsString(RSSFeed(Channel(title = "blah", description = "blah", image = RSSImage(url = "https://someurl.com", link = "somelink"), isExplicit = "yes", atomLink = AtomLink(href="blah.com"), newFeedUrl = "thisisthenewfeed"))))
However, when I print that out it ignores the namespace and localName properties and just uses the variable names. What am I doing wrong?
Xml output: <rss><channel><title>blah</title><description><![CDATA[blah]]></description><image><url>https://someurl.com</url><title/><link><![CDATA[somelink]]></link></image><publishDate/><episodeType/><atomLink href="blah.com"/><newFeedUrl>thisisthenewfeed</newFeedUrl></channel></rss>
edit: it also doesn't seem to be printing the isExplicit property for some reason, not sure what that's about.
Well I figured it out! It turns out this was a Kotlin issue. I just had to do this: val xmlMapper = XmlMapper().registerModule(KotlinModule())
...and make sure that this is in your dependencies (I use gradle):
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
Anyways I hope this helps someone in the future.
I have following XML to create nested Kotlin objects but it is not mapping all TimeCycleData element
<TimeCycle>
<Last>
<Date>2001-06-13T01:00:00.000Z</Date>
<TimeCycleData>
<Hours type="F">123</Hours>
</TimeCycleData>
<TimeCycleData>
<Cycles>1234</Cycles>
</TimeCycleData>
<TimeCycleData>
<Land>1234</Land>
</TimeCycleData>
</Last>
</TimeCycle>
and want to map following Kotlin data classes
data class TimeCycle(
#field:XmlElement(name = "Last")
val last: Last? = null
)
data class Last(
#field:XmlElement(name = "Date")
#field:XmlJavaTypeAdapter(value = LocalDateTimeAdapter::class, type = LocalDateTime::class)
val date: LocalDateTime? = null,
#field:XmlElement(name = "TimeCycleData")
val timeCycleData: TimeCycleData? = null
)
data class TimeCycleData(
#field:XmlElement(name = "Hours")
val hours: DurationDetails? = null,
#field:XmlElement(name = "Cycles")
val cycles: Int? = null,
#field:XmlElement(name = "Land")
val land: Int? = null
)
data class DurationDetails(
#field:XmlValue
#field:XmlJavaTypeAdapter(value = DurationAdapter::class, type = Duration::class)
val value: Duration? = null,
#field:XmlAttribute(name = "type")
val type: String = ""
)
when I unmarshal the XML, only the first TimeCycleData with Hours is filled. How can I merge all TimeCycleData into one single object?
UPDATE: corrected submitted xml
I guess
#field:XmlElement(name = "TimeCycleData")
val timeCycleData: TimeCycleData? = null
should be declared in somehow this way
#field:XmlElement(name = "TimeCycleData")
#field:XmlElementWrapper(name = "TimeCycleInfo")
val timeCycleInfo: List<TimeCycleData>? = null
.
I want to change a xml string to a object but it seems like keeps throwing error to me and I'm not sure how to use those #XmlRootElement stuff
Just view/reply it as JAVA also can although written in kotlin
Here it's the XML string
<xml>
<return_code><![CDATA[SUCCESS]]></return_code>
<return_msg><![CDATA[OK]]></return_msg>
<appid><![CDATA[wx0b6dc231d20b379f1]]></appid>
<mch_id><![CDATA[1508319851]]></mch_id>
<nonce_str><![CDATA[mqy4nB6xGoyC1QPY]]></nonce_str>
<sign><![CDATA[2D9A3195E196F679D3916C5DC74754B4]]></sign>
<result_code><![CDATA[SUCCESS]]></result_code>
<prepay_id><![CDATA[wx2116190646297891sfae86747980208850875]]></prepay_id>
<trade_type><![CDATA[JSAPI]]></trade_type>
</xml>
Here it's my data class
#XmlRootElement
data class WxPayResult(
val return_code: String = "",
val return_msg: String = "",
//return_code as SUCCESS will only return the following params
val appid: String? = null,
val mch_id: String? = null,
val device_info: String? = null,
val nonce_str: String? = null,
val sign: String? = null,
val result_code: String? = null,
val err_code: String? = null,
val err_code_des: String? = null,
//return_code and result_code both as success will only return the following params
val trade_type: String? = null,
val prepay_id: Int? = null,
val code_url: String? = null
)
Here it's my code, "xmlreturn" is the xml string
val jaxbContext = JAXBContext.newInstance(WxPayResult::class.java)
val unmarshaller = jaxbContext.createUnmarshaller()
val reader = StringReader(xmlreturn)
val person = unmarshaller.unmarshal(reader)
Here it's the error
javax.xml.bind.UnmarshalException: unexpected element (uri:"", local:"xml"). Expected elements are <{}wxPayResult>
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.handleEvent(UnmarshallingContext.java:741)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportError(Loader.java:262)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportError(Loader.java:257)
at com.sun.xml.bind.v2.runtime.unmarshaller.Loader.reportUnexpectedChildElement(Loader.java:124)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext$DefaultRootLoader.childElement(UnmarshallingContext.java:1149)
at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext._startElement(UnmarshallingContext.java:574)
I know probably i need to add something into the data class but i don know what to add. Thanks in advance.
Your root element's name differs from your xml root element, which also the message suggests. It expects wxPayResult, but you give it a xml.
Either deliver an XML with wxPayResult as root element or supply the name to XMLRootElement, e.g.
#XmlRootElement(name = "xml")