in my case i use a dataset(dataframe) in JavaSparkSQL.
This dataset result from an JSON file. The json file is formed from key-value.When i lunch a query for see the value i write for examle:
SELECT key1.name from table
example JSON file
{
"key1":
{ "name": ".....",....}
"key2":
{ "name":"....",....}
}
my question is, when i want acceding at all key,I believe I should use a REGEX like
select key*.name from table
but i don't know the regex!
please help
I am afraid no such syntax is available in (spark) SQL.
You may want to construct your query programmatically though.
Something like :
String sql = Stream.of(ds.schema().fieldNames()).filter(name -> name.startsWith("key")).collect(Collectors.joining(", ", "select ", " from table"));
System.out.println(sql);
or even
Dataset<Row> result = spark.table("table").select(Stream.of(ds.schema().fieldNames()).filter(name -> name.startsWith("key")).map(name -> ds.col(name))
.toArray(Column[]::new));
result.show();
HTH!
Related
Context
I need to rewrite some code previoulsy with Jongo but using Springframework MongoDb. Previous code was:
eventsCollection
.aggregate("{$match:" + query + "}")
.and("{$group: {_id: '$domain', domain: {$first: '$domain'}, codes: {$push: '$code'}}}")
.and("{$project : { _id: 0, domain: 1 , codes: 1 } }")
.as(DomainCodes.class);
where eventsCollection is Jongo MongoCollection, and query is a String containing with criteria.
Problem
New code must probably look like :
Aggregation myAggregation = Aggregation.newAggregation(
Aggregation.match(/* something here */),
Aggregation.group("domain").first("domain").as("domain").push("code").as("codes")
);
mongoTemplate.aggregate(myAggregation, "collectionName", DomainCodes.class);
but I don't find a way to create match criteria using String (similare as BasicQuery that can take a query as String for argument)
Question
In order to change as little code as possible, is there anyway to use query String as in Jongo ?
Thank you,
Hi how can I use database stored GString definitions for dynamically generated data. I was able to use GString for pick and choose row attributes if format is defined in the code
code_format = "${-> row.ACCOUNT} ${-> row.ACCOUNT_OWNER}"
However if same definition is extracted from database my code is not working.
Sql sql = Sql.newInstance(url, login, password, driver);
sql.eachRow(SQL) { row ->
code_format = "${-> row.ACCOUNT} ${-> row.ACCOUNT_OWNER}"
database_format = "${-> row.REPORT_ATTRIBUTES}"
println "1- " + code_format
println "2- " + database_format
println "CODE : " + code_format.dump()
println "DB : " + database_format.dump()
}
When I run this code I am getting following output;
1- FlowerHouse Joe
2- ${-> row.ACCOUNT} ${-> row.ACCOUNT_OWNER}
CODE : <org.codehaus.groovy.runtime.GStringImpl#463cf024 strings=[, , ] values=[GString$_run_closure1_closure2#44f289ee, GString$_run_closure1_closure3#f3d8b9f]>
DB : org.codehaus.groovy.runtime.GStringImpl#4f5e9da9 strings=[, ] values=[GString$_run_closure1_closure4#11997b8a]
row.REPORT_ATTRIBUTES returns String because database doesn't know groovy stings format.
GString is template, which can be created from string.
So you can do something like:
def engine = new groovy.text.SimpleTemplateEngine()
println engine.createTemplate(row.REPORT_ATTRIBUTES).make([row:row]).toString()
I am trying to add a sort/ordering query.
At my java:
StructuredQueryBuilder qb = new StructuredQueryBuilder();
QueryDefinition queryDef = qb.and(qb.value(qb.jsonProperty("status"), "Active"));
SearchHandle resultsHandle = new SearchHandle();
queryManager.setPageLength(PAGE_SIZE_TEN);
int start = PAGE_SIZE_TEN * (pageNumber - 1) + 1;
queryManager.search(queryDef, resultsHandle, start);
The above will return the resultsHandle with 10 json files found for each page specified for the variable "start", with status "Active".
My question is how do I include a sorting query like maybe something along the line of the following:
QueryDefinition queryDef = qb.and(qb.value(qb.jsonProperty("status"), "Active"),
qb.sort?(qb.jsonProperty("dateCreated"));
I want it to get me the 1st 10 json files in order of latest date. It is too late to do a Comparator after getting the result, as the result returns a random 10 json files not in any particular order.
A few samples of the json files will look as such:
1.json
[
{
"id":"1",
"dateCreated":"2017-10-01 12:00:00",
"status":"Active"
"body":"This is a test"
}
]
2.json
[
{
"id":"2",
"dateCreated":"2017-10-02 12:00:00",
"status":"Active"
"body":"This is a test 2"
}
]
I realized there's a enum StructuredQueryBuilder.Ordering, how do I use it?
StructuredQueryBuilder.Ordering is specifically for use with near-query and is unrelated to what you want to do. You need to use query options to define a sort order for your search results. See the sort-order query option:
http://docs.marklogic.com/guide/search-dev/appendixa#id_44212
Options can be pre-defined and installed on MarkLogic and then referenced in your search, or you can define them at runtime and combine them with your structured query in a combined query.
Predefined: http://docs.marklogic.com/guide/java/query-options#chapter
Dynamic: http://docs.marklogic.com/guide/java/searches#id_76144
I'm trying to create a multi-value parameter in SpagoBI.
Here is my data set query whose last line appears to be causing an issue.
select C."CUSTOMERNAME", C."CITY", D."YEAR", P."NAME"
from "CUSTOMER" C, "DAY" D, "PRODUCT" P, "TRANSACTIONS" T
where C."CUSTOMERID" = T."CUSTOMERID"
and D."DAYID" = T."DAYID"
and P."PRODUCTID" = T."PRODUCTID"
and _CITY_
I created before open script in my dataset which looks like this:
this.queryText = this.queryText.replace(_CITY_, " CUSTOMER.CITY in ( "+params["cp"].value+" ) ");
My parameter is set as string, display type dynamic list box.
When I run the report I'm getting that error.
org.eclipse.birt.report.engine.api.EngineException: There are errors evaluating script "
this.queryText = this.queryText.replace(_CITY_, " CUSTOMER.CITY in ( "+params["cp"].value+" ) ");
":
Fail to execute script in function __bm_beforeOpen(). Source:
Could anyone please help me?
Hello I managed to solve the problem. Here is my code:
var substring = "" ;
var strParamValsSelected=reportContext.getParameterValue("citytext");
substring += "?," + strParamValsSelected ;
this.queryText = this.queryText.replace("'xxx'",substring);
As You can see the "?" is necessary before my parameter. Maybe It will help somebody. Thank You so much for Your comments.
If you are using SpagoBI server and High charts (JFreeChart Engine) / JSChat Engine you can just use ($P{param_url}) in query,
or build dynamic query using Java script / groovy Script
so your query could also be:
select C."CUSTOMERNAME", C."CITY", D."YEAR", P."NAME"
from "CUSTOMER" C, "DAY" D, "PRODUCT" P, "TRANSACTIONS" T
where C."CUSTOMERID" = T."CUSTOMERID"
and D."DAYID" = T."DAYID"
and P."PRODUCTID" = T."PRODUCTID"
and CUSTOMER."CITY" in ('$P{param_url}')
I'm new in mongodb. I have following data as a JSON format in mongodb. I need to search the bookLabel or the shortLabel for the book and it should show me all the information about the book. For example: if I query for 'Cosmos' it'll show all the description about the book, like: bookLabel, writer, yearPublish, url. How can I do that in java? Need query, please help.
"Class":"Science",
"Description":[
{
"bookLabel":"Cosmos (Mass Market Paperback)",
"shortLabel":"Cosmos",
"writer":"Carl Sagan",
"yearPublish":[
"2002"
],
"url":"https://www.goodreads.com/book/show/55030.Cosmos"
},
{
"bookLabel":"The Immortal Life of Henrietta Lacks",
"shortLabel":"Immortal Life",
"writer":"Rebecca Skloot",
"yearPublish":[
"2010, 2011"
],
"url":"https://www.goodreads.com/book/show/6493208-the-immortal-life-of-henrietta-lacks"
}
],
"Class":"History",
"Description":[
{
"bookLabel":"The Rise and Fall of the Third Reich",
"shortLabel":"Rise and Fall",
"writer":"William L. Shirer",
"yearPublish":[
"1960"
],
"url":"https://www"
}
]
}
With MongoDB Java Driver v3.2.2 you can do something like this:
FindIterable<Document> iterable = collection.find(Document.parse("{\"Description.shortLabel\": {$regex: \"Cosmos\"}"));
This returns all documents containing Cosmos in the Description.shortLabel nested field. For an exact match, try this {"Description.shortLabel": "Cosmos"}. Replace shortLabel with bookLabelto search the bookLabel field. Then you can do iterable.forEach(new Block<Document>()) on the returned documents. To search both bookLabel and shortLabel, you can do a $or{}. My syntax could be wrong so check the MongoDB manual. But this is the general idea.
For this, you can use MongoDB's Text Search Capabilities. You'll have to create a text index on your collection for that.
First of all create a text index on your collection on fields bookLabel and shortLabel.
db.books.createIndex({ "Description.bookLabel" : "text", "Description.shortLabel" : "text" })
Note that this is done in the Mongo shell
Then
DBObject command = BasicDBObjectBuilder
.start("text", "books")
.append("search", "Cosmos")
.get();
CommandResult result = db.command(command);
BasicDBList results = (BasicDBList) result.get("results");
for(Object o : results) {
DBObject dbo = (DBObject) ((DBObject) o).get("obj");
String id = (String) dbo.get("_ID");
System.out.println(id);
}
Haven't really tested this. But just give it a try. Should work.