Get a JSON from a Dataset in Spark SQL (java) - java

I have a Spark SQL application running on a server. It takes data from .parquet files and in each request performs an SQL query on those data. I need to send the JSON corresponding to the output of the query in the response.
This is what I do
Dataset<Row> sqlDF = spark.sql(query);
sqlDF.show();
So I know that the query works.
I tried returning sqlDF.toJSON().collect(), but in the other end I only receive [Ljava.lang.String;#1cd86ff9.
I tried writing sqlDF as a JSON file, but then I don't know how to add its content to the response, and it saves a structure of files that have nothing to do with a JSON file.
Any idea/suggestion?

You can return JSON String using the below code.
List<String> stringDataset = sqlDF.toJSON().collectAsList();
return stringDataset;
Jackson will return the JSON string in this case.
If you want to return proper JSONObject the you can use the below code :
List<Map<String,Object>> result= new ArrayList<>();
List<String> stringDataset = sqlDF.toJSON().collectAsList();
for(String s : stringDataset){
Map<String,Object> map = new HashMap<>();
map = mapper.readValue(s, new TypeReference<Map<String, String>>(){});
result.add(map);
}

Related

How can I convert ArrayList/Set to JSON and post data using postforobject method?

I have set which contains string ["a" , "b" , "c"] , I want to POST json data like (comma seperated and one string)
Here is
JSON
{"view" : "a,b,c",
"fruits" : "apple"}
to the endpoing using Resttemplate postForObject method? I have used GSON but that is not working in my project. Are there any other alternatives?
Here is my code
private run(set<data> datas) {
Set<string> stack = new hashset<>();
iterator<data> itr = datas.iterator();
while (itr.hasnext()) {
data macro = itr.next();
if (//some condition) {
stack.add(macro);
}
}
}
}
Resttemplate.getmessageconverters().add(stringconvertor);
String result = resttemplate.postforobject(endpoint, request, String.class);
}
If the data is in a specific class like format, you could go with the POJO approach that is encouraged by Spring Boot. But looking at your example, it seems like you want to achieve a one time JSON Object response.
import org.json.simple.JSONObject;
public static void run(set<data> datas, string endpoint){
// build your 'stack' set
String joined = String.join(",", stack);
JSONObject obj=new JSONObject();
obj.put("view",joined);
obj.put("fruits","apple");
//return the jsonObject as the response to your entrypoint using your method
}
You could also try the following if you use #ResponseBody annotation in Spring Boot that will convert the Response Body to the appropriate (JSON) format.
HashMap<String, String> map = new HashMap<>();
map.put("view", joined);
map.put("fruits", "apple");
return map;

Handling data return types with Mybatis SQL Builder

I am relatively new to Java MyBatis. I came across SQL Builder class in MyBatis. However, I don't understand how to handle the result of the SELECT SQL query, especially if the columns are going to be different in each case while using SQL Builder. Is there an example which can help me understand how to write this?
Usually, I use Mapper XML files with ResultMap to retrieve the output of an SQL statement.
I figured out the way to get it to work. I am not sure if it is the correct way.
In the XML I made the following entry
<select id="readSignals" resultType="map">
${query}
</select>
The ${query} is passed from a QueryBuilder class and the resultType is set to "map". This cause myBatis to return a List> where each Map in the list is a row. The String contains the column name and Object contains the data.
I use the following code to convert the List> into JSON.
public static JSONObject convertToJSON(List<Map<String, Object>> queryData) {
JSONObject queryJSONOutput = new JSONObject();
JSONArray outputArray = new JSONArray();
queryData.stream().forEach(d -> {
JSONObject jsonObject = new JSONObject();
for (String key: d.keySet()) {
jsonObject.put(key, d.get(key));
}
outputArray.put(jsonObject);
});
queryJSONOutput.put("data", outputArray);
return queryJSONOutput;
}

How to get the elasticsearch json response using aggregations in spring-data-elasticsearch?

I have the following:
I notice that at the end of running the code, if I print out aggregations.asMap().get('subjects');
I am getting:
org.elasticsearch.search.aggregations.bucket.terms.StringTerms#6cff59fa
Printing out "aggregations" gives me: org.elasticsearch.search.aggregations.InternalAggregations#65cf321d
What I really want is the entire string/json response that is normally returned if you were to curl on elasticsearch to get aggregations. How do I get to the raw response from the aggregation query? Also, is there a way to iterate and print out what's in those "wrapped up" objects?
https://github.com/spring-projects/spring-data-elasticsearch/blob/ab7e870d5f82f6c0de236048bd7001e8e7d2a680/src/test/java/org/springframework/data/elasticsearch/core/aggregation/ElasticsearchTemplateAggregationTests.java
#Test
public void shouldReturnAggregatedResponseForGivenSearchQuery() {
// given
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.withSearchType(COUNT)
.withIndices("articles").withTypes("article")
.addAggregation(terms("subjects").field("subject"))
.build();
// when
Aggregations aggregations = elasticsearchTemplate.query(searchQuery, new ResultsExtractor<Aggregations>() {
#Override
public Aggregations extract(SearchResponse response) {
return response.getAggregations();
}
});
// then
System.out.println(aggregations); // gives me some cryptic InternalAggregations object, how do I get to the raw JSON normally returned by elasticsearch?
System.out.println(aggregations.asMap().get("subjects")); // gives me some StringTerms object I have no idea how to iterate over to get results
}
You cannot get the raw JSON response this way, since Spring Data Elasticsearch will take care of parsing it for you, that's the whole point.
If you need to parse those buckets, you can do it like this easily:
...
StringTerms subjects = aggregations.asMap().get("subjects");
for (Terms.Bucket bucket : subjects.getBuckets()) {
String key = bucket.getKey();
long docCount = bucket.getDocCount();
// do something with the key and the doc count
}
If you really want to see the JSON being returned, what you can do is to re-write the parsed Aggregations object into JSON using serialization, but that won't really be helpful:
InternalAggregations aggregations = ...;
XContentBuilder jsonBuilder = JsonXContent.contentBuilder();
aggregations.toXContent(jsonBuilder, ToXContent.EMPTY_PARAMS);
String rawJson = jsonBuilder.string();
Set Size of EsRequest to Zero
Get Esresponse.toString()
Convert String to Json
Get aggregation field from Json.

Neo4j Cypher results as JSON: Getting JsonMappingException when using JsonHelper.createJsonFrom() but only with certain queries

I'm working on neo4vertx, a module that makes it possible to talk to a Neo4j database using Vert.x. Specifically, I'm working on feature called "query" which allows a Vert.x user
to send a Cypher query as an eventbus message and get a JSON resultset back.
However, I seem to run into an unexpected problem when serializing to JSON using JsonHelper.createJsonFrom() with certain queries.
A quick example (database has stuff in it of course):
// This Fails with JsonMappingException (see below):
String query="MATCH (n) RETURN n";
// This Succeeds:
String query="MATCH (n) RETURN n.something";
//Rest of code:
engine = new ExecutionEngine(graphDatabaseService);
ExecutionResult result;
result = engine.execute(query);
Object object = result.iterator();
String foo = JsonHelper.createJsonFrom(object);
System.out.println("DEBUG (foo): " + foo);
Does this look familiar to anyone? We essentially want to be able to
send any kind of query and either return an empty json string or a json
representation not unlike the result.json that you can retrieve from the
web interface of neo4j!
The Exception:
testQuery(org.openpcf.neo4vertx.neo4j.Neo4jGraphTest) Time elapsed: 2.362 sec <<< ERROR!
org.neo4j.server.rest.domain.JsonBuildRuntimeException: org.codehaus.jackson.map.JsonMappingException: No serializer found for class org.neo4j.kernel.InternalAbstractGraphDatabase$DependencyResolverImpl and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS) ) (through reference chain: scala.collection.convert.MapWrapper["content"]->org.neo4j.kernel.impl.core.NodeProxy["graphDatabase"]->org.neo4j.test.["dependencyResolver"])
...
Although the ExecutionResult is an implementation of Iterator<Map<String,Object>> it is currently just a java wrapper around scala classes, as well as Neo4j classes (like Node, Relationship, Path).
So you have probably to do 2 things:
recursively replace neo4j classes with appropriate maps and lists
probably: recursively replace scala lists and maps with java lists and maps, e.g. new LinkedHashMap(row)
I did some of that some time ago here in my cypher-websocket-experiments
You can serialize the ExecutionResult of Neo4J to JSON this way:
ExecutionResult result ...
CypherResultRepresentation repr = new CypherResultRepresentation(result, false, false);
OutputFormat format = new OutputFormat(new JsonFormat(), null, null);
String json = format.assemble(repr);
The next thing you have to do is to create the appropriate vert.x JsonObject objects.
#Rubin: I'm currently adding this to neo4vertx ;)
By using RESTAPI:
String query = "MATCH (a)-[r]-(b) RETURN a,r,b";
RestAPI restAPI = new RestAPIFacade(URI);
CypherResult result = restAPI.query(query, Collections.<String, Object>emptyMap());
Gson gson = new GsonBuilder().disableHtmlEscaping().create();
String stringResult = gson.toJson(result.asMap());
JSONObject jsonResult = new JSONObject(stringResult);
By using Bolt
String query = "MATCH (a)-[r]-(b) RETURN a,r,b";
String stringConnection = "bolt://" + HOST + ":7687";
Driver driver = GraphDatabase.driver(stringConnection, AuthTokens.basic(USER, PASS));
Session session = driver.session();
StatementResult result = session.run(query);
session.close();
driver.close();
Gson gson = new GsonBuilder().disableHtmlEscaping().create();
while (result.hasNext()) {
Record record = result.next();
stringResult = gson.toJson(record.asMap());
}
JSONObject jsonResult = new JSONObject(stringResult);

How to get column names from HQL query&result to json with Key as columns

How to convert hql query result directly to Json.I tried this one
Query query=session.createQuery(SQLUtilsConstants.fQuery);
query.setParameter(StringConstants.TRPID, trId);
List list=query.list();
gson = new Gson();
String jsonStudents = gson.toJson(list);
System.out.println("jsonStudents = " + jsonStudents);
When i converted.I get a json with list of values without properties.Query result contains data from multiple table .I want to generate result with properties as key and value as output.If query contains customerid and customername then i need a result like this.
[{"customerid" : "abc", "customername" : "rose"}]
But using above code i getting like this..
[{"abc", "rose"}]
How can i do this???
You should create a Map, how ever you want it. then parse the map into json.
Map<String,WhatEver> newMap = new HashMap<String,WhatEver>();
foreach(WhatEver item: list){
//create your map how ever you like it to be
}
String jsonStudents = gson.toJson(newMap);

Categories

Resources