I am unable to convert a text field ts1 in my ES index (sample value 2017-09-30 04:53:39.412496Z) to timestamp by using scripted fields in Kibana. The SimpleDateFormat expression works in Java, just not in Elasticsearch. This mentions a similar problem but the solution doesn't work for me.
Query
GET <index_name>/_search
{
"query" : {
"match_all": {}
},
"script_fields" : {
"test1" : {
"script" : {
"lang": "painless",
"source": """new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'").parse(doc['ts1'].value).getTime()"""
}
}
}
}
Output
{
"took": 76,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 4,
"skipped": 0,
"failed": 1,
"failures": [
{
"shard": 3,
"index": "<index_name>",
"node": "-TjsxK4-QsqgLQRQ-DSKGQ",
"reason": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"java.text.DateFormat.parse(DateFormat.java:366)",
"""new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'").parse(doc['ts1'].value).getTime()""",
" ^---- HERE"
],
"script": """new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'").parse(doc['ts1'].value).getTime()""",
"lang": "painless",
"caused_by": {
"type": "parse_exception",
"reason": """Unparseable date: "04""""
}
}
}
]
},
"hits": {
"total": 1,
"max_score": 1,
"hits": []
}
}
Java Code that works -
import java.text.SimpleDateFormat;
public class Main{
public static void main(String []args){
String text2 = "2017-09-30 04:53:39.123123Z";
SimpleDateFormat s2 = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSSSS'Z'");
try {
System.out.println(s2.parse(text2).getTime());
} catch (Exception e) {
System.out.println(e);
}
}
}
Related
I have a response which contains values which I need.
private void getDataFromDistanceMatrixToSaveAndAddToList(
GRATestDataImport place, String response) {
JSONObject responseAsJson = new JSONObject(response).getJSONArray("rows")
.getJSONObject(0)
.getJSONArray("elements")
.getJSONObject(0);
}
'response' in params includes:
{
"destination": [
"54.375,18.59"
],
"origin": [
"54.001,21.721"
],
"rows": [
{
"elements": [
{
"distance": {
"text": "304.4 km",
"value": 304403
},
"duration": {
"text": "3 h 53 min",
"value": 14003
},
"status": "OK"
}
]
}
],
"status": "OK"
}
value: 'responseAsJson' in my method is:
{
"duration": {
"text": "3 h 53 min",
"value": 14003
},
"distance": {
"text": "304.4 km",
"value": 304403
},
"status": "OK"
}
How can I get value from the duration and the distance ?
you'll just have to navigate along
int duration = responseAsJson.getJSONObject("duration").getInt("value");
int distance = responseAsJson.getJSONObject("distance").getInt("value");
public class YourResponse
{
public string duration { get; set; }
public string distance { get; set; }
}
YourResponse response = JsonSerializer.Deserialize<YourResponse>(result);
Elasticsearch Java client SearchResponse is unable to parse aggregation results. I've come across articles online which suggest adding aggregation types prefixed to the keys. I've added what I thought applies in my use case, such as "sterms# and sum#" but I am unable to figure out which type applies to the main filter (key:'matched' in my case). I am expecting the buckets object to be populated but it is currently being returned as an empty array despite response from elasticsearch containing aggregations.
Note: This is to be able to unit test.
Json response:
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 75,
"successful": 75,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 776329,
"max_score": 0,
"hits": []
},
"aggregations": {
"sterms#matched": {
"doc_count": 15,
"sterms#id": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "mykey",
"doc_count": 15,
"sterms#manufacturerName": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "subkey",
"doc_count": 15
}
]
},
"sum#totalCars": {
"value": 214244
},
"sum#totalTrucks": {
"value": 155738
}
}
]
}
}
}
}
Parser in Java
public static SearchResponse getSearchResponseFromJson(final String jsonFileName) {
try {
String jsonResponse = IOUtils.toString(new FileInputStream(jsonFileName));
NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
XContentParser parser = JsonXContent.jsonXContent.createParser(registry, jsonResponse);
return SearchResponse.fromXContent(parser);
} catch (IOException e) {
throw new RuntimeException("Failed to get resource: " + jsonFileName, e);
} catch (Exception e) {
throw new RuntimeException(("exception " + e));
}
}
private static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
map.put(TopHitsAggregationBuilder.NAME, (parser, content) -> ParsedTopHits.fromXContent(parser, (String) content));
map.put(StringTerms.NAME, (parser, content) -> ParsedStringTerms.fromXContent(parser, (String) content));
map.put(SumAggregationBuilder.NAME, (parser, content) -> ParsedSum.fromXContent(parser, (String) content));
//map.put(ScriptedMetricAggregationBuilder.NAME, (parser, content) -> ParsedScriptedMetric.fromXContent(parser, (String) content));
List<NamedXContentRegistry.Entry> entries = map.entrySet()
.stream()
.map(entry -> new NamedXContentRegistry.Entry(Aggregation.class,
new ParseField(entry.getKey()),
entry.getValue()))
.collect(Collectors.toList());
return entries;
}
Parsed response with empty buckets (expectation is for the buckets to have aggregations totalCars and totalTrucks)
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 75,
"successful": 75,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 776329,
"max_score": 0,
"hits": []
},
"aggregations": {
"sterms#matched": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": []
}
}
}
I am not sure if sterms is the right type for the filter
The answer for what type needs to be used is 'filter' (filter#matched). And Java class counterpart is FilterAggregationBuilder
response:
[
{
"id": "e9299032e8a34d168def176af7d62da3",
"createdAt": "Nov 8, 2017 9:46:40 AM",
"model": {
"id": "eeed0b6733a644cea07cf4c60f87ebb7",
"name": "color",
"app_id": "main",
"created_at": "May 11, 2016 11:35:45 PM",
"model_version": {}
},
"input": {
"id": "df6eae07cd86483f811c5a2202e782eb",
"data": {
"concepts": [],
"metadata": {},
"image": {
"url": "http://www.sachinmittal.com/wp-content/uploads/2017/04/47559184-image.jpg"
}
}
},
"data": [
{
"hex": "#f59b2d",
"webSafeHex": "#ffa500",
"webSafeColorName": "Orange",
"value": 0.0605
},
{
"hex": "#3f1303",
"webSafeHex": "#000000",
"webSafeColorName": "Black",
"value": 0.2085
},
{
"hex": "#a33303",
"webSafeHex": "#8b0000",
"webSafeColorName": "DarkRed",
"value": 0.3815
},
{
"hex": "#000000",
"webSafeHex": "#000000",
"webSafeColorName": "Black",
"value": 0.34275
},
{
"hex": "#f7ce93",
"webSafeHex": "#ffdead",
"webSafeColorName": "NavajoWhite",
"value": 0.00675
}
],
"status": {}
}
]
need to parse this reponse in json. Please help me out.
You can try something like this..
try{
JSONArray array= new JSONArray(Yourresponse);
for(int i=0; i<=array.length();i++){
JSONObject jsonObject=array.getJSONObject(i);
String id= jsonObject.getString("id");
String created_at= jsonObject.getString("createdAt");
String model_id = jsonObject.getJSONObject("model").getString("id");
String app_id=jsonObject.getJSONObject("model").getString("app_id");
//So On... Depends on your requirements. It's just an idea!
}
}
catch (Exception e){
e.printStackTrace();
}
I am a novice in Java and I am looking for a way to flatten json documents.
I have tried Object mapper but without success and I have also tried to do with json node but still get no success .
I found this link but the results is not what I need :https://github.com/wnameless/json-flattener
I have also been helped before but the example was too specific and I cannot do the same things because the documents is too long this is why I am looking for a way to make it generic: Flatten json documents in Java
I need to transform "any" json documents like in the example below :
Here is an example of my documents
Documents recieved:
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 10,
"max_score": 0,
"hits": []
},
"aggregations": {
"groupe": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "a",
"doc_count": 1,
"date": {
"buckets": [
{
"key_as_string": "2017-05-03T00:00:00.000Z",
"key": 1493769600000,
"doc_count": 1,
"value": {
"value": 1
}
},
{
"key_as_string": "2017-05-03T01:00:00.000Z",
"key": 1493776800000,
"doc_count": 1,
"value": {
"value": 3
}
}
]
}
},
{
"key": "b",
"doc_count": 4,
"date": {
"buckets": [
{
"key_as_string": "2017-05-03T00:00:00.000Z",
"key": 1493769600000,
"doc_count": 1,
"value": {
"value": 4
}
},
{
"key_as_string": "2017-05-03T01:00:00.000Z",
"key": 1493773200000,
"doc_count": 1,
"value": {
"value": 3
}
}
]
}
}
]
}
}
}
Document Transformed:
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 10,
"max_score": 0,
"hits": []
},
"aggregations": {
"groupe": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "a",
"doc_count": 1,
"date": {
"buckets": [
{
"key_as_string": "2017-05-03T00:00:00.000Z",
"key": 1493769600000,
"doc_count": 1,
"value": {
"value": 1
}
}
]
}
},
{
"key": "a",
"doc_count": 1,
"date": {
"buckets": [
{
"key_as_string": "2017-05-03T02:00:00.000Z",
"key": 1493776800000,
"doc_count": 1,
"value": {
"value": 3
}
}
]
}
},
{
"key": "b",
"doc_count": 1,
"date": {
"buckets": [
{
"key_as_string": "2017-05-03T02:00:00.000Z",
"key": 1493776800000,
"doc_count": 1,
"value": {
"value": 4
}
}
]
}
},
"key": "b",
"doc_count": 1,
"date": {
"buckets": [
{
"key_as_string": "2017-05-03T02:00:00.000Z",
"key": 1493776800000,
"doc_count": 1,
"value": {
"value": 4
}
}
]
}
}
]
}
}
}
I am going to get facebook read_books
The file is in this format:
{
"data": [
{
"id": "270170479804513",
"from": {
"name": "L I",
"id": "1000022"
},
"start_time": "2014-01-22T09:31:00+0000",
"publish_time": "2014-01-22T09:31:00+0000",
"application": {
"name": "Books",
"id": "174275722710475"
},
"data": {
"book": {
"id": "133556360140246",
"url": "https://www.facebook.com/pages/Pride-and-Prejudice/133556360140246",
"type": "books.book",
"title": "Pride and Prejudice"
}
},
"type": "books.reads",
"no_feed_story": false,
"likes": {
"count": 0,
"can_like": true,
"user_likes": false
},
"comments": {
"count": 0,
"can_comment": true,
"comment_order": "chronological"
}
},
{
"id": "270170328",
"from": {
"name": "h",
"id": "100004346894022"
},
"start_time": "2014-01-22T09:29:42+0000",
"publish_time": "2014-01-22T09:29:42+0000",
"application": {
"name": "Books",
"id": "174275722710475"
},
"data": {
"book": {
"id": "104081659627680",
"url": "https://www.facebook.com/pages/Gulistan-of-Sadi/104081659627680",
"type": "books.book",
"title": "Gulistan of Sa'di"
}
},
"type": "books.reads",
"no_feed_story": false,
"likes": {
"count": 0,
"can_like": true,
"user_likes": false
},
"comments": {
"count": 0,
"can_comment": true,
"comment_order": "chronological"
}
}
],
I need books titles and their URL. I run the below code but I get Exception in thread "main" java.lang.ClassCastException: org.json.simple.JSONObject cannot be cast to java.lang.String
while ((inputLine = in.readLine()) != null)
{
s = s + inputLine + "\r\n";
if (s == null) {
break;
}
t = t + inputLine + "\r\n";
}
in.close();
t = t.substring(0, t.length() - 2);
System.out.println(t);
Object dataObj =JSONValue.parse(t);
System.out.println(dataObj);
JSONObject dataJson = (JSONObject) dataObj;
JSONArray data = (JSONArray) dataJson.get("data");
for (Object o: data)
{
JSONObject indata= (JSONObject) o;
Object indatafirst=(JSONObjec`enter code here`t).get("0");
String inndata=(String) indatafirst.get("data");
System.out.println("inndata"+inndata);
}}
but it is not true
The problem is with the following line:
String inndata=(String) indatafirst.get("data");
The data field in the JSON is not a String, it's a nested JSON object.
"data": {
"book": {
"id": "104081659627680",
"url": "https://www.facebook.com/pages/Gulistan-of-Sadi/104081659627680",
"type": "books.book",
"title": "Gulistan of Sa'di"
}
}
This explains your ClassCastException.
Instead you should do something like:
JSONObject data = (JSONObject) indatafirst.get("data");
JSONObject book = (JSONObject) data.get("book");
String bookTitle = book.get("title");