Spring Data Elasticsearch Query Date Format - java
We are using the spring data elasticsearch library to query our elasticsearch server. We are currently using a rest call to get the results and have been successful, but we want to use the library.
The working rest query we are sending resembles
{
"query": {
"bool": {
"must": [
{ "range" : { "startDateTime" : { "from" : "2016-01-31T00:00:00", "to" : "2016-02-01T00:00:00" }}},
{ "match_phrase" : { "keyword" : "task" }}
]
}
}
}
Using the spring data elasicsearch query derivation tool we created the method
findMessagesByKeywordAndStartDateTimeBetween(String keyword, String start, String end);
Which derives to the query
{
"from": 0,
"query": {
"bool": {
"must": [
{"query_string":{"query":"\"tasks\"","fields":["keyword"]}},
{"range":{"startDateTime":{"from":"2016-01-31T00:00:00","to":"2016-02-01T00:00:00","include_lower":true,"include_upper":true}}}
]
}
}
}
I can run this query in a rest client and receive data, however, when the library attempts to query the database, I receive an error
{
"timestamp": 1454360466934,
"status": 500,
"error": "Internal Server Error",
"exception": "org.elasticsearch.action.search.SearchPhaseExecutionException",
"message": "Failed to execute phase [query_fetch], all shards failed; shardFailures {
[██████████████████████][████████████][0]: RemoteTransportException[
[██████-██-███-███████][inet[/██.███.███.███:9300]]
[indices:data/read/search[phase/query+fetch]]]; nested: SearchParseException[
[████████████][0]: from[0],size[10]: Parse Failure [
Failed to parse source [
{\"from\":0,\"size\":10,\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"\"/tasks\"\",\"fields\":[\"method\"]}},{\"range\":{\"startDateTime\":{\"from\":\"2016-01-31T00:00:00.000Z\",\"to\":\"2016-02-01T00:00:00.000Z\",\"include_lower\":true,\"include_upper\":true}}}]}}}
]
]
];
nested: NumberFormatException[For input string: \"2016-01-31T00:00:00.000Z\"];
}",
"path": "/report/tasks"
}
This leads us to believe that the date we are asking for is not in the correct format to reference against database items, but a sample result looks like
{
"_index": "████████████",
"_type": "████████████",
"_id": "████████████",
"_score": 0.000,
"_source": {
"keyword": "tasks",
"endDateTime": "2016-01-15T00:57:31.427Z",
"startDateTime": "2016-01-15T00:57:30.201Z",
"#timestamp": "2016-01-15T00:57:31+00:00",
"responseBody": "{...stuff goes here...}"
}
},...
So you would think that you would be able to query using that format.
We decided to attempt to get all results with the tasks keyword using a new query
findMessagesByKeyword(String keyword);
which derives to
{
"from": 0,
"query": {
"bool": {
"must": [
{"query_string":{"query":"\"tasks\"","fields":["keyword"]}}
]
}
}
}
This returns all the results in a page and after printing the mapped objects startDateTime and responseBody fields to the console
10: [
[Thu Oct 15 18:55:53 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 18:56:38 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 18:56:49 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 18:58:59 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 18:59:16 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 18:59:33 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 18:59:54 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 19:00:02 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 19:00:02 EDT 2015, {...stuff goes here...}]
[Thu Oct 15 19:00:11 EDT 2015, {...stuff goes here...}]
] //These are paged results, there are results as recently as last week, just not in this page
I notice that the date time field is now in a different format, so I use the format
public String DATETIME_FORMAT = "EE MMM dd HH:mm:ss zz yyyy";
instead of
public String DATETIME_FORMAT = "yyyy-MM-dd'T'00:00:00.000'Z'";
and get the error
NumberFormatException[For input string: \"Sun Jan 31 00:00:00 EST 2016\"]
The mapping for the field, if that helps, is
"startDateTime": {
"type": "date",
"format": "dateOptionalTime"
},
We have tried many formats and datatypes. When we changed the format
to
yyyMMddHHmmss
We no longer receive the error, but get no results.
At this point we know we must be doing something wrong, but are not sure where to go. All help is greatly appreciated.
Feb 2, 2016 10:15AM
Thanks to #Richa: After converting the date to long (in milliseconds) the query seems to run, but there are no results returned.
This is run on the default timerange of from yesterday until today, and on a manual timerange of about 10 days which I know have about 300 records.
I am also able to verify using the current rest implementation that there is data, and I am able to use a rest client to triple check, but no data for spring data impl.
Thoughts?
I got this error in my project too.Got it resolved using Long datatype. The dynamic finder provided by spring Data which you are using is taking date argument in String. Convert date to milliseconds and pass date in Long. Use method as:
findMessagesByKeywordAndStartDateTimeBetween(String keyword, Long start, Long end);
Hope this helps.
Related
Spring - Treat incoming date as UTC
I have application written in spring that allows user to filter data by date. Filters model: data class FilterCommand( #field:DateTimeFormat(pattern = "dd/MM/yyyy") val dateFrom: Date?, #field:DateTimeFormat(pattern = "dd/MM/yyyy") val dateTo: Date?, val author: String?, val state: String? ) Everything works fine, except when user chooses 07/06/2019 in UI, I get exactly same value in Spring in current timezone. Is it possible to tell spring to treat this incoming data as UTC? Or to convert it somehow on model level? Date I receive: Fri Jun 07 00:00:00 CEST 2019 Date I would like to get: Fri Jun 07 00:00:00 UTC 2019
You can configure the timezone directly in the JVM, so if you set it to UTC, all dates will be on UTC.
IBM Watson - Natural Language Classifier service (Java SDK) - deleteClassifier() method does not delete the classifier
I'm trying to invoke the deleteClassifier() method in the Natural Language Classifier service in IBM Watson platform, the operation completes but does not delete the classifier. Here's my code and the output:
Code:
NaturalLanguageClassifier service = new NaturalLanguageClassifier();
service.setUsernameAndPassword("xxxxxxx", "yyyyyyy");
System.out.println("Before deleting: ");
Classifier classifier;
classifier = service.getClassifier("90e7acx197-nlc-38920").execute();
System.out.println(classifier);
service.deleteClassifier("90e7acx197-nlc-38920");
System.out.println("After deleting: ");
classifier = service.getClassifier("90e7acx197-nlc-38920").execute();
System.out.println(classifier);
Output:
Before deleting:
Apr 18, 2017 7:16:08 PM okhttp3.internal.platform.Platform log
INFO: --> GET https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/90e7acx197-nlc-38920 http/1.1
Apr 18, 2017 7:16:08 PM okhttp3.internal.platform.Platform log
INFO: <-- 200 OK https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/90e7acx197-nlc-38920 (167ms, unknown-length body)
{
"classifier_id": "90e7acx197-nlc-38920",
"language": "en",
"name": "TutorialClassifier",
"status": "Available",
"created": "2017-04-18T01:26:14.630",
"status_description": "The classifier instance is now available and is ready to take classifier requests.",
"url": "https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/90e7acx197-nlc-38920"
}
After deleting:
Apr 18, 2017 7:16:08 PM okhttp3.internal.platform.Platform log
INFO: --> GET https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/90e7acx197-nlc-38920 http/1.1
Apr 18, 2017 7:16:09 PM okhttp3.internal.platform.Platform log
INFO: <-- 200 OK https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/90e7acx197-nlc-38920 (142ms, unknown-length body)
{
"classifier_id": "90e7acx197-nlc-38920",
"language": "en",
"name": "TutorialClassifier",
"status": "Available",
"created": "2017-04-18T01:26:14.630",
"status_description": "The classifier instance is now available and is ready to take classifier requests.",
"url": "https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/90e7acx197-nlc-38920"
}
Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M
Am I doing something wrong here?
The deleteClassifier needs to be invoked using .execute()
Add .execute() to
service.deleteClassifier("90e7acx197-nlc-38920");
The method below will delete your classifier
service.deleteClassifier("90e7acx197-nlc-38920").execute();
Can WireMock play back requests from multiple domains?
I am building a Dockerised record-playback system to help me record websites, so I can design scrapers aginst a local version rather than the real thing. This means that I do not swamp a website with automated requests, and has the added advantage that I do not need to be connected to the web to work.
I have used the Java-based WireMock internally, which records from a queue of site scrapes using Wget. I am using the WireMock API to read various pieces information from the mappings it records.
However, I have spotted from a mapping response that domain information does not seem to be recorded (except where it is in response headers by accident). See the following response from __admin/mappings:
{
"result": {
"ok": true,
"list": [
{
"id": "794d609f-99b9-376d-b6b8-04dab161c023",
"uuid": "794d609f-99b9-376d-b6b8-04dab161c023",
"request": {
"url": "/robots.txt",
"method": "GET"
},
"response": {
"status": 404,
"bodyFileName": "body-robots.txt-j9qqJ.txt",
"headers": {
"Server": "nginx/1.0.15",
"Date": "Wed, 04 Jan 2017 21:04:40 GMT",
"Content-Type": "text/html",
"Connection": "keep-alive"
}
}
},
{
"id": "e246fac2-f9ad-3799-b7b7-066941408b8b",
"uuid": "e246fac2-f9ad-3799-b7b7-066941408b8b",
"request": {
"url": "/about/careers/",
"method": "GET"
},
"response": {
"status": 200,
"bodyFileName": "body-about-careers-GhVqy.txt",
"headers": {
"Server": "nginx/1.0.15",
"Date": "Wed, 04 Jan 2017 21:04:35 GMT",
"Content-Type": "text/html",
"Last-Modified": "Wed, 04 Jan 2017 12:52:12 GMT",
"Connection": "keep-alive",
"X-CACHE-URI": "/about/careers/",
"Accept-Ranges": "bytes"
}
}
},
{
"id": "def378f5-a93c-333e-9663-edcd30c936d7",
"uuid": "def378f5-a93c-333e-9663-edcd30c936d7",
"request": {
"url": "/about/careers/feed/",
"method": "GET"
},
"response": {
"status": 200,
"bodyFileName": "body-careers-feed-Fd2fO.xml",
"headers": {
"Server": "nginx/1.0.15",
"Date": "Wed, 04 Jan 2017 21:04:45 GMT",
"Content-Type": "application/rss+xml; charset=UTF-8",
"Transfer-Encoding": "chunked",
"Connection": "keep-alive",
"X-Powered-By": "PHP/5.3.3",
"Vary": "Cookie",
"X-Pingback": "http://www.example.com/xmlrpc.php",
"Last-Modified": "Thu, 06 Jun 2013 14:01:52 GMT",
"ETag": "\"765fc03186b121a764133349f8b716df\"",
"X-Robots-Tag": "noindex, follow",
"Link": "<http://www.example.com/?p=2680>; rel=shortlink",
"X-CACHE-URI": "null cache"
}
}
},
{
"id": "616ca6d7-6e57-4c10-8b57-f6f3dabc0930",
"uuid": "616ca6d7-6e57-4c10-8b57-f6f3dabc0930",
"request": {
"method": "ANY"
},
"response": {
"status": 200,
"proxyBaseUrl": "http://www.example.com"
},
"priority": 10
}
]
}
}
The only clear recording of a URL is in the final entry against proxyBaseUrl, and given that I had to specify a URL in the console call I am now worried that if I record against a different domain, the domain that each one is from will be lost.
That would mean that in playback mode, WireMock would only be able to play back from one domain, and I'd have to restart it and point it to another cache in order to play back different sites. This is not workable for my use case, so is there a way around this problem?
(I have done a little work with Mountebank, and would be willing to switch to it, though I find WireMock generally easier to use. My limited understanding of Mountebank is that it suffers from the same single-domain problem, though I am happy to be corrected on that. I'd be happy to swap to any robust open-source API-driven recorder HTTP proxy, if dropping WireMock is the only way forward).
It's possible to serve WireMock stubs for multiple domains by adding a Host header criterion in your requests. Assuming your DNS/host file maps all the relevant domains to your WireMock server's IP, then this will cause it to behave like virtual hosting on an ordinary web server. The main issue is that the recorder won't add the host header to your mappings so you'd need to do this yourself afterwards, or hack the recorder to do it on the fly. I've been considering adding better support for this, so watch this space. I'd also suggest checking out Hoverfly, which seems to solve this problem pretty well already.
How to get conference call sid from partcipant call sid using rest api in Twilio
I am using Twilio Rest API to get detail about participant using the participant call sid by sending request to/2010-04-01/Accounts/{AccountSid}/Calls/{ParticipantCallSid}.
In response I can't find any parameter for getting conference call sid of participant. Can anyone suggest me some way to figure it out from participant call sid?
The Participants list resource is a subresource of a Conference instance resource.
Resource URI:
/2010-04-01/Accounts/{AccountSid}/Conferences/{ConferenceSid}/Participants/{CallSid}
You will find the first example here returning a representation of the participant with both a call_sid and conference_sid.
{
"account_sid": "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"call_sid": "CA386025c9bf5d6052a1d1ea42b4d16662",
"conference_sid": "CFbbe46ff1274e283f7e3ac1df0072ab39",
"date_created": "Wed, 18 Aug 2010 20:20:10 +0000",
"date_updated": "Wed, 18 Aug 2010 20:20:10 +0000",
"end_conference_on_exit": true,
"muted": false,
"start_conference_on_enter": true,
"uri": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Conferences/CFbbe46ff1274e283f7e3ac1df0072ab39/Participants/CA386025c9bf5d6052a1d1ea42b4d16662.json"
}
how to get an pinterest access token
i am using rest api to get an access toke of my user using following link
https://api.pinterest.com/v1/oauth/token?grant_type=authorization_code&client_id=my id&client_secret=my secret&code="+oauthVerifier+
i am getting the response as
{"status": "failure", "code": 3, "host": "devplatform-devapi-prod-4a8937b3", "generated_at": "Tue, 28 Jun 2016 06:06:56 +0000", "message": "Authorization failed.", "data": null}
what shod i do to get my users access token