I am new to Elasticsearch. I read Elasticsearch's Java client APIs and am able to build query and send it to the Elasticsearch server via the transport client.
Because my query is quite complex with multi-level filters and I notice that it is cumbersome to build a query via the Java client. I feel that it is much simpler to build a JSON query string and then send it over to the Elasticsearch server via a Java client.
Is this something Elasticsearch offers?
I do like what Elasticsearch Java API can do after receiving results such as scrolling over the results. I want to keep these features.
Thanks for any input and links!
Regards.
Did further research on Elasticsearch API and found out that Elasticsearch does offer this capability. Here is how:
SearchResponse scrollResp = client.prepareSearch("my-index")
.setTypes("my-type")
.setSearchType(SearchType.SCAN)
.setQuery(query) // **<-- Query string in JSON format**
.execute().actionGet();
You can no longer pass in string to the .setQuery function, however you can use a WrapperQueryBuilder like this:
WrapperQueryBuilder builder = QueryBuilders.wrapperQuery(searchQuery);
SearchRequestBuilder sr = client.prepareSearch().setIndices(index).setTypes(mapping).setQuery(builder);
I'd recommend using the Java API, it is very good once you get used to it and in most cases it is less cumbersome. If you look through the Elasticsearch source code you will see that the Java API Builds the JSON under the hood. Here is an example from the MatchAllQueryBuilder:
#Override
public void doXContent(XContentBuilder builder, Params params) throws IOException {
builder.startObject(MatchAllQueryParser.NAME);
if (boost != -1) {
builder.field("boost", boost);
}
if (normsField != null) {
builder.field("norms_field", normsField);
}
builder.endObject();
}
ElasticSearch has built in capabilities to do exactly what you need, in an organized manner.
To answer your question, please see this link (the material is gone on elastic's site, so it might no longer work):
https://web.archive.org/web/20150906215934/https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/search.html
All you have to do is build a simple file which contains your search template i.e complex search query.
It can be a simple json file, or a text file.
Now you simply pass in your parameters, through your java code.
See the example in the link, it makes things amply clear.
Bhargav.
Related
Good day!
I'm using cloud dlp api to inspect bigquery views by converting chunks of the data into ContentItem and passing it to the inspect request. However, I am having trouble converting the findings and saving it to a bigquery table. Before, I used an airflow DLP operator for this and it is being done automatically by passing output storage config in an InspectConfig. However, that approach won't be applicable anymore as I'm calling the DLP api per chunks of data using apache beam in java.
I saw that the finding object has a writeTo() method but I'm not sure how to use it and how to save the findings with correct types into a bigquery table. can you help me with this? I'm currently stuck. thank you!
what I want to do is something like this
for (Finding res : result.getFindingsList()){
TableRow bqRow = new TableRow();
Object data = res.getLocation();
bqRow.set("field", data);
context.output(bqRow);
}
but this approach wouldn't save it in bigquery with correct types, especially for getLocation as it returns something like a protobuf message type.
I was trying to see if I can use the writeTo() method but I'm not sure how to use it. Thank you in advance for the help!
for (Finding res : result.getFindingsList()){
res.writeTo(...)
...
context.output(...);
}
If you use HybridInspect we'll store the findings for you to BigQuery.
https://cloud.google.com/dlp/docs/how-to-hybrid-jobs
If you do it yourself you will need to convert to a native BQ format like json
Load protobuf data to bigquery
A new Java API Client was released in the 7.16 version of ES and the Java Rest Client was deprecated. There was the ability to convert a query to JSON in the deprecated client. It was convenient for debugging/tuning/profiling of the query in Kibana.
Unfortunately, I don't see any mention of this in the new client documentation. So my question is:
Is it possible to get a JSON representation of the query that was constructed via the Java API Client? It can be either some utility class or log config of the client that prints outgoing requests.
It looks like this feature will be directly supported in an upcoming version: https://discuss.elastic.co/t/elaticsearch-java-client-output-dsl/300952
In the meantime, there is a workaround given in the above discussion, but it does not quite work. Here is a snippet that works for me:
StringWriter writer = new StringWriter();
JsonGenerator generator = JacksonJsonProvider.provider().createGenerator(writer);
request.serialize(generator, new JacksonJsonpMapper());
generator.flush();
return writer.toString();
The only way I found so far is to put a conditional breakpoint in
co/elastic/clients/transport/rest_client/RestClientTransport.java:215 from elasticsearch-java-7.16.2.jar and check the value of baos variable.
As a temporary method waiting for pull request mentionned below, i just used
// TODO delete when https://github.com/elastic/elasticsearch-java/pull/213
public static String serializeToString(Query q) {
return q._get().toString();
}
It does not print anything pretty but it covers my use case of string comparison for tests.
I'm using the Twitter4j library to develop a proyect that works with Twitter, one of the things what I need is to get the Direct messages, I'm using the following code:
try{
List<DirectMessage> loStatusList = loTwitter.getDirectMessages();
for (DirectMessage loStatus : loStatusList) {
System.out.println(loStatus.getId() + ",#" + loStatus.getSenderScreenName() + "," + loStatus.getText() + "|");
}
}
catch(Exception e)
It works fine, but what the code returns is a list of the most recent messages in general. What I want is to get those direct messages using some kind of filter that allows finding them by a user that I indicate.
For example, I need to see the DM only from user #TwitterUser.
Is this posible with this library?
All kinds of suggestions are accepted, even if I should use another library I would be grateful if you let me know.
It looks like the actual Twitter API doesn't support a direct filter on that API, by username anyway. (See Twitter API doc: GET direct_messages.)
Which means, you'd have to make multiple calls to the API with pagination enabled, and cache the responses into a list.
Here is an example of pagination wtih Twitter4J getDirectMessages().
In that example, use the existing:
List<DirectMessage> messages;
But inside the loop, do:
messages.addAll(twitter.getDirectMessages(paging));
Note: you only would have to do this once. And in fact, you should persist these to a durable local cache like Redis or something. Because once you have the last message id, you can ask the Twitter API to only return "messages since id" with the since_id param.
Anyway, then on the client side you'd just do your filtering with the usual means in Java. For example:
// Joe is on twitter as #joe
private static final String AT_JOE = "Joe";
// Java 8 Lambda to filter by screen name
List<DirectMessage> messagesFromJoe = messages.stream()
.filter(message -> message.getSenderScreenName().equals(AT_JOE))
.collect(Collectors.toList());
Above, getSenderScreenName() was discovered by reading the Twitter4J API doc for DirectMessage.
hi i want to get all issues stored in jira from java using jql or any othere way.
i try to use this code:
for(String name:getProjectsNames()){
String jqlRequest = "project = \""+name+"\"";
SearchResult result = restClient.getSearchClient().searchJql(
jqlRequest, 10000,0, pm);
final Iterable<BasicIssue> issues = result.getIssues();
for (BasicIssue is : issues) {
Issue issue = restClient.getIssueClient().getIssue(is.getKey(), pm);
...........
}
it give me the result but it take a very long time.
is there a query or a rest API URL or any other way that give me all issues?
please help me
The JIRA REST API will give you all the info from each issue at a rate of a few issues/second. The Inquisitor add-on at https://marketplace.atlassian.com/plugins/com.citrix.jira.inquisitor will give you thousands of issues per second but only the standard JIRA fields.
There is one other way. There is one table in JIRA database named "dbo.jiraissue". If you have access to that database then you can fetch all the ids of all issues. After fetching this data you can send this REST request "**localhost/rest/api/2/issue/issue_id" and get JSON response. Of course you have to write some code for this but this is one way I know to get all issues.
I'm very new in using web services. Appreciate if anyone can help me on this.
In my PHP codes, I'm trying to use the SOAP web services from another server (JIRA, java). The JIRA SOAP API is shown here.
$jirasoap = new SoapClient($jiraserver['url']);
$token = $jirasoap->login($jiraserver['username'], $jiraserver['password']);
$remoteissue = $jirasoap->getIssue($token, "issuekey");
I found that my codes have no problem to call the functions listed on that page. However, I don't know how to use the objects returned by the API calls.
My question are:
In my PHP codes, how can I use the methods in the Java class objects returned by SOAP API calls?
For example, the function $remoteissue = $jirasoap->getIssue($a, $b) will return a RemoteIssue. Based on this (http://docs.atlassian.com/rpc-jira-plugin/latest/com/atlassian/jira/rpc/soap/beans/RemoteIssue.html), there are methods like getSummary, getKey, etc. How can I use these functions in my codes?
Based on some PHP examples I found from the internet, it seems that everyone is using something like this:
$remoteissue = $jirasoap->getIssue($token, "issuekey");
$key = $remoteissue->key;
They are not using the object's methods.
Refer to this example, it seems that someone is able to do this in other languages. Can it be done in PHP too?
The problem I'm facing is that, I am trying to get the ID of an Attachment. However, it seems that we can't get the Attachment ID using this method: $attachmentid = $remoteattachment->id;. I am trying to use the $remoteattachment->getId() method.
In PHP codes, after we made a SOAP API call and received the returned objects, how do we know what data fields are available in that object?
For example,
$remoteissue = $jirasoap->getIssue($token, "issuekey");
$summary = $remoteissue->summary;
How do we know ->summary is available in $remoteissue?
When i refer to this document (http://docs.atlassian.com/rpc-jira-plugin/latest/com/atlassian/jira/rpc/soap/beans/RemoteIssue.html), I don't see it mention any data fields in RemoteIssue. How do we know we can get key, summary, etc, from this object? How do we know it is ->summary, not ->getsummary? We need to use a web browser to open the WSDL URL?
Thanks.
This question is over one year old, but to share knowledge and provide an answer to people who have this same question and found this page, here are my findings.
The document mentioned in the question is an overview of the JiraSoapService interface. This is a good reference for what functions can be called with which arguments and what they return.
If you use Java for your Jira SoapClient the returned objects are implemented, but if you use PHP, the returned objects aren't of the type stated in this documentation and do not have any of the methods mentioned. The returned objects are instances of the internal PHP class stdClass, which is a placeholder for undefined objects. The best way to know what is returned is to use var_dump() on the objects returned from the SoapCalls.
$jirasoap = new SoapClient($jiraserver['url']);
$token = $jirasoap->login($jiraserver['username'], $jiraserver['password']);
$remoteissue = $jirasoap->getIssue($token, "PROJ-1");
var_dump($remoteissue);
/* -- You will get something like this ---
object(stdClass)#2 (21) {
["id"]=> string(3) "100"
["affectsVersions"]=> array(0) { }
["assignee"]=> string(4) "user"
...
["created"]=> string(24) "2012-12-13T09:27:49.934Z"
...
["description"]=> string(17) "issue description"
....
["key"]=> string(6) "PROJ-1"
["priority"]=> string(1) "3"
["project"]=> string(4) "PROJ"
["reporter"]=> string(4) "user"
["resolution"]=> NULL
["status"]=> string(1) "1"
["summary"]=> string(15) "Project issue 1"
["type"]=> string(1) "3"
["updated"]=> string(24) "2013-01-21T16:11:43.073Z"
["votes"]=> int(0)
}
*/
// You can access data like this:
$jiraKey = $remoteissue->key;
$jiraProject = $remoteissue->project;
The document you referred to in #2 is to a Java implementation and really doesn't give you any help with PHP. If they do not publish a public API for their service (which would be unusual), then using the WSDL as a reference will let you know what objects and methods are accepted by the service and you can plan your method calls accordingly.
The technique you used to call getIssue(...) seems fine, although you should consider using try...catch in case of a SoapException.
I have used Jira SOAP in .NET project and IntelliSense hinted me what fields are available for returned object.
You can use something like VS.Php for Visual Studio or Php for Visual Studio if you are using Visual Studio.
Or you can choose one of the IDEs from here with support of IntelliSense.