Develop Reference

Latency in retrieving Elastic search response to Java

I am searching my Elastic index from my Java backend using Elastic's high level REST client for JAVA. I notice that it takes 700 to 800 milliseconds to receive the response from Elastic.
I checked the actual query time in Elastic and it is only 7 milliseconds.
I have built filters and aggregations into my query and also am returning many fields.
However, if I remove all filters and aggregations and limit the result set to a single document and only return a single field, the time it takes my Java code to receive the response from Elastic is still > 700ms. Why might this be? My server code is running in California. My Elastic index is served in North Virginia. Perhaps this explains the latency? What else could be the cause?
This is a multisearch containing two search queries.

Activiti: Rest Api paging

I have 500 process instances in activiti. I am retrieving them by using REST Api pagination
/runtime/tasks?processDefinitionId=test:1:1212&includeProcessVariables=true&start=0&size=10
It works until start=125&size=10. However I use high value for start=105&size=10, return nothing. I remove includeProcessVariables=true from query param, pagination works again.
This question is also asked in alfresco community, no answers reply.

How to query + Reindex in AWS hosted Elasticsearch outside of Kibana

I have a problem in which I need to query for a subset of records on a large index containing a high volume of records, whilst running a Painless script with the search query to augment the result. The (much smaller) result is to be saved in a secondary index for later use. In a different SO question: Reindex part of Elasticsearch index onto new index via Jest, I mentioned this is possible through the Kibana interface, but there does not seem to be a Java library that can accomplish what I need. Has anyone ever accomplished a query within a _reindex operation outside of Kibana? I am leaning toward using the URLConnection family in Java, but am looking for suggestions and advice at this point.

Query past the 500 limit in Gerrit REST API

I'm trying to get 2000 change results from a specific branch with a query request using Gerrit REST API in Java. The problem is that I'm only getting 500 results no matter what I add to the query search.
I have tried the options listed here but I'm not getting the 2000 results that I need. I also read that an admin can increase this limit but would prefer a method that doesn't require this detour.
So what I'm wondering is:
Is it possible to increase the limit without the need to contact the admin?
If not. Is it possible to continue/repeat the query in order to get the remaining 1500 results that I want, using a loop that performs the query on the following 500 results from the previous query until I finally get 2000 results in total?

When using the list changes REST API, the results are returned as a list of ChangeInfo Elements. If there are more results than were returned, the last entry in that list will have a _more_changes field with value true. You can then query again and set the start option to skip over the ones that you've already received.

I want to add a minor workaround to David's great answer.
If you want to crawl Gerrit instances hosted on Google servers (such as Android, Chromium, Golang), you will notice that they block queries with more than 10000 results. You can check this e.g. with
curl "https://android-review.googlesource.com/changes/?q=status:closed&S=10000"
I solved the problem in such a way, that I split up these list of changes with before: and until: in a query string, for example lie
_url_/changes/?q=after:{2018-01-01 00:00:00.000} AND before:{2018-01-01 00:59:99.999}
_url_/changes/?q=after:{2018-01-01 01:00:00.000} AND before:{2018-01-01 01:59:99.999}
_url_/changes/?q=after:{2018-01-01 02:00:00.000} AND before:{2018-01-01 02:59:99.999}
and so on. I think you get the idea. ;-) Please notice, that both limits (before: and after:) are inclusive! For each day I use the pagination described by David.
A nice side effect is, that you can track the progress of the crawling.
I wrote a small Python tool named "Gerry" to crawl open source instances. Feel free to use, adopt it and send me pull requests!

I almost had the same problem. But there is no way as you mentioned you don't want admin to increase the query limit and also you don't want to fire the rest query in a loop with the counter. I will suggest you to follow the second approach firing the query in a loop with a counter set. That's the way I have implemented the rest client in Java.

How to query using EasticSearch using java Client without worrying about client java heap memory

I am new to elastic search, I was reading that we can use elasticsearch to query using its rest API calls.
I was reading the following link :
http://blogs.justenougharchitecture.com/using-jest-as-a-rest-based-java-client-with-elasticsearch/
Is this the right way to do it??
Also, I donot want to put a limit to the number of results that my search will return(it can return millions of records).
So just how ResultSet in java works, where the table might have millions of row, but we can iterate one row at a time and just process it, and not storing it in my java heap anywhere), hence not worrying about the java heap space,.. Similarly I want to do something similar with Elastic Search Querying if possible, ( where I want all the records in the query), but not putting them all together in my memory while iterating them.
Is it possible to do so using any java client(via rest API), if not via rest API, then is there a method of solving this problem.
Thanks

First, if you use a Java or another JVM language, you could also use the native client. Jest is a good option if you want to keep your dependencies small (the java client is essentially the same as the complete server) or if you want or can access Elasticsearch only via the HTTP interface and not via its binary interface.
Second, what you want to use is the scroll API: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html (didn't found a quick reference on the Jest documentation though).
It doesn't exactly work like ResultSet, but allows you to iterate in chunks over all your results. An example, copied from the documentation
QueryBuilder query = ...;
SearchResponse scrollResponse = client.prepareSearch(index)
.setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(60000)) // timeout
.setQuery(query)
.setSize(100) // bulk size
.execute().actionGet();
//Scroll until no hits are returned
while (!scrollResp.getHits().getHits().isEmpty()) {
for (SearchHit hit : scrollResp.getHits().getHits()) {
//Handle the hit...
}
scrollResp = client.prepareSearchScroll(scrollResp.getScrollId())
.setScroll(new TimeValue(60000))
.execute().actionGet();
}