Develop Reference

Reindex part of Elasticsearch index onto new index via Jest

I have a test ElasticSearch 6.0 index populated with millions of records, likely to be in the billions in production. I need to search for a subset of these records, then save this subset of the original set into a secondary index for later searching. I have proven this out via querying ES on Kibana, the challenge is to find appropriate APIs in Java 8 using my Jest client (searchbox.io, version 5.3.3) to do the same. The ElasticSearch cluster is on AWS, so using a transport client is out.
POST _reindex?slices=10&wait_for_completion=false
{ "conflicts": "proceed",
"source":{
"index": "my_source_idx",
"size": 5000,
"query": { "bool": {
"filter": { "bool" : { "must" : [
{ "nested": { "path": "test", "query": { "bool": { "must":[
{ "terms" : { "test.RowKey": ["abc"]} },
{ "range" : { "test.dates" : { "lte": "2018-01-01", "gte": "2010-08-01"} } },
{ "range" : { "test.DatesCount" : { "gte": 2} } },
{ "script" : { "script" : { "id": "my_painless_script",
"params" : {"min_occurs" : 1, "dateField": "test.dates", "RowKey": ["abc"], "fromDate": "2010-08-01", "toDate": "2018-01-01"}}}}
]}}}}
]}}
}}
},
"dest": {
"index": "my_dest_idx"
},
"script": {
"source": <My painless script>
} }
I am aware I can perform a search on the source index, then create and bulk load the response records onto the new index, but I want to be able to do this all in one shot, as I do have a painless script to glean off some information that is pertinent to the queries that will search the secondary index. Performance is a concern, as the application will be chaining subsequent queries together using the destination index to query against. Does anyone know how to do accomplish this using Jest?

It appears as if this particular functionality is not yet supported in Jest. The Jest API It has a way to pass in a script (not a query) as a parameter, but I even was having problems with that.
EDIT:
After some hacking with a coworker, we found a way around this...
Step 1) Extend the GenericResultAbstractionAction class with edits to the script:
public class GenericResultReindexActionHack extends GenericResultAbstractAction {
GenericResultReindexActionHack(GenericResultReindexActionHack.Builder builder) {
super(builder);
Map<String, Object> payload = new HashMap<>();
payload.put("source", builder.source);
payload.put("dest", builder.dest);
if (builder.conflicts != null) {
payload.put("conflicts", builder.conflicts);
}
if (builder.size != null) {
payload.put("size", builder.size);
}
if (builder.script != null) {
Script script = (Script) builder.script;
// Note the script parameter needs to be formatted differently to conform to the ES _reindex API:
payload.put("script", new Gson().toJson(ImmutableMap.of("id", script.getIdOrCode(), "params", script.getParams())));
}
this.payload = ImmutableMap.copyOf(payload);
setURI(buildURI());
}
#Override
protected String buildURI() {
return super.buildURI() + "/_reindex";
}
#Override
public String getRestMethodName() {
return "POST";
}
#Override
public String getData(Gson gson) {
if (payload == null) {
return null;
} else if (payload instanceof String) {
return (String) payload;
} else {
// We need to remove the incorrect formatting for the query, dest, and script fields:
// TODO: Need to consider spaces in the JSON
return gson.toJson(payload).replaceAll("\\\\n", "")
.replace("\\", "")
.replace("query\":\"", "query\":")
.replace("\"},\"dest\"", "},\"dest\"")
.replaceAll("\"script\":\"","\"script\":")
.replaceAll("\"}","}")
.replaceAll("},\"script\"","\"},\"script\"");
}
}
public static class Builder extends GenericResultAbstractAction.Builder<GenericResultReindexActionHack , GenericResultReindexActionHack.Builder> {
private Object source;
private Object dest;
private String conflicts;
private Long size;
private Object script;
public Builder(Object source, Object dest) {
this.source = source;
this.dest = dest;
}
public GenericResultReindexActionHack.Builder conflicts(String conflicts) {
this.conflicts = conflicts;
return this;
}
public GenericResultReindexActionHack.Builder size(Long size) {
this.size = size;
return this;
}
public GenericResultReindexActionHack.Builder script(Object script) {
this.script = script;
return this;
}
public GenericResultReindexActionHack.Builder waitForCompletion(boolean waitForCompletion) {
return setParameter("wait_for_completion", waitForCompletion);
}
public GenericResultReindexActionHack.Builder waitForActiveShards(int waitForActiveShards) {
return setParameter("wait_for_active_shards", waitForActiveShards);
}
public GenericResultReindexActionHack.Builder timeout(long timeout) {
return setParameter("timeout", timeout);
}
public GenericResultReindexActionHack.Builder requestsPerSecond(double requestsPerSecond) {
return setParameter("requests_per_second", requestsPerSecond);
}
public GenericResultReindexActionHack build() {
return new GenericResultReindexActionHack(this);
}
}
}
Step 2) Use of this class with a query then requires you to pass in the query as part of the source, then remove the '\n' characters:
ImmutableMap<String, Object> sourceMap = ImmutableMap.of("index", sourceIndex, "query", qb.toString().replaceAll("\\\\n", ""));
ImmutableMap<String, Object> destMap = ImmutableMap.of("index", destIndex);
GenericResultReindexActionHack reindex = new GenericResultReindexActionHack.Builder(sourceMap, destMap)
.waitForCompletion(false)
.conflicts("proceed")
.size(5000L)
.script(reindexScript)
.setParameter("slices", 10)
.build();
JestResult result = handleResult(reindex);
String task = result.getJsonString();
return (task);
Note the reindexScript parameter is of type org.elasticsearch.script.
This is a messy, hack-y way of getting around the limitations of Jest, but it seems to work. I understand that by doing it this way there may be some limitations to what may be acceptable in the input formatting...

fail using a variable to call a method in java-android

I've tried using a variable to invoke a java method, using method.invoke(), as suggested in this example. But it seems there should be an object or something as a parameter in method.invoke(). I've tried using null, but the method didn't get invoked. My code is as follows:
String ACTION = "cart";
Method method = SolverService.class.getDeclaredMethod("Method" + ACTION);
method.invoke(null);
I've got a method as:
public void Methodcart(){
Toast.makeText(this,"Method called",Toast.LENGTH_LONG).show();
}
PS: I HAVE TO make this method.invoke() work. Otherwise, I need to write a very long list of switch-case statements.I've gone through the documentation but couldn't understand much about the object instance i might need to use here as I'm new to android app developing.

You can try something similar to the code shown below (Java Reflection) -
Suppose I have a class ClassWithMethods.java with the methods I want to invoke in some other class as shown below -
public class ClassWithMethods {
private int counter;
public void printIt(){
System.out.println("printIt() no param");
}
public void printItString(String temp){
System.out.println("printIt() with param String : " + temp);
}
}
Now I also have another class TestApp.java which will invoke methods of the ClassWithMethods class at runtime using Java Reflection -
public class TestApp {
public static void main(String[] args) {
//no paramater
Class noparams[] = {};
//String parameter
Class[] paramString = new Class[1];
paramString[0] = String.class;
//int parameter
Class[] paramInt = new Class[1];
paramInt[0] = Integer.TYPE;
try{
//load the ClassWithMethods at runtime
Class cls = Class.forName("com.myapps.ClassWithMethods");
Object obj = cls.newInstance();
//call the printIt method
Method method = cls.getDeclaredMethod("printIt", noparams);
method.invoke(obj, null);
//call the printItString method, pass a String param
method = cls.getDeclaredMethod("printItString", paramString);
method.invoke(obj, new String("someString"));
}catch(Exception ex){
ex.printStackTrace();
}
}

I am using Java Reflection in my current project (since you mentioned you are using Android Studio) to get Battery Capacity of device from PowerProfile class which is internal to the Android OS.
public double getBatteryCapacity() {
Object mPowerProfile = null;
try {
mPowerProfile = Class.forName("com.android.internal.os.PowerProfile")
.getConstructor(Context.class)
.newInstance(getContext());
} catch (Exception e) {
e.printStackTrace();
}
try {
// get access to method named "getAveragePower()" in the class "PowerProfile"
Method getAveragePower = Class.forName("com.android.internal.os.PowerProfile").getMethod("getAveragePower", String.class);
//Get total battery capacity in mAh.
double batteryCapacity = (Double) getAveragePower.invoke(mPowerProfile, "battery.capacity");
return batteryCapacity;
} catch (Exception e) {
e.printStackTrace();
}
return 0.0;
}
Here is a screenshot of how the actual method structure looks like in the PowerProfile class -

Looking for Json-path/(any API) to update any value in given json string in Java

Inshort : I am trying to find some api that could just change the value by taking first parameter as jsonString , second parameter as JSONPath and third will be new value of that parameter. But, all I found is this..
https://code.google.com/p/json-path/
This api allows me to find any value in JSON String. But, I am not finding easy way to update the value of any key. For example, Here is a book.json.
{
"store":{
"book":[
{
"category":"reference",
"author":"Nigel Rees",
"title":"Sayings of the Century",
"price":8.95
},
{
"category":"fiction",
"author":"Evelyn Waugh",
"title":"Sword of Honour",
"price":12.99,
"isbn":"0-553-21311-3"
}
],
"bicycle":{
"color":"red",
"price":19.95
}
}
}
I can access color of bicycle by doing this.
String bicycleColor = JsonPath.read(json, "$.store.bicycle.color");
But I am looking for a method in JsonPath or other api some thing like this
JsonPath.changeNodeValue(json, "$.store.bicycle.color", "green");
String bicycleColor = JsonPath.read(json, "$.store.bicycle.color");
System.out.println(bicycleColor); // This should print "green" now.
I am excluding these options,
Create a new JSON String.
Create a JSON Object to deal with changing value and convert it back to jsonstring
Reason: I have about 500 different requests for different types of service which return different json structure. So, I do not want to manually create new JSON string always. Because, IDs are dynamic in json structure.
Any idea or direction is much appreciated.
Updating this question with following answer.
Copy MutableJson.java.
copy this little snippet and modify as per you need.
private static void updateJsonValue() {
JSONParser parser = new JSONParser();
JSONObject jsonObject = new JSONObject();
FileReader reader = null;
try {
File jsonFile = new File("path to book.json");
reader = new FileReader(jsonFile);
jsonObject = (JSONObject) parser.parse(reader);
} catch (Exception ex) {
System.out.println(ex.getLocalizedMessage());
}
Map<String, Object> userData = null;
try {
userData = new ObjectMapper().readValue(jsonObject.toJSONString(), Map.class);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
MutableJson json = new MutableJson(userData);
System.out.println("Before:\t" + json.map());
json.update("$.store.book[0].author", "jigish");
json.update("$.store.book[1].category", "action");
System.out.println("After:\t" + json.map().toString());
}
Use these libraries.
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.codehaus.jackson.map.ObjectMapper;

The thing is that the functionality you want is already an undocumented feature of JsonPath. Example using your json structure:
String json = "{ \"store\":{ \"book\":[ { \"category\":\"reference\", \"author\":\"Nigel Rees\", \"title\":\"Sayings of the Century\", \"price\":8.95 }, { \"category\":\"fiction\", \"author\":\"Evelyn Waugh\", \"title\":\"Sword of Honour\", \"price\":12.99, \"isbn\":\"0-553-21311-3\" } ], \"bicycle\":{ \"color\":\"red\", \"price\":19.95 } } }";
DocumentContext doc = JsonPath.parse(json).
set("$.store.bicycle.color", "green").
set("$.store.book[0].price", 9.5);
String newJson = new Gson().toJson(doc.read("$"));

Assuming that parsed JSON can be represented in memory as a Map, you can build an API similar to JsonPath that looks like:
void update(Map<String, Object> json, String path, Object newValue);
I've quickly done a gist of a dirty implementation for simple specific paths (no support for conditions and wildcards) that can traverse json tree, E.g. $.store.name, $.store.books[0].isbn. Here it is: MutableJson.java. It definitely needs improvement, but can give a good start.
Usage example:
import java.util.*;
public class MutableJson {
public static void main(String[] args) {
MutableJson json = new MutableJson(
new HashMap<String, Object>() {{
put("store", new HashMap<String, Object>() {{
put("name", "Some Store");
put("books", Arrays.asList(
new HashMap<String, Object>() {{
put("isbn", "111");
}},
new HashMap<String, Object>() {{
put("isbn", "222");
}}
));
}});
}}
);
System.out.println("Before:\t" + json.map());
json.update("$.store.name", "Book Store");
json.update("$.store.books[0].isbn", "444");
json.update("$.store.books[1].isbn", "555");
System.out.println("After:\t" + json.map());
}
private final Map<String, Object> json;
public MutableJson(Map<String, Object> json) {
this.json = json;
}
public Map<String, Object> map() {
return json;
}
public void update(String path, Object newValue) {
updateJson(this.json, Path.parse(path), newValue);
}
private void updateJson(Map<String, Object> data, Iterator<Token> path, Object newValue) {
Token token = path.next();
for (Map.Entry<String, Object> entry : data.entrySet()) {
if (!token.accept(entry.getKey(), entry.getValue())) {
continue;
}
if (path.hasNext()) {
Object value = token.value(entry.getValue());
if (value instanceof Map) {
updateJson((Map<String, Object>) value, path, newValue);
}
} else {
token.update(entry, newValue);
}
}
}
}
class Path {
public static Iterator<Token> parse(String path) {
if (path.isEmpty()) {
return Collections.<Token>emptyList().iterator();
}
if (path.startsWith("$.")) {
path = path.substring(2);
}
List<Token> tokens = new ArrayList<>();
for (String part : path.split("\\.")) {
if (part.matches("\\w+\\[\\d+\\]")) {
String fieldName = part.substring(0, part.indexOf('['));
int index = Integer.parseInt(part.substring(part.indexOf('[')+1, part.indexOf(']')));
tokens.add(new ArrayToken(fieldName, index));
} else {
tokens.add(new FieldToken(part));
}
};
return tokens.iterator();
}
}
abstract class Token {
protected final String fieldName;
Token(String fieldName) {
this.fieldName = fieldName;
}
public abstract Object value(Object value);
public abstract boolean accept(String key, Object value);
public abstract void update(Map.Entry<String, Object> entry, Object newValue);
}
class FieldToken extends Token {
FieldToken(String fieldName) {
super(fieldName);
}
#Override
public Object value(Object value) {
return value;
}
#Override
public boolean accept(String key, Object value) {
return fieldName.equals(key);
}
#Override
public void update(Map.Entry<String, Object> entry, Object newValue) {
entry.setValue(newValue);
}
}
class ArrayToken extends Token {
private final int index;
ArrayToken(String fieldName, int index) {
super(fieldName);
this.index = index;
}
#Override
public Object value(Object value) {
return ((List) value).get(index);
}
#Override
public boolean accept(String key, Object value) {
return fieldName.equals(key) && value instanceof List && ((List) value).size() > index;
}
#Override
public void update(Map.Entry<String, Object> entry, Object newValue) {
List list = (List) entry.getValue();
list.set(index, newValue);
}
}
A JSON string can be easily parsed into a Map using Jackson:
Map<String,Object> userData = new ObjectMapper().readValue("{ \"store\": ... }", Map.class);

Just answering for folks landing on this page in future for reference.
You could consider using a Java implementation of jsonpatch. RFC can be found here
JSON Patch is a format for describing changes to a JSON document. It can be used to avoid sending a whole document when only a part has changed. When used in combination with the HTTP PATCH method it allows partial updates for HTTP APIs in a standards compliant way.
You can specify the operation that needs to be performed (replace, add....), json path at which it has to be performed, and the value which should be used.
Again, taking example from the RFC :
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]
For Java implementation, I have not used it myself, but you can give a try to https://github.com/fge/json-patch

So in order to change a value within a JSon string, there are two steps:
Parse the JSon
Modify the appropriate field
You are trying to optimize step 2, but understand that you are not going to be able to avoid step 1. Looking at the Json-path source code (which, really, is just a wrapper around Jackson), note that it does do a full parse of the Json string before being able to spit out the read value. It does this parse every time you call read(), e.g. it is not cached.
I think this task is specific enough that you're going to have to write it yourself. Here is what I would do:
Create an object that represents the data in the parsed Json string.
Make sure this object has, as part of it's fields, the Json String pieces that you do not expect to change often.
Create a custom Deserializer in the Json framework of your choice that will populate the fields correctly.
Create a custom Serializer that uses the cached String pieces, plus the data that you expect to change
I think the exact scope of your problem is unusual enough that it is unlikely a library already exists for this. When a program receives a Json String, most of the time what it wants is the fully deserialized object - it is unusual that it needs to FORWARD this object on to somewhere else.

Java Reflection: "java.lang.NoSuchMethodException"

I am trying to get the method from another class using reflection, but for some reason it keeps on giving me a no such method exception. These are the classes im using:
ScriptTable class:
for(Class<?> script: Scripts) {
System.out.println(script.getName());
try {
Method c = script.getMethod("scriptInfo()", script);
} catch(Exception e) {
e.printStackTrace();
}
}
DummyScript class
public String[] scriptInfo() {
String[] ScriptInfo = {"DummyScript", "Chris", "Does nothing atm"};
return ScriptInfo;
}

This is your problem:
script.getMethod("scriptInfo()", script);
change it to:
script.getMethod("scriptInfo");
and look here to see why:
http://docs.oracle.com/javase/7/docs/api/java/lang/Class.html#getMethod%28java.lang.String,%20java.lang.Class...%29

Is it possible automatically instantiation of a nested Property with Commons Bean Utils?

I'm using PropertyUtils.setProperty(object, name, value) method of Apache Commons Bean Utils:
Giving these classes:
public class A {
B b;
}
public class B {
C c;
}
public class C {
}
And this:
A a = new A();
C c = new C();
PropertyUtils.setProperty(a, "b.c", c); //exception
If I try that I get:
org.apache.commons.beanutils.NestedNullException: Null property value for 'b.c' on bean class 'class A'
Is it possible to tell PropertyUtils that if a nested property has a null value try to instantiate it (default constructor) before trying to go deeper?
Any other approach?
Thank you

I solved it by doing this:
private void instantiateNestedProperties(Object obj, String fieldName) {
try {
String[] fieldNames = fieldName.split("\\.");
if (fieldNames.length > 1) {
StringBuffer nestedProperty = new StringBuffer();
for (int i = 0; i < fieldNames.length - 1; i++) {
String fn = fieldNames[i];
if (i != 0) {
nestedProperty.append(".");
}
nestedProperty.append(fn);
Object value = PropertyUtils.getProperty(obj, nestedProperty.toString());
if (value == null) {
PropertyDescriptor propertyDescriptor = PropertyUtils.getPropertyDescriptor(obj, nestedProperty.toString());
Class<?> propertyType = propertyDescriptor.getPropertyType();
Object newInstance = propertyType.newInstance();
PropertyUtils.setProperty(obj, nestedProperty.toString(), newInstance);
}
}
}
} catch (IllegalAccessException e) {
throw new RuntimeException(e);
} catch (InvocationTargetException e) {
throw new RuntimeException(e);
} catch (NoSuchMethodException e) {
throw new RuntimeException(e);
} catch (InstantiationException e) {
throw new RuntimeException(e);
}
}

I know the question is about apache commons PropertyUtils.setProperty but there is very similar functionality available in
Spring Expression Language "SpEL" which does exactly what you want. Better still it deals with lists and arrays too. The doc link above is for spring 4.x but the code below works for me in spring 3.2.9.
StockOrder stockOrder = new StockOrder(); // Your root class here
SpelParserConfiguration config = new SpelParserConfiguration(true,true); // auto create objects if null
ExpressionParser parser = new SpelExpressionParser(config);
StandardEvaluationContext modelContext = new StandardEvaluationContext(stockOrder);
parser.parseExpression("techId").setValue(modelContext, "XXXYYY1");
parser.parseExpression("orderLines[0].partNumber").setValue(modelContext, "65498");
parser.parseExpression("orderLines[0].inventories[0].serialNumber").setValue(modelContext, "54686513216");
System.out.println(ReflectionToStringBuilder.toString(stockOrder));

A little correction:
String fn = fieldNames[i];
if (i != 0) {
nestedProperty.append(".");
}
nestedProperty.append(fn);
Object value = PropertyUtils.getProperty(obj, nestedProperty.toString());

IMHO, the best solution is to get rid of the commons-beanutils and use Spring Framework org.springframework.beans.PropertyAccessorFactory
BeanWrapper wrapper = PropertyAccessorFactory.forBeanPropertyAccess(targetObject);
wrapper.setAutoGrowNestedPaths(true);
I won't delve into details of how it works, but if you want to check it out, go take a look at the link above, this API is quite intuitive, but you'll need to have Spring Framework Core configured on your classpath, so I wouldn't recommend that you add spring just for the sake of this feature.
However,
If you only have commons-beanutils as your ally, this following code snippet may help you to grow your nested paths, as you set the values, therefore, you won't need to concern about the null objects along the path properties.
In this example I used with JPA Tuple Query to construct a custom object with some specific property paths with its corresponding values to be set.
import java.util.ArrayList;
import java.util.List;
import javax.persistence.Tuple;
import javax.persistence.TupleElement;
import org.apache.commons.beanutils.PropertyUtils;
import org.apache.commons.beanutils.expression.DefaultResolver;
public class TupleToObject<T> {
public List<T> transformResult(List<Tuple> result, Class<T> targetClass) {
try {
List<T> objects = new ArrayList<>();
for (Tuple tuple : result) {
T target = targetClass.newInstance();
List<TupleElement<?>> elements = tuple.getElements();
for (TupleElement<?> tupleElement : elements) {
String alias = tupleElement.getAlias();
Object value = tuple.get(alias);
if (value != null) {
instantiateObject(target, alias);
PropertyUtils.setNestedProperty(target, alias, value);
}
}
objects.add(target);
}
return objects;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private void instantiateObject(T target, String propertyPath) throws Exception {
DefaultResolver resolver = new DefaultResolver();
Object currentTarget = target;
while (resolver.hasNested(propertyPath)) {
final String property = resolver.next(propertyPath);
Object value = PropertyUtils.getSimpleProperty(currentTarget, property);
if (value == null) {
Class<?> propertyType = PropertyUtils.getPropertyType(currentTarget, property);
value = propertyType.newInstance();
PropertyUtils.setSimpleProperty(currentTarget, property, value);
}
currentTarget = value;
propertyPath = resolver.remove(propertyPath);
}
}
}
This code is using commons-beanutils-1.9.3.jar
Hope it helps!

I have used only reflection w/o Apache library to achieve this. The assumption is that all object to be traversed are all POJOs, and default construction is publicly accessible. This way, there is no need to construct the reference path for each loop.
public Object getOrCreateEmbeddedObject(Object inputObj,String[] fieldNames) throws Exception {
Object cursor = inputObj;
//Loop until second last index
for (int i = 0; i < fieldNames.length - 1; i++){
Field ff = getClassFieldFrom(cursor,fieldNames[i]);
Object child = ff.get(cursor);
if(null == child) {
Class<?> cls=ff.getType();
child = cls.newInstance();
ff.set(cursor, child);
}
cursor = child;
}
return cursor;
}
private Field getClassFieldFrom(Object object, String fieldStr)
throws NoSuchFieldException {
java.lang.reflect.Field ff = object.getClass().getDeclaredField(fieldStr);
ff.setAccessible(true);
return ff;
}
If you have any suggestion to my solution , please let me know.

I went for the very basic approach of just instantiating each of the objects by default:
public class A {
B b = new B();
}
public class B {
C c = new C();
}
public class C {
}
Not ideal, but it worked for my situation and didn't involve complicated fixes.

After doing some research, the short answer to "Is it possible..." question is No.