I am working in an XPages application with the OpenNTF Domino API to explore the Graph data modelling capabilities. As example I have taken the Teamroom that ships with IBM Domino.
I have defined a method to migrate responses documents into the Grap db but I get the error message: Cannot make a static reference to the non-static method
Here is what the method looks like:
private void migrateResponses(DFramedTransactionalGraph<DGraph> profilesGraph) {
try {
Database db = Factory.getSession().getCurrentDatabase();
View view = db.getView("responsesOnly");
DocumentCollection col = view.getAllDocuments();
System.out.println("number of docs found " + col.getCount());
for (Document response : col) {
System.out.println("form:" + response.getFormName());
System.out.println("id:" + response.getUniversalID());
org.openntf.domino.ext.Document parent = response.getParentDocument();
if (null == parent.getParentDocument()){
//has no parent document so this parent document is a MainTopic/Post
Post post = profilesGraph.addVertex(parent.getMetaversalID(), Post.class);
Response vertexResponse = profilesGraph.addVertex(response.getUniversalID(), Response.class);
vertexResponse.setSubject(response.getItemValueString("Subject"));
Post.addResponse(vertexResponse);
}
}
profilesGraph.commit();
} catch (Throwable t) {
XspOpenLogUtil.logError(t);
}
}
The error occurs in line:
Post.addResponse(vertexResponse);
Here is what my Post class looks like:
package com.wordpress.quintessens.graph.teamroom;
import org.openntf.domino.graph2.annotations.AdjacencyUnique;
import org.openntf.domino.graph2.builtin.DVertexFrame;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.frames.Property;
import com.tinkerpop.frames.modules.typedgraph.TypeValue;
#TypeValue("post")
public interface Post extends DVertexFrame {
#Property("$$Key")
public String getKey();
#Property("subject")
public String getSubject();
#Property("subject")
public void setSubject(String n);
// real edges!
#AdjacencyUnique(label = "hasWritten", direction = Direction.OUT)
public Iterable<Profile> getAuthors();
#AdjacencyUnique(label = "hasReaction", direction = Direction.IN)
public void addResponse(Response response);
#AdjacencyUnique(label = "hasReaction", direction = Direction.IN)
public void removeResponse(Response response);
#AdjacencyUnique(label = "hasReaction", direction = Direction.IN)
public Iterable<Response> getResponses();
}
Do you have a suggestion how I should adapt my code to make it work?
Unless OpenNTF or TinkerPop are doing some kind of magic with the supplied annotations, you are attempting to call a non-static method on an interface. Are you sure that you don't want to change:
Post.addResponse(vertexResponse);
to
post.addResponse(vertexResponse);
Related
I want to make a POST request with URL Query Params set to the values of an object.
For example
http://test/data?a=1&b=2&c=3
I want to make a post request to this URL with a class like this:
public class Data {
private Integer a;
private Integer b;
private Integer c;
}
I do NOT want to do each field manually, like this:
public void sendRequest(Data data) {
String url = UriComponentsBuilder.fromHttpUrl("http://test/")
.queryParam("a", data.getA())
.queryParam("b", data.getB())
.queryParam("c", data.getC())
.toUriString();
restTemplate.postForObject(url, body, Void.class);
}
Instead, I want to use the entire object:
public void sendRequest(Data data) {
String url = UriComponentsBuilder.fromHttpUrl("http://test/")
.queryParamsAll(data) //pseudo
.toUriString();
restTemplate.postForObject(url, body, Void.class);
}
Your requirement is like QS in js. Thx qianshui423/qs . It is implementation QS in java. It is coded by a Chinese guy. At first git clone it and use below cmd to build. You will get a jar called "qs-1.0.0.jar" in build/libs (JDK required version 8)
# cd qs directory
./gradlew build -x test
Import it, I do a simple demo as below. For your requirement, you can build class to transfer your Obj into QSObject. Besides toQString, QS can parse string to QSObject. I think it powerful.
import com.qs.core.QS;
import com.qs.core.model.QSObject;
public class Demo {
public static void main(String[] args) throws Exception{
QSObject qsobj = new QSObject();
qsobj.put("a",1);
qsobj.put("b",2);
qsobj.put("c",3);
String str = QS.toQString(qsobj);
System.out.println(str); // output is a=1&b=2&c=3
}
}
When working with gRPC, we need to generate the gRPC client and server interfaces from our .proto service definition via protocol buffer compiler (protoc) or using Gradle or Maven protoc build plugin.
Flow now: protobuf file -> java code -> gRPC client.
So, is there any way to skip this step?
How to create a generic gRPC client that can call the server directly from the protobuf file without compile into java code?
Or, is there a way to Generated Code at runtime?
Flow expect: protobuf file -> gRPC client.
I want to build a generic gRPC client system with the input are protobuf files along with description of method, package, message request ... without having to compile again for each protobuf.
Thank you very much.
Protobuf systems really need protoc to be run. However, the generated code could be skipped. Instead of passing something like --java_out and --grpc_java_out to protoc you can pass --descriptor_set_out=FILE which will parse the .proto file into a descriptor file. A descriptor file is a proto-encoded FileDescriptorSet. This is the same basic format as used with the reflection service.
Once you have a descriptor, you can load it a FileDescriptor at a time and create a DynamicMessage.
Then for the gRPC piece, you need to create a gRPC MethodDescriptor.
static MethodDescriptor from(
Descriptors.MethodDescriptor methodDesc
) {
return MethodDescriptor.<DynamicMessage, DynamicMessage>newBuilder()
// UNKNOWN is fine, but the "correct" value can be computed from
// methodDesc.toProto().getClientStreaming()/getServerStreaming()
.setType(getMethodTypeFromDesc(methodDesc))
.setFullMethodName(MethodDescriptor.generateFullMethodName(
serviceDesc.getFullName(), methodDesc.getName()))
.setRequestMarshaller(ProtoUtils.marshaller(
DynamicMessage.getDefaultInstance(methodDesc.getInputType())))
.setResponseMarshaller(ProtoUtils.marshaller(
DynamicMessage.getDefaultInstance(methodDesc.getOutputType())))
.build();
static MethodDescriptor.MethodType getMethodTypeFromDesc(
Descriptors.MethodDescriptor methodDesc
) {
if (!methodDesc.isServerStreaming()
&& !methodDesc.isClientStreaming()) {
return MethodDescriptor.MethodType.UNARY;
} else if (methodDesc.isServerStreaming()
&& !methodDesc.isClientStreaming()) {
return MethodDescriptor.MethodType.SERVER_STREAMING;
} else if (!methodDesc.isServerStreaming()) {
return MethodDescriptor.MethodType.CLIENT_STREAMING);
} else {
return MethodDescriptor.MethodType.BIDI_STREAMING);
}
}
At that point you have everything you need and can call Channel.newCall(method, CallOptions.DEFAULT) in gRPC. You're also free to use ClientCalls to use something more similar to the stub APIs.
So dynamic calls are definitely possible, and is used for things like grpcurl. But it also is not easy and so is generally only done when necessary.
I did it in Java, and the step is:
Call reflection service to get FileDescriptorProto list by method name
Get FileDescriptor of method from FileDescriptorProto list by package name, service name
Get MethodDescriptor from ServiceDescriptor which get from the FileDescriptor
Generate a MethodDescriptor<DynamicMessage, DynamicMessage> by MethodDescriptor
Build request DynamicMessage from content like JSON or others
Call method
Parse response content to JSON from DynamicMessage response
You can reference the full sample in project helloworlde/grpc-java-sample#reflection
And proto is:
syntax = "proto3";
package io.github.helloworlde.grpc;
option go_package = "api;grpc_gateway";
option java_package = "io.github.helloworlde.grpc";
option java_multiple_files = true;
option java_outer_classname = "HelloWorldGrpc";
service HelloService{
rpc SayHello(HelloMessage) returns (HelloResponse){
}
}
message HelloMessage {
string message = 2;
}
message HelloResponse {
string message = 1;
}
Start server for this proto by yourself, and the full code in Java just like:
import com.google.protobuf.ByteString;
import com.google.protobuf.DescriptorProtos;
import com.google.protobuf.Descriptors;
import com.google.protobuf.DynamicMessage;
import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.TypeRegistry;
import com.google.protobuf.util.JsonFormat;
import io.grpc.CallOptions;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.grpc.MethodDescriptor;
import io.grpc.protobuf.ProtoUtils;
import io.grpc.reflection.v1alpha.ServerReflectionGrpc;
import io.grpc.reflection.v1alpha.ServerReflectionRequest;
import io.grpc.reflection.v1alpha.ServerReflectionResponse;
import io.grpc.stub.ClientCalls;
import io.grpc.stub.StreamObserver;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
#Slf4j
public class ReflectionCall {
public static void main(String[] args) throws InterruptedException {
// 反射方法的格式只支持 package.service.method 或者 package.service
String methodSymbol = "io.github.helloworlde.grpc.HelloService.SayHello";
String requestContent = "{\"message\": \"Reflection\"}";
// 构建 Channel
ManagedChannel channel = ManagedChannelBuilder.forAddress("127.0.0.1", 9090)
.usePlaintext()
.build();
// 使用 Channel 构建 BlockingStub
ServerReflectionGrpc.ServerReflectionStub reflectionStub = ServerReflectionGrpc.newStub(channel);
// 响应观察器
StreamObserver<ServerReflectionResponse> streamObserver = new StreamObserver<ServerReflectionResponse>() {
#Override
public void onNext(ServerReflectionResponse response) {
try {
// 只需要关注文件描述类型的响应
if (response.getMessageResponseCase() == ServerReflectionResponse.MessageResponseCase.FILE_DESCRIPTOR_RESPONSE) {
List<ByteString> fileDescriptorProtoList = response.getFileDescriptorResponse().getFileDescriptorProtoList();
handleResponse(fileDescriptorProtoList, channel, methodSymbol, requestContent);
} else {
log.warn("未知响应类型: " + response.getMessageResponseCase());
}
} catch (Exception e) {
log.error("处理响应失败: {}", e.getMessage(), e);
}
}
#Override
public void onError(Throwable t) {
}
#Override
public void onCompleted() {
log.info("Complete");
}
};
// 请求观察器
StreamObserver<ServerReflectionRequest> requestStreamObserver = reflectionStub.serverReflectionInfo(streamObserver);
// 构建并发送获取方法文件描述请求
ServerReflectionRequest getFileContainingSymbolRequest = ServerReflectionRequest.newBuilder()
.setFileContainingSymbol(methodSymbol)
.build();
requestStreamObserver.onNext(getFileContainingSymbolRequest);
channel.awaitTermination(10, TimeUnit.SECONDS);
}
/**
* 处理响应
*/
private static void handleResponse(List<ByteString> fileDescriptorProtoList,
ManagedChannel channel,
String methodFullName,
String requestContent) {
try {
// 解析方法和服务名称
String fullServiceName = extraPrefix(methodFullName);
String methodName = extraSuffix(methodFullName);
String packageName = extraPrefix(fullServiceName);
String serviceName = extraSuffix(fullServiceName);
// 根据响应解析 FileDescriptor
Descriptors.FileDescriptor fileDescriptor = getFileDescriptor(fileDescriptorProtoList, packageName, serviceName);
// 查找服务描述
Descriptors.ServiceDescriptor serviceDescriptor = fileDescriptor.getFile().findServiceByName(serviceName);
// 查找方法描述
Descriptors.MethodDescriptor methodDescriptor = serviceDescriptor.findMethodByName(methodName);
// 发起请求
executeCall(channel, fileDescriptor, methodDescriptor, requestContent);
} catch (Exception e) {
log.error(e.getMessage(), e);
}
}
/**
* 解析并查找方法对应的文件描述
*/
private static Descriptors.FileDescriptor getFileDescriptor(List<ByteString> fileDescriptorProtoList,
String packageName,
String serviceName) throws Exception {
Map<String, DescriptorProtos.FileDescriptorProto> fileDescriptorProtoMap =
fileDescriptorProtoList.stream()
.map(bs -> {
try {
return DescriptorProtos.FileDescriptorProto.parseFrom(bs);
} catch (InvalidProtocolBufferException e) {
e.printStackTrace();
}
return null;
})
.filter(Objects::nonNull)
.collect(Collectors.toMap(DescriptorProtos.FileDescriptorProto::getName, f -> f));
if (fileDescriptorProtoMap.isEmpty()) {
log.error("服务不存在");
throw new IllegalArgumentException("方法的文件描述不存在");
}
// 查找服务对应的 Proto 描述
DescriptorProtos.FileDescriptorProto fileDescriptorProto = findServiceFileDescriptorProto(packageName, serviceName, fileDescriptorProtoMap);
// 获取这个 Proto 的依赖
Descriptors.FileDescriptor[] dependencies = getDependencies(fileDescriptorProto, fileDescriptorProtoMap);
// 生成 Proto 的 FileDescriptor
return Descriptors.FileDescriptor.buildFrom(fileDescriptorProto, dependencies);
}
/**
* 根据包名和服务名查找相应的文件描述
*/
private static DescriptorProtos.FileDescriptorProto findServiceFileDescriptorProto(String packageName,
String serviceName,
Map<String, DescriptorProtos.FileDescriptorProto> fileDescriptorProtoMap) {
for (DescriptorProtos.FileDescriptorProto proto : fileDescriptorProtoMap.values()) {
if (proto.getPackage().equals(packageName)) {
boolean exist = proto.getServiceList()
.stream()
.anyMatch(s -> serviceName.equals(s.getName()));
if (exist) {
return proto;
}
}
}
throw new IllegalArgumentException("服务不存在");
}
/**
* 获取前缀
*/
private static String extraPrefix(String content) {
int index = content.lastIndexOf(".");
return content.substring(0, index);
}
/**
* 获取后缀
*/
private static String extraSuffix(String content) {
int index = content.lastIndexOf(".");
return content.substring(index + 1);
}
/**
* 获取依赖类型
*/
private static Descriptors.FileDescriptor[] getDependencies(DescriptorProtos.FileDescriptorProto proto,
Map<String, DescriptorProtos.FileDescriptorProto> finalDescriptorProtoMap) {
return proto.getDependencyList()
.stream()
.map(finalDescriptorProtoMap::get)
.map(f -> toFileDescriptor(f, getDependencies(f, finalDescriptorProtoMap)))
.toArray(Descriptors.FileDescriptor[]::new);
}
/**
* 将 FileDescriptorProto 转为 FileDescriptor
*/
#SneakyThrows
private static Descriptors.FileDescriptor toFileDescriptor(DescriptorProtos.FileDescriptorProto fileDescriptorProto,
Descriptors.FileDescriptor[] dependencies) {
return Descriptors.FileDescriptor.buildFrom(fileDescriptorProto, dependencies);
}
/**
* 执行方法调用
*/
private static void executeCall(ManagedChannel channel,
Descriptors.FileDescriptor fileDescriptor,
Descriptors.MethodDescriptor originMethodDescriptor,
String requestContent) throws Exception {
// 重新生成 MethodDescriptor
MethodDescriptor<DynamicMessage, DynamicMessage> methodDescriptor = generateMethodDescriptor(originMethodDescriptor);
CallOptions callOptions = CallOptions.DEFAULT;
TypeRegistry registry = TypeRegistry.newBuilder()
.add(fileDescriptor.getMessageTypes())
.build();
// 将请求内容由 JSON 字符串转为相应的类型
JsonFormat.Parser parser = JsonFormat.parser().usingTypeRegistry(registry);
DynamicMessage.Builder messageBuilder = DynamicMessage.newBuilder(originMethodDescriptor.getInputType());
parser.merge(requestContent, messageBuilder);
DynamicMessage requestMessage = messageBuilder.build();
// 调用，调用方式可以通过 originMethodDescriptor.isClientStreaming() 和 originMethodDescriptor.isServerStreaming() 推断
DynamicMessage response = ClientCalls.blockingUnaryCall(channel, methodDescriptor, callOptions, requestMessage);
// 将响应解析为 JSON 字符串
JsonFormat.Printer printer = JsonFormat.printer()
.usingTypeRegistry(registry)
.includingDefaultValueFields();
String responseContent = printer.print(response);
log.info("响应: {}", responseContent);
}
/**
* 重新生成方法描述
*/
private static MethodDescriptor<DynamicMessage, DynamicMessage> generateMethodDescriptor(Descriptors.MethodDescriptor originMethodDescriptor) {
// 生成方法全名
String fullMethodName = MethodDescriptor.generateFullMethodName(originMethodDescriptor.getService().getFullName(), originMethodDescriptor.getName());
// 请求和响应类型
MethodDescriptor.Marshaller<DynamicMessage> inputTypeMarshaller = ProtoUtils.marshaller(DynamicMessage.newBuilder(originMethodDescriptor.getInputType())
.buildPartial());
MethodDescriptor.Marshaller<DynamicMessage> outputTypeMarshaller = ProtoUtils.marshaller(DynamicMessage.newBuilder(originMethodDescriptor.getOutputType())
.buildPartial());
// 生成方法描述, originMethodDescriptor 的 fullMethodName 不正确
return MethodDescriptor.<DynamicMessage, DynamicMessage>newBuilder()
.setFullMethodName(fullMethodName)
.setRequestMarshaller(inputTypeMarshaller)
.setResponseMarshaller(outputTypeMarshaller)
// 使用 UNKNOWN，自动修改
.setType(MethodDescriptor.MethodType.UNKNOWN)
.build();
}
}
There isn't much to prevent this technically. The two big hurdles are:
having a runtime-callable parser for reading the .proto, and
having a general purpose gRPC client available that takes things like the service method name as literals
Both are possible, but neither is trivial.
For 1, the crude way would be to shell/invoke protoc using the descriptor-set option to generate a schema binary, then deserialize that as a FileDescriptorSet (from descriptor.proto); this model gives you access to how protoc sees the file. Some platforms also have native parsers (essentially reimplementing protoc as a library in that platform), for example protobuf-net.Reflection does this in .NET-land
For 2, here's an implementation of that in C#. The approach should be fairly portable to Java, even if the details vary. You can look at a generated implementation to see how it works in any particular language.
(Sorry that the specific examples are C#/.NET, but that's where I live; the approaches should be portable, even if the specific code: not directly)
technically both are possible.
The codegen is simply generating a handful of classes; mainly protobuf messages, grpc method descriptors and stubs. You can implement it or check in the generated code to bypass the codegen. i am not sure what is the benefit of doing this tbh. Also, it will be very annoying if the proto is changed.
It is also possible to do it dynamically using byte codegen as long as you check-in some interfaces/abstract classes to represent those generated stub/method descriptors and protobuf messages. you have to make sure those non dynamic code is in sync with the proto definition though (most likely runtime check/exception).
I try to get some data from a dbus service and work with it in Java.
I can get the information in cli with the following command:
dbus-send --print-reply --system --dest=com.victronenergy.solarcharger.ttyUSB0 /Dc/0/Voltage com.victronenergy.BusItem.GetValue
The result is:
method return time=1538903662.321580 sender=:1.14 -> destination=:1.806 serial=335692 reply_serial=2
variant double 13.43
What I tried to get this data in Java, is:
After hours of reading, I created an Interface.
package javadbus;
import java.util.Map;
import org.freedesktop.dbus.DBusInterface;
import org.freedesktop.dbus.DBusSignal;
import org.freedesktop.dbus.Variant;
import org.freedesktop.dbus.exceptions.DBusException;
public interface BusItem extends DBusInterface
{
public static class PropertiesChanged extends DBusSignal
{
public final Map<String,Variant> changes;
public PropertiesChanged(String path, Map<String,Variant> changes) throws DBusException
{
super(path, changes);
this.changes = changes;
}
}
public String GetDescription(String language, int length);
public Variant GetValue();
public String GetText();
public int SetValue(Variant value);
public Variant GetMin();
public Variant GetMax();
public int SetDefault();
public Variant GetDefault();
}
Here I call getConnection() and getRemoteObject() successfully.
package javadbus;
import org.freedesktop.dbus.DBusConnection;
import org.freedesktop.dbus.exceptions.DBusException;
import org.freedesktop.dbus.Variant;
public class VictronEnergyDBusSolarCharger {
private String port;
private DBusConnection conn;
public VictronEnergyDBusSolarCharger(String port) {
this.port = port;
try {
this.conn = DBusConnection.getConnection(DBusConnection.SYSTEM);
} catch (DBusException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private String getData(String item) {
BusItem bi;
String data = null;
Variant vData = null;
try {
bi = (BusItem)conn.getRemoteObject("com.victronenergy.solarcharger." + this.port, item, BusItem.class);
vData = bi.GetValue();
//data = bi.GetText();
} catch (DBusException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return data;
}
...
}
It was a big task to resolve all dependecies and get the code compiled. But finaly I did it. So, javac now runs without errors.
But if I try to call the Method GetValue(), I get the following Exception:
[Sender] INFO org.freedesktop.dbus.MessageWriter - <= MethodCall(0,1) { Path=>/org/freedesktop/DBus, Interface=>org.freedesktop.DBus, Member=>Hello, Destination=>org.freedesktop.DBus } { }
[Sender] INFO org.freedesktop.dbus.MessageWriter - <= MethodCall(0,3) { Path=>/Dc/0/Voltage, Interface=>javadbus.BusItem, Member=>GetValue, Destination=>com.victronenergy.solarcharger.ttyUSB0 } { }
Exception in thread "main" org.freedesktop.DBus$Error$UnknownMethod: Method "GetValue" with signature "" on interface "javadbus.BusItem" doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.freedesktop.dbus.Error.getException(Error.java:141)
at org.freedesktop.dbus.Error.throwException(Error.java:171)
at org.freedesktop.dbus.RemoteInvocationHandler.executeRemoteMethod(RemoteInvocationHandler.java:158)
at org.freedesktop.dbus.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:222)
at com.sun.proxy.$Proxy1.GetValue(Unknown Source)
at javadbus.VictronEnergyDBusSolarCharger.getData(VictronEnergyDBusSolarCharger.java:28)
at javadbus.VictronEnergyDBusSolarCharger.getDcV(VictronEnergyDBusSolarCharger.java:38)
at javadbus.MainClass.main(MainClass.java:7)
Is it necessary to make a implementation of this Method GetValue? But why e.g. how should I do this? I only want to get this Information and not provide it like a Server.
Why was it a big task to get all dependencies?
dbus-java library and dependencies are all available at maven central, so a proper maven project should just work out-of-the-box.
Back to topic:
You don't have to implement GetValue(), but you need a suitable java interface for BusItem.
As far as I can see in the documentation of victronenergy (https://www.victronenergy.com/live/open_source:ccgx:d-bus) , your interface is not correct.
You provide SetDefault()/GetDefault() methods, which are only available on com.victronenergy.settings Objects, but you want to retrieve a com.victronenergy.BusItem (no part of the com.victronenergy.settings package).
This is one error. The second error is: you use the wrong package name for your BusItem class.
In your case DBus will try to resolve an object with the path javadbus.BusItem which is not provided by the connected BusAddress com.victronenergy.solarcharger.ttyUSB0.
The BusItem class has to be in package com.victronenergy or you have to use the annotation #DBusInterfaceName("com.victronenergy.BusItem").
The annotation will tell the DBus library to ignore the java package/class name and use the one provided in the annotation.
The Inteface BusItem had been created by CreateInterface-Script from https://dbus.freedesktop.org/doc/dbus-java/dbus-java/dbus-javase10.html and the XML from Introspect()
But you solved my real problem. I used the annotation #DBusInterfaceName("com.victronenergy.BusItem") now. No Exception anymore an i get data from my solarcharger. Thank you so much!
I am trying several examples from Jest to use as a POC for ElasticSearch integration.
Right now, I am trying just a basic GET. I created a POJO called Document. In there are some basic setters and getters are some fields. I populate it and then use GSON to generate the JSON text.
From this generated JSON, I go into ElasticSearch Sense and do the following:
PUT /reports/documents/3
{
// JSON code
}
This generates just fine. I then try using Get to pull the values out from Java, like so:
JestClientFactory factory = new JestClientFactory();
factory.setHttpClientConfig(new HttpClientConfig
.Builder("http://localhost:9200")
.multiThreaded(true)
.build());
client = factory.getObject();
Get get = new Get.Builder("reports", "3").type("documents").build();
try {
JestResult result = client.execute(get);
String json = result.getJsonString();
System.out.println(json);
Document doc = null;
doc = result.getSourceAsObject(Document.class);
System.out.println("is doc null? " + doc == null);
}catch (Exception e) {
System.err.println("Error getting document");
e.printStackTrace();
}
The String json returns what I would expect (showing _index, _type, _id and of course _source). However, doc always comes out as NULL. I am not sure why that is happening.
Just to see if this was just a Get problem, I proceeded to try to Search.I did the following code snippet:
try {
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.matchQuery("reportNumber", "101221895CRT-004"));
Search search = new Search.Builder(searchSourceBuilder.toString())
// multiple index or types can be added.
.addIndex("reports")
.addType("documents")
.build();
SearchResult result = client.execute(search);
//List<Document> results = result.getSourceAsObjectList(Document.class);
List<SearchResult.Hit<Document, Void>> hits = result.getHits(Document.class);
for (SearchResult.Hit hit : hits) {
Document source = (Document) hit.source;
Void ex = (Void) hit.explanation;
System.out.println();
}
System.out.println("Result size: " + hits.size());
}catch (Exception e) {
System.err.println("Error searching");
e.printStackTrace();
}
When looking at result, the JSON of the object is shown. However, the List<Document> results comes out as NULL. When using hits, the size of hits is correct, but the "source" and "ex" are both NULL.
Any ideas on what I am doing wrong with this?
UPDATE
After reading Cihat's comment, I went ahead and added in logging. It turns out I am getting an error when trying to convert a date (hence why it's always coming back as NULL).
I get the following error message:
Unhandled exception occurred while converting source to the object .com.someCompanyName.data.Document
com.google.gson.JsonSyntaxException: java.text.ParseException: Unparseable date: "Nov 6, 2014 8:29:00 AM"
I have tried all different formats:
11/06/2014 8:29:00 AM (and without time and making year just 14)
06-NOV-2014 8:29:00 AM (and without time and making year just 14)
2014-11-06 8:29:00 AM (same thing with time and year changes)
2014-NOV-06 8:29:00 AM (same thing with time and year changes)
06/11/2014 8:29:00 AM (same thing)
All of those failed. I am sure I tried some other formats, so not sure what format the date should be in. I even tried the exact date from DateFormat JavaDocs and it still failed. Every time I do a search, it says to define the Dateformat in the GsonBuilder, but in Jest I do not have access to that.
This test case demonstrates indexing a document with Jest and then getting the same document back out. Not a complete answer, but hopefully it is useful to see something that is known to work.
import io.searchbox.client.JestClient;
import io.searchbox.client.JestClientFactory;
import io.searchbox.client.JestResult;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.core.Get;
import io.searchbox.core.Index;
import static org.hamcrest.Matchers.*;
import static org.hamcrest.MatcherAssert.*;
import org.junit.Test;
public class JestRoundtripIT {
public static final String INDEX = "reports";
public static final String TYPE = "documents";
public static final String ID = "3";
#Test
public void documentRoundTrip() throws Exception {
JestClientFactory factory = new JestClientFactory();
factory.setHttpClientConfig(new HttpClientConfig
.Builder("http://localhost:9200")
.multiThreaded(true)
.build());
JestClient client = factory.getObject();
Document original = new Document()
.withAuthor("Shay Banon")
.withContent("You know, for search...");
JestResult indexResult = client.execute(
new Index.Builder(original)
.index(INDEX)
.type(TYPE)
.id(ID)
.build());
assertThat(indexResult.isSucceeded(), equalTo(true));
JestResult getResult = client.execute(
new Get.Builder(INDEX, ID)
.type(TYPE)
.build());
assertThat(getResult.isSucceeded(), equalTo(true));
Document fromEs = getResult.getSourceAsObject(Document.class);
assertThat(fromEs, notNullValue());
assertThat(fromEs.getAuthor(), equalTo(original.getAuthor()));
assertThat(fromEs.getContent(), equalTo(original.getContent()));
}
public static class Document {
protected String author;
protected String content;
public Document withAuthor( String author ) {
this.author = author;
return this;
}
public Document withContent( String content ) {
this.content = content;
return this;
}
public String getAuthor() {
return author;
}
public void setAuthor( String author ) {
this.author = author;
}
public String getContent() {
return content;
}
public void setContent( String content ) {
this.content = content;
}
}
}
In a Java servlet container (preferably Tomcat, but if this can be done in a different container then say so) I desire something which is theoretically possible. My question here is whether tools exist to support it, and if so what tools (or what names I should research further).
Here is my problem: in one servlet container I want to run a large number of different WAR files. They share some large common libraries (such as Spring). At first blush, I have two unacceptable alternatives:
Include the large library (Spring, for example) in each WAR file. This is unacceptable because it will load a large number of copies of Spring, exhausting the memory on the server.
Place the large library in the container classpath. Now all of the WAR files share one instance of the library (good). But this is unacceptable because I cannot upgrade the Spring version without upgrading ALL of the WAR files at once, and such a large change is difficult verging on impossible.
In theory, though, there is an alternative which could work:
Put each version of the large library into the container-level classpath. Do some container level magic so that each WAR file declares which version it wishes to use and it will find that on its classpath.
The "magic" must be done at the container level (I think) because this can only be achieved by loading each version of the library with a different classloader, then adjusting what classloaders are visible to each WAR file.
So, have you ever heard of doing this? If so, how? Or tell me what it is called so I can research further.
Regarding Tomcat, for the 7th version you can use VirtualWebappLocader like so
<Context>
<Loader className="org.apache.catalina.loader.VirtualWebappLoader"
virtualClasspath="/usr/shared/lib/spring-3/*.jar,/usr/shared/classes" />
</Context>
For the 8th version Pre- & Post- Resources should be used instead
<Context>
<Resources>
<PostResources className="org.apache.catalina.webresources.DirResourceSet"
base="/usr/shared/lib/spring-3" webAppMount="/WEB-INF/lib" />
<PostResources className="org.apache.catalina.webresources.DirResourceSet"
base="/usr/shared/classes" webAppMount="/WEB-INF/classes" />
</Resources>
</Context>
Don't forget to put the corresponding context.xml into the META-INF of your webapp.
For the jetty as well as other containers the same technique may be used.
The only difference is in how to specify extra classpath elements for the webapp.
UPDATE
The samples above does not share the loaded classes, but the idea is the same - use custom classloader. Here is just the pretty ugly sample that also tries to prevent classloader leaks during undeployment.
SharedWebappLoader
package com.foo.bar;
import org.apache.catalina.LifecycleException;
import org.apache.catalina.loader.WebappLoader;
public class SharedWebappLoader extends WebappLoader {
private String pathID;
private String pathConfig;
static final ThreadLocal<ClassLoaderFactory> classLoaderFactory = new ThreadLocal<>();
public SharedWebappLoader() {
this(null);
}
public SharedWebappLoader(ClassLoader parent) {
super(parent);
setLoaderClass(SharedWebappClassLoader.class.getName());
}
public String getPathID() {
return pathID;
}
public void setPathID(String pathID) {
this.pathID = pathID;
}
public String getPathConfig() {
return pathConfig;
}
public void setPathConfig(String pathConfig) {
this.pathConfig = pathConfig;
}
#Override
protected void startInternal() throws LifecycleException {
classLoaderFactory.set(new ClassLoaderFactory(pathConfig, pathID));
try {
super.startInternal();
} finally {
classLoaderFactory.remove();
}
}
}
SharedWebappClassLoader
package com.foo.bar;
import org.apache.catalina.LifecycleException;
import org.apache.catalina.loader.ResourceEntry;
import org.apache.catalina.loader.WebappClassLoader;
import java.net.URL;
public class SharedWebappClassLoader extends WebappClassLoader {
public SharedWebappClassLoader(ClassLoader parent) {
super(SharedWebappLoader.classLoaderFactory.get().create(parent));
}
#Override
protected ResourceEntry findResourceInternal(String name, String path) {
ResourceEntry entry = super.findResourceInternal(name, path);
if(entry == null) {
URL url = parent.getResource(name);
if (url == null) {
return null;
}
entry = new ResourceEntry();
entry.source = url;
entry.codeBase = entry.source;
}
return entry;
}
#Override
public void stop() throws LifecycleException {
ClassLoaderFactory.removeLoader(parent);
}
}
ClassLoaderFactory
package com.foo.bar;
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
public class ClassLoaderFactory {
private static final class ConfigKey {
private final String pathConfig;
private final String pathID;
private ConfigKey(String pathConfig, String pathID) {
this.pathConfig = pathConfig;
this.pathID = pathID;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ConfigKey configKey = (ConfigKey) o;
if (pathConfig != null ? !pathConfig.equals(configKey.pathConfig) : configKey.pathConfig != null)
return false;
if (pathID != null ? !pathID.equals(configKey.pathID) : configKey.pathID != null) return false;
return true;
}
#Override
public int hashCode() {
int result = pathConfig != null ? pathConfig.hashCode() : 0;
result = 31 * result + (pathID != null ? pathID.hashCode() : 0);
return result;
}
}
private static final Map<ConfigKey, ClassLoader> loaders = new HashMap<>();
private static final Map<ClassLoader, ConfigKey> revLoaders = new HashMap<>();
private static final Map<ClassLoader, Integer> usages = new HashMap<>();
private final ConfigKey key;
public ClassLoaderFactory(String pathConfig, String pathID) {
this.key = new ConfigKey(pathConfig, pathID);
}
public ClassLoader create(ClassLoader parent) {
synchronized (loaders) {
ClassLoader loader = loaders.get(key);
if(loader != null) {
Integer usageCount = usages.get(loader);
usages.put(loader, ++usageCount);
return loader;
}
Properties props = new Properties();
try (InputStream is = new BufferedInputStream(new FileInputStream(key.pathConfig))) {
props.load(is);
} catch (IOException e) {
throw new RuntimeException(e);
}
String libsStr = props.getProperty(key.pathID);
String[] libs = libsStr.split(File.pathSeparator);
URL[] urls = new URL[libs.length];
try {
for(int i = 0, len = libs.length; i < len; i++) {
urls[i] = new URL(libs[i]);
}
} catch (MalformedURLException e) {
throw new RuntimeException(e);
}
loader = new URLClassLoader(urls, parent);
loaders.put(key, loader);
revLoaders.put(loader, key);
usages.put(loader, 1);
return loader;
}
}
public static void removeLoader(ClassLoader parent) {
synchronized (loaders) {
Integer val = usages.get(parent);
if(val > 1) {
usages.put(parent, --val);
} else {
usages.remove(parent);
ConfigKey key = revLoaders.remove(parent);
loaders.remove(key);
}
}
}
}
context.xml of the first app
<Context>
<Loader className="com.foo.bar.SharedWebappLoader"
pathConfig="${catalina.base}/conf/shared.properties"
pathID="commons_2_1"/>
</Context>
context.xml of the second app
<Context>
<Loader className="com.foo.bar.SharedWebappLoader"
pathConfig="${catalina.base}/conf/shared.properties"
pathID="commons_2_6"/>
</Context>
$TOMCAT_HOME/conf/shared.properties
commons_2_1=file:/home/xxx/.m2/repository/commons-lang/commons-lang/2.1/commons-lang-2.1.jar
commons_2_6=file:/home/xxx/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
I was able to implement this for Tomcat (Tested on Tomcat 7.0.52). My solution involves implementing custom version of WebAppLoader which extends standard Tomcat's WebAppLoader. Thanks to this solution you can pass custom classloader to load classes for each of web application.
To utilize this new loader you need to declare it for each application (either in Context.xml file placed in each war or in Tomcat's server.xml file). This loader takes an extra custom parameter webappName which is later passed to LibrariesStorage class to determine which libraries should be used by which application.
<Context path="/pl-app" >
<Loader className="web.DynamicWebappLoader" webappName="pl-app"/>
</Context>
<Context path="/my-webapp" >
<Loader className="web.DynamicWebappLoader" webappName="myApplication2"/>
</Context>
Once this is defined you need to install this DynamicWebappLoader to Tomcat. To do this copy all copiled classes to lib directory of Tomcat (so you should have following files [tomcat dir]/lib/web/DynamicWebappLoader.class, [tomcat dir]/lib/web/LibrariesStorage.class, [tomcat dir]/lib/web/LibraryAndVersion.class, [tomcat dir]/lib/web/WebAppAwareClassLoader.class).
You need also to download xbean-classloader-4.0.jar and place it in Tomcat's lib dir (so you should have [tomcat dir]/lib/xbean-classloader-4.0.jar. NOTE:xbean-classloader provides special implementation of classloader (org.apache.xbean.classloader.JarFileClassLoader) which allowes to load needed jars at runtime.
Main trick is made in LibraryStorgeClass (full implementation is at the end). It stores a mapping between each application (defined by webappName) and libraries which this application is allowed to load. In current implementation this is hardcoded, but this can be rewritten to dynamically generate list of libs needed by each application. Each library has its own instance of JarFileClassLoader which ensures that each library is only loaded one time (the mapping between library and its classloader is stored in static field "libraryToClassLoader", so this mapping is the same for every web application because of static nature of the field)
class LibrariesStorage {
private static final String JARS_DIR = "D:/temp/idea_temp_proj2_/some_jars";
private static Map<LibraryAndVersion, JarFileClassLoader> libraryToClassLoader = new HashMap<>();
private static Map<String, List<LibraryAndVersion>> webappLibraries = new HashMap<>();
static {
try {
addLibrary("commons-lang3", "3.3.2", "commons-lang3-3.3.2.jar"); // instead of this lines add some intelligent directory scanner which will detect all jars and their versions in JAR_DIR
addLibrary("commons-lang3", "3.3.1", "commons-lang3-3.3.1.jar");
addLibrary("commons-lang3", "3.3.0", "commons-lang3-3.3.0.jar");
mapApplicationToLibrary("pl-app", "commons-lang3", "3.3.2"); // instead of manually mapping application to library version, some more intelligent code should be here (for example you can scann Web-Inf/lib of each application and detect needed jars
mapApplicationToLibrary("myApplication2", "commons-lang3", "3.3.0");
(...)
}
In above example, suppose that in directory with all the jars (defined here by JARS_DIR) we have only a commons-lang3-3.3.2.jar file. This would mean that application identified by "pl-app" name (the name comes from webappName attribute in tag in Context.xml as mentioned above) will be able to load classes from commons-lang jar. Application identified by "myApplication2" will get ClassNotFoundException at this point because it has access only to commons-lang3-3.3.0.jar, but this file is not present in JARS_DIR directory.
Full implementation here:
package web;
import org.apache.catalina.loader.WebappLoader;
import org.apache.xbean.classloader.JarFileClassLoader;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class DynamicWebappLoader extends WebappLoader {
private String webappName;
private WebAppAwareClassLoader webAppAwareClassLoader;
public static final ThreadLocal lastCreatedClassLoader = new ThreadLocal();
public DynamicWebappLoader() {
super(new WebAppAwareClassLoader(Thread.currentThread().getContextClassLoader()));
webAppAwareClassLoader = (WebAppAwareClassLoader) lastCreatedClassLoader.get(); // unfortunately I did not find better solution to access new instance of WebAppAwareClassLoader created in previous line so I passed it via thread local
lastCreatedClassLoader.remove();
}
// (this method is called by Tomcat because of Loader attribute in Context.xml - <Context> <Loader className="..." webappName="myApplication2"/> )
public void setWebappName(String name) {
System.out.println("Setting webapp name: " + name);
this.webappName = name;
webAppAwareClassLoader.setWebAppName(name); // pass web app name to ClassLoader
}
}
class WebAppAwareClassLoader extends ClassLoader {
private String webAppName;
public WebAppAwareClassLoader(ClassLoader parent) {
super(parent);
DynamicWebappLoader.lastCreatedClassLoader.set(this); // store newly created instance in ThreadLocal .. did not find better way to access the reference later in code
}
#Override
public Class<?> loadClass(String className) throws ClassNotFoundException {
System.out.println("Load class: " + className + " for webapp: " + webAppName);
try {
return LibrariesStorage.loadClassForWebapp(webAppName, className);
} catch (ClassNotFoundException e) {
System.out.println("JarFileClassLoader did not find class: " + className + " " + e.getMessage());
return super.loadClass(className);
}
}
public void setWebAppName(String webAppName) {
this.webAppName = webAppName;
}
}
class LibrariesStorage {
private static final String JARS_DIR = "D:/temp/idea_temp_proj2_/some_jars";
private static Map<LibraryAndVersion, JarFileClassLoader> libraryToClassLoader = new HashMap<>();
private static Map<String, List<LibraryAndVersion>> webappLibraries = new HashMap<>();
static {
try {
addLibrary("commons-lang3", "3.3.2", "commons-lang3-3.3.2.jar"); // instead of this lines add some intelligent directory scanner which will detect all jars and their versions in JAR_DIR
addLibrary("commons-lang3", "3.3.1", "commons-lang3-3.3.1.jar");
addLibrary("commons-lang3", "3.3.0", "commons-lang3-3.3.0.jar");
mapApplicationToLibrary("pl-app", "commons-lang3", "3.3.2"); // instead of manually mapping application to library version, some more intelligent code should be here (for example you can scann Web-Inf/lib of each application and detect needed jars
mapApplicationToLibrary("myApplication2", "commons-lang3", "3.3.0");
} catch (MalformedURLException e) {
throw new RuntimeException(e.getMessage(), e);
}
}
private static void mapApplicationToLibrary(String applicationName, String libraryName, String libraryVersion) {
LibraryAndVersion libraryAndVersion = new LibraryAndVersion(libraryName, libraryVersion);
if (!webappLibraries.containsKey(applicationName)) {
webappLibraries.put(applicationName, new ArrayList<LibraryAndVersion>());
}
webappLibraries.get(applicationName).add(libraryAndVersion);
}
private static void addLibrary(String libraryName, String libraryVersion, String filename)
throws MalformedURLException {
LibraryAndVersion libraryAndVersion = new LibraryAndVersion(libraryName, libraryVersion);
URL libraryLocation = new File(JARS_DIR + File.separator + filename).toURI().toURL();
libraryToClassLoader.put(libraryAndVersion,
new JarFileClassLoader("JarFileClassLoader for lib: " + libraryAndVersion,
new URL[] { libraryLocation }));
}
private LibrariesStorage() {
}
public static Class<?> loadClassForWebapp(String webappName, String className) throws ClassNotFoundException {
System.out.println("Loading class: " + className + " for web application: " + webappName);
List<LibraryAndVersion> webappLibraries = LibrariesStorage.webappLibraries.get(webappName);
for (LibraryAndVersion libraryAndVersion : webappLibraries) {
JarFileClassLoader libraryClassLoader = libraryToClassLoader.get(libraryAndVersion);
try {
return libraryClassLoader.loadClass(className); // ok current lib contained class to load
} catch (ClassNotFoundException e) {
// ok.. continue in loop... try to load the class from classloader connected to next library
}
}
throw new ClassNotFoundException("Class " + className + " was not found in any jar connected to webapp: " +
webappLibraries);
}
}
class LibraryAndVersion {
private final String name;
private final String version;
LibraryAndVersion(String name, String version) {
this.name = name;
this.version = version;
}
#Override
public boolean equals(Object o) {
if (this == o) {
return true;
}
if ((o == null) || (getClass() != o.getClass())) {
return false;
}
LibraryAndVersion that = (LibraryAndVersion) o;
if ((name != null) ? (!name.equals(that.name)) : (that.name != null)) {
return false;
}
if ((version != null) ? (!version.equals(that.version)) : (that.version != null)) {
return false;
}
return true;
}
#Override
public int hashCode() {
int result = (name != null) ? name.hashCode() : 0;
result = (31 * result) + ((version != null) ? version.hashCode() : 0);
return result;
}
#Override
public String toString() {
return "LibraryAndVersion{" +
"name='" + name + '\'' +
", version='" + version + '\'' +
'}';
}
}
JBoss has a framework called Modules that solves this problem. You can save the shared library with its version and reference it from your war-file.
I have no idea if it works on Tomcat, but it works as a charm on Wildfly.