I am a beginner who just started developing pulsar-client with spring boot.
First of all, I learned the basics through pulsar doc and git, but I was stuck testing batch transmission of messages from the pulsar-client producer.
In particular, I want to send JsonArray data in batches, but I keep getting a JsonArray.getAsInt error.
Please take a look at my code and tell me what's wrong
package com.refactorizando.example.pulsar.producer;
import static java.util.stream.Collectors.toList;
import com.refactorizando.example.pulsar.config.PulsarConfiguration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.stream.IntStream;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import net.sf.json.JSONArray;
import org.apache.pulsar.client.api.CompressionType;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.impl.schema.JSONSchema;
import org.apache.pulsar.shade.com.google.gson.JsonArray;
import org.apache.pulsar.shade.com.google.gson.JsonElement;
import org.apache.pulsar.shade.com.google.gson.JsonObject;
import org.apache.pulsar.shade.com.google.gson.JsonParser;
import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;
#Component
#RequiredArgsConstructor
#Slf4j
public class PulsarProducer {
private static final String TOPIC_NAME = "Json_Test";
private final PulsarClient client;
#Bean(name = "producer")
public void producer() throws PulsarClientException {
// batching
Producer<JsonArray> producer = client.newProducer(JSONSchema.of(JsonArray.class))
.topic(TOPIC_NAME)
.batchingMaxPublishDelay(60, TimeUnit.SECONDS)
.batchingMaxMessages(2)
.enableBatching(true)
.compressionType(CompressionType.LZ4)
.create();
String data = "{'users': [{'userId': 1,'firstName': 'AAAAA'},{'userId': 2,'firstName': 'BBBB'},{'userId': 3,'firstName': 'CCCCC'},{'userId': 4,'firstName': 'DDDDD'},{'userId': 5,'firstName': 'EEEEE'}]}";
JsonElement element = JsonParser.parseString(data);
JsonObject obj = element.getAsJsonObject();
JsonArray arr = obj.getAsJsonArray("users");
try {
producer.send(arr);
} catch (Exception e) {
log.error("Error sending mesasage");
e.printStackTrace();
}
producer.close();
}
}
I'm still a beginner developer, so I couldn't find it on stackOverflow because I couldn't search well. If you have anything related to it, please leave a link and I'll delete the question.
Thanks for reading my question and have a nice day!
I tried several things, such as converting to JsonObject and sending, converting to String and sending, etc., but the same error came out.
cho ,
Welcome to Pulsar and Spring Pulsar! I believe there are a few things to cover to fully answer your question.
Spring Pulsar Usage
In your example you are crafting a Producer directly from the PulsarClient. There is absolutely nothing wrong/bad about using that API directly. However, if you want to use Spring Pulsar, the recommended approach to send messages in a Spring Boot app using Spring Pulsar is via the auto-configured PulsarTemplate (or ReactivePulsarTemplate if using Reactive). It simplifies usage and allows configuring the template/producer using configuration properties. For example, instead of building up and then using Producer.send() you would instead inject the pulsar template and use it as follows:
pulsarTemplate.newMessage(foo)
.withTopic("Json_Test")
.withSchema(Schema.JSON(Foo.class))
.withProducerCustomizer((producerBuilder) -> {
producerBuilder
.batchingMaxPublishDelay(60, TimeUnit.SECONDS)
.batchingMaxMessages(2)
.enableBatching(true)
.compressionType(CompressionType.LZ4);
})
.send();
Furthermore you can replace the builder calls w/ configuration properties like:
spring:
pulsar:
producer:
batching-enabled: true
batching-max-publish-delay: 60s
batching-max-messages: 2
compression-type: lz4
and then your code becomes:
pulsarTemplate.newMessage(foo)
.withTopic("Json_Test")
.withSchema(Schema.JSON(Foo.class))
.send();
NOTE: I replace json array w/ Foo for simplicity.
Schemas
In Pulsar, the Schema knows how to de/serialize the data. The built-in Pulsar Schema.JSON by default uses the Jackson json lib to de/serialize the data. This requires that the data must be able to be handled by Jackson ObjectMapper.readValue/writeValue methods. It handles POJOs really well, but does not handle the JSON impl you are using.
I noticed the latest json-lib is 2.4 and (AFAICT) has 9 CVEs against it and was last released in 2010. If I had to use a Json level API for my data I would pick a more contemporary and well supported / used lib such as Jackson or Gson.
I switched your sample to use Jackson ArrayNode and it worked well. I did have to replace the single quotes in your data string to backslash double-quote as Jackson by default does not like single-quoted data. Here is the re-worked sample app using Jackson ArrayNode:
#SpringBootApplication
public class HyunginChoSpringPulsarUserApp {
public static void main(String[] args) {
SpringApplication.run(HyunginChoSpringPulsarUserApp.class, args);
}
#Bean
ApplicationRunner sendDataOnStartup(PulsarTemplate<ArrayNode> pulsarTemplate) {
return (args) -> {
String data2 = "{\"users\": [{\"userId\": 1,\"firstName\": \"AAAAA\"},{\"userId\": 2,\"firstName\": \"BBBB\"},{\"userId\": 3,\"firstName\": \"CCCCC\"},{\"userId\": 4,\"firstName\": \"DDDDD\"},{\"userId\": 5,\"firstName\": \"EEEEE\"}]}";
ArrayNode jsonArray = (ArrayNode) ObjectMapperFactory.create().readTree(data2).get("users");
System.out.printf("*** SENDING: %s%n", jsonArray);
pulsarTemplate.newMessage(jsonArray)
.withTopic("Json_Test")
.withSchema(Schema.JSON(ArrayNode.class))
.send();
};
}
#PulsarListener(topics = "Json_Test", schemaType = SchemaType.JSON, batch = true)
public void listenForData(List<ArrayNode> user) {
System.out.printf("***** LISTEN: %s%n".formatted(user));
}
}
The output looks like:
*** SENDING: [{"userId":1,"firstName":"AAAAA"},{"userId":2,"firstName":"BBBB"},{"userId":3,"firstName":"CCCCC"},{"userId":4,"firstName":"DDDDD"},{"userId":5,"firstName":"EEEEE"}]
***** LISTEN: [{"userId":1,"firstName":"AAAAA"},{"userId":2,"firstName":"BBBB"},{"userId":3,"firstName":"CCCCC"},{"userId":4,"firstName":"DDDDD"},{"userId":5,"firstName":"EEEEE"}]
Data Model
Your data is an array of users. Do you have a requirement to use a Json level API or you instead deal with a List<User> POJOs? This would simplify things and make it much better experience to use. The Java record is a great choice such as:
public record(String userId, String firstName) {}
then you can pass in a List<User> to your PulsarTemplate and everything will work well. For example:
#SpringBootApplication
public class HyunginChoSpringPulsarUserApp {
public static void main(String[] args) {
SpringApplication.run(HyunginChoSpringPulsarUserApp.class, args);
}
#Bean
ApplicationRunner sendDataOnStartup(PulsarTemplate<User> pulsarTemplate) {
return (args) -> {
String data2 = "{\"users\": [{\"userId\": 1,\"firstName\": \"AAAAA\"},{\"userId\": 2,\"firstName\": \"BBBB\"},{\"userId\": 3,\"firstName\": \"CCCCC\"},{\"userId\": 4,\"firstName\": \"DDDDD\"},{\"userId\": 5,\"firstName\": \"EEEEE\"}]}";
ObjectMapper objectMapper = ObjectMapperFactory.create();
JsonNode usersNode = objectMapper.readTree(data2).get("users");
List<User> users = objectMapper.convertValue(usersNode, new TypeReference<>() {});
System.out.printf("*** SENDING: %s%n", users);
for (User user : users) {
pulsarTemplate.newMessage(user)
.withTopic("Json_Test2")
.withSchema(Schema.JSON(User.class))
.send();
}
};
}
#PulsarListener(topics = "Json_Test2", schemaType = SchemaType.JSON, batch = true)
public void listenForData(List<User> users) {
users.forEach((user) -> System.out.printf("***** LISTEN: %s%n".formatted(user)));
}
public record User(String userId, String firstName) {}
}
*** SENDING: [User[userId=1, firstName=AAAAA], User[userId=2, firstName=BBBB], User[userId=3, firstName=CCCCC], User[userId=4, firstName=DDDDD], User[userId=5, firstName=EEEEE]]
...
***** LISTEN: User[userId=1, firstName=AAAAA]
***** LISTEN: User[userId=2, firstName=BBBB]
***** LISTEN: User[userId=3, firstName=CCCCC]
***** LISTEN: User[userId=4, firstName=DDDDD]
***** LISTEN: User[userId=5, firstName=EEEEE]
I hope this helps. Take care.
Related
I have a Kafka Topic wit JSON data. Now im trying to send those JSON strings to an ES topic using the new "Java API Client" (https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/7.17/index.html), but im running into a parser exception:
co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/index] failed: [mapper_parsing_exception] failed to parse
at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:281)
at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
at co.elastic.clients.elasticsearch.ElasticsearchClient.index(ElasticsearchClient.java:953)
This exception occurs in the last line of the following code:
final IndexRequest<String> request =
new IndexRequest.Builder<String>()
.index("myIndex")
.id(String.valueOf(UUID.randomUUID()))
.document(consumerRecord.value()) //already serialized json data
.build();
elasticsearchClient.index(request);
As far as I understand this exception occurs, because the ES client tries to serialize the data im providing, which is already serialized, resulting in a malformed JSON string.
Is there anyway to get around this and just send simple JSON strings? Also I believe this was possible with the earlier "Low Level Java Library", right? And yes, I know there are ways to allow communication between Kafka and ES without writing a Consumer.
Thanks for any hints.
If you use a JacksonJsonpMapper when creating your ElasticsearchTransport, you can use a custom PreserializedJson class to send already-serialized JSON.
ElasticsearchTransport transport = new RestClientTransport(
createLowLevelRestClient(), // supply your own!
new JacksonJsonpMapper()
);
ElasticsearchClient client = new ElasticsearchClient(transport);
IndexResponse response = client.index(indexReq -> indexReq
.index("my-index")
.id("docId")
.document(new PreserializedJson("{\"foo\":\"bar\"}"))
);
System.out.println(response);
Here is the source for PreserializedJson:
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.ser.std.StdSerializer;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import static java.util.Objects.requireNonNull;
#JsonSerialize(using = PreserializedJson.Serializer.class)
public class PreserializedJson {
private final String value;
public PreserializedJson(String value) {
this.value = requireNonNull(value);
}
public PreserializedJson(byte[] value) {
this(new String(value, StandardCharsets.UTF_8));
}
public static class Serializer extends StdSerializer<PreserializedJson> {
public Serializer() {
super(PreserializedJson.class);
}
#Override
public void serialize(PreserializedJson value, JsonGenerator gen, SerializerProvider provider) throws IOException {
gen.writeRaw(value.value);
}
}
}
I solved the problem by substituting "Java API Client" (https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/introduction.html) with "Java Low Level Rest Client" (https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/java-rest-low.html).
This Library allows sending of arbitrary JSON-Strings to ES:
final Request request = new Request("POST", "/twitter/_doc");
request.setJsonEntity(record.value());
restClient.performRequest(request);
With the new API Client, you can natively insert raw json into it.
As specified here : Using raw json data
IndexRequest<JsonData> request = IndexRequest.of(i -> i
.index("logs")
.withJson(input)
);
I am trying to create a simple application where the app will consume Kafka message do some cql transform and publish to Kafka and below is the code:
JAVA: 1.8
Flink: 1.13
Scala: 2.11
flink-siddhi: 2.11-0.2.2-SNAPSHOT
I am using library: https://github.com/haoch/flink-siddhi
input json to Kafka:
{
"awsS3":{
"ResourceType":"aws.S3",
"Details":{
"Name":"crossplane-test",
"CreationDate":"2020-08-17T11:28:05+00:00"
},
"AccessBlock":{
"PublicAccessBlockConfiguration":{
"BlockPublicAcls":true,
"IgnorePublicAcls":true,
"BlockPublicPolicy":true,
"RestrictPublicBuckets":true
}
},
"Location":{
"LocationConstraint":"us-west-2"
}
}
}
main class:
public class S3SidhiApp {
public static void main(String[] args) {
internalStreamSiddhiApp.start();
//kafkaStreamApp.start();
}
}
App class:
package flinksidhi.app;
import com.google.gson.JsonObject;
import flinksidhi.event.s3.source.S3EventSource;
import io.siddhi.core.SiddhiManager;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.core.fs.FileSystem;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;
import org.apache.flink.streaming.siddhi.SiddhiCEP;
import org.json.JSONObject;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import static flinksidhi.app.connector.Consumers.createInputMessageConsumer;
import static flinksidhi.app.connector.Producer.*;
public class internalStreamSiddhiApp {
private static final String inputTopic = "EVENT_STREAM_INPUT";
private static final String outputTopic = "EVENT_STREAM_OUTPUT";
private static final String consumerGroup = "EVENT_STREAM1";
private static final String kafkaAddress = "localhost:9092";
private static final String zkAddress = "localhost:2181";
private static final String S3_CQL1 = "from inputStream select * insert into temp";
private static final String S3_CQL = "from inputStream select json:toObject(awsS3) as obj insert into temp;" +
"from temp select json:getString(obj,'$.awsS3.ResourceType') as affected_resource_type," +
"json:getString(obj,'$.awsS3.Details.Name') as affected_resource_name," +
"json:getString(obj,'$.awsS3.Encryption.ServerSideEncryptionConfiguration') as encryption," +
"json:getString(obj,'$.awsS3.Encryption.ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault.SSEAlgorithm') as algorithm insert into temp2; " +
"from temp2 select affected_resource_name,affected_resource_type, " +
"ifThenElse(encryption == ' ','Fail','Pass') as state," +
"ifThenElse(encryption != ' ' and algorithm == 'aws:kms','None','Critical') as severity insert into outputStream";
public static void start(){
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//DataStream<String> inputS = env.addSource(new S3EventSource());
//Flink kafka stream consumer
FlinkKafkaConsumer<String> flinkKafkaConsumer =
createInputMessageConsumer(inputTopic, kafkaAddress,zkAddress, consumerGroup);
//Add Data stream source -- flink consumer
DataStream<String> inputS = env.addSource(flinkKafkaConsumer);
SiddhiCEP cep = SiddhiCEP.getSiddhiEnvironment(env);
cep.registerExtension("json:toObject", io.siddhi.extension.execution.json.function.ToJSONObjectFunctionExtension.class);
cep.registerExtension( "json:getString", io.siddhi.extension.execution.json.function.GetStringJSONFunctionExtension.class);
cep.registerStream("inputStream", inputS, "awsS3");
inputS.print();
System.out.println(cep.getDataStreamSchemas());
//json needs extension jars to present during runtime.
DataStream<Map<String,Object>> output = cep
.from("inputStream")
.cql(S3_CQL1)
.returnAsMap("temp");
//Flink kafka stream Producer
FlinkKafkaProducer<Map<String, Object>> flinkKafkaProducer =
createMapProducer(env,outputTopic, kafkaAddress);
//Add Data stream sink -- flink producer
output.addSink(flinkKafkaProducer);
output.print();
try {
env.execute();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Consumer class:
package flinksidhi.app.connector;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.json.JSONObject;
import java.util.Properties;
public class Consumers {
public static FlinkKafkaConsumer<String> createInputMessageConsumer(String topic, String kafkaAddress, String zookeeprAddr, String kafkaGroup ) {
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", kafkaAddress);
properties.setProperty("zookeeper.connect", zookeeprAddr);
properties.setProperty("group.id",kafkaGroup);
FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<String>(
topic,new SimpleStringSchema(),properties);
return consumer;
}
}
Producer class:
package flinksidhi.app.connector;
import flinksidhi.app.util.ConvertJavaMapToJson;
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;
import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
import org.json.JSONObject;
import java.util.Map;
public class Producer {
public static FlinkKafkaProducer<Tuple2> createStringProducer(StreamExecutionEnvironment env, String topic, String kafkaAddress) {
return new FlinkKafkaProducer<Tuple2>(kafkaAddress, topic, new AverageSerializer());
}
public static FlinkKafkaProducer<Map<String,Object>> createMapProducer(StreamExecutionEnvironment env, String topic, String kafkaAddress) {
return new FlinkKafkaProducer<Map<String,Object>>(kafkaAddress, topic, new SerializationSchema<Map<String, Object>>() {
#Override
public void open(InitializationContext context) throws Exception {
}
#Override
public byte[] serialize(Map<String, Object> stringObjectMap) {
String json = ConvertJavaMapToJson.convert(stringObjectMap);
return json.getBytes();
}
});
}
}
I have tried many things but the code where the CQL is invoked is never called and doesn't even give any error not sure where is it going wrong.
The same thing if I do creating an internal stream source and use the same input json to return as string it works.
Initial guess: if you are using event time, are you sure you have defined watermarks correctly? As stated in the docs:
(...) an incoming element is initially put in a buffer where elements are sorted in ascending order based on their timestamp, and when a watermark arrives, all the elements in this buffer with timestamps smaller than that of the watermark are processed (...)
If this doesn't help, I would suggest to decompose/simplify the job to a bare minimum, for example just a source operator and some naive sink printing/logging elements. And if that works, start adding back operators one by one. You could also start by simplifying your CEP pattern as much as possible.
First of all thanks a lot #Piotr Nowojski , just because of your small pointer which no matter how many times I pondered over about event time , it did not came in my mind. So yes while debugging the two cases:
With internal datasource , where it was processing successfully, while debugging the flow , I identified that it was processing a watermark after it was processing the data, but it did not catch me that it was somehow managing the event time of the data implicitly.
With kafka as a datasource , while I was debugging I could very clearly see that it was not processing any watermark in the flow, but it did not occur to me that , it is happening because of the event time and watermark not handled properly.
Just adding a single line of code in the application code which I understood from below Flink code snippet:
#deprecated In Flink 1.12 the default stream time characteristic has been changed to {#link
* TimeCharacteristic#EventTime}, thus you don't need to call this method for enabling
* event-time support anymore. Explicitly using processing-time windows and timers works in
* event-time mode. If you need to disable watermarks, please use {#link
* ExecutionConfig#setAutoWatermarkInterval(long)}. If you are using {#link
* TimeCharacteristic#IngestionTime}, please manually set an appropriate {#link
* WatermarkStrategy}. If you are using generic "time window" operations (for example {#link
* org.apache.flink.streaming.api.datastream.KeyedStream#timeWindow(org.apache.flink.streaming.api.windowing.time.Time)}
* that change behaviour based on the time characteristic, please use equivalent operations
* that explicitly specify processing time or event time.
*/
I got to know that by default flink considers event time and for that watermark needs to be handled properly which I didn't so I added below link for setting the time characteristics of the flink execution environment:
env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime);
and kaboom ... it started working , while this is deprecated and needs some other configuration, but thanks a lot , it was a great pointer and helped me a lot and I solved the issue..
Thanks again #Piotr Nowojski
I've been getting into Quarkus and trying to utilize the Mutiny Vertx WebClient. My code works but I prefer not to have to rely on unsafe/unchecked assignments which is how I've currently written the code with bodyAsJson method on HttpResponse. Is there a better way, or more standard way to decode JSON from the Mutiny Vertx client? I realize I could just call bodyAsJsonObject and return that, but I need to do processing on the data the comes back from API calls so I need to decode it to a class representing the data shape/structure.
package com.something.app.language;
import com.something.app.model.Language;
import io.micrometer.core.annotation.Timed;
import io.smallrye.mutiny.Uni;
import io.vertx.mutiny.core.Vertx;
import io.vertx.mutiny.ext.web.client.WebClient;
import javax.annotation.PostConstruct;
import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import java.util.List;
#ApplicationScoped
public class LanguageService {
#Inject
Vertx vertx;
private WebClient client;
#PostConstruct
void init() {
this.client = WebClient.create(vertx);
}
#Timed
public Uni<List<Language>> getLanguages() {
return this.client
.get(80, "somehost.com", "/languages")
.timeout(1000)
.send()
.onItem()
.transform(resp -> {
if (resp.statusCode() == 200) {
return resp.bodyAsJson(List.class);
} else {
throw new RuntimeException("");
}
});
}
}
There are several ways. First, Vert.x uses Jackson under the hood, so everything that can be done with Jackson is possible.
You can use resp.bodyAsJson(MyStructure.class), which would create an instance of MyStructure.
If you have a JSON Array, you can map every element to the object class.
Finally, you can implement your own body codec (See https://vertx.io/docs/apidocs/io/vertx/ext/web/codec/BodyCodec.html).
I am following this source:
Elastic Search Example
and I created the piece of code:
import static org.elasticsearch.node.NodeBuilder.nodeBuilder;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import org.elasticsearch.client.Client;
import org.elasticsearch.node.Node;
public class ElasticSearchAPI {
public static void main(String[] args) {
Node node = nodeBuilder().clusterName("yourclustername").node();
Client client = node.client();
client.prepareIndex("kodcucom", "article", "1")
.setSource(
putJsonDocument(
"ElasticSearch: Java API",
"ElasticSearch provides the Java API, all operations "
+ "can be executed asynchronously using a client object.",
new Date(), new String[] { "elasticsearch" },
"Huseyin Akdogan")).execute().actionGet();
node.close();
}
public static Map<String, Object> putJsonDocument(String title,
String content, Date postDate, String[] tags, String author) {
Map<String, Object> jsonDocument = new HashMap<String, Object>();
jsonDocument.put("title", title);
jsonDocument.put("conten", content);
jsonDocument.put("postDate", postDate);
jsonDocument.put("tags", tags);
jsonDocument.put("author", author);
return jsonDocument;
}
}
I run ElasticSearch with command line:
elasticsearch.bat
and it runs correctly:
After that, I run my Java code and here is a log from Eclipse and server:
Should I configure something? I saw few tutorials like this and everytime is really similar code which never works for me.
Thanks
Your jsonDocument has a typo:
jsonDocument.put("conten", content);
Should be
jsonDocument.put("content", content);
I presume
Ok, I solved this problem. In fact, the problem was with the versions of ElasticSearch Client and ES Java API.
Upgrade ES Java API to the same version as ES Client solved this problem.
More info here:
Java API 1.x Client
Important:
Please note that you are encouraged to use the same version on client
and cluster sides. You may hit some incompatibilities issues when
mixing major versions.
For example I have a response with the following JSON:
{
response: {
id: 20,
name: Stas
}
}
And I want to parse it to the following object:
class User {
private int id;
private String name;
}
How to skip the response node?
I use Google Http Java Client and it will be good if someone will answer how to do it there.
How will this lines have changed?
request.setParser(new JacksonFactory().createJsonObjectParser());
return request.execute().parseAs(getResultType());
You can now implement this in one step:
new JsonObjectParser.Builder(jsonFactory)
.setWrapperKeys(Arrays.asList("response"))
.build()
http://javadoc.google-http-java-client.googlecode.com/hg/1.15.0-rc/index.html
I do not know the Google Http Java Client, but if you can access the Jackson ObjectMapper you could do the following:
1.) Enable root unwrapping:
objectMapper.enable(DeserializationFeature.UNWRAP_ROOT_VALUE);
2.) Add annotation to User.class:
#JsonRootName("response")
class User {
…
}
I hope you can use this approach.
Edit: I dug through the google-http-java-client API and have come to the conclusion that you cannot access the ObjectMapper directly. In order to use the full power of Jackson you would have to write your own implementation of JsonObjectParser to wrap a 'real' Jackson parser. Sorry about that, maybe someone else could come up with a better solution.
I didn't find a native way (for this library) to solve my task. As a result I solved this problem by extending the functionality of JsonObjectParser. It entails expanding of the JacksonFactory, but it's a final class, so I used aggregation.
I wrote the following classes:
JacksonFilteringFactory
import com.google.api.client.json.JsonObjectParser;
import com.google.api.client.json.jackson2.JacksonFactory;
public class JacksonFilteringFactory {
private final JacksonFactory factory = new JacksonFactory();
public JsonObjectParser createJsonObjectParser(String filteringNode) {
return new FilteringJsonObjectParser(factory, filteringNode);
}
}
FilteringJsonObjectParser
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Type;
import java.nio.charset.Charset;
import org.json.JSONException;
import org.json.JSONObject;
import org.json.JSONTokener;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.JsonObjectParser;
import com.vkredmessenger.AppController;
import com.vkredmessenger.util.StringUtils;
public class FilteringJsonObjectParser extends JsonObjectParser {
private String mFilteringNode;
public FilteringJsonObjectParser(JsonFactory jsonFactory,
String filteringNode) {
super(jsonFactory);
mFilteringNode = filteringNode;
}
#Override
public Object parseAndClose(InputStream in,
Charset charset, Type dataType)
throws IOException {
String originalResponse =
StringUtils.convertStreamToString(in, charset);
String response = null;
try {
JSONTokener tokener = new JSONTokener(originalResponse);
JSONObject originalResponseObject =
(JSONObject) tokener.nextValue();
JSONObject responseObject =
originalResponseObject.getJSONObject(mFilteringNode);
response = responseObject.toString();
} catch (JSONException e) {
e.printStackTrace();
}
InputStream filteredIn =
new ByteArrayInputStream(response.getBytes(charset));
return super.parseAndClose(filteredIn, charset, dataType);
}
}
So, for example from my question, the result parsing code will be the following:
request.setParser(new JacksonFilteringFactory().createJsonObjectParser("response"));
return request.execute().parseAs(getResultType());