Flink SerializationSchema: Could not serialize row error - java

I have some trouble using flink's SerializationSchema.
Here is my main code :
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DeserializationSchema<Row> sourceDeserializer = new JsonRowDeserializationSchema.Builder( /*Extract TypeInformation<Row> from an avsc schema file*/ ).build();
DataStream<Row> myDataStream = env.addSource( new MyCustomSource(sourceDeserializer) ) ;
final SinkFunction<Row> sink = new MyCustomSink(new JsonRowSerializationSchema.Builder(myDataStream.getType()).build());
myDataStream.addSink(sink).name("MyCustomSink");
env.execute("MyJob");
Here is my custom Sink Function :
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.types.Row;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
#SuppressWarnings("serial")
public class MyCustomSink implements SinkFunction<Row> {
private static final Logger LOGGER = LoggerFactory.getLogger(MyCustomSink.class);
private final boolean print;
private final SerializationSchema<Row> serializationSchema;
public MyCustomSink(final SerializationSchema<Row> serializationSchema) {
this.serializationSchema = serializationSchema;
}
#Override
public void invoke(final Row value, final Context context) throws Exception {
try {
LOGGER.info("MyCustomSink- invoke : [{}]", new String(serializationSchema.serialize(value)));
}catch (Exception e){
LOGGER.error("MyCustomSink- Error while sending data : " + e);
}
}
}
And here is my custom Source Function (not sure it is useful for the problem I have) :
import org.apache.flink.api.common.serialization.DeserializationSchema;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.shaded.guava18.com.google.common.io.ByteStreams;
import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MyCustomSource<T> extends RichSourceFunction<T> implements ResultTypeQueryable<T> {
/** logger */
private static final Logger LOGGER = LoggerFactory.getLogger(MyCustomSource.class);
/** the JSON deserializer */
private final DeserializationSchema<T> deserializationSchema;
public MyCustomSource(final DeserializationSchema<T> deserializer) {
this.deserializationSchema = deserializer;
}
#Override
public void open(final Configuration parameters) {
...
}
#Override
public void run(final SourceContext<T> ctx) throws Exception {
LOGGER.info("run");
InputStream data = ...; // Retrieve the input json data
final T row = deserializationSchema
.deserialize(ByteStreams.toByteArray(data));
ctx.collect(row);
}
#Override
public void cancel() {
...
}
#Override
public TypeInformation<T> getProducedType() {
return deserializationSchema.getProducedType();
}
}
Now I run my code and I send some data sequentially to my pipeline :
==>
{
"id": "sensor1",
"data":{
"rotation": 250
}
}
Here, the data is correctly printed by my sink : MyCustomSink- invoke : [{"id":"sensor1","data":{"rotation":250}}]
==>
{
"id": "sensor1"
}
Here, the data is correctly printed by my sink : MyCustomSink- invoke : [{"id":"sensor1","data":null}]
==>
{
"id": "sensor1",
"data":{
"rotation": 250
}
}
Here, there is an error on serialization. The error log printed is :
MyCustomSink- Error while sending data : java.lang.RuntimeException: Could not serialize row 'sensor1,250'. Make sure that the schema matches the input.
I do not understand at all why I have this behavior. Someone have an idea ?
Notes:
Using Flink 1.9.2
-- EDIT --
I added the CustomSource part
-- EDIT 2 --
After more investigations, it looks like this behavior is caused by the private transient ObjectNode node of the JsonRowSerializationSchema. If I understand correctly, this is used for optimization, but seems to be the cause of my problem.
Is it the normal behavior, and if it is, what would be the correct use of this class in my case ? (Else, is there any way to bypass this problem ?)

This is a JsonRowSerializationSchema bug which has been fixed in most recent Flink versions - I believe, this PR addresses the issue above.

Related

Prometheus requires that all meters with the same name have the same set of tag keys

If #Around only #Timed annotated method like this:
package ru.fabit.visor.config.aop;
import io.micrometer.core.annotation.Timed;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tag;
import io.micrometer.core.instrument.Tags;
import io.micrometer.core.instrument.Timer;
import io.micrometer.core.lang.NonNullApi;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.scheduling.annotation.Scheduled;
import java.lang.reflect.Method;
import java.util.function.Function;
/**
* The type Targeted timed aspect.
*/
#Aspect
#NonNullApi
public class TargetedTimedAspect {
public static final String DEFAULT_METRIC_NAME = "method.timed";
public static final String EXCEPTION_TAG = "exception";
public static final String BINDING_TAG = "binding";
public static final String SCHEDULED_CRON_TAG = "cron";
private final MeterRegistry registry;
private final Function<ProceedingJoinPoint, Iterable<Tag>> tagsBasedOnJoinPoint;
public TargetedTimedAspect(MeterRegistry registry) {
this(registry, pjp ->
Tags.of("class", pjp.getStaticPart().getSignature().getDeclaringTypeName(),
"method", pjp.getStaticPart().getSignature().getName())
);
}
public TargetedTimedAspect(MeterRegistry registry, Function<ProceedingJoinPoint, Iterable<Tag>> tagsBasedOnJoinPoint) {
this.registry = registry;
this.tagsBasedOnJoinPoint = tagsBasedOnJoinPoint;
}
// enable TimedAspect only for #StreamListener and #Scheduled annotated methods or allowed methods pointcut
#Around("timedAnnotatedPointcut() )")
public Object timedMethod(ProceedingJoinPoint pjp) throws Throwable {
Method method = ((MethodSignature) pjp.getSignature()).getMethod();
StreamListener streamListener = method.getAnnotation(StreamListener.class);
Scheduled scheduled = method.getAnnotation(Scheduled.class);
// timed can be on method or class
Timed timed = method.getAnnotation(Timed.class);
if (timed == null) {
method = pjp.getTarget().getClass().getMethod(method.getName(), method.getParameterTypes());
timed = method.getAnnotation(Timed.class);
}
final String metricName = timed.value().isEmpty() ? DEFAULT_METRIC_NAME : timed.value();
Timer.Sample sample = Timer.start(registry);
String exceptionClass = "none";
try {
return pjp.proceed();
} catch (Exception ex) {
exceptionClass = ex.getClass().getSimpleName();
throw ex;
} finally {
try {
Timer.Builder timerBuilder = Timer.builder(metricName)
.description(timed.description().isEmpty() ? null : timed.description())
.tags(timed.extraTags())
.tags(EXCEPTION_TAG, exceptionClass)
.tags(tagsBasedOnJoinPoint.apply(pjp))
.publishPercentileHistogram(timed.histogram())
.publishPercentiles(timed.percentiles().length == 0 ? null : timed.percentiles());
if (streamListener != null) {
timerBuilder.tags(
BINDING_TAG,
streamListener.value().isEmpty() ? streamListener.target() : streamListener.value()
);
} else if (scheduled != null) {
timerBuilder.tags(SCHEDULED_CRON_TAG, scheduled.cron());
}
sample.stop(timerBuilder.register(registry));
} catch (Exception e) {
// ignoring on purpose
}
}
}
#Pointcut(
"(#annotation(org.springframework.cloud.stream.annotation.StreamListener) ||" +
"#annotation(org.springframework.scheduling.annotation.Scheduled))"
)
public void asyncAnnotatedPointcut() {
// Method is empty as this is just a Pointcut, the implementations are in the advices.
}
#Pointcut("execution(public * ru.fabit.visor.service.impl.StorageClientImpl.*(..)) ||" +
"execution(public * ru.fabit.visor.service.s3storage.S3StorageClientImpl.*(..))")
public void allowedMethodPointcut() {
// Method is empty as this is just a Pointcut, the implementations are in the advices.
}
#Pointcut("#annotation(io.micrometer.core.annotation.Timed)")
public void timedAnnotatedPointcut() {
// Method is empty as this is just a Pointcut, the implementations are in the advices.
}
}
Then return: java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'web_photos_gotten_list_seconds' containing tag keys [class, exception, method]. The meter you are attempting to register has keys [exception, method, outcome, status, uri].
But, if add all #Timed method in Pointcut, all good work, i dont understand, why we need all annotated method add to Pointcut separately?
This work:
package ru.fabit.visor.config.aop;
import io.micrometer.core.annotation.Timed;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tag;
import io.micrometer.core.instrument.Tags;
import io.micrometer.core.instrument.Timer;
import io.micrometer.core.lang.NonNullApi;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.scheduling.annotation.Scheduled;
import java.lang.reflect.Method;
import java.util.function.Function;
/**
* The type Targeted timed aspect.
*/
#Aspect
#NonNullApi
public class TargetedTimedAspect {
public static final String DEFAULT_METRIC_NAME = "method.timed";
public static final String EXCEPTION_TAG = "exception";
public static final String BINDING_TAG = "binding";
public static final String SCHEDULED_CRON_TAG = "cron";
private final MeterRegistry registry;
private final Function<ProceedingJoinPoint, Iterable<Tag>> tagsBasedOnJoinPoint;
public TargetedTimedAspect(MeterRegistry registry) {
this(registry, pjp ->
Tags.of("class", pjp.getStaticPart().getSignature().getDeclaringTypeName(),
"method", pjp.getStaticPart().getSignature().getName())
);
}
public TargetedTimedAspect(MeterRegistry registry, Function<ProceedingJoinPoint, Iterable<Tag>> tagsBasedOnJoinPoint) {
this.registry = registry;
this.tagsBasedOnJoinPoint = tagsBasedOnJoinPoint;
}
// enable TimedAspect only for #StreamListener and #Scheduled annotated methods or allowed methods pointcut
#Around("timedAnnotatedPointcut() && (asyncAnnotatedPointcut() || allowedMethodPointcut())")
public Object timedMethod(ProceedingJoinPoint pjp) throws Throwable {
Method method = ((MethodSignature) pjp.getSignature()).getMethod();
StreamListener streamListener = method.getAnnotation(StreamListener.class);
Scheduled scheduled = method.getAnnotation(Scheduled.class);
// timed can be on method or class
Timed timed = method.getAnnotation(Timed.class);
if (timed == null) {
method = pjp.getTarget().getClass().getMethod(method.getName(), method.getParameterTypes());
timed = method.getAnnotation(Timed.class);
}
final String metricName = timed.value().isEmpty() ? DEFAULT_METRIC_NAME : timed.value();
Timer.Sample sample = Timer.start(registry);
String exceptionClass = "none";
try {
return pjp.proceed();
} catch (Exception ex) {
exceptionClass = ex.getClass().getSimpleName();
throw ex;
} finally {
try {
Timer.Builder timerBuilder = Timer.builder(metricName)
.description(timed.description().isEmpty() ? null : timed.description())
.tags(timed.extraTags())
.tags(EXCEPTION_TAG, exceptionClass)
.tags(tagsBasedOnJoinPoint.apply(pjp))
.publishPercentileHistogram(timed.histogram())
.publishPercentiles(timed.percentiles().length == 0 ? null : timed.percentiles());
if (streamListener != null) {
timerBuilder.tags(
BINDING_TAG,
streamListener.value().isEmpty() ? streamListener.target() : streamListener.value()
);
} else if (scheduled != null) {
timerBuilder.tags(SCHEDULED_CRON_TAG, scheduled.cron());
}
sample.stop(timerBuilder.register(registry));
} catch (Exception e) {
// ignoring on purpose
}
}
}
#Pointcut(
"(#annotation(org.springframework.cloud.stream.annotation.StreamListener) ||" +
"#annotation(org.springframework.scheduling.annotation.Scheduled))"
)
public void asyncAnnotatedPointcut() {
// Method is empty as this is just a Pointcut, the implementations are in the advices.
}
#Pointcut("execution(public * ru.fabit.visor.service.impl.StorageClientImpl.*(..)) ||" +
"execution(public * ru.fabit.visor.service.s3storage.S3StorageClientImpl.*(..))")
public void allowedMethodPointcut() {
// Method is empty as this is just a Pointcut, the implementations are in the advices.
}
#Pointcut("#annotation(io.micrometer.core.annotation.Timed)")
public void timedAnnotatedPointcut() {
// Method is empty as this is just a Pointcut, the implementations are in the advices.
}
}
pom.xml:
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
The problem youre discribing has nothing to do with pointcuts. There is one piece of code that generates a Timer with three tags (class, exception, method) and another one creating the timer with the exact same same with 5 tags (exception, method, outcome, status, uri) and the framework clearly says, that this is now allowed.
There are several possibilites to solve the issue:
simply use another name for the timer (if you need the other one)
find the other piece of code that generates that timer and deactivate it. Maybe you need to use the debugger and set an conditional breakpoint in MeterRegistry.register()` that registers the meters with the condition that the meter name matches.
PS: using the URI as a tag is not a good practice. The issue is that anyone access your service using different URIs (e.g. by just adding a random number) that will end up in a very high number of meters, which will finally kill your prometheus.

Block texture load only when placed, but not in inventory

I've read some other 'similar' questions but their problems is exactly the opposite. I've also read the docs but they won't provide anything useful to this problem.
When I /give myself the block, it shows a missing texture in my inventory as a item. But when I place it, its texture is shown in the world as a block.
Screenshot:
Main mod class:
package com.byethost8.code2828.mcmods.chemc;
import net.minecraft.block.AbstractBlock;
import net.minecraft.block.Block;
import net.minecraft.block.OreBlock;
import net.minecraft.block.material.Material;
import net.minecraft.block.material.MaterialColor;
import net.minecraft.item.BlockItem;
import net.minecraft.item.Item.Properties;
import net.minecraft.item.ItemGroup;
import net.minecraftforge.common.MinecraftForge;
import net.minecraftforge.event.RegistryEvent;
import net.minecraftforge.eventbus.api.SubscribeEvent;
import net.minecraftforge.fml.common.Mod;
import net.minecraftforge.fml.event.lifecycle.FMLCommonSetupEvent;
import net.minecraftforge.fml.javafmlmod.FMLJavaModLoadingContext;
#Mod(CheMC_.modid)
public class CheMC_ {
public static final String modid = "chemc";
public static OreBlock ore_lithium = (OreBlock) new OreBlock(
AbstractBlock.Properties
.create(Material.ROCK, MaterialColor.PINK_TERRACOTTA)
.harvestLevel(1)
.hardnessAndResistance(1, 1)
.setLightLevel(
light -> {
return 1;
}
)
)
.setRegistryName("chemc", "lithium_ore");
public static BlockItem i_ore_lithium = (BlockItem) new BlockItem(
ore_lithium,
new Properties().group(ItemGroup.BUILDING_BLOCKS)
)
.setRegistryName(ore_lithium.getRegistryName());
public static Block block_lithium = new Block(
AbstractBlock.Properties
.create(Material.IRON, MaterialColor.PINK_TERRACOTTA)
.harvestLevel(1)
.hardnessAndResistance(1.2F, 1)
.setLightLevel(
light -> {
return 1;
}
)
)
.setRegistryName("chemc", "lithium_block");
public static BlockItem i_block_lithium = (BlockItem) new BlockItem(
block_lithium,
new Properties().group(ItemGroup.BUILDING_BLOCKS)
)
.setRegistryName(block_lithium.getRegistryName());
public CheMC_() {
FMLJavaModLoadingContext
.get()
.getModEventBus()
.addListener(this::setup);
MinecraftForge.EVENT_BUS.register(this);
}
private void setup(final FMLCommonSetupEvent event) {}
// You can use EventBusSubscriber to automatically subscribe events on the
// contained class (this is subscribing to the MOD
// Event bus for receiving Registry Events)
#Mod.EventBusSubscriber(bus = Mod.EventBusSubscriber.Bus.MOD)
public static class RegistryEvents {
#SubscribeEvent
public static void onBlocksRegistry(
final RegistryEvent.Register<Block> blockRegistryEvent
) {
// register a new block here
blockRegistryEvent
.getRegistry()
.registerAll(ore_lithium, block_lithium);
}
}
}
Some codes are removed to make the main problem clear.
Following texts only will say about Lithium Block, but same things apply for Lithium Ore.
Model File:
{
"parent": "block/cube_all",
"textures": {
"all": "chemc:block/lithium_block"
}
}
Folder structure of src/main/resources:
Blockstate:
{
"variants": {
"": [
{ "model": "chemc:block/lithium_block" }
]
}
}
I can't believe that I was stupid enough to register a Item and do nothing to assets/chemc/resources/models/item/ folder. See this for more. I have the exactly same problem as that OP.

How to add "text/plain" MIME type to DataHandler

I have been struggling with getting this test to work for awhile, the relevant code executes fine in production my assumption is that it has some additional configuration, lots of searching seems to be related specifically to email handling and additional libraries, I don't want to include anything else, what am I missing to link DataHandler to a relevant way of handling "text/plain" ?
Expected result: DataHandler allows me to stream the input "Value" back into a result.
Reproduce issue with this code:
import java.io.IOException;
import java.io.InputStream;
import javax.activation.CommandInfo;
import javax.activation.CommandMap;
import javax.activation.DataHandler;
import org.apache.commons.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class DataHandlerTest {
#Before
public void setUp() throws Exception {
}
#After
public void tearDown() throws Exception {
}
#Test
public void test() throws IOException {
printDefaultCommandMap();
DataHandler dh = new DataHandler("Value", "text/plain");
System.out.println("DataHandler commands:");
printDataHandlerCommands(dh);
dh.setCommandMap(CommandMap.getDefaultCommandMap());
System.out.println("DataHandler commands:");
printDataHandlerCommands(dh);
final InputStream in = dh.getInputStream();
String result = new String(IOUtils.toByteArray(in));
System.out.println("Returned String: " + result);
}
private void printDataHandlerCommands(DataHandler dh) {
CommandInfo[] infos = dh.getAllCommands();
printCommands(infos);
}
private void printDefaultCommandMap() {
CommandMap currentMap = CommandMap.getDefaultCommandMap();
String[] mimeTypes = currentMap.getMimeTypes();
System.out.println("Found " + mimeTypes.length + " MIME types.");
for (String mimeType : mimeTypes) {
System.out.println("Commands for: " + mimeType);
printCommands(currentMap.getAllCommands(mimeType));
}
}
private void printCommands(CommandInfo[] infos) {
for (CommandInfo info : infos) {
System.out.println(" Command Class: " +info.getCommandClass());
System.out.println(" Command Name: " + info.getCommandName());
}
}
}
Exception:
javax.activation.UnsupportedDataTypeException: no object DCH for MIME
type text/plain at
javax.activation.DataHandler.getInputStream(DataHandler.java:249)
Help much appreciated, I hope this is a well formed question!
========================
Update 25th February
I have found if i know I stored a String in DataHandler, then I can cast the result to String and return the object that was stored, example:
#Test
public void testGetWithoutStream() throws IOException {
String inputString = "Value";
DataHandler dh = new DataHandler(inputString, "text/plain");
String rawResult = (String) dh.getContent();
assertEquals(inputString, rawResult);
}
But the code under test uses an InputStream, so my 'real' tests still fail when executed locally.
Continuing my investigation and still hoping for someone's assistance/guidance on this one...
Answering my own question for anyone's future reference.
All credit goes to: https://community.oracle.com/thread/1675030?start=0
The principle here is that you need to provide DataHandler a factory that contains a DataContentHandler that will behave as you would like it to for your MIME type, setting this is via a static method that seems to affect all DataHandler instances.
I declared a new class (SystemDataHandlerConfigurator), which has a single public method that creates my factory and provides it the static DataHandler.setDataContentHandlerFactory() function.
My tests now work correctly if I do this before they run:
SystemDataHandlerConfigurator configurator = new SystemDataHandlerConfigurator();
configurator.setupCustomDataContentHandlers();
SystemDataHandlerConfigurator
import java.io.IOException;
import javax.activation.*;
public class SystemDataHandlerConfigurator {
public void setupCustomDataContentHandlers() {
DataHandler.setDataContentHandlerFactory(new CustomDCHFactory());
}
private class CustomDCHFactory implements DataContentHandlerFactory {
#Override
public DataContentHandler createDataContentHandler(String mimeType) {
return new BinaryDataHandler();
}
}
private class BinaryDataHandler implements DataContentHandler {
/** Creates a new instance of BinaryDataHandler */
public BinaryDataHandler() {
}
/** This is the key, it just returns the data uninterpreted. */
public Object getContent(javax.activation.DataSource dataSource) throws java.io.IOException {
return dataSource.getInputStream();
}
public Object getTransferData(java.awt.datatransfer.DataFlavor dataFlavor,
javax.activation.DataSource dataSource)
throws java.awt.datatransfer.UnsupportedFlavorException,
java.io.IOException {
return null;
}
public java.awt.datatransfer.DataFlavor[] getTransferDataFlavors() {
return new java.awt.datatransfer.DataFlavor[0];
}
public void writeTo(Object obj, String mimeType, java.io.OutputStream outputStream)
throws java.io.IOException {
if (mimeType == "text/plain") {
byte[] stringByte = (byte[]) ((String) obj).getBytes("UTF-8");
outputStream.write(stringByte);
}
else {
throw new IOException("Unsupported Data Type: " + mimeType);
}
}
}
}

Kafka : Alter number of partitions for a specific topic using java

I am new to Kafka and working with new KafkaProducer and KafkaConsumer, version : 0.9.0.1
Is there any way in java to alter/update the number of partitions for a specific topic after it has been created.
I am not using zookeeper to create topic.
My KafkaProducer is automatically creating topics when publish request arrives.
I can also provide more details if these are not enough
Yes, it's possible. You have to access the AdminUtils scala class in kafka_2.11-0.9.0.1.jar to add partitions.
AdminUtils supports the number of partitions in the topic can only be increased. You may need kafka_2.11-0.9.0.1.jar, zk-client-0.8.jar, scala-library-2.11.8.jar and scala-parser-combinators_2.11-1.0.4.jar jars in your classpath.
Portions of the below code is borrowed / inspired from kafka-cloudera examples.
package org.apache.kafka.examples;
import java.io.Closeable;
import org.I0Itec.zkclient.ZkClient;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import kafka.admin.AdminOperationException;
import kafka.admin.AdminUtils;
import kafka.admin.RackAwareMode.Enforced$;
import kafka.utils.ZKStringSerializer$;
import kafka.utils.ZkUtils;
public class Test {
static final Logger logger = LogManager.getLogger();
public Test() {
// TODO Auto-generated constructor stub
}
public static void addPartitions(String zkServers, String topic, int partitions) {
try (AutoZkClient zkClient = new AutoZkClient(zkServers)) {
ZkUtils zkUtils = ZkUtils.apply(zkClient, false);
if (AdminUtils.topicExists(zkUtils, topic)) {
logger.info("Altering topic {}", topic);
try {
AdminUtils.addPartitions(zkUtils, topic, partitions, "", true, Enforced$.MODULE$);
logger.info("Topic {} altered with partitions : {}", topic, partitions);
} catch (AdminOperationException aoe) {
logger.info("Error while altering partitions for topic : {}", topic, aoe);
}
} else {
logger.info("Topic {} doesn't exists", topic);
}
}
}
// Just exists for Closeable convenience
private static final class AutoZkClient extends ZkClient implements Closeable {
static int sessionTimeout = 30_000;
static int connectionTimeout = 6_000;
AutoZkClient(String zkServers) {
super(zkServers, sessionTimeout, connectionTimeout, ZKStringSerializer$.MODULE$);
}
}
public static void main(String[] args) {
addPartitions("localhost:2181", "hello", 20);
}
}

Good Zookeeper Hello world Program with Java client

I was trying to use Zookeeper in our project. Could run the server..Even test it using zkcli.sh .. All good..
But couldn't find a good tutorial for me to connect to this server using Java ! All I need in Java API is a method
public String getServiceURL ( String serviceName )
I tried https://cwiki.apache.org/confluence/display/ZOOKEEPER/Index --> Not good for me.
http://zookeeper.apache.org/doc/trunk/javaExample.html : Sort of ok; but couldnt understand concepts clearly ! I feel it is not explained well..
Finally, this is the simplest and most basic program I came up with which will help you with ZooKeeper "Getting Started":
package core.framework.zookeeper;
import java.util.Date;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.Watcher.Event.KeeperState;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;
public class ZkConnect {
private ZooKeeper zk;
private CountDownLatch connSignal = new CountDownLatch(0);
//host should be 127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002
public ZooKeeper connect(String host) throws Exception {
zk = new ZooKeeper(host, 3000, new Watcher() {
public void process(WatchedEvent event) {
if (event.getState() == KeeperState.SyncConnected) {
connSignal.countDown();
}
}
});
connSignal.await();
return zk;
}
public void close() throws InterruptedException {
zk.close();
}
public void createNode(String path, byte[] data) throws Exception
{
zk.create(path, data, Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}
public void updateNode(String path, byte[] data) throws Exception
{
zk.setData(path, data, zk.exists(path, true).getVersion());
}
public void deleteNode(String path) throws Exception
{
zk.delete(path, zk.exists(path, true).getVersion());
}
public static void main (String args[]) throws Exception
{
ZkConnect connector = new ZkConnect();
ZooKeeper zk = connector.connect("54.169.132.0,52.74.51.0");
String newNode = "/deepakDate"+new Date();
connector.createNode(newNode, new Date().toString().getBytes());
List<String> zNodes = zk.getChildren("/", true);
for (String zNode: zNodes)
{
System.out.println("ChildrenNode " + zNode);
}
byte[] data = zk.getData(newNode, true, zk.exists(newNode, true));
System.out.println("GetData before setting");
for ( byte dataPoint : data)
{
System.out.print ((char)dataPoint);
}
System.out.println("GetData after setting");
connector.updateNode(newNode, "Modified data".getBytes());
data = zk.getData(newNode, true, zk.exists(newNode, true));
for ( byte dataPoint : data)
{
System.out.print ((char)dataPoint);
}
connector.deleteNode(newNode);
}
}
This post has almost all operations required to interact with Zookeeper.
https://www.tutorialspoint.com/zookeeper/zookeeper_api.htm
Create ZNode with data
Delete ZNode
Get list of ZNodes(Children)
Check an ZNode exists or not
Edit the content of a ZNode...
This blog post, Zookeeper Java API examples, includes some good examples if you are looking for Java examples to start with. Zookeeper also provides a client API library( C and Java) that is very easy to use.
Zookeeper is one of the best open source server and service that helps to reliably coordinates distributed processes. Zookeeper is a CP system (Refer CAP Theorem) that provides Consistency and Partition tolerance. Replication of Zookeeper state across all the nods makes it an eventually consistent distributed service.
This is about as simple as you can get. I am building a tool which will use ZK to lock files that are being processed (hence the class name):
package mypackage;
import java.io.IOException;
import java.util.List;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.Watcher;
public class ZooKeeperFileLock {
public static void main(String[] args) throws IOException, KeeperException, InterruptedException {
String zkConnString = "<zknode1>:2181,<zknode2>:2181,<zknode3>:2181";
ZooKeeperWatcher zkWatcher = new ZooKeeperWatcher();
ZooKeeper client = new ZooKeeper(zkConnString, 10000, zkWatcher);
List<String> zkNodes = client.getChildren("/", true);
for(String node : zkNodes) {
System.out.println(node);
}
}
public static class ZooKeeperWatcher implements Watcher {
#Override
public void process(WatchedEvent event) {
}
}
If you are on AWS; now We can create internal ELB which supports redirection based on URI .. which can really solve this problem with High Availability already baked in.

Categories

Resources