Develop Reference

Google Cloud Speech NoSuchFieldError: CONTEXT_SPAN_KEY

I'm getting an error while trying to use the google-cloud-speech library.
It's probably about my dependencies. The library was working seamlessly before adding firebase authentication dependencies. So it's probably about the endpoints-management-control-appengine-all dependency.
Stackdriver logs:
com.google.api.server.spi.SystemService invokeServiceMethod: exception
occurred while calling backend method (SystemService.java:373)
java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError:
CONTEXT_SPAN_KEY at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:503)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:462)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
at
com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)
at
com.google.api.gax.longrunning.OperationFutureImpl.get(OperationFutureImpl.java:127)
at
com.codefan.backend.util.AudioRecognizer.getRecognizedSpeechResults(AudioRecognizer.java:45)
at
com.codefan.backend.util.SubtitleGenerator.generate(SubtitleGenerator.java:34)
at
com.codefan.backend.endpoint.SubtitleApi.generateSubtitles(SubtitleApi.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
com.google.api.server.spi.SystemService.invokeServiceMethod(SystemService.java:349)
at
com.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler.handle(EndpointsMethodHandler.java:127)
at
com.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler.handle(EndpointsMethodHandler.java:110)
at
com.google.api.server.spi.dispatcher.PathDispatcher.dispatch(PathDispatcher.java:50)
at
com.google.api.server.spi.EndpointsServlet.service(EndpointsServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at
com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at
com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.api.control.ConfigFilter.doFilter(ConfigFilter.java:125)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.google.api.control.ControlFilter.doFilter(ControlFilter.java:229)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at
com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle(ParseBlobUploadHandler.java:119)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1182)
at
com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle(AppEngineWebAppContext.java:183)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:293)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539) at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at
com.google.apphosting.runtime.jetty9.RpcConnection.handle(RpcConnection.java:213)
at
com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest(RpcConnector.java:81)
at
com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123)
at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:692)
at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:655)
at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:625)
at
com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run(JavaRuntime.java:817)
at
com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:269)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.lang.NoSuchFieldError: CONTEXT_SPAN_KEY at
io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:384)
at
io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at
io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:690)
at
io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at
com.google.api.gax.grpc.GrpcChannelUUIDInterceptor.interceptCall(GrpcChannelUUIDInterceptor.java:52)
at
io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at
com.google.api.gax.grpc.GrpcHeaderInterceptor.interceptCall(GrpcHeaderInterceptor.java:80)
at
io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at
com.google.api.gax.grpc.GrpcMetadataHandlerInterceptor.interceptCall(GrpcMetadataHandlerInterceptor.java:55)
at
io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at
io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:789)
at
io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
at
com.google.api.gax.grpc.GrpcClientCalls.newCall(GrpcClientCalls.java:90)
at
com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58)
at
com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
at
com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86)
at
com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
at
com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
at
com.google.api.gax.grpc.GrpcOperationSnapshotCallable.futureCall(GrpcOperationSnapshotCallable.java:58)
at
com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
at
com.google.api.gax.tracing.TracedOperationInitialCallable.futureCall(TracedOperationInitialCallable.java:61)
at
com.google.api.gax.rpc.OperationCallableImpl.futureCall(OperationCallableImpl.java:82)
at
com.google.api.gax.tracing.TracedOperationCallable.futureCall(TracedOperationCallable.java:78)
at
com.google.api.gax.rpc.OperationCallable$1.futureCall(OperationCallable.java:150)
at
com.google.api.gax.rpc.OperationCallable.futureCall(OperationCallable.java:67)
at
com.google.cloud.speech.v1.SpeechClient.longRunningRecognizeAsync(SpeechClient.java:360)
at
com.google.cloud.speech.v1.SpeechClient.longRunningRecognizeAsync(SpeechClient.java:320)
at
com.codefan.backend.util.AudioRecognizer.getRecognizedSpeechResults(AudioRecognizer.java:39)
... 57 more
Here's the AudioRecognizer class' only method:
#Override
public List<SpeechRecognitionResult> getRecognizedSpeechResults(String sourceLang, int sampleRate, String gcsUri) throws IOException, InterruptedException, ExecutionException {
SpeechClient speechClient = SpeechClient.create();
RecognitionConfig config = RecognitionConfig.newBuilder()
.setEncoding(AudioEncoding.FLAC)
.setLanguageCode(sourceLang)
.setSampleRateHertz(sampleRate)
.setEnableWordTimeOffsets(true)
.build();
RecognitionAudio audio = RecognitionAudio.newBuilder().setUri(gcsUri).build();
// Use non-blocking call for getting file transcription
OperationFuture<LongRunningRecognizeResponse, LongRunningRecognizeMetadata> speechRecResponse = speechClient.longRunningRecognizeAsync(config, audio);
while (!speechRecResponse.isDone()) {
logger.warning("Waiting for response...");
Thread.sleep(10000);
}
return speechRecResponse.get().getResultsList();
}
Project's dependencies:
<dependencies>
<dependency>
<groupId>com.google.endpoints</groupId>
<artifactId>endpoints-framework</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>com.google.endpoints</groupId>
<artifactId>endpoints-framework-guice</artifactId>
<version>2.2.0</version>
</dependency>
<!-- [START api_management] -->
<dependency>
<groupId>com.google.endpoints</groupId>
<artifactId>endpoints-management-control-appengine-all</artifactId>
<version>1.0.11</version>
</dependency>
<!-- [END api_management] -->
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
<type>jar</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-speech</artifactId>
<version>0.87.0-beta</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-translate</artifactId>
<version>1.69.0</version>
</dependency>
<!-- [START Objectify_Dependencies] -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>22.0</version>
</dependency>
<dependency>
<groupId>com.googlecode.objectify</groupId>
<artifactId>objectify</artifactId>
<version>5.1.21</version>
</dependency>
<!-- [END Objectify_Dependencies] -->
<!-- [START Test_Dependencies] -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<!-- [END Test_Dependencies] -->
</dependencies>

To add api management, I walked through this post. It indicates using endpoints-management-control-appengine-all library.
Later on, when I checked this library's compile dependencies, I couldn't see any. They probably stuffed the codes inside the library without using any dependencies, that's why it has the indicator -all, I guess. This leads to having multiple instances of the same classes. Since no dependencies are defined, maven can't pick a single one among multiple dependencies.
In a nutshell, this is caused by classes being scanned from multiple locations.
Luckily, there are alternatives to this library. Here are the dependencies, I am using for api management right now.
<dependency>
<groupId>com.google.endpoints</groupId>
<artifactId>endpoints-management-control-appengine</artifactId>
<version>1.0.11</version>
</dependency>
<dependency>
<groupId>com.google.endpoints</groupId>
<artifactId>endpoints-framework-auth</artifactId>
<version>1.0.11</version>
</dependency>
And probably having less code in my project, since I am not adding all the things.

java.lang.NoClassDefFoundError: javax/websocket/ClientEndpointConfig$Configurator from StandardWebsocketClient

I need to write websocket client.
I started following this tutorial: https://github.com/eugenp/tutorials/blob/master/spring-boot/src/main/java/org/baeldung/websocket
and after run StomClient I received:
java.lang.NoClassDefFoundError: javax/websocket/ClientEndpointConfig$Configurator
at WebSocketTester.testConnection(WebSocketTester.java:16)
Caused by: java.lang.ClassNotFoundException:
javax.websocket.ClientEndpointConfig$Configurator at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 23 more
My pom only have this dependencies:
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-websocket</artifactId>
<version>5.0.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-messaging</artifactId>
<version>5.0.2.RELEASE</version>
</dependency>
Client and session handler are copied from github under attached link.
Exception is thrown from this line WebSocketClient client = new StandardWebSocketClient(); in client class

Try adding the following dependency. It worked for me.
<dependency>
<groupId>org.glassfish.tyrus.bundles</groupId>
<artifactId>tyrus-standalone-client</artifactId>
<version>1.9</version>
</dependency>

The Spring Boot Starter for WebSockets has the dependencies you need:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
</dependency>

Was getting same error. Resolved it by adding below dependency.
<dependency>
<groupId>javax.websocket</groupId>
<artifactId>javax.websocket-api</artifactId>
<version>1.1</version>
</dependency>

Spark (1.5.2) & Cassandra (2.2.5) Java Connector exception

I'm using spark 1.5.2 & cassandra 2.2.5. So based on the recommendation, I used cassandra connector 1.5.0. While running my application (save operation), it's throwing this exception:
Exception in thread "main" java.lang.IllegalArgumentException: Failed to get converter for field "getCompanyid" of type java.lang.String in SampleDBOperation$Employee mapped to column "companyid" of "test.employee"
at com.datastax.spark.connector.writer.MappedToGettableDataConverter$$anon$1$$anonfun$5.apply(MappedToGettableDataConverter.scala:157)
at com.datastax.spark.connector.writer.MappedToGettableDataConverter$$anon$1$$anonfun$5.apply(MappedToGettableDataConverter.scala:150)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at com.datastax.spark.connector.writer.MappedToGettableDataConverter$$anon$1.<init>(MappedToGettableDataConverter.scala:150)
at com.datastax.spark.connector.writer.MappedToGettableDataConverter$.apply(MappedToGettableDataConverter.scala:20)
at com.datastax.spark.connector.writer.DefaultRowWriter.<init>(DefaultRowWriter.scala:17)
at com.datastax.spark.connector.writer.DefaultRowWriter$$anon$1.rowWriter(DefaultRowWriter.scala:31)
at com.datastax.spark.connector.writer.DefaultRowWriter$$anon$1.rowWriter(DefaultRowWriter.scala:29)
at com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:269)
at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
at com.datastax.spark.connector.japi.RDDJavaFunctions.saveToCassandra(RDDJavaFunctions.java:61)
at com.datastax.spark.connector.japi.RDDAndDStreamCommonJavaFunctions$WriterBuilder.saveToCassandra(RDDAndDStreamCommonJavaFunctions.java:443)
Not sure, what's going wrong. Please find my pom.xml details:
<project>
<groupId>edu.berkeley</groupId>
<artifactId>sampledboperation</artifactId>
<modelVersion>4.0.0</modelVersion>
<name>Simple Project for Connecting to Cassandra and doing operations</name>
<packaging>jar</packaging>
<version>1.0</version>
<dependencies>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<!--Spark Cassandra Connector-->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.5.2</version>
</dependency>
</dependencies>
</project>

HBase Java API for HBaseTable creation shows Error

I am a beginner in Hadoop and Hbase. I am getting error when I try to create an HBase Table using Java API. My project is a maven Project. I am using cloudera machine for running my application. Code sample and error and POM.xml is given below
public static void main(String[] args) throws IOException {
// Instantiating configuration class
Configuration con = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(con);
// Instantiating table descriptor class
HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf("empdata"));
// Adding column families to table descriptor
tableDescriptor.addFamily(new HColumnDescriptor("personal"));
tableDescriptor.addFamily(new HColumnDescriptor("official"));
// Execute the table through admin
admin.createTable(tableDescriptor);
System.out.println(" Table created ");
}
Command I am giving like this :
-bash-4.1$ hadoop jar HBaseTableCreation-1.0-SNAPSHOT.jar com.HBaseTable
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at com.Feathersoft.HBaseTable.main(HBaseTable.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 6 more
-bash-4.1$
POM.xml dependancies
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>0.95.0</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shell</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-protocol</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-thrift</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.4.0a</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>11.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-cdh4.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.98.8-hadoop2</version>
</dependency>
How can I set classpath in Cloudera Machine. I am not so much familiar in Linux.
Can any one help, It would be great ....

Kafka Quickstart: What Dependencies do I need?

I am working through the kafka quickstart:
http://kafka.apache.org/07/quickstart.html
and the basic Consumer Group example:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
I have coded up the Consumer and ConsumerThreadPool as above:
import kafka.consumer.KafkaStream;
import kafka.consumer.ConsumerIterator;
public class Consumer implements Runnable {
private KafkaStream m_stream;
private Integer m_threadNumber;
public Consumer(KafkaStream a_stream, Integer a_threadNumber) {
m_threadNumber = a_threadNumber;
m_stream = a_stream;
}
public void run() {
ConsumerIterator<byte[], byte[]> it = m_stream.iterator();
while (it.hasNext()) {
System.out.println("Thread " + m_threadNumber + ": " + new String(it.next().message()));
}
System.out.println("Shutting down Thread: " + m_threadNumber);
}
}
A couple of other facets: I am using spring to manage my zookeeper:
import javax.inject.Named;
import java.util.Properties;
import kafka.consumer.ConsumerConfig;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
#Configuration
#ComponentScan("com.truecar.inventory.worker.core")
public class AppConfig {
#Bean
#Named("consumerConfig")
private static ConsumerConfig createConsumerConfig() {
String zookeeperAddress = "127.0.0.1:2181";
String groupId = "inventory";
Properties props = new Properties();
props.put("zookeeper.connect", zookeeperAddress);
props.put("group.id", groupId);
props.put("zookeeper.session.timeout.ms", "400");
props.put("zookeeper.sync.time.ms", "200");
props.put("auto.commit.interval.ms", "1000");
return new ConsumerConfig(props);
}
}
And I am compiling with Maven and the OneJar Maven plugin. However, I compile and then run the resulting one jar I get the following error:
Aug 26, 2013 6:15:41 PM org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider registerDefaultFilters
INFO: JSR-330 'javax.inject.Named' annotation found and supported for component scanning
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.NoClassDefFoundError: scala/ScalaObject
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
at com.simontuffs.onejar.JarClassLoader.defineClass(JarClassLoader.java:803)
at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:710)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at com.simontuffs.onejar.JarClassLoader.loadClass(JarClassLoader.java:630)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
at java.lang.Class.getDeclaredMethods(Class.java:1845)
at org.springframework.core.type.StandardAnnotationMetadata.getAnnotatedMethods(StandardAnnotationMetadata.java:180)
at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:222)
at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:165)
at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:140)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:282)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:223)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:630)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:461)
at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:73)
at com.truecar.inventory.worker.core.consumer.ConsumerThreadPool.<clinit>(ConsumerThreadPool.java:31)
at com.truecar.inventory.worker.core.application.Starter.main(Starter.java:20)
... 6 more
Caused by: java.lang.ClassNotFoundException: scala.ScalaObject
at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:713)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at com.simontuffs.onejar.JarClassLoader.loadClass(JarClassLoader.java:630)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 27 more
Now, I know little about Kafka, and nothing about Scala. How do I fix this? What should i try next? Is this a known issue? Do I need other dependencies? Here is the kafka version in my pom.xml:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.9.2</artifactId>
<version>0.8.0-beta1</version>
</dependency>
Update: I contacted the Kafka dev mailing list, and they let me know some specific version requirements for the scala dependencies. However, there is also an undocumented log4j dependency, which results in another runtime, not compile time, exception.
Exception in thread "main" java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoSuchMethodError: ch.qos.logback.classic.Logger.filterAndLog(Ljava/lang/String;Lorg/slf4j/Marker;Lch/qos/logback/classic/Level;Ljava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
at org.apache.log4j.Category.log(Category.java:333)
at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
Another Update:
I found the correct log4j dependency:
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
But now I am met with an even more cryptic runtime exception:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.NoClassDefFoundError: org/I0Itec/zkclient/IZkStateListener
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:64)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:66)
at kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100)
at kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala)
At this point I got the WTF kind of feeling. So I added another dependency:
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.3</version>
</dependency>
But this exposed yet another runtime exception:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
at kafka.consumer.ZookeeperConsumerConnector.createFetcher(ZookeeperConsumerConnector.scala:146)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:113)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:64)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:66)
at kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100)
at kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala)
I am hoping to be able to get this baby example up and running, but maybe this is the price to pay for using beta products? Maybe I should switch to Apache Active MQ. But that sounds less fun. Am I missing something?

The problem is that kafka beta was built in a way that pom generated with a jar isn't valid and maven could not recognize it and parse properly, thus fetching transitive dependencies. We've managed to mitigate this problem by enlisting all of the dependencies from that pom (scala, zk, etc) in our pom definition. We're waiting for next beta builds of kafka, in which problem will be fixed.
Full dependencies list is below. Note that you have to change scala version dependency accordingly to the postfix of your kafka artifact.
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.8.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.15</version>
<exclusions>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>net.sf.jopt-simple</groupId>
<artifactId>jopt-simple</artifactId>
<version>3.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.6.4</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>2.8.0</version>
</dependency>
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-annotation</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<version>3.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest</artifactId>
<version>1.2</version>
<scope>test</scope>
</dependency>
As for the
Maybe I should switch to Apache Active MQ. But that sounds less fun.
Am I missing something?
Well, don't you forget that this is the beta release? Some bad things are happening, indeed, but currently we're running kafka 0.7 without any efforts.

I found this configuration of dependencies to be functional:
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>3.2.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>3.2.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.9.2</artifactId>
<version>0.8.0-beta1</version>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>2.2.0</version>
</dependency>
</dependencies>

This seems to work :
$ git clone https://github.com/buildlackey/cep
$ cd cep/kafka-0.8.x
$ mvn package
$ mvn exec:java -Dexec.mainClass=TestKafkaProducer
(via where can I find maven repository for kafka? )