Develop Reference

Gremlin-Driver update causes NoSuchMethodError

I have a Java Spring Boot application that connects to an Amazon Neptune graph database running on engine version 1.1.1.0.
After upgrading the gremlin-driver and TinkerPop dependencies to 3.5.2 from 3.4.6 (working on this version), the application can no longer make a connection to the graph database on AWS and it throws this exception
io.netty.channel.ChannelInitializer : Failed to initialize a channel. Closing: [id: 0xf213a752]ecs/XYZ
java.lang.NoSuchMethodError: io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.<init>(Ljava/net/URI;Lio/netty/handler/codec/http/websocketx/WebSocketVersion;Ljava/lang/String;ZLio/netty/handler/codec/http/HttpHeaders;IZZJ)V
I haven't made any changes with the builder besides the code-breaking change with one of the imports and the method name change. Did I miss something in this update?
This is the builder configuration that I am using from 3.4.6
Cluster.Builder builder = Cluster.build();
builder.addContactPoints(gremlinProperties.getContactPoints());
builder.port(gremlinProperties.getPort());
builder.nioPoolSize(gremlinProperties.getNioPoolSize());
builder.workerPoolSize(gremlinProperties.getWorkerPoolSize());
builder.minConnectionPoolSize(gremlinProperties.getMinConnectionPoolSize());
builder.maxConnectionPoolSize(gremlinProperties.getMaxConnectionPoolSize());
builder.minSimultaneousUsagePerConnection(gremlinProperties.getMinSimultaneousUsagePerConnection());
builder.maxSimultaneousUsagePerConnection(gremlinProperties.getMaxSimultaneousUsagePerConnection());
builder.maxInProcessPerConnection(gremlinProperties.getMaxInProcessPerConnection());
builder.minInProcessPerConnection(gremlinProperties.getMinInProcessPerConnection());
builder.maxWaitForConnection(gremlinProperties.getMaxWaitForConnection());
builder.maxWaitForClose(gremlinProperties.getMaxWaitForSessionClose());
builder.maxContentLength(gremlinProperties.getMaxContentLength());
builder.reconnectInterval(gremlinProperties.getReconnectInterval());
builder.resultIterationBatchSize(gremlinProperties.getResultIterationBatchSize());
builder.keepAliveInterval(gremlinProperties.getKeepAliveInterval());
builder.channelizer(Channelizer.WebSocketChannelizer.class);
builder.enableSsl(gremlinProperties.isEnableSsl());
return builder.create();
The values are extracted from a property file

Since the code that handles the gremlin connection and queries is located in a dependency jar project, the netty.version declared in the main project using that jars overrides the netty.io version that is used in the said jar project. I just have to declare a netty.version property in the main project pom so that it matches the netty version used in the dependency.

Missing dependency on EmbeddedKafka on integration test with groovy and spock

I've been trying to create an integration test using the embeddedKafka, but I'm getting problem of missing dependency when trying to run it, this is the error:
Unable to load class org.springframework.kafka.test.EmbeddedKafkaBroker due to missing dependency org/I0Itec/zkclient/serialize/ZkSerializer
I saw some stuff saying that this is related to my dependencies, so here is my dependencies:
springBootVersion = '2.3.5.RELEASE'
compile("org.springframework.boot:spring-boot-starter-web:${springBootVersion}")
compile("org.springframework.kafka:spring-kafka:${springBootVersion}")
testCompile("org.springframework.boot:spring-boot-starter-test:${springBootVersion}",
'org.spockframework:spock-core:1.2-groovy-2.4',
'org.spockframework:spock-spring:1.2-groovy-2.4',
'com.microsoft.azure:spring-data-cosmosdb:2.3.0',
'com.nimbusds:oauth2-oidc-sdk:5.64.4',
)
testCompile("org.springframework.kafka:spring-kafka-test:${springBootVersion}")
So, my question is, am I missing something?
EDIT
After changed the versions as indicated, I got a different error:
Error creating bean with name 'embeddedKafka': Invocation of init
method failed; nested exception is java.lang.NoClassDefFoundError:
scala/math/Ordering$$anon$7
I've added the scala dependencies, but still having the same issue:
testImplementation("org.scala-lang:scala-library:2.12.11")
testImplementation("org.scala-lang:scala-reflect:2.12.11")

You somehow have a mismatched kafka Vs. kafka-clients jars on the classpath; they all must be the same version.
You generally should not specify a version on boot's dependencies and use its dependency management instead.
You are pulling in spring-kafka 2.3.5 whereas spring-boot 2.3.5 requires spring-kafka 2.5.7.
Spring-kafka 2.5.x uses the kafka-clients 2.5.1.
See here for how to override versions of kafka jars when using a different version to the version that Boot prescribes.

the Kafka Client libraries for a time were inlining a particular version of the Scala library. This caused problems for those of us wanting to use the kafka client library with a slightly different version of Scala than that inline version.
In this cases the version of Scala they were using inline is Scala 2.12.10
They removed this dependency in later versions and this was backported as fixes (the earliest being 2.8.0) https://archive.apache.org/dist/kafka/2.8.0/RELEASE_NOTES.html)

How to use org.apache.httpcomponents inside a spark job on Hadoop/Spark?

I am trying to run a spark job on a Hadoop cluster that also makes an http request to another server. I am using org.apache.httpcomponents to make this request, which works fine locally on my machine. However this fails the moment I submit the job to the cluster (managed by Cloudera) with the following error:
User class threw exception: java.lang.NoSuchFieldError: INSTANCE
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.(SSLConnectionSocketFactory.java:151)
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:977)
at org.apache.http.impl.client.HttpClients.createDefault(HttpClients.java:56)
From all the reading I have done, this error is caused by multiple versions of Apache Http client jar. It appears that Hadoop/Spark engine has it's own dependency to Apache Http client and that is a different version than the one I am using. Because my jar is run as part of the hadoop/spark engine it ends up including both my version of http as well as the one Hadoop requires.
If I add 'compileOnly' for org.apache.httpcomponents in my build.gradle and submit, I get this error instead:
User class threw exception: java.lang.NoClassDefFoundError: org/apache/http/impl/client/HttpClients
Is there a way for me to configure this in gradle so that when I build my jar, it will use the already existing version on Hadoop? ie. A way to declare a temporary dependency (when running locally download and use latest version, but when building UberJar drop the dependency)?
UPDATE
I decided to try swapping to a different http library (okhttp3) to see if that would resolve the issue. However I get a very similar exception when trying to run through the cluster here too:
User class threw exception: java.lang.NoSuchFieldError: Companion
at okhttp3.internal.Util.(Util.kt:70)
at okhttp3.OkHttpClient.(OkHttpClient.kt:959)
Looks like Cloudera also supplies a version of okhttp with it's spark2 client which is unfortunate.

REST client inside of OSGi application

I need to integrate a REST client into an existing OSGi application implemented using Apache Felix. The REST service is based on RESTeasy implementation (version 2.3.2.Final) of JAX-RS. I created a separate bundle with clients' dependencies, exporting required RESTeasy packages and importing them in the bundle where the client is used, but unfortunately I cannot get it working inside of the OSGi context.
I tried two different approaches. First one using the generic ClientRequest:
ClientRequest request = new ClientRequest(MyService.URL_TEST+"/stats");
request.body(javax.ws.rs.core.MediaType.APPLICATION_XML, stats);
ClientResponse<String> response = request.post(String.class);
The error that I get in this case is pretty weird:
[java] java.lang.RuntimeException: java.lang.ClassCastException:
org.jboss.resteasy.client.core.executors.ApacheHttpClient4Executor cannot be cast to
org.jboss.resteasy.client.ClientExecutor
where I it is known for sure that ApacheHttpClient4Executor implements the ClientExecutor interface.
When I try to use my own REST client wrapper around RESTeasy like this:
MyService myService = MyServiceClient.getInstance();
myService.saveStatistics(stats);
I get a different exception:
[java] java.lang.LinkageError: ClassCastException: attempting to
castjar:file:/D:/Development/Eclipses/eclipse_4.2_j2ee_x64/lib/jaxrs-api-2.3.2.Final.jar
!/javax/ws/rs/ext/RuntimeDelegate.classtobundle:
//78.0:1/javax/ws/rs/ext/RuntimeDelegate.class
As far as I understand, the LinkageError most probably has to do with the way RESTeasy initializes the RuntimeDelegate using some classloader tricks, which probably fall under the restrictions of OSGi framework. I get the suspicion that the java.lang.ClassCastException mentioned first has the same source.
Is there any way to get RESTeasy working inside of OSGi?
PS: discussion about a similar issue with RESTeasy, but outside of OSGi: java.lang.LinkageError: ClassCastException
Update:
these are the libraries included into restclient bundle:
activation-1.1.jar commons-codec-1.2.jar commons-httpclient-3.1.jar commons-io-2.1.jar commons-logging-1.0.4.jar flexjson-2.1.jar httpclient-4.1.2.jar httpcore-4.1.2.jar javassist-3.12.1.GA.jar jaxb-api-2.2.3.jar jaxb-impl-2.2.4.jar jaxrs-api-2.3.2.Final.jar jcip-annotations-1.0.jar jettison-1.3.1.jar jsr250-api-1.0.jar junit-4.10.jar log4j-1.2.14.jar resteasy-jaxb-provider-2.3.2.Final.jar resteasy-jaxrs-2.3.2.Final.jar resteasy-jettison-provider-2.3.2.Final.jar scannotation-1.0.3.jar slf4j-api-1.6.4.jar slf4j-log4j12-1.6.4.jar myservice-common-0.1.0.3.jar my-service-client-0.1.0.3-SNAPSHOT.jar stax-api-1.0-2.jar xmlpull-1.1.3.1.jar xpp3_min-1.1.4c.jar xstream-1.4.2.jar
These are the exports from the restclient bundle: javax.ws.rs, javax.ws.rs.ext, javax.ws.rs.core, org.jboss.resteasy.client, org.jboss.resteasy.client.cache, org.jboss.resteasy.client.extractors, org.jboss.resteasy.client.marshallers, org.jboss.resteasy.client.core.executors, javax.xml.bind.annotation, org.jboss.resteasy.plugins.providers, org.jboss.resteasy.plugins.providers.jaxb, org.jboss.resteasy.spi

Have a look at the SpringSource Bundle Repo, it's got some very useful pre-built bundles of common libraries including the Apache HTTP Client which we are using (in conjunction with gson) to do our RESTful comms.

(unfortunately a legacy module of my project still uses OSGi, but using RESTeasy 3.0.16 now)
When I need to OSGify a dependency my preferred solution now is to wrap it using the excellent Apache Ops4j Pax Tipi project.
The project provides a preconfigured Maven setup (parent POM handles the bundling) and you just have to adapt the GAV coordinates of the original project in a Tipi sub module with a org.apache.ops4j.pax.tipi prefix and build the new bundle project which draws in the original dependency, unpacks and wraps it as OSGi bundle.
You can start from an existing Tipi sub project that best matches your project setup (dependencies, etc.) and adapt any OSGi imports/exports missing (most often, these are created automatically by the maven-bundle-plugin anyway).
This worked quite well for me as long as the original project did not contain too many exotic or malformed dependencies.
However you may run into snags like transitive dependencies using the root package, as I currently experience, which can be a real show stopper (finding out which library is a real nightmare).
Unfortunately, RESTeasy seems to be affected by this, as I get exactly the same error (default package , even after declaring non-test and non-provided dependencies as optional:
The default package '.' is not permitted by the Import-Package syntax.
Upgrading the maven-bundle-plugin to the latest release 3.0.1 yields a different error (even less helpful):
[ERROR] Bundle org.ops4j.pax.tipi:org.ops4j.pax.tipi.resteasy-jaxrs:bundle:3.0.16.Final.1 : Can not parse name from bundle native code header:
[ERROR] Error(s) found in bundle configuration
Update seems to be solved by upping Tipi version in POM to 1.4.0, testing...

Is RESTEasy mandatory ?
I personally use jersey in OSGi and it is working perfectly, both as client and server.

This problem isn't limited to RESTeasy. It also occurs with Jersey.
It is occurring because you have two copies of the JAX-RS classes on the classpath.
You can see this in the LinkageError:
[java] java.lang.LinkageError: ClassCastException: attempting to cast jar:file:/D:/Development/Eclipses/eclipse_4.2_j2ee_x64/lib/jaxrs-api-2.3.2.Final.jar!/javax/ws/rs/ext/RuntimeDelegate.class to bundle://78.0:1/javax/ws/rs/ext/RuntimeDelegate.class
i.e. one copy is coming from:
D:/Development/Eclipses/eclipse_4.2_j2ee_x64/lib/jaxrs-api-2.3.2.Final.jar
and the other from the OSGI bundle.
This causes problems for the RuntimeDelegate class, which by default uses the system class loader to create the RuntimeDelegate implementation (see javax.ws.rs.ext.FactoryFinder).
The problem can also occur if the same jar is loaded via two different class loaders.
There are a couple of workarounds:
remove the jaxrs-api-2.3.2.Final.jar from the system class path
set the thread context class loader to that of your bundle, prior to making any JAX-RS calls.
The FactoryFinder will use this to load the RuntimeDelegate.
To avoid polluting your code with calls to Thread.currentThread().setContextClassLoader(myBundleClassLoader), you can wrap your JAX-RS client using a Proxy. e.g. see the Thread context classloader section of https://puredanger.github.io/tech.puredanger.com/2007/06/15/classloaders/

Maven / Spring: AopNamespaceUtils NoSuchMethod Error

Using Maven to build my project under windows works fine, but when I build it under Linux I get a NoSuchMethodError regarding one of the spring libs.
I am guessing it is something to do with class loader differences cross-platform and somewhere in my dependencies and I may have the same class twice but windows is loading one while linux loads the other?
Has anyone come across this problem before, or have any advice on how I can debug this error further?
nested exception is java.lang.NoSuchMethodError:
org.springframework.aop.config.AopNamespaceUtils.registerAutoProxyCreatorIfNecessary(Lorg/springframework/beans/factory/xml/ParserContext;Lorg/w3c/dom/Element;)V:
java.lang.NoSuchMethodError:
org.springframework.aop.config.AopNamespaceUtils.registerAutoProxyCreatorIfNecessary(Lorg/springframework/beans/factory/xml/ParserContext;Lorg/w3c/dom/Element;)V
at org.springframework.transaction.config.AnnotationDrivenBeanDefinitionParser$AopAutoProxyConf
Stack Trace Pastie
POM.xml Pastie

Your spring dependencies have too many different versions:
org.springframework:spring-beans:jar:2.5.6:compile
org.springframework:spring-aop:jar:2.0.8:compile
org.springframework:spring-jms:jar:3.0.5.RELEASE:compile
Make all spring dependencies have the same version (preferably 3.0.5.RELEASE)