Develop Reference

Flink Reading a S3 file causing Jackson dependency issue

I a reading a config YAML file in my link application. I want to keep this config file on S3 filesystem but when using aws-sdk in my pom and trying to read I am getting this error. I know this is due to Jackson's conflict dependency but I am not able to resolve it. Please help me to get it to resolve.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
at com.amazonaws.partitions.PartitionsLoader.(PartitionsLoader.java:54)
at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30)
at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:65)
at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:53)
at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:107)
at com.amazonaws.services.s3.AmazonS3Client.createSigner(AmazonS3Client.java:4016)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4913)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1472)
at com.bounce.processor.EventProcessor.main(EventProcessor.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
This this code that I am using to read the file
AmazonS3 amazonS3Client = new AmazonS3Client(credentials);
S3Object object = amazonS3Client.getObject(new GetObjectRequest(S3_PROD_BUCKET, para.get("topology")));
InputStream awsinputStream = object.getObjectContent();
This is my pom.xml
<!-- Flink dependencies -->
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-filesystem_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<version>${flink.format.parquet.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.7</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-parquet_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.10.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
<version>2.10.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.esotericsoftware.yamlbeans/yamlbeans -->
<dependency>
<groupId>com.esotericsoftware.yamlbeans</groupId>
<artifactId>yamlbeans</artifactId>
<version>1.13</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.uber/h3 -->
<dependency>
<groupId>com.uber</groupId>
<artifactId>h3</artifactId>
<version>3.6.3</version>
</dependency>
<dependency>
<groupId>com.github.davidmoten</groupId>
<artifactId>geo</artifactId>
<version>0.7.7</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-elasticsearch6 -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch7_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>tech.allegro.schema.json2avro</groupId>
<artifactId>converter</artifactId>
<version>0.2.9</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.12</artifactId>
<version>1.10.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-avro-confluent-registry</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch7_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
<version>1.11.756</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.10.0</version>
</dependency>

Getting the answer about conflict solely from POM is highly unlikely. Instead, You should refer to the maven dependency plugin, by invoking the following command:
mvn dependency:tree
This will print all the dependencies and the dependencies of those dependencies, this way You will be able to locate which of libraries You are importing has the transitive dependency on Jackson and You will be able to mark it as excluded.
Note: What You are really looking for in this dependency tree are the Jackson dependencies with different versions, so You don't need to exclude them all.

Libraries conflict after upgrade of google-cloud library in my java software running on dataproc

I have a problem after upgrade google-cloud library from 0.8.0 to 0.32.0-alpha version on my java software running on google dataproc.
Here my maven dependencies:
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud</artifactId>
<version>0.32.0-alpha</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.19</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.19</version>
</dependency>
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-core</artifactId>
<version>1.12</version>
</dependency>
<dependency>
<groupId>args4j</groupId>
<artifactId>args4j</artifactId>
<version>2.33</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>2.0.2-beta</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.mail</groupId>
<artifactId>mail</artifactId>
<version>1.4</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.39</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
</dependencies>
This is the error I can see in the dataproc job output
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:77)
at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:73)
at com.google.cloud.RetryHelper.run(RetryHelper.java:73)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:375)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:366)
at com.finscience.job.link.LinkAnalyzerProcess.process(LinkAnalyzerProcess.java:210)
at com.finscience.job.link.LinkAnalyzerJob.main(LinkAnalyzerJob.java:140)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
It seems that the problem is related to google.api.gax library.
After a search I found people that solved the problem excluding guava from maven dependencies.
So I modified my google cloud dependency in this way:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud</artifactId>
<version>0.32.0-alpha</version>
<exclusions>
<exclusion>
<artifactId>com.google</artifactId>
<groupId>guava</groupId>
</exclusion>
</exclusions>
</dependency>
But unfortunately this do not solve my problem.
The github page of the google cloud library (https://github.com/googlecloudplatform/google-cloud-java) says that:
The easiest way to solve version conflicts is to use google-cloud's
BOM
So I add the following dependency to my pom:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bom</artifactId>
<version>0.41.0-alpha</version>
<type>pom</type>
<scope>import</scope>
</dependency>
but even in this case the problem has not been solved.
Could someone help me?

You are most likely running into conflicts with Spark and Hadoop jars on the Dataproc cluster. This answer goes into how to repackage jars to deal with Hadoop / Spark conflicts.

Hibernate Issue with Spring Maven & Tomcat

I have been scratching my head for a few hours and can't seem to solve this issue. I have been following several tutorials on getting started with Maven & Hibernate, following this and this. Each time I deploy on Virgo Tomcat, I get the following error spitting out
An Import-Package could not be resolved. Caused by missing constraint in bundle <com.stuart.contextmanagement_0.1.0>
constraint: <Import-Package: javassist.util.proxy; version="[3.18.0,4.0.0)">
So I added Java Assist as a dependency in my pom, then is spat out
An Import-Package could not be resolved. Caused by missing constraint in bundle
com.stuart.contextmanagement_0.1.0>
constraint: <Import-Package: com.sun.jdi.request; version="0.0.0">
I can't find the above package when searching through repo's. I have also experienced a similar issue before, and every time I add a new dependency which it can't find, it can't find another one and so on...
A snippet of my pom is as follows;
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.5.Final</version>
</dependency>
<dependency>
<groupId>javassist</groupId>
<artifactId>javassist</artifactId>
<version>3.12.1.GA</version>
</dependency>
Thanks for your help.

hm, have you tried also the jboss javassist?

I ended up finding a solution. For any one who's interested, I ended up creating another maven project and exporting several other dependencies than I could see any any documentation or tutorial. Below is the snippet of my final pom.xml.
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>org.springframework.spring-library</artifactId>
<type>libd</type>
<version>3.0.5.RELEASE</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.jboss.javassist</groupId>
<artifactId>com.springsource.javassist</artifactId>
<version>3.15.0.GA</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>com.springsource.javax.persistence</artifactId>
<version>2.0.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>1.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>jsr250-api</artifactId>
<version>1.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>com.springsource.org.apache.commons.dbcp</artifactId>
<version>1.2.2.osgi</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>com.springsource.org.apache.derby</artifactId>
<version>10.5.1000001.764942</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>com.springsource.org.hibernate</artifactId>
<version>3.3.2.GA</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>com.springsource.org.hibernate.annotations</artifactId>
<version>3.4.0.GA-A</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>com.springsource.org.hibernate.annotations.common</artifactId>
<version>3.3.0.ga</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.dom4j</groupId>
<artifactId>com.springsource.org.dom4j</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.9</version>
<scope>compile</scope>
</dependency>
</dependencies>

NoSuchMethodError: JsonFactory.fromInputStream

I am trying to get a clean code sample of Google API OAuth2 authentication.
My code is mostly based on the explanation at https://code.google.com/p/google-api-java-client/wiki/OAuth2
I deviate from this tutorial because I want my libraries be managed by Maven and because I want to create a project without Google Apps Engine dependencies.
Each time I try to the run the project on my Glassfish server, I get the following error, probably indicating a dependency conflict.
java.lang.NoSuchMethodError: com.google.api.client.json.JsonFactory.fromInputStream(Ljava/io/InputStream;Ljava/lang/Class;)Ljava/lang/Object;
The relevant piece of my pom.xml:
<dependencies>
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.api-jackson2</groupId>
<artifactId>google-api-jackson2</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-calendar</artifactId>
<version>v3-rev34-1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.google.oauth-client</groupId>
<artifactId>google-oauth-client</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.1.3</version>
</dependency>
<dependency>
<groupId>com.google.oauth-client</groupId>
<artifactId>google-oauth-client-servlet</artifactId>
<version>1.14.1-beta</version>
</dependency>
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>javax.validation</groupId>
<artifactId>validation-api</artifactId>
<version>1.1.0.CR2</version>
</dependency>
<dependency>
<groupId>javax.faces</groupId>
<artifactId>jsf-api</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>javax.enterprise</groupId>
<artifactId>cdi-api</artifactId>
<version>1.1-PRD</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>jsr250-api</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-web-api</artifactId>
<version>6.0</version>
</dependency>
</dependencies>
Please help me find the root cause of this error, because the used dependencies seem just fine to me and there's no similar problem case I could find.

The correct dependency for jackson2 is:
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client-jackson2</artifactId>
<version>1.14.1-beta</version>
</dependency>
Once I changed that, the project built successfully, and I was able to call the method in question just fine.
There could be something else on the classpath that's interfering. I suggest you run the program with -verbose: class (see here for a more detailed explanation), which will show any conflicts.

Replacing JWSDP jar files with up-to-date equivalents from Maven repo

I inherited an old project that uses JWSDP, and would like to upgrade it to Maven 2 and Java 6.
The project uses the following JWSDP jar files:
jwsdp-jax-qname-1.5.jar
jwsdp-jaxrpc-api-1.5.jar
jwsdp-jaxrpc-impl-1.5-patched-1.10.jar
jwsdp-jaxrpc-spi-1.5.jar
jwsdp-namespace-1.5.jar
jwsdp-relaxngDatatype-1.5.jar
jwsdp-saaj-api-1.5.jar
jwsdp-saaj-impl-1.5.jar
jwsdp-xalan-1.5.jar
jwsdp-xercesImpl-1.5.jar
jwsdp-xsdlib-1.5.jar
As far as I understand, the up-to-date equivalents for these jars should be part of Glassfish , but which ones exactly do I need, and are they available in a Maven 2 repository?

If you just want to upgrade to the JWSPD 1.6 versions of the given artifacts, you'll need the following dependencies (that you can find in http://mvnrepository.com/):
<dependencies>
<dependency>
<groupId>javax.xml</groupId>
<artifactId>jax-qname</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>javax.xml</groupId>
<artifactId>jaxrpc-api</artifactId>
<version>1.1.3</version>
</dependency>
<dependency>
<groupId>javax.xml</groupId>
<artifactId>jaxb-impl</artifactId>
<version>1.0.5</version>
</dependency>
<dependency>
<groupId>com.sun.xml.rpc</groupId>
<artifactId>jaxrpc-spi</artifactId>
<version>1.1.3_01</version>
</dependency>
<dependency>
<groupId>javax.xml</groupId>
<artifactId>namespace</artifactId>
<version>1.0.1</version>
</dependency>
<dependency>
<groupId>com.sun.xml</groupId>
<artifactId>relaxngDatatype</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>javax.xml</groupId>
<artifactId>saaj-api</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>com.sun.xml</groupId>
<artifactId>saaj-impl</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>xalan</groupId>
<artifactId>xalan</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>com.sun.xml</groupId>
<artifactId>xsdlib</artifactId>
<version>20050614</version>
</dependency>
But note that there is no further development of JWSPD 1.6. The replacement stack is Metro which is part of GlassFish v3.