Develop Reference

Auth fail : Running maven plugin with JGit

I am facing this issue while building a maven project
[INFO] Cloning git#bitbucket.org:bookkeeper/bookkeeper-openapi3-hosted-ui-configuration.git:refs/tags/v5.0.0 into /Users/shubhamjain/bookkeeper/de/bookkeeper-ui-service/service/bookkeeper-api/bookkeeper-platform-hosted-ui-configuration-5.0.0
[ERROR] Failed to fetch bookkeeper-platform-hosted-ui-configuration API (5.0.0):
org.eclipse.jgit.api.errors.TransportException: git#bitbucket.org:bookkeeper/bookkeeper-openapi3-hosted-ui-configuration.git: Auth fail
at org.eclipse.jgit.api.FetchCommand.call (FetchCommand.java:222)
----
----
Caused by: org.eclipse.jgit.errors.TransportException: git#bitbucket.org:bookkeeper/bookkeeper-openapi3-hosted-ui-configuration.git: Auth fail
at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession (JschConfigSessionFactory.java:162)
at org.eclipse.jgit.transport.SshTransport.getSession (SshTransport.java:107)
Initially I had issues in access to the mentioned repository but then I got required permissions.
In the pom.xml
-----
-----
<plugin>
<groupId>com.bookkeeper</groupId>
<artifactId>bookkeeper-api-maven-plugin</artifactId>
<executions>
<execution>
<id>generate-hosted-ui-configuration-openapi-hook</id>
<goals>
<goal>generate-client</goal>
</goals>
<configuration>
<apiName>bookkeeper-platform-hosted-ui-configuration</apiName>
<apiVersion>5.0.0</apiVersion>
<repositorySpecPath>bookkeeper-platform-hosted-ui-configuration/application-hosted-ui-configuration.yaml</repositorySpecPath>
<gitUrlTemplate>git#bitbucket.org:bookkeeper/bookkeeper-openapi3-hosted-ui-configuration.git</gitUrlTemplate>
<openapiConfigurationOverrides>
<isExperimental>false</isExperimental>
<apiName>hosted-ui-configuration</apiName>
</openapiConfigurationOverrides>
</configuration>
</execution>
</executions>
</plugin>
-----
-----
What could be missing ? Thanks in advance.

Issue was Maven couldn't find ssh binary path on the environment
Setting GIT_SSH environment variable worked.
export GIT_SSH=${SSH_BINARY_PATH}
In my case
export GIT_SSH=/usr/bin/ssh

Running Java Jar with included config via maven on flink yarn cluster

I am using flink in a maven/java project and need to include my configs internally in the created jar.
So, I have added the following in my pom file. This includes all my yml configs (located in src/main/resources folder) in the jar, whose name I will pass as argument while executing.
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.yml</include>
</includes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<finalName>${project.artifactId}-${project.version}</finalName>
<shadedArtifactAttached>true</shadedArtifactAttached>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.exmaple.MyApplication</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
The following main class code receives an arg based on which I decide what config to pick from resource, read(using snakeyaml) and use.
public static void main(String[] args) throws Exception {
final ParameterTool parameterTool = ParameterTool.fromArgs(args);
ClassLoader classLoader = MyApplication.class.getClassLoader();
Yaml yaml = new Yaml();
String filename = parameterTool.getRequired("configFilename");
InputStream in = classLoader.getSystemResourceAsStream(filename);
MyConfigClass = yaml.loadAs(in, MyConfigClass.class);
...
}
mvn clean install creates "my-shaded-jar.jar"
which I execute using command
java -jar /path/to/my-shaded-jar.jar --configFilename filename
It works on multiple systems, if I share the jar with others.
However I am facing issue, when I try to run the same jar in a yarn cluster on Hadoop, using the following command:-
HADOOP_CLASSPATH=`hadoop classpath` HADOOP_CONF_DIR=/etc/hadoop/conf ./flink-1.6.2/bin/flink run -m yarn-cluster -yd -yn 5 -ys 30 -yjm 10240 -ytm 10240 -yst -ynm some-job-name -yqu queue-name ./my-shaded-jar.jar --configFilename filename
I am getting following Error:
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:238)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:216)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1053)
at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1129)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1129)
Caused by: org.yaml.snakeyaml.error.YAMLException: java.io.IOException: Stream closed
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:200)
at org.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:60)
at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:444)
at com.example.MyApplication.main(MyApplication.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 13 more
Caused by: java.io.IOException: Stream closed
at java.io.PushbackInputStream.ensureOpen(PushbackInputStream.java:74)
at java.io.PushbackInputStream.read(PushbackInputStream.java:166)
at org.yaml.snakeyaml.reader.UnicodeReader.init(UnicodeReader.java:90)
at org.yaml.snakeyaml.reader.UnicodeReader.read(UnicodeReader.java:122)
at java.io.Reader.read(Reader.java:140)
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:184)
Why does my solution works on any normal linux/mac systems, however the same jar with same args fails when running with flink run command on yarn cluster.
Is there a difference between how we generally execute jars and how yarn does it.
Any help appreciated.

Replace classLoader.getSystemResourceAsStream(filename) with classLoader.getResourceAsStream(filename).
java.lang.ClassLoader#getSystemResourceAsStream locates the resource through the system class loader, which is typically used to start the application.
java.lang.ClassLoader#getResourceAsStream will first search the parent class loader. That failing, it will invoke findResource of the current class loader.
To avoid dependency conflicts, classes in Flink applications are divided into two domains [1], which is also applied to Flink client, e.g. CliFrontend.
The Java Classpath includes the classes of Apache Flink and its core dependencies.
The Dynamic User Code includes the classes (and resources) of user jars.
So in order to find your "config file", which is packaged in your jar file, we should use the user code class loader (you can find the details of userCodeClassLoader in org.apache.flink.client.program.PackagedProgram), instead of the system classloader.
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html

Can't connect to mySql docker container with JDBC

I use Docker Maven Plugin
When test-integration starts i can connect to mysql on container in terminal with this command:
mysql -h 127.0.0.1 -P 32795 -uroot -p
and everythings works good but when i want to connect mysql in java app with JDBC with this code:
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection connection = DriverManager.getConnection(
"jdbc:mysql://127.0.0.1:" + System.getProperty("mysqlPort") + "/dashboardmanager",
"root",
"root"
);
i get this error:
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:866) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:927) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:937) ~[spring-jdbc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
I tried:
export _JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"
and
System.setProperty("java.net.preferIPv4Stack" , "true");
but nothing changed.
Docker Maven Plugin Conf:
<plugin>
<groupId>org.jolokia</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>${docker-maven-plugin.version}</version>
<configuration>
<images>
<image>
<name>mysql:5.7.11</name>
<run>
<env>
<MYSQL_ROOT_PASSWORD>root</MYSQL_ROOT_PASSWORD>
<MYSQL_DATABASE>dashboardmanager</MYSQL_DATABASE>
</env>
<ports>
<port>mysqlPort:3306</port>
</ports>
</run>
</image>
</images>
</configuration>
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>

The problem was this:
MySql starting process takes about 40 seconds, so i should stay about 40 seconds and after that try to connecting to mySql, so simple :)
Or i can use these settings in pom.xml:
<image>
<name>mysql:5.7.11</name>
<alias>mysqlContainer</alias>
<run>
<env>
<MYSQL_ROOT_PASSWORD>root</MYSQL_ROOT_PASSWORD>
<MYSQL_DATABASE>dashboard</MYSQL_DATABASE>
</env>
<ports>
<port>mysqlPort:3306</port>
</ports>
<wait>
<log>.*port: 3306 MySQL Community Server.*</log>
<time>120000</time>
</wait>
</run>
</image>

Make sure your your MySQL config file (my.cnf) set in your mysql container uses:
bind-address = 0.0.0.0
As explained in this answer (for a reverse case: connecting to mysql running on host from a docker container, but the idea is the same here), in bridge mode, setting bind-address to broadcast mode would help validate that mysql is reacheable.
Note: if you use bind-address = 0.0.0.0 your MySQL server will listen for connections on all network interfaces. That means your MySQL server could be reached from the Internet ; make sure to setup firewall rules accordingly.
After this test, check "How to connect to mysql running in container from host machine"
By default, root only has access from the localhost, 127.0.0.1 & ::1, you need to specifically allow access from 192.168.99.1 or from anywhere using '%' in the user setup.
See "Securing the Initial MySQL Accounts".

JBAS012144 Connection timeout

When I try to deploy or undeploy my application(EAR) in JBoss using Maven and Jenkins I get the following error :
INFO: JBoss Remoting version 3.2.12.GA
[DEBUG]
java.io.IOException: java.net.ConnectException: JBAS012144: Could not
connect to remote://192.168.1.8:10099. The connection timed out
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
at org.jboss.as.plugin.common.AbstractServerConnection.isDomainServer(AbstractServerConnection.java:234)
at org.jboss.as.plugin.common.AbstractServerConnection.getClient(AbstractServerConnection.java:156)
...
My JBoss server is listening the port 10099 (9999+100) taking into account the following jboss configuration:
<socket-binding-group
name="standard-sockets"
default-interface="public"
port-offset="${jboss.socket.binding.port-offset:100}">
<socket-binding
name="management-native"
interface="management"
port="${jboss.management.native.port:9999}"/>
The Maven plugin configuration :
<plugin>
<groupId>org.jboss.as.plugins</groupId>
<artifactId>jboss-as-maven-plugin</artifactId>
<version>7.6.Final</version>
<inherited>true</inherited>
<configuration>
<!-- <skip>true</skip> -->
<hostname>${deploy.jboss.host}</hostname>
<port>${deploy.jboss.port}</port>
<username>${deploy.jboss.user}</username>
<password>${deploy.jboss.password}</password>
<filename>fitness-${stage}-${app.server}.ear</filename>
<name>fitness-${stage}-${app.server}.ear</name>
<skip>${skip.deployment}</skip>
<!-- Logging ??? not working-->
<execute-commands>
<commands>
<command>/subsystem=logging/file-handler=debug:add(level=DEBUG,autoflush=true,file={"relative-to"=>"jboss.server.log.dir", "path"=>"jenkins-deployment.log"})
</command>
<command>/subsystem=logging/logger=org.jboss.as:add(level=DEBUG,handlers=[debug])
</command>
</commands>
</execute-commands>
</configuration>
<executions>
<execution>
<id>deploy-application</id>
<goals>
<goal>deploy</goal>
</goals>
</execution>
<execution>
<id>undeploying-all-application</id>
<goals>
<goal>undeploy</goal>
</goals>
<configuration>
<match-pattern>fitness-.*</match-pattern>
<matchPatternStrategy>all</matchPatternStrategy>
</configuration>
</execution>
</executions>
</plugin>
With the following variables :
deploy.jboss.host = 192.168.1.8
deploy.jboss.port = 10099
Here is my configuration :
OS : Ubuntu 13.04
Java : 1.6.0_26
JBoss : 7.1.1.final
Maven : 3.0.3
Jboss-as-maven-plugin : 7.6.final
Jenkins and the target jboss server are running the same machine identified by the ip 192.168.1.8
My own diagnostic :
If I run
sudo netstat -nlp | grep :10099
I get :
tcp 0 0 0.0.0.0:10099 0.0.0.0:* LISTEN 25475/java
And 25475 is my Jboss instance. It seems that JBoss is listening behind the right port.
I can connect using another instance with CLI :
sh jboss-cli.sh controller=192.168.1.8:10099
Thanks in advance for your help

I had same problem and I solved it setting a higher timeout value.
For example:
<plugin>
<groupId>org.jboss.as.plugins</groupId>
<artifactId>jboss-as-maven-plugin</artifactId>
<version>7.9.Final</version>
<inherited>true</inherited>
<configuration>
<hostname>${jboss.hostname}</hostname>
<port>${jboss.port}</port>
<username>${jboss.user}</username>
<password>${jboss.pass}</password>
....
<timeout>30000</timeout>
....
</configuration>
</plugin>
Default "timeout" value is 5000ms. You can try with a higher value like 30000ms. It worked for me.
Edited:
As Pedro said, another option can be passing "timeout" to maven via a command line argument. eg. -Djboss-as.timeout=30000

The problem was JDK 7 related. My jenkins server was using Java 1.7
If you want to use the jdk 7 you need to use the following parameter :
-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.KQueueSelectorProvider

I had the similar issue, however the fix was different, it might help somebody. I had to change maven plugin to the following one:
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<version>1.1.0.Alpha11</version>
</plugin>

Could not send Message : SocketTimeoutException : NotAuthorizedException

I'm using Jetty and getting the following problem on my build server (I don't get it on my dev machine).
2013-07-22 14:19:20,647972350: WARN : org.apache.cxf.phase.PhaseInterceptorChain: Interceptor for {http://rs.rtiservice.esi.v21.xxxx.com/}RTIServiceRSI has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
<SNIP>
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking http://localhost:25000/esi-rtiservice/ESI/RTIService/rs-rtiservice/addLineItem/: Read timed out
<SNIP>
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 104 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
<SNIP>
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1308)
... 107 more
2013-07-22 14:19:20,654681221: ERROR : [qtp23135205-124] com.xxx.v21.mpos.MPOSEndPoint: System error
javax.ws.rs.NotAuthorizedException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
<SNIP>
at java.lang.Thread.run(Thread.java:662)
2013-07-22 14:19:20,656234277: WARN : org.apache.cxf.phase.PhaseInterceptorChain: Interceptor for {http://mpos.v21.xxx.com/}MPOSEndPoint has thrown exception, unwinding now
java.lang.IllegalStateException: WRITER
at org.eclipse.jetty.server.Response.getOutputStream(Response.java:673)
<SNIP>
at java.lang.Thread.run(Thread.java:662)
2013-07-22 14:19:20,658931461: WARN : [qtp23135205-124] org.apache.cxf.phase.PhaseInterceptorChain: Interceptor for {http://mpos.v21.xxx.com/}MPOSEndPoint has thrown exception, unwinding now
java.lang.IllegalStateException: WRITER
at org.eclipse.jetty.server.Response.getOutputStream(Response.java:673)
at org.apache.cxf.transport.http.AbstractHTTPDestination.flushHeaders(AbstractHTTPDestination.java:564)
Now I can see that some people say I need to set ReceiveTimeout on my connection.1
These people don't seem to have an NotAuthorizedException like I do.
All the examples I see are like this:
<http-conf:destination
name="{http://apache.org/hello_world_soap_http}SoapPort.http-destination">
<http-conf:server ReceiveTimeout="30000"
HonorKeepAlive="true" />
</http-conf:destination>
Conversely, I've defined mine like this:
<jaxrs:client id="itemService" address="http://${itemservice.address}/esi-itemservice/ESI/ItemService" serviceClass="com.xxx.v21.esi.itemservice.rs.ItemServiceRSI" inheritHeaders="true">
<jaxrs:headers>
<entry key="Accept" value="application/xml" />
</jaxrs:headers>
<jaxrs:providers>
<ref bean="jaxbextprovider" />
</jaxrs:providers>
</jaxrs:client>
Can someone tell me how to set the ReceiveTimeout on my connection - Or give me something else to go on please?

I've not found the reason for this, but the work around that I've taken is to switch from Jetty in my integration tests and to start using Cargo to download Tomcat, install the artifact/wars and run the tests.
Cargo is taking 5 seconds a test, Jetty was taking 5 minutes!! So they now all run and it takes 2 minutes to run, not almost an hour!
In short, have a look at Cargo (cargo.codehaus.org)
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<version>1.4.5</version>
<configuration>
<container>
<containerId>tomcat7x</containerId>
<zipUrlInstaller>
<url>ftp://x.x.x.x/install_pack/Tomcat7/tomcat7.zip</url>
<downloadDir>${project.build.directory}/cargo/downloads</downloadDir>
</zipUrlInstaller>
<output>${project.build.directory}/cargo/container.log</output>
<append>false</append>
<log>${project.build.directory}/cargo/cargo.log</log>
<timeout>240000</timeout>
<systemProperties>
<logLevel>INFO</logLevel>
<itemservice.address>localhost:9080</itemservice.address>
<deviceservice.address>localhost:9080</deviceservice.address>
<userservice.address>localhost:9080</userservice.address>
<rtiservice.address>localhost:9080</rtiservice.address>
<corporateservice.address>localhost:9080</corporateservice.address>
</systemProperties>
</container>
<configuration>
<properties>
<cargo.servlet.port>9080</cargo.servlet.port>
<cargo.tomcat.ajp.port>9085</cargo.tomcat.ajp.port>
<cargo.jvmargs>-XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled</cargo.jvmargs>
<cargo.logging>high</cargo.logging>
</properties>
</configuration>
<deployables>
<deployable>
<groupId>com.test</groupId>
<artifactId>test-war</artifactId>
<type>war</type>
</deployable>
</deployables>
</configuration>
<executions>
<execution>
<id>start-server</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop-server</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>