Develop Reference

Does grpc-core work when embedded in a shaded jar?

TL;DR
The grpc-core jar file doesn't work when embedded in a shaded jar but does work when placed on the classpath as a separate jar file. I can't tell if the issue is with grpc-core or grpc-netty (or grpc-netty-shaded) which seems to depend on grpc-core, but when grpc-core is embedded in my shaded jar, I get a java.nio.channels.UnsupportedAddressTypeException exception.
Longer Explanation
Until recently my Java web application (that deploys as a .war file) did not make use of shaded jar files. That is, in the WEB-INF folder contained all the jar files individually laid out. I recently made a change to use a shaded jar and everything works fine except for an integration that makes use of grpc-netty-shaded. I have tried using different versions of the grpc-* libraries and using both the shaded and unshaded version of grpc-netty, but neither of those work when I create a shaded jar.
However, if I produce a shaded jar and place the grpc-core library separately (as a sibling to the shaded jar in the WEB-INF folder) then everything works. I should also say that it doesn't matter if I run the application from within Tomcat or deploy it from the command line, the result is the same. I have also tried switching the JRE I am using but that made no difference.
Here is the exception I get:
Exception in thread "main" io.grpc.StatusRuntimeException: UNKNOWN
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at com.ibm.crypto.grep11.grpc.CryptoGrpc$CryptoBlockingStub.generateKey(CryptoGrpc.java:2964)
at ibm.TestHpcsGrep11.main(TestHpcsGrep11.java:77)
Caused by: java.nio.channels.UnsupportedAddressTypeException
at sun.nio.ch.Net.checkAddress(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at io.grpc.netty.shaded.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
at io.grpc.netty.shaded.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
at java.security.AccessController.doPrivileged(Native Method)
at io.grpc.netty.shaded.io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
at io.grpc.netty.shaded.io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:54)
at io.grpc.netty.shaded.io.grpc.netty.WriteBufferingAndExceptionHandler.connect(WriteBufferingAndExceptionHandler.java:157)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.access$1000(AbstractChannelHandlerContext.java:61)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext$9.run(AbstractChannelHandlerContext.java:538)
at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
This is the project I am integrating with. While the project makes use of version 1.44.0 of the grpc libraries, I have tried it with 1.44.0, 1.47.0, and 1.48.1 and always get the same result.
Here is the Maven plugin configuration I am using to produce the shaded jar:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>shade</goal></goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<artifactSet>
<excludes>
<exclude>junit:junit</exclude>
<exclude>org.testng:testng</exclude>
<exclude>javax.servlet:servlet-api</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>

I did spend hours resolving the same issue, but finally found the root cause.
When running your mvn package command, you most likely get a warning similar to the following:
[WARNING] grpc-core-1.48.1.jar, grpc-netty-shaded-1.48.1.jar define 1 overlapping resource:
[WARNING] - META-INF/services/io.grpc.NameResolverProvider
those 2 files need to be merged (otherwise maven won't take any of them).
this can be done by adding the following to your pom.xml within your maven shade plugin <configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
this will merge your duplicate files and also respect potential relocations from the shade plugin.
also see
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
Maven Shade - change file name and replace text

Running Java Jar with included config via maven on flink yarn cluster

I am using flink in a maven/java project and need to include my configs internally in the created jar.
So, I have added the following in my pom file. This includes all my yml configs (located in src/main/resources folder) in the jar, whose name I will pass as argument while executing.
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.yml</include>
</includes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<finalName>${project.artifactId}-${project.version}</finalName>
<shadedArtifactAttached>true</shadedArtifactAttached>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.exmaple.MyApplication</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
The following main class code receives an arg based on which I decide what config to pick from resource, read(using snakeyaml) and use.
public static void main(String[] args) throws Exception {
final ParameterTool parameterTool = ParameterTool.fromArgs(args);
ClassLoader classLoader = MyApplication.class.getClassLoader();
Yaml yaml = new Yaml();
String filename = parameterTool.getRequired("configFilename");
InputStream in = classLoader.getSystemResourceAsStream(filename);
MyConfigClass = yaml.loadAs(in, MyConfigClass.class);
...
}
mvn clean install creates "my-shaded-jar.jar"
which I execute using command
java -jar /path/to/my-shaded-jar.jar --configFilename filename
It works on multiple systems, if I share the jar with others.
However I am facing issue, when I try to run the same jar in a yarn cluster on Hadoop, using the following command:-
HADOOP_CLASSPATH=`hadoop classpath` HADOOP_CONF_DIR=/etc/hadoop/conf ./flink-1.6.2/bin/flink run -m yarn-cluster -yd -yn 5 -ys 30 -yjm 10240 -ytm 10240 -yst -ynm some-job-name -yqu queue-name ./my-shaded-jar.jar --configFilename filename
I am getting following Error:
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:546)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:238)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:216)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1053)
at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1129)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1129)
Caused by: org.yaml.snakeyaml.error.YAMLException: java.io.IOException: Stream closed
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:200)
at org.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:60)
at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:444)
at com.example.MyApplication.main(MyApplication.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
... 13 more
Caused by: java.io.IOException: Stream closed
at java.io.PushbackInputStream.ensureOpen(PushbackInputStream.java:74)
at java.io.PushbackInputStream.read(PushbackInputStream.java:166)
at org.yaml.snakeyaml.reader.UnicodeReader.init(UnicodeReader.java:90)
at org.yaml.snakeyaml.reader.UnicodeReader.read(UnicodeReader.java:122)
at java.io.Reader.read(Reader.java:140)
at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:184)
Why does my solution works on any normal linux/mac systems, however the same jar with same args fails when running with flink run command on yarn cluster.
Is there a difference between how we generally execute jars and how yarn does it.
Any help appreciated.

Replace classLoader.getSystemResourceAsStream(filename) with classLoader.getResourceAsStream(filename).
java.lang.ClassLoader#getSystemResourceAsStream locates the resource through the system class loader, which is typically used to start the application.
java.lang.ClassLoader#getResourceAsStream will first search the parent class loader. That failing, it will invoke findResource of the current class loader.
To avoid dependency conflicts, classes in Flink applications are divided into two domains [1], which is also applied to Flink client, e.g. CliFrontend.
The Java Classpath includes the classes of Apache Flink and its core dependencies.
The Dynamic User Code includes the classes (and resources) of user jars.
So in order to find your "config file", which is packaged in your jar file, we should use the user code class loader (you can find the details of userCodeClassLoader in org.apache.flink.client.program.PackagedProgram), instead of the system classloader.
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html

why app config lost,when akka used mvn package jar and load by typesafe factory? [duplicate]

I have a command line app that downloads some reports, processes them, then uploads data to Google Drive. I'm using Typesafe-Config for all the magic strings I need. Typesafe-Config looks on the classpath for my application.conf file and uses HOCON to map config objects to fields in my class, like this:
From ~/configs/application.conf:
my.homePageUrl = "https://my.home.com"
From MyClass.java:
private static Config config = ConfigFactory.load();
private static final String HOME_URL = config.getString("my.homePageUrl");
I'm using the maven-shade-plugin to build an executable .jar for easy deployment on a remote server. The plugin looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4</version>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.my.reports.ReportRunner</Main-Class>
<Class-Path>~/configs/application.conf</Class-Path>
</manifestEntries>
</transformer>
</transformers>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
The problem is, when I run the executable .jar, application.conf isn't found on my classpath (I guess this also could be a bug in the typesafe code). All this works just fine in Intellij.
dustinevan#iMac:~/bin$ java -jar my-reports-1.0-SNAPSHOT.jar
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'my'
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
at com.typesafe.config.impl.SimpleConfig.getList(SimpleConfig.java:212)
at com.typesafe.config.impl.SimpleConfig.getHomogeneousUnwrappedList(SimpleConfig.java:271)
at com.typesafe.config.impl.SimpleConfig.getStringList(SimpleConfig.java:329)
at com.stellarliving.reports.ecp.ReportRunner.<clinit>(ReportRunner.java:19)
dustinevan#iMac:~/configs$ ls
total 8
-rw-r--r--# 1 dustinevan staff 1520 Jun 13 01:16 application.conf
I've tried MANY permutations, and done lots of reading to solve this, any help would be greatly appreciated.

As I just had the same problem...
The -jar overrides all classpath settings, so only the jar is seen. -Dconfig.trace=loads will show what is seen by java.
We want the application.conf on the classpath, as well as the jar, so:
java -cp .:my-reports-1.0-SNAPSHOT.jar full.path.to.main.Main
did the trick for me. application.conf found and overrides reference.conf in the jar.

I also had that issue. I've noticed that you use a Class-Path declaration in shaded configuration, so I've merged the answer by Lothar with your info and added:
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<manifestEntries>
<Main-Class>com.my.reports.ReportRunner</Main-Class>
<Class-Path>.</Class-Path>
</manifestEntries>
</transformer>

Load .properties file from IDE and also from outside of JAR

I have a app.properties file in my maven project under resources folder as shown here (simplified):
myApp
|----src
| |
| |--main
| |--java
| | |--ApplicationInitializer.java
| |
| |--resources
| |--app.properties
|
|---target
|--myApp.jar
|--app.properties
In ApplicationInitializer class I want to load properties from the app.properties file with following piece of code:
Properties props = new Properties();
String path = "/app.properties";
try {
props.load(ApplicationInitializer.class.getResourceAsStream(path));
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(props.getProperty("property"));
This piece of code loads properties correctly when I run it from inside my IDE but fails with exception
Exception in thread "main" java.lang.NullPointerException
at java.util.Properties$LineReader.readLine(Properties.java:434)
at java.util.Properties.load0(Properties.java:353)
at java.util.Properties.load(Properties.java:341)
at cz.muni.fi.fits.ApplicationInitializer.main(ApplicationInitializer.java:18)
when trying to run in as JAR file.
For creating a jar file I am using combination of maven-shade-plugin, maven-jar-plugin (for excluding properties file outside of the JAR) and maven-resources-plugin (for copying properties file to specific folder) in pom.xml file as shown here:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>cz.muni.fi.fits.ApplicationInitializer</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
<configuration>
<excludes>
<exclude>**/*.properties</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.7</version>
<executions>
<execution>
<id>copy-resource</id>
<phase>package</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target</outputDirectory>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
I then switched the code in main method to this one:
Properties props = new Properties();
String path = "./app.properties";
try (FileInputStream file = new FileInputStream(path)) {
props.load(file);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(props.getProperty("property"));
and managed to load properties from file when running JAR but this time I was not able to load them when running in IDE, it ended with the same exception as above.
So my question is: How to set the filepath (or pom.xml file?) that I will be able to load properties running from both IDE and JAR file?
Thanks in advance :)

In your Java code read your app.properties file like this:
final String ROOT_PATH = "custom.path"; // Or whatever you most like
final String PROPERTIES_FILE = "app.properties";
// start :: try-catch here to manage I/O resources
File directory = new File(System.getProperty(ROOT_PATH), "conf");
File file = new File(directory, PROPERTIES_FILE);
BufferedInputStream bufferedInputStream = new BufferedInputStream(new FileInputStream(file));
// ...
// end :: try-catch here to manage I/O resources
Then you have to ways for setting/declaring custom.path:
Declaring it as an environment variable
Setting it at run-time (environment properties) with the -Dcustom.path=/somewhere/in/your/filesystem (I prefer this one, unless the path is fixed and/or shared across different applications)
Then, eventually you need to copy/put your app.properties file inside /somewhere/in/your/filesystem/conf. And why inside /conf? Because you use it when you declare the directory field. If you don't want it in there just don't set it and delete the , "conf" part.
Additionally, for running it "locally" (in your IDE) use the VM options setting (IntelliJ IDEA):

The way you're working at the moment you depend on the working directory of your java process. Typically that's the your command line (aka shell) points to, when starting the application.
On most IDE's you can configure this directory in your launcher settings. (For eclipse it's on the second tab 'Arguments') So you will need to the target directory (For eclipse there's a button 'Workspace...')
Specifically in intelliJ you can find this setting under Run/Edit Configurations... This will open a window like this:
There you can edit the working directory. You simply add target at the end.
Edit:
Actually you've add src/main/ressources at the end.

Jersey exception only thrown when depencencies assembled into a single jar

I'm writing a server that embeds Jetty w/ Jersey. When I execute from Eclipse, everything is great. However, if I assemble my server and all dependencies into a single jar using Maven's assembly:single goal, I get an exception:
Sep 26, 2012 5:35:59 PM com.sun.jersey.spi.container.ContainerResponse write
SEVERE: A message body writer for Java class com.acme.server.webservice.
exception.WebServiceFailure, and Java type class com.acme.server.webserv
ice.exception.WebServiceFailure, and MIME media type application/json was not fo
und
Sep 26, 2012 5:35:59 PM com.sun.jersey.spi.container.ContainerResponse write
SEVERE: The registered message body writers compatible with the MIME media type
are:
*/* ->
com.sun.jersey.server.impl.template.ViewableMessageBodyWriter
17:35:59.372 [qtp184245201-22 - /] ERROR o.a.h.ReflectorServletProcessor - onReq
uest()
javax.ws.rs.WebApplicationException: com.sun.jersey.api.MessageException: A mess
age body writer for Java class com.acme.server.webservice.exception.WebS
erviceFailure, and Java type class com.acme.server.webservice.exception.
WebServiceFailure, and MIME media type application/json was not found
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerRespons
e.java:285) ~[vma-server-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequ
est(WebApplicationImpl.java:1457) ~[server-0.0.1-SNAPSHOT-jar-with-dependenc
ies.jar:na]
...
The full trace is here, if it's useful:
https://gist.github.com/3790817
Maven throws no errors while creating the jar-with-dependencies.
I'm a novice with Maven and deployment of Java, and I'm really not sure how to proceed with debugging.
Also, while I need to solve this issue I'd also appreciate any suggested work-arounds as I need to produce an executable demo of my server ASAP that a Pointy-Haired Boss (tm) can execute without Eclipse.
Solution:
Based on Pavel's answer, I dropped the maven-assemly-plugin in favor of maven-shade-plugin. Here's the shade configuration that worked for me:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<!-- use transformer to handle merge of META-INF/services - see http://java.net/jira/browse/JERSEY-440?focusedCommentId=14822&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_14822 -->
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<!-- filter to address "Invalid signature file" issue - see http://stackoverflow.com/a/6743609/589215-->
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>

You are not merging Jersey jars correctly.
Jersey 1.x uses META-INF/services mechanism to discover its components and assembly:single probably just copies everything into single jar, overriding already present files BUT META-INF/services file(s) needs to be CONCATENATED.
Try using jersey-bundle (com.sun.jersey:jersey-bundle:1.14) or fix your assembly settings (or find another plugin to do it better).

Could you post your pom ?
Do you mark some dependencies as provided ?
It's something quite different to build a standalone app and a webapp, as some jars a supposed to be provided by the web container (tomcat or other).
As your container is "embedded" in your app (and not your app in the container) then maybe you don't manage correctly these dependencies.