I need to create a hadoop job jar file that uses mahout and a bunch of other libraries. I need ti be able to run the job without needing additional jar.files such that all referenced classes are packaged with the resultant jar file. How can this be done?
Configure your build file to copy all the referenced classes to the build directory. For example, in ant:
<path id="classpathunjar">
<fileset dir="${lib.dir}" includes="*.jar" excludes="sqljdbc4.jar"/>
</path>
<target name="compile" depends="clean">
...
<unjar dest="${build.dir}">
<path refid="classpathunjar" />
</unjar>
...
</target>
But it is better if you can manage without doing this. Use the libjars feature to load the jars into all nodes if you are doing this for running mapreduce jobs on a hadoop cluster
Hadoop has the ability to read jars-in-jar. Amend you Ant script to include all the dependency jars in a folder called lib, and add this lib folder to your output Jar. This is sometimes a better choice if you have number of larger jars as it decreases your jar build time.
See this article on a number of options you have when using 3rd party libs with hadoop
http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
Jar is just a Zip container.
You can manually unzip and modify your Jar file with the classes needed, or you can make use of e.g., the jar-with-dependencies descriptor of the Maven build system.
In the generic sense, it is sometimes impossible, as JAR files have resources that must be in particular locations, and two conflicting but necessary resources might prevent the combination (Think META-INF/MANIFEST.MF)
However, in many cases it is very easy. Basically you unzip the JAR file to be added (it is a zip file format) and "add" the classes and what-not to the existing JAR file.
A better choice if you are making an executable JAR file is to add a ClassPath entry into your launching MANIFEST.MF and ship both JAR files in a directory structure compatible with your added ClassPath entry.
Note that the additional jars have to be put under a lib/ subdirectory (Yes, jars within a jar). I use the following maven assembly, which I found somewhere else.
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>job</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>false</unpack>
<scope>runtime</scope>
<outputDirectory>lib</outputDirectory>
<excludes>
<exclude>org.apache.hadoop:hadoop-core</exclude>
<exclude>${artifact.groupId}:${artifact.artifactId}</exclude>
</excludes>
</dependencySet>
<dependencySet>
<unpack>false</unpack>
<scope>system</scope>
<outputDirectory>lib</outputDirectory>
<excludes>
<exclude>${artifact.groupId}:${artifact.artifactId}</exclude>
</excludes>
</dependencySet>
</dependencySets>
<fileSets>
<fileSet>
<directory>${basedir}/target/classes</directory>
<outputDirectory>/</outputDirectory>
<excludes>
<exclude>*.jar</exclude>
</excludes>
</fileSet>
</fileSets>
</assembly>
Related
My java program is packaged in a jar file and makes use of an external jar library, bouncy castle. My code compiles fine, but running the jar leads to the following error:
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
I've googled for over an hour searching for an explanation and found very little of value. If anyone has seen this error before and could offer some help, I would be obliged.
For those who got this error when trying to create a shaded uber-jar with maven-shade-plugin, the solution is to exclude manifest signature files by adding the following lines to the plugin configuration:
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<!-- Additional configuration. -->
</configuration>
For those using gradle and trying to create and use a fat jar, the following syntax might help.
jar {
doFirst {
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
}
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
}
Please use the following command
zip -d yourjar.jar 'META-INF/*.SF' 'META-INF/*.RSA' 'META-INF/*.DSA'
Some of your dependencies are likely signed jarfiles. When you combine them all into one big jarfile, the corresponding signature files are still present, and no longer match the "big combined" jarfile, so the runtime halts thinking the jar file has been tampered with (which it...has so to speak).
Assuming you're using ant, you can solve the problem by eliminating the signature files from your jarfile dependencies. Unfortunately, it's not possible to do this in one step in ant.
However, I was able to get this working with Ant in two steps, without specifically naming each jarfile dependency, by using:
<target name="jar" depends="compile" description="Create one big jarfile.">
<jar jarfile="${output.dir}/deps.jar">
<zipgroupfileset dir="jars">
<include name="**/*.jar" />
</zipgroupfileset>
</jar>
<sleep seconds="1" />
<jar jarfile="${output.dir}/myjar.jar" basedir="${classes.dir}">
<zipfileset src="${output.dir}/deps.jar" excludes="META-INF/*.SF" />
<manifest>
<attribute name="Main-Class" value="com.mycompany.MyMain" />
</manifest>
</jar>
</target>
The sleep element is supposed to prevent errors about files with modification dates in the future.
Other variations I found in the linked threads didn't work for me.
The solution listed here might provide a pointer.
Invalid signature file digest for Manifest main attributes
Bottom line :
It's probably best to keep the official jar as
is and just add it as a dependency in the manifest file for your
application jar file.
Security is already a tough topic, but I'm disappointed to see the most popular solution is to delete the security signatures. JCE requires these signatures. Maven shade explodes the BouncyCastle jar file which puts the signatures into META-INF, but the BouncyCastle signatures aren't valid for a new, uber-jar (only for the BC jar), and that's what causes the Invalid signature error in this thread.
Yes, excluding or deleting the signatures as suggested by #ruhsuzbaykus does indeed make the original error go away, but it can also lead to new, cryptic errors:
java.security.NoSuchAlgorithmException: PBEWithSHA256And256BitAES-CBC-BC SecretKeyFactory not available
By explicitly specifying where to find the algorithm like this:
SecretKeyFactory.getInstance("PBEWithSHA256And256BitAES-CBC-BC","BC");
I was able to get a different error:
java.security.NoSuchProviderException: JCE cannot authenticate the provider BC
JCE can't authenticate the provider because we've deleted the cryptographic signatures by following the suggestion elsewhere in this same thread.
The solution I found was the executable packer plugin that uses a jar-in-jar approach to preserve the BouncyCastle signature in a single, executable jar.
UPDATE:
Another way to do this (the correct way?) is to use Maven Jar signer. This allows you to keep using Maven shade without getting security errors. HOWEVER, you must have a code signing certificate (Oracle suggests searching for "Java Code Signing Certificate"). The POM config looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>org.bouncycastle:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>your.class.here</mainClass>
</transformer>
</transformers>
<shadedArtifactAttached>true</shadedArtifactAttached>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jarsigner-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<id>sign</id>
<goals>
<goal>sign</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<keystore>/path/to/myKeystore</keystore>
<alias>myfirstkey</alias>
<storepass>111111</storepass>
<keypass>111111</keypass>
</configuration>
</plugin>
No, there's no way to get JCE to recognize a self-signed cert, so if you need to preserve the BouncyCastle certs, you have to either use the jar-in-jar plugin or get a JCE cert.
I had this problem when using IntelliJ IDEA 14.01.
I was able to fix it by:
File->Project Structure->Add New (Artifacts)->jar->From Modules With Dependencies on the Create Jar From Module Window:
Select you main class
JAR File from Libraries
Select copy to the output directory and link via manifest
I faced the same issue, after reference somewhere, it worked as below changing:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
Assuming you build your jar file with ant, you can just instruct ant to leave out the META-INF dir. This is a simplified version of my ant target:
<jar destfile="app.jar" basedir="${classes.dir}">
<zipfileset excludes="META-INF/**/*" src="${lib.dir}/bcprov-jdk16-145.jar"></zipfileset>
<manifest>
<attribute name="Main-Class" value="app.Main"/>
</manifest>
</jar>
I've recently started using IntelliJ on my projects. However, some of my colleagues still use Eclipse on the same projects. Today, I've got the very same error after executing the jar-file created by my IntelliJ. While all the solutions in here talking about almost the same thing, none of them worked for me easily (possibly because I don't use ANT, maven build gave me other errors which referred me to http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException, and also I couldn't figure out what are the signed jars by myself!)
Finally, this helped me
zip -d demoSampler.jar 'META-INF/*.SF' 'META-INF/*.RSA' 'META-INF/*SF'
Guess what's been removed from my jar file?!
deleting: META-INF/ECLIPSE_.SF
deleting: META-INF/ECLIPSE_.RSA
It seems that the issue was relevant to some eclipse-relevant files.
I had the same issue in gradle when creating a fat Jar; updating the build.gradle file with an exclude line corrected the problem.
jar {
from {
configurations.compile.collect {
it.isDirectory() ? it : zipTree(it)
}
}
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
manifest {
attributes 'Main-Class': 'com.test.Main'
}
}
In case you're using gradle, here is a full farJar task:
version = '1.0'
//create a single Jar with all dependencies
task fatJar(type: Jar) {
manifest {
attributes 'Implementation-Title': 'Gradle Jar File Example',
'Implementation-Version': version,
'Main-Class': 'com.example.main'
}
baseName = project.name + '-all'
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
with jar
}
Compare the folder META-INF in new jar with old jar (before you added new libraries). It is possibility that there will be new files. If yes, you can remove them. It should helps.
Regards,
999michal
A strategy would consist in using ANT to simplify the removal of the signature from each Jar file. It would proceed with the following steps:
Copying the MANIFEST.MF in a temporary file
Removing the Name and SHA entries from the temporary file
Creating a temporary Jar file with the temporary manifest
Removing the temporary manifest
Swapping the original Jar file with the temporary one
Here is an ANT macrodef doing the work:
<macrodef name="unsignjar" description="To unsign a specific Jar file">
<attribute name="jarfile"
description="The jar file to unsign" />
<sequential>
<!-- Copying to the temporary manifest file -->
<copy toFile="#{jarFile}_MANIFEST.tmp">
<resources>
<zipentry zipfile="#{jarFile}" name="META-INF/MANIFEST.MF"/>
</resources>
</copy>
<!-- Removing the Name and SHA entries from the temporary file -->
<replaceregexp file="#{jarFile}_MANIFEST.tmp" match="\nName:(.+?)\nSH" replace="SH" flags="gis" byline="false"/>
<replaceregexp file="#{jarFile}_MANIFEST.tmp" match="SHA(.*)" replace="" flags="gis" byline="false"/>
<!-- Creating a temporary Jar file with the temporary manifest -->
<jar jarfile="#{jarFile}.tmp"
manifest="#{jarFile}_MANIFEST.tmp">
<zipfileset src="#{jarFile}">
<include name="**"/>
<exclude name="META-INF/*.SF"/>
<exclude name="META-INF/*.DSA"/>
<exclude name="META-INF/*.RSA"/>
</zipfileset>
</jar>
<!-- Removing the temporary manifest -->
<delete file="#{jarFile}_MANIFEST.tmp" />
<!-- Swapping the original Jar file with the temporary one -->
<move file="#{jarFile}.tmp"
tofile="#{jarFile}"
overwrite="true" />
</sequential>
`
The definition can then be called this way in an ANT task:
<target name="unsignJar">
<unsignjar jarFile="org.test.myjartounsign.jar" />
</target>
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:314)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:268)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
at java.util.jar.JarFile.getInputStream(JarFile.java:450)
at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:977)
at sun.misc.Resource.cachedInputStream(Resource.java:77)
at sun.misc.Resource.getByteBuffer(Resource.java:160)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
What helped me (IntelliJ IDEA 2016.3):
File -> Project Structure -> Artifacts -> Add JAR -> Select Main Class -> Choose "copy to the output directory and link via manifest" -> OK -> Apply -> Build -> Build Artifacts... -> Build
It's possible that two different signers mess up java mind.
Try removing META-INF folder from jar, adding manifest and signing JAR again, it helped me: http://jehy.ru/articles/2013/12/13/invalid-signature-file-digest-for-manifest-main-attributes/
If you are looking for a Fat JAR solution without unpacking or tampering with the original libraries but with a special JAR classloader, take a look at my project here.
Disclaimer: I did not write the code, just package it and publish it on Maven Central and describe in my read-me how to use it.
I personally use it for creating runnable uber JARs containing BouncyCastle dependencies. Maybe it is useful for you, too.
You can use Shadow to generate a jar.
Shadow is a Gradle plugin for combining a project's dependency classes and resources into a single output Jar. The combined Jar is often referred to a fat-jar or uber-jar.
Modify build.gradle
plugins {
...
// ① Add the shadow plugin
id "com.github.johnrengelman.shadow" version "5.2.0"
}
...
// ② Config the shadow jar, its name is baseName-1.0-classifier.jar
shadowJar {
archiveBaseName.set('baseName')
archiveClassifier.set('classifier')
archiveVersion.set('1.0')
manifest {
attributes 'Main-Class': 'Main'
}
}
// ③ Disable the default jar task
jar.enabled = false
// ④ Execute the shadowJar task when compiling
build.dependsOn(shadowJar)
Execute the command gradle build, the jar file will be generated:
<Project Directory>/build/libs/baseName-1.0-classifier.jar
For those who have trouble with the accepted solution, there is another way to exclude resource from shaded jar with DontIncludeResourceTransformer:
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#DontIncludeResourceTransformer
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resource>BC1024KE.DSA</resource>
</transformer>
</transformers>
From Shade 3.0, this transformer accepts a list of resources. Before that you just need to use multiple transformer each with one resource.
This happened to me in Intellij when I clicked "Add as a Maven Project" on bottom line when Intellij said "non-managed pom files found.". Meanwhile out folder was already generated. So it did not get recent changes.
Deleting out folder and running the program solved the issue for me. out folder was then recreated.
See the answer of Little Fox as well. The error I received was very similar to his.
For someone who using kotlin script(kts) as building script:
task("fatJar", type = Jar::class) {
baseName = "${project.name}-fat"
manifest {
attributes["Implementation-Title"] = "Watcher Jar File"
attributes["Implementation-Version"] = version
attributes["Main-Class"] = "MainKt"
}
from(configurations.runtimeClasspath.get().map { if (it.isDirectory) it else zipTree(it) }) {
exclude(listOf("META-INF/*.RSA", "META-INF/*.SF", "META-INF/*.DSA"))
}
with(tasks.jar.get() as CopySpec)
}
I had a similar problem. The reason was that I was compiling using a JDK with a different JRE than the default one in my Windows box.
Using the correct java.exe solved my problem.
If you're getting this when trying to bind JAR files for a Xamarin.Android bindings project like so:
JARTOXML : warning J2XA006: missing class error was raised while reflecting com.your.class : Invalid signature file digest for Manifest main attributes
Just open the JAR files using Winzip and delete the meta-inf directories. Rebuild - job done
I would like to add a file from outside of the Maven project folder to the ZIP that is generated when assembling. How can I do this in the bin.xml file?
Some folder
-Maven Project folder
-Random folder
-File I want to add
What I've tried:
<fileSet>
<directory>../${project.basedir}</directory>
<includes>
<include>Random folder/File I want to add</include>
</includes>
</fileSet>
EDIT: Other things I've tried:
../${basedir}
${project.basedir}/../
I have a multimodule project such as:
top
|----sub1/
|----sub2/
|----packager/
sub1 and sub2 have multiple assemblies attached. Let's say sub1's assemblies creates target/dist/artifactA.jar and target/dist/artifactB.jar
If I create another assembly in packager, how can I pull in artifactA and artifactB from sub1, etc?
If I use this including groupId:artifactId:type:classifier as below, I get the error: "You must set at least one file".
Even if this did work, this isn't exactly what I want, anyway. I want the versions of the files in sub1/target/dist, which don't have version or artifactIds on them (as per sub1's assembly configuration).
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
<id>my-distribution</id>
<formats>
<format>zip</format>
</formats>
<moduleSets>
<moduleSet>
<useAllReactorProjects>true</useAllReactorProjects>
<binaries>
<unpack>false</unpack>
<includes>
<include>com.example:sub1:jar:artifactA</include>
<include>com.example:sub1:jar:artifactB</include>
</includes>
</binaries>
</moduleSet>
</moduleSets>
edit: looks like <moduleset> works only if the sub modules are directly below the module doing the assembling. <dependencyset> is recommended in my case... but I still have the problem of rounding up attached jars that are not the primary jar.
Something that has always made me wonder with maven-assembly-plugin is that the jar-with-dependencies example file on the apache website has a TODO comment about a jarjar format.
Is this just an artifact of code that keeps getting copied through indefinitely or is there some actual reason behind that TODO on the sample file?
The documentation states the following
Use jar-with-dependencies as the descriptorRef of your assembly-plugin
configuration in order to create a JAR which contains the binary
output of your project, along its the unpacked dependencies.
So jar-with-dependencies creates a single JAR will all the class files of your project and also all the unpacked class files of your dependencies.
I'm guessing, "jarjar" refers to something like the Spring Boot Maven plugin does: it also creates a single JAR file, but the dependencies are included in their original JAR files, not unpacked. In contrast to the assembly plugin approach, this requires a custom class loader.
However, you should probably ask this question on the Maven mailing list. I just did so and will update this answer if I get any new information.
Updates
In found out that this particular part of the web page (including the TODO) was added over 10 years ago. Apparently the TODO hasn't been changed since.
I suspect this is referring to a jarjar format, which repackages Java libraries and embeds them into your own distribution, to allow for no dependencies.
see: jarjar
In the context of the example file, which contains:
<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
<!-- TODO: a jarjar format would be better -->
<id>jar-with-dependencies</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<useProjectArtifact>true</useProjectArtifact>
<unpack>true</unpack>
<scope>runtime</scope>
</dependencySet>
</dependencySets>
</assembly>
I think the todo is to indicate that this is how this certain method is done, but that a jarjar would be better. That is, "here's how you do it, but you should do it this other way".
My java program is packaged in a jar file and makes use of an external jar library, bouncy castle. My code compiles fine, but running the jar leads to the following error:
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
I've googled for over an hour searching for an explanation and found very little of value. If anyone has seen this error before and could offer some help, I would be obliged.
For those who got this error when trying to create a shaded uber-jar with maven-shade-plugin, the solution is to exclude manifest signature files by adding the following lines to the plugin configuration:
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<!-- Additional configuration. -->
</configuration>
For those using gradle and trying to create and use a fat jar, the following syntax might help.
jar {
doFirst {
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
}
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
}
Please use the following command
zip -d yourjar.jar 'META-INF/*.SF' 'META-INF/*.RSA' 'META-INF/*.DSA'
Some of your dependencies are likely signed jarfiles. When you combine them all into one big jarfile, the corresponding signature files are still present, and no longer match the "big combined" jarfile, so the runtime halts thinking the jar file has been tampered with (which it...has so to speak).
Assuming you're using ant, you can solve the problem by eliminating the signature files from your jarfile dependencies. Unfortunately, it's not possible to do this in one step in ant.
However, I was able to get this working with Ant in two steps, without specifically naming each jarfile dependency, by using:
<target name="jar" depends="compile" description="Create one big jarfile.">
<jar jarfile="${output.dir}/deps.jar">
<zipgroupfileset dir="jars">
<include name="**/*.jar" />
</zipgroupfileset>
</jar>
<sleep seconds="1" />
<jar jarfile="${output.dir}/myjar.jar" basedir="${classes.dir}">
<zipfileset src="${output.dir}/deps.jar" excludes="META-INF/*.SF" />
<manifest>
<attribute name="Main-Class" value="com.mycompany.MyMain" />
</manifest>
</jar>
</target>
The sleep element is supposed to prevent errors about files with modification dates in the future.
Other variations I found in the linked threads didn't work for me.
The solution listed here might provide a pointer.
Invalid signature file digest for Manifest main attributes
Bottom line :
It's probably best to keep the official jar as
is and just add it as a dependency in the manifest file for your
application jar file.
Security is already a tough topic, but I'm disappointed to see the most popular solution is to delete the security signatures. JCE requires these signatures. Maven shade explodes the BouncyCastle jar file which puts the signatures into META-INF, but the BouncyCastle signatures aren't valid for a new, uber-jar (only for the BC jar), and that's what causes the Invalid signature error in this thread.
Yes, excluding or deleting the signatures as suggested by #ruhsuzbaykus does indeed make the original error go away, but it can also lead to new, cryptic errors:
java.security.NoSuchAlgorithmException: PBEWithSHA256And256BitAES-CBC-BC SecretKeyFactory not available
By explicitly specifying where to find the algorithm like this:
SecretKeyFactory.getInstance("PBEWithSHA256And256BitAES-CBC-BC","BC");
I was able to get a different error:
java.security.NoSuchProviderException: JCE cannot authenticate the provider BC
JCE can't authenticate the provider because we've deleted the cryptographic signatures by following the suggestion elsewhere in this same thread.
The solution I found was the executable packer plugin that uses a jar-in-jar approach to preserve the BouncyCastle signature in a single, executable jar.
UPDATE:
Another way to do this (the correct way?) is to use Maven Jar signer. This allows you to keep using Maven shade without getting security errors. HOWEVER, you must have a code signing certificate (Oracle suggests searching for "Java Code Signing Certificate"). The POM config looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>org.bouncycastle:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>your.class.here</mainClass>
</transformer>
</transformers>
<shadedArtifactAttached>true</shadedArtifactAttached>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jarsigner-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<id>sign</id>
<goals>
<goal>sign</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<keystore>/path/to/myKeystore</keystore>
<alias>myfirstkey</alias>
<storepass>111111</storepass>
<keypass>111111</keypass>
</configuration>
</plugin>
No, there's no way to get JCE to recognize a self-signed cert, so if you need to preserve the BouncyCastle certs, you have to either use the jar-in-jar plugin or get a JCE cert.
I had this problem when using IntelliJ IDEA 14.01.
I was able to fix it by:
File->Project Structure->Add New (Artifacts)->jar->From Modules With Dependencies on the Create Jar From Module Window:
Select you main class
JAR File from Libraries
Select copy to the output directory and link via manifest
I faced the same issue, after reference somewhere, it worked as below changing:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
Assuming you build your jar file with ant, you can just instruct ant to leave out the META-INF dir. This is a simplified version of my ant target:
<jar destfile="app.jar" basedir="${classes.dir}">
<zipfileset excludes="META-INF/**/*" src="${lib.dir}/bcprov-jdk16-145.jar"></zipfileset>
<manifest>
<attribute name="Main-Class" value="app.Main"/>
</manifest>
</jar>
I've recently started using IntelliJ on my projects. However, some of my colleagues still use Eclipse on the same projects. Today, I've got the very same error after executing the jar-file created by my IntelliJ. While all the solutions in here talking about almost the same thing, none of them worked for me easily (possibly because I don't use ANT, maven build gave me other errors which referred me to http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException, and also I couldn't figure out what are the signed jars by myself!)
Finally, this helped me
zip -d demoSampler.jar 'META-INF/*.SF' 'META-INF/*.RSA' 'META-INF/*SF'
Guess what's been removed from my jar file?!
deleting: META-INF/ECLIPSE_.SF
deleting: META-INF/ECLIPSE_.RSA
It seems that the issue was relevant to some eclipse-relevant files.
I had the same issue in gradle when creating a fat Jar; updating the build.gradle file with an exclude line corrected the problem.
jar {
from {
configurations.compile.collect {
it.isDirectory() ? it : zipTree(it)
}
}
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
manifest {
attributes 'Main-Class': 'com.test.Main'
}
}
In case you're using gradle, here is a full farJar task:
version = '1.0'
//create a single Jar with all dependencies
task fatJar(type: Jar) {
manifest {
attributes 'Implementation-Title': 'Gradle Jar File Example',
'Implementation-Version': version,
'Main-Class': 'com.example.main'
}
baseName = project.name + '-all'
from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
with jar
}
Compare the folder META-INF in new jar with old jar (before you added new libraries). It is possibility that there will be new files. If yes, you can remove them. It should helps.
Regards,
999michal
A strategy would consist in using ANT to simplify the removal of the signature from each Jar file. It would proceed with the following steps:
Copying the MANIFEST.MF in a temporary file
Removing the Name and SHA entries from the temporary file
Creating a temporary Jar file with the temporary manifest
Removing the temporary manifest
Swapping the original Jar file with the temporary one
Here is an ANT macrodef doing the work:
<macrodef name="unsignjar" description="To unsign a specific Jar file">
<attribute name="jarfile"
description="The jar file to unsign" />
<sequential>
<!-- Copying to the temporary manifest file -->
<copy toFile="#{jarFile}_MANIFEST.tmp">
<resources>
<zipentry zipfile="#{jarFile}" name="META-INF/MANIFEST.MF"/>
</resources>
</copy>
<!-- Removing the Name and SHA entries from the temporary file -->
<replaceregexp file="#{jarFile}_MANIFEST.tmp" match="\nName:(.+?)\nSH" replace="SH" flags="gis" byline="false"/>
<replaceregexp file="#{jarFile}_MANIFEST.tmp" match="SHA(.*)" replace="" flags="gis" byline="false"/>
<!-- Creating a temporary Jar file with the temporary manifest -->
<jar jarfile="#{jarFile}.tmp"
manifest="#{jarFile}_MANIFEST.tmp">
<zipfileset src="#{jarFile}">
<include name="**"/>
<exclude name="META-INF/*.SF"/>
<exclude name="META-INF/*.DSA"/>
<exclude name="META-INF/*.RSA"/>
</zipfileset>
</jar>
<!-- Removing the temporary manifest -->
<delete file="#{jarFile}_MANIFEST.tmp" />
<!-- Swapping the original Jar file with the temporary one -->
<move file="#{jarFile}.tmp"
tofile="#{jarFile}"
overwrite="true" />
</sequential>
`
The definition can then be called this way in an ANT task:
<target name="unsignJar">
<unsignjar jarFile="org.test.myjartounsign.jar" />
</target>
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:314)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:268)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
at java.util.jar.JarVerifier.update(JarVerifier.java:228)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
at java.util.jar.JarFile.getInputStream(JarFile.java:450)
at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:977)
at sun.misc.Resource.cachedInputStream(Resource.java:77)
at sun.misc.Resource.getByteBuffer(Resource.java:160)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)
What helped me (IntelliJ IDEA 2016.3):
File -> Project Structure -> Artifacts -> Add JAR -> Select Main Class -> Choose "copy to the output directory and link via manifest" -> OK -> Apply -> Build -> Build Artifacts... -> Build
It's possible that two different signers mess up java mind.
Try removing META-INF folder from jar, adding manifest and signing JAR again, it helped me: http://jehy.ru/articles/2013/12/13/invalid-signature-file-digest-for-manifest-main-attributes/
If you are looking for a Fat JAR solution without unpacking or tampering with the original libraries but with a special JAR classloader, take a look at my project here.
Disclaimer: I did not write the code, just package it and publish it on Maven Central and describe in my read-me how to use it.
I personally use it for creating runnable uber JARs containing BouncyCastle dependencies. Maybe it is useful for you, too.
You can use Shadow to generate a jar.
Shadow is a Gradle plugin for combining a project's dependency classes and resources into a single output Jar. The combined Jar is often referred to a fat-jar or uber-jar.
Modify build.gradle
plugins {
...
// ① Add the shadow plugin
id "com.github.johnrengelman.shadow" version "5.2.0"
}
...
// ② Config the shadow jar, its name is baseName-1.0-classifier.jar
shadowJar {
archiveBaseName.set('baseName')
archiveClassifier.set('classifier')
archiveVersion.set('1.0')
manifest {
attributes 'Main-Class': 'Main'
}
}
// ③ Disable the default jar task
jar.enabled = false
// ④ Execute the shadowJar task when compiling
build.dependsOn(shadowJar)
Execute the command gradle build, the jar file will be generated:
<Project Directory>/build/libs/baseName-1.0-classifier.jar
For those who have trouble with the accepted solution, there is another way to exclude resource from shaded jar with DontIncludeResourceTransformer:
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#DontIncludeResourceTransformer
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resource>BC1024KE.DSA</resource>
</transformer>
</transformers>
From Shade 3.0, this transformer accepts a list of resources. Before that you just need to use multiple transformer each with one resource.
This happened to me in Intellij when I clicked "Add as a Maven Project" on bottom line when Intellij said "non-managed pom files found.". Meanwhile out folder was already generated. So it did not get recent changes.
Deleting out folder and running the program solved the issue for me. out folder was then recreated.
See the answer of Little Fox as well. The error I received was very similar to his.
For someone who using kotlin script(kts) as building script:
task("fatJar", type = Jar::class) {
baseName = "${project.name}-fat"
manifest {
attributes["Implementation-Title"] = "Watcher Jar File"
attributes["Implementation-Version"] = version
attributes["Main-Class"] = "MainKt"
}
from(configurations.runtimeClasspath.get().map { if (it.isDirectory) it else zipTree(it) }) {
exclude(listOf("META-INF/*.RSA", "META-INF/*.SF", "META-INF/*.DSA"))
}
with(tasks.jar.get() as CopySpec)
}
I had a similar problem. The reason was that I was compiling using a JDK with a different JRE than the default one in my Windows box.
Using the correct java.exe solved my problem.
If you're getting this when trying to bind JAR files for a Xamarin.Android bindings project like so:
JARTOXML : warning J2XA006: missing class error was raised while reflecting com.your.class : Invalid signature file digest for Manifest main attributes
Just open the JAR files using Winzip and delete the meta-inf directories. Rebuild - job done