Develop Reference

How to use Proguard to shrink Gradle ShadowJar output to create minimized fat Java jar?

I want to shrink a fat jar which has been created using the Gradle shadow plugin. The reason for this is that I need to keep some classes within a specific package within the finished jar as they are used for dynamic class generation (e.g. Class.forName(...)) and that plugin won't allow me to specify a package to keep when calling minimize() within the shadow jar task). I am therefore using the Proguard Gradle plugin.
buildscript {
dependencies {
classpath 'net.sf.proguard:proguard-gradle:6.2.2'
classpath 'net.sf.proguard:proguard-base:6.2.2'
}
}
plugins {
id 'java-library'
id 'application'
id 'com.github.johnrengelman.shadow' version '5.2.0'
}
sourceCompatibility = 1.11
shadowJar {
zip64 true
}
application {
mainClassName = 'com.example.MyApp'
}
task proguard(type: proguard.gradle.ProGuardTask) {
dependsOn shadowJar
injars shadowJar
outjars "${buildDir}/libs/${project.name}-${project.version}-proguard.jar"
dontobfuscate
dontoptimize
keep 'class com.example.packagetokeep.*'
// next block taken verbatim from Proguard's documentation examples:
// Automatically handle the Java version of this build.
if (System.getProperty('java.version').startsWith('1.')) {
// Before Java 9, the runtime classes were packaged in a single jar file.
libraryjars "${System.getProperty('java.home')}/lib/rt.jar"
} else {
// As of Java 9, the runtime classes are packaged in modular jmod files.
libraryjars "${System.getProperty('java.home')}/jmods/java.base.jmod", jarfilter: '!**.jar', filter: '!module-info.class'
}
}
So, what I am trying to do is create a fat jar with all dependencies in it, then pass to Proguard to minimize. For now, I don't want to optimise or obfuscate.
When I try to run the task, I get a lot of errors of the form can't find referenced class for classes like javax.xml.stream.events.Attribute, java.sql.Timestamp, java.util.logging.Level. I think it's because Proguard is not picking up the Java runtime libraries. Here is the error summary:
Warning: there were 75920 unresolved references to classes or interfaces.
You may need to add missing library jars or update their versions.
If your code works fine without the missing classes, you can suppress
the warnings with '-dontwarn' options.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedclass)
Warning: there were 670 unresolved references to program class members.
Your input classes appear to be inconsistent.
You may need to recompile the code.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedprogramclassmember)
Warning: there were 4 unresolved references to library class members.
You probably need to update the library versions.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedlibraryclassmember)
How do I make Proguard work with Gradle and Java 11?

Added Gradle to Java project "Exception ... java.lang.NoClassDefFoundError"

I had an existing project without Gradle and needed to add com.google.code.gson:gson:+ library to work with JSON objects. To begin with I ran either gradle init or gradle build, I'm not sure. This caused my java classes with a main() not to run as the source path was wrong/changed. I have changed the structure following advice to at least get the classes to compile and run, but I still have this warning in run configurations "Warning: Class 'Main' not found in module 'src'" ;
If I set Use classpath of module to src.main, the warning goes away but when I run Main.main() Gradle seems to execute Gradle tasks, like this - this will run indefinitely;
Here is my project structure;
This is my build.gradle file;
/*
* This file was generated by the Gradle 'init' task.
*
* This generated file contains a sample Java project to get you started.
* For more details take a look at the Java Quickstart chapter in the Gradle
* User Manual available at https://docs.gradle.org/6.3/userguide/tutorial_java_projects.html
*/
plugins {
// Apply the java plugin to add support for Java
id 'java'
// Apply the application plugin to add support for building a CLI application.
id 'application'
// idea plugin? // I added this to original build.gradle file
id 'idea'
}
repositories {
// Use jcenter for resolving dependencies.
// You can declare any Maven/Ivy/file repository here.
jcenter()
mavenCentral()
google()
}
dependencies {
// This dependency is used by the application.
implementation 'com.google.guava:guava:28.2-jre'
// Use JUnit test framework
testImplementation 'junit:junit:4.12'
// For use with JSONUtil class // I added this to original build.gradle file
compile 'com.google.code.gson:gson:+'
}
application {
// Define the main class for the application.
mainClassName = 'java.Main' // changed to 'Main' and I can `gradle run` seems to actually run Main.java
}
I have imported com.google.gson.JsonObject and com.google.gson.JsonParser from com.google.gson:gson:2.8.6 library, with no code inspection warnings, i.e available at compile time. If I run my code with a JsonObject jsonObject = new JsonObject I get the error;
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/gson/JsonParser
at HttpUtils.getAccessToken(HttpUtils.java:80)
at Main.auth(Main.java:75)
at Main.play(Main.java:36)
at Main.main(Main.java:17)
Caused by: java.lang.ClassNotFoundException: com.google.gson.JsonParser
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 4 more
Line 80 of HttpUtils.java;
JsonObject jsonResponse = JsonParser.parseString(response.body()).getAsJsonObject(); // todo: status 200 "success" else failed
accessToken = jsonResponse.get("access_token").getAsString();
System.out.println(accessToken);
I understand this means that JVM can't compile a .class for JsonParser? I suppose this means the compiler has no knowledge of the library existing, which makes me suspect that Gradle isn't configured properly with the project, as it has downloaded the library, but not added a path to it?
I have tried gradle cleanIdea and then gradle idea. I have rebuilt the the project. I have "Mark directory as source root" on various directories for testing being careful to revert when it failed to change behaviour.
Edit;
I have added a package com.example in the src.main.Java directory and added the java files.
I edited run configuration for Main.java to
Main class: com.example.Main
Use classpath of module: src.main
I also changed the build.gradle file to;
application {
// Define the main class for the application.
mainClassName = 'com.example.Main'
}
Main runs but I am stuck at this point, which seems to run indefinitely;
Also, I am sure I right clicked on build.gradle and selected import, although I can't recreate this as the option isn't available now.
Edit 2;
I have been able to get the classes Main and Test with main() to run by putting them in the test/java/src package, and using unusual run configuration with warnings. Although on closer inspection, it seems to be running code that is previously compiled somewhere, as any changes I make aren't reflected in output.
Here is my project structure at the moment;
This is my run configuration that actually runs main in the standard output console, rather than a Gradle Task. It's clearly wrong, as Main is not in the com.example package or src.main module. If I set it correctly using module src.test and main class src.Main Gradle runs as screenshot 5.
Edit 3;
I see now that Gradle has took over responsibility to build and run the java files. I didn't know running in the output could be done with another CLI app and I admit it confused me, so please forgive anything above that seems stupid, I'm learning and figuring this out as I go.
I found in InteliJ settings Build, Execution, Deployment > Build Tools > Gradle I can change the Build and run using option between InteliJ IDEA and Gradle. The only issue I'm having with Gradle now I understand what is happening is Gradle doesn't seem to update my .class files when I run my main() with Gradle. Maybe this is for another question though.

mainClassName = 'java.Main' // changed to 'Main' and I can "gradle run" seems to actually run Main.java
This is not correct. Based on screenshot - you have not package named java (also I doubld that this is a valid name for a Java package). Create proper package inside src/main/java directory and specify it in the Main source file and in build.gradle file.
Also make sure you have imported build.gradle file in IDE, see Link a Gradle project to an IntelliJ IDEA project

Including Local Jars In Groovy

I'm attempting to use "HTTPBuilder" within my simple Groovy script. When I use '#Grab' to import the dependency, everything works fine. Though, I'd like to keep the jar within a different directory and import it using the classLoader function. I've copied the 'http-builder-0.7.jar' that '#Grab' placed into my grapes directory and pasted it into the same directory my Groovy script is running (on Windows). I then comment out the '#Grab' statement and include the classLoader, but get this error:
org.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed: C:\Groovy Scripts\test.groovy: 9: unable to resolve
class HTTPBuilder
Any ideas why the classLoader wouldn't be working in the script? I printed out the path of the jar when importing with '#Grab' and it's definitely using the one within the grape directory. If I uncomment the '#Grab' statement, it works again. Here's the small script...
//#Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7')
this.getClass().classLoader.rootLoader.addURL(new File("http-builder-0.7.jar").toURL())
//return new File(groovyx.net.http.HTTPBuilder.class.getProtectionDomain().getCodeSource().getLocation().toURI().getPath());
def http = new HTTPBuilder('http://httpbin.org/get')

As mentioned, you would be wise to use another method, such as Gradle's application plugin.
However, this is one way to do what you're asking.
First, to get the jar and all dependencies, consider the following Gradle build.gradle script:
apply plugin: 'java'
dependencies {
compile 'org.codehaus.groovy.modules.http-builder:http-builder:0.7'
}
repositories {
jcenter()
}
clean {
doLast {
ant.delete(dir: 'runtime')
}
}
task getDeps(type: Copy) {
from sourceSets.main.runtimeClasspath
into 'runtime/'
doFirst {
ant.delete(dir: 'runtime')
ant.mkdir(dir: 'runtime')
}
}
If you run gradle getDeps, it will write all of the jars into runtime.
Then, in a Unix terminal (for example), you could set the classpath with this (using wildcard syntax from Java 6+, and assuming the path is the same runtime as above):
export CLASSPATH=.:"/user/foo/some/path/runtime/*"
In the same terminal, this will work:
import groovyx.net.http.*
def http = new HTTPBuilder('http://httpbin.org/get')
println "Ready."

Gradle exclude java class from lib replaced by own class to avoid duplicate

In Android Studio, there is a specific file (src/org/luaj/vm2/lib/jse/JavaMethod.java) that I need to overwrite from a package that is pulled in via Gradle (dependencies {compile 'org.luaj:luaj-jse:3.0.1'}).
I copied the file into my source directory with the exact same path and made my changes to it. This was working fine for an individual JUnit test case that was using it. It also looks like it is working for a normal compile of my project (unable to easily confirm at the moment).
However, when I try to run all my tests at once via a configuration of ProjectType="Android Tests", I get Error:Error converting bytecode to dex:
Cause: com.android.dex.DexException: Multiple dex files define Lorg/luaj/vm2/lib/jse/JavaMethod$Overload;.
Is there a specific task or command that I need to add to my Gradle file to make sure the project selects the file in my local source directory? I tried the Copy task and the sourceSets->main->java->exclude command, but neither seemed to work (I may have done them wrong). I also tried the "exclude module/group" directive under "compile" from this post.
The non-default settings for the Run/Debug Confirmation:
Type=Android Tests
Module=My module
Test: All in package
Package: "test"
All my JUnit test cases are in the "test" package.
Any answer that gets this to work is fine. If not Gradle, perhaps something in the android manifest or the local source file itself.
[Edit on 2016-07-24]
The error is also happening on a normal compile when my android emulator is running lower APIs. API 16 and 19 error out, but API 23 does not.

issue: when linking your app the linker finds two versions
org.luaj:luaj-jse:3.0.1:org.luaj.vm2.lib.jse.JavaMethod and
{localProject}:org.luaj.vm2.lib.jse.JavaMethod
howto fix: tell gradle to exclude org.luaj:luaj-jse:3.0.1:org.luaj.vm2.lib.jse.JavaMethod from building
android {
packagingOptions {
exclude '**/JavaMethod.class'
}
}
I have not tried this with "exclude class" but it works for removing duplicate gpl license files a la "COPYING".
If this "exclude" does not work you can
download the lib org.luaj:luaj-jse:3.0.1 to the local libs folder,
open jar/aar with a zip-app and manually remove the duplicate class.
remove org.luaj:luaj-jse:3.0.1 from dependencies since this is now loaded from lib folder

I am not completely sure I understand your problem; however, it sounds like a classpath ordering issue, not really a file overwrite one.
AFAIK, gradle does not make a 'guarantee' on the ordering from a 'dependencies' section, save for that it will be repeatable. As you are compiling a version of file that you want to customize, to make your test/system use that file, it must come earlier in the classpath than the jar file it is duplicated from.
Fortunately, gradle does allow a fairly easy method of 'prepending' to the classpath:
sourceSets.main.compileClasspath = file("path/to/builddir/named/classes") + sourceSets.main.compileClasspath
I don't know enough about your system to define that better. However, you should be able to easily customize to your needs. That is, you can change the 'compile' to one of the other classpath (runtime, testRuntime, etc) if needed. Also, you can specify the jarfile you build rather than the classes directory if that is better solution. Just remember, it may not be optimal, but it is fairly harmless to have something specified twice in the classpath definition.

This is rather convoluted but it is technically feasible. However it's not a single task as asked by the poster:
Exclude said dependency from build.gradle and make sure it's not indirectly included by another jar (hint: use ./gradlew dependencies to check it)
create a gradle task that downloads said dependency in a known folder
unpack such jar, remove offending .class file
include folder as compile dependency
If it's safe to assume that you're using Linux/Mac you can run a simple command line on item 3, it's only using widely available commands:
mkdir newFolder ; cd newFolder ; jar xf $filename ; rm $offendingFilePath
If you don't care about automatic dependency management you can download the jar file with curl, which I believe to be widely available on both linux and mac.
curl http://somehost.com/some.jar -o some.jar
For a more robust implementation you can substitute such simple command lines with groovy/java code. It's interesting to know that gradle can be seen as a superset of groovy, which is arguable a superset of java in most ways. That means you can put java/groovy code pretty much anywhere into a gradle.build file. It's not clean but it's effective, and it's just another option.
For 4 you can have something along either
sourceSets.main.java.srcDirs += ["newFolder/class"]
at the root level of build.gradle, or
dependencies {
. . .
compile fileTree(dir: 'newFolder', include: ['*.class'])
. . .

This is what I ended up adding after Fabio's suggestion:
//Get LUAJ
buildscript { dependencies { classpath 'de.undercouch:gradle-download-task:3.1.1' }}
apply plugin: 'de.undercouch.download'
task GetLuaJ {
//Configure
def JARDownloadURL='http://central.maven.org/maven2/org/luaj/luaj-jse/3.0.1/luaj-jse-3.0.1.jar' //compile 'org.luaj:luaj-jse:3.0.1'
def BaseDir="$projectDir/luaj"
def ExtractToDir='class'
def ConfirmAlreadyDownloadedFile="$BaseDir/$ExtractToDir/lua.class"
def JarFileName=JARDownloadURL.substring(JARDownloadURL.lastIndexOf('/')+1)
def ClassesToDeleteDir="$BaseDir/$ExtractToDir/org/luaj/vm2/lib/jse"
def ClassNamesToDelete=["JavaMethod", "LuajavaLib"]
//Only run if LuaJ does not already exist
if (!file(ConfirmAlreadyDownloadedFile).exists()) {
//Download and extract the source files to /luaj
println 'Setting up LuaJ' //TODO: For some reason, print statements are not working when the "copy" directive is included below
mkdir BaseDir
download {
src JARDownloadURL
dest BaseDir
}
copy {
from(zipTree("$BaseDir/$JarFileName"))
into("$BaseDir/$ExtractToDir")
}
//Remove the unneeded class files
ClassNamesToDelete=ClassNamesToDelete.join("|")
file(ClassesToDeleteDir).listFiles().each {
if(it.getPath().replace('\\', '/').matches('^.*?/(?:'+ClassNamesToDelete+')[^/]*\\.class$')) {
println "Deleting: $it"
it.delete()
}
}
}
}
I'll upload a version that works directly with the jar later.

Another solution if we got then source jar:
task downloadAndCopy {
def downloadDir = "${buildDir}/downloads"
def generatedSrcDir = "${buildDir}/depSrc"
copy {
from(configurations.detachedConfiguration(dependencies.add('implementation', 'xxx:source')))
file(downloadDir).mkdirs()
into(downloadDir)
}
println("downloading file into ${downloadDir}")
fileTree(downloadDir).visit { FileVisitDetails details ->
if (!details.file.name.endsWith("jar")) {
println("ignore ${details.file.name}")
return
}
println("downloaded ${details.file.name}")
def srcFiles = zipTree(details.file).matching {
include "**/*.java"
exclude "**/NeedEclude*java"
}
srcFiles.visit {FileVisitDetails sourceFile ->
println("include ${sourceFile}")
}
copy {
from(srcFiles)
into(generatedSrcDir)
}
}
}
and remember to add depSrc to srcDirs
android {
sourceSets {
`main.java.srcDirs = ['src/main/java', "${buildDir}/depSrc"]
}
}

Running a Gradle compiled Java application with dependencies via the java command

I’m trying to compile the Code Example 3 from this article explaining the Swing Application Framework (JSR 296) with Gradle and to run it from the command line with the java command.
My directory layout looks like this:
├── build.gradle
└── src/
└── main/
└── java/
└── demo/
└── BasicSingleFrameApp.java
The build.gradle file defines a dependency to the appframework:
apply plugin: 'java'
repositories {
mavenCentral()
}
dependencies {
compile 'org.jdesktop:appframework:1.0.3'
}
And this is the BasicSingleFrameApp.java file, a copy of the example of the article mentioned above, enhanced by a package declaration:
package demo;
import org.jdesktop.application.*;
import javax.swing.*;
import java.awt.*;
public class BasicSingleFrameApp extends SingleFrameApplication {
JLabel label;
#Override
protected void startup() {
getMainFrame().setTitle("BasicSingleFrameApp");
label = new JLabel("Hello, world!");
label.setFont(new Font("SansSerif", Font.PLAIN, 22));
show(label);
}
public static void main(String[] args) {
Application.launch(BasicSingleFrameApp.class, args);
}
}
Compiling with gradle build works fine and without errors.
But when I then try to run the BasicSingleFrameApp with
$ java -cp build/classes/main/ demo.BasicSingleFrameApp
I get the error message:
Error: Could not find or load main class demo.BasicSingleFrameApp
When I replace the BasicSingleFrameApp class with a simple “Hello, world!” printing class without dependencies, everything works fine.
I’m confused, because in my understanding I correctly set up the classpath and I don’t understand why the main method (which has the right signature) cannot be found.
This is my Java version:
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
which I am running an Mac OS X Yosemite 10.10.2.

OK, so #chuchikaeschtli helped me to get to the core of the problem, namely that Gradle manages dependencies for compiling and reports to do so for runtime, but more manual tasks are required to make these dependencies available during runtime.
What still confuses me is the unintuitive error message I got:
Error: Could not find or load main class demo.BasicSingleFrameApp
I would have expected a problem that has to do with missing dependencies at runtime to report an error like
error: package org.jdesktop.application does not exist
which is the kind of error I get when these dependencies are missing at compile time.
And as Gradle gave me the following report about runtime dependencies (by running gradle dependencies) I thought that Gradle somehow manages these:
compile - Compile classpath for source set 'main'.
\--- org.jdesktop:appframework:1.0.3
\--- org.jdesktop:swing-worker:1.1
…
runtime - Runtime classpath for source set 'main'.
\--- org.jdesktop:appframework:1.0.3
\--- org.jdesktop:swing-worker:1.1
…
In the end, after knowing that it is a problem of missing runtime dependencies I found several ways to provide them, which I will share.
First solution: Linking to the cache
As stated in this Stackoverflow answer Gradle caches dependencies in $HOME/.gradle, but the actual path to them is tricky. The answer describes a small Gradle task that outputs the full path of each dependency in the cache (in this example for the compile configuration, which is what I need):
task showMeCache << {
configurations.compile.each { println it }
}
In my case gradle showMeCache reports:
$HOME/.gradle/caches/modules-2/files-2.1/org.jdesktop/appframework/1.0.3/338045feff6e61df237aafd11b6f3fe1a3b4e60e/appframework-1.0.3.jar
$HOME/.gradle/caches/modules-2/files-2.1/org.jdesktop/swing-worker/1.1/dc9f8d6f7236087924aad28fbec794a087dd1b3d/swing-worker-1.1.jar
These are long and nasty file paths, but I’m able to construct a java command that works in the style #chuchikaeschtli suggested:
java \
-cp build/classes/main/\
:$HOME/.gradle/caches/modules-2/files-2.1/org.jdesktop/appframework/1.0.3/338045feff6e61df237aafd11b6f3fe1a3b4e60e/appframework-1.0.3.jar\
:$HOME/.gradle/caches/modules-2/files-2.1/org.jdesktop/swing-worker/1.1/dc9f8d6f7236087924aad28fbec794a087dd1b3d/swing-worker-1.1.jar \
demo.BasicSingleFrameApp
This works, but of course this does not feel very “right”. But it helps to understand the problem: it was really a matter of missing dependency jars.
Second solution: Syncing dependencies to the build directory
In the section Using the Sync task of the Gradle User Guide it explicitly uses an example that describes a better solution to the problem at hand:
Here is an example which maintains a copy of the project's runtime dependencies in the build/libs directory.
task libs(type: Sync) {
from configurations.runtime
into "$buildDir/libs"
}
After running this task with gradle libs I’m able to construct a much simpler working java command:
java -cp build/classes/main/:build/libs/appframework-1.0.3.jar:build/libs/swing-worker-1.1.jar demo.BasicSingleFrameApp
Remember: these dependencies haven’t been synced into the build directory by default!
Third solution: Using Gradle to run the application
With the Application Plugin Gradle itself provides a convenient solution to running an application. After adding these two lines to my build.gradle
apply plugin:'application'
mainClassName = 'demo.BasicSingleFrameApp'
I was able to succesfully start the application with just gradle run. Simplest solution so far!
Note that the application plugin also delivers a task named installDist that within build/install creates a runnable distribution of the app and all its dependencies, complete with start scripts for UNIX and Windows systems. This has the advantage that Gradle is not needed for execution of the application.
Also, if you like the gradle run task of the Application Plugin but are concerned about whether Gradle is available at the target system, have a look at the Gradle Wrapper.
Fourth solution: Creating a One-JAR, Fat Jar, or Uber Jar
There seem to exist Gradle-only ways and Gradle plugins that help with the creation of these jars that include all of the required dependencies for the application.
In the case of the gradle-one-jar-Plugin the application can then be started with a java command like:
java -jar build/libs/YOUR_APP_NAME-standalone.jar

from what I get is that you are starting the app just using java so you have to tell java where to find the swing application framework jar you downloaded. so the command should look something like
java -cp pathtojdektopjar/jdesktopjar.jar;build/classes/main/ demo.BasicSingleFrameApp