We're currently deploying our Flink applications as a fat-jar using the maven-shade-plugin. Problem is, each application jar ends up being approximately 130-140 MB which is a pain to build and deploy every time. Is there a way to exclude dependencies and just deploy a thin jar to the cluster which comes up to about 50 kB?
You can place the dependency JARs in the cluster beforehand in Flink's lib (see Avoid Dynamic Classloading) and just upload the thin JAR on each job submission.
Here's how we do it with Gradle!
We have two sub-projects:
job: For the stream job that we want to run
runtime: For additional runtime dependencies (e.g. a custom FileSystem implementation)
We create a new gradle configuration for dependencies that are provided at runtime:
configurations {
provided,
compile.extendsFrom provided
}
and then mark the provided dependencies as:
provided("org.apache.flink:flink-java:1.6.0") // flink java v1.6.0
Then, we modify the jar task to build a jar without any provided dependencies:
jar {
dependsOn configurations.runtime
from {
(configurations.runtime - configurations.provided).collect {
it.isDirectory()? it : zipTree(it)
}
} {
exclude 'META-INF/*.RSA'
exclude 'META-INF/*.SF'
exclude 'META-INF/*.DSA'
}
manifest {
attributes 'Main-Class': 'com.example.Entrypoint'
}
}
The result is a jar with required dependencies (compile) bundled which we then deploy using the Web UI.
As for the custom runtime dependencies, we build a custom Docker image and push the built artifact (runtime.jar, built using same configuration as above) to the libs/directory in Flink. You can do that manually too if you are not using Docker.
And, lastly, in our particular case there is no direct dependency defined between our job and the runtime dependency (which is discovered using reflection).
Related
I added this artifact which is a war to my gradle project dependencies.
I need to extend some classes, and use a modified servlet contexts from it.
I was expecting the war to be imported as is then I would use gradle tasks to manipulate to include the jars to dependencies, copy static resources to correct classpath etc.
But gradle actually added a bunch of jars to dependency.
Im not sure if gradle scanned recursively all paths for jars and poms or probably just the jars under the WEB-INF/classes folder in the war.
I can assume probably not the poms repositories as stated here.
Im I correct is assuming the jars in the WEB-INF/lib folder in the deflated war were not imported? its hard to tell as there are a lot of shared dependencies between my project and the war in question
Then whats the best way to declare a dependency on a war in the maven repo/jcenter if I need to extend and modify as I described at the top?
UPDATE:
I am now trying to use answer below and this solution https://discuss.gradle.org/t/how-to-add-an-artifactory-war-as-a-gradle-dependency/19804/2
, This only worked after moving the directory with the copied jars outside the buildDir
my build.gradle
configurations {
warOnly
}
dependencies {
// not working implementation fileTree('$buildDir/explodedWar/WEB-INF/classes')
implementation fileTree('anotherDir/explodedWar/WEB-INF/classes')
// implementation fileTree('$buildDir/explodedWar/WEB-INF/lib')
implementation fileTree('anotherDir/explodedWar/WEB-INF/lib')
warOnly 'ca.uhn.hapi.fhir:hapi-fhir-jpaserver-starter:4.2.0#war'
}
tasks.register("explodeWar",Copy) {
from zipTree(configurations.warOnly.singleFile)
// into "${buildDir}/explodedWar"
into "anotherDir/explodedWar"
}
compileJava {
dependsOn explodeWar
}
By declaring a dependency on a WAR, Gradle will simply add it to the list of files for the matching configuration. So if you add a WAR in implementation, it will simply be on the compileClasspath and runtimeClasspath without any processing.
So for sure Gradle will not transform your WAR dependency in a dependency on the JARs it contains.
If you want to use a WAR to copy and modify some of its content before repackaging it, you can use an isolated and custom configuration to resolve it from a remote repositories. Then you will define a Gradle task that will take the files of that configuration as the input and do the required processing on the WAR. Note that the task could also be the starting point of a series of tasks manipulating the WAR to one output, then that output to another one, etc ...
configurations {
warOnly
}
dependencies {
warOnly "com.pany:some-war:1.0"
}
tasks.register("copyWar", Copy) { // Register a copy task to modify the WAR
from(zipTree(configurations.warOnly)) // I did not run this, so you may have to get to the single file instead
// Regular copy configuration to decide a destination, perform on the fly changes, etc ...
}
It took some trail and error but there is better way. When using the dependency:
providedCompile 'ca.uhn.hapi.fhir:hapi-fhir-testpage-overlay:6.0.1'
Gradle will download a war file and place it you your classpath. However providing a classifier will help here. The following dependency will get the jar file.
providedCompile (group: 'ca.uhn.hapi.fhir', name: 'hapi-fhir-testpage-overlay', version:'6.0.1', classifier: 'classes')
I'm having a problem on a larger multi-project build that I was migrating from Gradle 4 to 5, and replicated the issue on a smaller, more concise build to demonstrate the issue.
I have 2 projects in the build. One is a dependency (basic library) used by the other.
demo (root project)
|- build.gradle
|
|--- demo-web
|---|- build.gradle
|
|--- demo-dependency
|---|- build.gradle
Snippet of demo-web: build.gradle
...
dependencies {
implementation project(':demo-dependency')
...
}
...
The dependency project defines one class that is used in the web project, DownstreamThing.
The web project attempts to use that to construct its own object, but during a build on the root project level, it fails.
> ./gradlew build
> Task :demo-web:test
com.example.demo.ThingTest > testThing FAILED
java.lang.NoClassDefFoundError at ThingTest.java:12
Caused by: java.lang.ClassNotFoundException at ThingTest.java:12
ThingTest.java
#Test
public void testThing() {
DownstreamThing t = new DownstreamThing(); //line 12, ClassNotFoundException
assertTrue(t != null);
}
I didn't have any trouble with this in Gradle 4, but only in Gradle 5. Why is the dependency not being found during the test task?
Full source for the example is here: https://bitbucket.org/travelsized/gradle-problem/src/master/
I believe he reason you are getting that exception is because you have applied the Spring Boot plugin to the demo-dependency project. What this plugin does is to repackage the jar file to a fat jar that needs a special classloader to load the content.
You can still use the Spring Boot dependencies (e.g. the starters) in the dependency project without the plugin. So if you can, just remove it.
If there is a particular reason why you have it, you will need to keep the original jar file so that is used as the actual dependency. For Spring Boot 1.5.x, you can do that with something like this:
bootRepackage {
classifier = "boot"
}
bootJar {
enabled = true
}
But do note that I don't think Spring Boot 1.5 is fully compatible with Gradle 5 and later (at this time we are on 6.0), so you might need to either downgrade Gradle or upgrade Spring Boot.
Bjørn Vester's answer got me pointed in the right direction. The spring-boot plugin was causing the jar tasks to go awry. I needed to make sure that the bootJar task was disabled for the dependency while the jar task was enabled.
The changes in the configuration to do this between versions of Gradle and the Spring Boot plugin made this get lost in the upgrades.
Previously, I could specify a classifier for the jar post-boot:
bootRepackage {
classifier = 'exec'
}
Now, I need to enable and disable the appropriate tasks:
bootJar {
enabled = false
}
jar {
enabled = true
archiveBaseName = "demo-dependency"
}
In the larger project, I previously had a jar task that specified the archiveBaseName, but didn't explicitly enable it to override the bootJar task. Once I made the above changes (while keeping the boot plugins in place), things started working.
I'm new to Gradle and I'm facing a problem ( using Gradle 2.14)
My project has 3 subprojects: assembly, sprinBootProject, E2E
The assembly module is responsible of zipping the jar of the spring boot application along with properties files.
The E2E module is an 'end to end' tests module for testing the spring boot application.
I'm using the distribution plugin in assembly/build.gradle like this:
apply plugin: 'distribution'
distributions {
main {
baseName "project-assembly"
contents{
from project(':springBootProject').fileTree(dir: 'build/libs', include: ['*.jar'])
from project(':springBootProject').fileTree(dir: 'build/resources/main', includes: ['*.properties', 'conf/*.properties'])
}
}
}
but I need to make sure that springBootProject will be evaluated before the assembly subproject, so I used dependsOn like that:
distZip.dependsOn(':springBootProject:build')
when I added this line I see that the E2E module is failing in compileTestJava task, so this line is effecting how the spring boot application is generated (I think)
E2E/build.gradle:
dependencies {
testCompile project(":springBootProject")
}
so my question is am I doing it right? why when I added the dependsOn line the E2E module is effected, and how can I force Gradle to evaluate the spring boot application before the assembly module?
I figure it out.
Gradle will evaluate modules by alphabetic order
so the evaluation will be: assembly -> E2E -> springBootProject
when adding this line to assembly/build.gradle:
distZip.dependsOn(':springBootProject:build')
the order of the evaluation changed to be: assembly -> springBootProject -> E2E
now in 'springBootProject' module I am using the spring boot plugin in order to generate an executable jar, this executable jar is generated by the task :springBootProject:bootRepackage which uses the jar generated by the :springBootProject:jar task and convert it to executable.
now because the 'E2E' module depends on 'springBootProject' it will use it's jar, but now this jar is executable, and because of that I had a compilation error in 'E2E'.
so I have to make sure that the 'E2E' module gets the original jar and not the executable one.
so I added this line in springBootProject/build.gradle:
bootRepackage.mustRunAfter(':E2E:test')
which seem to solve the problem
I have a java gradle project. I have a dependency.
dependencies {
compile project(":mymodule")
compile 'org.springframework:spring-context:4.1.2.RELEASE'
}
Where can I find and use exact jar file name of both my custom module and spring dependency jar to process it further in composing application's libs, folders, etc?
Stick this in a new task:
project.configurations.compile.each{ println it}
Or, for just one specific dependency:
println project.configurations.compile.find {it.name.startsWith("something") }
However, if you're looking to create distributable packages including dependencies, you really should look into the gradle application plugin.
In the gradle documentation (sec. 8.3) defined the java plugin classpath.
In Gradle dependencies are grouped into configurations. A
configuration is simply a named set of dependencies. We will refer to
them as dependency configurations. You can use them to declare the
external dependencies of your project. As we will see later, they are
also used to declare the publications of your project.
Is it the same as the Java classpath we define as export CLASSPATH = ...?
Adding comment as answer with an example
It is more than that. Think of it as a bucket which handles dependencies. From Java plugin, we get configurations like compile, runtime, testCompile etc which handles dependencies in respective manner. You can have your own configuration and use that configuration for any dependency. Now what you do with that configuration dictates how you handle the dependencies under that configuration.
For example, here is a sample configuration setup which extracts a zip archive and copies its content to a directory.
configurations {
extractZips // custom configuration
}
dependencies {
extractZips 'com.mycompany:my-dependency:0.1#zip'
}
// Task configured to copy contents from zip archive
task extractZip(type: Copy) {
from zipTree( configurations.extractZips.singleFile )
into "$projectDir/zips" // for example
}