Develop Reference

How to create a debian distro for a java project using gradle

I see someone has asked this question before. I would have loved to have seen the answer, but it was removed. At the risk of getting down-voted like that post..., I really need help with this, as I've spent a few days on it already and I'm thoroughly at a loss...
I have a java project that's fairly mature. We're preparing to go from alpha phase into a beta release. As a part of that, we want to release installable packages with a proper app with an icon, etc. Creating a (dmg) package for distribution on Mac was extremely easy using the macAppBundle gradle plugin and it works beautifully. I'm now attempting to address distribution on Linux. Ideally, the setupbuilder plugin would be the way to go, but there's a bug that's preventing me from creating a .deb or .rpm package. I submitted the bug to the developer and am currently trying to work around the issue by following this blog post.
I am running an Ubuntu 16.04.3 vm in VirtualBox on my Mac and I can successfully create a working executable by running gradle debianPrepareappname. But when I try to run gradle debian to create the .deb file, the build always fails (currently with this error:).
Process 'command 'debuild'' finished with non-zero exit value 255
When I run debuild manually, I see the following:
debuild: fatal error at line 679:
found debian/changelog in directory
/home/username/appname/build/debian/appname
but there's no debian/rules there! Are you in the source code tree?
No rules file is getting created by gradle. I know that the rules file is basically a makefile... and I'm not very familiar with makefiles in general, let alone creating .deb distros. I know makefiles do compilations and copy files to places in the system, but I don't know what needs to be done to create a .deb file or where things need to go. I mean, the necessary components are there and they work:
appname/build/debian/appname/debian/{bin,lib}
The bin has the working executable and the lib has all the necessary jar files. I just don't know what I need to do in the gradle build script to create the .deb file. Here's what I've got in the gradle build file (I've omitted the macAppBundle and setupbuilder stuff that's just vestigial in there right now, just to keep it simple):
apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'application'
defaultTasks "clean", "fatJar", "eclipse"
version = getVersionName()
sourceCompatibility = 1.7
targetCompatibility = 1.7
repositories {
mavenCentral()
}
dependencies {
compile 'com.miglayout:miglayout-swing:5.0'
compile 'com.googlecode.plist:dd-plist:1.3'
compile 'org.freehep:freehep-graphicsio:2.4'
compile 'org.freehep:freehep-graphicsio-pdf:2.4'
compile 'org.freehep:freehep-graphicsio-ps:2.4'
compile 'org.freehep:freehep-graphicsio-svg:2.4'
compile 'org.freehep:freehep-graphics2d:2.4'
compile 'org.swinglabs.swingx:swingx-autocomplete:1.6.5-1'
}
sourceSets {
main {
java {
srcDir 'src/main/java/'
}
}
}
task fatJar(type: Jar) {
manifest {
attributes 'Main-Class':'com.placeholder.appname'
}
baseName = project.name + '-all'
from {configurations.compile.collect {it.isDirectory() ? it : zipTree(it)}}
with jar
}
def getVersionName() {
def stdout = new ByteArrayOutputStream()
exec {
commandLine 'git', 'rev-parse', '--short', 'HEAD'
standardOutput = stdout
}
return stdout.toString().trim()
}
String applicationVersionFull = getVersionName()
task debianClean(type: Delete) {
delete 'build/debian'
}
tasks.addRule("Pattern: debianPrepare<distribution>") { String taskName ->
if (taskName.startsWith("debianPrepare")) {
task(taskName, dependsOn: [installDist, debianClean]){
String debianDistribution = (taskName - "debianPrepare").toLowerCase()
String debianApplicationVersionFull = getVersionName()
doLast {
copy {
from rootProject.files("build/install/appname")
into rootProject.file("build/debian/appname")
}
copy {
from rootProject.files("gradle/debian/debian")
into rootProject.file("build/debian/appname/debian")
}
}
}
}
}
task debian { // depends on debianPrepare*
doLast {
exec {
workingDir rootProject.file("build/debian/appname")
commandLine "debuild -i -us -uc -b".split()
}
}
}
Everything I've read says this is supposed to be really easy with gradle. The macAppBundle was definitely very easy - it was like 5 lines of code. I barely had to read anything to figure it out and it creates a dmg that has an executable with an icon and everything. I just copied & edited the example in the macAppBundle readme. setupbuilder looked similarly easy, if not for the bug I encountered. Is there a similar example out there for building .deb packages for java projects that doesn't use setupbuilder? I've tried a couple other plugins with no success. I've been googling and I can't find anything straightforward other than the blog post I mentioned. I eventually would like to apply an icon to the executable and other niceties, but first thing is to just get it to build. So why does the rules file not get created? That seems like a good place to start.

I think what you're missing is a "debian" directory with all the related files already present. If you look at syncany's repo https://github.com/syncany/syncany/tree/74c737d871d21dff5283edaac8c187a42c020b20/gradle/debian/debian on github from the blog post you mentioned, you'll see he has 8 files.
At the end of the day, debuild is just bundling a set of files up into an installer. They all have to be there to begin with. His scripts don't create any of these files, just modify some such as the changelog.

Gradle exclude java class from lib replaced by own class to avoid duplicate

In Android Studio, there is a specific file (src/org/luaj/vm2/lib/jse/JavaMethod.java) that I need to overwrite from a package that is pulled in via Gradle (dependencies {compile 'org.luaj:luaj-jse:3.0.1'}).
I copied the file into my source directory with the exact same path and made my changes to it. This was working fine for an individual JUnit test case that was using it. It also looks like it is working for a normal compile of my project (unable to easily confirm at the moment).
However, when I try to run all my tests at once via a configuration of ProjectType="Android Tests", I get Error:Error converting bytecode to dex:
Cause: com.android.dex.DexException: Multiple dex files define Lorg/luaj/vm2/lib/jse/JavaMethod$Overload;.
Is there a specific task or command that I need to add to my Gradle file to make sure the project selects the file in my local source directory? I tried the Copy task and the sourceSets->main->java->exclude command, but neither seemed to work (I may have done them wrong). I also tried the "exclude module/group" directive under "compile" from this post.
The non-default settings for the Run/Debug Confirmation:
Type=Android Tests
Module=My module
Test: All in package
Package: "test"
All my JUnit test cases are in the "test" package.
Any answer that gets this to work is fine. If not Gradle, perhaps something in the android manifest or the local source file itself.
[Edit on 2016-07-24]
The error is also happening on a normal compile when my android emulator is running lower APIs. API 16 and 19 error out, but API 23 does not.

issue: when linking your app the linker finds two versions
org.luaj:luaj-jse:3.0.1:org.luaj.vm2.lib.jse.JavaMethod and
{localProject}:org.luaj.vm2.lib.jse.JavaMethod
howto fix: tell gradle to exclude org.luaj:luaj-jse:3.0.1:org.luaj.vm2.lib.jse.JavaMethod from building
android {
packagingOptions {
exclude '**/JavaMethod.class'
}
}
I have not tried this with "exclude class" but it works for removing duplicate gpl license files a la "COPYING".
If this "exclude" does not work you can
download the lib org.luaj:luaj-jse:3.0.1 to the local libs folder,
open jar/aar with a zip-app and manually remove the duplicate class.
remove org.luaj:luaj-jse:3.0.1 from dependencies since this is now loaded from lib folder

I am not completely sure I understand your problem; however, it sounds like a classpath ordering issue, not really a file overwrite one.
AFAIK, gradle does not make a 'guarantee' on the ordering from a 'dependencies' section, save for that it will be repeatable. As you are compiling a version of file that you want to customize, to make your test/system use that file, it must come earlier in the classpath than the jar file it is duplicated from.
Fortunately, gradle does allow a fairly easy method of 'prepending' to the classpath:
sourceSets.main.compileClasspath = file("path/to/builddir/named/classes") + sourceSets.main.compileClasspath
I don't know enough about your system to define that better. However, you should be able to easily customize to your needs. That is, you can change the 'compile' to one of the other classpath (runtime, testRuntime, etc) if needed. Also, you can specify the jarfile you build rather than the classes directory if that is better solution. Just remember, it may not be optimal, but it is fairly harmless to have something specified twice in the classpath definition.

This is rather convoluted but it is technically feasible. However it's not a single task as asked by the poster:
Exclude said dependency from build.gradle and make sure it's not indirectly included by another jar (hint: use ./gradlew dependencies to check it)
create a gradle task that downloads said dependency in a known folder
unpack such jar, remove offending .class file
include folder as compile dependency
If it's safe to assume that you're using Linux/Mac you can run a simple command line on item 3, it's only using widely available commands:
mkdir newFolder ; cd newFolder ; jar xf $filename ; rm $offendingFilePath
If you don't care about automatic dependency management you can download the jar file with curl, which I believe to be widely available on both linux and mac.
curl http://somehost.com/some.jar -o some.jar
For a more robust implementation you can substitute such simple command lines with groovy/java code. It's interesting to know that gradle can be seen as a superset of groovy, which is arguable a superset of java in most ways. That means you can put java/groovy code pretty much anywhere into a gradle.build file. It's not clean but it's effective, and it's just another option.
For 4 you can have something along either
sourceSets.main.java.srcDirs += ["newFolder/class"]
at the root level of build.gradle, or
dependencies {
. . .
compile fileTree(dir: 'newFolder', include: ['*.class'])
. . .

This is what I ended up adding after Fabio's suggestion:
//Get LUAJ
buildscript { dependencies { classpath 'de.undercouch:gradle-download-task:3.1.1' }}
apply plugin: 'de.undercouch.download'
task GetLuaJ {
//Configure
def JARDownloadURL='http://central.maven.org/maven2/org/luaj/luaj-jse/3.0.1/luaj-jse-3.0.1.jar' //compile 'org.luaj:luaj-jse:3.0.1'
def BaseDir="$projectDir/luaj"
def ExtractToDir='class'
def ConfirmAlreadyDownloadedFile="$BaseDir/$ExtractToDir/lua.class"
def JarFileName=JARDownloadURL.substring(JARDownloadURL.lastIndexOf('/')+1)
def ClassesToDeleteDir="$BaseDir/$ExtractToDir/org/luaj/vm2/lib/jse"
def ClassNamesToDelete=["JavaMethod", "LuajavaLib"]
//Only run if LuaJ does not already exist
if (!file(ConfirmAlreadyDownloadedFile).exists()) {
//Download and extract the source files to /luaj
println 'Setting up LuaJ' //TODO: For some reason, print statements are not working when the "copy" directive is included below
mkdir BaseDir
download {
src JARDownloadURL
dest BaseDir
}
copy {
from(zipTree("$BaseDir/$JarFileName"))
into("$BaseDir/$ExtractToDir")
}
//Remove the unneeded class files
ClassNamesToDelete=ClassNamesToDelete.join("|")
file(ClassesToDeleteDir).listFiles().each {
if(it.getPath().replace('\\', '/').matches('^.*?/(?:'+ClassNamesToDelete+')[^/]*\\.class$')) {
println "Deleting: $it"
it.delete()
}
}
}
}
I'll upload a version that works directly with the jar later.

Another solution if we got then source jar:
task downloadAndCopy {
def downloadDir = "${buildDir}/downloads"
def generatedSrcDir = "${buildDir}/depSrc"
copy {
from(configurations.detachedConfiguration(dependencies.add('implementation', 'xxx:source')))
file(downloadDir).mkdirs()
into(downloadDir)
}
println("downloading file into ${downloadDir}")
fileTree(downloadDir).visit { FileVisitDetails details ->
if (!details.file.name.endsWith("jar")) {
println("ignore ${details.file.name}")
return
}
println("downloaded ${details.file.name}")
def srcFiles = zipTree(details.file).matching {
include "**/*.java"
exclude "**/NeedEclude*java"
}
srcFiles.visit {FileVisitDetails sourceFile ->
println("include ${sourceFile}")
}
copy {
from(srcFiles)
into(generatedSrcDir)
}
}
}
and remember to add depSrc to srcDirs
android {
sourceSets {
`main.java.srcDirs = ['src/main/java', "${buildDir}/depSrc"]
}
}

Intellij + Gradle: properties file

My configuration:
Gradle installed via Homebrew (Mac OS), Intellij Idea 13.
Preferences:
"Use default Gradle wrapper"
and gradle home: /Users/my_username/.gradle
For the following project:
/
build.gradle
gradle.properties
The gradle.properties file contains a variable NAME = PJ.
I also have a gradle.properties under /Users/my_username/.gradle/gradle.properties having a variable TEST=Hello.
For the given task, gradle prints Hello and null
task hello << {
println TEST
println NAME
}
Which means that it does not read by default the gradle.properties in the same directory. Is it a normal behavior or do I miss something in path variable so can the build.gradle script can read it?
A work-around I found is to add:
file("gradle.properties").withInputStream {
stream -> new Properties().load(stream)
}
When I used android studio the .properties file of the same dir was automatically imported.
PS: Although TEST var gets printed I get a warning for Cannot resolve symbol. I saw around and this is typical error (?).

With gradle 2.0 it works perfectly fine:
$HOME/.gradle/gradle.properties
TEST=Hello
gradle.properties
NAME = PJ
build.gradle
task hello << {
println TEST
println NAME
}
When gradle hello run I get the following output:
:hello
Hello
PJ
BUILD SUCCESSFUL
Total time: 2.992 secs

Tested with intelliJ 13.1.4 built-in Gradle 1.7 and downloaded Gradle 2.0 and 2.1 version
Works fine. Make sure that path HOME/.gradle/ is set properly
gradle.properties:
TEST=Hello
NAME=PJ

How can you override an existing Gradle plugin from a local repository?

I am building an application using Gradle, JDK 8, Java FX, and Test FX. I need to be on JDK 8 for our application to work on all platforms with our tech stack. My problem is that I am unable to get code coverage into our build. I found this link...
https://github.com/jacoco/jacoco/issues/74
...and using the Jacoco preview build at the top, I was able to replace my intellij JARs and successfully run my tests and get the coverage. However, I am having trouble putting this into my build.gradle. From what I can tell, I need to add a local repository in my build script...
...
apply plugin: "jacoco"
...
buildscript {
repositories {
// Local Repo
// MVN Repo(s)
}
dependencies {
// Classpaths
}
}
jacoco {
toolVersion = "0.6.4.201311160552" // I need this specific version, which isn't on a MVN repo
}
...I tried to add my local repo several ways including...
flatDir(dirs: "lib")
flatDir dirs: "${projectDir}/lib"
maven { url uri("lib") }
one or two other ways I forget
...my lib folder contains the exact contents, unchanged, from the preview build zip's lib folder in the link above. It doesn't seem like gradle is having a problem locating the local repo, but it is having trouble finding the JAR. I assume there is something wrong with the way I am naming it or the way that it is "packaged". I have tried modifying the JAR names but I keep getting the error...
What went wrong:
A problem occurred configuring root project 'myProject'.
Could not resolve all dependencies for configuration ':classpath'.
Could not find :org.jacoco.agent:.
Required by:
:myProject:unspecified
...any ideas why my JAR is not being found? Thanks!

"Answering" my own question, despite the fact that I still haven't quite figured it out. Anyways, here are two links I found that seem to solve my problem...
http://forums.gradle.org/gradle/topics/jacocotestreport_is_skipping
...following some of these instructions allow my tests to run, but I am still not able to run "gradle jacocoTestReport" without it failing.
UPDATE
OKAY! I figured it out, the link above did help me figure it out. My problem was with the asm-all JAR, since there were several, I did not know which one to use. In order to get jacoco working with Java 1.8, you do not need to specify the toolVersion property. All you need to do is add the following to your dependencies block (not the buildscript block, the code block)...
jacocoAgent files(
"$projectDir/lib/org.jacoco.agent-0.6.4.201311160552.jar")
jacocoAnt files(
"$projectDir/lib/org.jacoco.ant-0.6.4.201311160552.jar",
"$projectDir/lib/org.jacoco.core-0.6.4.201311160552.jar",
"$projectDir/lib/org.jacoco.report-0.6.4.201311160552.jar",
"$projectDir/lib/asm-all-5.0_BETA.jar")
...where the asm-all-5.0_BETA.jar is taken from the org.ow2.asm group found at...
http://mvnrepository.com/artifact/org.ow2.asm/asm-all/5.0_BETA
...hope this helps!

for reference, latest jacoco libs are changed so i'm sharing the following snippet:
dependencies{
jacocoAgent files("$rootProject.projectDir/lib/org.jacoco.agent-0.8.3.201904130250.jar")
jacocoAnt files("$rootProject.projectDir/lib/org.jacoco.ant-0.8.3.201904130250.jar",
"$rootProject.projectDir/lib/org.jacoco.core-0.8.3.201904130250.jar",
"$rootProject.projectDir/lib/org.jacoco.report-0.8.3.201904130250.jar",
"$rootProject.projectDir/lib/asm-7.0.jar",
"$rootProject.projectDir/lib/asm-tree-7.0.jar",
"$rootProject.projectDir/lib/asm-commons-7.0.jar"
)
}

Cannot run Flume because of JAR conflict

I've installed Flume and Hadoop manually (I mean, not CDH) and I'm trying to run the twitter example from Cloudera.
In the apache-flume-1.5.0-SNAPSHOT-bin directory, I start the agent with the following command:
bin/flume-ng agent -c conf -f conf/twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent
My conf/twitter.conf file uses the logger as the sink. The conf/flume-env.sh assigns to CLASSPATH the flume-sources-1.0-SNAPSHOT.jar that contains the definition of the twitter source. The resulting output is:
(...) [ERROR org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows. java.lang.NoSuchMethodError:
twitter4j.FilterQuery.setIncludeEntities(Z)Ltwitter4j/FilterQuery;
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:139)
The conflict results from a FilterQuery class that is defined elsewhere in the flume lib and that does not contain the setIncludeEntities method. For me, the file that contains this class is the twitter4j-stream-3.0.3.jar and I cannot exclude the file from the classpath as suggested here.

I believe this experience was quite frustrating for you, for me it was for sure. The main problem is, both the files, flume-sources-1.0-SNAPSHOT.jar and twitter4j-stream-3.0.3.jar contains the same FilterQuery.class. That is why the conflict message is generated in the log file.
I am not a Java or Big Data expert, but I can give you an alternate to this problem. Download the Twitter4j-stream-2.6.6.jar or lower version from here and replacethe twitter4j-stream-3.0.3.jar. All the 3.X.X uses this class. After replacing, everything should work fine. But you may get some heap error after downloading huge amount of tweets. Please google the solution as it was resolved in 3.X.X files.
-Edit
Also, please don't forget to download and replace all the twitter4j files in /usr/lib/flume-ng folder. Namely, twitter4j-media-support-2.2.6.jar, twitter4j-stream-2.2.6.jar and twitter4j-core-2.2.6.jar. Any mismatch related to version among these files will also create problem.

As suggested in the post a problematic file can be search-contrib-1.0.0-jar-with-dependencies.jar too.

You need to recompile flume-sources-1.0-SNAPSHOT.jar from the git:https://github.com/cloudera/cdh-twitter-example
Install Maven, then download the repository of cdh-twitter-example.
Unzip, then execute inside (as mentionned) :
$ cd flume-sources
$ mvn package
$ cd ..
This problem happened when the twitter4j version updated from 2.2.6 to 3.X, they removed the method setIncludeEntities, and the JAR is not up to date.
PS: Do not download the prebuilt version, it is still the old.

Simply rename all twitter4j-stream* jar files and rerun your flume. It will work with charm. :)

I had the same problem and at last I solved following these steps:
First I renamed all jar files in jarx: from twitter4j-stream-3.0.3.jar -> twitter4j-stream-3.0.3.jarx, ...
This solved the error, but when it tried to estabilish connection, I got error 404:
(Twitter Stream consumer-1[Establishing connection])
[INFO - Twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] 404:
The URI requested is invalid or the resource requested, such as a user, does not exist.)
After reading this page (https://twittercommunity.com/t/twitter-streaming-api-not-working-with-twitter4j-and-apache-flume/66612/11) finally I solved downloading a new version of twitter4j (in the page there's a link).
Probably not the best solution, but worked for me.