I have been having endless grief with Gradle for Android builds after I had to format my Macbook Pro for some stupid corporate domain migration. I have been using Gradle (both off the command line and with Android Studio) to build my projects for over an year now, and never had problems till the fresh install. I spent countless hours over the weekend trying to fix this, but with no luck, and I'm literally on the verge of tearing my hair out! Here is what I'm stuck with:
Gradle builds have slowed to a crawl
I have a multi-project (or multi-module in Android terms) project which used to take around 2 minutes for a clean build - plus uploading archives to the (local) Maven repository. Now,the Gradle configuration phase takes around 8 minutes! Nothing has changed, for after the fresh setup on the Mac, I just took a pull of the sources from the Github repo, and I'm building using the gradle wrapper (as before) which uses version 2.2.1. Not sure if this matters, but the Gradle version on my machine is 2.6. I use Maven - not for builds, but for the local M2 repository, and the Maven version is 3.3.3. Both Maven and Gradle were installed using Homebrew. The Gradle runtimes are the same whether of the command line or using Android Studio. I'm using Android Studio version 1.4-beta4. Here are the things I have set up:
I have set up the Maven settings.xml to point the local Maven repository to the default location `${user.home}/.m2/repository
I have set up Google repository using Android Studio and the Support repository (my project needs Play services and the support library)
We need to upload the archives to the corporate Maven repository on our build servers; to sidestep this, I use the gradle.properties to define the repository URL to be the local M2 repository that's set up in the Maven settings
The project defines Android build tools version 1.1.0, and while this is an older version, I tried with the latest 1.3.1 with no luck on the build times
Possibly related: my Mac Pro uses a good ol' HDD, not the newer solid-state storage. While that can impact build times, the disk was not updated during the format, and also, I presume that it shouldn't result in such multiple orders of magnitude impact
Failure in resolving artifacts from local M2 repo
The primary project that I work on is a library, and we have test clients that we use to verify functionality. The library and test clients are maintained as separate projects in the Github repo. To not have to make any changes in my local development setup, I prefer to deploy the artifacts from my library to my local M2 repo, and then have the test client define and resolve the dependency locally. I accomplish this using the global gradle.properties to override the repository URL (point it to the local M2 repo). This worked just fine till the disk format, but is broken since. Gradle is never able to resolve the artifacts, but I can see them in the local M2 repo. I have googled high and low (on Gradle forums, here on SO), but cannot seem to figure out what I'm missing or doing wrongly. As a work-around, I added the test client as a module to the library project, and am building it as a single multi-module project. BTW, even with this, I still run into the slow Gradle build times problem that I mentioned above.
Can someone help me out?
so there are a few Problems you got to adress:
1. gradle is not resolving the artefacts in the local repository
- maybe it corrupted during the formating (Setup a test Project using maven for
its build only, specify some existing dependencies if you want to verify).
A fresh setup of the repo could resolve this.
2. gradle build is slow
Well, without further Information hard to troubleshoot. Did you refresh your dependencies, clean up the Cache, have enough free disk space in the GRADLE_USER_HOME?
I figured it out. There's one little bit I'm still not able to grok, but that aside here is what was wrong:
Slow gradle builds
The project's build.gradle file had declared a dependency on our corporate SCM Maven repository. I replicate whatever dependencies my project actually needs in the local M2 repo. Once I replaced this with a file URL pointing to the local M2 repo, Gradle build times dropped to the earlier 30-40 seconds. The part I'm still confused about is that even with the remote repo, Gradle should have downloaded the artifacts once and cached them in the Gradle cache. Still need to figure out why it's not doing that.
Failure in resolving artifacts from local M2 repo
This was a bad error on my part. In my gradle.properties I'd listed the SNAPSHOT_REPOSITORY_URL override as /Users/me/.m2/repository (with me replaced with the username). I missed the file:// prefix here. I'm somewhat surprised that Gradle didn't call it out, and instead deployed the artifacts at the directory pointed by /Users/me/.m2/repository, but may be I don't understand how this works so well. Once I added the file:// prefix, all the test project builds worked like a charm!
Related
I have a library and a program, both under my control and built using Gradle. What's the best way to develop these two at the same time?
I have set up a private maven repository to distribute the library and that's working, but I don't want to release to that repository every little experiment I make during development. It's slow and disruptive to users of the library.
I tried installing the jar to the local maven repository as explained here: Gradle alternate to mvn install but the project that's using the library is not picking up that newly installed version.
I think, you can try to use multi-project builds for that if it's possible. But you will likely need to restructure both your current projects to become modules of the same new project.
What's the best way to develop these two at the same time?
It depends by how the team is organized and what are your policies.
Of course if the team can use a git repo and access to the source code you can just use git without pushing a new version on the maven server for each commit or push.
Otherwise if other users can only use the final library, you have to push the version on the maven server.
I have set up a private repository to distribute the library and that's working, but I don't want to release to that repository every little experiment I make during development. It's slow and disruptive to users of the library.
Every maven repo has 2 different repositories:
release
snapshot
Usually release repo is used only for stable releases and the snapshot repo is used to publish little change, beta release and so on.
In any case it is not required that every changes in the code is pushed in the maven repo (it is a your choice)
It's slow
The time to upload artifacts usually is not so big, in any case you can evaluate to push the release in the maven repo with a CI server.
The best method seems to be to make one project include the other one when present by adding:
if (file("../libraryproject").exists()) {
includeBuild "../libraryproject"
}
to the settings.gradle file of the project that uses the library. That can be committed to the source code repo because when that directory doesn't exist, the dependency will be included in the traditional way.
This is called Composite Build in the Gradle world, IntelliJ seems to handle properly and theres'a recorded webcast showing the whole setup: https://www.youtube.com/watch?v=grPJanXfRPg
Currently, my built structure for a plugin in is a bit messy: I'm using the normal IDEA project file to build the plugin locally. When I push it to the repo and travis-ci is building it, it uses the maven pom.xml because for travis to work, it always has to download the complete IDEA sources.
Although this works, this has several drawbacks:
I need to keep two built mechanisms up to date. This is
When a new IDEA version is out (every few weeks), I need to change the SDK in maven and in my IDEA settings
When I add a new library, change resources, etc. I need to do this for two the two settings as well
I ran into problems when I kept the IDEA Maven plugin turned on because it saw the pom.xml and interfered with my local built. Turning it off means, I cannot download libraries with Maven which has the feature of tracking dependencies.
I saw that Gradle has an 'idea' plugin and after googling, I got the impression that Gradle is the preferred choice these days. I have seen Best way to add Gradle support to IntelliJ IDEA and I'm sure I can use the answers there to turn my pom.xml into a valid build.gradle.
However, maybe someone else has already done this or can provide a better approach. What I'm looking for is a unified way to build my plugin locally and on Travis-CI.
Some Details
For compiling an IDEA plugin, you need its SDK which you can access through an installation of IDEA or a download of the complete package. Locally, I'm using my installation for the SDK. With Travis, my maven built has the rule to download the tar.gz and extract it.
It turns out that in particular for building an IntelliJ plugin, Gradle seems to have many advantages. This is mainly due to the great IntelliJ plugin for Gradle which makes compiling plugins so much easier. With Gradle, I could turn my >220 lines of Maven build into a few lines of easily readable Gradle code. The main advantages are that
It takes care of downloading and using the correct IDEA SDK while you only have to specify the IDEA version.
It can publish your plugin to your Jetbrains repository and make it instantly available to all users
It fixes items in your plugin.xml, e.g. you can use one central version number in gradle.build and it will keep plugin.xml up-to-date or it can include change-notes
It seamlessly integrates with Travis-CI
How to use Gradle with an existing IDEA plugin
Do it manually. It's much easier.
Create an empty build.gradle file
Look at an example and read through the README (there are many build.gradle of projects at the end) to see what each intellij property does.
Adapt it to your plugin by
Setting the intellij.version you want to build against
Setting your intellij.pluginName
Define where your sources and resources are
Define your plugin version
Define a Gradle wrapper that enables people (and Travis) to build your plugin without having Gradle
Create the gradle wrapper scripts with gradle wrapper
Test and fix your build process locally with ./gradlew assemble
If everything works well, you can push build.gradle, gradlew, gradlew.bat and the gradle-folder to your repo.
Building with Travis-CI
For Travis you want to use the gradlew script for building. To do so, you need to make it executable in the travis run. An example can be found here.
Recently, a project I need as a dependency for some of my programs has switched to providing patch files instead of actual jars (legal reasons).
They included a small tool to automatically patch your existing jars with the new update you downloaded.
I could write a small program that automatically downloads and patches the file, all I need is a way to tell maven not to download the file but to run a command and then use the file from my local repo.
Is there a way to do this (maybe using a custom plugin)?
You could write a small ant task run the tool and call this task from your maven build using maven-antrun plugin.
You could also write your own maven plugin but for such a simle task I would not recommend it.
The reason why your IDE does not find the dependency is that it is not published anymore (not the version you seek apparently).
Building the artifact and installing it locally as #Joe suggested is an elegant solution to solve your issue from my perspective.
The advantage is twofold:
Your IDE can seamlessly find the missing artifact
You can keep multiple version of that dependency (one for every new patch) and thus reproduce previous (working) build in case a new patch breaks something.
if you have a local repository, it is even better as this would maven the dependency available to all your team.
Of course you would still need a tool to download new patch, apply them and deploy the produced artifact to the repository automatically. But doing this as a separate task give you more flexibility concerning the tools you could use.
I have a Java-based GitHub project, fitnessjiffy-spring (I'm currently focused on the "bootstrap" branch). It depends on a library built from another GitHib project, fitnessjiff-etl. I am trying to configure both of these to be built by Travis CI.
Unfortunately, Travis is not as sophisticated as Jenkins or Hudson in dealing with Maven-based Java projects. Jenkins can easily handle dependencies between projects, but the same concept doesn't seem to exist with Travis. If one project depends on another, then that other project must already be built previously... and its artifact uploaded to some Maven repo where the first project can download it later.
My "fitnessjiffy-etl" library is building and deploying just fine. I'm using Bintray for Maven repository hosting, and you can clearly see my artifacts over plain HTTP at:
http://dl.bintray.com/steve-perkins/maven/
In my "fitnessjiffy-spring" project, I am adding this Maven repo location directly in the pom.xml, so that Travis will be able to find that artifact dependency. Here is the state of my POM at the time of this writing. Note the <repositories> element at the bottom of the file.
When I build this project locally, it works just fine. I can see it downloading the Maven artifact from "http://dl.bintray.com/...". However, when I try to build on Travis CI it fails every time. I can see in the console log that Travis is still trying to download the artifact from Maven Central rather than my specified repo.
Does this make sense to anyone else? Why does Maven utilize a custom repository location in a POM file when building locally, but ignores this configuration when running on a Travis CI build?
From digging into this further, I discovered that Travis uses its own proxy for Maven Central, and has configured Maven to force ALL dependency requests through their proxy. In other words, it does not seem possible at this time to use additional Maven repos specified in the POM file of a project built on Travis.
In my case, I ended up refactoring such that project would not need the outside JAR dependency. I also switched to Drone.io, so I could manage my settings on the build server rather than having to carry a YAML file in my repository (which always struck me as a bit daft).
However, even on Drone it's still a major hassle to manage dependencies between multiple projects (extremely common with Java development). For Java, I just don't think there's currently an adequate substitute for Jenkins or Hudson, maybe running on a cheap Digital Ocean droplet or some other VPS provider instance.
In your install phase add a $HOME/.m2/settings.xml define your custom repository.
cache:
directories:
- "$HOME/.m2"
install:
- curl -o $HOME/.m2/settings.xml
https://raw.githubusercontent.com/trajano/trajano/master/src/site/resources/settings.xml
- mvn dependency:go-offline
script:
- mvn clean install site
I am on Netbeans and don't know Maven much. Whenever I import, open some Maven project, it starts donwloading something from some central repository, sometimes huge. It downloads things in .m2\repository.cache\m2e. I have limited bandwidth and don't want this. How to stop this?
I have set Options>Java>Maven>Dependency Download Strategy to never. Also tried mvn -o install and mvn -o for offline. Not solved.
The Maven way is to get you what the project says it needs, but you have not already downloaded to your local repository.
The huge file is the list of what is actually available in Maven Central, and for some reason unknown to me it is downloaded on a regular basis. If you do it once, it should be kept for future sessions.
Maven will download all the dependency only once to the local repository and not again and again.
Weather you have limited or unlimited bandwidth you have to download it to execute your project.
Maven has a very modular architecture. That means the the thing you get when you download the Maven distribution is in reality small core functionality.
The rest is downloaded from a Maven artifact repository, like Maven Central (which is the default repo).
Note that this applies not only for dependencies (the library your project uses), but also your plugins (i.e. the stuff that compiles, packages, and otherwise builds the projects). Hence the large number of downloads.
Like the other answers said, if you don't delete your local repository it should eventually contain all the artifacts (dependencies and plugins) you need without re-downloading. The only exception are SNAPHSOT dependencies which can get re-downloaded periodically, depending what's in your POM and settings.
Ultimately, you have two possibilities:
If you have access to a higher-bandwith connection somewhere, you can build the projects while using it, and your local repo will still store the needed artifacts.
If you have several computers/configurations behind a local network, you can set up a Maven repository manager, like Nexus or Artifactory, and use it as a local mirror. Note that those still need to download the artifacts at first as well.
But there isn't much else you can do. "Maven downloading the Internet" is, unfortunately in your case, by design.