I need to build a custom Ant script that builds a project based on CI output. We use Atlassian Bamboo as CI server.
Normally our projects have a dependency to our platform module, managed via Ivy/Artifactory.
Our typical dependencies.xml file contains a dependency to that module, transitively. And other potential dependencies. As an example, our core module depends on lots of Spring packages, but not on Spring Boot. If a project needs Spring Boot too, it will define its dependency in its dependencies.xml file along with <depencency org="com.acme" name="core-platform"...
My goal now is to exclude com.acme#core-platform from resolution, because I am making a different task that uses Bamboo output artifact to take the latest build of the core module and its dependencies without going through Artifactory.
This is very important because during a build I may like to change the version of a dependent package (e.g. upgrade Spring 4.3.1 to 4.3.3) and test with the proper Spring. If I simply resolve dependencies to com.acme#core-platform#latest.release, which is released on Artifactory, I won't take 4.3.3 of Spring which was committed to Git and available in core-platform's currently-building dependencies.xml. I hope my explanation is easy to understand.
So let's say I have this dependency list as an example
com.acme#core-platform#${version}
org.hibernate#hibernate-java8#5.1.0.Final
org.springframework.boot#spring-boot-starter-web#1.3.1.RELEASE
commons-collections#commons-collections#3.2.2
.... others
Full dependency is
<dependencies>
<dependency org="com.acme" name="core-platform" rev="${version}" transitive="true" conf="runtime->runtime" changing="true"/>
<dependency org="com.acme" name="core-platform" rev="${version}" transitive="true" conf="compile->compile" changing="true"/>
<dependency org="com.acme" name="core-platform" rev="${version}" transitive="true" conf="provided->provided" changing="true"/>
<dependency org="com.acme" name="core-platform" rev="${version}" transitive="true" conf="junit->junit" changing="true"/>
<dependency org="com.acme" name="core-platform" rev="${version}" transitive="true" conf="test->test" changing="true"/>
<!-- https://mvnrepository.com/artifact/org.hibernate/hibernate-java8 -->
<dependency org="org.hibernate" name="hibernate-java8" rev="5.1.0.Final" transitive="false" />
<dependency org="org.springframework.boot" name="spring-boot-starter-web" rev="1.3.1.RELEASE" transitive="false" />
<dependency org="org.springframework.boot" name="spring-boot-starter-tomcat" rev="1.3.1.RELEASE" transitive="false" />
<dependency org="org.springframework.boot" name="spring-boot-starter-validation" rev="1.3.1.RELEASE" transitive="false" />
<dependency org="commons-collections" name="commons-collections" rev="3.2.2" transitive="false" />
<!-- jackson2 libs -->
<dependency org="com.fasterxml.jackson.datatype" name="jackson-datatype-jdk8" rev="2.8.1" transitive="false" conf="runtime->*"/>
<dependency org="com.fasterxml.jackson.datatype" name="jackson-datatype-jsr310" rev="2.8.1" transitive="false" conf="runtime->*"/>
<exclude module="joda-time" />
<exclude module="jackson-datatype-joda" />
</dependencies>
I simply want to take Hibernates' Java8, commons-collections, etc.
Creating a duplicate dependencies.xml is not an option
I was considering manipulating the dependencies.xml via Ant and have it exclude the acme modules by regex. Feasible but tricky
Unfortunately I can't combine Ant task's ivy:retrieve with attributes file and element exclude, because that would have helped a looooooot
Any ideas?
It is hard to understand your requirement. I suspect that your problem could be solved by creating an additional configuration and use configuration mappings to control the downloads.
Example
This build creates two directories. The first contains the log4j dependency without transitive dependencies, the second includes the remote module's optional dependencies. If you look at the remote POM you'll see they have a different scope.
├── build.xml
├── ivy.xml
├── lib1
│ └── log4j-1.2.17.jar
└── lib2
├── activation-1.1.jar
├── geronimo-jms_1.1_spec-1.0.jar
├── log4j-1.2.17.jar
└── mail-1.4.3.jar
build.xml
<project name="demo" default="resolve" xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="resolve">
<ivy:resolve/>
<ivy:retrieve pattern="lib1/[artifact]-[revision](-[classifier]).[ext]" conf="noDependencies"/>
<ivy:retrieve pattern="lib2/[artifact]-[revision](-[classifier]).[ext]" conf="withDependencies"/>
</target>
</project>
Notes:
Each "retrieve" task creates a directory containing the files that make up the configuration.
ivy.xml
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="noDependencies" description="File grouping that has no transitive dependencies"/>
<conf name="withDependencies" description="File grouping that contains dependencies"/>
</configurations>
<dependencies>
<dependency org="log4j" name="log4j" rev="1.2.17" conf="noDependencies->master; withDependencies->master,optional"/>
</dependencies>
</ivy-module>
Notes:
Note how the configuration is declared at the top of the ivy file and the dependency contains two configuration mappings
Additional
The following answer explains how ivy interprets Maven modules. It creates configurations that can be used to decide which files should be downloaded:
How are maven scopes mapped to ivy configurations by ivy
Ok, looks like the replace trick is very easy too.
Add the following markers <!-- DEPS START --> and <!-- DEPS END --> (or any of choice) between the parts of the dependencies.xml file to ignore
Hack via Ant
<copy file="dependencies.xml" tofile="ci/hacked-dependencies.xml" overwrite="true">
<filterchain>
<replacestring from="<!-- DEPS START -->" to="<!--" />
<replacestring from="<!-- DEPS END -->" to="-->" />
</filterchain>
</copy>
Example
<!-- DEPS START -->
<dependency org="com.acme" name="core-platfrom" rev="${version}" transitive="true" conf="runtime->runtime"/>
<!-- DEPS END -->
Related
My dependencies.xml contains a lot of packages required for my web application to run
Following are a few notable fragments
<configurations>
<conf name="test" visibility="public" extends="compile" />
<conf name="compile" visibility="public" extends="runtime" />
<conf name="runtime" visibility="public" />
<conf name="provided" visibility="public" />
<conf name="junit" visibility="public" extends="test" />
</configurations>
<publications>
<artifact name="${project.name}" type="jar" ext="jar" conf="compile" />
<artifact name="${project.name}" type="zip" ext="zip" conf="compile"/>
</publications>
<dependencies>
<dependency org="org.hibernate" name="hibernate-core" rev="5.1.3.Final" transitive="false" conf="runtime->*"/>
<dependency org="org.hibernate" name="hibernate-ehcache" rev="5.1.3.Final" transitive="false" conf="runtime->*"/>
<dependency org="org.hibernate.common" name="hibernate-commons-annotations" rev="5.0.1.Final" transitive="false" conf="runtime->*"/>
<dependency org="org.hibernate.javax.persistence" name="hibernate-jpa-2.1-api" rev="1.0.0.Final" transitive="false" conf="runtime->*"/>
<dependency org="org.javassist" name="javassist" rev="3.21.0-GA" transitive="false" conf="runtime->*"/>
<dependency org="org.jboss.logging" name="jboss-logging" rev="3.3.0.Final" transitive="false" conf="runtime->*"/>
<dependency org="javax.transaction" name="jta" rev="1.1" transitive="false" conf="runtime->*"/>
<dependency org="net.sf.ehcache" name="ehcache-core" rev="2.6.11" transitive="false" conf="runtime->*"/>
<dependency org="antlr" name="antlr" rev="2.7.7" transitive="false" conf="runtime->*"/>
<dependency org="org.antlr" name="antlr4-runtime" rev="4.5.2-1" transitive="false" conf="runtime->*"/>
</dependencies>
I have package JTA from javax.transaction that is required for the application to run under Tomcat and forbidden for the application to run under WebSphere.
I need to know how to make two different WAR files depending on the target platform. I don't exactly know how to use configurations, if that is the way.
Ant will do an ivy-retrieve for configuration runtime and build a WAR archive using the jars downloaded from Artifactory.
I could exclude thos jar(s) manually by doing a delete after Ivy has resolved the artifacts, but hey, we are cool developers and we like to do the things the cleaner way.
How would you suggest me to do an ivy-retrieve that targets Tomcat including JTA an another that targets Websphere excluding it?
build.xml : Building war files
The following fragment builds two war files:
../demo.war
../demo-websphere.war
The magic is that the tomcat retrieve task includes two configurations:
<ivy:retrieve pattern="${lib.dir}/tomcat/[artifact].[ext]" conf="runtime,tomcat_only"/>
<war destfile="${dist.dir}/demo.war" webxml="${resources.dir}/web.xml">
<fileset dir="${resources.dir}" excludes="web.xml"/>
<lib dir="${lib.dir}/tomcat"/>
</war>
<ivy:retrieve pattern="${lib.dir}/websphere/[artifact].[ext]" conf="runtime"/>
<war destfile="${dist.dir}/demo-websphere.war" webxml="${resources.dir}/web.xml">
<fileset dir="${resources.dir}" excludes="web.xml"/>
<lib dir="${lib.dir}/websphere"/>
</war>
build.xml : Publishing
The following answer contains more details on publishing multiple module artifacts to a Maven repository
how to publish 3rdparty artifacts with ivy and nexus
So we need a target to generate the POM file:
<target name="prepare" description="Generate POM">
<!-- Optional: Intermediate file containing resolved version numbers -->
<ivy:deliver deliverpattern="${build.dir}/ivy.xml" pubrevision="${publish.revision}" status="release"/>
<!-- Generate the Maven POM -->
<ivy:makepom ivyfile="${build.dir}/ivy.xml" pomfile="${build.dir}/demo.pom"/>
</target>
And a second target that publish the built files:
<target name="publish" depends="init,prepare" description="Upload to Nexus">
<ivy:publish resolver="nexus-deploy" pubrevision="${publish.revision}" overwrite="true" publishivy="false" >
<artifacts pattern="${build.dir}/[artifact](-[classifier]).[ext]"/>
</ivy:publish>
</target>
Take careful notice of the optional "classifier" attribute and note how the ivy file is structured next
ivy.xml
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="master"/>
<conf name="default" extends="master,runtime"/>
<conf name="compile"/>
<conf name="provided"/>
<conf name="runtime" extends="compile"/>
<conf name="test" extends="runtime"/>
<conf name="tomcat_only" description="A special configuration for special tomcat only dependencies"/>
</configurations>
<publications>
<artifact name="demo" type="war" conf="master"/>
<artifact name="demo" type="pom" conf="master"/>
<artifact name="demo" type="war" conf="master" e:classifier="websphere"/>
</publications>
<dependencies>
<!-- Compile dependencies -->
<dependency org="org.hibernate" name="hibernate-core" rev="5.1.3.Final" conf="compile->default"/>
<dependency org="org.api" name="slf4j-api" rev="1.7.22" conf="compile->default"/>
<!-- Runtime dependencies -->
<dependency org="org.slf4j" name="slf4j-log4j12" rev="1.7.22" conf="runtime->default"/>
<!-- Tomcat dependencies -->
<dependency org="javax.transaction" name="jta" rev="1.1" conf="tomcat_only->master"/>
</dependencies>
</ivy-module>
Few things going on here.
1)
Make configuration mappings work for you
config1->default # Remote artifact plus transitive dependencies
config2->master # Remote artifact only (Same as setting transitive=false)
2)
The "extends" attribute is a set operation, meaning a compile dependency is automatically included as a runtime configuration.
An example is the SLF4J libary. The slf4j-api jar is required when compile code, where as the slf4j-log4j12 jar contained bindings and dependencies on log4j, a runtime (and changeable) dependency.
3)
The "master" configuration in your module is special and by convention in Maven world corresponds to the files published by this module.
4)
The "classifier" attribute is an example of a extra attribute in ivy.
I have tried to find the right org=".. address for JaCoCo so that I can simply add the address to my ivy.xml file and it will take care of the dependency.
But I have trouble finding the right link and was wondering if anyone knew what it was?
I am going to use Ant to build my project, so I'd have to put JaCoCo to use in there.
None of the below are correct:
<!-- JaCoCo -->
<dependency org="org.jacoco.agent" name="jacoco-agent" rev="0.6.2.2" />
<dependency org="org.jacoco.core" name="jacoco-core" rev="0.6.2.2" />
<dependency org="org.jacoco.report" name="jacoco-report" rev="0.6.2.2" />
You'll only need a single dependency declaration:
<dependency org="org.jacoco" name="org.jacoco.ant" rev="0.7.2.201409121644" />
The "core", "agent" and "report" jars get pulled in as dependencies.
Jacoco ANT task jar in Maven Central
The following is an build example that uses ivy to manage Jacoco and Sonar plugins. It's worth noting how it uses a "build" configuration mapping to create a classpath for just the ANT task jars.
integrating JaCoCo in sonar using Ant
I use following dependencies:
<dependency org="org.jacoco" name="org.jacoco.ant" rev="0.6.3.201306030806"/>
<dependency org="org.jacoco" name="org.jacoco.core" rev="0.6.3.201306030806"/>
<dependency org="org.jacoco" name="org.jacoco.agent" rev="0.6.3.201306030806"/>
<dependency org="org.jacoco" name="org.jacoco.report" rev="0.6.3.201306030806"/>
I tried to configure Ivy to download source code for dependencies. It works, but only for the top-level dependency. Maven will get sources for all levels of dependencies. How can I do this using Ivy?
Here is my ivy.xml file:
<ivy-module version="2.0">
<info organisation="com.ubs" module="IMQuant"/>
<dependencies defaultconfmapping="*->default,sources">
<dependency org="junit" name="junit" rev="4.11"/>
<dependency org="org.apache.maven.surefire" name="common-java5" rev="2.17"/>
<dependency org="com.googlecode.kevinarpe-papaya" name="kevinarpe-papaya" rev="0.0.15"/>
</dependencies>
</ivy-module>
Here is my retreive statement:
<ivy:retrieve pattern="${libs.dir}/ivy/all/[artifact]-[revision]-[type].[ext]" />
Above, dependency kevinarpe-papaya requires Google Guava. I see the JAR downloaded (actually a bundle), but I don't see the sources. I checked my Ivy cache. The sources are also missing for Google Guava.
Also, dependency junit requires Hamcrest (core). Again, I see the JAR downloaded, but I don't see the sources. (Also, I checked my Ivy cache.)
You could try to specify the type:
type="sources"
E.g.:
<dependency org="junit" name="junit" type="sources" rev="4.11"/>
If you want to retrieve different types, mention the same dependendy multiple times:
<dependency org="junit" name="junit" type="sources" rev="4.11"/>
<dependency org="junit" name="junit" type="jar" rev="4.11"/>
I handle the build of a Java product with dependencies with ant.
Here is my project.properties file:
project.name=foo
project.version=1.0.0
thirdpart.commons-cli.version=1.2
thirdpart.guava.version=16.0.1
This is loaded using the following in my build.xml ant script:
<property file="project.properties"/>
I would like to loop over all properties starting by "thirdpart." and retrieve each time the name between "thirdpart." & ".version" and the value of the property.
Idea behind is then to retrieve the correct jar file from a shared server. Those informations will help me to build up the correct URL to retrieve them, while allowing me to change my dependencies version easily.
How to proceed with ant ? (Thanks for your help).
Instead of building your own dependency manager I would suggest using Apache ivy.
Several advantages. Instead of building and populating a shared server, you could download from Maven Central. Standard Maven repository managers (nexus, artifactory, archiva) can be used to host repositories inside your firewall.
Examples:
Class not found with Ant, Ivy and JUnit - error in build.xml?
How to avoid copying dependencies with Ivy
Your dependencies
Here's an example ivy.xml file to retrieve your dependencies
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Required to compile application"/>
</configurations>
<dependencies>
<!-- compile dependencies -->
<dependency org="commons-cli" name="commons-cli" rev="1.2" conf="compile->default"/>
<dependency org="com.google.guava" name="guava" rev="17.0-rc2" conf="compile->default"/>
</dependencies>
</ivy-module>
Relatively new to Ivy here.
I have a growing list of projects that all publish to the same repository under the same organization. And I'm trying to find out how, if at all possible, I can declare dependencies from another project to artifacts created by all those projects.
My ivisettings.xml already has a resolver that points to that repository location. All artifacts in that repo are from the same organization. I trying to determine if there's a way to easily include artifacts in that location as dependencies from another project. I was hoping that I could do this using a wildcard for its names, like:
<dependencies>
<dependency org="my.org.name" name="**" rev="${current.iteration}" conf="master" />
</dependencies>
But of course the above doesn't work. So, any pointer/comment/pointer suggestion would be welcomed. I can't be the first guy to want to do this, so there's got to be a way to do this and maybe I'm just not using the right keywords in my searches. Thanks.
Publish an extra ivy file that lists all the other projects as a list of dependencies.
<ivy-module version="2.0">
<info organisation="my.org.name" module="projects" revision="1.0.0" status="release" publication="20130215110241"/>
<dependencies>
<dependency org="my.org.name" name="projectA" rev="1.0.0"/>
<dependency org="my.org.name" name="projectB" rev="1.0.0"/>
<dependency org="my.org.name" name="projectC" rev="1.0.0"/>
..
..
</dependencies>
</ivy-module>
Creating a single dependency against this special ivy module will then pull in the other project artifacts as transitive dependencies.
<dependency org="my.org.name" name="projects" rev="1.0.0"/>
Enhancement
You could go one further and create different configurations to categorize your projects' artifacts:
<ivy-module version="2.0">
<info organisation="my.org.name" module="projects" revision="1.0.0" status="release" publication="20130215110241"/>
<configurations>
<conf name="web-apps" description="Projects which provide functionality for webapps"/>
<conf name="standalone" description="Projects which provide functionality for stand-alone java apps"/>
</configurations>
<dependencies>
<dependency org="my.org.name" name="projectA" rev="1.0.0" conf="webapps->default"/>
<dependency org="my.org.name" name="projectB" rev="1.0.0" conf="standalone->default"/>
<dependency org="my.org.name" name="projectC" rev="1.0.0" conf="webapps->default;standalone->default"/>
</dependencies>
</ivy-module>
Configurations are an extremely useful ivy feature. Client builds can use mappings to pull in artifacts associated with either webapp or standalone development. This feature is similar to "scopes" in Maven but much more powerful.
Update: Dynamic revisions
When publishing the "projects" module, you might want to use dynamic revisions to simplify maintenance.
<ivy-module version="2.0">
<info organisation="my.org.name" module="projects"/>
<dependencies>
<dependency org="my.org.name" name="projectA" rev="latest.release"/>
<dependency org="my.org.name" name="projectB" rev="latest.release"/>
<dependency org="my.org.name" name="projectC" rev="latest.release"/>
..
..
</dependencies>
</ivy-module>
The publish task will generate and push an ivy file resolved with the latest version of each project.
You will still need to list each project, but this is in fact a good thing. It means pulling in older versions of the "projects" module will result in same number of revisions at a point in time.
Generate ivy file
Finally to completely automate this process, using a variable number projects, perhaps you could generate the "projects" ivy file?
The following is a groovy snippet that you could use:
<groovy>
import groovy.xml.MarkupBuilder
new File("build/ivy.xml").withWriter { writer ->
def xml = new MarkupBuilder(writer)
xml."ivy-module"(version:"2.0") {
info(organisation:"my.org.name", module:"projects")
dependencies() {
new File("/path/to/projects/directory").listFiles().each { dir ->
dependency(org:"my.org.name", name:dir.name, rev:"latest.release")
}
}
}
}
</groovy>