Having resolved my ivy.xml file, I'd like to create a new resolved-ivy.xml file consisting of all of the transitive dependencies found in the resolve. Is it possible to do this?
This is different to a deliver, which (I believe) only writes out the immediate dependencies from your ivy.xml, not the transitive dependencies. The deliver Ant task does have a delivertarget attribute, which looks in the documentation like it should do this. In practice it works only for modules in the same organisation (so not generally for all dependencies) and generates a file per module.
It's also different from the ivy-report XML file that is generated during the resolve, but not hugely different. If what I'm trying is not possible then I'll just hack at this file directly, I suppose.
The context here is trying to enable repeatable reproducible builds, including in the presence of changes (new libraries, versions) to the repository. There are posts around the interwebs that try to do this, and none I've found can do it properly.
Additions to the Ivy repository can change resolve results, in particular if any dependencies anywhere in the repository (not just your project) have range dependencies. Example: A depends on B;[2.0,4.0] and B;3.1 is later added to the repository.
Idea is to resolve as normal, write the resolve as a flattened Ivy file, save that in your project's VCS for that tag (or whatever) and subsequently resolve against that file with transitive="false". Assuming that existing items in the repository do not change, this allows repeatable builds.
If anyone has any better ideas for doing this, I'm all ears. At the moment I'm expecting to have to hack some combination of the ResolveEngine to make the ResolveReport available, then add a custom DeliverEngine to use it.
artifactreport> might help.
Use the deliver task to create an ivy.xml with the dynamic version constraints replaced by the static version constraint (i.e [2.0,3.0[ becomes 2.2.1):
<ivy:deliver conf="*(public)" deliverpattern="${dist.dir}/ivy.xml"/>
Then use the resolve task on that file to prepare for artifactreport.
<ivy:resolve file="${dist.dir}/ivy.xml"
conf="*(public)"
refresh="true"
type="ivy" />
Finally, artifactreport will do the transient dependency resolution.
<ivy:artifactreport tofile="${dist.dir}/artifactreport.xml" />
artifactreport.xml will look like
<modules>
<module organisation="com.googlecode.flyway" name="flyway" rev="1.7" status="release"/>
<module organisation="org.postgresql" name="postgresql-jdbc" rev="9.0" status="release"/>
<module organisation="org.hibernate" name="hibernate" rev="3.3.2" status="release"/>
<module organisation="org.apache.commons" name="commons-daemon" rev="1.0.2" status="release"/>
...
Use XSLT to produce the ivy.xml form.
The feature you are looking for was added in Ivy 2.4: fixdeps. It reads an ivy.xml file, which in this case serves as an specification, and output an equivalent file, e.g. ivy-resolved.xml, with all transitive dependencies resolved.
Under normal circumstances I would say this is over-kill. The problem you're trying to work-around is that your repository is unreliable..... Perhaps you should consider using a repository manager to manage your repository?
Nexus
Artifactory
Archiva
(Artifacts published to Maven Central are deliberately never changed, this ensures that people using the repository do not encounter build instability).
Having said all this, if you can't completely trust the repository module configuration then what you're attempting to do is possible using the ivy artifactreport task. It generates an XML report, which can be transformed, using XSLT into an new ivy file.
Example
$ tree
.
|-- build.xml
|-- ivy.xml
`-- src
`-- main
`-- xsl
`-- artifactreport.xsl
build.xml
<project name="demo" default="build" xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="init">
<ivy:resolve/>
</target>
<target name="build" depends="init">
<ivy:artifactreport tofile="build/reports/artifacts.xml"/>
<xslt style="src/main/xsl/artifactreport.xsl" in="build/reports/artifacts.xml" out="build/ivy.xml"/>
</target>
</project>
artifactreport.xsl
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:template match="/">
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<dependencies>
<xsl:apply-templates select="modules/module"/>
</dependencies>
</ivy-module>
</xsl:template>
<xsl:template match="module">
<dependency org="{#organisation}" name="{#name}" rev="{#rev}"/>
</xsl:template>
</xsl:stylesheet>
Related
I am trying to use the ivy:publish task to publish my jars using the public ibiblio resolver which I understand corresponds to the Maven central repository.
It seems to sort of work, but I am eventually getting an HTTP 405: Method Not Allowed.
The relevant sections from my ivysettings.xml, ivy.xml and build.xml files are:
ivysettings.xml
This is actually the entire file:
<ivysettings>
<settings defaultResolver="public"/>
<resolvers>
<ibiblio name="public" m2compatible="true" />
</resolvers>
</ivysettings>
ivy.xml
...
<publications>
<artifact type="pom" ext="pom" conf="default"/>
<artifact type="jar" ext="jar" conf="default"/>
</publications>
...
build.xml
...
<target name="gen-pom" depends="ivy-resolve">
<ivy:makepom ivyfile="ivy.xml" pomfile="${dist.dir}/${ant.project.name}.pom">
<mapping conf="default" scope="compile"/>
</ivy:makepom>
</target>
<target name="ivy-publish" depends="jar, gen-pom" description="publish jar/source to maven repo">
<ivy:publish resolver="public" forcedeliver="true" overwrite="true" publishivy="false">
<artifacts pattern="${dist.dir}/[artifact].[ext]" />
</ivy:publish>
</target>
...
When I invoke ant ivy-publish I get:
impossible to publish artifacts for mant-tasks#mant-tasks;1.0: java.io.IOException: PUT operation to URL https://repo1.maven.org/maven2/mant-tasks/mant-tasks/1.0/mant-tasks-1.0.pom failed with status code 405: Method Not Allowed
I guess this makes sense otherwise anybody would be able to pollute the namespace. So my questions are:
is this due to my lack of credentials? I.e. would I need to add a
credentials element in my ivysettings.xml as shown in this
answer?
can I obtain credentials for ibiblio for my
insignificant module and if not what other freely available public
repositories are available to a Java developer who doesn't want to
build and maintain their own maven repository? I need to be able to pull-in my module (as an IVY dependency of other projects) from a globally available repository.
Maven central is operated by Sonatype, the creators of Maven:
http://central.sonatype.org/
JFrog operate a competing service called bintray:
https://bintray.com/
I suggest reading the documentation and obtain a credential that will allow to publish your module's file(s)
I handle the build of a Java product with dependencies with ant.
Here is my project.properties file:
project.name=foo
project.version=1.0.0
thirdpart.commons-cli.version=1.2
thirdpart.guava.version=16.0.1
This is loaded using the following in my build.xml ant script:
<property file="project.properties"/>
I would like to loop over all properties starting by "thirdpart." and retrieve each time the name between "thirdpart." & ".version" and the value of the property.
Idea behind is then to retrieve the correct jar file from a shared server. Those informations will help me to build up the correct URL to retrieve them, while allowing me to change my dependencies version easily.
How to proceed with ant ? (Thanks for your help).
Instead of building your own dependency manager I would suggest using Apache ivy.
Several advantages. Instead of building and populating a shared server, you could download from Maven Central. Standard Maven repository managers (nexus, artifactory, archiva) can be used to host repositories inside your firewall.
Examples:
Class not found with Ant, Ivy and JUnit - error in build.xml?
How to avoid copying dependencies with Ivy
Your dependencies
Here's an example ivy.xml file to retrieve your dependencies
<ivy-module version="2.0">
<info organisation="com.myspotontheweb" module="demo"/>
<configurations>
<conf name="compile" description="Required to compile application"/>
</configurations>
<dependencies>
<!-- compile dependencies -->
<dependency org="commons-cli" name="commons-cli" rev="1.2" conf="compile->default"/>
<dependency org="com.google.guava" name="guava" rev="17.0-rc2" conf="compile->default"/>
</dependencies>
</ivy-module>
I currently have ANT_HOME located at /home/<myuser>/ant/1.8.4/ant-1.8.4.
I just downloaded the Apache Ivy tarball that includes its dependencies. I extracted it to /home/<myuser>/ivy/2.3.0-rc1/ivy-2.3.0-rc1.
I then copied /home/<myuser>/ivy/2.3.0-rc1/ivy-2.3.0-rc1/lib/*.jar to ANT_HOME/lib. If my understanding of how Ant works with plugins/extensions is correct, then Ant should now be able to access all of Ivy's tasks at runtime.
My next question is, how do I define Ivy tasks inside my Ant buildfile? Say I want to use ivy-retrieve, ivy-resolve and ivy-publish tasks. What are all the configurations I need to do (in the XML) to get these tasks working when I run my Ant build from the command-line (I will not be building through the Ant-Eclipse plugin). Thanks in advance!
First, you have to define a <taskdef> to point to the Ivy tasks.
<property environment="env"/>
<property name="ivy.home" value="${env_IVY_HOME}"/>
<taskdef resource="org/apache/ivy/ant/antlib.xml">
<classpath>
<fileset dir="${ivy.home}">
<include name="*.jar"/>
</fileset>
</classpath>
</taskdef>
That will give you access to the Ivy tasks. You'd use these tasks like this:
<cachepath pathid="main.classpath" conf="compile"/>
The problem is that your Ivy tasks names might clash with other Ant tasks. For example, there's an Ivy task <report>. To solve this, you can create an Ivy namespace. To do that, you put a reference in your namespace in the <project> entity like this:
<project name="my.proj" default="package" basedir="."
xmlns:ivy="antlib:org.apache.ivy.ant"/>
Now, when you define the Ivy tasks, you can use that antlib:org.apache.ivy.ant reference to your ivy namespace. Same taskdef as before, but with a uri field:
<property environment="env"/>
<property name="ivy.home" value="${env_IVY_HOME}"/>
<taskdef resource="org/apache/ivy/ant/antlib.xml"
uri="antlib:org.apache.ivy.ant">
<classpath>
<fileset dir="${ivy.home}">
<include name="*.jar"/>
</fileset>
</classpath>
</taskdef>
By the way, there's nothing special about that uri. I could have done this:
<project name="my.proj" default="package" basename="."
xmlns:ivy="pastrami:with.mustard">
[...]
<taskdef resource="org/apache/ivy/ant/antlib.xml"
uri="pastrami:with.mustard">
<classpath>
<fileset dir="${ivy.home}">
<include name="*.jar"/>
</fileset>
</classpath>
</taskdef>
The point is now you can prefix your task names with ivy:. Instead of this:
<cachepath pathid="main.classpath" conf="compile"/>
You can now do this:
<ivy:cachepath pathid="main.classpath" conf="compile"/>
And that's how you gain access to your Ivy Ant tasks.
Now, you have access to your Ivy Ant tasks, you need to define an ivysettings.xml file and use the <ivy:settings/> task to point there:
<ivy:settings file="${ivy.home}/ivysettings.xml"/>
There is a default ivysettings.xml file embedded in Ivy that will point you to the world wide Maven repository system. If you don't have a company wide Maven repository, then you can use the default ivysettings.xml file:
<ivy:settings/>
That's pretty simple.
Once you've done that, you need to read in and resolve your ivy.xml file which usually sits in the root of your project in the same directory as your build.xml file.
Basically, your ivy.xml file contains references to the third party jars you want to bring into your project. For example:
<dependencies>
<dependency org="log4j" name="log4j" rev="1.2.17" conf="compile->default"/>
<dependency org="junit" name="junit" rev="4.10" conf="test->default"/>
</dependencies>
What this is saying is that I need the log4j.jar (revision 1.2.17) for compilation (and for compiling tests too) and I need junit.jar (revision.4.10) for compilation of my test code.
The compile->default is a mapping of my compile configuration to Maven's default configuration (which says I just want the Jar and any other jars that it might depend upon.
Where's does my compile configuration come from? I define it in my ivy.xml. There are ten standard configurations. This also goes into your ivy.xml file:
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
You can use any configuration name you want, but these map to the default Maven configurations and are widely used.
Once you have your ivy.xml file defined, you can use <ivy.resolve> to resolve your dependencies:
<ivy:resolve/>
So, we have the following:
How to use <taskdef> in your build.xml to incorporate the Ivy Ant tasks into your build.
How to use the Ivy Ant task <ivy:settings> to configure Ivy.
How to use <ivy:resolve/> to read in your ivy.xml file and resolve your third party jar dependencies.
Now, you probably want to actually use those jar files. There are three ways to do this:
<ivy:cachepath pathid="main.classpath" conf="compile"/>
The <ivy:cachepath/> task will create a classpath (in this case called main.classpath) that points to the jars you have in your ivy.xml file's compile configuration. This is used most of the time.
If you need a fileset, you can use this:
<ivy:cachefileset setid="compile.fileset" conf="compile"/>
In this case, it will create a fileset with a refid of compile.fileset.
Sometimes you have to bring the jars into your project. For example, if you create a war or ear file, you want to enclose your jars. In that case, you can use this:
<property name="lib.dir" value="${target.dir}/lib"/>
<ivy:retrieve pattern="${lib.dir}/[artifact].[ext]"
conf="runtime"/>
That will fetch your jars into the ${lib.dir} directory, so you can include them in wars or ears.
Sorry for the long answer, but there are a lot of steps to cover. I highly recommend Manning's book Ant in Action which has a whole chapter on Ivy.
David gave a very fine answer, but I'd like to point out that the taskdef is not required.
Provided the ivy.jar is in the expected location the namespace declaration at the top of the ANT file is enough:
<project ..... xmlns:ivy="antlib:org.apache.ivy.ant">
For more detail I'd recommend reading about how ANT libs work.
The following answer provides some more "setting up ivy" advice:
Ivy fails to resolve a dependency, unable to find cause
How to add a local path (not URL) to ivysettings.xml? I need to add my Maven local repository (/Users/me/.m2/repository to it.
Thanks
Try the following ivysettings.xml file:
<ivysettings>
<settings defaultResolver="default"/>
<property name="m2-pattern" value="${user.home}/.m2/repository/[organisation]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]" override="false" />
<resolvers>
<chain name="default">
<filesystem name="local-maven2" m2compatible="true" >
<artifact pattern="${m2-pattern}"/>
<ivy pattern="${m2-pattern}"/>
</filesystem>
<ibiblio name="central" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
It includes Maven central in case the dependency is missing from the local Maven repo.
Note:
The benefits of re-using a local Maven repository are limited. Ivy caches jars retrieved from repostories.
Ivy dependencies are resolved with "Resolvers".
This page is pretty good for understanding the basics of how they work.
http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html
Specifically : How can I "resolve" a local maven repository ?
Ivy has a "FileSystemResolver" which, rather than taking in a web address, can simply resolve from a local, root path. Note that there are some gotchas when things get complicated, like this one : http://ant.apache.org/ivy/history/latest-milestone/settings/resolvers.html . Resolvers are similar to maven Repository tags, in that they define a resource.
A quick word of advice
Remember that once you customize ivysettings.xml if you are using an IDE, you will have to tell it to specifically use YOUR ivysettings.xml file, rather than some internal default.
I found out that in the more recent versions of sbt you can do
sbt publish-m2
Prior to 0.13.7, SBT ignored the Maven's settings.xml to use the <localRepository> setting. See https://github.com/sbt/sbt/issues/1589. In some cases, this will obviate the need to change ivysettings.xml when using publish-m2.
Here is my ivy.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0">
...
<dependencies>
<dependency org="spring" name="richclient" rev="1.1.0"/>
</dependencies>
</ivy-module>
And ivy-settings.xml:
<property name="ivy.local.default.root" value="/home/---/dev/Java/_libraries/_ivy" override="false"/>
<property name="ivy.local.default.ivy.pattern" value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]" override="false"/>
<property name="ivy.local.default.artifact.pattern" value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]" override="false"/>
<resolvers>
<filesystem name="local">
<ivy pattern="${ivy.local.default.root}/${ivy.local.default.ivy.pattern}" />
<artifact pattern="${ivy.local.default.root}/${ivy.local.default.artifact.pattern}" />
</filesystem>
</resolvers>
Ivy try to find
/home/---/dev/Java/_libraries/_ivy/spring/richclient/1.1.0/jars/richclient.jar
And here is the problem. Library has 4 jar files.
How to include all jars in project from one dependency in ivy.xml?
Thx
I'm assuming you've just downloaded the jars locally? It won't work unless you also write an ivy.xml file for the downloaded files, listing the artifacts that are associated with the module (See publications section of the ivy.xml doco)
Why not avoid the hassle of maintaining the your own version of someone else's module by using the maven repository provided by Spring?
Add the following to your ivy-settings.xml file:
<resolvers>
<ibiblio name="spring-rcp" m2compatible="true" root="http://spring-rich-c.sourceforge.net/maven2repository"/>
</resolvers>
While Ivy can work using dependencies on individual JAR files, it works better if you define separate ivy.xml files for the dependencies themselves, which specifies the 4 separate JAR files. This ivy.xml defines what Ivy calls a module.
Your application's ivy.xml then expresses a dependency on that module, rather than on specific JAR files.
The Ivy website has a tutorial on modules, I highly recommend reading it
http://ant.apache.org/ivy/history/latest-milestone/tutorial/conf.html