Develop Reference

Building Nutch Plugin: class dependency

I wrote some Nutch plugin with using different extension points such as Protocol, Parser and etc. These plugins work perfectly inside eclipse. But in order to use them on hadoop cluster it should be built by using ANT. My problem is, I wrote some classes in some new packages inside core folder(src). These classes are shared cross different developed plugins. My problem is at build time of developed plugins, ANT can not find the mentioned shared classes, so I am unable to complete the build process successfully. For better understanding of my problem Here is build.xml of one of my plugins:
<project name="filter-news" default="jar-core">
<import file="../build-plugin.xml"/>
<!-- Build compilation dependencies -->
<target name="deps-jar">
<ant target="jar" inheritall="false" dir="../lib-xml"/>
</target>
<!-- Add compilation dependencies to classpath -->
<path id="plugin.deps">
<fileset dir="${nutch.root}/build">
<include name="**/lib-xml/*.jar" />
</fileset>
</path>
<!-- Deploy Unit test dependencies -->
<!-- Deploy Unit test dependencies -->
<!-- for junit test -->
</project>
ivy.xml:
<ivy-module version="1.0">
<info organisation="org.apache.nutch" module="${ant.project.name}">
<license name="Apache 2.0"/>
<ivyauthor name="Apache Nutch Team" url="http://nutch.apache.org"/>
<description>
Apache Nutch
</description>
</info>
<configurations>
<include file="../../..//ivy/ivy-configurations.xml"/>
</configurations>
<publications>
<!--get the artifact from our module name-->
<artifact conf="master"/>
</publications>
<dependencies>
<dependency org="mysql" name="mysql-connector-java" rev="5.1.31"/>
<dependency org="net.sourceforge.htmlcleaner" name="htmlcleaner" rev="2.2"/>
<dependency org="commons-jxpath" name="commons-jxpath" rev="1.3"/>
</dependencies>
</ivy-module>
plugin.xml:
<plugin id="filter-news" name="Apache Nutch XML/HTML Parser/Indexing Filter" version="1.4" provider-name="nutch.org">
<runtime>
<library name="filter-news.jar">
<export name="*"/>
</library>
<library name="ant-1.7.0.jar"/>
<library name="ant-launcher-1.7.0.jar"/>
<library name="jdom-1.1.jar"/>
<library name="commons-jxpath-1.3.jar"/>
<library name="htmlcleaner-2.2.jar"/>
<library name="mysql-connector-java-5.1.31.jar"/>
</runtime>
<requires>
<import plugin="nutch-extensionpoints"/>
</requires>
<extension id="org.apache.nutch.parse" name="Nutch XML/HTML Html parser filter" point="org.apache.nutch.parse.HtmlParseFilter">
<implementation id="com.ictcert.nutch.filter.news.NewsHtmlFilter" class="com.ictcert.nutch.filter.news.NewsHtmlFilter" />
</extension>
<extension id="org.apache.nutch.indexer" name="Nutch XML/HTML Indexing Filter" point="org.apache.nutch.indexer.IndexingFilter">
<implementation id="com.ictcert.nutch.filter.news.NewsIndexingFilter" class="com.ictcert.nutch.filter.news.NewsIndexingFilter"/>
</extension>
</plugin>
When I try to build this plugin ant can not find all of the class dependencies related to com.ictcert.nutch package which is located in core part of nutch (src). While for other classes located in org.apache.nutch I have not such problem. Would you please tell me what is wrong with my configuration that the default packages could be found by ANT but the new ones could not.

As far as I know from my experience the implementation id should have the same package structure as your extension point. Can you try that and see if it solves your problem.

Publishing a jar file that already exists

I have some problems understanding how the publication works. I have to publish a jar file to my web repository, but I have found some probably maybe by the fact that I missing something about the artifact and the publication.
These are my three files for the publication:
Build.xml
<project xmlns:ivy="antlib:org.apache.ivy.ant" name="pubblication"
default="pubblication" basedir=".">
<echo>inizio</echo>
<target name="pubblication" description="--> pubblicare un artifact">
<ivy:settings file="archivaIvySetting.xml" />
<ivy:publish resolver="publish-artifact" conf="publicConf" organisation="bbi"
module="resutil" revision="1.0">
<artifacts pattern="./[artifact]-[revision].[type]"/>
</ivy:publish>
</target>
</project>
Ivy.xml
<ivy-module version="2.0">
<info organisation="org.apache" module="central"/>
<configurations>
<conf name="publicConf" visibility="public" />
</configurations>
<publications>
<artifact name="[organisation]-resutil" ext="jar" conf="publicConf"/>
</publications>
</ivy-module>
archivaIvySetting.xml
<?xml version="1.0" encoding="UTF-8"?>
<ivysettings>
<property name="archiva-internal" value="http://host.com:8080/repository
/internal/"/>
<settings defaultResolver="central">
<credentials host="host.com" realm="Repository Archiva Managed internal
Repository" username="username" passwd="passwd" />
</settings>
<resolvers >
<ibiblio name="central" m2compatible="true" usepoms="true" root="${archiva-
internal}" />
</resolvers>
</ivysettings>
My problem is that when I do the build that ant says there is no module with that name in the cache. Now the question:
1) In the pattern do I set the jar that I want to publish?
2) If not how do I must to do practically that: take the jar give it the info params and publish it in the repo?
I repeat the file already exist, and this is a test file.

The pattern in the publish task should match a that is created locally in your build. Additionally the publish section of the ivy file must match the files your attempting to upload.
Hopefully some examples will help:
good ivy tutorial for local repository?
Issues using ivy:publish task
Convert ivy.xml to pom.xml
how to publish 3rdparty artifacts with ivy and nexus

Trying to convert ivy artifacts into maven - missing artifact exception for the pom, even though it is there

Not sure if I'm going about this the right way, but I have some artifacts that I'm trying to convert to maven using ivy ant tasks and push into my maven repo.
the component in question is mystuff.services.common.
First I make the pom...
<ivy:makepom ivyfile="${ivy.lib.dir}/ivy/cache/myorg/mystuff.services.common/ivy-mystuff.services.common.xml" pomfile="${ivy.lib.dir}/ivy/cache/myorg/mystuff.services.common/poms/mystuff.services.common.pom">
<mapping conf="default" scope="compile"/>
<mapping conf="runtime" scope="runtime"/>
</ivy:makepom>
Then a little hackery - I insert an artifact element in the ivy file using xml task. This works ok...
<xmltask source="${ivy.lib.dir}/ivy/cache/myorg/mystuff.services.common/ivy-${resolved.revision}.xml" dest="${ivy.lib.dir}/ivy/cache/myorg/mystuff.services.common/ivy-${resolved.revision}.xml">
<insert path="/ivy-module/publications" >
<![CDATA[
<artifact name="mystuff.services.common" type="pom"/>
]]>
</insert>
</xmltask>
Then I resolve/deliver/publish, as per various docs I've seen on how to do this.
<ivy:resolve file="${ivy.lib.dir}/ivy/cache/myorg/mystuff.services.common/ivy-${resolved.revision}.xml"/>
<!--<echoproperties/>-->
<ivy:deliver conf="*" delivertarget="recursive-deliver"/>
<ivy:publish resolver="myrepo-publish" publishivy="false" overwrite="true">
<artifacts pattern="lib/myorg/[module]/[type]s/[artifact].[ext]"/>
</ivy:publish>
And the error I get:
build.xml:235: impossible to publish artifacts for
myorg#mystuff.services.common;1.0.1: java.io.IOException: missing artifact
myorg#mystuff.services.common;1.0.1!mystuff.services.common.pom
If I leave out the pom from the artifacts in the ivy file, the other artifacts just publish fine.
What am I doing wrong?
This is what the ivy file looks like after inserting the pom entry for artifacts
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="../../ivy-doc.xsl"?>
<ivy-module version="1.0">
<info organisation="myorg" module="mystuff.services.common" revision="1.0.1" status="integration" publication="20130206204156"/>
<configurations>
<conf name="default"/>
<conf name="compile" extends="default"/>
</configurations>
<publications>
<artifact name="services.common" type="jar" conf="compile"/>
<artifact name="services.common~test" type="jar" conf="compile"/>
<artifact name="services.common" type="javadoc-zip" ext="zip" conf="compile"/>
<artifact name="services.common~test" type="javadoc-zip" ext="zip" conf="compile"/>
<artifact name="services.common" type="src-zip" ext="zip" conf="compile"/>
<artifact name="services.common~test" type="src-zip" ext="zip" conf="compile"/>
<artifact name="com.myorg.mystuffservices.common" type="osgi-module" ext="jar" conf="compile"/>
<artifact name="services.common" type="pom"/>
</publications>
<dependencies>
<dependency org="org.testng" name="testng" rev="5.11" conf="compile->compile-15"/>
</dependencies>
</ivy-module>

Your publish does not have an artifact pattern that finds the pom generated by your "makepom" task.
Either change the location or alternatively add an extra artifacts tag to your publish task:
<ivy:publish resolver="myrepo-publish" publishivy="false" overwrite="true">
<artifacts pattern="lib/myorg/[module]/[type]s/[artifact].[ext]"/>
<artifacts pattern="${ivy.lib.dir}/ivy/cache/myorg/mystuff.services.common/poms/mystuff.services.common.pom"/>
</ivy:publish>
I also don't understand why you're inserting a POM entry into you ivy file. Why don't you just list in your publications section?
For a detailed example see:
how to publish 3rdparty artifacts with ivy and nexus

How to configure Ivy for Ant build

I currently have ANT_HOME located at /home/<myuser>/ant/1.8.4/ant-1.8.4.
I just downloaded the Apache Ivy tarball that includes its dependencies. I extracted it to /home/<myuser>/ivy/2.3.0-rc1/ivy-2.3.0-rc1.
I then copied /home/<myuser>/ivy/2.3.0-rc1/ivy-2.3.0-rc1/lib/*.jar to ANT_HOME/lib. If my understanding of how Ant works with plugins/extensions is correct, then Ant should now be able to access all of Ivy's tasks at runtime.
My next question is, how do I define Ivy tasks inside my Ant buildfile? Say I want to use ivy-retrieve, ivy-resolve and ivy-publish tasks. What are all the configurations I need to do (in the XML) to get these tasks working when I run my Ant build from the command-line (I will not be building through the Ant-Eclipse plugin). Thanks in advance!

First, you have to define a <taskdef> to point to the Ivy tasks.
<property environment="env"/>
<property name="ivy.home" value="${env_IVY_HOME}"/>
<taskdef resource="org/apache/ivy/ant/antlib.xml">
<classpath>
<fileset dir="${ivy.home}">
<include name="*.jar"/>
</fileset>
</classpath>
</taskdef>
That will give you access to the Ivy tasks. You'd use these tasks like this:
<cachepath pathid="main.classpath" conf="compile"/>
The problem is that your Ivy tasks names might clash with other Ant tasks. For example, there's an Ivy task <report>. To solve this, you can create an Ivy namespace. To do that, you put a reference in your namespace in the <project> entity like this:
<project name="my.proj" default="package" basedir="."
xmlns:ivy="antlib:org.apache.ivy.ant"/>
Now, when you define the Ivy tasks, you can use that antlib:org.apache.ivy.ant reference to your ivy namespace. Same taskdef as before, but with a uri field:
<property environment="env"/>
<property name="ivy.home" value="${env_IVY_HOME}"/>
<taskdef resource="org/apache/ivy/ant/antlib.xml"
uri="antlib:org.apache.ivy.ant">
<classpath>
<fileset dir="${ivy.home}">
<include name="*.jar"/>
</fileset>
</classpath>
</taskdef>
By the way, there's nothing special about that uri. I could have done this:
<project name="my.proj" default="package" basename="."
xmlns:ivy="pastrami:with.mustard">
[...]
<taskdef resource="org/apache/ivy/ant/antlib.xml"
uri="pastrami:with.mustard">
<classpath>
<fileset dir="${ivy.home}">
<include name="*.jar"/>
</fileset>
</classpath>
</taskdef>
The point is now you can prefix your task names with ivy:. Instead of this:
<cachepath pathid="main.classpath" conf="compile"/>
You can now do this:
<ivy:cachepath pathid="main.classpath" conf="compile"/>
And that's how you gain access to your Ivy Ant tasks.
Now, you have access to your Ivy Ant tasks, you need to define an ivysettings.xml file and use the <ivy:settings/> task to point there:
<ivy:settings file="${ivy.home}/ivysettings.xml"/>
There is a default ivysettings.xml file embedded in Ivy that will point you to the world wide Maven repository system. If you don't have a company wide Maven repository, then you can use the default ivysettings.xml file:
<ivy:settings/>
That's pretty simple.
Once you've done that, you need to read in and resolve your ivy.xml file which usually sits in the root of your project in the same directory as your build.xml file.
Basically, your ivy.xml file contains references to the third party jars you want to bring into your project. For example:
<dependencies>
<dependency org="log4j" name="log4j" rev="1.2.17" conf="compile->default"/>
<dependency org="junit" name="junit" rev="4.10" conf="test->default"/>
</dependencies>
What this is saying is that I need the log4j.jar (revision 1.2.17) for compilation (and for compiling tests too) and I need junit.jar (revision.4.10) for compilation of my test code.
The compile->default is a mapping of my compile configuration to Maven's default configuration (which says I just want the Jar and any other jars that it might depend upon.
Where's does my compile configuration come from? I define it in my ivy.xml. There are ten standard configurations. This also goes into your ivy.xml file:
<configurations>
<conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
<conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
<conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
<conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
<conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
<conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
<conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
<conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
<conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
<conf name="optional" visibility="public" description="contains all optional dependencies"/>
</configurations>
You can use any configuration name you want, but these map to the default Maven configurations and are widely used.
Once you have your ivy.xml file defined, you can use <ivy.resolve> to resolve your dependencies:
<ivy:resolve/>
So, we have the following:
How to use <taskdef> in your build.xml to incorporate the Ivy Ant tasks into your build.
How to use the Ivy Ant task <ivy:settings> to configure Ivy.
How to use <ivy:resolve/> to read in your ivy.xml file and resolve your third party jar dependencies.
Now, you probably want to actually use those jar files. There are three ways to do this:
<ivy:cachepath pathid="main.classpath" conf="compile"/>
The <ivy:cachepath/> task will create a classpath (in this case called main.classpath) that points to the jars you have in your ivy.xml file's compile configuration. This is used most of the time.
If you need a fileset, you can use this:
<ivy:cachefileset setid="compile.fileset" conf="compile"/>
In this case, it will create a fileset with a refid of compile.fileset.
Sometimes you have to bring the jars into your project. For example, if you create a war or ear file, you want to enclose your jars. In that case, you can use this:
<property name="lib.dir" value="${target.dir}/lib"/>
<ivy:retrieve pattern="${lib.dir}/[artifact].[ext]"
conf="runtime"/>
That will fetch your jars into the ${lib.dir} directory, so you can include them in wars or ears.
Sorry for the long answer, but there are a lot of steps to cover. I highly recommend Manning's book Ant in Action which has a whole chapter on Ivy.

David gave a very fine answer, but I'd like to point out that the taskdef is not required.
Provided the ivy.jar is in the expected location the namespace declaration at the top of the ANT file is enough:
<project ..... xmlns:ivy="antlib:org.apache.ivy.ant">
For more detail I'd recommend reading about how ANT libs work.
The following answer provides some more "setting up ivy" advice:
Ivy fails to resolve a dependency, unable to find cause

How to use Apache ivy to resolve dependency with multiple files?

Here is my ivy.xml:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0">
...
<dependencies>
<dependency org="spring" name="richclient" rev="1.1.0"/>
</dependencies>
</ivy-module>
And ivy-settings.xml:
<property name="ivy.local.default.root" value="/home/---/dev/Java/_libraries/_ivy" override="false"/>
<property name="ivy.local.default.ivy.pattern" value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]" override="false"/>
<property name="ivy.local.default.artifact.pattern" value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]" override="false"/>
<resolvers>
<filesystem name="local">
<ivy pattern="${ivy.local.default.root}/${ivy.local.default.ivy.pattern}" />
<artifact pattern="${ivy.local.default.root}/${ivy.local.default.artifact.pattern}" />
</filesystem>
</resolvers>
Ivy try to find
/home/---/dev/Java/_libraries/_ivy/spring/richclient/1.1.0/jars/richclient.jar
And here is the problem. Library has 4 jar files.
How to include all jars in project from one dependency in ivy.xml?
Thx

I'm assuming you've just downloaded the jars locally? It won't work unless you also write an ivy.xml file for the downloaded files, listing the artifacts that are associated with the module (See publications section of the ivy.xml doco)
Why not avoid the hassle of maintaining the your own version of someone else's module by using the maven repository provided by Spring?
Add the following to your ivy-settings.xml file:
<resolvers>
<ibiblio name="spring-rcp" m2compatible="true" root="http://spring-rich-c.sourceforge.net/maven2repository"/>
</resolvers>

While Ivy can work using dependencies on individual JAR files, it works better if you define separate ivy.xml files for the dependencies themselves, which specifies the 4 separate JAR files. This ivy.xml defines what Ivy calls a module.
Your application's ivy.xml then expresses a dependency on that module, rather than on specific JAR files.
The Ivy website has a tutorial on modules, I highly recommend reading it
http://ant.apache.org/ivy/history/latest-milestone/tutorial/conf.html