Develop Reference

Nutch Crawler Error: Hadoop Path Permissions

I'm trying to run Apache Nutch (v2.3) with MongoDB (v2.6). I am following this tutorial to help me get things set up. I have already created my seed list and my gora.properties and nutch-site.xml are set up fine. However, when running the bin/nutch inject ../urls/test/ command, I keep getting a java.io.IOException error :
$ bin/nutch inject ./../../urls/test/
InjectorJob: starting at 2015-05-04 13:53:29
InjectorJob: Injecting urlDir: ../../urls/test
InjectorJob: Using class org.apache.gora.mongodb.store.MongoStore as the Gora storage class.
InjectorJob: java.io.IOException: Failed to set permissions of path: \tmp\hadoop-TColletti\mapred\staging\TColletti1801159571\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:50)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
I've read in some places this could be caused by not having the correct version of MongoDb or the Gora is wrong. It seems as though something is wrong with the permissions to a temp directory for hadoop(which im not using right now). I've looked at this S/O article but cannot find the core-site.xml file anywhere in my 2.3 version of Nutch.
Can someone help me finally run this command?

I'm not sure if this is the official answer or not, but it worked for me. I found another S/O post here. The general problem was the same. Other places kept mentioning about modifying core-site.xml which I did not have. However in one of the answers it mentions a patch to download and couple lines you need to add to the nutch-site.xml file in my runtime/local/conf directory. Tried it out and it solved my problems. It seems like the patch just ignored the errors and went around them. There may be a better solution to this but for now, it worked.

ClassDefNotFoundError while class in classpath

I'm trying to run ParSeMiS. According to the documentation, it requires ant, prefuse and antlr jars to be available in its lib directory. I've put all the required jars in it. However, when I try to run it, I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: antlr/TokenStreamException
at de.parsemis.miner.environment.Settings.parseFileName(Settings.java:198)
at de.parsemis.miner.environment.Settings.parseOption(Settings.java:312)
at de.parsemis.miner.environment.Settings.parse(Settings.java:170)
at de.parsemis.miner.environment.Settings.parse(Settings.java:122)
at de.parsemis.Miner.run(Miner.java:358)
at de.parsemis.Miner.main(Miner.java:61)
Caused by: java.lang.ClassNotFoundException: antlr.TokenStreamException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 6 more
Now, I've verified that antlr/TokenStreamException.class is present in the antlr jar. I have tried adding the jar manually to the classpath by both exporting the CLASSPATH variable and setting it via the -cp switch. However, none of that works, and I still get this exception. Can anybody help me figure out what's wrong? Thanks.

The problem is that you're using -jar which ignores your CLASSPATH environment variable. You should list your dependencies in the manifest, as shown here, e.g.
Class-Path: lib/ant.jar lib/antlr-3.4-complete.jar lib/prefuse.jar
(It should have still worked with an explicit -cp option, however. My guess is that you got something wrong when specifying that, and assumed it was the same underlying cause as the failure when using the environment variable.)

I realize this question is very old, but I just had exactly the same problem and found this thread. For posterity, I'm posting how I did eventually get it to run:
As mentioned above, when running with -jar, java apparently ignores the class path. So don't run it with -jar. instead include the jars in the path and run the class directly. Poking around, the following should work (paths are on my Ubuntu 12.10 system):
java -cp /usr/share/java/antlr.jar:/full/path/to/parsemis.jar de.parsemis.Miner
You can then pass in options to the above. Maye sure you use full paths, and no shortcuts like ~/foo, as they apparently don't expand.
Of course, if you're using a Dot-formatted graph like I am, it dies very early on complaining of "unexpected char 0xA", but at least it gets further.

Unable to read TLD "META-INF/stripes.tld" from JAR file error

I'm encountering a bit of an unusual error after a major upgrade of our web application to ivy for build management which also included updating all of the major libraries involved as well. While we are managing to pass our unit tests, when we deploy the web application to Tomcat and try to navigate to it in the brower we get the following error:
org.apache.jasper.JasperException: /index.jsp(1,1)
/layout/taglibs.jsp(2,77) Unable to read TLD "META-INF/stripes.tld"
from JAR file
"file:/Library/Tomcat/apache-tomcat-6.0.32/webapps/cga/WEB-INF/lib/stripes-1.5.6.jar":
org.apache.jasper.JasperException: Failed to load or instantiate
TagExtraInfo class: net.sourceforge.stripes.tag.ErrorsTagExtraInfo
I've checked the usual suspects and verified that the jar file is in the correct location, that the tld file is present in the jar, and so forth. Likewise, searching for the error shows only a limited number of returns, primarily related to a mailing list question with the same error. I've gone through that post and also tried those solutions with not much luck.
Any ideas as to what might be causing this error?
The full error trace that I'm receiving is as follows:
org.apache.jasper.JasperException: /index.jsp(1,1) /layout/taglibs.jsp(2,77) Unable to read TLD "META-INF/stripes.tld" from JAR file "file:/Library/Tomcat/apache-tomcat-6.0.32/webapps/cga/WEB-INF/lib/stripes-1.5.6.jar": org.apache.jasper.JasperException: Failed to load or instantiate TagExtraInfo class: net.sourceforge.stripes.tag.ErrorsTagExtraInfo
org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:40)
org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:407)
org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:88)
org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:300)
org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:333)
org.apache.jasper.compiler.Parser.parseDirective(Parser.java:442)
org.apache.jasper.compiler.Parser.parseElements(Parser.java:1397)
org.apache.jasper.compiler.Parser.parse(Parser.java:130)
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:255)
org.apache.jasper.compiler.ParserController.parse(ParserController.java:103)
org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:185)
org.apache.jasper.compiler.Compiler.compile(Compiler.java:347)
org.apache.jasper.compiler.Compiler.compile(Compiler.java:327)
org.apache.jasper.compiler.Compiler.compile(Compiler.java:314)
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:326)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

This actually took awhile to chase down as the error message is not entirely reflective of what the actual problem is, but in a nut shell this an similar sounding errors (i.e. "Unable to read TLD “META-INF/*.tld” from JAR file") seem to occur when there is a class conflict between one the JAR files that you have deployed and what is deployed with Tomcat itself. This turned out to be the case for my error as there were two files in place (javax.servlet.jsp-api-2.2.1.jar and el-api-1.0.jar) that were conflicting.
As this issue seems to popup quite a bit, here are a couple troubleshooting tips:
Double check what you are deploying to make sure there is nothing unexpected, if you are using a dependency manager make sure the dependency of a dependency isn't causing you problems.
Double check to make sure you have cleared everything out of previous version and deploy again, surprisingly this caught me more than once causing some frustration.
Don't include servlet-api.jar directly in your own lib but rather use an external JAR with a variable path - guide.
Don't expect JAR files to be named as you expect or for them to not contain something you don't think they would have, if you have to, open them up to double check the contents.

getting javax.mail.MessagingException

I am using javax.mail .jar file to read the mail messages. But when i m running the code i am getting the following exception.
I added mail.jar in classpath.
Exception in thread "main" java.lang.NoClassDefFoundError: javax/mail/MessagingE
xception
Caused by: java.lang.ClassNotFoundException: javax.mail.MessagingException
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
Could not find the main class: MasterProcess. Program will exit.
any suggetionsn to solve this issue please ...

You're not getting a MessagingException, the VM is complaining that it can't find MessagingException (although it's probably that it's looking for it because it wants to throw it, but those are issues for later).
Check if your mail.jar actually contains this class, and check if your mail.jar really is on the classpath.
The last thing that could happen is that the class is incompatible with your version of Java. Classes compiled for 1.5 won't run on 1.4, for example.

Working in eclipse and not in command line clearly say that there is multiple versions of mail jar present in the project.
Ex:
You have 3 Jars , Jar X, Jar Y and Jar Z.
With out you knowing JAR X might be already be bundled inside JAR Z ( in this case mail jar but some different version.
So what is the problem in having multiple version of same jar ?
You have two different versions and you will not know which will be referenced in your project. (so if you are looking for some class from mail-2.jar in your project you will get class not found exception if the reference is made to mail-1.jar by class loader)
So how come its working properly on Eclipse ?
In eclipse you can see there is an option for ORDER the library, these will be reference in the same order, but while running in command prompt we used to load the as lib/* which loads all to gather and we will not know which lib will get loaded first.
How to identify the culprit and fix the issue ?
Option 1 :
See docs page / user guide of the libs you are using to see what they have in them.
Option 2 :
In eclipse move the JAR in question (mail.jar here) to TOP in the class path order. (your program should work now)
Now bring down the order one by one to see where you get the error. (if you wish to find the jar which is causing this issue , duplicate reference)
Extract the jar which has the duplicate reference remove the duplicate inside the JAR and repackage it. (if required)

Also be sure you're having also the java activation.jar jar.
Are you by any chance working with an App Server (eg. Jboss), if so, just check classloading model (in jboss with single classloading model you may be loading a not so up-to-date version of the jars)

I've also faced the same error. It happens to me that while compiling i used the jars correctly. While running it on command line i noticed the mail.jar (javamail API) was not included.

Hibernate in Servlet causesNoClassDefFoundError: org/slf4j/LoggerFactory

I have a Java (6) application which uses Hibernate (V3.3.2) to read data from HSQLDB, which i build and debug/run with Eclipse (V3.5.1), and it works fine.
I then created a GWT (V1.7) Servlet web app, copied my hibernate classes into it, and added the same User Libraries dependencies. However when i run the servlet and try to access a URL which invokes my code i get this:
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.hibernate.cfg.Configuration.<clinit>(Configuration.java:152)
at xxx.daoimpl.DAOSession.initialise(DAOSession.java:40)
where DAOSession.java:40 is :
AnnotationConfiguration config = new AnnotationConfiguration ();
Googling for this error suggests i am missing slf4j-api.jar from the classpath, however if i look at the command line for the Debug properties i can see this jar there:
C:\java\jsedk_6\jre\bin\javaw.exe
-agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:54541
-Xmx512m -Dfile.encoding=Cp1252
-classpath
D:\dev\workspace\xxx\src;
D:\dev\workspace\xxx\resources;
D:\dev\workspace\xxx\war\WEB-INF\classes;
C:\java\eclipse\plugins\com.google.gwt.eclipse.sdkbundle.win32_1.7.1.v200909221731\gwt-windows-1.7.1\gwt-user.jar;
C:\java\eclipse\plugins\com.google.gwt.eclipse.sdkbundle.win32_1.7.1.v200909221731\gwt-windows-1.7.1\gwt-dev-windows.jar;
C:\java\hibernate-annotations-3.4.0.GA\hibernate-annotations.jar;
C:\java\hibernate-annotations-3.4.0.GA\lib\ejb3-persistence.jar;
C:\java\hibernate-annotations-3.4.0.GA\lib\hibernate-commons-annotations.jar;
C:\java\hibernate-distribution-3.3.2.GA\hibernate3.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\antlr-2.7.6.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\commons-collections-3.1.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\dom4j-1.6.1.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\javassist-3.9.0.GA.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\jta-1.1.jar;
C:\java\hibernate-validator-4.0.1.GA\hibernate-validator-4.0.1.GA.jar;
C:\java\hibernate-validator-4.0.1.GA\lib\validation-api-1.0.0.GA.jar;
C:\java\hibernate-validator-4.0.1.GA\lib\log4j-1.2.14.jar;
C:\java\hsqldb\lib\hsqldb.jar;
C:\java\restlet-jse-2.0m5\lib\org.restlet.jar;
C:\java\restlet-jee-2.0m5\lib\org.restlet.ext.servlet.jar;
C:\java\restlet-jse-2.0m5\lib\org.restlet.ext.xml.jar;
C:\java\slf4j-1.5.8\slf4j-api-1.5.8.jar;
C:\java\slf4j-1.5.8\slf4j-log4j12-1.5.8.jar
com.google.gwt.dev.HostedMode
...
If i open the jar i can see the LoggerFactory class in there.
Any idea why it isn't being found by the class loader?
Edit 1: If try to access org.slf4j.LoggerFactory from my code, Eclipse compiles it ok, but i get the same error at runtime.
Edit 2: If i add a Test class with a main which calls the same code and run it, it works. So this classpath problem seems specific to the Servlet.
thanks,
jon

It seems that copying the two slf4j jars to the war/WEB-INF/lib sub-project/dir fixed the problem. I'm not really sure why i should need to do this for these two jars and not for all the other Hibernate, Restlet etc jars that the project also uses, though i suppose for consistency i will do that anyway - i guess it will make deployment easier as well.
If someone can provide some sort of explanation as to why this worked and why exactly i need to do it i will select it as the "correct" answer, otherwise i'll select this one.

Can you please reconfirm that you have at least two slf4j.jar files in the classpath, the slf4j-api.jar and exactly one implementation, such as slf4j-jdk14.jar?
You must not have multiple sflj4-implementations in the classpath.

You have a runtime dependency issue so everything compiles fine but the dependency is in your jars. You need to visit the Hibernate site and look at the compatibility matrix and make sure you are matched up correctly then check the dependencies for annotations and core. You log4J jars look fine so it's definitely some quirk.
If it works in Eclipse then logically it's definitely some diff between the 2 runtimes (eclipse and non-eclipse), if the matrix checks out fine then see if you can factor out the commonalities in the 2 then work out the differences -- your answer should be in there.

I had a similar issue except I was using Tomcat and the NoClassDefFound error was on the juli logger. I fixed it by removing the Tomcat dependencies from my classpath when running in hosted mode because hosted mode embeds a Tomcat server that was conflicting. So I would see what happens if you remove some or all of the restlet jars from your classpath to see if they conflict.