Cannot run Flume because of JAR conflict - java

I've installed Flume and Hadoop manually (I mean, not CDH) and I'm trying to run the twitter example from Cloudera.
In the apache-flume-1.5.0-SNAPSHOT-bin directory, I start the agent with the following command:
bin/flume-ng agent -c conf -f conf/twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent
My conf/twitter.conf file uses the logger as the sink. The conf/flume-env.sh assigns to CLASSPATH the flume-sources-1.0-SNAPSHOT.jar that contains the definition of the twitter source. The resulting output is:
(...) [ERROR org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows. java.lang.NoSuchMethodError:
twitter4j.FilterQuery.setIncludeEntities(Z)Ltwitter4j/FilterQuery;
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:139)
The conflict results from a FilterQuery class that is defined elsewhere in the flume lib and that does not contain the setIncludeEntities method. For me, the file that contains this class is the twitter4j-stream-3.0.3.jar and I cannot exclude the file from the classpath as suggested here.

I believe this experience was quite frustrating for you, for me it was for sure. The main problem is, both the files, flume-sources-1.0-SNAPSHOT.jar and twitter4j-stream-3.0.3.jar contains the same FilterQuery.class. That is why the conflict message is generated in the log file.
I am not a Java or Big Data expert, but I can give you an alternate to this problem. Download the Twitter4j-stream-2.6.6.jar or lower version from here and replacethe twitter4j-stream-3.0.3.jar. All the 3.X.X uses this class. After replacing, everything should work fine. But you may get some heap error after downloading huge amount of tweets. Please google the solution as it was resolved in 3.X.X files.
-Edit
Also, please don't forget to download and replace all the twitter4j files in /usr/lib/flume-ng folder. Namely, twitter4j-media-support-2.2.6.jar, twitter4j-stream-2.2.6.jar and twitter4j-core-2.2.6.jar. Any mismatch related to version among these files will also create problem.

As suggested in the post a problematic file can be search-contrib-1.0.0-jar-with-dependencies.jar too.

You need to recompile flume-sources-1.0-SNAPSHOT.jar from the git:https://github.com/cloudera/cdh-twitter-example
Install Maven, then download the repository of cdh-twitter-example.
Unzip, then execute inside (as mentionned) :
$ cd flume-sources
$ mvn package
$ cd ..
This problem happened when the twitter4j version updated from 2.2.6 to 3.X, they removed the method setIncludeEntities, and the JAR is not up to date.
PS: Do not download the prebuilt version, it is still the old.

Simply rename all twitter4j-stream* jar files and rerun your flume. It will work with charm. :)

I had the same problem and at last I solved following these steps:
First I renamed all jar files in jarx: from twitter4j-stream-3.0.3.jar -> twitter4j-stream-3.0.3.jarx, ...
This solved the error, but when it tried to estabilish connection, I got error 404:
(Twitter Stream consumer-1[Establishing connection])
[INFO - Twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] 404:
The URI requested is invalid or the resource requested, such as a user, does not exist.)
After reading this page (https://twittercommunity.com/t/twitter-streaming-api-not-working-with-twitter4j-and-apache-flume/66612/11) finally I solved downloading a new version of twitter4j (in the page there's a link).
Probably not the best solution, but worked for me.

Related

jar: Package org.apache.catalina.ssi missing from ModulePackages class file attribute

I changed my tomcat version 9 to 8 but when I wrote jar uf ../catalina.jar org/apache/catalina/util/ServerInfo.properties it gave me the following error
jar: Package org.apache.catalina.ssi missing from ModulePackages class
file attribute
How can I solve this issue?
Had the same issue. Upgraded from apache 9.0.16 to 9.0.41, wanted to remove server version/infos, edited ServerInfo.properties, and when time came to
jar uf ../catalina.jar org/apache/catalina/util/ServerInfo.properties
got the same error than you.
I think I found the why :
I opened catalina.jar both on 9.0.16 and 9.0.41, and lo and behold : there is no ssi in org/apache/catalina/ for 9.0.41
Yet the MANIFEST.MF still includes it, as well as the module-info.class
From reading Tomcat changelog, it looks like this happened with tomcat 9.0.30 :
Moved server-side include (SSI) module into a separate JAR library. (schultz)
(looking in the lib directory, it seems like it is catalina-ssi.jar now)
Not sure how clean this is, but I went around the problem by opening catalina.jar with 7zip, modify the ServerInfo.properties from there, saved, restarted Tomcat, and it worked.
That is a really weird issue I'm facing myself right know, excluding
org.apache.catalina.ssi from the module-info.class
using ModiTect helped, hope it won't crash in a long run...

Optaplanner - Error when running executable jar but works fine from eclipse

I use eclipse for my optaplanner project development and I am able to execute my code from eclipse without any problem. To make this project as an executable, I exported it as a jar file using eclipse export wizard.
Then I run my jar from terminal using the command java -jar myOptaPlanner.jar, then, I get below error
WARN given scan urls are empty. set urls in the configuration
Exception in thread "main" java.lang.IllegalStateException: The scanAnnotatedClasses
(ScanAnnotatedClassesConfig()) did not find any classes with a PlanningSolution
annotation.
Maybe you forgot to annotate a class with a PlanningSolution annotation.
Maybe you're using special classloading mechanisms (OSGi, ...) and this is a bug.
If you can confirm that, report it to our issue tracker and workaround it by
defining the classes explicitly in the solver configuration.
at org.optaplanner.core.config.domain.ScanAnnotatedClassesConfig.loadSolutionClass(ScanAnnotatedClassesConfig.java:106)
at org.optaplanner.core.config.domain.ScanAnnotatedClassesConfig.buildSolutionDescriptor(ScanAnnotatedClassesConfig.java:86)
at org.optaplanner.core.config.solver.SolverConfig.buildSolutionDescriptor(SolverConfig.java:270)
at org.optaplanner.core.config.solver.SolverConfig.buildSolver(SolverConfig.java:216)
at org.optaplanner.core.impl.solver.AbstractSolverFactory.buildSolver(AbstractSolverFactory.java:57)
at org.roster.app.WorkerRosteringApp.main(WorkerRosteringApp.java:36)
Is this really a bug or am I exporting it wrong?
Update: I tried to export jar using maven-shade-plugin and here also I am getting the same error.
Thanks
scanAnnotatedClasses doesn't seem to work in a jar file. Instead, use following workaround:
<solutionClass>path.to.your.SolutionClass</solutionClass>
<entityClass>path.to.your.EntityClassI</entityClass>
<entityClass>path.to.your.EntityClassII</entityClass>
and delete the scanAnnotatedClasses in your config file.
If you don't want to declare the solution and entity classes manually, including the packages containing the solution and entity classes also works as workaround:
<scanAnnotatedClasses>
<packageInclude>your.package.here</packageInclude>
</scanAnnotatedClasses>
Documentation about <packageInclude> can be found here

Android Studio - Resource Entry already defined

After compiling my Android application using the Android Studio (Beta), I have the following error:
Error:error: Resource entry background is already defined.
res\drawable-xhdpi-v4\background.9.png:0: Originally defined here.
This occurred when I had a image file called "background" in my res folder.
I then made a 9 patch version of it and pasted it in the same folder. This meant that there was 2 files with the same name. So I unsurprisingly I had this error.
However I now deleted all instances of the background image file (including the 9 patch version and original image) in my code and layout file and I still have this error.
You could try to do one of the following:
Clean and Build or gradle clean build assemble
Invalidate and restart Android Studio
1 Build > Clean Project
2 Build > Rebuild Project
I am using Android studio and I run exactly into the same issue and none of the answers I found on the web helped me solve the problem, including the previous one. So reading through all details of the error messages and found this lines:
Error:Execution failed for task ':app:processDebugResources'.
com.android.ide.common.internal.LoggedErrorException: Failed to run command:
C:\Users\user\AppData\Local\Android\android-studio\sdk\build-tools\android-4.4W\aapt.exe package -f --no-crunch -I C:\Users\user\AppData\Local\Android\android-studio\sdk\platforms\android-20\android.jar -M C:\Users\user\AndroidStudioProjects\DartsApp\app\build\intermediates\manifests\debug\AndroidManifest.xml -S C:\Users\user\AndroidStudioProjects\DartsApp\app\build\intermediates\res\debug -A C:\Users\user\AndroidStudioProjects\DartsApp\app\build\intermediates\assets\debug -m -J C:\Users\user\AndroidStudioProjects\DartsApp\app\build\generated\source\r\debug -F C:\Users\user\AndroidStudioProjects\DartsApp\app\build\intermediates\libs\app-debug.ap_ --debug-mode --custom-package com.dartstransit.dartsapp -0 apk --output-text-symbols C:\Users\user\AndroidStudioProjects\DartsApp\app\build\intermediates\symbols\debug
Error Code:
1
Output:
res\drawable-hdpi-v4\bus_w.png:0: error: Resource entry bus_w is already defined.
res\drawable-hdpi-v4\bus_w.jpg:0: Originally defined here.
As you can see there are two files of name bus_w with different extensions on the same folder. I had already removed those files from the project and still getting the error, so I decided to check every single folder of the project that was reference in the compiler line in the error message. Finally found that there was a copy of the file bus_w.png in the folder app\build\intermediates\res\debug\drawable-mdpi-v4.
I just deleted it and voila, problem solved.
A simple Build > Rebuild project helped me
Did you try rename this file to other name? It seem one of your libraries have same resource.
I see that many devs(include me) had issue with ic_launcher too. I think you want to try mipmap instead drawable. So we will have something like this:
\res\mipmap-hdpi
\res\mipmap-mdpi
\res\mipmap-xhdpi
\res\mipmap-xxhdpi
Hope this help.
I have also encountered this error recently.Here is my solution that works,hope it will help you.
1.only keep 9 patch version image in the drawable-hdpi folder
2.make sure the code related to this problem in xml file remains the same: android:background="#drawable/msg_left"
instead of using the code:
android:background="#drawable/msg_left.9"
PS:My original image name is msg_left,and my 9 patch version image name is msg_left.9
I did the trick by moving the icon
drawable/ic_launcher
to
mipmap/ic_launcher
and it worked fine :)
In my case the problem was that I had more than one '.' in the filename. like:
a.png
a.bold.png
Solution was to rename to
a.png
a_bold.png
In our case, problem started occurring (error shows 'duplicate value for resource' in a library) when some other library was upgraded to it's latest version.
We just downgraded the version of the library that was upgraded.
Real pain was because that we have many dependencies which can be dynamically upgraded when there is a new version available in repositories. So my teammate had to go through all libs that can be dynamically upgraded and see if there is one that is recently upgraded, try to downgrade it and check if build goes fine. He identified many were upgraded on previous day. Downgrading one of them got the build back.
Conclusion we made is not to use Dynamic Versioning. Don't use dynamic versions for your dependencies.
This was the error by the way:
Task :app:generateDebugResources UP-TO-DATE
AGPBI: {"kind":"error","text":"error: duplicate value for resource \u0027attr/itemBackground\u0027 with config \u0027\u0027.","sources":[{"file":"/Users/unknownuser/.gradle/caches/transforms-1/files-1.1/singledateandtimepicker-1.2.2.aar/86ae3786d70c769e610c8a01d58aa638/res/values/values.xml","position":{"startLine":15,"startColumn":4,"startOffset":2427,"endColumn":48,"endOffset":2471}}],"original":"","tool":"AAPT"}
AGPBI: {"kind":"error","text":"error: resource previously defined here.","sources":[{"file":"/Users/unknownuser/.gradle/caches/transforms-1/files-1.1/singledateandtimepicker-1.2.2.aar/86ae3786d70c769e610c8a01d58aa638/res/values/values.xml","position":{"startLine":15,"startColumn":4,"startOffset":2427,"endColumn":48,"endOffset":2471}}],"original":"","tool":"AAPT"}
Task :app:mergeDebugResources FAILED
Error: java.util.concurrent.ExecutionException: com.android.builder.internal.aapt.v2.Aapt2Exception: AAPT2 error: check logs for details
FAILURE: Build failed with an exception.

Navigation Engine not able to process request

I get the following error in Endeca:
com.endeca.navigation.ENEException: Navigation Engine not able to process
request `http://myhost:15000/search?terms=remote&rank=0&offset=0&irversion=601`
My MDEX version is 6.3.0
The referenced jar files in my app is:
endeca_navigation.jar : Specification-IR-Version: 6.3.0 (Read from MANIFEST file)
endeca_logging.jar: Specification-Version: 6.1.2 (Read from MANIFEST file)
Even with these referenced jars, why is the request parameter still irversion=601?
I'm using java 1.5 and building my app in eclipse. At first, it was referencing older jar files (601), but i removed those completely and replaced them with the newer jar files listed above. I've rebuilt my app and I still see the parameter irversion=601. I've been trying to find out what the issue is for a few hours, but i'm having no luck.
Any ideas?
If nothing else I think that you need an N=0 parameter for you request to work.
Also if you are using an application server like Weblogic to run you app, it has a tendency of caching referenced jars which may have to be cleared for it to pick up the new versions of the jars.
Finally if all else fails there is a --back_compat 601 parameter on your dgraph which indicated the accepatble navigation.jar versions. You can use this parameter to determine if the issue is the jar version or something else in between.
Good Luck
Additionally to Wiszh's response, it is useful to take a look at the dgraph request query logs (located in /logs/dgraph/Dgraphxx/Dgraphxx.log and .reqlog) which will show a more specific error and the query that caused it.
Do your jars match the version of Endeca server you are querying?
This issue is due to an old navigation api jar in your class path
Check the ToolsAndFrameworks\version\assembler\lib to get the correct version of navigation api jar. That should fix this issue.
correct jar path fixed the issue for me
ToolsAndFrameworks\version\assembler\lib

Hibernate in Servlet causesNoClassDefFoundError: org/slf4j/LoggerFactory

I have a Java (6) application which uses Hibernate (V3.3.2) to read data from HSQLDB, which i build and debug/run with Eclipse (V3.5.1), and it works fine.
I then created a GWT (V1.7) Servlet web app, copied my hibernate classes into it, and added the same User Libraries dependencies. However when i run the servlet and try to access a URL which invokes my code i get this:
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.hibernate.cfg.Configuration.<clinit>(Configuration.java:152)
at xxx.daoimpl.DAOSession.initialise(DAOSession.java:40)
where DAOSession.java:40 is :
AnnotationConfiguration config = new AnnotationConfiguration ();
Googling for this error suggests i am missing slf4j-api.jar from the classpath, however if i look at the command line for the Debug properties i can see this jar there:
C:\java\jsedk_6\jre\bin\javaw.exe
-agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:54541
-Xmx512m -Dfile.encoding=Cp1252
-classpath
D:\dev\workspace\xxx\src;
D:\dev\workspace\xxx\resources;
D:\dev\workspace\xxx\war\WEB-INF\classes;
C:\java\eclipse\plugins\com.google.gwt.eclipse.sdkbundle.win32_1.7.1.v200909221731\gwt-windows-1.7.1\gwt-user.jar;
C:\java\eclipse\plugins\com.google.gwt.eclipse.sdkbundle.win32_1.7.1.v200909221731\gwt-windows-1.7.1\gwt-dev-windows.jar;
C:\java\hibernate-annotations-3.4.0.GA\hibernate-annotations.jar;
C:\java\hibernate-annotations-3.4.0.GA\lib\ejb3-persistence.jar;
C:\java\hibernate-annotations-3.4.0.GA\lib\hibernate-commons-annotations.jar;
C:\java\hibernate-distribution-3.3.2.GA\hibernate3.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\antlr-2.7.6.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\commons-collections-3.1.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\dom4j-1.6.1.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\javassist-3.9.0.GA.jar;
C:\java\hibernate-distribution-3.3.2.GA\lib\required\jta-1.1.jar;
C:\java\hibernate-validator-4.0.1.GA\hibernate-validator-4.0.1.GA.jar;
C:\java\hibernate-validator-4.0.1.GA\lib\validation-api-1.0.0.GA.jar;
C:\java\hibernate-validator-4.0.1.GA\lib\log4j-1.2.14.jar;
C:\java\hsqldb\lib\hsqldb.jar;
C:\java\restlet-jse-2.0m5\lib\org.restlet.jar;
C:\java\restlet-jee-2.0m5\lib\org.restlet.ext.servlet.jar;
C:\java\restlet-jse-2.0m5\lib\org.restlet.ext.xml.jar;
C:\java\slf4j-1.5.8\slf4j-api-1.5.8.jar;
C:\java\slf4j-1.5.8\slf4j-log4j12-1.5.8.jar
com.google.gwt.dev.HostedMode
...
If i open the jar i can see the LoggerFactory class in there.
Any idea why it isn't being found by the class loader?
Edit 1: If try to access org.slf4j.LoggerFactory from my code, Eclipse compiles it ok, but i get the same error at runtime.
Edit 2: If i add a Test class with a main which calls the same code and run it, it works. So this classpath problem seems specific to the Servlet.
thanks,
jon
It seems that copying the two slf4j jars to the war/WEB-INF/lib sub-project/dir fixed the problem. I'm not really sure why i should need to do this for these two jars and not for all the other Hibernate, Restlet etc jars that the project also uses, though i suppose for consistency i will do that anyway - i guess it will make deployment easier as well.
If someone can provide some sort of explanation as to why this worked and why exactly i need to do it i will select it as the "correct" answer, otherwise i'll select this one.
Can you please reconfirm that you have at least two slf4j.jar files in the classpath, the slf4j-api.jar and exactly one implementation, such as slf4j-jdk14.jar?
You must not have multiple sflj4-implementations in the classpath.
You have a runtime dependency issue so everything compiles fine but the dependency is in your jars. You need to visit the Hibernate site and look at the compatibility matrix and make sure you are matched up correctly then check the dependencies for annotations and core. You log4J jars look fine so it's definitely some quirk.
If it works in Eclipse then logically it's definitely some diff between the 2 runtimes (eclipse and non-eclipse), if the matrix checks out fine then see if you can factor out the commonalities in the 2 then work out the differences -- your answer should be in there.
I had a similar issue except I was using Tomcat and the NoClassDefFound error was on the juli logger. I fixed it by removing the Tomcat dependencies from my classpath when running in hosted mode because hosted mode embeds a Tomcat server that was conflicting. So I would see what happens if you remove some or all of the restlet jars from your classpath to see if they conflict.

Categories

Resources