I have the following environment:OS - Windows7
Environment variables set:
CLASSPATH = C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\gemfire.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\antlr.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\gfsh-dependencies.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\gfSecurityImpl.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\jackson-core-2.2.0.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\commons-logging-1.1.1.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\tomcat-embed-core.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\tomcat-embed-jasper.jar;C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows\lib\tomcat-embed-logging-juli.jar;%JAVA_HOME%\lib\tools.jar;C:\MyWorkspace\Repository\org\springframework\data\spring-data-gemfire\1.5.1.RELEASE\spring-data-gemfire-1.5.1.RELEASE.jar;%CLASSPATH%
GEMFIRE = C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows GF_JAVA = %JAVA_HOME%\bin\java.exe JAVA_HOME = C:\MyWorkspace\JDK8\jdk1.8.0_31 MAVEN_HOME = C:\MyWorkspace\Maven\apache-maven-3.3.1 PATH = %JAVA_HOME%\bin;%ERACOM_HOME%;%Cryptoki%;%WAS_HOME%\bin;%MAVEN_HOME%\bin;C:\Windows\System32;C:\Windows;C:\OtherProgramFiles\nodejs\;%GEMFIRE%\bin;
My Simple Project Files:
cache-xml-file:cache.xml
<cache>
<region name="Apps">
<region-attributes>
<cache-listener>
<class-name>mypackage.listener.TestListener</class-name>
</cache-listener>
</region-attributes>
</region>
<initializer>
<class-name>org.springframework.data.gemfire.support.SpringContextBootstrappingInitializer</class-name>
<parameter name="contextConfigLocations">
<string>spring-application-context.xml</string>
</parameter>
</initializer>
</cache>
mypackage.listener.TestListener
public class TestListener implements CacheListener, Declarable {
#
Override
public void init(Properties arg0) {
System.out.println("Inside TestListener.init()");
}
... //Other Overridden methods . . .
}
spring-application-context.xml: contains a single bean:
<?xml version="1.0" encoding="utf-8" ?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:gfe="http://www.springframework.org/schema/gemfire" xmlns:util="http://www.springframework.org/schema/util" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/gemfire http://www.springframework.org/schema/gemfire/spring-gemfire.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="myTestCacheLoader" class="mypackage.loader.MyTestCacheLoader">
</bean>
</beans>
**mypackage.loader.MyTestCacheLoader:
public class MyTestCacheLoader implements
CacheLoader, Declarable {
#
Override
public void close() {
}
#
Override
public Object load(LoaderHelper arg0) throws CacheLoaderException {
System.out.println("Inside MyTestCacheLoader.load()");
return null;
}
#
Override
public void init(Properties arg0) {
}
}
Steps:
1. Start gfsh from directory - C:\MyWorkspace\Pivotal_GemFire_800_b48398_Windows
start locator: start locator --name=locator1 --port=10334
start server:
start server --name=server1 --server-port=40411 --cache-xml-file=C:\MyWorkspace\Codebase\ContentServices\content-loaderwriter\src\test\resources\cache-loader-writer.xml --classpath=C:\MyWorkspace\Codebase\ContentServices\content-loaderwriter\target\content-loaderwriter-0.0.1-SNAPSHOT.jar
But its throwing: java.lang.ClassNotFoundException: org.springframework.data.gemfire.support.SpringContextBootstrappingInitializer
I even added the gemfire jars as well as the above spring-data-gemfire jar (from my repo) in CLASSPATH variable. Still getting the exception. It seems not to resolve the maven dependencies/classpath.
If I try to provide multiple jars in classpath in gfsh start server command:
start server --name=server1 --server-port=40411 --cache-xml-file="C:\MyWorkspace\Codebase\ContentServices\content-loaderwriter\src\test\resources\cache-loader-writer.xml" --classpath="C:\MyWorkspace\Repository\org\springframework\data\spring-data-gemfire\1.5.1.RELEASE\spring-data-gemfire-1.5.1.RELEASE.jar:C:\MyWorkspace\Codebase\ContentServices\content-loaderwriter\target\content-loaderwriter-0.0.1-SNAPSHOT.jar"
I still get the ClassNotFoundException. Its not resolving the jars provided in the classpath.
Can anyone please help?
I think the problem is simply that you have ':' as the separator character in your class path. Windows wants a ';' here. You do not need to set the CLASSPATH environment variable in this case.
However, you will most likely need other jars that spring-data-gemfire depends upon. It looks like you are using maven so here is an easy way to get the class path for all of your project dependencies including transitive ones.
In your project directory, run "mvn -DexcludeArtifactIds=gemfire dependency:build-classpath". This will give you all of your dependencies other than gemfire (which is provided automatically). Add your application jar to the resulting classpath and you should have everything you need.
Also, based on what you are doing, you would probably be interested in the "--spring-xml-location" option which was added to the "gfsh start" command in 8.1. This allows you to dispense with "cache.xml" completely and define everything in a spring context file using the "gfe" namespace provided by spring-data-gemfire.
#Suparno Karmakar - Not sure if you were able to resolve this issue yet, but in reviewing your CLASSPATH to the 'start server' command (i.e. the value for the --classpath option), assuming the path is correct, then one thing to keep in mind is that the --classpath option value is read into a String when passed as an argument to 'start server'.
From there, the CLASSPATH value is subsequently passed and appended to the eventual command-line String (java.exe -classpath ... JVM arg) that is used to "fork" the JVM process for the GemFire Server (using the java.lang.ProcessBuilder API).
Java is most likely treating the first forward slash () in the path information as an "escape" character", so you have to escape the escape character as in...
C:\\path\\to\\external\\file.jar
I believe backslashes in Windows will also work, therefore...
C:/path/to/external/file.jar
#Randy May is correct in stating that you can just use the 'start server' command's new '--spring-xml-location' option to indicate that you'd like to bootstrap your the GemFire Server with Spring, assuming you are using GemFire 8.1 of course.
By using this option, the appropriate Spring JARS from $GEMFIRE/lib are automatically appended to the Server's CLASSPATH on start (which otherwise are not used).
Unfortunately, there was a bug identified when using the '--spring-xml-location' option with 'start server' in that Gfsh was looking for the wrong version of the Spring JARS. It was looking for Spring core 3.2.11.RELEASE instead of 3.2.12.RELEASE, SDC 1.9.0.RELEASE instead of 1.9.1.RELEASE and SDG 1.5.1.BUILD-SNAPSHOT instead of 1.5.1.RELEASE.
Simply renaming the Spring JARs in $GEMFIRE/lib resolves this problem.
See https://svn.gemstone.com/trac/gemfire/ticket/51956, though you probably don't have access to see this bug until the release notes for GemFire 8.2 is out (~July).
Here are my bug notes...
Bug Title:
GemFire Shell (Gfsh) throws a
NoClassDefFoundError/ClassNotFoundException on
org/springframework/data/gemfire/support/SpringContextBootstrappingInitializer
when launching a GemFire Server using Gfsh with the 'start server ...
--spring-xml-location' option in GemFire 8.1
Bug Workaround:
When a user tries to launch a GemFire Server using Gfsh with the
'start server' command bootstrapped with Spring by specifying the
'--spring-xml-location' option, GemFire 8.1 (Gfsh) throws a
NoClassDefFoundError caused by a ClassNotFoundException... {{{ The
Cache Server process terminated unexpectedly with exit status 1.
Please refer to the log file in /Users/jblum/vmdev/lab/serverX for
full details. Exception in thread "main"
java.lang.NoClassDefFoundError:
org/springframework/data/gemfire/support/SpringContextBootstrappingInitializer
at
com.gemstone.gemfire.distributed.ServerLauncher.startWithSpring(ServerLauncher.java:764)
at
com.gemstone.gemfire.distributed.ServerLauncher.start(ServerLauncher.java:696)
at
com.gemstone.gemfire.distributed.ServerLauncher.run(ServerLauncher.java:626)
at
com.gemstone.gemfire.distributed.ServerLauncher.main(ServerLauncher.java:200)
Caused by: java.lang.ClassNotFoundException:
org.springframework.data.gemfire.support.SpringContextBootstrappingInitializer
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 4 more }}}
GemFire is incorrectly specifying the Spring LIBS as version
3.2.11.RELEASE when GemFire in fact shipped with Spring 3.2.12.RELEASE JAR files. In addition, the Spring Data Commons JAR file is
incorrectly identified as spring-data-commons-1.9.0.RELEASE.jar,
however, GemFire 8.1 shipped with
spring-data-commons-1.9.1.RELEASE.jar. Note that GemFire/Gfsh in 8.1
is correctly specifying the Spring Data GemFire JAR file with
spring-data-gemfire-1.5.1.RELEASE.jar See Bugnote workaround to
resolve this issue.
Bug Workaround:
There are 2 ways to resolve this issue: 1. First, and the tried and
tested procedure is to... 1.1 Download and copy the Spring
3.2.11.RELEASE JAR files into the $GEMFIRE/lib directory. For every Spring 3.2.12.RELEASE JAR file in $GEMFIRE/lib, copy the corresponding
3.2.11.RELEASE JAR into $GEMFIRE/lib. 1.2 Next, download and copy the spring-data-commons-1.9.0.RELEASE.jar file into $GEMFIRE/lib. 1.3
Finally, perform the following... $ cp
$GEMFIRE/lib/spring-data-gemfire-1.5.1.RELEASE.jar
$GEMFIRE/lib/spring-data-gemfire-1.5.1.BUILD-SNAPSHOT.jar --- NOTE:
Unfortunately the bug is that Gfsh is looking for the wrong version of
these particular Spring JAR files. --- 2. The other option, is to
manually specify the Spring JARS and versions yourself using the
'--classpath' option to the 'start server' command. Note, in GemFire
8.1, the '--classpath' option has been changed to a "prepend" option, and therefore, effectively overrides any GemFire lib dependency in the
$GEMFIRE/lib directory. Caution, however care should be taken when
overriding any of the GemFire lib dependencies. NOTE, this "prepend"
behavior does not apply to the 'gemfire.jar' file itself for security
reasons.
Hope this helps! Post back if you still have problems.
Cheers!
Please replace the separator ":" to "\;"
You need to add the escape character before the windows separator. Otherwise, gfsh will treat the ";" as the ending of one command.
Related
I have a java-application (standard springboot from default tutorial: https://spring.io/guides/gs/spring-boot-for-azure/ ) that I "successfuly" deploy to my WebApp (created during deployment) via Eclipse/maven plugin azure-webapp:deploy
Once deployed, files are inside the WebApp, I can see them. If start-up is successful I do get running application, but if it is not - I do not know how to troubleshoot. I don't know where to find error logs, what caused the problem and as consequence, how to solve it.
as an example of how to make it fail, add this line:
throw new RuntimeException("Doomed to fail");
I tried enabling logs from "diagnostic logs tab" and expected to see them under LogFiles/Applications but that folder remains empty.
How do I troubleshoot java-application that fails to start in WebApps of Azure?
edit: additional example of Exception to troubleshoot:
public static void main(String[] args) {
throw new RuntimeException("start failure #21");
//SpringApplication.run(Application.class, args);
}
It sounds like you followed the springboot tutorial Deploying a Spring Boot app to Azure to build the GitHub project microsoft/gs-spring-boot and deploy to Azure, but not works.
Here is my steps which I also followed the tutorial, but deployed via my own way.
I created a directory SpringBoot on my local machine, and to do the commands cd SpringBoot and git clone https://github.com/microsoft/gs-spring-boot.
Then, to build it via commands cd gs-spring-boot/complete and mvnw clean package
Note: I reviewed the sections of the tutorial under Create a sample Spring Boot web app which seems to do on Linux, but the web.config file in microsoft/gs-spring-boot/complete is ready to Azure WebApp for Windows. However, there is not any comments to describe the deployment target that be Azure WebApp for Windows or Linux.
So I used my existing WebApp for Windows to test my deployment. I open my Kudu console in browser via the url https://<webapp name>.scm.azurewebsites.net/DebugConsole and drag the files complete/web.config and complete/target/gs-spring-boot-0.1.0.jar to site/wwwroot as the figure below. Then, I started my webapp, and it works fine.
Note: Please check the JAVA_HOME environment variable which has been configured on Azure via command echo %JAVA_HOME% as the figure below.
If not, you need to set Java runtime in the Application settings tab of Azure portal.
Or you can also configure the web.config file to replace the reference %JAVA_HOME% with an existing Java runtime installed in the path D:\Program Files\Java of Azure WebApp, as below.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="httpPlatformHandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified"/>
</handlers>
<!-- <httpPlatform processPath="%JAVA_HOME%\bin\java.exe" -->
<httpPlatform processPath="D:\Program Files\Java\jre1.8.0_181\bin\java.exe"
arguments="-Djava.net.preferIPv4Stack=true -Dserver.port=%HTTP_PLATFORM_PORT% -jar "%HOME%\site\wwwroot\gs-spring-boot-0.1.0.jar"">
</httpPlatform>
</system.webServer>
</configuration>
I didn't manage to find logs in windows based machine, but If you enable logs on linux-based-machine you will see them in the "Diagnostic logs" output. There is a catch though.
There is 230 timeout. It will wait full "timeout" time, until producing logs in the log file, after which you can access them via log file or through "Diagnostic logs". Make sure to enable logging before you start application. This applies to linux based machine, I don't know if it can apply to windows based machine.
Then it waits for an answer trigger. Answer trigger, as it turns out, is a phrase in console output "Application is started in X seconds". I increased the timeout to 500 seconds, because although it starts in 60 seconds on my local machine, it takes 430 seconds on remote-linux-based machine of Microsoft Azure*
Second, I changed the name of my main class from "GameStart" to "Application" and after that it actually caught the trigger. After that the application started. Nowhere in manuals did I find the "until timeout - no logs" and "trigger phrase" clauses mentioned.
ps: For reference, it took me 20 to upload application, 5 minutes to start it. I was using centralUS server, cuz centralEU did not work out for me, as it was even longer, although I'm in central EU myself
-
*using test account. It might happen that on payed account number are either different or similar.
I am trying to setup kafka-connect-cassandra on an AWS instance.
I have setup plugin.path in connect-avro-distributed.properties file:
plugin.path=/home/ubuntu/kafka_2.11-1.0.0/plugins
And I have kafka-connect-cassandra-1.0.0-1.0.0-all.jar in:
/home/ubuntu/kafka_2.11-1.0.0/plugins/lib
This is the traceback:
[2018-02-18 10:28:33,268] INFO Kafka Connect distributed worker initializing ... (org.apache.kafka.connect.cli.ConnectDistributed:60)
[2018-02-18 10:28:33,278] INFO WorkerInfo values:
jvm.args = -Xmx256M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/log/kafka, -Dlog4j.configuration=file:/etc/kafka/connect-log4j.properties
jvm.spec = Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_151, 25.151-b12
jvm.classpath = /home/ubuntu/kafka_2.11-1.0.0/plugins:/usr/share/java/kafka/jackson-jaxrs-json-provider-2.9.1.jar:/usr/share/java/kafka/paranamer-2.7.jar:/usr/share/java/kafka/rocksdbjni-5.7.3.jar:/usr/share/java/kafka/jackson-jaxrs-base-2.9.1.jar:/usr/share/java/kafka/jetty-servlets-9.2.22.v20170606.jar:/usr/share/java/kafka/kafka-clients-1.0.0-cp1.jar:/usr/share/java/kafka/xz-1.5.jar:/usr/share/java/kafka/commons-lang3-3.1.jar:/usr/share/java/kafka/jetty-security-9.2.22.v20170606.jar:/usr/share/java/kafka/httpclient-4.5.2.jar:/usr/share/java/kafka/jackson-core-asl-1.9.13.jar:/usr/share/java/kafka/connect-json-1.0.0-cp1.jar:/usr/share/java/kafka/connect-runtime-1.0.0-cp1.jar:/usr/share/java/kafka/jersey-container-servlet-2.25.1.jar:/usr/share/java/kafka/commons-collections-3.2.1.jar:/usr/share/java/kafka/connect-transforms-1.0.0-cp1.jar:/usr/share/java/kafka/jersey-common-2.25.1.jar:/usr/share/java/kafka/zookeeper-3.4.10.jar:/usr/share/java/kafka/scala-library-2.11.11.jar:/usr/share/java/kafka/jackson-core-2.9.1.jar:/usr/share/java/kafka/argparse4j-0.7.0.jar:/usr/share/java/kafka/maven-artifact-3.5.0.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-test-sources.jar:/usr/share/java/kafka/commons-codec-1.9.jar:/usr/share/java/kafka/kafka-connect-cassandra-1.0.0-1.0.0-all.jar:/usr/share/java/kafka/httpcore-4.4.4.jar:/usr/share/java/kafka/hk2-utils-2.5.0-b32.jar:/usr/share/java/kafka/connect-api-1.0.0-cp1.jar:/usr/share/java/kafka/javassist-3.21.0-GA.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-sources.jar:/usr/share/java/kafka/support-metrics-client-4.0.0.jar:/usr/share/java/kafka/commons-compress-1.8.1.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-scaladoc.jar:/usr/share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/share/java/kafka/jersey-media-jaxb-2.25.1.jar:/usr/share/java/kafka/kafka-streams-1.0.0-cp1.jar:/usr/share/java/kafka/zkclient-0.10.jar:/usr/share/java/kafka/hk2-locator-2.5.0-b32.jar:/usr/share/java/kafka/slf4j-api-1.7.25.jar:/usr/share/java/kafka/support-metrics-common-4.0.0.jar:/usr/share/java/kafka/kafka.jar:/usr/share/java/kafka/jersey-server-2.25.1.jar:/usr/share/java/kafka/jackson-module-jaxb-annotations-2.9.1.jar:/usr/share/java/kafka/jetty-io-9.2.22.v20170606.jar:/usr/share/java/kafka/kafka-log4j-appender-1.0.0-cp1.jar:/usr/share/java/kafka/avro-1.8.2.jar:/usr/share/java/kafka/jackson-annotations-2.9.1.jar:/usr/share/java/kafka/guava-20.0.jar:/usr/share/java/kafka/hk2-api-2.5.0-b32.jar:/usr/share/java/kafka/lz4-java-1.4.jar:/usr/share/java/kafka/reflections-0.9.11.jar:/usr/share/java/kafka/commons-digester-1.8.1.jar:/usr/share/java/kafka/slf4j-log4j12-1.7.25.jar:/usr/share/java/kafka/jersey-client-2.25.1.jar:/usr/share/java/kafka/commons-lang3-3.5.jar:/usr/share/java/kafka/jackson-mapper-asl-1.9.13.jar:/usr/share/java/kafka/javax.annotation-api-1.2.jar:/usr/share/java/kafka/snappy-java-1.1.4.jar:/usr/share/java/kafka/javax.inject-2.5.0-b32.jar:/usr/share/java/kafka/jackson-databind-2.9.1.jar:/usr/share/java/kafka/jetty-http-9.2.22.v20170606.jar:/usr/share/java/kafka/kafka-streams-examples-1.0.0-cp1.jar:/usr/share/java/kafka/plexus-utils-3.0.24.jar:/usr/share/java/kafka/metrics-core-2.2.0.jar:/usr/share/java/kafka/connect-file-1.0.0-cp1.jar:/usr/share/java/kafka/kafka-tools-1.0.0-cp1.jar:/usr/share/java/kafka/jersey-guava-2.25.1.jar:/usr/share/java/kafka/commons-logging-1.2.jar:/usr/share/java/kafka/jetty-server-9.2.22.v20170606.jar:/usr/share/java/kafka/validation-api-1.1.0.Final.jar:/usr/share/java/kafka/jetty-continuation-9.2.22.v20170606.jar:/usr/share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/share/java/kafka/httpmime-4.5.2.jar:/usr/share/java/kafka/log4j-1.2.17.jar:/usr/share/java/kafka/jopt-simple-5.0.4.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-javadoc.jar:/usr/share/java/kafka/javax.ws.rs-api-2.0.1.jar:/usr/share/java/kafka/jersey-container-servlet-core-2.25.1.jar:/usr/share/java/kafka/jetty-servlet-9.2.22.v20170606.jar:/usr/share/java/kafka/jetty-util-9.2.22.v20170606.jar:/usr/share/java/kafka/commons-validator-1.4.1.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1.jar:/usr/share/java/kafka/javax.inject-1.jar:/usr/share/java/kafka/commons-beanutils-1.8.3.jar:/usr/share/java/kafka/javassist-3.20.0-GA.jar:/usr/share/java/kafka/aopalliance-repackaged-2.5.0-b32.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-test.jar:/usr/share/java/confluent-common/zookeeper-3.4.10.jar:/usr/share/java/confluent-common/common-metrics-4.0.0.jar:/usr/share/java/confluent-common/build-tools-4.0.0.jar:/usr/share/java/confluent-common/zkclient-0.10.jar:/usr/share/java/confluent-common/slf4j-api-1.7.25.jar:/usr/share/java/confluent-common/common-utils-4.0.0.jar:/usr/share/java/confluent-common/netty-3.10.5.Final.jar:/usr/share/java/confluent-common/common-config-4.0.0.jar:/usr/share/java/confluent-common/jline-0.9.94.jar:/usr/share/java/confluent-common/log4j-1.2.17.jar:/usr/share/java/kafka-serde-tools/kafka-json-serializer-4.0.0.jar:/usr/share/java/kafka-serde-tools/paranamer-2.7.jar:/usr/share/java/kafka-serde-tools/xz-1.5.jar:/usr/share/java/kafka-serde-tools/jackson-core-asl-1.9.13.jar:/usr/share/java/kafka-serde-tools/jackson-core-2.9.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-avro-converter-4.0.0.jar:/usr/share/java/kafka-serde-tools/commons-compress-1.8.1.jar:/usr/share/java/kafka-serde-tools/avro-1.8.2.jar:/usr/share/java/kafka-serde-tools/jackson-annotations-2.9.1.jar:/usr/share/java/kafka-serde-tools/kafka-schema-registry-client-4.0.0.jar:/usr/share/java/kafka-serde-tools/kafka-avro-serializer-4.0.0.jar:/usr/share/java/kafka-serde-tools/jackson-mapper-asl-1.9.13.jar:/usr/share/java/kafka-serde-tools/jackson-databind-2.9.1.jar:/usr/share/java/kafka-serde-tools/snappy-java-1.1.1.3.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.9.1.jar:/usr/bin/../share/java/kafka/paranamer-2.7.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.7.3.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.9.1.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/kafka-clients-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/xz-1.5.jar:/usr/bin/../share/java/kafka/commons-lang3-3.1.jar:/usr/bin/../share/java/kafka/jetty-security-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/httpclient-4.5.2.jar:/usr/bin/../share/java/kafka/jackson-core-asl-1.9.13.jar:/usr/bin/../share/java/kafka/connect-json-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/connect-runtime-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.25.1.jar:/usr/bin/../share/java/kafka/commons-collections-3.2.1.jar:/usr/bin/../share/java/kafka/connect-transforms-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/jersey-common-2.25.1.jar:/usr/bin/../share/java/kafka/zookeeper-3.4.10.jar:/usr/bin/../share/java/kafka/scala-library-2.11.11.jar:/usr/bin/../share/java/kafka/jackson-core-2.9.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/maven-artifact-3.5.0.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-test-sources.jar:/usr/bin/../share/java/kafka/commons-codec-1.9.jar:/usr/bin/../share/java/kafka/kafka-connect-cassandra-1.0.0-1.0.0-all.jar:/usr/bin/../share/java/kafka/httpcore-4.4.4.jar:/usr/bin/../share/java/kafka/hk2-utils-2.5.0-b32.jar:/usr/bin/../share/java/kafka/connect-api-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/javassist-3.21.0-GA.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-sources.jar:/usr/bin/../share/java/kafka/support-metrics-client-4.0.0.jar:/usr/bin/../share/java/kafka/commons-compress-1.8.1.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-scaladoc.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.25.1.jar:/usr/bin/../share/java/kafka/kafka-streams-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/zkclient-0.10.jar:/usr/bin/../share/java/kafka/hk2-locator-2.5.0-b32.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.25.jar:/usr/bin/../share/java/kafka/support-metrics-common-4.0.0.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/jersey-server-2.25.1.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.9.1.jar:/usr/bin/../share/java/kafka/jetty-io-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/avro-1.8.2.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.9.1.jar:/usr/bin/../share/java/kafka/guava-20.0.jar:/usr/bin/../share/java/kafka/hk2-api-2.5.0-b32.jar:/usr/bin/../share/java/kafka/lz4-java-1.4.jar:/usr/bin/../share/java/kafka/reflections-0.9.11.jar:/usr/bin/../share/java/kafka/commons-digester-1.8.1.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.25.jar:/usr/bin/../share/java/kafka/jersey-client-2.25.1.jar:/usr/bin/../share/java/kafka/commons-lang3-3.5.jar:/usr/bin/../share/java/kafka/jackson-mapper-asl-1.9.13.jar:/usr/bin/../share/java/kafka/javax.annotation-api-1.2.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.4.jar:/usr/bin/../share/java/kafka/javax.inject-2.5.0-b32.jar:/usr/bin/../share/java/kafka/jackson-databind-2.9.1.jar:/usr/bin/../share/java/kafka/jetty-http-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/plexus-utils-3.0.24.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/connect-file-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/kafka-tools-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/jersey-guava-2.25.1.jar:/usr/bin/../share/java/kafka/commons-logging-1.2.jar:/usr/bin/../share/java/kafka/jetty-server-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/validation-api-1.1.0.Final.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/bin/../share/java/kafka/httpmime-4.5.2.jar:/usr/bin/../share/java/kafka/log4j-1.2.17.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-javadoc.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.0.1.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.25.1.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/jetty-util-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/commons-validator-1.4.1.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/javax.inject-1.jar:/usr/bin/../share/java/kafka/commons-beanutils-1.8.3.jar:/usr/bin/../share/java/kafka/javassist-3.20.0-GA.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.5.0-b32.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-test.jar:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/*
os.spec = Linux, amd64, 4.4.0-1049-aws
os.vcpus = 2
(org.apache.kafka.connect.runtime.WorkerInfo:71)
[2018-02-18 10:28:33,279] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed:69)
[2018-02-18 10:28:33,290] INFO Loading plugin from: /home/ubuntu/kafka_2.11-1.0.0/plugins/lib (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:179)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)
The first entry in jvm.classpath is the location where I have kafka-connect-cassandra.jar, inside plugin/lib.
guava.jars:
These are the paths of guava jars. Where should I place the kafka-connect-cassandra.jar or should I just remove any of these jars?
/usr/share/java/kafka-connect-elasticsearch/guava-18.0.jar
/usr/share/java/kafka-connect-storage-common/guava-14.0.1.jar
/usr/share/java/kafka/guava-20.0.jar
/home/ubuntu/cassandra/apache-cassandra-3.11.1/lib/guava-18.0.jar
Kindly help me out.
The Connect framework currently carries guava-20.0.jar in order to use Reflections and be able to scan your plugin.path for Connect plugins.
Connect framework's dependencies are added to your CLASSPATH if you are starting Connect using one of its bin scripts: connect-distributed or connect-standalone.
However, somehow an older version of guava is residing first in your CLASSPATH (maybe because you've set the CLASSPATH to include such a dependency), and then Connect is unable to scan your plugin.path with a guava version < 20. That's what your exception indicates. That Connect can't scan the plugin.path with the version of guava it finds in your CLASSPATH.
Your plugin.path should contain your plugins, each on its own directory. For example, similarly to what you have above:
plugin.path=/home/ubuntu/connect with
/home/ubuntu/connect/kafka_2.11-1.0.0/ and
/home/ubuntu/connect/kafka-connect-cassandra-1.0.0-1.0.0-all.jar
The layout above is used to make the point that plugins should not contain other plugins according to your example, but it's not recommended to set your plugin.path to a user's home directory.
This is a classpath issue. Looks like maybe you have an incompatible version of guava in the classpath? If your plugin path isn't including this method in any of the jars it has, that's a connector packaging issue. If it is, then you probably have two versions hanging around. Double check that plugin path with a find command to inspect all jars for that class in the message as a first step. Ultimately, you'll need to figure out what version of the dependency the connector expects and get that version and only that version into the plugin path.
I downloaded sqljdbc4.jar. I'm invoking sqoop like so from the folder (where the jar is stored):
sqoop list-tables --driver com.microsoft.jdbc.sqlserver.SQLServerDriver --connect jdbc:sqlserver://localhost:1433;user=me;password=myPassword; -libjars=./sqljdbc4.jar
I'm getting the following warning & error:
13/10/25 18:38:13 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
13/10/25 18:38:13 INFO manager.SqlManager: Using default fetchSize of 1000
13/10/25 18:38:13 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.jdbc.sqlserver.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.jdbc.sqlserver.SQLServerDriver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:727)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.SqlManager.listTables(SqlManager.java:418)
at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
UPDATE
I changed the command line to reflect the comments below, I get the same error:
sqoop list-databases -libjars=<ABSOLUTE_PATH>/jars/sqljdbc4.jar --connect jdbc:sqlserver://localhost:1433;user=me;password=password
13/10/28 17:00:33 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
java.lang.RuntimeException: Could not load db driver class: com.microsoft.sqlserver.jdbc.SQLServerDriver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:727)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
When I look at the listing of sqljdbc4.jar, I do see the class in that path... Is it possible that libjars option isn't doing what I think it is supposed to do?
In vast majority of cases using parameter --driver is not required and even more will lead to an undesirable behaviour. I would strongly recommend dropping this argument entirely from your command line. Check out Connectors vs Drivers blog post for more details.
Also in addition you are specifying a nonexistent JDBC Driver class. The correct one is:
com.microsoft.sqlserver.jdbc.SQLServerDriver
You can see it in the official docs, whereas you are specifying
com.microsoft.jdbc.sqlserver.SQLServerDriver
Notice the different order of jdbc and sqlserver packages. This is one of the reasons why it's recommended to not use the --driver option at all.
You need to put sqljdbc4.jar in $SQOOP_HOME/lib and also add sqoop-1.4.4.jar or whatever version you are using along with sqljdbc4.jar to $HADOOP_HOME/lib.
I'm using Hadoop-2.2.0, so i put it inside $HADOOP_HOME/share/hadoop/common/lib directory, and use the following command to do the import:
export HCAT_HOME=/home/Kuntal/BIG_DATA/hive-0.12.0/hcatalog
(sometimes HCatlog of Hive needs to be exported or set.)
./sqoop-import --connect "jdbc:sqlserver://IP\INSTANCE;port=1433;username=USERNAME;password=PASSWORD;database=DATABASE_NAME" --table TABLE_NAME --target-dir hdfs://localhost:50315/sqoop --m 1
Sometimes you have to specify the port, otherwise default works. Hope you find it useful.
According to this sqoop documentation, generic options like -libjars must come before tool-specific options:
Generic Hadoop command-line arguments:
(must preceed any tool-specific arguments)
...
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
Recently came across this same problem. Even though documentation says that it will pick up additional jar files. The problem is I believe propagated from Hadoop jar command line option. -libjars is not reliable option to set additional jar file path.
Instead, choose HADOOP_CLASSPATH option to setup additional jar files.
In my case, I had multiple different versions of driver JAR, but using -libjars was not correctly picking up file for me.
To resolve this, I specified
export HADOOP_CLASSPATH=/$SQOOP_HOME/<path_to_driver>.jar
This makes sure that correct JAR file gets loaded.
I want to do something with the HypergraphDB in Java. I'm not very familiar with this language and IDE Netbeans.
I followed steps describing the HGDB installation:Link to tutorial
Now I want to write an example in Java (using NetbeansIDE).
So I created new project, added refferences to the downloaded libraries. OK, now I'm able to import packages with HGDB. The only thing I have in my main method is HyperGraph HG = new HyperGraph(dbLocation);. This code compiles. But get following runtime error:
run:
checkpoint kbytes:0
checkpoint minutes:0
java.lang.IllegalStateException: There is 1 existing transaction opened against the Environment.
Aborting open transactions ...
aborting <Transaction id="28">
at com.sleepycat.je.Environment.close(Environment.java:383)
at org.hypergraphdb.storage.bje.BJEStorageImplementation.shutdown(BJEStorageImplementation.java:178)
at org.hypergraphdb.HGStore.close(HGStore.java:355)
at org.hypergraphdb.HyperGraph.open(HyperGraph.java:392)
at org.hypergraphdb.HyperGraph.open(HyperGraph.java:213)
at org.hypergraphdb.HyperGraph.<init>(HyperGraph.java:200)
at hgdbtest.HgdbTest.main(HgdbTest.java:16)
Exception in thread "main" org.hypergraphdb.HGException: java.lang.NoSuchMethodError: org.hypergraphdb.storage.bje.LinkBinding.objectToEntry(Ljava/lang/Object;Lcom/sleepycat/je/DatabaseEntry;)V
at org.hypergraphdb.HyperGraph.open(HyperGraph.java:395)
at org.hypergraphdb.HyperGraph.open(HyperGraph.java:213)
at org.hypergraphdb.HyperGraph.<init>(HyperGraph.java:200)
at hgdbtest.HgdbTest.main(HgdbTest.java:16)
Caused by: java.lang.NoSuchMethodError: org.hypergraphdb.storage.bje.LinkBinding.objectToEntry(Ljava/lang/Object;Lcom/sleepycat/je/DatabaseEntry;)V
at org.hypergraphdb.storage.bje.BJEStorageImplementation.store(BJEStorageImplementation.java:234)
at org.hypergraphdb.HGStore.store(HGStore.java:119)
at org.hypergraphdb.HGTypeSystem.addPrimitiveTypeToStore(HGTypeSystem.java:185)
at org.hypergraphdb.HGTypeSystem.bootstrap(HGTypeSystem.java:234)
at org.hypergraphdb.HyperGraph.open(HyperGraph.java:355)
... 3 more
Java Result: 1
BUILD SUCCESSFUL (total time: 4 seconds)
I guess it's caused by the native API of the BerkeleyDB.
According to the tutorial, in project options I supply the -Djava.library.path=$HGDB_ROOT/lib/native/$PLATFORM argument on Run card as a VM Option. Instead of the system variables I use full-path to the libraries.
My OS in Windows7. I'm not sure about using slashes or backslashes or double-backslashes as a path-separator.?
So a simply question is: how have I to set up the IDE for using the HypergraphDB?
Thank you for advices!
Judging from the stack trace, you are using the latest version of HyperGraphDB. In that version, the native API is not used. Rather, it's Java only libraries. You have to include the BerkeleyDB Java Edition 5.0.34 jar in your runtime classpath. You can get it either from the Oracle http://www.oracle.com/technetwork/products/berkeleydb/downloads/index.html or from the HyperGraphDB Maven repository: http://hypergraphdb.org/maven/org/hypergraphdb/hgbdbje/1.2/hgbdbje-1.2.jar
I had the same issue. I fixed it (by accident) by fixing the order of the libraries on which you're HyperGraphDB application relies on. The library je-[version].jar has to be ABOVE (and hence to be referenced before) db-[version].jar
Please see the passage "RMI Registry Issue" of this article for the background on Java Update 1.6.0_29 first.
If I understand correctly (I'm german), the update introduces a bug in the rmiregistry which fails to work with the file: pattern in the codebase.
I.E. the following won't work any more with 1.6.0_29:
-Djava.rmi.server.codebase="file:myproject/bin/ ..."
We are currently using the feature of having a codebase with file: syntax. Does anyone know a workaround for making this work?
Note: No, we do not want to start a local webserver or ftp server.
Update:
On Naming.bind this exception is thrown:
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.lang.ClassNotFoundException: access to class loader denied
at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:400)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:248)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
I had the same problem, and can confirm that downgrading JDK to earlier version solves the problem. I know, it's not a solution you're looking for, but at least it makes it to work.
Take running in windows as an example:
Step 1. In C:\Users\Jimmy.java.policy (create it if not exist), append below content:
grant { permission java.security.AllPermission; };
Of course "C:\Users\Jimmy\" is the user home, please change to your home accordingly.
Adding AllPermission is just for quick resolving your issue. you'd better config a more accurate FilePermission here.
Step 2. Start rmiregistry:
C:\JDK\bin>rmiregistry -J-Djava.rmi.server.codebase=file://C:/workspaces/MyLab/target/classes/
(Please note codebase must ended with "/")
Step 3. Run your server and client program.
References:
http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/codebase.html
http://docs.oracle.com/javase/7/docs/technotes/guides/security/spec/security-spec.doc3.html
http://docs.oracle.com/javase/7/docs/technotes/guides/rmi/enhancements-7.html
It looks like there is no workaround because it is a bug, so wait for the fix
See details at
https://bugzilla.redhat.com/show_bug.cgi?id=751203
Code fix
http://icedtea.classpath.org/hg/icedtea6/rev/67df573b0734
If you do not need dynamic code downloading (in which case you can use ftp codebase) the solution is simply to set CLASSPATH environment variable to the path to your jar file:
Windows:
set CLASSPATH="path_to_jarfile"
Linux (batch):
CLASSPATH="path_to_jarfile"
export CLASSPATH
Best place to do it is in some script that invokes the RMI server.
Setting class path in the command line (-cp option) when starting RMI server does not help because it does not affect rmiregistry classpath!
If you start the rmiregistry in the working directory of your project, it works.
So essentially working directory of your project and current directory for rmiregistry should be same.
I recently encountered this issue as well. I can confirm that when using the file: protocol the rmiregistry must either:
be started in the root of the directory containing the shared classes; or
set the classpath to point to the shared classes or shared class jar; or
use a protocol other than file:// (I set up ngnix and served the jar from that).
Maybe not what you want, but you could resolve this with classpath rather than codebase. The client JVM will work fine if you add the required classes to its classpath. If you are using the file: URL scheme, then the classes must already be available on the localhost.
I had the same problem but I couldn't change the JDK version. Turns out you can solve it by running/starting the rmiregistry from the same directory as your code base, which in my case was target/classes. So cd project/target/classes and then run rmiregistry &