I have installed Kuberenetes (minikube for Windows 10) and added Spark there using helm:
.\helm.exe install --name spark-test stable/spark
Then I exposed Spark master port 7077 using
.\kubectl.exe expose deployment spark-test-master --port=7070 --name=spark-master-ext --type=NodePort
For example, my UI runs on http://<MINIKUBE_IP>:31905/ and spark master is exposed to <MINIKUBE_IP>:32473. In order to check, I do:
.\minikube-windows-amd64.exe service spark-master-ext
But when I do in Java:
SparkConf conf = new SparkConf().setMaster("spark://192.168.1.168:32473").setAppName("Data Extractor");
I've got:
18/03/19 13:57:29 WARN AppClient$ClientEndpoint: Could not connect to 192.168.1.168:32473: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster#192.168.1.168:32473]
18/03/19 13:57:29 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster#192.168.1.168:32473] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://sparkMaster#192.168.1.168:32473]] Caused by: [Connection refused: no further information: /192.168.1.168:32473]
18/03/19 13:57:29 WARN AppClient$ClientEndpoint: Failed to connect to master 192.168.1.168:32473
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkMaster#192.168.1.168:32473/), Path(/user/Master)]
Any ideas, how to run Java Spark jobs on Spark running in Minikube?
It looks like Helm chart for Spark is really outdated (1.5.1), so I have installed 2.3.0 locally and it runs without any issues. Case closed, sorry :)
Related
I'm trying to run sample cordapp-example code by cloning from Github repository using:
git clone https://github.com/corda/samples
I followed all the steps as mentioned in the documentation for running the application from IntelliJ.
[ERROR] 14:54:18,832 [main] internal.DriverDSLImpl. - Driver shutting down because of exception [errorCode=1crywct, moreInformationAt=https://errors.corda.net/OS/4.3/1crywct]
java.lang.IllegalStateException: Unable to start notaries. A required port might be bound already.
at net.corda.testing.node.internal.DriverDSLImpl.start(DriverDSLImpl.kt:390) ~[corda-node-driver-4.3.jar:?]
at net.corda.testing.node.internal.DriverDSLImplKt.genericDriver(DriverDSLImpl.kt:1048) ~[corda-node-driver-4.3.jar:?]
at net.corda.testing.driver.Driver.driver(Driver.kt:185) ~[corda-node-driver-4.3.jar:?]
at com.example.test.NodeDriverKt.main(NodeDriver.kt:15) ~[test/:?]
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[?:1.8.0_231]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_231]
at net.corda.core.internal.concurrent.CordaFutureImpl.get(CordaFutureImpl.kt) ~[corda-core-4.3.jar:?]
at net.corda.core.internal.concurrent.CordaFutureImplKt.get(CordaFutureImpl.kt:172) ~[corda-core-4.3.jar:?]
at net.corda.core.utilities.KotlinUtilsKt.getOrThrow(KotlinUtils.kt:134) ~[corda-core-4.3.jar:?]
at net.corda.testing.node.internal.DriverDSLImpl.start(DriverDSLImpl.kt:379) ~[corda-node-driver-4.3.jar:?]
... 3 more
[WARN] 14:54:19,251 [driver-pool-thread-0] internal.InternalTestUtils. - Been polling address localhost:10040 to bind for 60 seconds...
[INFO] 14:54:57,702 [driver-pool-thread-0] internal.RPCClient. - Startup took 10512 msec
[INFO] 14:54:58,015 [driver-pool-thread-1] internal.DriverDSLImpl. - Node handle is ready. NodeInfo: NodeInfo(addresses=[localhost:10040], legalIdentitiesAndCerts=[O=Notary Service, L=Zurich, C=CH], platformVersion=5, serial=1578902078740), WebAddress: localhost:10043
Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
I see this description:
[ERROR] 14:13:50,501 [main] internal.DriverDSLImpl. - Driver shutting down because of exception [errorCode=1crywct, moreInformationAt=https://errors.corda.net/OS/4.3/1crywct]
Has anyone else seen this before and are there any recommendations to fixing the issue or clues as to how we can debug it further?
From the error message, I'd see this:
"Unable to start notaries. A required port might be bound already"
which means that the port(s) used by notary is being used by other application, or, most likely being used by another running notary.
How to fix?
Open node.conf in your notary folder, and check the ports listed, such as
address : "localhost:10006"
then check the ports usage in you system, either kill the running process or change the port in notary node.conf and run again.
Good luck.
Task state of a connector is getting failed with error:
org.apache.kafka.connect.errors.ConnectException: java.lang.NoClassDefFoundError
I am running kafka connect cluster in distributed mode and I am using kafka(0.10.0.2.5) connect of ambari deployment.
I gave debezium mysql connector path using export CLASSPATH=/path to connector/.
Loaded connector configuration into Kafka Connect using the following command:
curl -i -X POST -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/ \
-d '{
"name": "MYSQL_CONNECTOR",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "10.224.21.36",
"database.port": "3306",
"database.user": "root",
"database.password": "shobhna",
"database.server.id": "1",
"database.server.name": "demo",
"database.history.kafka.bootstrap.servers": "slnxhadoop04.noid.in:6669",
"database.history.kafka.topic": "dbhistory.demo" ,
"include.schema.changes": "true"
}
}'
Now after checking connector status, I am getting error:
- {"name":"MYSQL_CONNECTOR","connector":{"state":"RUNNING","worker_id":"172.26.177.115:8083"},
"tasks":[{"state":"FAILED","trace":"org.apache.kafka.connect.errors.ConnectException:
java.lang.NoClassDefFoundError:
org/apache/kafka/clients/admin/AdminClient\n\tat
io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:218)\n\tat
io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:45)\n\tat
org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:137)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)\n\tat
java.util.concurrent.Executors$RunnableAdapter.cal(Executors.java:511)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
java.lang.Thread.run(Thread.java:745)\nCaused by:
java.lang.NoClassDefFoundError:
org/apache/kafka/clients/admin/AdminClient\n\tat
io.debezium.relational.history.KafkaDatabaseHistory.initializeStorage(KafkaDatabaseHistory.java:336)\n\tat
io.debezium.connector.mysql.MySqlSchema.intializeHistoryStorage(MySqlSchema.java:260)\n\tat
io.debezium.connector.mysql.MySqlTaskContext.initializeHistoryStorage(MySqlTaskContext.java:194)\n\tat
io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:126)\n\t...
9 more\nCaused by: java.lang.ClassNotFoundException:
org.apache.kafka.clients.admin.AdminClient \n\tat
java.net.URLClassLoader.findClass(URLClassLoader.java:381)\n\tat
java.lang.ClassLoader.loadClass(ClassLoader.java:424)\n\tat
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)\n\tat
java.lang.ClassLoader.loadClass(ClassLoader.java:357)\n\t
It can't find a builtin Kafka class, not your Connector
NoClassDefFoundError:
org/apache/kafka/clients/admin/AdminClient
...
i am using kafka(0.10.0.2.5)
Make sure you're running 1) a Connect Server version that matches your Kafka broker 2) using a Connector that uses code for that version of Connect
For example, AdminClient only exists in Kafka 0.11+.
In the recent HDP releases, you get Kafka 1.1 (different than 0.11), and this is the version that the latest Debezium is built and tested against https://debezium.io/docs/releases/
Debezium needs the AdminClient to make and register topic information, so I'm not sure if it'll work on old version such as 0.10
Its stated in the Kafka wiki that newer versions of Connect Server can communicate with old brokers, but the protocol used by the Connector classes is up for debate.
I am using Spring Tool Suite. I check out demo projects in STS. It works fine without any problem before I installed updates in STS today or I installed Oracle SQL Developer recently.
The steps to reproduce my bugļ¼
In STS, in "File" -> "New" -> "Import Spring Getting Started Content", then check out "Building a RESTful Web Service" this project.
https://spring.io/guides/gs/rest-service/ I go to my project folder, type 'mvnw spring-root:run' (I am using Windows). Then got following error.
I do not if this bug related to I installed two updated in STS today or I installed Oracle SQL Developer recently.
Here is the error:
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:1.5.6.RELEASE:run (default-cli) on project gs-rest-service: An exception occurred while running. null: InvocationTargetException: Connector configured to listen on port 8080 failed to start -> [Help 1]
Then, I checked out the solution in here:
https://stackoverflow.com/a/27416379/8229192
It works after I kill the task which uses the port 8080.
c:\>netstat -ano | find "8080"
TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING 3088
TCP [::]:8080 [::]:0 LISTENING 3088
c:\>taskkill /F /PID 3088
SUCCESS: The process with PID 1196 has been terminated.
My questions are:
Why will I have port conflict? Because I installed Oracle SQL Developer? How can I know exactly which software is using port 8080 also?
I want to know if I kill the task (A) which uses that port 8080, will it cause an issue when this task (A) run later?
I have checked out other projects (like: https://spring.io/guides/gs/scheduling-tasks/). I did not need to kill the task which is also using port 8080, I just directly run "mvnw spring-boot:run". It works and it does not have port number 8080 conflict. Why? Why some have port 8080 conflict, why some are not? This is very confused me. Thanks.
Oracle-XE, OracleXETNSListener service, uses port 8080 to serve its
Application Express.
You kill OracleXETNSListener service, it has no problem at all because you
use SQL Developer not Application Express. Or you can disable its auto start configuration.
Spring Boot's project, serves web server, uses port 8080 by default, you can run with different port Spring Boot - how to configure port, and https://spring.io/guides/gs/scheduling-tasks is not web serving project so it doesn't use any port.
hope this can help you
I am trying to submit my spark-mongo code jar through spark on windows.I am using spark in standalone mode. I have configured spark master and two workers on same machine. I want to execute my jar with one master and two workers.I am trying to execute following command: spark-submit --master spark://localhost:7077 --deploy-mode cluster --executor-memory 5G --class spark.mongohadoop.testing3 G:\sparkmon1.jar
I am facing following error:
Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/28 17:09:13 INFO RestSubmissionClient: Submitting a request to launch an application in spark://192.168.242.1:7077.
17/02/28 17:09:24 WARN RestSubmissionClient: Unable to connect to server spark://192.168.242.1:7077.
Warning: Master endpoint spark://192.168.242.1:7077 was not a REST server. Falling back to legacy submission gateway instead.
17/02/28 17:09:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/02/28 17:09:32 ERROR ClientEndpoint: Exception from cluster was: java.lang.NullPointerException
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)
at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:474)
at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:154)
at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:83
I have already set winutil path in env.
why I am getting this error and what is the solution?
I encountered the same error on Linux but with me it was coming when the driver was getting initiated from a particular machine in my cluster, if request to launch driver was going to any other machine in cluster, ten it was working fine. So, in my cased seemed to be as an environmental issue.
I then checked the code at org.apache.hadoop.util.Shell$ShellCommandExecutor class and got that it is trying to run a command but before that it tries lo run "bash" for that machine. I observed that my bash was responding slow.made some changes in bashrc and restarted my cluster.
Now its working fine.
I am trying to setup a JMeter server/client instance on a remote database server. I have extracted JMeter on the server, copied the MySQL java connector into $JMETER_HOME/lib and started the JMeter-server listener. On the client (my local desktop), I have pointed the JMeterclient to the remote server (via the remote_hosts entry in $JMETER_HOME/bin/jmeter.properties). Whenever I try to run a MySQL load job on the client (Run -> Remote Start -> ), I get the following error:
Response message: java.sql.SQLException: No suitable driver found for jdbc:mysql://1.2.3.4/database
And this is on the server side:
2013/04/10 12:44:43 WARN - jmeter.protocol.jdbc.config.DataSourceElement: Could not return Connection java.sql.SQLException: No suitable driver found for jdbc:mysql://1.2.3.4/dbname at java.sql.DriverManager.getConnection(DriverManager.java:602) at java.sql.DriverManager.getConnection(DriverManager.java:185) at org.apache.avalon.excalibur.datasource.JdbcConnectionFactory.newInstance(JdbcConnectionFactory.java:185) at org.apache.avalon.excalibur.pool.ResourceLimitingPool.newPoolable(ResourceLimitingPool.java:672) at org.apache.avalon.excalibur.pool.ValidatedResourceLimitingPool.newPoolable(ValidatedResourceLimitingPool.java:178) at org.apache.avalon.excalibur.datasource.ResourceLimitingJdbcConnectionPool.newPoolable(ResourceLimitingJdbcConnectionPool.java:123) at org.apache.avalon.excalibur.pool.ResourceLimitingPool.get(ResourceLimitingPool.java:402) at org.apache.avalon.excalibur.pool.ValidatedResourceLimitingPool.get(ValidatedResourceLimitingPool.java:130) at org.apache.avalon.excalibur.datasource.ResourceLimitingJdbcDataSource.getConnection(ResourceLimitingJdbcDataSource.java:222) at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:286) at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:146) at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:85) at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) at java.lang.Thread.run(Thread.java:662)
Does anyone have any idea why this wouldn't work even though the Java connector is in the jmeter lib dir?
So it turns out that the Percona MySQL version that we are running required a specific version of the connector. Dropped that in and it all worked out.