I've installed Apache Zookeeper and Apache Kafka (kafka_2.13-2.7.0) on my macOS Big Sur 11.1. Where the Zookeeper is running fine but not the Kafka.
Java version:
openjdk 14.0.2 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.2+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode, sharing)
I'm following this get started page. The zookeeper runs fine but then when I try to run Kafka it gives me the following error.
MacBook-Pro:kafka saad$ bin/kafka-server-start.sh config/server.properties
[2021-04-16 06:19:56,021] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-04-16 06:19:56,450] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-04-16 06:19:56,465] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT:localhost:9092': Unable to parse PLAINTEXT:localhost:9092 to a broker endpoint
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:274)
at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:1703)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1778)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1756)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1312)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:68)
at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse PLAINTEXT:localhost:9092 to a broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:57)
at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:271)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:99)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:86)
at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:38)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:271)
... 7 more
I tried to get an idea from the error but as I'm not a java guy I was unable to understand
I was able to solve the issue with Kafka issue thread. As I'm not running the servers on docker also made it difficult to find a solution.
The problem was basically using a strict localhost IP address (I'm not clearly sure about this). Just by changing the strict ip server.properties in kafka/config directory.
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://localhost:9092
Related
I am trying to run pyspark on jupyter(via anaconda) in windows.Facing the below mentioned error while trying to create a SparkSession.
Exception: Java gateway process exited before sending its port number
Error snapshot 1
Error snapshot 2
I even tried adding JAVA_HOME,SPARK_HOME and HADOOP_HOME path into environment variable:
JAVA_HOME: C:\Java\jdk-11.0.16.1
SPARK_HOME:C:\Spark\spark-3.1.3-bin-hadoop3.2
HADOOP_HOME:C:\Spark\spark-3.1.3-bin-hadoop3.2
Even after this I am facing the same issue.
PS: My pyspark version is 3.3.1 and python version is 3.8.6.
As per spark documentation, the string for setting master should be "local[*]" or "local[N]" for only using N cores. If you leave out the master setting, it defaults to "local[*]".
After several attempts, I finally figured out the issue. It was because the windows firewall had blocked java that caused this error. Once I gave the access permission the error was rectified!
I'm try to run 2 Spring Boot applications within the same Pod (essentially one is a reverse proxy for the other one - a small implementation of the sidecar pattern) and I've that one of the containers can't start. In fact, it crashes with the following error:
Starting the Java application using /opt/jboss/container/java/run/run-java.sh ...
INFO exec java -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -javaagent:/usr/share/java/prometheus-jmx-exporter/jmx_prometheus_javaagent.jar=9779:/opt/jboss/container/prometheus/etc/jmx-exporter-config.yaml -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/app-backend-1.0.0-SNAPSHOT.jar
Could not start Jolokia agent: java.lang.IllegalStateException: Cannot open keystore for https communication: java.net.BindException: Address already in use
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:555)
at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
at jdk.httpserver/sun.net.httpserver.ServerImpl.bind(ServerImpl.java:134)
at jdk.httpserver/sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f3cf9ef7e91, pid=1, tid=154
#
# JRE version: OpenJDK Runtime Environment 21.9 (17.0.1+12) (build 17.0.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM 21.9 (17.0.1+12-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x21e91] abort+0x203
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /deployments/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
# https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-17-openjdk
#
I've the impression the issue is caused by the way the plugin I'm using builds the final image, but I'm not sure.
I suspect that both the containers use the same port (since the images are built in the same way) for the Jolokia and Prometheus JMX agents, but I didn't find a way to disable them.
Does somebody have some ideas?
when running 2 containers in one pod, the containers recognize the pod as their localhost environment. as a result, if both containers try to bind the same port, they will interfere each other.
try to configure one of the containers to bind another port. this should resolve the error above. :)
edit: I checked the provided link to the github issue you opened for the plugin. jkube and spring boot docs both list several different ways to provide build- and runtime configs. I don't think this is a bug but a configuration issue that can be solved with the options the framework and plugin provides.
I propose you dig the docs further to check how to configure the ports for the processes that are started in both containers via springs application properties, ENVs or something like that. Afterwards, you can check how to provide these values during build/runtime using jkube generated manifests.
docs I've found:
https://www.eclipse.org/jkube/docs/kubernetes-maven-plugin#_configuration
https://docs.spring.io/spring-boot/docs/2.1.1.RELEASE/reference/html/production-ready-jmx.html
https://blog.kubernauts.io/https-blog-kubernauts-io-monitoring-java-spring-boot-applications-with-prometheus-part-2-b883063486b5
In the end it was effectively a bug in JKube. See https://github.com/eclipse/jkube/issues/1294#event-6132755248
We recently migrated our systems to open-jdk from oracle jdk. When I tried deploying my war application, Facing some issues with the same.
C:\Softwares\apache-tomcat-8.0.24\bin\catalina.bat run
[2019-06-14 05:19:12,449] Artifact expresso:war exploded: Waiting for server connection to start artifact deployment...
Using CATALINA_BASE: "C:\Users\Rohit.Bansal.IntelliJIdea2019.1\system\tomcat\Unnamed_expresso_3"
Using CATALINA_HOME: "C:\Softwares\apache-tomcat-8.0.24"
Using CATALINA_TMPDIR: "C:\Users\Rohit.Bansal.IntelliJIdea2019.1\system\tomcat\Unnamed_expresso_3\temp"
Using JRE_HOME: "C:\Program Files\AdoptOpenJDK\jdk-8.0.212.04-openj9"
Using CLASSPATH: "C:\Softwares\apache-tomcat-8.0.24\bin\bootstrap.jar;C:\Softwares\apache-tomcat-8.0.24\bin\tomcat-juli.jar"
port = 81
Error: Password file read access must be restricted:
C:\Users\Rohit.Bansal\.IntelliJIdea2019.1\system\tomcat\Unnamed_expresso_3\jmxremote.password
sun.management.AgentConfigurationError
at sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:577)
at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:426)
at sun.management.Agent.startAgent(Agent.java:262)
at sun.management.Agent.startAgent(Agent.java:452)
at java.lang.System.startSNMPAgent(Native Method)
at java.lang.Thread.completeInitialization(Thread.java:168)
at java.lang.J9VMInternals.completeInitialization(J9VMInternals.java:74)
Exception in thread "main" java/lang/RuntimeException: sun.management.AgentConfigurationError
at sun/management/Agent.error (Agent.java:526)
at sun/management/Agent.startAgent (Agent.java:269)
at sun/management/Agent.startAgent (Agent.java:452)
at java/lang/System.startSNMPAgent (NativeMethod:4294967295)
at java/lang/Thread.completeInitialization (Thread.java:168)
at java/lang/J9VMInternals.completeInitialization (J9VMInternals.java:74)
Disconnected from server
I encountered the same issue here: debug a Webapp in Tomcat from IntelliJ, after updating AdoptOpenJDK to jdk-8.0.212.04-openj9 under Windows (in my case Windows 7).
My solution was to add the following parameter to the VM Options of my run config:
-Dcom.sun.management.jmxremote.authenticate=false
Now it works as before.
It was somewhat difficult for me to find references to the problem, as in my case the error message was in German (cited here to augment the chances to get a result for other people searching for the German version):
Fehler: Lesezugriff auf Kennwortdatei muss eingeschränkt werden
Obviously, the chances to get meaningful results are somewhat lower. I have then changed the language to en by setting the VM Option -Duser.language=en.
I haven't found a central location to set these parameters, but what probably comes closest is to set the parameters in the template of the local tomcat run configuration.
I'm getting this error when I try to run
RACK_ENV=test be rails test test/system/service_provider_map_test.rb —seed 48088
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file logs/gc.log due to No such file or directory
The tests are then timing out with this error
Timeout while waiting for cluster status [green] and [1] nodes
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:551:in sleep': execution expired (Timeout::Error) from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:551:inblock (2 levels) in __wait_for_status' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:542:in loop' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:542:inblock in __wait_for_status' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:108:in
timeout' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:541:in__wait_for_status' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:364:in wait_for_green' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:280:instart' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/elasticsearch-extensions-0.0.26/lib/elasticsearch/extensions/test/cluster.rb:52:in
start' from
/Users/patrickward/Desktop/esh/SDR/test/support/elasticsearch_helpers.rb:38:in
start_elasticsearch' from
/Users/patrickward/Desktop/esh/SDR/test/application_system_test_case.rb:12:in
<class:ApplicationSystemTestCase>' from
/Users/patrickward/Desktop/esh/SDR/test/application_system_test_case.rb:6:in
' from
/Users/patrickward/Desktop/esh/SDR/test/system/service_provider_map_test.rb:2:in
require' from
/Users/patrickward/Desktop/esh/SDR/test/system/service_provider_map_test.rb:2:in
' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.1.2/lib/rails/test_unit/test_requirer.rb:14:in
require' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.1.2/lib/rails/test_unit/test_requirer.rb:14:in
block in require_files' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.1.2/lib/rails/test_unit/test_requirer.rb:13:in
each' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.1.2/lib/rails/test_unit/test_requirer.rb:13:in
require_files' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.1.2/lib/rails/test_unit/minitest_plugin.rb:96:in plugin_rails_init' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.11.3/lib/minitest.rb:81:in
block in init_plugins' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.11.3/lib/minitest.rb:79:in
each' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.11.3/lib/minitest.rb:79:in
init_plugins' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.11.3/lib/minitest.rb:130:in
run' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/railties-5.1.2/lib/rails/test_unit/minitest_plugin.rb:77:inrun' from
/Users/patrickward/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/minitest-5.11.3/lib/minitest.rb:63:in
`block in autorun'
For context: I'm running on a Mac OS, any help would be greatly appreciated
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file logs/gc.log due to No such file or directory
is not an error it is a warning and misleading. The real error message is
Timeout while waiting for cluster status [green] and [1] nodes
elasticsearch-ruby library has a test code which is starting a cluster itself.
and it comes from this test. https://github.com/elastic/elasticsearch-ruby/blob/6.x/elasticsearch-extensions/lib/elasticsearch/extensions/test/cluster.rb Line : 579
message = "\nTimeout while waiting for cluster status [#{status}]"
message += " and [#{arguments[:number_of_nodes]}] nodes" if arguments[:number_of_nodes]
__log message.ansi(:red, :bold)
However there was a bug which is pointing your problem.
https://github.com/elastic/elasticsearch-ruby/issues/371
You may need to upgrade your elasticsearch-ruby version to newer one
Could someone help me figure out my mistake? Thank you in advance :) I'm trying to set up the Command Line Tool (CLT) on my Mac OS X Yosemite and I'm getting error messages.
My problem seems similar to the one in the link below, but not identical; I have changed "http" to "https" in the murk.properties file after installing the CLT.
getBalance in Amazon Turk gives error
CODE: This is what I entered in Terminal (initially thinking that my problem was Java location):
$ export MTURK_CMD_HOME=/Applications/aws-mturk-clt-1.3.1
$ java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
$ which java
/usr/bin/java
$ export JAVA_HOME=/usr
$ cd /Applications/aws-mturk-clt-1.3.1/bin/
$ ./getBalance.sh
ERROR: This is an excerpt of the error message I received
Unable to find a $JAVA_HOME at "/usr", continuing with system-provided Java...
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: HTTPS hostname invalid: expected '176.32.98.23', received 'mechanicalturk.amazonaws.com'
Retrying request
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: HTTPS hostname invalid: expected '176.32.98.23', received 'mechanicalturk.amazonaws.com'
Retrying request
I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: HTTPS hostname invalid: expected '176.32.98.23', received 'mechanicalturk.amazonaws.com'
Retrying request
An error occurred while fetching your balance: javax.net.ssl.SSLPeerUnverifiedException: HTTPS hostname invalid: expected '176.32.98.23', received 'mechanicalturk.amazonaws.com'
com.amazonaws.mturk.service.exception.InternalServiceException: javax.net.ssl.SSLPeerUnverifiedException: HTTPS hostname invalid: expected '176.32.98.23', received 'mechanicalturk.amazonaws.com'
at com.amazonaws.mturk.service.axis.AWSService.executeRequestMessage(AWSService.java:243)
at com.amazonaws.mturk.filter.FinalFilter.execute(FinalFilter.java:38)
at com.amazonaws.mturk.filter.Filter.passMessage(Filter.java:56)
at com.amazonaws.mturk.filter.ErrorProcessingFilter.execute(ErrorProcessingFilter.java:46)
at com.amazonaws.mturk.filter.Filter.passMessage(Filter.java:56)
at com.amazonaws.mturk.filter.RetryFilter.execute(RetryFilter.java:115)
at com.amazonaws.mturk.filter.Filter.passMessage(Filter.java:56)
at com.amazonaws.mturk.util.CLTExceptionFilter.sendMessage(CLTExceptionFilter.java:77)
at com.amazonaws.mturk.util.CLTExceptionFilter.execute(CLTExceptionFilter.java:62)
at com.amazonaws.mturk.service.axis.FilteredAWSService.executeRequests(FilteredAWSService.java:172)
at com.amazonaws.mturk.service.axis.FilteredAWSService.executeRequest(FilteredAWSService.java:152)
at com.amazonaws.mturk.service.axis.FilteredAWSService.executeRequest(FilteredAWSService.java:116)
at com.amazonaws.mturk.service.axis.RequesterServiceRaw.getAccountBalance(RequesterServiceRaw.java:1193)
at com.amazonaws.mturk.service.axis.RequesterService.getAccountBalance(RequesterService.java:922)
at com.amazonaws.mturk.cmd.GetBalance.getBalance(GetBalance.java:50)
at com.amazonaws.mturk.cmd.GetBalance.runCommand(GetBalance.java:41)
at com.amazonaws.mturk.cmd.AbstractCmd.run(AbstractCmd.java:148)
at com.amazonaws.mturk.cmd.GetBalance.main(GetBalance.java:28)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: HTTPS hostname invalid: expected '176.32.98.23', received 'mechanicalturk.amazonaws.com'
at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
The error message continues a bit. So, what do you think? Thanks again for looking at this. - Soon
mturk.properties file
# -------------------
# ADVANCED PROPERTIES
# -------------------
#
# If you want to test your solution in the Amazon Mechanical Turk Developers Sandbox (http://sandbox.mturk.com)
# use the service_url defined below:
#service_url=https://mechanicalturk.sandbox.amazonaws.com/?Service=AWSMechanicalTurkRequester
# If you want to have your solution work against the Amazon Mechnical Turk Production site (http://www.mturk.com)
# use the service_url defined below:
service_url=https://mechanicalturk.amazonaws.com/?Service=AWSMechanicalTurkRequester
# The settings below should only be modified under special circumstances.
# You should not need to adjust these values.
retriable_errors=Server.ServiceUnavailable,503
retry_attempts=6
retry_delay_millis=500'
Guan, have you tried with an earlier version of the JDK (e.g., JDK-1.5?). I realize it's much older, but I'm curious if it's related to using the CLT on JDK 1.8. Just an idea.
Additionally, it would help if we could see the turk.properties file (please don't share your access keys or secret key) to make sure the endpoints are well-formed. Thanks!