Develop Reference

Issues running apache Kafka on Mac

I've installed Apache Zookeeper and Apache Kafka (kafka_2.13-2.7.0) on my macOS Big Sur 11.1. Where the Zookeeper is running fine but not the Kafka.
Java version:
openjdk 14.0.2 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.2+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode, sharing)
I'm following this get started page. The zookeeper runs fine but then when I try to run Kafka it gives me the following error.
MacBook-Pro:kafka saad$ bin/kafka-server-start.sh config/server.properties
[2021-04-16 06:19:56,021] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-04-16 06:19:56,450] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-04-16 06:19:56,465] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT:localhost:9092': Unable to parse PLAINTEXT:localhost:9092 to a broker endpoint
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:274)
at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:1703)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1778)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1756)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1312)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:68)
at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse PLAINTEXT:localhost:9092 to a broker endpoint
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:57)
at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:271)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:99)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:86)
at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:38)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:271)
... 7 more
I tried to get an idea from the error but as I'm not a java guy I was unable to understand

I was able to solve the issue with Kafka issue thread. As I'm not running the servers on docker also made it difficult to find a solution.
The problem was basically using a strict localhost IP address (I'm not clearly sure about this). Just by changing the strict ip server.properties in kafka/config directory.
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://localhost:9092

Jmeter plugin execution throws ArrayIndexOutOfBoundsException

I've looked for answers to this problem, but couldn't find any over the internet. Maybe someone here ran into this issue before.
I have a CentOS machine with Jmeter 3.1. On this machine everything works fine. I created a new VM, and copied the jmeter directory to the new machine with everything set up. Test execution works fine, but when I try to use any of the plugins (cmdrunner-2.0.jar or by JMeterPluginsCMD.sh)
I get an
Exception back, with not much information what's wrong:
[root#box bin]# java -jar "/opt/apache-jmeter-3.1/lib/cmdrunner-2.0.jar" -n --tool Reporter --input-jtl "/tmp/data.csv" --plugin-type SynthesisReport --generate-csv "/tmp/report.csv"
WARN 2017-10-22 12:41:57.204 [jmeter.u] (): Exception 'null' occurred when fetching String property:'sampleresult.default.encoding', defaulting to:ISO-8859-1
WARN 2017-10-22 12:41:57.224 [jmeter.u] (): Exception 'null' occurred when fetching String property:'jmeterPlugin.prefixPlugins'
INFO 2017-10-22 12:41:57.224 [kg.apc.j] (): Using JMeterPluginsCMD v. N/A
INFO 2017-10-22 12:41:57.229 [jmeter.u] (): Setting Locale to en_US
INFO 2017-10-22 12:41:57.238 [kg.apc.j] (): Loading user properties from: /opt/apache-jmeter-3.1/bin/user.properties
INFO 2017-10-22 12:41:57.238 [kg.apc.j] (): Loading system properties from: /opt/apache-jmeter-3.1/bin/system.properties
ERROR: java.lang.ArrayIndexOutOfBoundsException: 0
*** Problem's technical details go below ***
Home directory was detected as: /opt/apache-jmeter-3.1/lib
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75)
at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
...
...
ERROR: java.lang.ArrayIndexOutOfBoundsException: 0
That's all I get. The only differences between the two machines is:
Working platform:
kernel 3.10.0-514.26.2.el7.x86_64
java (build 1.8.0_131-b12)
Not working:
kernel 3.10.0-693.2.2.el7.x86_64,
java (build 1.8.0_144-b01)
There are no environment variables missing.
Any suggestions are more than welcomed

This is a java bug on this particular platform:
https://bugzilla.redhat.com/show_bug.cgi?id=1484079

I can't believe it - yum update java solved my issue. The thing is, I just updated java last week..

CentOS Java application error with SELinux

I have a CentOS box hosting a Drupal 7 site. I've attempted to run a Java application called Tika on it, to index files using Apache Solr search.
I keep running into an issue only when SELinux is enabled:
extract using tika: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1ed9000000, 2555904, 1) failed; error='Permission denied' (errno=13)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2555904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/jvm-2356/hs_error.log
This does not happen if I disable selinux. If I run the command from SSH, it works fine -- but not in browser. This is the command it is running:
java '-Dfile.encoding=UTF8' -cp '/var/www/drupal/sites/all/modules/contrib/apachesolr_attachments/tika' -jar '/var/www/drupal/sites/all/modules/contrib/apachesolr_attachments/tika/tika-app-1.11.jar' -t '/var/www/drupal/sites/all/modules/contrib/apachesolr_attachments/tests/test-tika.pdf'
Here is the log from SELinux at /var/log/audit/audit.log:
type=AVC msg=audit(1454636072.494:3351): avc: denied { execmem } for pid=11285 comm="java" scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:system_r:httpd_t:s0 tclass=process
type=SYSCALL msg=audit(1454636072.494:3351): arch=c000003e syscall=9 success=no exit=-13 a0=7fdfe5000000 a1=270000 a2=7 a3=32 items=0 ppid=2377 pid=11285 auid=506 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=1 comm="java" exe="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.95.x86_64/jre/bin/java" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
Is there a way I can run this with SELinux enabled? I do not know the policy name of Tika (or should I use Java?) so I'm unsure where to go from here...

This worked for me...
I have tika at /var/apache-tika/tika-app-1.14.jar
setsebool -P httpd_execmem 1
chcon -t httpd_exec_t /var/apache-tika/tika-app-1.14.jar
Using the sealert tools (https://wiki.centos.org/HowTos/SELinux) helped track down the correct selinux type.

All of your context messages reference httpd_t, so I would run
/usr/sbin/getsebool -a | grep httpd
And experiment with enabling properties that show as off. It's been a while since I ran a database-backed website (Drupal, WordPress, etc.) on CentOS, but as I recall, these two were required to be enabled:
httpd_can_network_connect
httpd_can_network_connect_db
to enable a property with persistence, run
setsebool -P httpd_can_network_connect on
etc.

The booleans you're looking for are:
httpd_execmem
httpd_read_user_content
How to find:
audit2why -i /var/log/audit/audit.log will tell you this.
Part of package: policycoreutils-python-utils

HBase: MiniDFSCluster.java Fails in Certain Environments

I'm working on some code to access HBase and I am writing unit tests that create a MiniDFSCluster as part of the test setup.
(defn test-config [& options]
(let [testing-utility (HBaseTestingUtility.)]
(.startMiniCluster testing-utility 1)
(let [config (.getConfiguration testing-utility)]
(if (not= options nil)
(doseq [[key value] options]
(.set config key value)))
config)))
;; For those who don't read Clojure, lines 2 and 3 cause
;; the failure and are equivalent to the following Java
;;
;; HBaseTestingUtility testingUtility = new HBaseTestingUtility();
;; testingUtility.startMiniCluster(1); // blows up on Linux but not Mac OSX
This runs fine on Mac OSX with Java HotSpot:
$ java -version
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-457, mixed mode)
$ lein test
lein test hbase.config-test
lein test hbase.table-test
2013-07-12 17:44:13.488 java[27384:1203] Unable to load realm info from SCDynamicStore
Starting DataNode 0 with dfs.data.dir: /Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/fe0199fd-0168-48d9-98ce-b4a5e62d3257/dfscluster_bbad1095-58d1-4571-ba12-4d4f1c24203f/dfs/data/data1,/Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/fe0199fd-0168-48d9-98ce-b4a5e62d3257/dfscluster_bbad1095-58d1-4571-ba12-4d4f1c24203f/dfs/data/data2
Cluster is active
Ran 11 tests containing 14 assertions.
0 failures, 0 errors.
But when this is run in a Linux environment, the following error occurs:
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
actual: java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes (MiniDFSCluster.java:426)
org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster (HBaseTestingUtility.java:444)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:612)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:568)
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:555)
I filed a travis-ci ticket, since this first manifested itself there and I thought it might be due to their environment.
https://github.com/travis-ci/travis-ci/issues/1240
However, after discussion with travis support I was able to reproduce the error on CentOS. I tried both the Sun JDK and the OpenJDK on Linux and both produced the same error. Whats going on here? Is this a trivial configuration problem? Perhaps something not set in the Linux ENV that is set in Mac OSX's ENV?
If you would like to run the tests, please clone the repo
https://github.com/mobiusinversion/hbase
And run lein test. Help is greatly appreciated!
Update:
Filed this HBASE Jira ticket
https://issues.apache.org/jira/browse/HBASE-8944

Short answer: set "umask 022" prior to running the tests.
Long answer: This is a common environmental issue with running MiniDFSCluster from Hadoop 1.x, releases, which HBaseTestingUtility uses internally. It has been effectively fixed in Hadoop 0.22+ (including 2.0+, but not 1.x at the moment).
The underlying problem is https://issues.apache.org/jira/browse/HDFS-2556.
When the MiniDFSCluster starts up, it creates the temporary storage directories to use for the datanode processes (configured as "dfs.data.dir"). These will be created with your currently set umask. When each datanode starts up, it checks that the directories configured in "dfs.data.dir" both exist and that the directory permissions match the expected value (set as "dfs.datanode.data.dir.perm"). If the directories permissions do not match the expected value ("755" by default), then the datanode process exits.
By default, in Hadoop 1.x, this value is set to "755", so if you set your umask to "022", the data directories will wind up with the correct permissions. If however, the permissions do not match the expected value, the datanode will abort and you will see errors like the following in the test log file:
WARN [main] datanode.DataNode(1577): Invalid directory in dfs.data.dir: Incorrect permission for /.../dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x
In later versions of Hadoop, the datanode will attempt to change the directory permissions to the expected value if they do not match. Only if this operation fails will the datanode abort. HDFS-2556 proposes backporting this change to the 1.x releases, but has not yet been fixed.

Java Appcrash Problem

when I start my Java-Program, I get this error (using Xuggle-library with Webcam):
Problem signature:
Problem Event Name: APPCRASH
Application Name: java.exe
Application Version: 6.0.260.3
Application Timestamp: 4dc11607
Fault Module Name: libxuggle-ferry-4.dll
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 4b9493c5
Exception Code: c0000005
Exception Offset: 0000822a
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1031
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
I have already found out, that c0000005 is an access violation.
Is there a way to solve such errors? Or how can I trace this error (Stacktrace or something, or maybe some tools for this)? Does Java create logfiles for such errors. Where can I find them? My Java-Program does not throw any exception. I just get this appcrash and "Java Result: -xxxxxxx" (where xxxxxx is a large number).
Thank you.

Java didn't crash, your dll did, and it took the Java process down with it. You won't see an exception. Typically when the JVM is about to die it creates a file that lists what was going on at the time, and it writes a message to stdout with a path to this file. Here's the beginning of one I got not too long back:
A fatal error has been detected by the Java Runtime Environment:
EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x1002ff3a,
pid=4312, tid=2120
JRE version: 6.0_24-b07
Java VM: Java HotSpot(TM) Client VM (19.1-b02 mixed mode, sharing windows-x86 )
Problematic frame:
C [namechangedtoprotecttheguilty.dll+0x2ff3a]
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.
--------------- T H R E A D ---------------
and it goes on to list what's in the registers, what the stack frames look like, what's on the heap, environment variable values, etc.

The developer of libxuggle-ferry-4.dll should fix the library.