Develop Reference

Kafka Error reading field 'correlation_id': java.nio.BufferUnderflowException

Getting this kafka exception on consumer :
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'correlation_id': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
at org.apache.kafka.common.requests.ResponseHeader.parse(ResponseHeader.java:53)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:435)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:134)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorKnown(AbstractCoordinator.java:184)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:886)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853)
No client-server version mismatch.

Be sure your client connect to a real Kafka port !
this specific error happens while parsing (one of?) the first header field of the expected kafka message, as shown by the invocation of ResponseHeader.java in stack-trace.
So this can occurs if you target a listening port that has nothing to do with kafka server.
just a 1 minute check !
Otherwise, you should check for a client-server version mismatch.

For me, I was having trouble with unit test failure with above exception. When I inspected the port(9092) being used on local machine, it was bound to already running process, worth checking if there is process for Kafka running locally. If you are sure you are not expecting it to be running, kill it by finding its pid.
(Don't try on production though :P )
lsof -i:9092
kill -9 <PID_FROM_ABOVE_IF_ANY>

kafka + zookeeper remote = error

I am trying to install a kafka & zookeeper instance on a remote server. I only need 1 node of each actually because i only want to provide remote kafka for test purposes.
Kafka and Zookeeper are running from the Apache Kafka tarball you can find there (v0.0.9), inside a Docker image.
Trying to consume / produce using the provided scripts. And trying to produce using own java application. Everythinf is working fine if Kafka & ZK are installed on the local server.
Here is the error I get while trying to produce :
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
No partition metadata for topic RSS due to kafka.common.LeaderNotAvailableException}] for topic [RSS]: class kafka.common.LeaderNotAvailableException
Kafka properties tested
First :
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=localhost:<PORT>
Second:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
Third:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Last:
borker.id=0
port=9092
host.name=</etc/host name>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Here is my "/etc/hosts"
127.0.0.1 kafka kafka
127.0.0.1 localhost
I followed the Getting Started, which if I understood is a localhost / signle server configurations. I cannot understand what I have to do to get this work with remote calls...
Thanks for your help !
EDIT 1
host.name=localhost
advertised.host.name=politik.cm-cloud.fr
Seems to allow a local consumer (on the server) and producer. But if we want to do the same from a remote server we get
[2015-12-09 12:44:10,826] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host

The error does not look like connectivity problem with Zookeeper / Kafka.
Just follow the instruction in "quickstart" from http://kafka.apache.org/
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
Additionally the error indicates there is no partition info i.e topic not yet created . Try creating topics first and then try to produce/consume because when producing to a non existent topic kafka will create the topic based on auto.create.topics.enable in server.properties but remotely it is better to create topics rathen than relying on auto create

Connect from Hive to HDFS (JSON Files)

I have already built up a hadoop cluster with apache flume to import twitter data, it works fine.
Now I wanna start analytics with apache hive on the twitter data. On the web I found following example from cloudera.
https://github.com/cloudera/cdh-twitter-example
But now, by creating the table, hive returns the following error message:
java.net.URISyntaxException: Relative path in absolute URI: text:STRING, Query returned non-zero code: 1,
cause: java.net.URISyntaxException: Relative path in absolute URI: text:STRING,
On web i didn't found something about this (only by starting hive), maybe someone here can help me!
Thanks!

Okay, first problem are solved by myself. forgot a semicolon on the command. sorry for this.
But now I get another error message after start jobs over hive. All Query Jobs on Hive abort after some seconds. In the Log I found only this:
2015-03-25 14:47:40,680 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1427105751169_0006_01_000030
Any Ideas here?

Inconsistency in Hbase table[Region not deployed on any region server]

In a small HBase cluster, all the slave nodes got restarted. When I started HBase services, one of the tables (test) became inconsistent.
In HDFS some blocks were missing(hbase blocks). So it was in safe mode. I gave safemode -leave command.
Then HBase table (test) became inconsistent.
I performed below mentioned actions:
I executed "hbase hbck" several times. 2 inconsistencies found for table "test".
ERROR: Region { meta=>test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862., hdfs => hdfs://master:9000/hbase/test/4c213a47bba83c47075f21fec7c6d862, deployed => } not deployed on any region server.
hbase hbck -fixMeta -fixAssignments HBaseFsckRepair: Region still in transition, waiting for it to become assigned:
{NAME => 'test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862.', STARTKEY => '1m\x00\x03\x1B\x15', ENDKEY => '', ENCODED => 4c213a47bba83c47075f21fec7c6d862,}
hbase hbck -repair HBaseFsckRepair: Region still in transition, waiting for it to become assigned:
{NAME => 'test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862.', STARTKEY => '1m\x00\x03\x1B\x15', ENDKEY => '', ENCODED => 4c213a47bba83c47075f21fec7c6d862,}
I checked datanode logs in parallel.
Logs:
org.apache.hadoop.hdfs.server.datanode.DataNode: opReadBlock BP-1015188871-192.168.1.11-1391187113543:blk_7616957984716737802_27846 received exception java.io.EOFException
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.1.12, storageID=DS-831971799-192.168.1.12-50010-1391193910800, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-7f99a9de-258c-493c-9db0-46b9e84b4c12;nsid=1286773982;c=0):Got exception while serving BP-1015188871-192.168.1.11-1391187113543:blk_7616957984716737802_27846 to /192.168.1.12:36127
Checked Namenode logs
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ubuntu (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C 2014-02-28 14:13:15,738
INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from
10.10.242.31:42149: error: java.io.FileNotFoundException: File does not exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C java.io.FileNotFoundException: File does not exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1301)
But, I am able to browse and download the file from HDFS. How can recover the data?
How can I make the "test" table consistent?

In HBase 2.0 (and possibly in previous versions), "not deployed on any region server" is typically solved by getting the region assigned.
Authenticate if you're on a secured cluster. You are on a secured cluster, aren't you? ;)
kinit [keytab] [principal]
Run HBase check to see which regions specifically are unassigned
hbase hbck -details
If you see an error like this:
ERROR: Region {
meta => my.tablename,,1500001112222.abcdef123456789abcdef12345678912.,
hdfs => hdfs://cluster/apps/hbase/data/data/default/my.tablename/abcdef123456789abcdef12345678912,
deployed => ,
replicaId => 0
} not deployed on any region server.
(the key being "not deployed on any region server"), then you should assign the region. This, it turns out, is pretty simple. Proceed to step 4.
Open an hbase shell
hbase shell
Assign the region by passing the encoded regionname to the assign method. As noted in the help documentation, this should not be called without the previous due diligence as this command will do a force reassign. The docs say, and I caution: for experts only.
hbase(main):001:0> assign 'abcdef123456789abcdef12345678912'
Double-check your work by running hbase check for your table that had the unassigned regions.
hbase hbck my.tablename
If you did everything correctly and if there's no underlying HDFS issue, you should see this message near the bottom of the hbck output:
0 inconsistencies detected.
Status: OK

In Hbase 2.0.2 version there is no repair option to recover inconsistencies.
Run hbase hbck command.
If the error mesaage look like mentioned below:
ERROR: Region { meta => EMP_NMAE,\x02\x00\x00\x00\x00,1571419090798.054b393c37a80563ae1aa60f29e3e4df., hdfs => hdfs://node1:8020/apps/hbase/data/data/LEVEL_RESULT/054b393c37a80563ae1aa60f29e3e4df, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => TABLE_3,\x02174\x0011100383\x00496\x001,1571324271429.6959c7157693956825be65676ced605c., hdfs => hdfs://node1:8020/apps/hbase/data/data/TABLE_NAME/6959c7157693956825be65676ced605c, deployed => , replicaId => 0 } not deployed on any region server.
copy this error inconsistancy to an file and pull the alphanumeric value by using the below command.
If our inconsistancy count is less we can take value manually if the number is more it would be hectic to retrive the entire value. so use the below command to narrow down to alphanemeric alone which can be copied and put in hbase shell at a stretch.
cat inconsistant.out|awk -F'.' '{print $2}'
Open hbase hbase shell and assign these consistancy manually. LIKE BELOW:
assign '054b393c37a80563ae1aa60f29e3e4df'
assign '6959c7157693956825be65676ced605c'
assign '7058dfe0da0699865a5b63be9d3799ab'
assign 'd25529539bae49eb078c7d0ca6ce84e4'
assign 'e4ad94f58e310a771a0f5a1eade884cc'
once the assigning is completed run the hbase hbck command again

I had the same problem. It turned out there were regions overlappings. How I fixed:
Try to assign region which is not deployed in hbase shell: assign 'Abcd...'
Check HBase Master log for ERROR AssingmentManager [something like that: Trying to assign region {ENCODED => Abcd..., NAME => ..., ts=1591351130943, server=server1,6020,1581641930622}]
Turn off region server on server1
Run hbase hbck -repair my_table
Repeat for every undeployed region
Or you can just restart hbase and run 'hbase hbck -repair'

Error when trying to connect to Jacorb naming service

I'm hoping to get some help with this weird problem. We're running the Jacorb name server and I have a simple client that I'm using to try to connect and do awesome CORBA voodoo. The name server is running, but when I try to start my java app, I get a "Connection failure" error (org.omg.CORBA.COMM_FAILURE, minor code 201, "caused by java.net.ConnectionException: Connection refused: connect").
Here's the weird part. The error reports that it's trying to connect using the default port 900, but I'm passing in an argument to try to override the port number of the name service to match the one being used by the name server. My java command is like this:
java -classpath . HelloClient -Djava.endorsed.dirs="bla bla bla" <br>
-Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB
-Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton
-DORBInitRef.NameService=corbaloc::localhost:2809/StandardNS/NameServer-POA/_root
I also tried the parameters without the first capital D (I've seen it both ways and I don't know the difference).
Now, if I put in -ORBInitialPort 2809, then the client does appear to try to connect, but then I get a corba.OBJECT_NOT_EXIST error.
I could use any help or advice anyone has.

Connection Refused. This sounds like a firewall/program not running issue.
try a telnet <machine> 2809. You should get a "Connected to "
and not a refusal, if everything is running/enabled correctly.
I'm running on a UNIX client so the paths use UNIX style.
jacORB installed properly ? e.g. get the nameservice entry from the
orb.properties file (in ${JAVA_HOME}/jre/lib/
I use "ORBInitRef.NameService=corbaloc::localhost:2809/NameServer"
as "NameServer" is used on the production name server and not the other
string of "Standard...."
The other changes in the properties files are setting the paths to UNIX
style (i.e. e:\NS_Ref -> /tmp/NS_Ref)
jacorb.naming.ior_filename=/tmp/NS_Ref
1a. Setting the http:// in the properties file didn't seem to do anything
in regards to resolving on the client side.
1b. NOTE: start ns with:
ns -DOAPort=2809
Log will show:
2010-05-27 10:00:47.777 FINE Created socket listener on 0.0.0.0/0.0.0.0:2809
2010-05-27 10:00:47.777 FINE Using port 2809
Running:
$ lsof | grep 2809
java 27529 jbsymolo 15u IPv6 693300 TCP *:2809 (LISTEN)
$ lsof -Pnl +M -i6
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
Naming_Se 9678 1000 7u IPv6 45779 TCP *:51148 (LISTEN)
java 27959 1000 15u IPv6 696092 TCP *:2809 (LISTEN)
Not Running: (shows nothing)
ns when started will log where it reads the properties from and it shouldn't
throw any errors. If it does your properties files have issues.
VM arguments. The -D is used to set system properties. Any Java code can
then access any property so defined via System.getProperty(). Even though
I've also seen the "non-D" used, I've been using the D.
-DORBInitRef.NameService=corbaloc::localhost:2809/NameService
-Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB
-Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton
When running the client in Eclipse, I see the following in the Console:
May 27, 2010 10:01:06 AM org.jacorb.config.JacORBConfiguration init
INFO: base configuration loaded from file /usr/lib/java/jdk1.6.0_18/jre/lib/orb.properties
...
2010-05-27 10:01:09.836 FINE Trying to connect to 127.0.0.1:2809 with timeout=90000.
2010-05-27 10:01:09.844 INFO Connected to 127.0.0.1:2809 from local port 45745
2010-05-27 10:01:09.846 FINE wrote 12 bytes to 127.0.0.1:2809
...
Skipping lots of other read/write traffic

I can't be sure without seeing the rest of the code, but I'm pretty sure you need to change the InitRef string to be:
-DORBInitRef.NameService=corbaloc::localhost:2809
When your client connects, this should give you the root naming context for the naming service and then you can traverse the NameContext tree to get to your desired server object.