Develop Reference

Spring Cloud message serialisation problem with Kafka running in docker container

I'm trying to write a test for my Spring Cloud service while it runs against Kafka and Schema Registry which run inside Docker containers.
Kafka and Schema Registry communicate with each other via a docker network, and have ports that are exposed on the host. The service I am testing is running on the host - it is able to communicate with both the docker kafka broker and docker schema registry. I am starting it up from a JUnit test which is annotated as shown below.
#ExtendWith(SpringExtension.class)
#SpringBootTest
#EnableAutoConfiguration(exclude = TestSupportBinderAutoConfiguration.class)
#AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
public class MyTest {
...
}
My service spins up and is able to write a message to the Kafka broker running inside the Docker container, however when my service is started using the various Spring / JUnit test annotations, there appears to be something different about the way the message it writes is serialized compared to when my service runs in 'production mode' (i.e. if I run it using using java -jar com.xyz.MyService).
The message needs to be written in Avro format, so I've configured the binder in application.yml as
my-topic:
destination: my-topic
contentType: application/*+avro
producer:
useNativeEncoding: true
When attempting to consume the message that my service has written, AbstractKafkaAvroDeserializer blows up, complaining that it was unable to marshal it into a completely unrelated Avro type:
{"logger_name":"org.apache.kafka.streams.errors.LogAndFailExceptionHandler","message":"Exception caught during Deserialization, taskId: 0_0, topic: my-topic, partition: 0, offset: 1","stack_trace":"org.apache.kafka.common.errors.SerializationException: Could not find class com.xyz.SomeOtherMessageType specified in writer's schema whilst finding reader's schema for a SpecificRecord.
at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.getSpecificReaderSchema(AbstractKafkaAvroDeserializer.java:265)
at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.getReaderSchema(AbstractKafkaAvroDeserializer.java:247)
at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.getDatumReader(AbstractKafkaAvroDeserializer.java:194)
...
This does not happen if my service runs in 'production mode'.
I think therefore that some setting is being applied to my service when I spin it up in 'test mode', which changes the way messages are encoded or serialized.
Can anyone suggest some things I can try to resolve this?
Update 1
So, it turns out that the messages looks pretty much identical when they are written to the topic and then read back (UUIDs are random for each test run):
Written to topic by service running in 'test mode':
Address 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
------- -------- -------- -------- -------- ----------------
000000: 00000000 01483335 63616366 62642D30 .....H35cacfbd-0
000010: 3165642D 34653564 2D613936 652D6665 1ed-4e5d-a96e-fe
000020: 30626339 65313033 34664832 35313436 0bc9e1034fH25146
000030: 6237392D 66643334 2D346430 322D6261 b79-fd34-4d02-ba
000040: 37362D36 61396535 62623861 31343448 76-6a9e5bb8a144H
000050: 30653364 30326536 2D383732 372D3466 0e3d02e6-8727-4f
000060: 64312D38 3730662D 33646633 35353166 d1-870f-3df3551f
000070: 37343861 084D7220 54064D72 730A4A69 748a.Mr T.Mrs.Ji
000080: 6D6D790A 57686974 6514536E 6F772068 mmy.White.Snow h
000090: 6F757365 00000012 4C697665 72706F6F ouse....Liverpoo
0000A0: 6C0C4C4C 32335252 0E456E67 6C616E64 l.XXXXXX.England
0000B0: 16303735 31323334 35363738 021E4D72 .XXXXXXXXXXX..Mr
0000C0: 20542773 20427573 696E6573 73483737 T's BusinessH77
0000D0: 32383064 36352D36 3633362D 34376565 280d65-6636-47ee
0000E0: 2D393864 302D6361 36646531 32373838 -98d0-ca6de12788
0000F0: 63610000 ca..
Written to topic by service running in 'production mode':
Address 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
------- -------- -------- -------- -------- ----------------
000000: 00000000 57483433 64343264 61372D30 ....WH43d42da7-0
000010: 6533392D 34646665 2D383966 362D6531 e39-4dfe-89f6-e1
000020: 37363036 34383730 61344833 38663864 76064870a4H38f8d
000030: 3561342D 65386532 2D346134 372D6235 5a4-e8e2-4a47-b5
000040: 30662D37 31623435 36653837 33393348 0f-71b456e87393H
000050: 63666463 33653964 2D303362 612D3464 cfdc3e9d-03ba-4d
000060: 62372D62 3034622D 31393137 37323634 b7-b04b-19177264
000070: 36623665 084D7220 54064D72 730A4A69 6b6e.Mr T.Mrs.Ji
000080: 6D6D790A 57686974 6514536E 6F772068 mmy.White.Snow h
000090: 6F757365 00000012 4C697665 72706F6F ouse....Liverpoo
0000A0: 6C0C4C4C 32335252 0E456E67 6C616E64 l.XXXXXX.England
0000B0: 16303735 31323334 35363738 021E4D72 .XXXXXXXXXXX..Mr
0000C0: 20542773 20427573 696E6573 73486161 T's BusinessHaa
0000D0: 35326636 34662D36 6131642D 34393030 52f64f-6a1d-4900
0000E0: 2D616537 612D3432 33326333 65613938 -ae7a-4232c3ea98
0000F0: 38330000 83..

Testcontainers Kafka module runs a single node Kafka installation. It doesn't spin up a Schema Registry. Which I suspect might be a problem for Avro serialization.
You can add it manually to the tests. Testcontainers allows to run any Docker image programmatically with a simple API call:
var schemaRegistry = new GenericContainer(DockerImageName.parse("confluentcp/cp-schema-registry:version"));
I don't know for certain, but you probably need to connect Kafka and the schema registry, which you can do with the Network, see the Advanced networking chapter in the docs.
Unfortunately, I don't have a good example to refer to.
You can also look at something like this: https://github.com/kreuzwerker/kafka-consumer-testing.
They mock schema registry url so there's no separate schema registry container.

Failed to delete the state directory in IDE for Kafka Stream Application

I am developing a simple Kafka Stream application which extracting messages from a topic and put it into another topic after transformation. I am using Intelij for my development.
When I debug/run this application, it works perfect if my IDE and the Kafka Server sitting in the SAME machine
(i.e. with the BOOTSTRAP_SERVERS_CONFIG = localhost:9092 and
SCHEMA_REGISTRY_URL_CONFIG = localhost:8081)
However, when I try to use another machine to do the development
(i.e. with the BOOTSTRAP_SERVERS_CONFIG = XXX.XXX.XXX:9092 and
SCHEMA_REGISTRY_URL_CONFIG = XXX.XXX.XXX:8081 where XXX.XXX.XXX is the
ip address of my Kafka),
the debug process run without problem at the 1st time. However, when I run 2nd time after resetting the offset, I received the following error:
ERROR stream-thread [main] Failed to delete the state directory. (org.apache.kafka.streams.processor.internals.StateDirectory:297)
java.nio.file.DirectoryNotEmptyException: \tmp\kafka-streams\my_application_id\0_0
Exception in thread "main" org.apache.kafka.streams.errors.StreamsException: java.nio.file.DirectoryNotEmptyException:
If I changed my_application_id as my_application_id2, and run it, it works again at the 1st time but receiving error again if I run it again.
I have the following code in my last sentence in my application:
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
Any advice how to solve this problem?
UPDATE:
I have reviewed the state directory created in my development machine (Windows Platform) and if I delete these directory manually before running 2nd time, no error found. I have tried to run my IDE as Administrator because I think this could be something about the permission on the folder. However, this doesn't help.
Full stack for reference:
INFO Kafka version : 1.1.0 (org.apache.kafka.common.utils.AppInfoParser:109)
INFO Kafka commitId : fdcf75ea326b8e07 (org.apache.kafka.common.utils.AppInfoParser:110)
INFO stream-thread [main] Deleting state directory 0_0 for task 0_0 as user calling cleanup. (org.apache.kafka.streams.processor.internals.StateDirectory:281)
Disconnected from the target VM, address: '127.0.0.1:16552', transport: 'socket'
Exception in thread "main" org.apache.kafka.streams.errors.StreamsException: java.nio.file.DirectoryNotEmptyException: C:\workspace\bennychan\kafka-streams\my_application_001\0_0
at org.apache.kafka.streams.processor.internals.StateDirectory.clean(StateDirectory.java:231)
at org.apache.kafka.streams.KafkaStreams.cleanUp(KafkaStreams.java:931)
at com.macroviewhk.financialreport.simpleStream.start(simpleStream.java:60)
at com.macroviewhk.financialreport.simpleStream.main(simpleStream.java:45)
Caused by: java.nio.file.DirectoryNotEmptyException: C:\workspace\bennychan\kafka-streams\my_application_001\0_0
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at org.apache.kafka.common.utils.Utils$1.postVisitDirectory(Utils.java:651)
at org.apache.kafka.common.utils.Utils$1.postVisitDirectory(Utils.java:634)
at java.nio.file.Files.walkFileTree(Files.java:2688)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.apache.kafka.common.utils.Utils.delete(Utils.java:634)
ERROR stream-thread [main] Failed to delete the state directory. (org.apache.kafka.streams.processor.internals.StateDirectory:297)
at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:287)
java.nio.file.DirectoryNotEmptyException: C:\workspace\bennychan\kafka-streams\my_application_001\0_0
at org.apache.kafka.streams.processor.internals.StateDirectory.clean(StateDirectory.java:228)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:266)
... 3 more
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at org.apache.kafka.common.utils.Utils$1.postVisitDirectory(Utils.java:651)
at org.apache.kafka.common.utils.Utils$1.postVisitDirectory(Utils.java:634)
at java.nio.file.Files.walkFileTree(Files.java:2688)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at org.apache.kafka.common.utils.Utils.delete(Utils.java:634)
at org.apache.kafka.streams.processor.internals.StateDirectory.cleanRemovedTasks(StateDirectory.java:287)
at org.apache.kafka.streams.processor.internals.StateDirectory.clean(StateDirectory.java:228)
at org.apache.kafka.streams.KafkaStreams.cleanUp(KafkaStreams.java:931)
at com.macroviewhk.financialreport.simpleStream.start(simpleStream.java:60)
at com.macroviewhk.financialreport.simpleStream.main(simpleStream.java:45)
UPDATE 2 :
After another detailed check, the line below throwing IOException
Files.walkFileTree(file.toPath(), new SimpleFileVisitor<Path>() {
This line is located at kafka-clients-1.1.0.jar org.apache.kafka.common.utilsUtils.class
May be this is the problem with Windows system (sorry that I am not an experienced JAVA programmer).

For googlers..
I'm currently using this Scala code for helping windows guys to handle deletion of state store.
if (System.getProperty("os.name").toLowerCase.contains("windows")) {
logger.info("WINDOWS OS MODE - Cleanup state store.")
try {
FileUtils.deleteDirectory(new File("/tmp/kafka-streams/" + config.getProperty("application.id")))
FileUtils.forceMkdir(new File("/tmp/kafka-streams/" + config.getProperty("application.id")))
} catch {
case e: Exception => logger.error(e.toString)
}
}
else {
streams.cleanUp()
}

I agree with #ideano1 that is seems to be related to https://issues.apache.org/jira/browse/KAFKA-6647 -- what you can try is, to explicitly call KafkaStreams#cleanUp() between tests. It's unclear why there are issues at Window-OS. Atm, all testing happens on Linux.

This is what we've implemented that works on Windows. This is written in Kotlin.
Version used : kafka-streams-test-utils:2.3.0.
The key is to catch the exception. The tests will pass as long as you catch the exception raised by testDriver.close()even if you don't delete the directory. However, cleaning up the directory makes your unit tests independent and repeatable.
val directory = "test"
#BeforeEach
fun setup(){
//other code omitted for setting the props
props.setProperty(StreamsConfig.STATE_DIR_CONFIG,directory)
}
#AfterEach
fun tearDown(){
try{
testDriver.close()
}catch(exception: Exception){
FileUtils.deleteDirectory(File(directory)) //there is a bug on Windows that does not delete the state directory properly. In order for the test to pass, the directory must be deleted manually
}
}

For tests (but not only if you afford so) one could use an IN_MEMORY("in-memory") store for each KTable created (directly or indirectly, by e.g. aggregations); this avoids the creation of any directory such that the error no longer occurs.

Cassandra NoHostAvailableException: All host(s) tried for query failed in Production

We have 10 Cassandra nodes in production running Cassandra-2.1.8. We recently upgraded to 2.1.8 version. Previously we were using only 3 nodes running Cassandra-2.1.2. First we upgraded the initial 3 nodes from 2.1.2 to 2.1.8 (following the procedure as described in Upgrading Cassandra). Then we added 7 more nodes running Cassandra-2.1.8 in cluster. Then we started our client programs. For first few hours everything worked fine, but after few hours, we saw some errors in client program logs like
Thread-0 [29/07/15 17:41:23.356] ERROR com.cleartrail.entityprofiling.engine.InterpretationWriter - Error:com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: [/172.50.33.161:9041, /172.50.33.162:9041, /172.50.33.95:9041, /172.50.33.96:9041, /172.50.33.165:9041, /172.50.33.166:9041, /172.50.33.163:9041, /172.50.33.164:9041, /172.50.33.42:9041, /172.50.33.167:9041] - use getErrors() for details)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65)
at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:259)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:175)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
at com.cleartrail.entityprofiling.engine.InterpretationWriter.WriteInterpretation(InterpretationWriter.java:430)
at com.cleartrail.entityprofiling.engine.Profiler.buildProfile(Profiler.java:1042)
at com.cleartrail.messageconsumer.consumer.KafkaConsumer.run(KafkaConsumer.java:336)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: [/172.50.33.161:9041, /172.50.33.162:9041, /172.50.33.95:9041, /172.50.33.96:9041, /172.50.33.165:9041, /172.50.33.166:9041, /172.50.33.163:9041, /172.50.33.164:9041, /172.50.33.42:9041, /172.50.33.167:9041] - use getErrors() for details)
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:102)
at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:176)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Now, I double checked the Firewall (as suggested in few posts), ports, timeouts in client as well as nodes and they all are correct.
I am also not closing the connection anywhere in between. I am using batch queries with batch size of 1000 and the queries are update queries updating counters in my table with three columns
entity , twfwv , cvalue
where entity and twfwv columns are text and primary key and cvalue is counter column.
I even restarted all my nodes (because this trick helped me in my dev environment when I faced the same exception) but its not helping. Please suggest what can be the probable problem here.

My issue was resolved by checking the errors collection of NoHostAvailableException as advised by Olivier Michallat in the comments. For me it was the protocol version on the cluster configuration. Mine was null, setting it to 3 fixed the problem.

My issue was resolved by removing/using a property to set or unset the custom load balancing TokenAwarePolicy my connection was using, and relying on the default.
Specifically, I was trying to get a local spring boot app talking to a single dockerized Cassandra instance.
Cluster.Builder builder = Cluster.builder()
.addContactPoints(cassandraProperties.getHosts())
.withPort(cassandraProperties.getPort())
.withProtocolVersion(ProtocolVersion.V4)
.withRetryPolicy(new LoggingRetryPolicy(DefaultRetryPolicy.INSTANCE))
.withCredentials(cassandraProperties.getUsername(), cassandraProperties.getPassword())
.withCodecRegistry(codecRegistry);
if (loadBalanced) {
builder.withLoadBalancingPolicy(
new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(localDc).build()));
}

Testing java HBase connection

I am trying to use the HBase Java APIs to write data into HBase. I installed Hadoop/HBase through Ambari.
Here is how the configuration is currently set up:
final Configuration CONFIGURATION = HBaseConfiguration.create();
final HBaseAdmin HBASE_ADMIN;
HBASE_ADMIN = new HBaseAdmin(CONFIGURATION)
When I try to write to HBase, I check to make sure that the table exists
!HBASE_ADMIN.tableExists(tableName)
If not, create a new one. However, it appears that when attempting to check if the table exists exceptions are being thrown.
I'm wondering if I'm not correctly connected to HBase...is there any good way to verify that the configuration is correct and that I am connecting to HBase? The exception I'm getting is below.
Thanks.
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:209)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:288)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135)
at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:597)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:802)
at org.apache.hadoop.hbase.catalog.MetaReader.tableExists(MetaReader.java:359)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:287)
at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:301)
at com.business.project.hbase.HBaseMessageWriter.getTable(HBaseMessageWriter.java:40)
at com.business.project.hbase.HBaseMessageWriter.write(HBaseMessageWriter.java:59)
at com.business.project.hbase.HBaseMessageWriter.write(HBaseMessageWriter.java:54)
at com.business.project.storm.bolt.package.exampleBolt.execute(exampleBolt.java:19)
at backtype.storm.daemon.executor$fn__5697$tuple_action_fn__5699.invoke(executor.clj:659)
at backtype.storm.daemon.executor$mk_task_receiver$fn__5620.invoke(executor.clj:415)
at backtype.storm.disruptor$clojure_handler$reify__1741.onEvent(disruptor.clj:58)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at backtype.storm.daemon.executor$fn__5697$fn__5710$fn__5761.invoke(executor.clj:794)
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:465)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:269)
at org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:241)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:62)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1203)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1164)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:294)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:130)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:55)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:201)

In addition to the configuration parameters suggested by Yosr, specifying
conf.set("zookeeper.znode.parent", "VALUE")
would help resolve the issue.

The property below resolved my issue
For Hortonworks:
hconfig.set("zookeeper.znode.parent", "/hbase-unsecure")
For cloudera:
hconfig.set("zookeeper.znode.parent", "/hbase")

You can use HBaseAdmin.checkHBaseAvailable(conf);

Configuration conf = HBaseConfiguration.create();
conf.set("hbase.master", "ip_address:60000");
conf.set("hbase.zookeeper.quorum","ip_address");
conf.set("hbase.zookeeper.property.clientPort", "2181");
HBaseAdmin admin = new HBaseAdmin(conf);
boolean bool = admin.tableExists("table_name");
System.out.println( bool);
ip_address : this is the ip_adress of your hbase cluster, change your hbase zookeeper port (2181) if it is not the same on your configuration files.

Writing to HDFS from Java, getting "could only be replicated to 0 nodes instead of minReplication"

I’ve downloaded and started up Cloudera's Hadoop Demo VM for CDH4 (running Hadoop 2.0.0). I’m trying to write a Java program that will run from my windows 7 machine (The same machine/OS that the VM is running in). I have a sample program like:
public static void main(String[] args) {
try{
Configuration conf = new Configuration();
conf.addResource("config.xml");
FileSystem fs = FileSystem.get(conf);
FSDataOutputStream fdos=fs.create(new Path("/testing/file01.txt"), true);
fdos.writeBytes("Test text for the txt file");
fdos.flush();
fdos.close();
fs.close();
}catch(Exception e){
e.printStackTrace();
}
}
My config.xml file only has on property defined: fs.default.name=hdfs://CDH4_IP:8020.
When I run it I’m getting the following exception:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testing/file01.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
at org.apache.hadoop.ipc.Client.call(Client.java:1160)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:290)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1150)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1003)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463)
I’ve looked around the internet and it seem that this happens when disk space is low but that’s not the case for me when I run "hdfs dfsadmin -report" I get the following:
Configured Capacity: 25197727744 (23.47 GB)
Present Capacity: 21771988992 (20.28 GB)
DFS Remaining: 21770715136 (20.28 GB)
DFS Used: 1273856 (1.21 MB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Live datanodes:
Name: 127.0.0.1:50010 (localhost.localdomain)
Hostname: localhost.localdomain
Decommission Status : Normal
Configured Capacity: 25197727744 (23.47 GB)
DFS Used: 1273856 (1.21 MB)
Non DFS Used: 3425738752 (3.19 GB)
DFS Remaining: 21770715136 (20.28 GB)
DFS Used%: 0.01%
DFS Remaining%: 86.4%
Last contact: Fri Jan 11 17:30:56 EST 201323 EST 2013
I can also run this code just fine from with in the VM. I’m not sure what the problem is or how to fix it. This is my first time using hadoop so I’m probably missing something basic. Any ideas?
Update
The only thing I see in the logs is an exception similar to the one on get on the client:
java.io.IOException: File /testing/file01.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
I tried changing the permissions on the data directory (/var/lib/hadoop-hdfs/cache/hdfs/dfs/data) and that didn't fix it (I went so far as giving full access to everyone).
I notice that when I'm browsing the HDFS via the HUE web app I see that the folder structure was created and that the file does exist but it is empty. I tried putting the file under the default user directory by using
FSDataOutputStream fdos=fs.create(new Path("testing/file04.txt"), true);
instead of
FSDataOutputStream fdos=fs.create(new Path("/testing/file04.txt"), true);
Which makes the file path become "/user/dharris/testing/file04.txt" ('dharris' is my windows user). But that gave me the same kind of error.

I got a same problem.
In my case, a key of the problem was following error message.
There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
It means that your hdfs-client couldn't connect to your datanode with 50010 port.
As you connected to hdfs namenode, you could got a datanode's status. But, your hdfs-client would failed to connect to your datanode.
(In hdfs, a namenode manages file directories, and datanodes. If hdfs-client connect to a namnenode, it will find a target file path and address of datanode that have the data. Then hdfs-client will communicate with datanode. (You can check those datanode uri by using netstat. because, hdfs-client will be trying to communicate with datanodes using by address informed by namenode)
I solved that problem by:
opening 50010(dfs.datanode.address) port in a firewall.
adding propertiy "dfs.client.use.datanode.hostname", "true"
adding hostname to hostfile in my client PC.
I'm sorry for my poor English skill.

Go to linux VM and check the hostname and iP ADDRESS(use ifconfig cmd).
Then in the linux vm edit /etc/host file with
IPADDRESS (SPALCE) hostname
example :
192.168.110.27 clouderavm
and change the all your hadoop configuration files like
core-site.xml
hdfs-site.xml
mapred-site.xml
yarn-site.xml
change localhost or localhost.localdomain or 0.0.0.0 to your hostname
then Restart cloudera manger.
in the windows machine edit C:\Windows\System32\Drivers\etc\hosts
add one line at the end with
you vm machine ip and hostname (same as you done on the /etc/host file in the vm)
VMIPADRESS VMHOSTNAME
example :
192.168.110.27 clouderavm
then check now, it should work, for detail configuration check following VIDEO from you tube
https://www.youtube.com/watch?v=fSGpYHjGIRY

add given property in hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
and add this file also in your program
conf.addResource("hdfs-site.xml");
stop hadoop
stop-all.sh
then start
start-all.sh

I ran into the similar issue and have two pieces of information may help you.
The first thing I realized is I was using ssh tunnel to access the name node and when the client code tries to access data node it can not find the data node due to the tunnel somehow messed up the communication. I then run the client on the same box as the hadoop name node and it solved the problem. In short, non-standard network configuration confused hadoop to find the data node.
The reason I used ssh tunnel is I can't access name node remotely and I thought it was due to port restriction by admin, so I used ssh tunnel to bypass the restriction. But it turns out to be a misconfiguration of hadoop.
In core-site.xml after I changed
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
to
<value>hdfs://host_name:9000</value>
I no longer need the ssh turnnel and I can access the hdfs remotely.

Since I found many questions like this one in my search for having the exact same issue I thought I would share what finally worked for me. I found this forum post on Hortonworks: https://community.hortonworks.com/questions/16837/cannot-copy-from-local-machine-to-vm-datanode-via.html
The answer was truly understanding what calling new Configuration() means and setting the correct parameters as I needed them. In my case it was exactly the one mentioned in that post. So my working code looks like this.
try {
Configuration config = new Configuration();
config.set("dfs.client.use.datanode.hostname", "true");
Path pdFile = new Path("stgicp-" + pd);
FileSystem dFS = FileSystem.get(new URI("hdfs://" + HadoopProperties.HIVE_HOST + ":" + HadoopProperties.HDFS_DEFAULT_PORT), config,
HadoopProperties.HIVE_DEFAULT_USER);
if (dFS.exists(pdFile)) {
dFS.delete(pdFile, false);
}
FSDataOutputStream outStream = dFS.create(pdFile);
for (String sjWLR : processWLR.get(pd)) {
outStream.writeBytes(sjWLR);
}
outStream.flush();
outStream.close();
dFS.delete(pdFile, false);
dFS.close();
} catch (IOException | URISyntaxException | InterruptedException e) {
log.error("WLR file processing error: " + e.getMessage());
}

in the hadoop configuration, default replication is set to 3. check it once and change accordingly to your requirements

You can try deleting the data (dfs/data) folder manually and formating the namenode. You can then start hadoop.

From error message replication factor seems to be fine i.e.1.
It Seems datanode is properly functioning or have permission issues.
Check the permissions and check the status of datanode form the user, you are trying to run hadoop.

I had a similar problem, in my case I just emptied the following folder ${hadoop.tmp.dir}/nm-local-dir/usercache/{{hdfs_user}}/appcache/

It appears to be some issue with the FS.
Either the parameters in cross-site.xml are not matching the file it is trying to read
OR
there is some common mismatch in the path (I see there being a WINDOWS reference).
you can use cygwin tool to setup the path and place it where the datanodes and temp file locations are placed and that should sufficiently do the trick
Location : $/bin/cygpath.exe
P.S. Replication does NOT seem to be the primary issue here according to me

Here is how I create files in the HDFS:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
FileSystem hdfs = FileSystem.get(context.getConfiguration());
Path outFile=new Path("/path to store the output file");
String line1=null;
if (!hdfs.exists(outFile)){
OutputStream out = hdfs.create(outFile);
BufferedWriter br = new BufferedWriter(new OutputStreamWriter(out, "UTF-8"));
br.write("whatever data"+"\n");
br.close();
hdfs.close();
}
else{
String line2=null;
BufferedReader br1 = new BufferedReader(new InputStreamReader(hdfs.open(outFile)));
while((line2=br1.readLine())!=null){
line1=line1.concat(line2)+"\n";
}
br1.close();
hdfs.delete(outFile, true);
OutputStream out = hdfs.create(outFile);
BufferedWriter br2 = new BufferedWriter(new OutputStreamWriter(out, "UTF-8"));
br2.write(line1+"new data"+"\n");
br2.close();
hdfs.close();
}