problem connecting to Apache Kafka from Java - java

I am trying to connect to one Kafka server.
in Java, I have configured:
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
.
.
Properties properties = new Properties();
String bootstrapServers = "localhost:9092";
properties.put("bootstrap.servers", bootstrapServers);
properties.put("connections.max.idle.ms", 10000);
properties.put("request.timeout.ms", 5000);
try {
AdminClient adminClient = AdminClient.create(properties);
NewTopic newTopic = new NewTopic("testtopic", 1, (short)1);
.
.
Although there is a configuration to use localhost, I get errors:
INFO: AdminClientConfig values:
bootstrap.servers = [localhost:9092]
Jan 22, 2021 9:48:01 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo <init>
INFO: Kafka version: 2.7.0
Jan 22, 2021 9:48:01 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo <init>
INFO: Kafka commitId: 448719dc99a19793
Jan 22, 2021 9:48:01 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo <init>
INFO: Kafka startTimeMs: 1611348481265
Jan 22, 2021 9:48:03 PM org.apache.kafka.clients.NetworkClient initiateConnect
WARNING: [AdminClient clientId=adminclient-1] Error connecting to node e4f0851caf23:9092 (id: 1001 rack: null)
java.net.UnknownHostException: No such host is known (e4f0851caf23)
Is there any additional configuration that I am missing?
This is only a sample code, I am trying to create one topic.
thank you!

Thank you yuppie-flu!
I have to add:
KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9093,CLIENT://kafka:9092

Related

how overcome this error "com.mongodb.diagnostics.logging.JULLogger log"

I am working with java program connected with MongoDB when I run the program it will show an error but code is working.
MongoDB has a database called MongoDB and inside that, there is a collection called seatbooking with two columns(name,seatnumber).
This is my code:
MongoClient mongoClient = new MongoClient("localhost", 27017);
System.out.println("connection is established");
MongoDatabase mongoDatabase = mongoClient.getDatabase("MongoDB");
MongoCollection mongoCollection = mongoDatabase.getCollection("seatbooking");
Document document = new Document("name","shenal");
document.append("seatnumber",20);
mongoCollection.insertOne(document);
when I run this code my output is:
> Mar 09, 2020 12:41:36 PM
> com.mongodb.diagnostics.logging.JULLogger log INFO: Cluster created
> with settings {hosts=[localhost:27017], mode=SINGLE,
> requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
> maxWaitQueueSize=500}
> **connection is established** Mar 09, 2020 12:41:36 PM com.mongodb.diagnostics.logging.JULLogger log INFO: Cluster
> description not yet available. Waiting for 30000 ms before timing out
> Mar 09, 2020 12:41:36 PM com.mongodb.diagnostics.logging.JULLogger log
> INFO: Opened connection [connectionId{localValue:1, serverValue:309}]
> to localhost:27017 Mar 09, 2020 12:41:36 PM
> com.mongodb.diagnostics.logging.JULLogger log INFO: Monitor thread
> successfully connected to server with description
> ServerDescription{address=localhost:27017, type=STANDALONE,
> state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2,
> 2]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216,
> logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5168100} Mar 09,
> 2020 12:41:36 PM com.mongodb.diagnostics.logging.JULLogger log INFO:
> Opened connection [connectionId{localValue:2, serverValue:310}] to
> localhost:27017
The logging output doesn't show any result for the insertOne operation.
If you are using MongoDB Java Driver version prior to 4.0, there is no acknowledgement object returned. But, the version 4's insertOne method returns a InsertOneResult object. This is a new feature in 4.0 (in prior version's the method returned a void).
You can use the following code to check the insert's outcome as follows (with version 4.0):
try {
InsertOneResult insertResult = collection.insertOne(document);
System.out.println("Document inserted with ID: " + insertResult.getInsertedId());
}
catch(MongoWriteException e) {
// write failure happened, handle it here...
}

Creating topic on pubsub emulator

I started to use the pubsub emulator to test my basic implementations and ran into an issue while trying to create a new topic.
My emulator listens on localhost:8085 and if i create the topic via the api
PUT http://localhost:8085/v1/projects/testproject/topics/test
everything works fine and the topic gets created.
But if i run the following snippet nothing works as intended and no topic gets created:
TopicName topicName = TopicName.create("testproject", "test");
ChannelProvider channelProvider =
TopicAdminSettings.defaultChannelProviderBuilder()
.setEndpoint("localhost:8085")
.setCredentialsProvider(
FixedCredentialsProvider.create(NoCredentials.getInstance()))
.build();
TopicAdminClient topicClient = TopicAdminClient.create(
TopicAdminSettings.defaultBuilder().setChannelProvider(channelProvider).build());
topicClient.createTopic(topicName);
while running this the emulator logs
[pubsub] Apr 27, 2017 1:10:47 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[pubsub] INFORMATION: Adding handler(s) to newly registered Channel.
[pubsub] Apr 27, 2017 1:10:47 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFORMATION: Detected non-HTTP/2 connection.
[pubsub] Apr 27, 2017 1:10:47 PM io.gapi.emulators.netty.NotFoundHandler handleRequest
[pubsub] INFORMATION: Unknown request URI: /bad-request
[pubsub] Apr 27, 2017 1:10:47 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[pubsub] INFORMATION: Adding handler(s) to newly registered Channel.
[pubsub] Apr 27, 2017 1:10:47 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFORMATION: Detected non-HTTP/2 connection.
[pubsub] Apr 27, 2017 1:10:47 PM io.gapi.emulators.netty.NotFoundHandler handleRequest
[pubsub] INFORMATION: Unknown request URI: /bad-request
...
[pubsub] Apr 27, 2017 1:10:49 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[pubsub] INFORMATION: Adding handler(s) to newly registered Channel.
[pubsub] Apr 27, 2017 1:10:49 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFORMATION: Detected non-HTTP/2 connection.
Am i missing something on my ChannelProvider? Or didn't I configure my TopicAdminClient correctly? I don't see whats wrong since i used
this as reference.
Maybe someone can help me out with this.
Channels used to communicate with the emulator need to set the negotiationType property to NegotiationType.PLAINTEXT. That means you need to create a custom ChannelProvider. Something like the following should work:
public class PlainTextChannelProvider implements ChannelProvider {
#Override
public boolean shouldAutoClose() {
return false;
}
#Override
public boolean needsExecutor() {
return false;
}
#Override
public ManagedChannel getChannel() throws IOException {
return NettyChannelBuilder.forAddress("localhost", 8085)
.negotiationType(NegotiationType.PLAINTEXT)
.build();
}
#Override
public ManagedChannel getChannel(Executor executor) throws IOException {
return getChannel();
}
}
This post is a little old, hope this serves as an update.
The code snippet from Testing apps locally with the emulator also works. The full snippet is on GitHub if you follow the "View on GitHub" link on the linked page.
String hostport = System.getenv("PUBSUB_EMULATOR_HOST");
ManagedChannel channel = ManagedChannelBuilder.forTarget(hostport).usePlaintext().build();
try {
TransportChannelProvider channelProvider =
FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));
CredentialsProvider credentialsProvider = NoCredentialsProvider.create();
TopicAdminClient topicClient =
TopicAdminClient.create(
TopicAdminSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build());
try {
response = topicClient.createTopic(topicName);
System.out.printf("Topic %s created.\n", response);
} catch (ApiException e) {
System.out.println(e.getStatusCode().getCode());
System.out.println(e.isRetryable());
System.out.println("No topic was created.");
}
} finally {
channel.shutdown();
}

Use Google Cloud PubSub emulator from Java

I've setup Pubsub.Builder with root url pointed to local PubSub emulator (localhost:8036 at my case). It seems to be working, I see that emulator is receiving my test pushes, but gives me error 400 Bad Request.
Publish code:
auth = GoogleCredential.getApplicationDefault(HTTP_TRANSPORT, JSON_FACTORY)
.createScoped(PubsubScopes.all());
client = new Pubsub.Builder(HTTP_TRANSPORT, JSON_FACTORY, auth)
.setApplicationName "test"
.setRootUrl("http://localhost:8036/")
.build();
msg = new PubsubMessage().encodeData("{\"a\": 1, \"b\": 2}".getBytes());
req = new PublishRequest().setMessages(Arrays.asList(msg));
client.projects()
.topics()
.publish("projects/GCLOUD-DEFAULT-PROJECT/topics/test-topic", req)
.execute();
In emulator console output I see following (that's for very basic PublishRequest):
[pubsub] jan 27, 2016 9:03:20 PM com.google.cloud.pubsub.testing.v1.FakePubsubGrpcServer$2 operationComplete
[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] jan 27, 2016 9:03:20 PM com.google.cloud.pubsub.testing.v1.NettyUtil$HttpVersionDecider channelRead
[pubsub] INFO: Detected non-HTTP/2 connection.
[pubsub] jan 27, 2016 9:03:20 PM com.google.cloud.pubsub.testing.v1.NettyUtil$HttpJsonAdapter channelRead
[pubsub] INFO: Invalid input: Expect message object but got: "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000�V�M-.NLO-V���VJI"
[pubsub] com.google.protobuf.InvalidProtocolBufferException: Expect message object but got: "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000�V�M-.NLO-V���VJI"
[pubsub] at com.google.protobuf.util.JsonFormat$ParserImpl.mergeMessage(JsonFormat.java:1099)
[pubsub] at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1075)
[pubsub] at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:973)
[pubsub] at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:201)
[pubsub] at com.google.cloud.pubsub.testing.v1.PubsubJsonGrpcAdapters$PublisherAdapter.handleRequest(PubsubJsonGrpcAdapters.java:231)
[pubsub] at com.google.cloud.pubsub.testing.v1.NettyUtil$HttpJsonAdapter.channelRead(NettyUtil.java:94)
[pubsub] at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
Seems that default PubSub client uses different protocol, but I don't see any way to configure it.
How can I use com.google.apis:google-api-services-pubsub library with local emulator?
The emulator does not require or handle authentication or authorization; I'm guessing that's where the problem lies. Can you try passing null as the last parameter to the builder?
Also, it appears that Gzip compression must be explicitly disabled. Please set this on the builder:
.setGoogleClientRequestInitializer(new GoogleClientRequestInitializer() {
#Override
public void initialize(AbstractGoogleClientRequest<?> request) throws IOException {
request.setDisableGZipContent(true);
}
})

Connection immediately is aborted when trying to connect to remote cluster-node

I am trying to connect to an Elasticsearch cluster form a Java application by using the TransportClient.
I have successfully tested connected to a cluster running locally.
Now we have set up a remote cluster in the EC2-Cloud. Accessing it via REST intefaces work, also Kibana works.
I can run my application on the Elasticsearch Node directly successfully.
However when I try to connect remotely from another host, the log indicates that the connection succeeds, but then immediately fails with "An established connection was aborted by the software in your host machine".
I have verified that both server and client run the same version of Elasticsearch. Also the firewall configuration should allow communication on port 9300 as this is also needed for the cluster-nodes to communicate between each other.
Okt 21, 2015 6:50:24 PM com.example.elasticsearch.visit.nativescript.ElasticsearchTest main
INFO: Setting adress to myhost.com:9300
Okt 21, 2015 6:50:24 PM org.elasticsearch.client.transport.TransportClientNodesService addTransportAddresses
FINE: [Master Man] adding address [[#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]]]
Okt 21, 2015 6:50:24 PM org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler doSample
FINEST: [Master Man] connecting to listed node (light) [[#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]]]
Okt 21, 2015 6:50:24 PM org.elasticsearch.transport.netty.NettyTransport connectToNode
FINE: [Master Man] connected to node [[#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]]]
Okt 21, 2015 6:50:24 PM org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker addWithoutBreaking
FINEST: [Master Man] [REQUEST] Adjusted breaker by [16440] bytes, now [16440]
Okt 21, 2015 6:50:24 PM org.elasticsearch.transport.TransportService$Adapter traceRequestSent
FINEST: [Master Man] [0][cluster:monitor/nodes/info] sent to [[#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]]] (timeout: [5s])
Okt 21, 2015 6:50:24 PM org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker addWithoutBreaking
FINEST: [Master Man] [REQUEST] Adjusted breaker by [-16440] bytes, now [0]
Okt 21, 2015 6:50:24 PM org.elasticsearch.transport.netty.NettyTransport exceptionCaught
FINEST: [Master Man] close connection exception caught on transport layer [[id: 0xd938d867, /x.x.x.x:60058 => myhost.com/1.1.1.1:9300]], disconnecting from relevant node
java.io.IOException: An established connection was aborted by the software in your host machine
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Okt 21, 2015 6:50:24 PM org.elasticsearch.transport.netty.NettyTransport disconnectFromNode
FINE: [Master Man] disconnecting from [[#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]]], channel closed event
Okt 21, 2015 6:50:24 PM org.elasticsearch.transport.netty.NettyTransport disconnectFromNode
FINEST: [Master Man] disconnected from [[#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]]], channel closed event
Okt 21, 2015 6:50:24 PM org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler doSample
INFO: [Master Man] failed to get node info for [#transport#-1][LNZ123675N02][inet[myhost.com/1.1.1.1:9300]], disconnecting...
org.elasticsearch.transport.NodeDisconnectedException: [][inet[myhost.com/1.1.1.1:9300]][cluster:monitor/nodes/info] disconnected
Okt 21, 2015 6:50:24 PM com.example.elasticsearch.visit.nativescript.ElasticsearchTest main
INFO: Having: []
Exception in thread "main" org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: []
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:200)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:167)
at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:370)
at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:166)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65)
at com.example.elasticsearch.visit.nativescript.ElasticsearchTest.main(ElasticsearchTest.java:68)
My simple test looks like this:
Settings settings = ImmutableSettings.settingsBuilder()
.put("cluster.name", "cluster-name")
.build();
logger.info("Creating client");
try (final TransportClient client = new TransportClient(settings)) {
logger.info("Setting adress to " + host + ":" + port);
client.addTransportAddress(new InetSocketTransportAddress(host, port));
logger.info("Having: " + client.connectedNodes());
BulkRequestBuilder bulkRequest = client.prepareBulk();
String json= "{ id: \"12345678\", value: \"value\" }";
bulkRequest.add(client.prepareIndex(INDEX, TYPE, "12345678").
setSource(json));
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) {
logger.info("Could not write bulk: " + bulkResponse.buildFailureMessage());
} else {
logger.info("Wrote");
}
}
I also tried to verify some common things like cluster-name not set and others.
Also when I use a completely different port, I get a timeout as expected, not this error-message. So the port is definitely behaving different.
Where would I start to look? Can this still be a firewall issue where connections are dropped only after a short while instead of being rejected immediately?
With more testing I actually found out that a company-firewall was interfering in a strange way. When I run the same client-code on a machine outside the company-netowrk, it works fine.

Issue with JacORB connection?

I'm new to CORBA and JacORB. I'm struggling with my first Hello World app using CORBA with JacORB.
This is the newest error I got:
Nov 04, 2013 9:52:19 AM com.sun.corba.se.impl.transport.SocketOrChannelConnectionIm <init>
WARNING: "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: localhost; port: 7070"
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
This is my server causes to that error:
Properties props = new Properties();
props.put("org.omg.CORBA.ORBInitialPort","7070");
props.put("org.omg.CORBA.ORBInitialHost","localhost");
ORB orb = ORB.init(args, props);
try {
//
POA poa = POAHelper.narrow(orb
.resolve_initial_references("RootPOA"));
poa.the_POAManager().activate();
// Servant
HelloVnImpl hello = new HelloVnImpl();
// get servant reference
Object o = poa.servant_to_reference(hello);
HelloVn hVnRef = HelloVnHelper.narrow(o);
// Get root naming context
Object objRef = orb.resolve_initial_references("NameService");//Error
NamingContextExt ncRef = NamingContextExtHelper.narrow(objRef);
Look around on the internet, the issue is apparently my nameservice, but I think I start it.
C:\Users\Paul>ns -ORBInitRef.NameService=corbaloc::localhost:7070/NameService
Nov 04, 2013 9:38:25 AM org.jacorb.config.JacORBConfiguration <init>
WARNING: no properties found for configuration jacorb
Nov 04, 2013 9:38:25 AM org.jacorb.orb.ORBSingleton <init>
INFO: created ORBSingleton
Nov 04, 2013 9:38:27 AM org.jacorb.orb.giop.CodeSet getTCSDefault
WARNING: Warning - unknown codeset (Cp1252) - defaulting to ISO-8859-1
Nov 04, 2013 9:38:27 AM org.jacorb.naming.NameServer main
INFO: NS up
Nov 04, 2013 9:38:27 AM org.jacorb.orb.ORB run
INFO: ORB run
Now, I have no idea to do next. Please give me an advice. Thank you. :)
UPDATE:
This line code give me a null value:
System.out.println(System.getProperty("org.omg.CORBA.ORBClass"));
What does it mean?. Does it tell me that there is something wrong with my nameserver?.
Hope to see your advice. Thanks
There are a couple of problems with the way that you setup your server and the Naming Service (NS):
1) The two properties (org.omg.CORBA.ORBInitialPort and org.omg.CORBA.ORBInitialHost) that you set in the server code are not JacORB properties. Assuming that you want your server to listen for requests on port 7070,then you want to set the "OAAddress" property like this:
props.put("OAAdress", "iiop://localhost:7070");
2) The option -ORBInitRef.NameService=corbaloc::localhost:XXX/NameService should be for the server to locate the NS and is not the NS. Please note that "XXX" is the listen port of the NS and is should not be "7070".
I would recommend that you take a look at the hello demo which includes both a working server and client and they are much simpler.

Categories

Resources