Develop Reference

Flink: TaskManager cannot connect to the JobManager - Could not resolve ResourceManager address

I'm using the Apache Flink Kubernetes operator to deploy a standalone job on an Application cluster setup.
I have setup the following files using the Flink official documentation - Link
jobmanager-application-non-ha.yaml
taskmanager-job-deployment.yaml
flink-configuration-configmap.yaml
jobmanager-service.yaml
I have not changed any of the configurations in these files and am trying to run a simple WordCount example from the Flink examples using the Apache Flink Operator.
After running the kubectl commands to setting up the job manager and the task manager - the job manager goes into a NotReady state while the task manager goes into a CrashLoopBackOff loop.
NAME READY STATUS RESTARTS AGE
flink-jobmanager-28k4b 1/2 NotReady 2 (4m24s ago) 16m
flink-kubernetes-operator-6585dddd97-9hjp4 2/2 Running 0 10d
flink-taskmanager-6bb88468d7-ggx8t 1/2 CrashLoopBackOff 9 (2m21s ago) 15m
The job manager logs look like this
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
at org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulkCheckerImpl.lambda$schedulePendingRequestBulkWithTimestampCheck$0(PhysicalSlotRequestBulkCheckerImpl.java:86) ~[flink-dist-1.16.0.jar:1.16.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:453) ~[flink-rpc-akka_be40712e-8b2e-47cd-baaf-f0149cf2604d.jar:1.16.0]
at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-rpc-akka_be40712e-8b2e-47cd-baaf-f0149cf2604d.jar:1.16.0]
The Task manager it seems cannot connect to the job manager
2023-01-28 19:21:47,647 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Connecting to ResourceManager akka.tcp://flink#flink-jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000).
2023-01-28 19:21:57,766 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://flink#flink-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink#flink-jobmanager:6123/user/rpc/resourcemanager_*.
2023-01-28 19:22:08,036 INFO akka.remote.transport.ProtocolStateActor [] - No response from remote for outbound association. Associate timed out after [20000 ms].
2023-01-28 19:22:08,057 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink#flink-jobmanager:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#flink-jobmanager:6123]] Caused by: [No response from remote for outbound association. Associate timed out after [20000 ms].]
2023-01-28 19:22:08,069 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://flink#flink-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink#flink-jobmanager:6123/user/rpc/resourcemanager_*.
2023-01-28 19:22:08,308 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with org.jboss.netty.channel.ConnectTimeoutException: connection timed out: flink-jobmanager/100.127.18.9:6123
The flink-configuration-configmap.yaml looks like this
flink-conf.yaml: |+
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 2
blob.server.port: 6124
jobmanager.rpc.port: 6123
taskmanager.rpc.port: 6122
queryable-state.proxy.ports: 6125
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
parallelism.default: 2
This is what the pom.xml looks like - Link

You deployed the Kubernetes Operator in the namespace, but you did not create the CRDs the Operator requires. Instead you tried to create a standalone Flink Kubernetes cluster.
The Flink Operator makes it a lot easier to deploy your Flink jobs, you only need to deploy the operator itself and FlinkDeployment/FlinkSessionJob CRDs. The operator will manage your deployment after.
Please use this documentation for the Kubernetes Operator: Link

Error launching simple Eureka-Service server

I am following Josh Long's presentation at https://www.youtube.com/watch?v=5q8B6lYhFvE&feature=youtu.be . I am following his examples. At about 34 minutes in he demos setting up the Eureka service.
This service uses the config-service. My config service is running and exposes the information the eureka service needs. I have checked my bootstrap.properties and they look correct according to his presentation. However the eureka service does not complete and does not seem to be accessing the config service. My other service does reach the config service and works fine.
The eureka service should come up at http://localhost:8761/ but does not. It is at 8080.
eureka service bootstrap.properties:
spring.application.name= eureka-service
spring.cloud.config.uri= http://localhost:8888
First error:
2016-10-13 18:43:00.088 ERROR 7464 --- [ main] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused: connect
Second error
2016-10-13 18:43:00.097 ERROR 7464 --- [ main] com.netflix.discovery.DiscoveryClient : DiscoveryClient_EUREKA-SERVICE/RB-64-PC.Home:eureka-service - was unable to refresh its cache! status = Cannot execute request on any known server
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server

Found the problem. I had accidently added the "config server" dependency into the project when I created it. I am using the site "http://start.spring.io/" as per the video. Adding the config server dependencies kept the initialization from accessing the actual config server running on another port.

Connecting to couchbase running on alternate ports using java client

I'm using couchbase-client 2.3.2 for Java and Couchbase server Community 4.0
So I'm experimenting with running Couchbase on non-standard ports using the documentation at Couchbase website
I've managed to start Couchbase using these alternate ports but I've only managed to change some of the ports in the java client, here's my code:
final CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
.bootstrapCarrierDirectPort(21210)
.bootstrapHttpDirectPort(9091)
.build();
return CouchbaseCluster.create(env, "10.0.2.15");
My program is able to connect to couchbase and so some things, however I still need to change the view port (default 8092) and the query port (default 8093) in the client. as a result I'm met with these errors:
2016-09-30 14:03:49.696 [] WARN c.c.c.c.e.Endpoint - [null][QueryEndpoint]: Could not connect to endpoint, retrying with delay 32 MILLISECONDS: ! java.net.ConnectException: Connection refused: /10.0.2.15:8093
2016-09-30 14:03:52.077 [] WARN c.c.c.c.e.Endpoint - [null][ViewEndpoint]: Could not connect to endpoint, retrying with delay 2048 MILLISECONDS: ! java.net.ConnectException: Connection refused: /10.0.2.15:8092
So the client still tries to connect to 8092 and 8093 when in fact I've changed those to 9092 and 9093

From the JavaDoc on 2.3.4 (http://docs.couchbase.com/sdk-api/couchbase-java-client-2.3.4/), I believe what you want is this:
DefaultCouchbaseEnvironment.Builder viewEndpoints(int viewServiceEndpoints)

Even though it's completely undocumented, you need to add those ports to static_config as well:
{capi_port, 9092}.
{query_port, 9093}.
and then it works, hope someone at couchbase sees this and updates their documentation :)

kafka + zookeeper remote = error

I am trying to install a kafka & zookeeper instance on a remote server. I only need 1 node of each actually because i only want to provide remote kafka for test purposes.
Kafka and Zookeeper are running from the Apache Kafka tarball you can find there (v0.0.9), inside a Docker image.
Trying to consume / produce using the provided scripts. And trying to produce using own java application. Everythinf is working fine if Kafka & ZK are installed on the local server.
Here is the error I get while trying to produce :
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
No partition metadata for topic RSS due to kafka.common.LeaderNotAvailableException}] for topic [RSS]: class kafka.common.LeaderNotAvailableException
Kafka properties tested
First :
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=localhost:<PORT>
Second:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
Third:
borker.id=0
port=9092
host.name=<external-ip>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Last:
borker.id=0
port=9092
host.name=</etc/host name>
zookeeper.connect=<external-ip>:<PORT>
advertised.host.name=<external-ip>
advertised.host.port=<external-ip>
Here is my "/etc/hosts"
127.0.0.1 kafka kafka
127.0.0.1 localhost
I followed the Getting Started, which if I understood is a localhost / signle server configurations. I cannot understand what I have to do to get this work with remote calls...
Thanks for your help !
EDIT 1
host.name=localhost
advertised.host.name=politik.cm-cloud.fr
Seems to allow a local consumer (on the server) and producer. But if we want to do the same from a remote server we get
[2015-12-09 12:44:10,826] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host

The error does not look like connectivity problem with Zookeeper / Kafka.
Just follow the instruction in "quickstart" from http://kafka.apache.org/
BrokerPartitionInfo:83 - Error while fetching metadata [{TopicMetadata for topic RSS ->
Additionally the error indicates there is no partition info i.e topic not yet created . Try creating topics first and then try to produce/consume because when producing to a non existent topic kafka will create the topic based on auto.create.topics.enable in server.properties but remotely it is better to create topics rathen than relying on auto create

how can we add a document using solr cloud server

While adding a document using solr cloud server I'm getting following exception
60 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Waiting for client to connect to ZooKeeper
65 [main-SendThread(jmajeed.ibsorb.com:8982)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server jmajeed.ibsorb.com/192.168.70.91:8982. Will not attempt to authenticate using SASL (unknown error)
69 [main-SendThread(jmajeed.ibsorb.com:8982)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to jmajeed.ibsorb.com/192.168.70.91:8982, initiating session
Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 192.168.70.91:8982/#/hotelcontent within 10000 ms
Does anybody has any idea why this is happening??
Thanks.

Have the disturbed the default configuration of solr nodes because by default if you do not specify the port the first node in the cluster will start in 8983 port so check this first. If this is not the problem then check whether the cluster is up or not by accessing admin UI of solr cloud. Then see whether all the shards in the cluster are alive by clicking on the cloud tab.
If everything is fine and still you are facing the above problem then are you trying to access a remote solr cloud server and it is firewall issue.