Spring Cloud Cannot Load Config - java

I deployed this Spring application]1 on a kubernetes cluster with kompose up. Once it was deployed, the pods for a number of the services would enter a CrashLoopBackoff state. The services I've had problems with are the account service, monitoring service, notification service, and statistics service. It appears that when the services start up, they run into the following error:
2017-11-15 22:10:52.857 INFO 1 --- [ main] c.p.monitoring.MonitoringApplication : Started MonitoringApplication in 0.511 seconds (JVM running for 8.579)
2017-11-15 22:10:52.882 INFO 1 --- [ main] o.s.c.s.b.r.RabbitMessageChannelBinder : declaring queue for inbound: springCloudHystrixStream.anonymous.oCdNGHmBRPO_Z3LMusKS2Q, bound to: springCloudHystrixStream
2017-11-15 22:11:52.953 WARN 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'outputBindingLifecycle'; nested exception is org.springframework.context.ApplicationContextException: Failed to start bean 'inputBindingLifecycle'; nested exception is org.springframework.amqp.AmqpIOException: java.net.SocketTimeoutException: connect timed out
The full log can be found here. Any help is greatly appreciated. I assume this is a Spring/Java related issue but it may be a Kubernetes related issue as well given that the application was designed to run on docker swarm.

Related

How to Shutdown Spring's STOMP Server (SimpleBrokerMessage) programmatically?

I've developed a Spring application that uses STOMP Server, and I would like to shutdown the STOMP server programmatically.
I've tried to shutdown the application using the following code:
SpringApplication.exit(context, new ExitCodeGenerator() {
#Override
public int getExitCode() {
return 0;
}
});
However, the code produces the following error:
...
WARN 15444 --- [ main] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [MessageBroker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base/jdk.internal.misc.Unsafe.park(Native Method)
...
To avoid the error, my plan is to shutdown the STOMP server first and then execute the
SpringApplication.exit( ... )
So, my question is how to shutdown the STOMP server programmatically?
Probably, the code should produce output like this one:
2023-01-13 20:28:05.869 INFO 4380 --- [ionShutdownHook] o.s.m.s.b.SimpleBrokerMessageHandler : Stopping...
2023-01-13 20:28:05.870 INFO 4380 --- [ionShutdownHook] o.s.m.s.b.SimpleBrokerMessageHandler : BrokerAvailabilityEvent[available=false, SimpleBrokerMessageHandler [org.springframework.messaging.simp.broker.DefaultSubscriptionRegistry#5d68db92]]
2023-01-13 20:28:05.870 INFO 4380 --- [ionShutdownHook] o.s.m.s.b.SimpleBrokerMessageHandler : Stopped.

Spring Boot Application shown as UNKNOWN in Eureka server

I'm running an application with Spring (as Discovery Client app), also have a Discovery Server with Eureka, and Spring Cloud Config Server. When the client app is started it's registered as "UNKNOWN" in Eureka, despite its status is set to"UP", isn't able to get the properties from the configuration server.
Client App, Eureka server,and Config Server Spring Boot Version: 2.4.2
Client bootstrap.properties:
spring.application.name=config-client-app
spring.cloud.config.discovery.enabled=true
eureka.client.service-url.defaultZone=http://localhost:8761/eureka
eureka.instance.instance-id=${spring.application.name}
Client application.properties file:
logging.level.=debug
server.port=8900
eureka.client.healthcheck.enabled=true
Client Application Class:
#SpringBootApplication
#EnableDiscoveryClient
#RestController
public class ConfigClientAppApplication {
public static void main(String[] args) {
SpringApplication.run(ConfigClientAppApplication.class, args);
}
}
Eureka Serverproperties file:
spring.application.name=discovery-server
server.port=8761
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
Eureka Application class:
#EnableEurekaServer
#SpringBootApplication
public class DiscoveryServerApplication {
public static void main(String[] args) {
SpringApplication.run(DiscoveryServerApplication.class, args);
}
}
The log info:
restartedMain] c.n.discovery.InstanceInfoReplicator : InstanceInfoReplicator onDemand update allowed rate per min is 4
2021-02-09 16:02:50.388 INFO 2845 --- [ restartedMain] com.netflix.discovery.DiscoveryClient : Discovery Client initialized at timestamp 1612904570388 with initial instances count: 1
2021-02-09 16:02:50.390 INFO 2845 --- [ restartedMain] o.s.c.n.e.s.EurekaServiceRegistry : Registering application UNKNOWN with eureka with status UP
2021-02-09 16:02:50.391 INFO 2845 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : Saw local status change event StatusChangeEvent [timestamp=1612904570391, current=UP, previous=STARTING]
2021-02-09 16:02:50.391 INFO 2845 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_UNKNOWN/192.168.10.22:8900: registering service...
2021-02-09 16:02:50.391 WARN 2845 --- [ restartedMain] c.n.discovery.InstanceInfoReplicator : Ignoring onDemand update due to rate limiter
2021-02-09 16:02:50.393 INFO 2845 --- [ restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8900 (http) with context path ''
2021-02-09 16:02:50.394 INFO 2845 --- [ restartedMain] .s.c.n.e.s.EurekaAutoServiceRegistration : Updating port to 8900
2021-02-09 16:02:50.413 INFO 2845 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_UNKNOWN/192.168.10.22:8900 - registration status: 204
I am using Spring cloud version
<spring-cloud.version>2021.0.5</spring-cloud.version>
and spring boot version
<version>2.7.5</version>
In application.properties, add the following lines to resolve the error:
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
By default, the Eureka Server registers itself into the discovery. The following 2 lines will tell eureka server that there is only one eureka server present in this context.
in the absence of these, our Eureka Server will try to find and register itself on other Eureka Servers in the context and throw a discovery client exception. i.e our own system.
If you encounter the below errors after adding the above properties only you need to add the below properties.
We must add this to the properties, if configuration is not required.
spring.cloud.config.enabled=false
Otherwise application will throw below exception.
No spring.config.import property has been defined
(or) If you have a other server, such as any configuration server we can add this optional server configuration to application.properties,
spring.config.import=optional:configserver: Your server url
Spring Cloud Config provides server and client-side support for
externalized configuration in a distributed system

Consul and Spring Boot services in Docker - not deregistering

So we have Java microservices written with Spring-Boot, using Consul for service discovery and config management and running in Docker containers. All of it is working, but when a container dies or a service restarts the old service-id never goes away in Consul and the service forever after shows as "Failing" in the Consul UI, even though the new container has registered and shows all Green.
We are not using heartbeat - but I cannot find much documentation on what the difference between heartbeat and healthcheck are for Consul.
Here's my bootstrp.yml
spring:
application:
name: my-service
cloud:
config:
enabled: false
consul:
host: ${discovery.host:localhost}
port: ${discovery.port:8500}
config:
watch:
wait-time: 30
delay: 10000
profile-separator: "-"
format: FILES
discovery:
prefer-ip-address: true
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
There are other settings to enable heartbeat, but the docs say something about this putting more stress on the Consul cluster.
Has anyone managed to get Consul and Spring Boot/Docker services to actually de-register automatically? It actually doesn't cause any real problems, but it makes the Consul UI pretty useless to actually monitor for up/down services.
Consul doesn't automatically deregister services.
See https://groups.google.com/forum/#!topic/consul-tool/slV5xfWRpEE for the hint about the same question. According to that thread you need to either update the config or perform an Agent API call. Since the agent is the source of truth, you shouldn't try to update via Catalog API. See GitHub for details. They also mention at the Google group that you don't necessarily have to deregister services if the node goes down gracefully, but that doesn't seem to be your use case.
Please have a look at Consul not deregistering zombie services for hints about automating the service de-registration using either the api or tools like registrator.
You have mentioned you are using a docker container to run the microservice. Are you trapping the SIGTERM in your entrypoint script in docker container ? If a SIGTERM is sent, the boot application will get it and you will see the below log showing that the microservice is deregistering with Consul.
2017-04-27 09:20:19.854 INFO 6852 --- [on(6)-127.0.0.1] inMXBeanRegistrar$SpringApplicationAdmin : Application shutdown requested.
2017-04-27 09:20:19.857 INFO 6852 --- [on(6)-127.0.0.1] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#afb5821: startup date [Thu Apr 27 09:20:00 EDT 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#130c12b7
2017-04-27 09:20:19.859 INFO 6852 --- [on(6)-127.0.0.1] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647
2017-04-27 09:20:19.863 INFO 6852 --- [on(6)-127.0.0.1] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 0
2017-04-27 09:20:19.863 INFO 6852 --- [on(6)-127.0.0.1] o.s.c.c.s.ConsulServiceRegistry : Deregistering service with consul: xxxxxxxxxxxxx
This blog post discusses this.

Spring boot connection reset after a few minutes in docker container

The backend runs in a docker container. It runs with spring boot & security and works fine. The frontend runs in another container as well. All works fine until out of nowhere this exception is thrown. I only noticed this exception after I idled a few minutes (normaly between 5 and 10 min).
WARN 1 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08S01
ERROR 1 --- [nio-8080-exec-2] o.h.engine.jdbc.spi.SqlExceptionHelper : I/O Error: Connection reset
ERROR 1 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.dao.DataAccessResourceFailureException: could not extract ResultSet; nested exception is org.hibernate.exception.JDBCConnectionException: could not extract ResultSet] with root cause
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[na:1.8.0_111-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111-internal]
at java.io.DataInputStream.readFully(DataInputStream.java:195) ~[na:1.8.0_111-internal]
.....
The connection to the Database is available all the time. The weird thing is, that this error is only thrown when I run my project with docker. If I run it with eclipse & node.js the error don't occurs. The only thing I noticed was, that a few minutes after I loged in and idled, I got automaticly loged out when I tried to do something. I don't know if this has any connection with the orgiginal error.
application.properties
spring.datasource.url=databaseURL
spring.datasource.username=SQLusername
spring.datasource.password=Password
spring.datasource.driver-class-name=net.sourceforge.jtds.jdbc.Driver
spring.datasource.initialSize=100
spring.datasource.minIdle=10
spring.jpa.hibernate.naming_strategy=org.hibernate.cfg.EJB3NamingStrategy
spring.jpa.hibernate.naming.implicit-strategy=org.hibernate.boot.model.naming.ImplicitNamingStrategyLegacyJpaImpl
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
spring.datasource.tomcat.max-active=250
spring.datasource.tomcat.max-wait=30000
spring.datasource.test-on-borrow=true
spring.datasource.test-while-idle=true
spring.datasource.minEvictableIdleTimeMillis=21600000
spring.datasource.timeBetweenEvictionRunsMillis=43200000
spring.datasource.connection-test-query=SELECT 1
spring.datasource.validationQuery=SELECT 1
I'm glad for every suggestion that could help to fix this proplem.
I had the same problem months ago!
This happens because the jvm doesn’t have memory enough to accept this request!
This error is intermittent and is very difficult to find your root cause!
Try this:
Put in your Dockerfile some flags to solve it
ENTRYPOINT ["java","-XX:+TieredCompilation","-XX:TieredStopAtLevel=1", "-jar","yourJar.jar"]
More details here: https://jpbempel.github.io/2020/05/22/startup-containers-tieredcompilation.html
Sometimes the problem can be memory. When you start your container with java, the jvm default heap memory is 1/4 of total memory of container.
Then you can use the flag -XX:MaxRAMFraction where your value has a range 1 to 4.
More details here: https://merikan.com/2019/04/jvm-in-a-container/#java-8u131-and-java-9
Good luck!

Error launching simple Eureka-Service server

I am following Josh Long's presentation at https://www.youtube.com/watch?v=5q8B6lYhFvE&feature=youtu.be . I am following his examples. At about 34 minutes in he demos setting up the Eureka service.
This service uses the config-service. My config service is running and exposes the information the eureka service needs. I have checked my bootstrap.properties and they look correct according to his presentation. However the eureka service does not complete and does not seem to be accessing the config service. My other service does reach the config service and works fine.
The eureka service should come up at http://localhost:8761/ but does not. It is at 8080.
eureka service bootstrap.properties:
spring.application.name= eureka-service
spring.cloud.config.uri= http://localhost:8888
First error:
2016-10-13 18:43:00.088 ERROR 7464 --- [ main] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused: connect
Second error
2016-10-13 18:43:00.097 ERROR 7464 --- [ main] com.netflix.discovery.DiscoveryClient : DiscoveryClient_EUREKA-SERVICE/RB-64-PC.Home:eureka-service - was unable to refresh its cache! status = Cannot execute request on any known server
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
Found the problem. I had accidently added the "config server" dependency into the project when I created it. I am using the site "http://start.spring.io/" as per the video. Adding the config server dependencies kept the initialization from accessing the actual config server running on another port.

Categories

Resources