Develop Reference

How to use stable RestHighLevelClient with Elasticsearch?

I have searched for so many posts but I couldn't find a proper way to use Elastic Search with spring boot application because I am totally new to elastic search.
My only dependency is:
org.springframework.boot
spring-boot-starter-data-elasticsearch
2.7.3
My config class is:
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.RestClients;
import org.springframework.data.elasticsearch.config.AbstractElasticsearchConfiguration;
import org.springframework.data.elasticsearch.repository.config.EnableElasticsearchRepositories;
#Configuration
#EnableElasticsearchRepositories(basePackages = "com.backend.repository.elasticsearchrepository")
#ComponentScan(basePackages = {"com.backend.model.elasticsearchmodel"})
public class ElasticSearchConfig extends AbstractElasticsearchConfiguration {
#Value("${spring.elasticsearch.url}")
public String elasticsearchUrl;
#Value("${spring.elasticsearch.username}")
public String username;
#Value("${spring.elasticsearch.password}")
public String password;
#Bean
#Override
public RestHighLevelClient elasticsearchClient() {
final ClientConfiguration config = ClientConfiguration.builder()
.connectedTo(elasticsearchUrl)
.withBasicAuth(username, password)
.build();
return RestClients.create(config).rest();
}
}
Here RestHighLevelClient is shown as deprecated. And my repository class is:
package com.backend.repository.elasticsearchrepository;
import com.backend.model.elasticsearchmodel.EsOffice;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import java.util.UUID;
public interface ESOfficeRepository extends ElasticsearchRepository<EsOffice, UUID> {
}
When I call the methods of this repository then it works fine but while storing the data it is returning error message even if it adds the data successfully.
2022-10-15 00:00:15.608 ERROR 51607 --- [nio-8080-exec-2] c.a.a.exception.GlobalExceptionHandler : Unable to parse response body for Response{requestLine=POST /office/_doc?timeout=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 201 Created}; nested exception is java.lang.RuntimeException: Unable to parse response body for Response{requestLine=POST /office/_doc?timeout=1m HTTP/1.1, host=http://localhost:9200, response=HTTP/1.1 201 Created}
Which POM dependency + what kind of repository should I use and How can I configure it in my config file ? I need these 3 that compatible with each other ?

Spring Data Elasticsearch 4.4 (which is pulled in by Spring Boot 2.7.3) is build with the Elasticsearch libraries in version 7.17, this is problematic when running against an Elasticsearch cluster in version 8. Youhave basically two options:
Downgrade your cluster to version 7.17.6 (the latest 7.17 currently available) i f this is possible.
You can try and see if setting the compatibility headers (see the Spring Data Elasticsearch documentation section 5.3.1 for more info). This should work, but I encountered cases where the response from the cluster still wasn't readable with a 7.17 client. - I had issues opened with Elasticsearch and they were resolved, but there still might be hidden traps.

How can I work with micronaut 3 and google secret manager?

Currently migrating my application to Micronaut 3, I encountered one problem with micronaut-gcp. Especially with the google secret manager. I am using gradle with Kotlin DSL.
Actual configuration: (not working, using plugin io.micronaut.library version 2.0.4)
gradle 7.2
micronaut 3.0.1
Previous configuration: (working with no plugin, using micronaut-bom)
gradle 6.5.1
micronaut 2.4.0
micronautGcp 3.5.0
I/ The Problem
My goal is to load some secrets as key/value pairs into a property source.
I followed the documentation that says to use a bootstrap.yml as follows:
micronaut:
config-client:
enabled: true
gcp:
secret-manager:
keys:
- MY_SECRET
Then in my class I should be able to inject that secret value using the #Value annotation. Actually I am doing this in my tests. I am using the #MicronautTest annotation from jUnit5 so I have something that looks like this:
#MicronautTest
public class MyClass {
#Value("${my.secret}")
private String mySecret;
}
then comes the problem, when running any test, it fails in the initialization saying this:
Error instantiating bean of type [io.micronaut.gcp.secretmanager.client.DefaultSecretManagerClient]
Message: getTransportChannel() called when needsExecutor() is true
Path Taken: new DistributedPropertySourceLocator(ConfigurationClient configurationClient,Duration readTimeout) --> new DistributedPropertySourceLocator([ConfigurationClient configurationClient],Duration readTimeout) --> new DefaultCompositeConfigurationClient([ConfigurationClient[] configurationClients]) --> new SecretManagerConfigurationClient([SecretManagerClient secretManagerClient],SecretManagerConfigurationProperties configurationProperties) --> new DefaultSecretManagerClient([SecretManagerServiceClient client],Environment environment,GoogleCloudConfiguration googleCloudConfiguration,ExecutorService executorService)
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [io.micronaut.gcp.secretmanager.client.DefaultSecretManagerClient]
II/ What I've tried ?
When I've seen that, I was thinking that maybe the bootstrap.yaml thing doesn't work the I tried to use the lower lever access to secret manager using injecting SecretManagerServiceClient as follows:
#MicronautTest
public class MyClass {
#Inject
private SecretManagerServiceClient client;
private void someTest() {
String mySecret = client.getSecret("MY_SECRET");
}
}
but got same error Message: getTransportChannel() called when needsExecutor() is true with same path taken.
I also tried to upgrade to micronaut 3.0.1 but didn't change anything.
III/ My solution to handle this
As my problem was to retrieve a secret during the testing stage of my ci/cd process and, as I am using Google Cloud Build. I could pass the secret using the availableSecrets feature in my cloudbuild.yaml:
steps:
...
- name: 'gradle'
entrypoint: 'bash'
args: ['./gradlew', 'test']
secretEnv: ['MY_SECRET']
...
availableSecrets:
secretManager:
- versionName: projects/PROJECT_ID/secrets/MY_SECRET/versions/latest
env: 'MY_SECRET'
then passing it in my application.yml:
my:
secret: ${MY_SECRET}
and then in my code:
#MicronautTest
public class MyClass {
#Value("${my.secret}")
private String mySecret;
}
But I dont find this solution very satisfying as I find the bootstrap.yaml one more convenient.
If someone gets a solution for me or an idea/advice, I'll be happy to take it.
Have a nice day !

I've been down that rabbit hole. Long story short I got past this by upgrading the google-cloud-secretmanager dependency from 1.6.4 to e.g. 2.0.2
Like so:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-secretmanager</artifactId>
<version>2.0.2</version>
</dependency>

Spring Boot Actuator not showing any information

I'm using Spring Boot version 2.2.5.RELEASE and have trouble to receive any information from the actuator endpoint. What I'm receiving is only the empty structure of Actuator as following:
This is my configuration:
implementation "org.springframework.boot:spring-boot-starter-actuator"
#Bean
public HttpTraceRepository httpTraceRepository() {
return new InMemoryHttpTraceRepository();
}
management:
endpoints:
web:
exposure:
include: "*"
What do I miss?

Hallelujah, finally found the issue of my above described problem!
The problem was that i configured my spring boot application to use Gson instead of the default Jackson json parser. Not completely clear why, but with the following lines of code it results in issues to show the data at the actuator endpoints.
implementation ("org.springframework.boot:spring-boot-starter-web") {
exclude group: "org.springframework.boot", module: "spring-boot-starter-json"
}
implementation "com.google.code.gson:gson:2.8.6"
The following above described Bean is also unnecessary!
#Bean
public HttpTraceRepository httpTraceRepository() {
return new InMemoryHttpTraceRepository();
}
The following configuration was hence as well a problem:
spring:
http:
converters:
preferred-json-mapper: gson
gson:
exclude-fields-without-expose-annotation: true
And finally with removing of all of my #Expose annotations it started to work perfectly fine again.
#Expose
private String content;
If anyone can explain to me why this happened, or how to configure Gson to work properly with Spring Boot Actuator in combination, I'm here to listen and understand.

I had the same problem and I fixed by removing the following line
gsonBuilder.excludeFieldsWithoutExposeAnnotation()

Turning off Spring Boot AWS Autoconfiguration

I'm using spring-cloud-aws-autoconfigure:2.1.0.RELEASE to connect to AWS. However when the app is running in an enviromnent other than AWS, I don't want the auto configuration to take place.
I tried turning off the auto configuration as suggested here and here with java configuration class, and also with spring.autoconfigure.excludes property in my yml file like this:
spring:
autoconfigure:
exclude:
- org.springframework.cloud.aws.autoconfigure.context.ContextCredentialsAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextInstanceDataAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextStackAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.messaging.MessagingAutoConfiguration
But none of those solutions seems to work. The autoconfiguration still takes place and consequently, the app fails to start.

Found a solution: I added this directly to my main application class:
import org.springframework.cloud.aws.autoconfigure.context.*;
#SpringBootApplication
#EnableAutoConfiguration(exclude = {
ContextCredentialsAutoConfiguration.class,
ContextInstanceDataAutoConfiguration.class,
ContextRegionProviderAutoConfiguration.class,
ContextResourceLoaderAutoConfiguration.class,
ContextStackAutoConfiguration.class,
MailSenderAutoConfiguration.class,
})
public class MyApplication {
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
}

Found solution: I excluded every class I found in the autoconfiguration jar:
spring:
autoconfigure:
exclude:
- org.springframework.cloud.aws.autoconfigure.cache.ElastiCacheAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextCredentialsAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextInstanceDataAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.jdbc.AmazonRdsDatabaseAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.mail.MailSenderAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.messaging.MessagingAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.metrics.CloudWatchExportAutoConfiguration

Spring Cloud Config Server lookup through Eureka using Java instead of bootstrap.yml

I'm attempting to build code into our base pom which autoconfigures Spring Cloud Config server lookup through Eureka. We're doing this to avoid templating .yml properties for developers building microservices. For example, we want to java config all behavior triggered from these properties:
spring:
application:
name: MyMicroservice
cloud:
config:
enabled: true
server:
prefix: /diagnostics/admin/config
failFast: true
discovery:
enabled: true
serviceId: echo
management:
context-path: /diagnostics/admin
eureka:
password: password
client:
serviceUrl:
defaultZone: http://user:${eureka.password}#localhost:8761/eureka/
instance:
leaseRenewalIntervalInSeconds: 10
statusPageUrlPath: /diagnostics/admin/info
healthCheckUrlPath: /diagnostics/admin/health
After much experimenting, the following approach mostly works except for the Eureka-discovered config server (resulting in no overridden config properties):
#Order(-1)
public class AdditionalBootstrapPropertySourceLocator implements PropertySourceLocator {
#Override
public PropertySource<?> locate(Environment environment) {
Map<String, Object> theBootstrapYmlConfig = new HashMap<>();
theBootstrapYmlConfig.put("spring.cloud.config.enabled", new Boolean(true));
theBootstrapYmlConfig.put("spring.cloud.config.server.prefix", "/diagnostics/admin/config");
theBootstrapYmlConfig.put("spring.cloud.config.failFast", new Boolean(true));
theBootstrapYmlConfig.put("spring.cloud.config.discovery.enabled", new Boolean(true));
theBootstrapYmlConfig.put("spring.cloud.config.discovery.serviceId", "echo");
theBootstrapYmlConfig.put("management.context-path", "/diagnostics/admin");
theBootstrapYmlConfig.put("eureka.client.serviceUrl.defaultZone", "http://user:password#localhost:8761/eureka/");
theBootstrapYmlConfig.put("eureka.instance.leaseRenewalIntervalInSeconds", new Integer(10));
theBootstrapYmlConfig.put("eureka.instance.statusPageUrlPath", "/diagnostics/admin/info");
theBootstrapYmlConfig.put("eureka.instance.healthCheckUrlPath", "/diagnostics/admin/health");
return new MapPropertySource("myExtraBootstrap", theBootstrapYmlConfig);
}
}
And I seem to need this Bean as well:
#ConditionalOnWebApplication
#Configuration
#Import(EurekaClientAutoConfiguration.class)
public class WorkfrontDiscoveryClientConfigServiceBootstrapConfiguration {
#Bean
#ConditionalOnClass({ DiscoveryClient.class, ConfigServicePropertySourceLocator.class })
#ConditionalOnMissingBean
DiscoveryClientConfigServiceBootstrapConfiguration discoveryClientConfigServiceBootstrapConfiguration() {
DiscoveryClientConfigServiceBootstrapConfiguration discoveryClientConfigServiceBootstrapConfiguration =
new DiscoveryClientConfigServiceBootstrapConfiguration();
return discoveryClientConfigServiceBootstrapConfiguration;
}
}
Finally, I put both into spring.factories to ensure they are constructed. The problem is that the PropertySourceLocator is never used to construct the call within ConfigServicePropertySourceLocator to retrieve the properties. No matter what I do, I cant seem to match the behaviors that specifying the properties within bootstrap.yml would produce.
Edit 4 days later
The key factor (and restriction) here is the ability to look up the config server through Eureka. In the current spring cloud release (1.0.2), the property source is retrieved and constructed too early in the spring initialization cycle for the config-lookup-through-eureka java property source config I have above. Plus if the Eureka server is slow or not available at bootstrap startup time, the Config server property source is never reconstructed when Eureka finally comes up. This in my mind is a bug.
I solved this all by eliminating the concept of looking up the config server through Eureka, and requiring this minimum config in bootstrap.yml:
spring:
application:
name: MyMicroservice
cloud:
config:
uri: http://localhost:8888/diagnostics/admin/config
eureka:
client:
serviceUrl:
defaultZone: http://user:password#localhost:8761/eureka/
and then setting the rest in the java AdditionalBootstrapPropertySourceLocator
Edit 30 days later
Java-configing bootstrap properties continues to be a challenge. I'm doing this because I'm developing a framework without templating or code generation (the premise of spring boot). I've added spring-retry to the mix and client-to-config gets retried but re-registration to Eureka does not. This is why Eureka-first had to be abandoned for me. I'd put my vote in for integrating spring-retry into the Eureka registration process so I can go back to Eureka-first for my framework. Still on Spring Cloud 1.0.2.
Edit 100 days later
Update for where we ended up. Continuing along our mantra of avoiding property templating, enforcing policies and practices within code .. and continuing without a Eureka-first concept, we abandoned PropertySourceLocator and simply used a SpringApplicationRunListener as follows:
public class OurFrameworkProperties implements SpringApplicationRunListener {
:
public void started() {
if (TestCaseUtils.isRunningFromTestCase()) {
System.setProperty("spring.cloud.config.failFast", "false");
System.setProperty("spring.cloud.config.enabled", "false");
System.setProperty("eureka.client.enabled", "false");
} else {
// set production values same way
}
}
}
A caution that this started() actually gets called twice inside the spring code (once not passing any program arguments btw) everytime your Spring application runs or gets an Actuator refresh().

If your PropertySourceLocator is listed inspring.factories (I assume as a BootstrapConfiguration) then it needs to be a #Component (or maybe even a #Configuration).

you have to set this properties in the boostrap.properties
eureka.instance.metadataMap.configPath: /your-app-name
and comment this one
#spring.cloud.config.uri=http://localhost:8888/
and obviously it must be also this
eureka.client.serviceUrl.defaultZone: ${EUREKA_URI:http://localhost:8761/eureka}
eureka.client.instance.preferIpAddress: true
according with the documentation
https://cloud.spring.io/spring-cloud-config/multi/multi__spring_cloud_config_client.html#discovery-first-bootstrap