Develop Reference

ES 7.4.1 - Authentication [Rest API]

I’m a newbie in ES and I have a task in my new job to upgrade from 6.4.2 to 7.4.1 – From TCP client to Rest High Level API.
Previously we built the client like this:
Settings settings = Settings.builder()
.put("xpack.security.user", String.format("%s:%s",esJavaUser,esJavaPassword))
.put("cluster.name", esClusterName)
.put("xpack.security.transport.ssl.enabled", xpackSecurityTransportSslEnabled)
.put("xpack.ssl.certificate_authorities", xpackSslCertificateAuthorities)
.build();
client = new PreBuiltXPackTransportClient(settings);
Now, in rest API, it’s changed to this:
final CredentialsProvider credentialsProvider =
new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(esJavaUser, esJavaPassword));
RestClientBuilder restClientBuilder = RestClient.builder(hosts)
.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
.setDefaultCredentialsProvider(credentialsProvider));
restHighLevelClient = new RestHighLevelClient(restClientBuilder);
With this build I set ES user and password by CredentialsProvider but what about ssl.enabled and certificate_authorities”? how should I provided them with rest API?

I got an answer from ES forum (didn't thought to ask there first..)
Because, as developer, I always looking for answer here, in stackoverflow, I decide to not delete this question and copy TimV answer:
The documentation you are looking for is here: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.4/_encrypted_communication.html
SSL is automatically enabled (or not) based on the scheme (protocol) in the HttpHost objects you pass to the builder.
RestClient.builder(hosts)
If you are using SSL, you want to pass "https" as the scheme (3rd argument) when you construct the HttpHost objects (hosts).
Unfortunately there is no simple means to pass certificate_authorities to the Rest client, you need to turn those certificates into a standard Java truststore.
You can probably find some sample code on the web ("convert PEM certificates to Java truststore"), but the gist of it is:
Open the certificate authority files as an InputStream
Create a X.509 certificate factory: java.security.cert.CertificateFactory.getInstance("X.509")
Call generateCertificates on the certificate factory to read those certificate files into java Certificate objects
Construct an empty KeyStore object
Add the loaded certificates as trusted entries
Pass that to SSLContextBuilder.loadTrustMaterial
Link: https://discuss.elastic.co/t/es-7-4-1-authentication-rest-api/211969

Is there a working code in Java to connect with Azure Keyvault to retrieve secrets using managed service identities

I am creating an application which connects with Azure keyvault to retrieve its secrets using managed service identity.
I have tried to connect with keyvault without using managed identities and it worked but with msi, I am not able to connect.
Few things I tried:
1) This piece of code is timing out:
MSICredentials credentials = new MSICredentials(AzureEnvironment.AZURE);
log.debug("Credentials acquired");
KeyVaultClient keyVaultClient = new KeyVaultClient(credentials);
log.debug("Key Vault client created");
String secret =
keyVaultClient.getSecret("https://<key-vault-name>.vault.azure.net/",
"<secret-name>").value();
2) This piece of code is throwing error in the 1st line itself saying endpoint==null. which by looking at library means that I need to furnish MSI_ENDPOINT which I am not sure what it is.
AppServiceMSICredentials credentials =
new AppServiceMSICredentials(AzureEnvironment.AZURE);
log.debug("Credentials acquired");
KeyVaultClient keyVaultClient = new KeyVaultClient(credentials);
log.debug("Key Vault client created");
String secret =
keyVaultClient.getSecret("https://<key-vault-name>.vault.azure.net/",
"<secret-name>").value();
Any help with this is appreciated.

We can use AppServiceMsiCredential class to achieve the result
you can read all about the param here
import com.microsoft.azure.AzureEnvironment;
import com.microsoft.azure.credentials.AppServiceMSICredentials;
import com.microsoft.azure.keyvault.KeyVaultClient;
import com.microsoft.azure.keyvault.models.KeyBundle;
AppServiceMSICredentials credentials = new AppServiceMSICredentials(AzureEnvironment.AZURE);
KeyVaultClient keyVaultClient = new KeyVaultClient(credentials);
keyVaultClient.getSecret("https://xxxx.vault.azure.net","secretName");
When MSI is enabled for an App Service/any resource, two environment variables MSI_ENDPOINT and MSI_SECRET are available (note that they change every time you restart the App Service) and can be used to obtain an access token for a given resource.
Hope it helps.

I followed this link to fetch database credentials from Azure Key Vault in my App Service. Used Managed Service Identity. Workekd like a charm
https://github.com/gandhirajan/Azure_AppService_ManagedIdentity

Google Dialogflow: The Application Default Credentials are not available

Hi I've an issue with Java SDK library for Google Cloud.
I need to query Dialogflow V2 API and I'm using this SDK (https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master)
I've followed the instructions in order to set GOOGLE_APPLICATION_CREDENTIALS as environment variable.
I did several attempts (I'm using a Mac):
export GOOGLE_APPLICATION_CREDENTIALS=path/to/my.json
doesn't work
putting
export GOOGLE_APPLICATION_CREDENTIALS=path/to/my.json
in .bash_profile
doesn't work
In my Java code:
System.setProperty("GOOGLE_APPLICATION_CREDENTIALS", "/path/to/my.json");
GoogleCredential credential = GoogleCredential.getApplicationDefault();
doesn't work
String jsonPath = "/path/to/my.json";
GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonPath));
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();
doesn't work
putting GOOGLE_APPLICATION_CREDENTIALS as environment variable in Eclipse by "Maven build > Environment > New variable" and restarted IDE
doesn't work
The same error always occurs:
An error occurred during Dialogflow http request: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information
Really I can't undestand what is wrong.
This is the code snippet to query Dialogflow never reached due to the above error:
SessionsClient sessionsClient = SessionsClient.create();
SessionName session = SessionName.of("my-project-id", sessionId);
InputStream stream = new ByteArrayInputStream(requestBody.getBytes(StandardCharsets.UTF_8));
QueryInput queryInput = QueryInput.newBuilder().build().parseFrom(stream);
DetectIntentResponse response = sessionsClient.detectIntent(session, queryInput);
Exception is raised on
SessionsClient sessionsClient = SessionsClient.create();
Thanks in advance.

The solution is to set GOOGLE_APPLICATION_CREDENTIALS as environment variable for the Application Server.

Unlike mentioned in question, I've got 'credentials' working when provided that way
Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();
But same approach with FirestoreOptions failed with mentioned error "The Application Default Credentials are not available".
Note: I'm willing to not configure my app via environment variable and prefer doing it in code. One more reason for that - I need multiple different connections from one app.
After debugging into Google's code I've found that they do not use supplied credentials, they rather call getCredentialsProvider() on provided options. I think this is bug in their implementation, but workaround is rather simple:
Create own CredentialsProvider
Set it into FirestoreOptions
Dummy sample:
public static class MyCredentialsProvider implements CredentialsProvider {
GoogleCredentials credentials;
public MyCredentialsProvider(GoogleCredentials credentials) {
this.credentials = credentials;
}
#Override
public Credentials getCredentials() throws IOException {
return credentials;
}
}
...
FirestoreOptions.Builder builder = FirestoreOptions.newBuilder()
.setProjectId(serviceAccount.project_id)
.setCredentialsProvider(new MyCredentialsProvider(credentials));

TLS on Embedded Jetty without JKS

My goal: I am working on an integration of Jetty Embedded that would make it simple to use. The interface would allow, among others, for integration of external sources for TLS certificates, without the use of the Java KeyStore.
This would allow for greater flexibility when building distributed web services (in my case an experimental, self-hosted CDN).
However, I am having problems building the integration. The stub implementation is in this repository.
What I've tried: I have tried replacing the key manager and the trust manager and set break points to every function in it. However, when trying to access the server, these break points are never triggered. Instead, I'm encountering this error:
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:621)
at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:322)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:231)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:261)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:150)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
I have tried analyzing the "standard" Jetty setup which a certificate from a keystore, but without much luck. I'm failing to find the point where Jetty is obtaining the cipher / certificate information that I should override.
My question: How can I get Jetty to use my own certificate source instead of the Java KeyStore and TrustStore?

#EJP pointed me in the right direction, so here's how to do it:
Here's how it needs to be done.
First, set up Jetty for TLS:
HttpConfiguration https = new HttpConfiguration();
https.addCustomizer(new SecureRequestCustomizer());
SslContextFactory sslContextFactory = new JettySslContextFactory(configuration.getSslProviders());
ServerConnector sslConnector = new ServerConnector(
server,
new SslConnectionFactory(sslContextFactory, "http/1.1"),
new HttpConnectionFactory(https)
);
sslConnector.setPort(httpsPort);
Note the class JettySslContextFactory. This class extends the built-in X509ExtendedKeyManager and needs to override the protected KeyManager[] getKeyManagers(KeyStore keyStore) throws Exception method in order to provider a custom KeyManager, like this:
#Override
protected KeyManager[] getKeyManagers(KeyStore keyStore) throws Exception {
return new KeyManager[] {
new JettyX509ExtendedKeyManager(certificateProviders)
};
}
In addition to that, the following steps are run in every connection:
The SNI matcher is consulted with the SNI host name. This seems to be the only place where the SNI host name is even available.
The key manager is consulted to get the alias (sort of a key ID) for a certain key type (EC or RSA). Here we need grab the host name from the SNI matcher, because otherwise we wouldn't know which host name to match on.
Based on the alias (key ID) we can then return the private key and certificate.
At least this is what I gathered from debugging this issue. The full code is online here.

Connecting to ldap using GSSAPI. Wrong service principal

I'm trying to connect to ldap server using SASL. I'm connecting using url ldaps://ldap.example.com but server hostname is host.example.com. ldap.example.com is cname for host.example.com. My program is trying to get service ticket for ldap/ldap.example.com instead of performing reverse dns request and getting ticket for ldap/host.example.com. Everything works fine when I'm using ldap://host.example.com but I prefer to use service CNAME.
There is my code for creating connection factory:
public DefaultConnectionFactory connectionFactory(){
return new DefaultConnectionFactory(connectionConfig());
}
private ConnectionConfig connectionConfig(){
final SaslConfig saslConfig = new SaslConfig();
saslConfig.setMechanism(Mechanism.GSSAPI);
final BindConnectionInitializer connectionInitializer = new BindConnectionInitializer();
connectionInitializer.setBindSaslConfig(saslConfig);
ConnectionConfig connConfig = new ConnectionConfig("ldaps://ldap.example.com");
connConfig.setConnectionInitializer(connectionInitializer);
return connConfig;
}
and jaas.config:
com.sun.security.jgss.initiate {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
keyTab="/etc/ldap.keytab"
principal="ldap#EXAMPLE.COM"
storeKey=true
useKeyTab=true
debug=true
;
};
Is there any way to change this behavior?

You should request a new certificate with ldap.example.com as the subject name and with host.example.com as a subject alternative name. The certificate negotiation is handled right before Kerberos.
A couple more suggestions:
All SPNs should be defined in your KDC:
LDAP/ldap.example.com
LDAP/host.example.com
Both of these A records should be set in DNS. Avoid use of CNAMES, while it might be OK at any given time, different browser versions and future updates could cause inconsistent behavior:
ldap.example.com
host.example.com
The principal in jaas.config and the keytab should match. You have:
principal="ldap#EXAMPLE.COM"
I suggest it should be: principal=“ldap/host.example.com“;
Finally, ldap/host.example.com should be defined as the SPN in your keytab. If it is not, it might be OK, as long as you either (1) add it as an additional SPN related in the keytab: How do you add multiple SPNs to the same keytab file for Spnego or Kerberos Configuration? or (2) see Setspn if you are using Active Directory and you application server supports it.
See further reading on GSSAPI.