Develop Reference

Unable to connect hive through JDBC java

I am trying to connect hive through java(eclipse). Hive is working on the linux server, where I have installed it.
I have imported all the jar files.
I am using this code
BasicConfigurator.configure();
try {
Class.forName("org.apache.hive.jdbc.HiveDriver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
final String keyPath = "Here is the keypath";
final String user = "myuser/FQDN#REALM.COM";
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "kerberos");
System.setProperty("java.security.krb5.conf", "PATH:/krb5.conf");
System.setProperty("sun.security.krb5.debug", "true");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
UserGroupInformation ugi;
UserGroupInformation.setConfiguration(conf);
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(user, keyPath);
UserGroupInformation.setLoginUser(ugi);
System.out.println("TRYING TO CONNECT");
Connection cnct = null;
cnct = DriverManager.getConnection(
"jdbc:hive2://FQDN:10000/default;principal=ADDED PRINCIPAL HERE",
"user", "pass");
System.out.println("Connected");
cnct.close();
}
The error I am getting is:
Exception in thread "main" java.sql.SQLException: Could not open connection to jdbc:hive2://FQDN:10000/default;principal=PRINCIPAL HERE: Peer indicated failure: GSS initiate failed
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:146)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:123)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
at TestHiveJdbc.main(TestHiveJdbc.java:56)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:144)
... 5 more
Kindly help me with this.

How to force Spring boot embedded tomcat to use SNI (Server Name Indication)

I need to make sure that my client supports SNI and it works. To do that, I want to start some dummy server (Spring boot application with it's embedded Tomcat 9) that requires hostname from client. The question is how to force Spring boot embedded Tomcat to require SNI from client?
I've read that to do that, we need to provide to server 2 (or more) certificates.
I tried to follow example from this topic. But it does not work for me.
I generated 2 keystores mycompany1.keystore and mycompany2.keystore with 1 certificate in each of them. And 1 truststore mycompany.truststore that stores both these certificates. Certificates have CN my.hostname.com and my.another.hostname.com.
This is the way how I configure embedded tomcat:
#Component
public class MultipleHostsTomcatFactory {
#Value("${abc.com.key-store}")
String abcComKeyStore;
#Value("${xyz.com.key-store}")
String xyzComKeyStore;
#Value("${server.port}")
int serverPort;
#Value("${server.http.port:8080}")
private int httpPort;
#Bean
public ServletWebServerFactory servletContainer() throws Exception {
TomcatServletWebServerFactory factor = new TomcatServletWebServerFactory();
factor.addAdditionalTomcatConnectors(createSSLConnectorForMultipleHosts());
return factor;
}
private Connector createSSLConnectorForMultipleHosts() {
Connector connector = null;
try {
connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
connector.setScheme("https");
connector.setSecure(true);
connector.setProperty("SSLEnabled", "true");
connector.setProperty("defaultSSLHostConfigName", /*"*.abc.com"*/ "*.hostname.com");
connector.setPort(8444);
// *.abc.com
SSLHostConfig sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("*.abc.com");
SSLHostConfigCertificate sslHostConfigCertificate = new SSLHostConfigCertificate(sslHostConfig, SSLHostConfigCertificate.Type.EC);
sslHostConfigCertificate.setCertificateKeystoreFile(abcComKeyStore);
sslHostConfig.addCertificate(sslHostConfigCertificate);
connector.addSslHostConfig(sslHostConfig);
// *.xyz.com
sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("*.xyz.com");
sslHostConfigCertificate = new SSLHostConfigCertificate(sslHostConfig, SSLHostConfigCertificate.Type.EC);
sslHostConfigCertificate.setCertificateKeystoreFile(xyzComKeyStore);
sslHostConfig.addCertificate(sslHostConfigCertificate);
connector.addSslHostConfig(sslHostConfig);
} catch (Exception e) {
System.out.println("Catched exception!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
e.printStackTrace();
}
return connector;
}
}
This code is based on this topic.
But I have an error:
2022-01-13 16:10:21.301 ERROR 22364 --- [ main] org.apache.catalina.util.LifecycleBase : Failed to initialize component [Connector[HTTP/1.1-8444]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
...
Caused by: java.lang.IllegalArgumentException: Keystore was tampered with, or password was incorrect
...
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
...
Caused by: java.security.UnrecoverableKeyException: Password verification failed
Looks like I miss somethimg.
So, how can I force embedded Tomcat to enable SNI?

I solved the problem. Maybe my explanation will help somebody. I just followed the errors that I saw and configured all data that server missed. I don't configure certificate, because I have keystore that stores this certificate. My final server has 2 hosts that support only TLSv1.3 (for my needs):
private Connector createSSLConnectorForMultipleHosts() {
Connector connector = null;
try {
Http2Protocol http2Protocol = new Http2Protocol();
Http11NioProtocol http11Protocol = new Http11NioProtocol();
http2Protocol.setHttp11Protocol(http11Protocol);
connector = new Connector(http11Protocol);
connector.addUpgradeProtocol(http2Protocol);
connector.setScheme("https");
connector.setSecure(true);
connector.setProperty("SSLEnabled", "true");
connector.setProperty("defaultSSLHostConfigName", "my.hostname.com");
connector.setPort(8445);
// first
SSLHostConfig sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("my.hostname.com");
sslHostConfig.setCertificateKeystorePassword("mypassword");
sslHostConfig.setCertificateKeystoreFile(firstKeyStore);
sslHostConfig.setProtocols("TLSv1.3");
connector.addSslHostConfig(sslHostConfig);
// second
sslHostConfig = new SSLHostConfig();
sslHostConfig.setHostName("my.another.hostname.com");
sslHostConfig.setCertificateKeystorePassword("mypassword");
sslHostConfig.setCertificateKeystoreFile(secondKeyStore);
sslHostConfig.setProtocols("TLSv1.3");
connector.addSslHostConfig(sslHostConfig);
} catch (Exception e) {
e.printStackTrace();
}
return connector;
}

SmbException failed to connect hostname/IP_address throwing with proper credentials in Java

I need to connect to a shared folder with proper user credentials (username, password, domain).
Then when I have access to the folder, I need to list the subfolders and files in it.
Im trying with the jcifs.smb.SmbFile class and jcifs.smb.NtlmPasswordAuthentication for the authentication.
My code is the following:
NtlmPasswordAuthentication auth = new NtlmPasswordAuthentication("domainName", "userName", "password");
SmbFile smbFile = new SmbFile("smb://servername/someFolder", auth);
for (String fileName : smbFile.list()) {
System.out.println(fileName);
}
I would be able to connect to the server with these credentials, but I'm getting this error:
Exception in thread "main" jcifs.smb.SmbException: Failed to connect: servername/IP_ADDR
jcifs.util.transport.TransportException
java.net.SocketException: Connection reset
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:981)
...
Anyone has any idea why am I unable to connect?
SmbFile - https://www.jcifs.org/src/docs/api/jcifs/smb/SmbFile.html
NtlmPasswordAuthentication - https://javadoc.io/static/eu.agno3.jcifs/jcifs-ng/2.1.3/jcifs/smb/NtlmPasswordAuthentication.html

I found the solution!
Because of my OS (Windows 10) I needed to use SMB2 not SMB1 (this is the default).
Solution:
Open powershell as an administrator
You need to set a property: Set -SmbServerConfiguration -EnableSMB2Protocol $true
Optional: I think it isn't necessarry but I turned off the SMB1 protocol wit the
Set -SmbServerConfiguration -EnableSMB1Protocol $false command.
Then you can check the properties with: Get -SmbServerConfiguration command, and be sure about that all of the properties have the right value.
Import the proper dependency to the pom.xml:
<dependency>
<groupId>eu.agno3.jcifs</groupId>
<artifactId>jcifs-ng</artifactId>
<version>2.1.6</version>
</dependency>
https://github.com/AgNO3/jcifs-ng
Finally the code:
public static void sendRequest() throws Exception {
CIFSContext base = SingletonContext.getInstance();
CIFSContext authed1 = base.withCredentials(new NtlmPasswordAuthentication(base, "domainName",
"userName", "password"));
try (SmbFile f = new SmbFile("smb:\\serverName\folder", authed1)) {
if (f.exists()) {
for (SmbFile file : f.listFiles()) {
System.out.println(file.getName());
}
}
}
}

Resource FileNotFoundException when using executable jar command with spring boot

To enable https in my spring-boot app I did the following.
#Bean
#Inject
public EmbeddedServletContainerCustomizer containerCustomizer() throws FileNotFoundException
{
final String absoluteKeystoreFile = ZenoTuringServiceApp.class.getClassLoader().getResource("test.jks").getFile();
return (ConfigurableEmbeddedServletContainer factory) -> {
TomcatEmbeddedServletContainerFactory containerFactory = (TomcatEmbeddedServletContainerFactory) factory;
containerFactory.addConnectorCustomizers((TomcatConnectorCustomizer) (Connector connector) -> {
connector.setSecure(true);
connector.setScheme("https");
connector.setAttribute("keystoreFile", absoluteKeystoreFile);
connector.setAttribute("keystorePass", "test");
connector.setAttribute("keystoreType", "JKS");
connector.setAttribute("clientAuth", "false");
connector.setAttribute("sslProtocol", "TLS");
connector.setAttribute("SSLEnabled", true);
});
};
}
If I run it using mvn spring-boot:run it works as expected.
But when I run using the executable jar with java -jar target/xxx-xxx-service-0.1.17-SNAPSHOT.jar I get the FileNotFoundException.
Caused by: org.apache.catalina.LifecycleException: service.getName(): "Tomcat"; Protocol handler start failed
at org.apache.catalina.connector.Connector.startInternal(Connector.java:993)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 18 common frames omitted
Caused by: java.io.FileNotFoundException: /private/var/folders/wl/xxx77_523z44yjdgx2y7xxxc217h/T/tomcat.8061417798873093914.8091/file:/Users/xxx/Work/Himalay/xxx/xxx-xxx-service/target/xxx-xxx-service-0.1.17-SNAPSHOT.jar!/test.jks (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore(JSSESocketFactory.java:433)
at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeystore(JSSESocketFactory.java:339)
at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:597)

The comments above and this code helped me..
// support loading the JKS from the classpath (to get around Tomcat limitation)
private static File getTuringKeyStoreFile() throws IOException {
ClassPathResource resource = new ClassPathResource("test.jks");
// Tomcat won't allow reading File from classpath so read as InputStream into temp File
File jks = File.createTempFile("ssl_keystore", ".jks");
InputStream inputStream = resource.getInputStream();
try {
FileUtils.copyInputStreamToFile(inputStream, jks);
} finally {
IOUtils.closeQuietly(inputStream);
}
return jks;
}
Reference: https://github.com/robinhowlett/everything-ssl

Kafka Java Producer with kerberos

Getting error while sending message to kafka topic in kerberosed enviornment. We have cluster on hdp 2.3
I followed this http://henning.kropponline.de/2016/02/21/secure-kafka-java-producer-with-kerberos/
But for sending messages, I have to do kinit explicitly first, then only I am able to send message to kafka topic.
I tried to do knit through java class but that also doesn't work.
PFB code:
package com.ct.test.kafka;
import java.util.Date;
import java.util.Properties;
import java.util.Random;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class TestProducer {
public static void main(String[] args) {
String principalName = "ctadmin";
String keyTabPath = "/etc/security/keytabs/ctadmin.keytab";
boolean authStatus = CTSecurityUtil.loginUserFromKeytab(principalName, keyTabPath);
if (!authStatus) {
System.out.println("Authntication fails, try something else " + authStatus);
} else {
System.out.println("Authntication successfull " + authStatus);
}
System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");
System.setProperty("java.security.auth.login.config", "/etc/kafka/2.3.4.0-3485/0/kafka_jaas.conf");
System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
System.setProperty("sun.security.krb5.debug", "true");
try {
long events = Long.parseLong("3");
Random rnd = new Random();
Properties props = new Properties();
System.out.println("After broker list- " + args[0]);
props.put("metadata.broker.list", args[0]);
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
props.put("security.protocol", "PLAINTEXTSASL");
//props.put("partitioner.class", "com.ct.test.kafka.SimplePartitioner");
System.out.println("After config prop -1");
ProducerConfig config = new ProducerConfig(props);
System.out.println("After config prop -2 config" + config);
Producer<String, String> producer = new Producer<String, String>(config);
System.out.println("After config prop -3");
for (long nEvents = 0L; nEvents < events; nEvents += 1L) {
Date runtime = new Date();
String ip = "192.168.2" + rnd.nextInt(255);
String msg = runtime + " www.example.com, " + ip;
KeyedMessage<String, String> data = new KeyedMessage<String, String>("test_march4", ip, msg);
System.out.println("After config prop -1 data" + data);
producer.send(data);
}
producer.close();
} catch (Throwable th) {
th.printStackTrace();
}
}
}
Pom.xml : All dependency downloaded from hortonworks repo.
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.9.0.2.3.4.0-3485</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.2.3.4.0-3485</version>
</dependency>
<dependency>
<groupId>org.jasypt</groupId>
<artifactId>jasypt-spring31</artifactId>
<version>1.9.2</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.1.2.3.4.0-3485</version>
</dependency>
</dependencies>
Error :
Case1 : when I specify myuser kafka_jass.conf
log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
After config prop -2 configkafka.producer.ProducerConfig#643293ae
java.lang.SecurityException: Configuration Error:
Line 6: expected [controlFlag]
at com.sun.security.auth.login.ConfigFile.<init>(ConfigFile.java:110)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:379)
at javax.security.auth.login.Configuration$2.run(Configuration.java:258)
at javax.security.auth.login.Configuration$2.run(Configuration.java:250)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:249)
at org.apache.kafka.common.security.kerberos.Login.login(Login.java:291)
at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104)
at kafka.common.security.LoginManager$.init(LoginManager.scala:36)
at kafka.producer.Producer.<init>(Producer.scala:50)
at kafka.producer.Producer.<init>(Producer.scala:73)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:26)
at com.ct.test.kafka.TestProducer.main(TestProducer.java:51)
Caused by: java.io.IOException: Configuration Error:
Line 6: expected [controlFlag]
at com.sun.security.auth.login.ConfigFile.match(ConfigFile.java:563)
at com.sun.security.auth.login.ConfigFile.parseLoginEntry(ConfigFile.java:413)
at com.sun.security.auth.login.ConfigFile.readConfig(ConfigFile.java:383)
at com.sun.security.auth.login.ConfigFile.init(ConfigFile.java:283)
at com.sun.security.auth.login.ConfigFile.init(ConfigFile.java:219)
at com.sun.security.auth.login.ConfigFile.<init>(ConfigFile.java:108)
MyUser_Kafka_jass.conf
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useTicketCache=true
renewTicket=true
principal="ctadmin/prod-dev1-dn1#PROD.COM";
useKeyTab=true
serviceName="kafka"
keyTab="/etc/security/keytabs/ctadmin.keytab"
client=true;
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/ctadmin.keytab"
storeKey=true
useTicketCache=true
serviceName="zookeeper"
principal="ctadmin/prod-dev1-dn1#PROD.COM";
};
case2 : When I specify Kafkas own jaas file
Java config name: /etc/krb5.conf
Loaded from Java config
javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. Make sure -Djava.security.auth.login.config property passed to JVM and the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)'. Make sure you are using FQDN of the Kafka broker you are trying to connect to. not available to garner authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:899)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.kafka.common.security.kerberos.Login.login(Login.java:298)
at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104)
at kafka.common.security.LoginManager$.init(LoginManager.scala:36)
at kafka.producer.Producer.<init>(Producer.scala:50)
at kafka.producer.Producer.<init>(Producer.scala:73)
at kafka.javaapi.producer.Producer.<init>(Producer.scala:26)
at com.ct.test.kafka.TestProducer.main(TestProducer.java:51)
This works fine, if I do kinit before running this app, else it will through above error.
I cant do this in my production environment, if there is any way to do this by our app itself then please help me out.
Please let me know if you need any more details.
Thanks:)

The error is in a semicolon you have in your jaas file as you can see in this piece of output:
Line 6: expected [controlFlag]
This line cannot have the semicolon:
principal="ctadmin/prod-dev1-dn1#PROD.COM";
it can only exist in the last line:

I don't know what mistake did first time, below things I did again, and it works fine.
First give all access to topic:
bin/kafka-acls.sh --add --allow-principals user:ctadmin --operation ALL --topic marchTesting --authorizer-properties zookeeper.connect={hostname}:2181
create jass file:
kafka-jaas.conf
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useTicketCache=true
principal="ctadmin#HSCALE.COM"
useKeyTab=true
serviceName="kafka"
keyTab="/etc/security/keytabs/ctadmin.keytab"
client=true;
};
Java Program:
package com.ct.test.kafka;
import java.util.Date;
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class KafkaProducer {
public static void main(String[] args) {
String topic = args[0];
Properties props = new Properties();
props.put("metadata.broker.list", "{Hostname}:6667");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("request.required.acks", "1");
props.put("security.protocol", "PLAINTEXTSASL");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
for (int i = 0; i < 10; i++){
producer.send(new KeyedMessage<String, String>(topic, "Test Date: " + new Date()));
}
}
}
Run application:
java -Djava.security.auth.login.config=/home/ctadmin/kafka-jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=true -cp kafka-testing-0.0.1-jar-with-dependencies.jar com.ct.test.kafka.KafkaProducer