Develop Reference

Hive JDBC connection problems

I am trying to connect to Hive2 server via JDBC with kerberos authentication. After numerous attempts to make it work, I can't get it to work with the Cloudera driver.
If someone can help me to solve the problem, I can greatly appreciate it.
I have this method:
private Connection establishConnection() {
final String driverPropertyClassName = "driver";
final String urlProperty = "url";
Properties hiveProperties = config.getMatchingProperties("hive.jdbc");
String driverClassName = (String) hiveProperties.remove(driverPropertyClassName);
String url = (String) hiveProperties.remove(urlProperty);
Configuration hadoopConfig = new Configuration();
hadoopConfig.set("hadoop.security.authentication", "Kerberos");
String p = config.getProperty("hadoop.core.site.path");
Path path = new Path(p);
hadoopConfig.addResource(path);
UserGroupInformation.setConfiguration(hadoopConfig);
Connection conn = null;
if (driverClassName != null) {
try {
UserGroupInformation.loginUserFromKeytab(config.getProperty("login.user"), config.getProperty("keytab.file"));
Driver driver = (Driver) Class.forName(driverClassName).newInstance();
DriverManager.registerDriver(driver);
conn = DriverManager.getConnection(url, hiveProperties);
} catch (Throwable e) {
LOG.error("Failed to establish Hive connection", e);
}
}
return conn;
}
URL for the server, that I am getting from the properties in the format described in Cloudera documentation
I am getting an exception:
2018-05-05 18:26:49 ERROR HiveReader:147 - Failed to establish Hive connection
java.sql.SQLException: [Cloudera][HiveJDBCDriver](500164) Error initialized or created transport for authentication: Peer indicated failure: Unsupported mechanism type PLAIN.
at com.cloudera.hiveserver2.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
at com.cloudera.hiveserver2.hivecommon.api.ZooKeeperEnabledExtendedHS2Factory.createClient(Unknown Source)
...
I thought, that it is missing AuthMech attribute and added AuthMech=1 to the URL. Now I am getting:
java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10100) Connection Refused: [Cloudera][JDBC](11640) Required Connection Key(s): KrbHostFQDN, KrbServiceName; [Cloudera][JDBC](11480) Optional Connection Key(s): AsyncExecPollInterval, AutomaticColumnRename, CatalogSchemaSwitch, DecimalColumnScale, DefaultStringColumnLength, DelegationToken, DelegationUID, krbAuthType, KrbRealm, PreparedMetaLimitZero, RowsFetchedPerBlock, SocketTimeOut, ssl, StripCatalogName, transportMode, UseCustomTypeCoercionMap, UseNativeQuery, zk
at com.cloudera.hiveserver2.exceptions.ExceptionConverter.toSQLException(Unknown Source)
at com.cloudera.hiveserver2.jdbc.common.BaseConnectionFactory.checkResponseMap(Unknown Source)
...
But KrbHostFQDN is already specified in the principal property as required in the documentation.
Am I missing something or is this documentation wrong?

Below is the one of the similar kind of problem statement in Impala (just JDBC engine changes others are same) that is resolved by setting "KrbHostFQDN" related properties in JDBC connection string itself.
Try to use the URL below. Hopefully works for u.
String jdbcConnStr = "jdbc:impala://myserver.mycompany.corp:21050/default;SSL=1;AuthMech=1;KrbHostFQDN=myserver.mycompany.corp;KrbRealm=MYCOMPANY.CORP;KrbServiceName=impala"
I suppose that if you are not using SSL=1 but only Kerberos, you just drop that part from the connection string and don't worry about setting up SSL certificates in the java key store, which is yet another hassle.
However in order to get Kerberos to work properly we did the following:
Install MIT Kerberos 4.0.1, which is a kerberos ticket manager. (This is for Windows)
This ticket manager asks you for authentication every time you initiate a connection, creates a ticket and stores it in a kerberos_ticket.dat binary file, whose location can be configured somehow but I do not recall exactly how.
Finally, before launching your JAVA app you have to set an environment variable KRB5CCNAME=C:/path/to/kerberos_ticket.dat. In your java app, you can check that the variable was correctly set by doing System.out.println( "KRB5CCNAME = " + System.getenv( "KRB5CCNAME" ) ). If you are working with eclipse or other IDE you might even have to close the IDE,set up the environment variable and start the IDE again.
NOTE: this last bit is very important, I have observed that if this variable is not properly set up, the connection wont be established...
In Linux, instead MIT Kerberos 4.0.1, there is a program called kinit which does the same thing, although without a graphical interface, which is even more convenient for automation.

I wanted to put it in the comment but it was too long for the comment, therefore I am placing it here:
I tried your suggestion and got another exception:
java.sql.SQLException: [Cloudera]HiveJDBCDriver Error
creating login context using ticket cache: Unable to obtain Principal
Name for authentication .
May be my problem is, that I do not have environment variable KRB5CCNAME set.
I, honestly, never heard about it before.
What is supposed to be in that ticket file.
I do have, however, following line in my main method:
System.setProperty("java.security.krb5.conf", "path/to/krb5.conf");
Which is supposed to be used by
UserGroupInformation.loginUserFromKeytab(config.getProperty("login.user"), config.getProperty("keytab.file"));
to obtain the kerberos ticket.

To solve this issue update Java Cryptography Extension for the Java version that you use in your system.
Here's the link when you can download JCE for Java 1.7
Uncompress and overwrite those files in $JDK_HOME/jre/lib/security
Restart your computer.

Exception in thread "main" com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server

I am trying to learn the SOAP protocol. So after generating the net.webservicex code with the aid of the wsimport by using the folowing command
C:\Program Files\Java\jdk1.8.0_74\bin>wsimport -keep -s src http://www.webservicex.net/geoipservice.asmx?WSDL
I placed the generated code in my project (Please take a look at the screen shoot underneath). I have added some the ip of google in run configurations --> java application --> arguments --> programm arguments--> 216.58.213.238
But when executing the main method in the IPLocationFinder class, I am getting the following error:
IPLocationFinder class
import net.webservicex.GeoIP;
import net.webservicex.GeoIPService;
import net.webservicex.GeoIPServiceSoap;
public class IPLocationFinder {
public static void main(String[] args) {
if (args.length != 1) {
System.out.println("You need to pass in one IP address");
} else {
String ipAddress = args[0];
GeoIPService ipService = new GeoIPService();
GeoIPServiceSoap geoIPServiceSoap = ipService.getGeoIPServiceSoap();
// Here is line 14.
GeoIP geoIP = geoIPServiceSoap.getGeoIP(ipAddress);
System.out.println(geoIP.getCountryName());
}
}
}
error
Exception in thread "main" com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at WebserviceX.Service.Adapter.IPAdapter.CheckIP(String IP)
at WebserviceX.Service.GeoIPService.GetGeoIP(String IPAddress)
--- End of inner exception stack trace --- Please see the server log to find more detail regarding exact cause of the failure.
at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(Unknown Source)
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(Unknown Source)
at com.sun.xml.internal.ws.client.sei.StubHandler.readResponse(Unknown Source)
at com.sun.xml.internal.ws.db.DatabindingImpl.deserializeResponse(Unknown Source)
at com.sun.xml.internal.ws.db.DatabindingImpl.deserializeResponse(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
at com.sun.proxy.$Proxy31.getGeoIP(Unknown Source)
at IPLocationFinder.main(IPLocationFinder.java:14)

Some addresses will not appear in the database and therefore cannot be mapped. This is one of the limitation of IP Address location (Geolocation).
You should try with this IP Address
212.58.246.79
It should display "United Kingdom"
You can read more about geolocation in general here: https://www.lifewire.com/does-ip-address-geolocation-really-work-818154
Hope this helps you.

This question is pretty challenging to analyze; However this exceptions simple means the service client Soap web services does not longer support the request. You are probably using the latest JDK version where the client is using the oldest one.
For example: I was using JDK1.8 and the client use jdk1.6 also the JavaBrains uses JDK 1.6.
Solution: Check the JDK version on you computer

This may also due to something wrong in the soap request that is getting passed. For example: the request username or password value might be having extra "\" or some other character. You can validate it using the SOAPUI and correct the request in the code.

TokenResponseException: missing scope when using the RemoteAPI of AppEngine in Java with OAuth 2.0 on stand-alone application

My goal is for my stand-alone application to access the datastore of a Google App Engine application so that I can query it. My application used to work with ClientLogin, but I have been asked to use OAuth 2.0 for the authentication (and using ClientLogin doesn't work anymore).
I follow the instructions on this page: https://cloud.google.com/appengine/docs/java/tools/remoteapi
I use the provided code, have made an service account, downloaded the json key, made an environment variable pointing to this key. The result is that I get the following exception:
Exception in thread "main" java.lang.ExceptionInInitializerError
at myApplication.myClass4.moveResultsOfFeature(myClass4.java:51)
at myApplication.myClass2.migrate(MyClass3.java:32)
at myApplication.myClass1.main(Starter.java:11)
Caused by: java.lang.RuntimeException: Failed to acquire Google Application Default credential.
at com.google.appengine.tools.remoteapi.RemoteApiOptions.useApplicationDefaultCredential(RemoteApiOptions.java:163)
at commonMigration.RemoteOptions.<clinit>(RemoteOptions.java:18)
... 3 more
Caused by: com.google.appengine.repackaged.com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
"error" : "invalid_scope",
"error_description" : "Empty or missing scope not allowed."
}
at com.google.appengine.repackaged.com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at com.google.appengine.repackaged.com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
at com.google.appengine.repackaged.com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
at com.google.appengine.repackaged.com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:384)
at com.google.appengine.repackaged.com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
at com.google.appengine.tools.remoteapi.RemoteApiOptions.useApplicationDefaultCredential(RemoteApiOptions.java:160)
... 4 more
which seems to point to a missing scope argument, a concern which isn't mentioned in the explication on the web page. Is there an easy way to fix this issue?
Per request, my code (simplified):
public class StackOverflow {
private static RemoteApiOptions REMOTE_OPTIONS = new RemoteApiOptions().server(
<application-id>.appspot.com, 443)
.useApplicationDefaultCredential();
public static void main(String[] args0) throws IOException {
// MAKING THE CONNECTION
RemoteApiInstaller installer = new RemoteApiInstaller();
// LOAD FROM Local
installer.install(REMOTE_OPTIONS);
try {
// MY OPERATIONS
} finally {
installer.uninstall();
}
}
}

This is a current limitation of the Remote API. See the note here:
Note: The Remote API call to useApplicationDefaultCredential() can only use credentials provided by the gcloud command.
(It's possible you followed the instructions before the note was added, since it is a recently discovered limitation). The limitation will be fixed in a future release. For now, you should either run:
gcloud auth login
And use your user account to authenticate using useApplicationDefaultCredential(). Or, you can use a service account with .useServiceAccountCredential, which accepts the service account email and a path to a p12 file instead of the json file.

Using external server restlet framework

I am using the Restlet Framework, but now I want to change to a proper server instead of using localhost.
I have already added my php files (they access the java files using the rest_server URL) to the server's folder and my java files as well, but I am not sure how to change the code so it identifies where the new location of the files is.
Here is the code from IdentiscopeServer (constructor empty):
public static void main(String[] args) throws Exception {
//setsup our security manager
if (System.getSecurityManager() == null){
System.setSecurityManager(new SecurityManager());
}
identiscopeServerApp = new IdentiscopeServerApplication();
IdentiscopeServer server = new IdentiscopeServer();
server.getServers().add(Protocol.HTTP,8888);
server.getDefaultHost().attach("", identiscopeServerApp);
server.start();
}
I guess that the correct line to change is the one with "Protocol.HTTP, 8888". If the address of my new server is http://devweb2013.co.uk/research/Identiscope, how exactly do I set this up? Is there anything else necessary for it to work apart from just moving the files to a folder in the server?
The IdensticopeServerApplication is the following:
public class IdentiscopeServerApplication extends Application {
public IdentiscopeServerApplication() {
}
public Restlet createInboundRoot() {
Router router = new Router(getContext());
//attaches the /tweet path to the TweetRest class
router.attach("/collectionPublic", CollectionPublicREST.class);
router.attach("/collectionPrivate", CollectionPrivateREST.class);
router.attach("/analysis", AnalysisREST.class);
return router;
}
}
Thank you in advance, it is my first time using this Framework.

If I understand you correctly, you just want to run your main() method as the server, correct? In this case, the code for main() needs to be in a location that -- when running -- can provide the service at http://devweb2013.co.uk/research/Identiscope. Since you haven't stated what kind of server you are putting the code, I can't say where the best place to put the code would be. I assume you have superuser privileges on your deployment server, since the URL you provided implies port 80 will be serving your Identiscope web service (port 80 is a privileged port on most OS's). So as an answer, I can only provide general information.
On your deployment server, port 80 must be free (i.e. nothing else should be acting as a web server on port 80 on that machine) and the IdentiscopeApplication must be running on port 80. To do that, you need only change the line:
server.getServers().add(Protocol.HTTP,8888);
to:
server.getServers().add(Protocol.HTTP, 80);
then run the application as a user that is allowed to start servers on port 80 (preferably NOT the superuser). If you haven't already, you will need to get Java running on your deployment server and make sure all Restlet libraries are in the classpath where you plan to run your application.
If I understand what you are trying to do, then this should do the trick.

Communication between two Machines using java

I have Gui Application written which running on windows,and i want to connect to remote unix machine and perform actions there such like API's ,go over the log file in the machines and send back to the application the last log file or others API that i want to perform on the remote machine.
In the remote machine i don;t have application server i just have Java which installed there.
I want to use Java in order to perform remote API over the remote machine;
what is the advice ,can i use web services ,can any one please advise.
Thanks in advance.

If Java can perform the actions you're talking about, I would use Sockets to communicate with the UNIX-Machine (over TCP/IP).
Your Windows-PC would be the client sending commands to the Unix-PC.

Web services would be a bit heavy handed option, esp if you opt for the SOAP ones. If you don't have a problem with the client and server always being Java, RMI seems to be the simplest solution to this problem since it's communication between two different JVM's using the normal method calling mechanism (with some additional interfaces and rules to be followed to please the RMI specification).

The Spring Framework ships with a number of remoting options that are all very easy to setup. You can use their classes for simpler configuration of something standard like RMI or JMS, or use a lightweight web services protocol such as Spring's HTTP invoker or Hessian.

For analyzing log files of remote machines you can always use Apache Commons sftp programmatically to FTP a copy of the remote log file to your PC.
If you configure the log files to be rotatable or to rotate each time they reach a specific size, you can avoid reloading the same information over and over.

You can use Ganymed SSH-2 for Java to ssh to the remote host from Client Java App and run the commands. No need to run any additional components on remote server. You can do password based authentication or key based authentication to login to remote host. We had successfully used it to administer (start/stop/grep log files, etc.) applications running on remote UNIX hosts. You can capture output of the remote command using the StreamGobbler class provided in the package. You can pass multiple commands separated by semi-colon in one remote call.
Basic Example included in the package:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import ch.ethz.ssh2.Connection;
import ch.ethz.ssh2.Session;
import ch.ethz.ssh2.StreamGobbler;
public class Basic
{
public static void main(String[] args)
{
String hostname = "127.0.0.1";
String username = "joe";
String password = "joespass";
try
{
/* Create a connection instance */
Connection conn = new Connection(hostname);
/* Now connect */
conn.connect();
/* Authenticate.
* If you get an IOException saying something like
* "Authentication method password not supported by the server at this stage."
* then please check the FAQ.
*/
boolean isAuthenticated = conn.authenticateWithPassword(username, password);
if (isAuthenticated == false)
throw new IOException("Authentication failed.");
/* Create a session */
Session sess = conn.openSession();
sess.execCommand("uname -a && date && uptime && who");
System.out.println("Here is some information about the remote host:");
/*
* This basic example does not handle stderr, which is sometimes dangerous
* (please read the FAQ).
*/
InputStream stdout = new StreamGobbler(sess.getStdout());
BufferedReader br = new BufferedReader(new InputStreamReader(stdout));
while (true)
{
String line = br.readLine();
if (line == null)
break;
System.out.println(line);
}
/* Show exit status, if available (otherwise "null") */
System.out.println("ExitCode: " + sess.getExitStatus());
/* Close this session */
sess.close();
/* Close the connection */
conn.close();
}
catch (IOException e)
{
e.printStackTrace(System.err);
System.exit(2);
}
}
}