I have a use case where the Saprk Data Set API has to connect to SQL server using jdbc to retrieve the data.
The DB is supporting the kerberos authentication thats why using Spring JTDS driver.
The code for JDBC connectivity is as :
/**
* Returns the data source for db connection
* #return
* #throws Exception
*/
private static DriverManagerDataSource getDataSource() throws Exception {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("net.sourceforge.jtds.jdbc.Driver");
String dataSourceUrl = "jdbc:jtds:sqlserver://" + "DBDEV.abc.com" + "/"
+ "TestDB";
dataSource.setUrl(dataSourceUrl);
Properties connProps = new Properties();
connProps.setProperty(DescapDataConstants.APP_NAME_PROPERTY, "Test");
connProps.setProperty(DescapDataConstants.USE_KERBEROS, Boolean.TRUE.toString());
connProps.setProperty(DescapDataConstants.LOGIN_TIMEOUT, "60");
connProps.setProperty(DescapDataConstants.SOCKET_TIMEOUT, "7200");
dataSource.setConnectionProperties(connProps);
return dataSource;
}
But there is no provision to use this Data source as per Spark JDBC API when checked on this page :
https://spark.apache.org/docs/2.3.1/api/java/org/apache/spark/sql/DataFrameReader.html#jdbc-java.lang.String-java.lang.String-java.util.Properties-
Is there any way to use Data Source for connecting to the JDBC via Spark API's.
As already noted in the comments and from a quick look at the source code here, it looks like the implementation does not indeed support a datasource lookup.
One option is to extend below source code and to extend it to accept a connection object itself and to figure out error handling and remember to close it!
Related
I am trying to connect Sybase which has set encrypt_password property to true. The JTDS version I am using is newest version I can find in Maven: 1.3.1 released in 2013 Jun.
I am getting error:ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Adaptive Server requires encryption of the login password on the network
Here is the DB Config code:
#Bean
public DataSource dataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName(dbConn.getDriverClassName());
dataSource.setUrl(dbConn.getUrl());
dataSource.setUsername(dbConn.getUsername());
dataSource.setPassword(dbConn.getPassword());
// setting driver properties
Properties props = new Properties();
for(String propPair: dbConn.getDriverProperties().split(",")) {
String[] prop = propPair.split("=");
props.setProperty(prop[0], prop[1]);
}
dataSource.setConnectionProperties(props);
return dataSource;
}
and in application.yml file:
connection:
driverProperties: "ENCRYPT_PASSWORD=true"
However this still does not work and I am getting this error:
ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Adaptive Server requires encryption of the login password on the network
I have made some search but the answer people left seems frustrating:
https://sourceforge.net/p/jtds/discussion/104389/thread/90e1bb7f/
Is there anyway I can do this property manually? Thanks in advance for reading this question and potentially if you have the answer.
I am unable to connect to cloud SQL from inside a custom DoFn while running in cloud dataflow. The errors that show up in the log are:
Connecting to Cloud SQL instance [] via ssl socket.
[Docbuilder-worker-exception]: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed
to initialize pool: Could not create connection to database server.
The same code and config work fine when connecting to cloud sql from the appenginer handle.
I have explicitly given the compute engine service account - -compute#developer.gserviceaccount.com - the Cloud SQL client, Cloud SQL viewer and Editor roles.
Any help to troubleshoot this is greatly appreciated!
To connect to Cloud SQL from external applications there are some methods that could follow in the document How to connect to Cloud SQL from external applications[1] you can find the alternatives and the steps to achieve your goal.
[1]https://cloud.google.com/sql/docs/postgres/connect-external-app
I've also run into a lot of issues when trying to use connection pooling with cloud dataflow to cloud sql with custom DoFn. Now I do not remember if my error was the same as yours, but my solution was to create an #Setup method in the DoFn class like this:
static class ProcessDatabaseEvent extends DoFn<String, String> {
#Setup
public void createConnectionPool() throws IOException {
final Properties properties = new Properties();
properties.load(Thread.currentThread().getContextClassLoader().getResourceAsStream("config.properties"));
final String JDBC_URL = properties.getProperty("jdbc.url");
final String JDBC_USER = properties.getProperty("jdbc.username");
final String JDBC_PASS = properties.getProperty("jdbc.password");
final HikariConfig config = new HikariConfig();
config.setMinimumIdle(5);
config.setMaximumPoolSize(50);
config.setConnectionTimeout(10000);
config.setIdleTimeout(600000);
config.setMaxLifetime(1800000);
config.setJdbcUrl(JDBC_URL);
config.setUsername(JDBC_USER);
config.setPassword(JDBC_PASS);
pool = new HikariDataSource(config);
}
#ProcessElement
public void processElement(final ProcessContext context) throws IOException, SQLException {
//Your DoFn code here...
}
As part of securing all the connection we are enabling ASO in Oracle database, i was able to make secure connection from my java using oracle.jdbc.pool.OracleDataSource. But we have projects using apache basic datasource. I tried the below but still the data is not getting encrypted.
BasicDataSource ods = new BasicDataSource();
ods.setUrl(URL);
ods.setUsername(user);
ods.setPassword(password);
ods.setConnectionProperties("AutoCommit=false;");
ods.setConnectionProperties("OracleConnection.CONNECTION_PROPERTY_THIN_NET_ENCRYPTION_LEVEL=REQUIRED;");
ods.setConnectionProperties("OracleConnection.CONNECTION_PROPERTY_THIN_NET_ENCRYPTION_TYPES=(AES256);");
ods.setConnectionProperties("OracleConnection.CONNECTION_PROPERTY_THIN_NET_CRYPTO_SEED=(sfdsvcfdssegdsvg);");
Please let me know if I am missing something in the connection properties. Thanks.
Method BasicDataSource#setConnectionProperties(String) overrides all properties that been added to the DataSource previously
You need to use method BasicDataSource#addConnectionProperty(String, String) to add single property:
BasicDataSource ods = new BasicDataSource();
ods.addConnectionProperty("OracleConnection.CONNECTION_PROPERTY_THIN_NET_ENCRYPTION_LEVEL", "REQUIRED")
or build Properties instance with relevant data, and only then add it to the DataSource:
Properties prop = new Properties();
prop.setProperty("OracleConnection.CONNECTION_PROPERTY_THIN_NET_ENCRYPTION_LEVEL", "REQUIRED");
prop.setProperty("OracleConnection.CONNECTION_PROPERTY_THIN_NET_ENCRYPTION_TYPES", "(AES256)")
...
BasicDataSource ods = new BasicDataSource();
ods.setConnectionProperties(prop);
I have an application that needs to use connection pool when connection to the database. Problem is, the application is designed to configure and change the connection settings on the fly. I have written the following code that will allow me to dynamically create the data source and then use that to open the connections to the database. However, this code was written for the DataDirect driver and I have used it to connect to Oracle and MS SQL. Unfortunately, the DataDirect Mysql driver will only connect to the commercial version of mysql, not the free version. So now I am attempting to accomplish the same task with connect/j from mysql. I cannot find how to set the connection properties found in a external (static) datasource definition via method calls like for the DD driver. Any help would be appreciated.
BaseDataSource bds = (BaseDataSource)ds;
// Populate the DataSource
bds.setDescription("Driver Data Source");
bds.setServerName(connectUrl);
bds.setUser(userName);
bds.setPassword(password);
if ( spyAttr.length() > 0 ) bds.setSpyAttributes(spyAttr);
// Create the PooledConnection DataSource. Pass the data source created above
// to the PooledConnection DataSource
pds = new PooledConnectionDataSource();
pds.setDescription("Pooled Data Source");
pds.setDataSourceName("myDataSource", bds);
pds.setPoolName("myPool");
pds.setInitialPoolSize(nPoolSize);
pds.setMinPoolSize(minPoolSize);
pds.setMaxPoolSize(maxPoolSize);
pds.setPropertyCycle(propCycle);
pds.setMaxIdleTime(maxIdleTime);
pds.setTracing(tracing);
// Get connection
pds.getConnection();
Apparently the MySQL Connector/J driver doesn't allow those parameters to be passed in with method calls so you will have to pass them in as parameters in the URL (jdbc:mysql://localhost/test?user=foo&password=bar).
I'm trying to follow Java's JDBC tutorials to write a Java program that can connect to SQL Server 2008. I'm getting lost at the point of making a connection.
The following snippet is from the tutorial:
InitialContext ic = new InitialContext();
DataSource ds = ic.lookup("java:comp/env/jdbc/myDB");
Connection con = ds.getConnection();
DataSource ds = (DataSource) org.apache.derby.jdbc.ClientDataSource()
ds.setPort(1527);
ds.setHost("localhost");
ds.setUser("APP")
ds.setPassword("APP");
Connection con = ds.getConnection();
There's no explanation of what comp/env/jdbc/myDB should point to, and I don't know how I should choose a port. Also, the object ds seems to be defined twice.
I'm using the JSQLDataSource driver, for the record. Can anyone point me in the right direction here?
http://java.sun.com/docs/books/tutorial/jdbc/basics/connecting.html
I'm not sure anyone above has really answered the question.
I found this microsoft sample useful.
The key information in there is really that the class you need is SQLServerDataSource
that is basically a configuration object - you use it something like this:
SQLServerDataSource dataSource = new SQLServerDataSource();
dataSource.setUser("aUser");
dataSource.setPassword("password");
dataSource.setServerName("hostname");
dataSource.setDatabaseName("db");
You would then call
dataSource.getConnection();
to get a connection object which is basically the thing you use to talk to the database.
Use
connection.prepareStatement("some sql with ? substitutions");
to make something for firing off sql and:
connection.prepareCall
for calling stored procedures.
Start with the JDBC tutorial or the Microsoft docs.
and this:
String driver = "com.microsoft.jdbc.sqlserver.SQLServerDriver";
Class.forName(driver);
String url = "jdbc:microsoft:sqlserver://host:1433/database";
Connection conn = DriverManager.getConnection(url, "username", "password");
Fill in your values for host, database, username, and password. The default port for SQL server is 1433.
UPDATE: Good point below. JDBC drivers can be had from both Microsoft and jTDS. I prefer the latter.
JNDI lookups have to do with Java EE app servers that support connection pooling. You can ask the app server to create a pool of connections, which can be an expensive thing to do, and loan them out to clients like library books as needed.
If you aren't using a Java EE app server or connection pooling, you have to create the connection on your own. That's where manual processes and DriverManager come in.
EXPLANATION: As for why the Sun tutorial shows DataSource twice, I'd say it's a case of poor editing. If you look above the code sample it says you can get a DataSource "by lookup or manually". The code snippet below shows both together, when it should be one or the other.
You know it's an inadvertent error because there's no way the code as written could compile. You have "ds" declared twice.
So it should read "...lookup", followed by its code snippet, and then "...manually", followed by its code snippet.
I like the jTDS driver for connecting to SQL Server. A URL will look like this:
jdbc:jtds:sqlserver://localhost/Finance;instance=sqlexpress
Check this for jTDS Url Info.
This also has some interesting information to help troubleshoot jtds to sql express sorts of problems.
DataSource ds = new SimpleDriverDataSource(new com.mysql.jdbc.Driver(),
"jdbc:mysql://database:1433;databaseName=name", "username", "password");
JdbcTemplate jdbc = new JdbcTemplate(ds);
This question has already been answered long time ago. The question was asked about JNDI lookup. With lookup you have to see the application server log to see what the connection is bound to. For example in Jboss startup, I can see:
[ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=myDB' to JNDI name 'java:myDB'
Using that name=myDB you lookup
InitialContext ic = new InitialContext();
DataSource ds = ic.lookup("java:myDB");
Notice how the server log and the code both point to the JNDI name java:myDB.