I'm using official JDBC driver for PostgreSQL, but I'm stuck with the following issues:
No support for PostgreSQL-ish data structures such as UUIDs.
Common JDBC weirdness, such as:
No function to escape values for consuming by PostgreSQL.
Limited support for executing heterogeneous statements in batch.
No rewriting of multiple insert statements into single insert statement when inserting many rows in one table.
So, the question — is there any PostgreSQL database driver which can leverage full power of PostgreSQL without much boilerplate? I'm also use Scala language for development, so if driver is designed specifically for Scala it would be so much awesome awesome.
Some of this seems to be (unless I'm not understanding) user error in using JDBC. JDBC is a pretty ugly API, so never ask if you can do it elegantly, just ask if you can do it at all.
Escaping and inserting multiple rows should be handled, as #ColinD and #a_horse pointed out, with Prepared statements and batch operations. Under the hood, I would expect a good JDBC implementation to do the things you want (I am not familiar with PostgreSQL's implementation).
Regarding UUIDs, here is a solution:
All that PostgreSQL can do is convert string literals to uuid.
You can make use of this by using the data type
org.postgresql.util.PGobject, which is a general class used to
represent data types unknown to JDBC.
You can define a helper class:
public class UUID extends org.postgresql.util.PGobject {
public static final long serialVersionUID = 668353936136517917L;
public UUID(String s) throws java.sql.SQLException {
super();
this.setType("uuid");
this.setValue(s);
}
}
Then the following piece of code will succeed:
java.sql.PreparedStatement stmt =
conn.prepareStatement("UPDATE t SET uid = ? WHERE id = 1");
stmt.setObject(1, new UUID("a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11"));
stmt.executeUpdate();
The driver supports batched statements to speed up bulk inserts.
And using batched statements is a lot more portable than using proprietary INSERT syntax (and as far as I can tell, there is no big different between a multi-row insert and batched inserts)
Check out PreparedStatement.addBatch()
The reason why UUID is not supported is probably that UUID is not part of the Postgres core, just a contrib module.
Edit
Regarding the execute heterogeneous statements
The Postgres driver does support different types of statements in the a batch.
The following works fine:
Connection con = DriverManager.getConnection("jdbc:postgresql://localhost/postgres", "foo", "bar");
con.setAutoCommit(false);
Statement stmt = con.createStatement();
stmt.addBatch("create table foo (id integer, data varchar(100))");
stmt.addBatch("insert into foo values (1, 'one')");
stmt.addBatch("insert into foo values (2, 'two')");
stmt.addBatch("update foo set data = 'one_other' where id = 1");
stmt.executeBatch();
con.commit();
Although you do lose the automatic escaping that PreparedStatement gives you.
I realise this doesn't answer your entire question, but hopefully it will be useful all the same.
I'm using Java 6 and Postgres 8.4. The driver I'm using is in my Maven POM file as:
<dependency>
<groupId>postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>8.4-702.jdbc4</version>
</dependency>
I'm using PreparedStatement.getObject() and PreparedStatement.setObject() with Java's java.util.UUID class to retrieve and store UUIDs.
For example:
pstm.setObject(1, guid); //where pstm is a PreparedStatement and guid is a UUID
and:
//where rs is a ResultSet
UUID myGuid = (UUID) rs.getObject("my_uuid_column_name");
Works fine.
With newer drivers, the following is alsow supported
UUID myGuid = rs.getObject("my_uuid_column_name", UUID.class);
No support for PostgreSQL-ish data structures such as UUIDs.
On the contrary, the current JDBC driver (9.2-1002 JDBC 4) for Postgres 9.x does indeed support UUID via the setObject and getObject commands. You cannot get any more direct or simpler than that (in any database, Postgres or any other) because JDBC does not recognize UUID as a data type.
As far as I can tell, there is no need to create a helper class as suggest in another answer by Yishai.
No need to do any casting or go through strings.
See my blog post for more discussion and code example.
Code example excerpt:
java.util.UUID uuid = java.util.UUID.randomUUID();
…
preparedStatement.setObject( nthPlaceholder++, uuid ); // Pass UUID to database.
Take a look at O/R Broker, which is a Scala JDBC-based library for relational database access.
Related
I want to integrate jOOQ in a project which uses native SQL and org.springframework.jdbc.core.JdbcTemplate. The database schema changes often and developers have to search the code for a column that has been removed and update the queries. Hibernate or another ORM solution isn't an option as queries are sometimes very large and developers get them and just have to insert into the application code. This is very error-prune. So I thought to integrate jOOQ but gradually.
Can jOOQ throw compile time errors if a native SQL cannot be executed?
I've tried the following:
// Existing code
JdbcTemplate jdbcTemplate = ...
String sql = "select ...";
// Check code
try {
DSLContext dslContext = DSL.using(jdbcTemplate.getDataSource().getConnection());
Query query = dslContext.parser().parseQuery(sql + " order by NON_EXISTING_COLUMN");
} catch (SQLException e) {
...
}
// Existing code
return jdbcTemplate.query(sql, ...)
But it compiles well although NON_EXISTING_COLUMN doesn't really exist. Maybe I know the jOOQ API not very well yet. But I thought that it should be capable of doing it. The table classes are generated, so it can check whether the query which is build from a native SQL string is executable without executing it. Isn't it?
why don't use generated class files rather than typing native SQL?
dslContext.selectFrom(GENERATED_TABLE)...
PS: If you use JOOQ DSL(with generated classes) for generating SQL it would be work but not completely in some cases if you do mistakes even on generated classes (relations or some aggregate functions) it will not complain about it and will compile successfully.
Older versions of jOOQ did not implement meta data lookups in the parser, or only to some limited extent, and just accepted all valid identifiers. Starting with jOOQ 3.14, there will be an improved implementation as of:
https://github.com/jOOQ/jOOQ/issues/9061
You will need to provide jOOQ with a Configuration.metaProvider() (it defaults to the JDBC DatabaseMetaData backed implementation. You might prefer using generated code), and then jOOQ will try to resolve all identifiers.
Prior to jOOQ 3.14, you could implement a VisitListener which traverses the parsed SQL query, and validate all the identifiers yourself.
This is just a theoretical question, but I am building a program that gets data from Facebook using the CDATA JDBC Plugin, I wanted to know if all JDBC Plugins have the same syntax. For example, if I just change the JAR file for the driver to a Twitter one, and change the names of the tables and columns I am accessing, would it still work?
By a plugin I mean a driver, also, to put it more clearly, if I was developing a MySQL app and switched from the stock Connector/J Driver to the CData driver, would I need to change the code?
Until the underlying schema where you store remains same, the use of JDBC driver will yield the same result.
Note: Twitter/FB... both has to support the JDBC Model...
However, if you have changes in Drivers, you can consider using ApacheMetamodel Link for reference
JDBC is a standard that has been established and vetted over the years. As long as the drivers you're working with are written to that standard (which as a CData employee, I can say that ours are) you can expect your code referencing a JDBC driver to be essentially identical, regardless of the manufacturer of the driver or the data source you're connecting to.
//optional, register the driver with the DriverManager
Class.forName(myDriverName).newInstance();
//obtain a Connection instance from the DriverManager
Connection conn = null;
try {
conn = DriverManager.getConnection(myJDBCurl);
//execute a select query
Statement stmt = conn.createStatement();
Result rs = stmt.executeQuery("SELECT foo FROM bar");
} catch (SQLException ex) {
//handle any errors
}
As you can see, the code to utilize the JDBC driver can be generalized with variables to use any driver or to use different connections under a single driver (if, for instance, you wanted to connect to different Facebook accounts).
JDBC is an interesting standard. It was intentionally designed to load the driver at run-time, so no vendor classes are used during compilation.
It also has some JDBC own mechanisms for schema data (DatabaseMetaData), and for such things as doing an INSERT with an autoincrement key, and retrieving that key (getGeneratedKeys).
However the SQL is far from standardized by vendor, despite standardisation efforts. For example just getting the first 10 rows.
Unfortunately the visionaries of JDBC seem no longer to exist.
But it a sound basis for professional usage.
I am working with some legacy code that performs database operations in a generic way, so that the User/developer can work with a different database by changing only the JDBC driver.
I have a problem with PostgreSQL JDBC driver. My test case:
//ddl
CREATE TABLE test
(
id numeric,
name text,
)
//java code
String sqlCmd = "INSERT INTO test values (?, ?)";
PreparedStatement ps = connection.prepareStatement( sqlCmd );
ps.setString( 1, "1" );
ps.setString( 1, "name1" );
ps.executeUpdate();
With Postgres, the result of this case is an exception with message: "can't cast string to int..."
Is it inappropriate to use PreparedStatement.setString() to set values that database expects to be numeric?
Should I expect the JDBC driver to automatically convert java types to database types?
This test passes with other databases, including H2 and MySQL. Does the failure with PostgreSQL reflect a bug in the JDBC driver? Is it possible to make this case work without changing code?
The documentation for java.sql.PreparedStatement has this to say:
Note: The setter methods (setShort, setString, and so on) for setting IN parameter values must specify types that are compatible with the defined SQL type of the input parameter. For instance, if the IN parameter has SQL type INTEGER, then the method setInt should be used.
Whether a particular database or JDBC driver allows you to be sloppy about that is its own affair, but you are not justified in expecting that all drivers will allow such slop, even if certain ones do.
while migrating from oracle database to postgresql, I found that may help you to use setString with numeric types and date types as well.
Just you have to use the connection parameter stringtype=specified as mentioned in the documentation.
https://jdbc.postgresql.org/documentation/head/connect.html
You are using setString() method to insert integers and postgres could not do that, Use
ps.setInt(1, INTEGER_VALUE);
The latest Java JDBC drivers for postgres claim to support UUIDs natively; working against Postgres 9.2 (mac).
Indeed, when a PreparedStatement is used, I can step through the driver code, and even walk
through the specialised 'setUuid' function in AbstractJdbc3gStatement.java. By all indications, it should 'just work'.
However, it does not work. The database flings back an error, which I receive thus:
Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: uuid = bytea
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Position: 139
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) ~[postgresql-9.2-1002.jdbc4.jar:na]
Yes, indeed, setUuid in the JDBC driver does send that as a bytea :
private void setUuid(int parameterIndex, UUID uuid) throws SQLException {
if (connection.binaryTransferSend(Oid.UUID)) {
byte[] val = new byte[16];
ByteConverter.int8(val, 0, uuid.getMostSignificantBits());
ByteConverter.int8(val, 8, uuid.getLeastSignificantBits());
bindBytes(parameterIndex, val, Oid.UUID);
} else {
bindLiteral(parameterIndex, uuid.toString(), Oid.UUID);
}
}
What gives?
Is there some magic rune required in the actual database to bless this conversion ?
tl;dr
myPreparedStatement.setObject(
… ,
java.util.UUID.randomUUID()
)
Details
(a) Show us your code.
PreparedStatement::setObject does work when passing a java.util.UUID. You likely have some other issue in your code.
(b) See my blog post UUID Values From JDBC to Postgres for a bit of discussion and example code.
// Generate or obtain data to store in database.
java.util.UUID uuid = java.util.UUID.randomUUID(); // Generate a random UUID.
String foodName = "Croissant";
// JDBC Prepared Statement.
PreparedStatement preparedStatement = conn.prepareStatement( "INSERT INTO food_ (pkey_, food_name_ ) VALUES (?,?)" );
int nthPlaceholder = 1; // 1-based counting (not an index).
preparedStatement.setObject( nthPlaceholder++, uuid );
preparedStatement.setString( nthPlaceholder++, foodName );
// Execute SQL.
if ( !( preparedStatement.executeUpdate() == 1 ) ) {
// If the SQL reports other than one row inserted…
this.logger.error( "Failed to insert row into database." );
}
(c) I'm not sure what you mean by
The latest Java JDBC drivers for postgres claim to support UUIDs natively
Which driver? There are at least two open-source JDBC drivers for Postgres, the current/legacy one and a new rewrite "next generation" one. And there are other commercial drivers as well.
"natively"? Can you link to the documentation you read? The SQL spec has no data type for UUID (unfortunately ☹), therefore the JDBC spec has no data type for UUID. As a workaround, the JDBC driver for Postgres uses the setObject and getObject methods on PreparedStatement move the UUID across the chasm between Java ↔ SQL ↔ Postgres. See the example code above.
As the PreparedStatement JDBC doc says:
If arbitrary parameter type conversions are required, the method setObject should be used with a target SQL type.
Perhaps by "natively", you confused Postgres' native support for UUID as a data type with JDBC having a UUID data type. Postgres does indeed support UUID as a data type, which means the value is stored as 128-bits rather than multiple times that if it were stored as as ASCII or Unicode hex string. And being native also means Postgres knows how to build an index on a column of that type.
The point of my blog post mentioned above was that I was pleasantly surprised by how simple it is to bridge that chasm between Java ↔ SQL ↔ Postgres. In my first uneducated attempts, I was working too hard.
Another note about Postgres supporting UUID… Postgres knows how to store, index, and retrieve existing UUID values. To generate UUID values, you must enable the Postgres extension (plugin) uuid-ossp. This extension wraps a library provided by The OSSP Project for generating a variety of kinds of UUID values. See my blog for instructions.
By the way…
If I knew how to petition the JDBC expert group or JSR team to make JDBC aware of UUID, I certainly would. They are doing just that for the new date-time types being defined in JSR 310: Date and Time API.
Similarly, if I knew how to petition the SQL standards committee to add a data type of UUID, I would. But apparently that committee is more secretive than the Soviet Politburo and slower than a glacier.
I used the following approach to add UUID and other objects to postgres:
PGobject toInsertUUID = new PGobject();
toInsertUUID.setType("uuid");
toInsertUUID.setValue(uuid.toString());
PreparedStmt stmt = conn.prepareStatement(query);
stmt.setObject(placeHolder,toInsertUUID);
stmt.execute();
This way you will be stopping yourself from doing type casting. This piece of code worked perfectly for me for any time for example even json.
This worked for me using the org.postgresql.postgresql 42.2.5
myPreparedStatement.setObject(4, UUID.randomUUID(),java.sql.Types.OTHER)
Without java.sql.Types.OTHER I got an error
try
.setParameter("uuid", uuid, PostgresUUIDType.INSTANCE);
I'm trying to determine the best way to ping a database via JDBC. By 'best' I mean fast and low overhead. For example, I've considered executing this:
"SELECT 1 FROM DUAL"
but I believe the DUAL table is Oracle-specific, and I need something more generic.
Note that Connection has an isClosed() method, but the javadoc states that this cannot be used to test the validity of the connection.
With JDBC 4 you can use isValid(int) (JavaDoc) from the Connection Interface. This basically does the trial statement for you.
Some driver implement this by sending the correct dummy SQL to the database and some directly uses low level operations which reduces the parsing overhead.
However beware of the timeout, some drivers (DB/400 and Oracle Thin) do spawn a new time thread for each invocation, which is not really acceptable for most Pool validation scenarios). And Oracle also does not seem to use a prepared statement, so it’s kind of relying on the implicit cache.
Yes, that would be Oracle-only, but there is no generic way to do this in JDBC.
Most connection pool implementations have a configuration parameter where you can specify the SQL that will be used for ping, thus pushing the responsiblity to figure out how to do it to the user.
That seems like the best approach unless someone comes up with a little helper tool for this (of course, it precludes using potentially even faster non-SQL-based methods like Oracle's internal ping function)
MySQL has a nice mechanism, documented in this SO answer. From the answer:
"/* ping */ SELECT 1"
This will actually cause the driver send a ping to the server and return a fake, light-weight, result set.
Having said that, #eckes answer is the best (using JDBC 4's Connection.isValid(int)).
I'm not aware of a generic solution, either. For IBM's UDB on iSeries (and perhaps other DB2 systems) it would be
select 1 from SYSIBM.SYSDUMMY1;
You could try to get the db name from the connection meta data and execute a matching sql staement. E.g.
Connection con = null;
Statement st = null;
ResultSet rs = null;
try {
con = dataSource.getConnection();
String dbProductName = con.getMetaData().getDatabaseProductName();
Statement st = con.createStatement();
if ( "PostgreSQL".equalsIgnoreCase(dbProductName) ) {
rs = st.executeQuery("select version();");
} else if ( "Oracle".equalsIgnoreCase(dbProductName) ) {
rs = st.executeQuery("select 1 from dual");
} else {
...
}
} catch ( Exception ex ) {
System.out.prinln("DB not reachable");
} finally {
// close statement, connection etc.
...
}
I may be out to lunch on this one, but could you simply execute some non-sense query, such as:
SELECT * FROM donkey_giraffe_87
I don't know very much about JDBC's error handling, but perhaps you could check to see if the database is at least telling you that the table does not exist. If JDBC's error codes are vendor-specific, the Spring Framework has some utilities for mapping these codes to more meaningful exceptions.