I've an application created using dropwizard framework where I've registered a quartz-scheduler job scheduled to run after every specified duration. This job fires a SQL query to SQL Server DB and iterates the ResultSet and sets the data to a POJO class which is later pushed to a queue.
The SQL query has UNION joining multiple tables which fetches the data for the records modified in a delta time using the last_modified_time column of the related table in where clause. DB jar included in pom.xml is sqljdbc-4.4.0 and quartz version is 2.2.1
The query looks like this:
SELECT
u.last_modified_date,
u.account_id,
u.user_id,
ud.is_active
FROM user u WITH (NOLOCK)
JOIN user_details ud with (NOLOCK) ON u.account_id = ud.account_id AND u.user_id = ud.user_id
WHERE u.last_modifed_date > ? AND ud.last_modifed_date <= ?
UNION
SELECT
u.last_modified_date,
u.account_id,
u.user_id,
ud.is_active
FROM user u WITH (NOLOCK)
JOIN user_details ud with (NOLOCK) ON u.account_id = ud.account_id AND u.user_id = ud.user_id
JOIN user_registration_details urd WITH (NOLOCK) ON urd.account_id = u.account_id AND urd.user_id = u.user_id AND urd.reg_id = ud.reg_id
WHERE urd.last_modifed_date > ? AND urd.last_modifed_date <= ?
This query is called by simple connection statement and resultset like this
final ManagedDataSource datasource configuration.getDatabase().build(environment.metrics(), "sql");
// configuration is the configuration class in a drop wizard application and configuration.getDatabase() returns
// the DataSourceFactory with all credentials like user, password and url set into it
try (Connection conn = dataSource.getConnection()) {
int resultSetType = SQLServerResultSet.TYPE_SS_SERVER_CURSOR_FORWARD_ONLY;
int resultSetConcurrency = ResultSet.CONCUR_READ_ONLY;
LOGGER.info("Starting execution: ");
try (PreparedStatement pstmt = conn.prepareStatement(getQuery(), resultSetType,resultSetConcurrency))
{
setQueryParameters(pstmt);
try (ResultSet rs = pstmt.executeQuery();)
{
//process results
}
}
} catch (SQLException | IOException ex) {
LOGGER.error(“Error occurred “ + ex);
}
LOGGER.info("Completed execution: ");
In a simple execution, it prints the logs "Starting execution" and then processes the records and prints "completed execution". But sometimes during the execution, it's printing the logs "Starting execution" and "completed execution" but this query is not actually fired to the SQL DB.
As I didn't get the records which I modified in that delta time, I put the profiler to check if the query is actually fired and didn't found this query firing to the DB. Also, I've tried adding log4jdbc library http://code.google.com/p/log4jdbc/wiki/FAQ to print the query to the logs but no logs were printed for this query.
with (NOLOCK) is not MySQL syntax. Look at the settings in the wizard and see if you have specified the correct RDBMS engine. In particular, it sounds like SQL Server syntax.
The equivalent may involve setting the TRANSACTION ISOLATION LEVEL to something like READ UNCOMMITTED.
I turned the SQL Profiler on this query and filtered it for my server to check if the query was actually hitting the DB from my application and found that the profiler could only find it rarely hitting the DB. So, I thought there might be some caching happening at mybatis level. Then I added more logs and performed debug analysis on mybatis by enabling all kind of logging and to check if there are any caching at local level or second level cache mybatis configuration but that wasn't the cause.
Then I used spy jdbc driver log4jdbc driver to log all the queries, parameters and all db information requests at db level.
My organization is using Spunk application to display logs from all the applications deployed to the different host servers. While checking the logs in splunk, I observed the same query was printed twice in the logs but when I noticed deeply it was printing one from my instance and another execution from the different instance deployed on some other server. I logged in to that server and found the same application deployed over there not updated since months. It was found to be multiple application instances running in the same environment but on two different servers and there is no way I could figure out that an application was deployed on multiple hosts.
Thank you #halfer for all the help and bounty.
Related
I am new to databases and transactions. I find a lot of different information up-to-date and am currently trying to organize my thoughts.
Regarding the context, I'm trying to test the current isolation level with the SQL Server, but I don't succeed.
For this I use Spring Transaction together with Eclipse Link. I found some information about Spring transactions (https://www.marcobehler.com/guides/spring-transaction-management-transactional-in-depth). However, there is still a concept of the works unit and now I don't know what is currently being used (https://wiki.eclipse.org/Introduction_to_EclipseLink_Transactions_(ELUG)#Unit_of_Work_Architecture)
What I am trying to test:
I have an Entity User (id, firstname, lastname). I have an entry in table id = 1, firstname = foo lastname = bar
I have a service and transaction. IMHO default is isolation READ_COMMIT for SQL Server
#Transactional
public User updateUser(){
User updateUser = new User(1, "new firstname", "new lastname");
User updatedUser = userRepository.save(updateUser); --> em.merge
return updatedUser;
}
So far so good. What I do not understand now. I set a breakpoint in the return.
At the same time I opened a second SQL client and executed the following SQL.
SET IMPLICIT_TRANSACTIONS ON
UPDATE
User
SET
lastname = 'complete new',
WHERE
id = 1
COMMIT TRAN
What I would expect is that the SQL statement will wait for the Spring transaction to complete. BUT this is currently not the case, the SQL statement is simply carried out.
Then lastname is in the table lastname "complete new" then I resume the breakpoint and then the lastname is "new lastname". This behavior I cannot understand. Is this normal or is this because of the unit work of eclipse link?
EclipseLink buffers SQL statements that need to be executed for as long as possible in order to reduce the lock time in the RDBMS. In your particular case the JDBC driver will receive the UPDATE statement when Spring Data JPA commits the transaction. You can verify it by enabling SQL logging in EclispeLink: stackoverflow.com/q/2374395/17695211
After enabling SQL logging you'll see that there won't be any SQL debug output in the console at your breakpoint. It will appear after the return. If you really want to see the locking effect, you need to write the repository without Spring Data JPA with a #PersistenceContext-injected EntityManager and call EntityManager.flush which will flush SQL statement buffer of EclipseLink to the JDBC driver before the breakpoint.
Alternatively (and if you feel adventurous), you may try looking for a place in the EclipseLink source code where it executes the corresponding PreparedStatement, and set the breakpoint right after it.
I have a java application that does a SQL query against an Oracle database, that for some reason gives way less values when executed from the SQL Developer and from the application itself.
Now to the technicalities. The application produces a connection to the db using a wrapper library that employs c3p0. The c3p0 config has been checked, so we know that this things can't be:
-Pointing to wrong database/schema
-Restricted user
Then there's the query:
select to_char(AGEPINDW.TRANSACTION.TS_TRANSACTION,'yyyy-mm') as Time,result, count(*) as TOTAL, sum(face_value) as TOTAL_AMOUNT
from AGEPINDW.TRANSACTION
where (ts_transaction >= to_timestamp(to_char(add_months(sysdate,-1),'yyyy-mm'),'yyyy-mm')
and ts_transaction < to_timestamp(to_char(sysdate,'yyyy-mm'),'yyyy-mm')) and service_id in (2,23)
group by to_char(AGEPINDW.TRANSACTION.TS_TRANSACTION,'yyyy-mm'), result;
It doesn't have any parameter and is executed via your standard PreparedStatement. Yet the return from the app is wrong and I don't know what may be. Any suggestions?
At the first stage i'm managing my app via JDBC, so i'm the resposible to build and validate all the SQL. But i was wondering if JPA could give me a hand on these tasks.
So, at this moment i've already have a DbaUser model, which was generated from the DBA_USERS table on the OracleDB, and i can actually list all of them.
However, i'm trying to manage to create or update more, but whenever i try to create using
em.createQuery("CREATE USER C##ANTONIO IDENTIFIED BY Antionio123").executeUpdate();
An syntax exception is launched: The query does not start with a valid identifier, has to be either SELECT, UPDATE or DELETE FROM.
Could you guys enlighten me a bit more or pointing me to some proper tutorials? I've been googling but nothing concrete apprears on Oracle DBs system tables.
Update1 (Thanks to #JB Nizet)
After replacing the execution of the query from JPQL to Native SQL, i've got an error such as:
Query: DataModifyQuery(sql="CREATE USER C? IDENTIFIED BY ANTONIO123").
I've replaced the hashtags with a scape character "...C##..." with "...C\#\#..." but the issue earns a different flavour:
Query: DataModifyQuery(sql="CREATE USER C\? IDENTIFIED BY ANTONIO123")
... and i really need to send the "##" to the Oracle DB. How can i force these special characters?
Update 2
So...after googling a bit more, i've found out positional parameters, and i've also discovered that we cannot have named paramteres on JPA native queries. After this, i've tried:
em.createNativeQuery("CREATE USER ?1 IDENTIFIED BY ANTONIO123").setParameter(1, "C##ANTONIO").executeUpdate();
Which triggers: ORA-01935: missing user or role name
alongside with
Error Code: 1935
Call: CREATE USER ? IDENTIFIED BY ANTONIO123
Which tells me that this binding doesn't work. Is there another way to do it?
Kind regards and thanks in advance,
Sammy
createQuery() expects a JPQL query. What you passed is not JPQL. It's SQL.
Use createNativeQuery() to execute SQL.
To create a common user (prefixed with C##) you (i.e. your JPA connection pool user) need a specific priviledges.
CREATE ROLE and
SET CONTAINER
Those are not a typical privileges granted to a JPA connection, so I'm guessing you will fail with the creation of a new common user.
Additionally you need to be connected to the root container.
The further example are plain JDBC called from Groovy, it should be easy possible to pass it to JPA if you get the DB connection.
def stmt = con.prepareStatement("SELECT SYS_CONTEXT('USERENV', 'CON_NAME') CON_NAME FROM dual")
def rs = stmt.executeQuery()
while(rs.next())
{
println "container name= ${rs.getString('CON_NAME')}"
}
gives
container name= CDB$ROOT
Note that if you are connected to a local DB, you get an error while trying to create a user prefixed with C##
ORA-65094: invalid local user or role name
If both conditions are fulfilled, it is possible to create / drop the common user:
String cu = "create user \"C##TEST\" identified by password123 profile \"DEFAULT\" account unlock"
con.createStatement().execute(cu)
resp.
cu = "drop user \"C##TEST\""
con.createStatement().execute(cu)
Finally should be stated, that this exercise was done for the aim of completeness only. I do not see a real use case for a JPA pool connection to be granted such privileges and connecting the root container. The database maintenance is typically done not using JPA.
Another option would be to create a stored procedure in the database and then invoke stored procedure from JPA, thus you do not need to bother about caveats and syntax.
E.g.assume that JPA provider is EclipseLink
Database stored procedure
CREATE OR REPLACE PROCEDURE p_user_creation (p_username IN VARCHAR2,
p_password IN VARCHAR2,
p_return OUT NUMBER)
IS
v_syntax VARCHAR2 (256);
BEGIN
IF (p_username IS NOT NULL)
THEN
v_syntax :=
'CREATE USER '''
|| p_username
|| ''' IDENTIFIED BY '''
|| p_password
|| '''';
EXECUTE IMMEDIATE v_syntax;
p_return := 0;
END IF;
EXCEPTION
WHEN OTHERS
THEN
raise_application_error (-20002, 'An error has occurred!');
END;
Java code snippet to invoke stored procedure
try {
Integer returnValue = null;
StoredProcedureQuery storedProcedureQuery =
getEntityManager().createStoredProcedureQuery("p_user_creation");
storedProcedureQuery.registerStoredProcedureParameter("p_username", String.class, ParameterMode.IN);
storedProcedureQuery.registerStoredProcedureParameter("p_password", String.class, ParameterMode.IN);
storedProcedureQuery.registerStoredProcedureParameter("p_return", Integer.class, ParameterMode.OUT);
storedProcedureQuery.setParameter("p_username", "SCOTT");
storedProcedureQuery.setParameter("p_password", "tiger");
storedProcedureQuery.execute();
returnValue = (Integer) storedProcedureQuery.getOutputParameterValue("p_return");
} catch (Exception e) {
log.error("Error " + e.getMessage());
}
i have the following query:
String updatequery = "UPDATE tbl_page SET linkCount = ?, pageProcessed = 1 WHERE pageUrl =?";
PreparedStatement updatestmt = kon.prepareStatement(updatequery);
updatestmt.clearParameters();
//updatestmt.setQueryTimeout(10);
updatestmt.setInt(1, linkCount);
updatestmt.setString(2, urlLink);
updatestmt.executeUpdate();
When i set the query timeout for 10 seconds it will catch an exception the query timed out. but when i dont it goes on waiting. Whats wrong with the query? pageUrl column is the Primary Key with varchar(900)
I know something might be wrong with the prepared statement because when i run this query in MS SQl Server Management Studio ('?' replaced with its value) it works fine.
Am i missing something in Java or MSSQL?
Since the code looks just fine, this could be an issue at database side. May be someone else has blocked the row by updating it and not doing a commit/rollback (most possibly from you MS-SQL Server Management studio !). You could look for locks owned by other processes for the same record so that you can be sure that this is not a database issue.
Create an index on pageUrl:
create index tbl_page_pageUrl_index on tbl_page(pageUrl);
That will allow speedy access to the rows you want to update.
Without this index, the database must do a full table scan, and when combined with an update command, if likely to lead to lock contention and possibly even deadlocks, depending on your locking options.
With SQL Plus for Oracle Database, I can call
SET autotrace on
and then see Execution Plan, statistics, etc.
The problem is that I want access to information about the Execution Plan and statistics in my Java program. I typically have done something like this to execute a sql statement,
Connection connection = //INITIALIZE HERE;
Statement getColumn = connection.createStatement();
ResultSet results = getColumn.executeQuery("INSERT SQL QUERY HERE");
while(results.next())
{
//view results
}
Is there a way I can get the Execution Plan and Statistics? Thanks.
You can query the V$SQL_PLAN table to get the explain plain. Alternatively you can query the PLAN_TABLE, you can see more details on this HERE.