I have a table named Person, My select sql usually brings number of lets say 100K person since It takes so much time I am having readtimeout exception.
So I know that I have to use ROWNUM to limit the result size.
Class MyService {
#Transactional(rollbackFor = Exception.class)
doJob(){
jobService.process();
}
}
Class JobService {
public void process() {
List<Person> personlList= jdbcQuery.query ("Select * from ... ... where rownum<1000" , ROWMAPPAR, parameter);
//Process all record list
}
Everything is ok till know But I want to be sure all record lets say 100K are processed and if there is an error while processing one of the batch ,rollback should be occured.
Do I need to invode process() method recursively?
Using
Spring 3.5
Oracle 11g
Using ROWNUM as shown in your query may very well not give you the results you expect. (But on the other hand it may, at least sometimes :-). ROWNUM is generated as rows are are emitted from the query, AFTER the WHERE clause is evaluated, but BEFORE any ORDER BY or HAVING clauses are applied. This can cause your query to return results which may surprise you.
Try creating the following table:
create table t(n number);
And populating it with:
insert into t (n)
select n from
(select rownum n from dual connect by level <= 2000)
where n > 1234;
Thus the table will have rows with values of 1235 through 2000.
The run each of the following queries in order:
select *
from t
order by n;
select n, rownum
from t
where rownum < 100
order by n;
select n, rownum as r from
(select n
from t
order by n);
select n, r from
(select n, rownum as r from
(select n
from t
order by n))
where r < 100
order by n;
and observe the differences in the output you get.
For those who don't have an Oracle instance handy, here's an SQLFiddle with the above in it.
Share and enjoy.
Do I need to invode process() method recursively?
I wouldn't do that. Simply rewrite your code to this:
class MyService {
#Transactional(rollbackFor = Exception.class)
void doJob(){
// Continue processing within the same transaction, until process() returns false
while (jobService.process());
}
}
class JobService {
public boolean process() {
List<Person> personlList= jdbcQuery.query(
"Select * from ... ... where rownum<=1000" , ROWMAPPAR, parameter);
// I've changed your predicate ------^^
// process() returns false when the above select returns less than 1000 records
return personList.size() == 1000;
}
}
Beware, though, that one of the problems that you may be experiencing is the fact that you're keeping a very long-running transaction alive. This will cause a lot of concurrency inside your database and might contribute to the batch job running slow. If you don't absolutely need an atomic batch job (everything committed or everything rolled back), you might consider running each sub-job in its own transaction.
Related
I have the following method where I am doing db insertions. I want to perform the inserts in a transaction.
Meaning when there are 100 values in records, I wan to insert them all and commit once.
How could I amend the following such that I can get record.value() info for each insert queries below.
This would essentially equate to having andThen() a 100 times but of course I do not want to write andThen() a 100 times nor do I know the number of records which can vary.
To note: Using RX Java 1.
Please advice. Thank you.
public Observable<?> insert(Observable<Record<String, String>> records) {
// I am looking for a way to get this record.value() into the following return block.
records.flatMap(record -> {
String value = record.value();
return null;
});
return client.rxGetConnection()
// making it transactional by setting to false
.flatMap(connection -> connection.rxSetAutoCommit(false)
// was looking to insert above records and flatMap operations here but it is not possible from what I have explored.
.toCompletable()
// .andThen(connection.rxExecute("INSERT (name) VALUES " + record.value()) // trying to achieve this, to be able to get record.value for each insert
.andThen(connection.rxExecute("INSERT (name) VALUES some_value"))
.flatMap(rows -> connection.rxCommit())).toObservable();
}
I am making the following query which works and updates the info into the database as expected. But is there a way I can get an output from Single < UpdateResult >?
public Single<UpdateResult> update(UpdateEventRequest request) {
return client.rxUpdateWithParams(
"UPDATE mydb.sample SET sta_cd=?, some_ts=current_timestamp WHERE id=? RETURNING sta_cd",
new JsonArray(Arrays.asList(sta_cd, id)));
}
From the following e variable, I was hoping to get the value "10012". But it doesn't seem possible. Tried with map, flatmap and just see options available in e. The only result data in e is 'keys' which is an empty list and 'updated' which is an integer value of 1. My DB is postgres and was expecting results from from Single < UpdateResult > since am using 'RETURNING' in the query.
I have done the same for an insert operation which works but that is via the method rxQueryWithParams() and that returns a Single < ResultSet > instead. Thus wondering if this is even possible. Been having a look at docs and maybe this is not possible as an update query is returning a Single < UpdateResult > . Looking for advice if this is possible, to return data from an update query or a way around this. Please advice. Thanks.
Single<UpdateResult> result = someClass.update("10012", "78632");
result.subscribe(
e -> {
System.out.println("success: " + e); // I land here as expected
},
error -> {
System.out.println("error: " + error);
}
);
Because you are using RETURNING in these commands, treat these INSERT and UPDATE commands as queries.
Run them through rxQueryWithParams() so you can retrieve the results.
When you run rxUpdateWithParams(), the UpdateResult contains only the number of rows affected.
I understand pagination to improve performance on queries that, naturally, have a very large resultset.
I have the implemented paginatedList method that successfully returns the correct amount. But it seems more of a MySQL's LIMIT or SQLServer's TOP clause.
The method is thus implemented:
#SuppressWarnings("unchecked")
public List<T> paginatedList(int pageSize) {
Criteria criteria = getSession().createCriteria(persistentClass);
criteria.setFirstResult(0);
criteria.setMaxResults(pageSize);
return (List<T>) criteria.list();
}
And I call like this:
#Override
public List<AlertLog> findAll() {
return super.paginatedList(1000);
}
This implementation returns results 0 to 1000.
How to, on runtime, get the 1001st to 2000th and so on?
to get the next 1000 result you need to set firstResult to 1000 and maxresults to 1000, so it becomes..
criteria.setFirstResult(1000);
criteria.setMaxResults(noOfEntries);
Please make sure that the noOfEntries is less that the total count.
I am not hardcore Hibernate Programmer.
Create a native query return bulk data million records:
return super.em.createNativeQuery(query).getResultList();
Fetching data and build Objects which is persisted using DAO in Loop which have 1 million persists.
persist(object)
When simultaneously I run select query on table it shows me 0 Results.
Select count(*) from Audit_Log;
After all records are inserted successfully, MySQL shows me result.
Earlier I was named query for fetching values from DAO and it worked well. Now I opted native query and got this behavior. Is there something I need to change?
Code:
public abstract class GenericDaoImpl<T, PK> implements GenericDao<T, PK> {
#Override
public T create(final T t) {
this.em.persist(t);
return t;
}
#Override
public void flush() {
org.hibernate.Session session = (org.hibernate.Session)em.getDelegate();
session.flush();
}
DAO:
#Override
public Person create(Person person) {
return (Person) create((MainRecord) person);
}
Are you persisting 10L (a.k.a. 1M) records in single transaction? Looks like this is this case. Now since TX is open for considerably very long period of time, firing a query on TOAD (or any other SQL client) won't return it since data isn't committed in the DB yet.
Try flushing the data in between.
Also, I hope you're using batching on hibernate as well as driver level. There is no way I will persist 1M records w/o batching.
The first thing you need to try is to use method flush or close the session directy while the persist was done.
And the explain was here:
persist() makes a transient instance persistent. However, it does not guarantee that the identifier value will be assigned to the persistent instance immediately, the assignment might happen at flush time. persist() also guarantees that it will not execute an INSERT statement if it is called outside of transaction boundaries. This is useful in long-running conversations with an extended Session/persistence context.
Try this code snippet, it was was similar to approach we using in JDBC batching..
public Long save(HttpServletRequest request) {
//Further business logic here....
for ( int i=0; i<count; i++ ) {
getEntityManager().persist((ABC) model);
if ( i > 0 && i % 2500== 0 ) {
getEntityManager().flush();
getEntityManager().clear();
}
}
tx.commit();
((EntityManager) session).close();
}
I like the idea of Named Queries in JPA for static queries I'm going to do, but I often want to get the count result for the query as well as a result list from some subset of the query. I'd rather not write two nearly identical NamedQueries. Ideally, what I'd like to have is something like:
#NamedQuery(name = "getAccounts", query = "SELECT a FROM Account")
.
.
Query q = em.createNamedQuery("getAccounts");
List r = q.setFirstResult(s).setMaxResults(m).getResultList();
int count = q.getCount();
So let's say m is 10, s is 0 and there are 400 rows in Account. I would expect r to have a list of 10 items in it, but I'd want to know there are 400 rows total. I could write a second #NamedQuery:
#NamedQuery(name = "getAccountCount", query = "SELECT COUNT(a) FROM Account")
but it seems a DRY violation to do that if I'm always just going to want the count. In this simple case it is easy to keep the two in sync, but if the query changes, it seems less than ideal that I have to update both #NamedQueries to keep the values in line.
A common use case here would be fetching some subset of the items, but needing some way of indicating total count ("Displaying 1-10 of 400").
So the solution I ended up using was to create two #NamedQuerys, one for the result set and one for the count, but capturing the base query in a static string to maintain DRY and ensure that both queries remain consistent. So for the above, I'd have something like:
#NamedQuery(name = "getAccounts", query = "SELECT a" + accountQuery)
#NamedQuery(name = "getAccounts.count", query = "SELECT COUNT(a)" + accountQuery)
.
static final String accountQuery = " FROM Account";
.
Query q = em.createNamedQuery("getAccounts");
List r = q.setFirstResult(s).setMaxResults(m).getResultList();
int count = ((Long)em.createNamedQuery("getAccounts.count").getSingleResult()).intValue();
Obviously, with this example, the query body is trivial and this is overkill. But with much more complex queries, you end up with a single definition of the query body and can ensure you have the two queries in sync. You also get the advantage that the queries are precompiled and at least with Eclipselink, you get validation at startup time instead of when you call the query.
By doing consistent naming between the two queries, it is possible to wrap the body of the code to run both sets just by basing the base name of the query.
Using setFirstResult/setMaxResults do not return a subset of a result set, the query hasn't even been run when you call these methods, they affect the generated SELECT query that will be executed when calling getResultList. If you want to get the total records count, you'll have to SELECT COUNT your entities in a separate query (typically before to paginate).
For a complete example, check out Pagination of Data Sets in a Sample Application using JSF, Catalog Facade Stateless Session, and Java Persistence APIs.
oh well you can use introspection to get named queries annotations like:
String getNamedQueryCode(Class<? extends Object> clazz, String namedQueryKey) {
NamedQueries namedQueriesAnnotation = clazz.getAnnotation(NamedQueries.class);
NamedQuery[] namedQueryAnnotations = namedQueriesAnnotation.value();
String code = null;
for (NamedQuery namedQuery : namedQueryAnnotations) {
if (namedQuery.name().equals(namedQueryKey)) {
code = namedQuery.query();
break;
}
}
if (code == null) {
if (clazz.getSuperclass().getAnnotation(MappedSuperclass.class) != null) {
code = getNamedQueryCode(clazz.getSuperclass(), namedQueryKey);
}
}
//if not found
return code;
}