Develop Reference

Database insertions in a reactive loop

I have the following method where I am doing db insertions. I want to perform the inserts in a transaction.
Meaning when there are 100 values in records, I wan to insert them all and commit once.
How could I amend the following such that I can get record.value() info for each insert queries below.
This would essentially equate to having andThen() a 100 times but of course I do not want to write andThen() a 100 times nor do I know the number of records which can vary.
To note: Using RX Java 1.
Please advice. Thank you.
public Observable<?> insert(Observable<Record<String, String>> records) {
// I am looking for a way to get this record.value() into the following return block.
records.flatMap(record -> {
String value = record.value();
return null;
});
return client.rxGetConnection()
// making it transactional by setting to false
.flatMap(connection -> connection.rxSetAutoCommit(false)
// was looking to insert above records and flatMap operations here but it is not possible from what I have explored.
.toCompletable()
// .andThen(connection.rxExecute("INSERT (name) VALUES " + record.value()) // trying to achieve this, to be able to get record.value for each insert
.andThen(connection.rxExecute("INSERT (name) VALUES some_value"))
.flatMap(rows -> connection.rxCommit())).toObservable();
}

Page<> vs Slice<> when to use which?

I've read in Spring Jpa Data documentation about two different types of objects when you 'page' your dynamic queries made out of repositories.
Page and Slice
Page<User> findByLastname(String lastname, Pageable pageable);
Slice<User> findByLastname(String lastname, Pageable pageable);
So, I've tried to find some articles or anything talking about main difference and different usages of both, how performance changes and how sorting affercts both type of queries.
Does anyone has this type of knowledge, articles or some good source of information?

Page extends Slice and knows the total number of elements and pages available by triggering a count query. From the Spring Data JPA documentation:
A Page knows about the total number of elements and pages available. It does so by the infrastructure triggering a count query to calculate the overall number. As this might be expensive depending on the store used, Slice can be used as return instead. A Slice only knows about whether there’s a next Slice available which might be just sufficient when walking through a larger result set.

The main difference between Slice and Page is the latter provides non-trivial pagination details such as total number of records(getTotalElements()), total number of pages(getTotalPages()), and next-page availability status(hasNext()) that satisfies the query conditions, on the other hand, the former only provides pagination details such as next-page availability status(hasNext()) compared to its counterpart Page. Slice gives significant performance benefits when you deal with a colossal table with burgeoning records.
Let's dig deeper into its technical implementation of both variants.
Page
static class PagedExecution extends JpaQueryExecution {
#Override
protected Object doExecute(final AbstractJpaQuery repositoryQuery, JpaParametersParameterAccessor accessor) {
Query query = repositoryQuery.createQuery(accessor);
return PageableExecutionUtils.getPage(query.getResultList(), accessor.getPageable(),
() -> count(repositoryQuery, accessor));
}
private long count(AbstractJpaQuery repositoryQuery, JpaParametersParameterAccessor accessor) {
List<?> totals = repositoryQuery.createCountQuery(accessor).getResultList();
return (totals.size() == 1 ? CONVERSION_SERVICE.convert(totals.get(0), Long.class) : totals.size());
}
}
If you observe the above code snippet, PagedExecution#doExecute method underlyingly calls PagedExecution#count method to get the total number of records satisfying the condition.
Slice
static class SlicedExecution extends JpaQueryExecution {
#Override
protected Object doExecute(AbstractJpaQuery query, JpaParametersParameterAccessor accessor) {
Pageable pageable = accessor.getPageable();
Query createQuery = query.createQuery(accessor);
int pageSize = 0;
if (pageable.isPaged()) {
pageSize = pageable.getPageSize();
createQuery.setMaxResults(pageSize + 1);
}
List<Object> resultList = createQuery.getResultList();
boolean hasNext = pageable.isPaged() && resultList.size() > pageSize;
return new SliceImpl<>(hasNext ? resultList.subList(0, pageSize) : resultList, pageable, hasNext);
}
}
If you observe the above code snippet, to findout whether next set of results present or not (for hasNext()) the SlicedExecution#doExecute method always fetch extra one element(createQuery.setMaxResults(pageSize + 1)) and skip it based on the pageSize condition(hasNext ? resultList.subList(0, pageSize) : resultList).
Application:
Page
Use when UI/GUI expects to displays all the results at the initial stage of the search/query itself, with page numbers to traverse(ex., bankStatement with pagenumbers)
Slice
Use when UI/GUI expects to doesnot interested to show all the results at the initial stage of the search/query itself, but intent to show the records to traverse based on scrolling or next button click event (ex., facebook feed search)

Recursive ROWNUM usage with transactional Method

I have a table named Person, My select sql usually brings number of lets say 100K person since It takes so much time I am having readtimeout exception.
So I know that I have to use ROWNUM to limit the result size.
Class MyService {
#Transactional(rollbackFor = Exception.class)
doJob(){
jobService.process();
}
}
Class JobService {
public void process() {
List<Person> personlList= jdbcQuery.query ("Select * from ... ... where rownum<1000" , ROWMAPPAR, parameter);
//Process all record list
}
Everything is ok till know But I want to be sure all record lets say 100K are processed and if there is an error while processing one of the batch ,rollback should be occured.
Do I need to invode process() method recursively?
Using
Spring 3.5
Oracle 11g

Using ROWNUM as shown in your query may very well not give you the results you expect. (But on the other hand it may, at least sometimes :-). ROWNUM is generated as rows are are emitted from the query, AFTER the WHERE clause is evaluated, but BEFORE any ORDER BY or HAVING clauses are applied. This can cause your query to return results which may surprise you.
Try creating the following table:
create table t(n number);
And populating it with:
insert into t (n)
select n from
(select rownum n from dual connect by level <= 2000)
where n > 1234;
Thus the table will have rows with values of 1235 through 2000.
The run each of the following queries in order:
select *
from t
order by n;
select n, rownum
from t
where rownum < 100
order by n;
select n, rownum as r from
(select n
from t
order by n);
select n, r from
(select n, rownum as r from
(select n
from t
order by n))
where r < 100
order by n;
and observe the differences in the output you get.
For those who don't have an Oracle instance handy, here's an SQLFiddle with the above in it.
Share and enjoy.

Do I need to invode process() method recursively?
I wouldn't do that. Simply rewrite your code to this:
class MyService {
#Transactional(rollbackFor = Exception.class)
void doJob(){
// Continue processing within the same transaction, until process() returns false
while (jobService.process());
}
}
class JobService {
public boolean process() {
List<Person> personlList= jdbcQuery.query(
"Select * from ... ... where rownum<=1000" , ROWMAPPAR, parameter);
// I've changed your predicate ------^^
// process() returns false when the above select returns less than 1000 records
return personList.size() == 1000;
}
}
Beware, though, that one of the problems that you may be experiencing is the fact that you're keeping a very long-running transaction alive. This will cause a lot of concurrency inside your database and might contribute to the batch job running slow. If you don't absolutely need an atomic batch job (everything committed or everything rolled back), you might consider running each sub-job in its own transaction.

Hibernate setMaxResults paging

If I don't set setFirstResult(-) and recursively call criteria.setmaxresults(10) each time, will it automatically grab the next 10 items from the database?

No. You have to use criteria.setFirstResult(0) and page through yourself, something like this:
public List getCarters(final int firstResult, final int maxResults) {
final Criteria criteria = sessionFactory.getCurrentSession()
.createCriteria(SomePersistentClass.class);
.add(Restrictions.eq("name", "Carter"))
criteria.setFirstResult(firstResult);
criteria.setMaxResults(maxResults);
return criteria.list();
}

Of course, no. Criteria grabs data from database only when you call .list() or .uniqueResult()

Is there a way to get the count size for a JPA Named Query with a result set?

I like the idea of Named Queries in JPA for static queries I'm going to do, but I often want to get the count result for the query as well as a result list from some subset of the query. I'd rather not write two nearly identical NamedQueries. Ideally, what I'd like to have is something like:
#NamedQuery(name = "getAccounts", query = "SELECT a FROM Account")
.
.
Query q = em.createNamedQuery("getAccounts");
List r = q.setFirstResult(s).setMaxResults(m).getResultList();
int count = q.getCount();
So let's say m is 10, s is 0 and there are 400 rows in Account. I would expect r to have a list of 10 items in it, but I'd want to know there are 400 rows total. I could write a second #NamedQuery:
#NamedQuery(name = "getAccountCount", query = "SELECT COUNT(a) FROM Account")
but it seems a DRY violation to do that if I'm always just going to want the count. In this simple case it is easy to keep the two in sync, but if the query changes, it seems less than ideal that I have to update both #NamedQueries to keep the values in line.
A common use case here would be fetching some subset of the items, but needing some way of indicating total count ("Displaying 1-10 of 400").

So the solution I ended up using was to create two #NamedQuerys, one for the result set and one for the count, but capturing the base query in a static string to maintain DRY and ensure that both queries remain consistent. So for the above, I'd have something like:
#NamedQuery(name = "getAccounts", query = "SELECT a" + accountQuery)
#NamedQuery(name = "getAccounts.count", query = "SELECT COUNT(a)" + accountQuery)
.
static final String accountQuery = " FROM Account";
.
Query q = em.createNamedQuery("getAccounts");
List r = q.setFirstResult(s).setMaxResults(m).getResultList();
int count = ((Long)em.createNamedQuery("getAccounts.count").getSingleResult()).intValue();
Obviously, with this example, the query body is trivial and this is overkill. But with much more complex queries, you end up with a single definition of the query body and can ensure you have the two queries in sync. You also get the advantage that the queries are precompiled and at least with Eclipselink, you get validation at startup time instead of when you call the query.
By doing consistent naming between the two queries, it is possible to wrap the body of the code to run both sets just by basing the base name of the query.

Using setFirstResult/setMaxResults do not return a subset of a result set, the query hasn't even been run when you call these methods, they affect the generated SELECT query that will be executed when calling getResultList. If you want to get the total records count, you'll have to SELECT COUNT your entities in a separate query (typically before to paginate).
For a complete example, check out Pagination of Data Sets in a Sample Application using JSF, Catalog Facade Stateless Session, and Java Persistence APIs.

oh well you can use introspection to get named queries annotations like:
String getNamedQueryCode(Class<? extends Object> clazz, String namedQueryKey) {
NamedQueries namedQueriesAnnotation = clazz.getAnnotation(NamedQueries.class);
NamedQuery[] namedQueryAnnotations = namedQueriesAnnotation.value();
String code = null;
for (NamedQuery namedQuery : namedQueryAnnotations) {
if (namedQuery.name().equals(namedQueryKey)) {
code = namedQuery.query();
break;
}
}
if (code == null) {
if (clazz.getSuperclass().getAnnotation(MappedSuperclass.class) != null) {
code = getNamedQueryCode(clazz.getSuperclass(), namedQueryKey);
}
}
//if not found
return code;
}