Develop Reference

Couchbase query does not see documents add recently

I'm performing a test with CouchBase 4.0 and java sdk 2.2. I'm inserting 10 documents whose keys always start by "190".
After inserting these 10 documents I query them with:
cb.restore("190", cache);
Thread.sleep(100);
cb.restore("190", cache);
The query within the 'restore' method is:
Statement st = Select.select("meta(c).id, c.*").from(this.bucketName + " c").where(Expression.x("meta(c).id").like(Expression.s(callId + "_%")));
N1qlQueryResult result = bucket.query(st);
The first call to restore returns 0 documents:
Query 'SELECT meta(c).id, c.* FROM cache c WHERE meta(c).id LIKE "190_%"' --> Size = 0
The second call (100ms later) returns the 10 documents:
Query 'SELECT meta(c).id, c.* FROM cache c WHERE meta(c).id LIKE "190_%"' --> Size = 10
I tried adding PersistTo.MASTER in the 'insert' statement, but it neither works.
It seems that the 'insert' is not persisted immediately.
Any help would be really appreciated.
Joan.

You're using N1QL to query the data - and N1QL is only eventually consistent (by default), so it only shows up after the indices are recalculated. This isn't related to whether or not the data is persisted (meaning: written from RAM to disc).
You can try to change the scan_consitency level from its default - NOT_BOUNDED - to get consistent results, but that would take longer to return.
read more here
java scan_consitency options

ORMLite groupByRaw and groupBy issue on android SQLite db

I have a SQLite table content with following columns:
-----------------------------------------------
|id|book_name|chapter_nr|verse_nr|word_nr|word|
-----------------------------------------------
the sql query
select count(*) from content where book_name = 'John'
group by book_name, chapter_nr
in DB Browser returns 21 rows (which is the count of chapters)
the equivalent with ORMLite android:
long count = getHelper().getWordDao().queryBuilder()
.groupByRaw("book_name, chapter_nr")
.where()
.eq("book_name", book_name)
.countOf();
returns 828 rows (which is the count of verse numbers)
as far as I know the above code is translated to:
select count(*) from content
where book_name = 'John'
group by book_name, chapter_nr
result of this in DB Browser:
| count(*)
------------
1 | 828
2 | 430
3 | 653
...
21| 542
---------
21 Rows returned from: select count(*)...
so it seems to me that ORMLite returns the first row of the query as the result of countOf().
I've searched stackoverflow and google a lot. I found this question (and more interestingly the answer)
You can also count the number of rows in a custom query by calling the > countOf() method on the Where or QueryBuilder object.
// count the number of lines in this custom query
int numRows = dao.queryBuilder().where().eq("name", "Joe Smith").countOf();
this is (correct me if I'm wrong) exactly what I'm doing, but somehow I just get the wrong number of rows.
So... either I'm doing something wrong here or countOf() is not working the way it is supposed to.
Note: It's the same with groupBy instead of groupByRaw (according to ORMLite documentation joining groupBy's should work)
...
.groupBy("book_name")
.groupBy("chapter_nr")
.where(...)
.countOf()
EDIT: getWordDao returns from class Word:
#DatabaseTable(tableName = "content")
public class Word { ... }

returns 828 rows (which is the count of verse numbers)
This seems to be a limitation of the QueryBuilder.countOf() mechanism. It is expecting a single value and does not understand the addition of GROUP BY to the count query. You can tell that it doesn't because that method returns a single long.
If you want to extract the counts for each of the groups it looks like you will need to do a raw query check out the docs.

How to select the field value if field in all child records are the same

I don't know if I'm wording this question correctly, but here it goes.
This is a web application using java, oracle, hibernate.
I have 2 tables in a one (items) to many (tasks) relationship.
Items
item_id name
active_status
etc
Tasks
task_id
item_id
active_status
progress_status
etc
The item's status is made up of the statuses of all of its tasks. Here's the logic...
If Item Status is Canceled or On Hold...return Item Active Status
If there are no tasks, return Completed
If All Tasks are Active and NOT Superseded, then
...return Not Started if all tasks are Not Started
...return Completed if all tasks are Completed
...return On Hold if all tasks are On Hold
Otherwise return Started
I want to do this using SQL and map it to a field in my hibernate mapping file.
I've tried many things over the past several days, and can't seem to get it to work. I tried grouping the records and if 1 record was found, return that status. I've used decode, case, etc.
Here are a few examples of things I've tried. In the second example I get a 'not a single group group function' error.
Any thoughts?
select decode(i.active_status_id, 'OH', i.active_status_id, 'Ca', i.active_status_id,t.progress_status_id)
from tasks t
LEFT OUTER JOIN Items i
ON i.item_id = t.item_id
where t.item_id = 10927815 and t.active_status_id = 'Ac' and t.active_status_id != 'Su'
group by i.active_status_id, t.progress_status_id;
select case
when (count(*) = 1) then progress_status_id
else 'St'
end
from
(select progress_status_id
from tasks t
where t.item_id = 10927815 and (t.active_status_id = 'Ac' and t.active_status_id != 'Su') group by t.progress_status_id)

perhaps somthing like this
SELECT
item_id
, CASE
WHEN active_status IN ('Canceled', 'On Hold') THEN active_status
WHEN t_num = 0 THEN 'Completed'
WHEN flag_all_active = 1 AND flag_all_not_started = 1 THEN 'Not Started'
WHEN flag_all_active = 1 AND flag_all_completed = 1 THEN 'Completed'
WHEN flag_all_active = 1 AND flag_all_on_hold = 1 THEN 'On Hold'
ELSE 'Started'
END
FROM
(
SELECT
i.item_id
, i.active_status
, sum(CASE WHEN t.task_id is NULL THEN 0 ELSE 1 end ) as t_num
, MIN( CASE t.active_status WHEN 'Ac' THEN 1 ELSE 0 END ) as flag_all_active
, MIN( CASE t.progress_status_id WHEN 'Not Starten' THEN 1 ELSE 0 END ) as flag_all_not_started
, MIN( CASE t.progress_status_id WHEN 'Completed' THEN 1 ELSE 0 END ) as flag_all_completed
, MIN( CASE t.progress_status_id WHEN 'On Hold' THEN 1 ELSE 0 END ) as flag_all_on_hold
FROM
items i
left outer join tasks t on (t.item_id = i.item_id )
group by i.item_id
)
;

If you're using annotations you can use #Formula("sql query here") for your derived properties.
See Hibernate formula docs for a (surprisingly brief) explanation.
Alternatively, since you're dealing with relatively large lists of items, it would be better to make the status calculations part of your initial query thus avoiding the database getting hammered by hundreds or thousands of requests. This is what will probably happen if you iterate over each item in the list.
I would recommend joining the status calculation to whatever query you are using to generate your list (presumably in a NamedQuery). This lets your database do all the heavy lifting without being slowed down by the network, which is what it is best at. The Hibernate docs give lots of helpful examples of queries you can try.

More Efficient Way of Doing This SQL Query? A time comparison query?

I have this SQL query which queries the database every 5 seconds to determine who is currently actively using the software. Active users have pinged the server in the last 10 seconds. (The table gets updated correctly on user activity and a I have a thread evicting entries on session timeouts, that all works correctly).
What I'm looking for is a more efficient/quicker way to do this, since it gets called frequently, about every 5 seconds. In addition, there may be up to 500 users in the database. The language is Java, but the question really pertains to any language.
List<String> r = new ArrayList<String>();
Calendar c = Calendar.getInstance();
long threshold = c.get(Calendar.SECOND) + c.get(Calendar.MINUTE)*60 + c.get(Calendar.HOUR_OF_DAY)*60*60 - 10;
String tmpSql = "SELECT user_name, EXTRACT(HOUR FROM last_access_ts) as hour, EXTRACT(MINUTE FROM last_access_ts) as minute, EXTRACT(SECOND FROM last_access_ts) as second FROM user_sessions";
DBResult rs = DB.select(tmpSql);
for (int i=0; i<rs.size(); i++)
{
Map<String, Object> result = rs.get(i);
long hour = (Long)result.get("hour");
long minute = (Long)result.get("minute");
long second = (Long)result.get("second");
if (hour*60*60 + minute*60 + second > threshold)
r.add(result.get("user_name").toString());
}
return r;

If you want this to run faster, then create an index on user_sessions(last_access_ts, user_name), and do the date logic in the query:
select user_name
from user_sessions
where last_access_ts >= now() - 5/(24*60*60);
This does have a downside. You are, presumably, updating the last_access_ts field quite often. An index on the field will also have to be updated. On the positive side, this is a covering index, so the index itself can satisfy the query without resorting to the original data pages.

I would move the logic from Java to DB. This mean you translate if into where, and just select the name of valid result.
SELECT user_name FROM user_sessions WHERE last_access_ts > ?
In your example the c represent current time. It is highly possible that result will be empty.
So your question should be more about date time operation on your database.

Just let the database do the comparison for you by using this query:
SELECT
user_name
FROM user_sessions
where TIMESTAMPDIFF(SECOND, last_access_ts, current_timestamp) > 10
Complete example:
List<String> r = new ArrayList<String>();
Calendar c = Calendar.getInstance();
long threshold = c.get(Calendar.SECOND) + c.get(Calendar.MINUTE)*60 + c.get(Calendar.HOUR_OF_DAY)*60*60 - 10;
// this will return all users that were inactive for longer than 10 seconds
String tmpSql = "SELECT
user_name
FROM user_sessions
where TIMESTAMPDIFF(SECOND, last_access_ts, current_timestamp) > 10";
DBResult rs = DB.select(tmpSql);
for (int i=0; i<rs.size(); i++)
{
Map<String, Object> result = rs.get(i);
r.add(result.get("user_name").toString());
}
return r;
SQLFiddle

The solution is to remove the logic from your code to the sql query to only get the active users from that select, using a where clause.
It is faster to use the sql built-in functions to get fewer records and iterate less in your code.
Add this to your sql query to get the active users only:
Where TIMESTAMPDIFF(SECOND, last_access_ts, current_timestamp) > 10
This will get you all the records whose date is 10 seconds ago or sooner.

Try the MySQL TimeDiff function in your select. This way you can select only the results that are active without having to do any other calculations.
Link: MySQL: how to get the difference between two timestamps in seconds

If I get you right, then you got only 500 entries in your user_sessions table. In this case I wouldn't even care about indexes. Throw them away. The DB engine probably won't use them anyway for such a low record count. The performance gain due to not updating the indexes on every record update could be probably higher than the query overhead.
If you care about DB stress, then lengthen the query/update intervals to 1 minute or more, if your application allows this. Gordon Linoff's answer should give you the best query performance though.
As a side note (because it has bitten me before): If you don't use the same synchronized time for all user callbacks, then your "active users logic" is flawed by design.

hibernate paging is resulting in select and update calls

I am trying to implement paging in hibernate and i am seeing some weird behavior from hibernate. I have tried two queries with the same result
List<SomeData> dataList = (List<SomeData>) session.getCurrentSession()
.createQuery("from SomeData ad where ad.bar = :bar order by ad.id.name")
.setString("bar", foo)
.setFirstResult(i*PAGE_SIZE)
.setMaxResults(PAGE_SIZE)
.setFetchSize(PAGE_SIZE) // page_size is 1000 in my case
.list();
and
List<SomeData> datalist= (List<SomeData>) session.getCurrentSession()
.createCriteria(SomeData.class)
.addOrder(Order.asc("id.name"))
.add(Expression.eq("bar", foo))
.setFirstResult(i*PAGE_SIZE)
.setMaxResults(PAGE_SIZE)
.list();
I have this in a for loop and each time this query runs, the run time increases. The first call returns in 100 ms, the second in 150 and the 5th call takes 2 seconds and so on.
Looking in the server (MySql 5.1.36) logs, I see that the select query does get generated properly with the LIMIT clause but for each record that is returned, hibernate for some reason also emits an update query. after the first result, it updates 1000 records, after the second result, it updates 2000 records and so on. So for a page size of 1000 and 5 iterations of the loop, the database is getting hit with 15,000 queries (5K + 4K + 3K + 2K + 1K ) Why is that happening?
I tried making a native SQL query and it worked as expected. The query is
List asins = (List) session.getCurrentSession()
.createSQLQuery("SELECT * FROM some_data where foo = :foo order by bar
LIMIT :from , :page")
.addScalar(..)
.setInteger("page", PAGE_SIZE)
.setInteger("from", (i*PAGE_SIZE))
... // set other params
.list();
My mapping class has setters/getters for the blob object as
void setSomeBlob(Blob blob){
this.someByteArray = this.toByteArray(blob)
}
void Blob getSomeBlob(){
return Hibernate.createBlob(someByteArray)
}

Turn on bound parameters logging (you can do that by setting "org.hibernate.type" log level to "TRACE") to see what specifically is being updated.
Most likely you're modifying the entities after they've been loaded - either explicitly or implicitly (e.g. returning different value from getter or using a default value somewhere).
Another possibility is that you've recently altered (one of) the table(s) you're selecting from and column default in the table doesn't match default value in the entity.