I have problems with understanding of way couchbase query plan works.
I use SpringData with Couchbase 4.1 and I provide custom implementation of Couchbase Repository. Inside my custom implememtnation of Couchbase Repository I have below method:
String queryAsString = "SELECT MyDatabase.*, META().id as _ID, META().cas as _CAS FROM MyDatabase WHERE segmentId = $id AND _class = $class ORDER BY executionTime DESC LIMIT 1";
JsonObject params = JsonObject.create()
.put(CLASS_VARIABLE, MyClass.class.getCanonicalName())
.put(ID_VARIABLE, segmentId);
N1qlQuery query = N1qlQuery.parameterized(queryAsString, params);
List<MyClass> resultList = couchbaseTemplate.findByN1QL(query, SegmentMembers.class);
return resultList.isEmpty() ? null : resultList.get(0);
In a result, Spring Data produces following json object represented query to Couchbase:
{
"$class":"path/MyClass",
"statement":"SELECT MyDatabase.*, META().id as _ID, META().cas as _CAS from MyDatabase where segmentId = $id AND _class = $class ORDER BY executionTime DESC LIMIT 1",
"id":"6592c16a-c8ae-4a74-bc17-7e18bf73b3f8"
}
And the problem is with performance when I execute it via Java and N1QL Rest Api or via cbq consol. For execute this query in cbq I simply replace parameters reference with exact values.
After adding EXPLAIN clause before select statement I mentioned different execution plans. Execution this query as parameterized query via Java Spring Data or N1QL Rest Api I've mentioned that query doesn't use index that I created exactly for this case. Index definiton can be found below:
CREATE INDEX `testMembers` ON MyDatabase `m`(`_class`,`segmentId`,`executionTime`) WHERE (`_class` = "path/MyClass") USING GSI;
So, when I execute query via cbq consol, Couchbase uses my idnex and query performance is very good. But, when I execute this query via N1QL rest api or Java i see that query doesn't use my index. Below you can find part of execution plan that proves this fact:
"~children": [
{
"#operator": "PrimaryScan",
"index": "#primary",
"keyspace": "CSM",
"namespace": "default",
"using": "gsi"
},
So, the question is that the right and legal behavior of couchbase query optimizer? And does it mean that query plan does not take into account real values of parameters? And have I manually put values into query string or exist eny other way to use N1Ql parameterized query with correct index selection?
EDIT
According to shashi raj answer I add N1qlParams.build().adhoc(false) parameter to parameterized N1QL query. This doesn't solve my problem, because I still have performance problem with this query. Moreover, when I print query I see that it is the same as I described earlier. So, my query still wrong analyzed and cause performance decline.
first of all you need to know how N1QL parameterized queries works query should be passed as:
String query= select * from bucketName where _class=$_class and segmentId=$segmentId LIMIT $limit ;
Now the query should be passed as:
N1QL.parameterized(query,jsonObject,N1qlParams.build().adhoc(false));
where jsonObject will have all the placeholder values.
JsonObject jsonObject=JsonObject.create().put("_class","com.entity,user").put("segmentId","12345").put("limit",100);
N1qlParams.build().adhoc(false) is optional since if you want your query to be optimized it will make use of it. It makes use of LRU where it keeps track of previously query entered and it keeps record of it so that next time it doesn't need to parse query and fetch it from previous what we call as prepared statement.
The only problem is couchbase only keeps record of last 5000 queried.
The problem in your case is caused by the fact that you have an index with 'where' clause WHERE ( _class = "path/MyClass"), and at the same time, you passing the _class as a parameter in your query.
Thus, the query optimizer analyzing the parametrized query has no idea that this query might use an index created for _class = "path/MyClass", cause it's _class = $class in a select's where. Pretty simple, right?
So, don't pass any fields mentioned in your index's 'where' as select parameters. Instead, hardcode _class = "path/MyClass" in your select in the same way you did for create index. And everything should be fine.
Here's the ticket in the couchbase issue tracking system about that.
https://issues.couchbase.com/browse/MB-22185?jql=text%20~%20%22parameters%20does%20not%20use%20index%22
Related
I want to build a cosmosdb sql query, because I'm using a rest interface, which accepts SQL querys (don't ask why, I can't change that :( )...
Now I want to build that query with some parameters, which affects the WHERE clause.
I think it is a good idea to escape these parameters, to prevent sql injection.
But I just found these way to build a query:
var param = new SqlParameter();
param.add("#test", "here some string to inject");
var query = new SqlQuerySpec("SELECT #test FROM table", param);
Now I could do sql calls to the cosmos's without sql injection. But I don't want this. I just want to get the query string.
But I need the full query from "query". But there seems to be just the method query.getQueryText(). But this just returns the string "SELECT #test FROM table".
Do know a workaround for me? Or maybe just a good package I can use to to my own string escapes.
T
I found the information that this escalation stuff doesn't happen on client site. It happens in the dbms. So I need a rest interface, where I can pass the parameters.
Azure Cosmos DB SQL Like, prepared statements
I wish to implement pagingation with FIQL support.
I am using apache cxf with JPA(Hibernate).
Here is sample example given for it http://cxf.apache.org/docs/jax-rs-search.html#JAX-RSSearch-JPA2.0
SearchConditionVisitor<Order, TypedQuery<Order>> visitor
= new JPATypedQueryVisitor<>(em, Order.class);
// connect FIQL cxf SearchCondition with our JPA visitor
searchCondition.accept(visitor);
// creeate JPA specific TypedQuery by our visitor
TypedQuery<Order> typedQuery = visitor.getQuery();
typedQuery.setFirstResult((page * perPage) - perPage);
typedQuery.setMaxResults(perPage);
// Run the query and return matching a complex FIQL criteria
return typedQuery.getResultList();
Every thing looks working including searching and pagination.
It looks have no order by clause being use from generate sql log and seems follow database insertion order.
Now I wish to change the default sorting order. For example I wish to sort by Order id field in descending order. How can I achieve that?
I can get it working with use of CriteriaQuery.
JPACriteriaQueryVisitor<Order, Order> jpa
= new JPACriteriaQueryVisitor<>(em, Order.class, Order.class);
searchCondition.accept(jpa);
CriteriaQuery<Order> cq = jpa.getCriteriaQuery();
CriteriaBuilder cb = em.getCriteriaBuilder();
Root<Order> root = (Root<Order>) cq.getRoots().iterator().next();
cq.orderBy(cb.desc(root.get("id")));
TypedQuery<Order> query = jpa.getTypedQuery();
query.setFirstResult((page * perPage) - perPage);
query.setMaxResults(perPage);
return query.getResultList();
You cannot change the order after creating the query.
If you use MySQL, then maybe you can use an integer parameter in ORDER BY :colnum DESC to specify the column number by which to sort (starting at 1 from the selected columns only), but you cannot change direction.
PostgreSQL does not allow you to do this. I do not know how it is on other databases, though. With MySQL, the parameters are replaced by the SQL Driver, so it always receives the full query with escaped sequences. With PG, the query is first parsed by the server, the execution plan is created (including if and how to order the results) and then the parameters are sent.
Is any possibility to execute N1QL update query using spring-data?
I.e. I have following update query:
"UPDATE USERS USE KEYS $id SET location = $location"
I tried to use couchbaseTemplate.queryN1QL method, but it doesn't work.
Is there any solution of this problem using spring data or even native couchbase java SDK?
the CouchbaseTemplate is more tailored towards Spring Data document's use case, so it expects to run queries that return specific elements and will end up unmarshalled into entities (eg. a User object).
if you want to run a more freeform query, like your update, you can always access the native SDK from the template. In your case:
String paStatement = "UPDATE USERS USE KEYS $id SET location = $location";
JsonObject paramValues = JsonObject.create().put("id", theId).put("location", "theLocation");
N1qlQuery query = N1qlQuery.parametrized(paStatement, paramValues);
template.getCouchbaseBucket().query(query);
I think you can add RETURNING * at the end of your UPDATE statement. It would make the statement returns something. It works for me.
I recently encountered the following problem with buiding queries in jooq (version 3.1.0):
I want to build delete statement with order and limit constraints. So, my aim is to build something like this:
DELETE FROM table ORDER BY field DESC LIMIT 1 (this is MySql syntax)
But i haven't found nesessary methods in result delete query object:
DSLContext context = createContext();
DeleteWhereStep delete = context.delete(createTable(table));
DeleteConditionStep whereStep = delete.where(condition);
whereStep.orderBy(...)//and no such method here
There are all nesessary methods in select statements and none for delete.
Is it possible to set order and limit for delete request in jooq?
As of jOOQ 3.2, these sorts of extensions are currently not implemented yet. Chances are, that #203 could be implemented in jOOQ 3.3, though.
In the mean time, you have two options:
Resort to plain SQL
i.e. write something like:
context.execute("DELETE FROM {0} ORDER BY {1} DESC LIMIT 1",
createTable(table),
field);
Manually transform your SQL statement into something equivalent
I suspect that the ORDER BY .. LIMIT extension to the MySQL DELETE statement is just sugar for:
DELETE FROM table t
WHERE t.id IN (
SELECT id FROM table
ORDER BY field LIMIT 1
)
Or with jOOQ:
context.delete(TABLE)
.where(TABLE.ID.in(
select(TABLE.ID)
.from(TABLE)
.orderBy(TABLE.FIELD)
.limit(1)
))
Is it possible to create a sqlite prepared statement in OrmLite?
If so, how to bind the query values which may change across different queries.
Is it possible to create a sqlite prepared statement in OrmLite?
You need to RTFM since ORMLite's online documentation is pretty extensive. If you look in the index for "prepared statement" you find out about the QueryBuilder which #Egor pointed out.
how to bind the query values which may change across different queries.
A little further in that section you learn about select arguments which is how you bind query values that change across queries. This is in the index under "arguments to queries".
To quote from the docs here's how you prepare a custom query:
QueryBuilder<Account, String> queryBuilder = dao.queryBuilder();
Where<Account, String> where = queryBuilder.where();
SelectArg selectArg = new SelectArg();
// define our query as 'name = ?'
where.eq("name", selectArg);
// prepare it so it is ready for later query or iterator calls
PreparedQuery<Account> preparedQuery = queryBuilder.prepare();
When you are ready to run the query you set the select argument and issue the query:
selectArg.setValue("foo");
List<Account> accounts = dao.query(preparedQuery);
Later, you can set the select argument to another value and re-run the query:
selectArg.setValue("bar");
accounts = accountDao.query(preparedQuery);