I have a list of status enum values which I am currently iterating over and using a basic counter to store how many in my list have the specific value that I am looking for. I want to improve greatly on this however and think that there may be a way to use some kind of JPA query on a paging and sorting repository to accomplish the same thing.
My current version which isn't as optimized as I would like is as follows.
public enum MailStatus {
SENT("SENT"),
DELETED("DELETED"),
SENDING("SENDING"),
}
val mails = mailService.getAllMailForUser(userId).toMutableList()
mails.forEach { mail ->
if (mail.status === MailStatus.SENT) {
mailCounter++
}
}
With a paging and sorting JPA repository is there some way to query this instead and get a count of all mail that has a status of sent only?
I tried the following but seem to be getting everything rather than just the 'SENT' status.
fun countByUserIdAndMailStatusIn(userId: UUID, mailStatus: List<MailStatus>): Long
Related
I'm writing an android app that supports exporting the app database to various formats. I don't want to run out of memory, but I want to page the results easily without receiving updates when it changes. So I put it in a service, and came up with the following method of paging.
I use a limit clause in my query to limit the number of results returned and I'm sorting on the primary key. So it should be fast. I use a set of nested for loops to execute the series of queries until no results are returned, and walk through the given results, so that's linear. It's in a service, so it doesn't matter that I'm using immediate result things here.
I feel like I might be doing something bad here. Am I?
// page through all results
for (List<CountedEventType> typeEvents = dao.getEventTypesPaged2(0);
typeEvents.size() > 0;
typeEvents = dao.getEventTypesPaged2(typeEvents.get(typeEvents.size() - 1).uid)
) {
for (CountedEventType type : typeEvents) {
// Do something for every result.
}
}
Here's my dao method.
#Dao
interface ExportDao {
#Query("SELECT * FROM CountedEventType WHERE uid > :lastUid ORDER BY uid ASC LIMIT 4")
List<CountedEventType> getEventTypesPaged2(int lastUid);
}
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I'm new to spring framework so the questions might come up as silly.
I have a database with almost 5000 entries in it. I need to create a GET request which takes 5 parameters to filter the data. Depending on what parameters are present, the request will filter the data. I was able to do it but I don't think I am doing it efficiently. So here are my questions:
First, Which one is a better approach? Retrieving all data from database using repository.findAll() and then using stream + filter to filter out the data OR writing the query in JPA repository interface and then simply calling those methods? Which one would be more efficient?
Second, What is the best way to retrieve a huge amount of data? Like in my case, there are 5000 entries. So how should I retrieve them? I've read something about Pageable but not 100% sure. Is that the way to go or is there any other better option?
Any help appreciated. Thanks :)
For the first question is better to retrieve only required records from DB, instead of retrieve all entries and then filter them on Java .Writing the query in JPA repository one of the options , but also you can use CriteriaQuery to do this . CriteriaQuery given you more manipulate on fillture items on programmatically way . Also it help you with your second question .
Yes Pagination is one of approach , special for Web Applications . The Main idea of pagination is to dividing large records of data to smaller chunks (Pages) , user search for his record on first chuck (Page) then he/she will request the a second page if he/she did found it .
Below example summarize your two queries . In this example am trying to retrive/search on large number of orders .
Bean OrderSearchCriteria.java , use to identify filter parameter .
public class OrderSearchCriteria {
private String user ;
private Date periodFrom ;
private Date periodTo ;
private String status ;
private Integer pageLimit ;
private Integer page ;
private Integer offset ;
private String sortOrder ;
.....
}
Repository
public interface OrderRepository extends JpaRepository<Order, Integer> , JpaSpecificationExecutor<Order>{}
Below using CriteriaQuery to filter orders based on submitted criteria .
#Service
public class OrderServiceImpl implements OrderService{
......
#Override
public Page<Order> orderSearch(OrderSearchCriteria orderSearchCriteria) {
if (orderSearchCriteria.getPage() == null)
orderSearchCriteria.setPage(orderSearchCriteria.getOffset() / orderSearchCriteria.getPageLimit());
return orderRepository.findAll(OrderSearchSpecificaton.orderSearch(orderSearchCriteria) ,
PageRequest.of(orderSearchCriteria.getPage(), orderSearchCriteria.getPageLimit()));
}
private static class OrderSearchSpecificaton {
public static Specification<Order> orderSearch(OrderSearchCriteria orderSearchCriteria) {
return new Specification<Order>() {
private static final long serialVersionUID = 1L;
#Override
public Predicate toPredicate(Root<Order> root, CriteriaQuery<?> query, CriteriaBuilder criteriaBuilder) {
List<Predicate> predicates = new ArrayList<>();
if (!StringUtils.isEmpty(orderSearchCriteria.getUser()) && !orderSearchCriteria.getUser().toUpperCase().equals("ALL")) {
Join<Order, User> userJoin = root.join("user") ;
predicates.add(criteriaBuilder.equal(userJoin.get("name") ,orderSearchCriteria.getUser()));
}
if (!StringUtils.isEmpty(orderSearchCriteria.getStatus()) && !orderSearchCriteria.getStatus().toUpperCase().equals("ALL")) {
predicates.add(criteriaBuilder.equal(root.get("status") ,orderSearchCriteria.getStatus()));
}
if (orderSearchCriteria.getPeriodFrom() != null) {
predicates.add(criteriaBuilder.greaterThanOrEqualTo(root.get("entryDate"), orderSearchCriteria.getPeriodFrom())) ;
}
if (orderSearchCriteria.getPeriodTo()!= null) {
predicates.add(criteriaBuilder.lessThan(root.get("entryDate"), orderSearchCriteria.getPeriodTo())) ;
}
if (!StringUtils.isEmpty(orderSearchCriteria.getSortOrder())) {
if (orderSearchCriteria.getSortOrder().toUpperCase().equals("DESC")) {
query.orderBy(criteriaBuilder.desc(root.get("entryDate"))) ;
}
else {
query.orderBy(criteriaBuilder.asc(root.get("entryDate"))) ;
}
}
return criteriaBuilder.and(predicates.toArray(new Predicate[predicates.size()]));
}
};
}
}
Call orderSearch from Controller
#ResponseBody
#RequestMapping(path = "/order/search" , method = RequestMethod.POST)
public HashMap<String, Object> orderSearch(#RequestBody OrderSearchCriteria orderSearchCriteria) {
Page<Order> page = getOrderService().orderSearch(orderSearchCriteria) ;
HashMap<String, Object> result = new HashMap<>() ;
result.put("total", page.getTotalElements());
result.put("rows", page.getContent());
return result ;
}
I hope this can help you .
What is better depends on context. Only you know what is better in your context. Nevertheless I'd suggest you to consider following solution.
1) Use Spring Data JPA Specifications
You say that some of 5 parameters can be present, some not. I'd suggest you to use Spring Data JPA Specifications. Here is a good article and examples.
The idea is following. For each of your 5 parameters you create a specification. In this example these are methods customerHasBirthday() and isLongTermCustomer() in the class CustomerSpecifications.
Then you create a query dynamically, depending on what parameters are present:
if (parameter1 is present){
add specification 1 to the "where" clause
}
...
if (parameter5 is present){
add specification 5 to the "where" clause
}
Then calls findAll() using the resulting aggregated specification.
Of course other solutions are possible: You can build a JPQL Query as a string dynamically, depending on what parameters are present. Or you can dynamically build a native SQL query. But specifications have one more advantage: pageable queries in Spring accept only specifications.
2) Use Paging
If your application has only 2-3 users that send only a few requests per hour, then loading 5000 items per request might work well. But if all the results need to be rendered in browser, this can take a lot of resources on the client and can be a performance problem.
If you have more users that send more requests, then also on the server side CPU and RAM can be insufficient and you can face performance problems and, as a consequence, very long response time for users.
That's why I'd suggest you to use Paging. You can limit the number of elements in the response. Suppose you set page size to 100. Then each request will need less resources:
On database level: Instead of 5000 database will return only 100 elements, it will be better performance
Application will create from JDBC response only 100 Java objects instead of 5000 -> less memory and less CPU used
Application will have less overhead with converting Java objects to JSON, again less memory and less CPU
The response time will be better, because sending of 100 elements from application to the user takes less time than sending 5000 elements
Browser performance can be better. It depends on the client logic. In case client application is not smart and renders every response element, this will be a higher performance, because rendering of 100 elements will be faster than rendering of 5000 elements.
There are many tutorials about paging, do one or two that you like.
I'am writing a software for the data-synchronization of a custom software and sugarCRM. Therefore I need an updateOrCreate() function. My Problem is, that the custom software uses other uuid´s than sugarCRM so i can´t look for the uuid to check on update or create.So I want to save the custom-uuid in a custom field of sugarCRM.
But i have no idea how to do that over the REST-API of sugarCRM.
By the way I wrote a java-application.
Thank you for help!
As far as I'm aware there is no update-or-create API (see https://your-sugarsite/rest/v10/help), howewer if you just want to use the API (rather than customize it) you could sync data like this:
1) Fetch all ids of records that have a custom uuid by using the POST /rest/v10/<module>/filter endpoint and a payload similar to:
{
offset: 0,
max_num: 1000,
fields: ["id", "custom_uuid_c"],
filter: [{"custom_uuid_c": {"$not_empty": ""}}],
]
}
or if you just need a specific custom uuid at a time:
{
offset: 0,
max_num: 1000,
fields: ["id"],
filter: [{"custom_uuid_c": {"$equals": "example-custom-uuid"}}],
]
}
The response will look something like this:
{
next_offset: -1,
records: [
{"id": "example-sugar-uuid", "custom_uuid_c": "example-custom-uuid"},
...
],
}
Notes:
Make sure to evaluate next_offset as even with a high max_num you may not get all records at once because of server limits. As long as next_offset isn't -1 you should use its value as offset in a new request to get the remaining records.
You can supply all field names you need to sync in the fields array, so that you get that information early and can check whether or not an update is required at all (maybe data is still up-to-date?)
Sugar also always include certain fields in the response, no matter if they were requested or not. (E.g. id and date_modified). I did not include them all in the response snippets for the sake of simplicity.
2)
Based on the information received in the previous step you know which sugar ID belongs to which custom UUID and you can detect/prepare data for updates.
If you need to sync all and retrieve the complete list first, I suggest you create a lookup table custom-uuid => sugar-id, so that you do not have to loop through the data array and compare fields when looking for a specific number.Don't forget to consider the possibility of a custom-uuid being present in one than more Sugar-record at a time, unless you enforce them being unique on the server/database side.
3)
Now that you have all the information you need you can update and create records as needed:
Update existing record: PUT /rest/v10/<module>/<record_id>
Create missing record: POST /rest/v10/<module>
If want to send a lot of creates and/or updates in a single request, have a look at the POST /rest/v10/bulk API - if your version of Sugar has it.
Final notes:
The filter operators definition on /rest/v10/help seems incomplete, for more info you can check the filter docs
I came across this blog post in looking for a way to organize relationships. What I'm getting confused on is the syntax behind the following statement. I realize by virtue of the javascript variables, the following is possible..
var party = {
_id: "chessparty",
name: "Chess Party!",
attendees: ["seanhess", "bob"]
}
var user = { _id: "seanhess", name: "Sean Hess", events: ["chessparty"]}
db.events.save(party)
db.users.save(user)
db.events.find({_id: {$in: user.events}}) // events for user
db.users.find({_id: {$in: party.attendees}}) // users for event
What is throwing me for a spin in the last two lines though, since what I'm trying to do is something like this in Java. So I understand the idea, but I want to accomplish this in Java, more specifically, the Camel/MongoDB component.
I've been referencing the following documentation and looking at the "findAll" operation. So would I need to first run a query to get the array, for example the "user.events" and then run a second query to find the list of events? Or is there a way to reference the field "events" in collection "db.user" as part of the query on "db.events"?
Something to the tune of the following with a single query..
pseudo idea: db.events.find({_id: {$in: [db.user.events]}})
Ultimately I'm looking to translate this into something like the following..
from("direct:findAll")
.setBody().constant("{ \"_id\": {$in :\"user.events\" }}")
.to("mongodb:myDb?database=sample&collection=events&operation=findAll")
.to("mock:resultFindAll");
I'm a bit new to the mongodb camel component, so I'm wondering if there are any gurus that have already been there done that sort of thing?? And have any advice on the subject. Or to find out without 2 days of trial and error that this simple isn't possible..?
Thanks!
I thought I'd wrap this question up, it has been some time now and a few weeks ago I was able to work past this.
Basically I would up storing an array of userId's in the events collection..
example:
{
_id : 22bjh2345j2k3v235,
eventName : "something",
eventDate : ISODate(...),
attendees : [
"abc123",
"def098",
"etc..."
]
}
essentially assigning users to events. This way I could find all events a user was participating in, and I wound up with a list of users per event.
if I wanted to find all events for a user:
from("direct:findAll")
.setBody().simple("{ \"attendees\": \"${header.userId}\" }")
.to("mongodb:myDb?database=sample&collection=events&operation=findAll")
.to("mock:resultFindAll");
I am new to ElasticSearch and Couchbase. I am building a sample Java application to learn more about ElasticSearch and Couchbase.
Reading the ElasticSearch Java API, Filters are better used in cases where sort on score is not necessary and for caching.
I still haven't figured out how to use FilterBuilders and have following questions:
Can FilterBuilders be used alone to search?
Or Do they always have to be used with a Query? ( If true, can someone please list an example? )
Going through a documentation, if I want to perform a search based on field values and want to use FilterBuilders, how can I accomplish that? (using AndFilterBuilder or TermFilterBuilder or InFilterBuilder? I am not clear about the differences between them.)
For the 3rd question, I actually tested it with search using queries and using filters as shown below.
I got empty result (no rows) when I tried search using FilterBuilders. I am not sure what am I doing wrong.
Any examples will be helpful. I have had a tough time going through documentation which I found sparse and even searching led to various unreliable user forums.
private void processQuery() {
SearchRequestBuilder srb = getSearchRequestBuilder(BUCKET);
QueryBuilder qb = QueryBuilders.fieldQuery("doc.address.state", "TX");
srb.setQuery(qb);
SearchResponse resp = srb.execute().actionGet();
System.out.println("response :" + resp);
}
private void searchWithFilters(){
SearchRequestBuilder srb = getSearchRequestBuilder(BUCKET);
srb.setFilter(FilterBuilders.termFilter("doc.address.state", "tx"));
//AndFilterBuilder andFb = FilterBuilders.andFilter();
//andFb.add(FilterBuilders.termFilter("doc.address.state", "TX"));
//srb.setFilter(andFb);
SearchResponse resp = srb.execute().actionGet();
System.out.println("response :" + resp);
}
--UPDATE--
As suggested in the answer, changing to lowercase "tx" works. With this question resolved. I still have following questions:
In what scenario(s), are filters used with query? What purpose will this serve?
Difference between InFilter, TermFilter and MatchAllFilter. Any illustration will help.
Right, you should use filters to exclude documents from being even considered when executing the query. Filters are faster since they don't involve any scoring, and cacheable as well.
That said, it's pretty obvious that you have to use a filter with the search api, which does execute a query and accepts an optional filter. If you only have a filter you can just use the match_all query together with your filter. A filter can be a simple one, or a compund one in order to combine multiple filters together.
Regarding the Java API, the names used are the names of the filters available, no big difference. Have a look at this search example for instance. In your code I don't see where you do setFilter on your SearchRequestBuilder object. You don't seem to need the and filter either, since you are using a single filter. Furthermore, it might be that you are indexing using the default mappings, thus the term "TX" is lowercased. That's why when you search using the term filter you don't find any match. Try searching for "tx" lowercased.
You can either change your mapping if you want to keep the "TX" term as it is while indexing, probably setting the field as not_analyzed if it should only be a single token. Otherwise you can change filter, you might want to have a look at a query that is analyzed, so that your query wil be analyzed the same way the content was indexed.
Have a look at the query DSL documentation for more information regarding queries and filters:
MatchAllFilter: matches all your document, not that useful I'd say
TermFilter: Filters documents that have fields that contain a term (not analyzed)
AndFilter: compound filter used to put in and two or more filters
Don't know what you mean by InFilterBuilder, couldn't find any filter with this name.
The query usually contains what the user types in through the text search box. Filters are more way to refine the search, for example clicking on facet entries. That's why you would still have the query plus one or more filters.
To append to what #javanna said:
A lot of confusion can come from the fact that filters can be defined in several ways:
standalone (with a required query, for instance match_all if all you need is the filters) (http://www.elasticsearch.org/guide/reference/api/search/filter/)
or as part of a filtered query (http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query/)
What's the difference you might ask. And indeed you can construct exactly the same logic in both ways.
The difference is that a query operates on BOTH the resultset as well as any facets you have defined. Whereas, a Filter (when defined standalone) only operates on the resultset and NOT on any facets you may have defined (explained here: http://www.elasticsearch.org/guide/reference/api/search/filter/)
To add to the other answers, InFilter is only used with FilterBuilders. The definition is, InFilter: A filter for a field based on several terms matching on any of them.
The query Java API uses FilterBuilders, which is a factory for filter builders that can dynamically create a query from Java code. We do this using a form and we build our query based on user selections from it with checkboxes, options, and dropdowns.
Here is some Example code for FilterBuilders and there is a snippet from that link that uses InFilter as shown below:
FilterBuilder filterBuilder;
User user = (User) auth.getPrincipal();
if (user.getGroups() != null && !user.getGroups().isEmpty()) {
filterBuilder = FilterBuilders.boolFilter()
.should(FilterBuilders.nestedFilter("userRoles", FilterBuilders.termFilter("userRoles.key", auth.getName())))
.should(FilterBuilders.nestedFilter("groupRoles", FilterBuilders.inFilter("groupRoles.key", user.getGroups().toArray())));
} else {
filterBuilder = FilterBuilders.nestedFilter("userRoles", FilterBuilders.termFilter("userRoles.key", auth.getName()));
}
...