Utilizing the Spring Framework's Page interface I'm able to retrieve 50 users at a time with the code below:
int page = 0;
int pageSize = 50;
Page<User> userPage = userRepo.findByFirstName("Bob", page, pageSize);
List<User> userList = userPage.getContent();
This runs some query in the background to give me my first 50 users in my database. What I want to do is populate my userList without paging so that every is returned from the underlying query - is this possible?
I've tried the hasNext() method on my Page object but it runs in an endless loop
while(userPage.hasNext()){
logger.info("Next page found");
}
Any ideas?
If you don't want paging you can ignore spring Page and directly call userRepo.findByFirstName("Bob");
This will return all users in database whose firstName is bob
Your method signature should be
List<User> users = userRepo.findByFirstName("Bob");
Hope this solves your question
If you always want all users and not them by page, you need to adjust this on the Repository side. Change the signature of findByFirstName to return List<User> instead of Page<User>, and remove the Pageable parameter. Then, it will always return all. See more information on how parameters are handled in the Spring Data documentation.
If you need to sometimes paginate and sometimes get all data, then just add the List<User> version. The return type will depend on whether you pass a Pageable parameter or not based on Java's usual overloading.
The hasNext() method on Page merely tells you if a next page is available (for use in presenting navigation), it doesn't run another query. You would need to construct a new Pageable object and iterate through several calls to the Page<User> version in order to access all data through the Pageable interface, which is less efficient than just querying for what you actually want in the first place.
Related
I want to fetch all records from a table using findAll and do some processing on each of them, but I'm not sure if it will give memory issues if the number of records is huge like in millions.
I have looked into the Pageable but I'm not sure how to iterate over all the data using Pageable approach. Is it even possible to fetch few records at a time process them and fetch them again until all the records are processed?
And what would be better? Fetch all the records in Iterable using findAll() method or the Pageable approach?
Don't use findAll if there is a lot of entities.
If you want to use pagination you can do something like this:
Pageable pageRequest = PageRequest.of(0, 200);
Page<Qmail> onePage = repository.findAll(pageRequest);
while (!onePage.isEmpty()) {
pageRequest = pageRequest.next();
//DO SOMETHING WITH ENTITIES
onePage.forEach(entity -> System.out.println(entity.getId()));
onePage = repository.findAll(pageRequest);
}
Since Spring Data 1.8 you can Stream over results.
Stream<Record> findAll();
Important is here that you add a QueryHint about the fetch size for the database. If set it internally uses pages for streaming over the results.
Use this for MySQL databases:
#QueryHints(value = #QueryHint(name = org.hibernate.jpa.QueryHints.HINT_FETCH_SIZE, value = "-2147483648"))
Stream<Record> findAll();
For none-MySQL databases you can play with the fetch size:
#QueryHints(value = #QueryHint(name = org.hibernate.jpa.QueryHints.HINT_FETCH_SIZE, value = "5000"))
Stream<Record> findAll();
And, if you do not update / delete the records, do not forget to set your transaction to read-only:
#Transactional(readOnly = true)
If it can be millions..
1) Do not use findAll() and retrieve a list of actual managed entities. If you only need to read the data then use a projection query along with Spring Data JPA projection interface. This will bypass the persistence context and save a lot of time and memory.
2) Use Paging (to save memory) and make sure make each call in a new transaction (#Transactional(propagation = REQUIRES_NEW)). This will allow other transactions not to hang forever which might be the case if you did NOT use paging and triggered only one, give me all, query.
3) It looks like a candidate for an overnight batch job also. Think about that.
What you need is read data as a batch and process each of them and may be persist same somewhere else or generate report out of it.
This the ETL use case.
Spring Batch can be used for this case which can handle it very well.
Reader reads the data one at a time and process it in processor. Writer will persist or generate report based on chunk/batch size you set.
This way you are not holding a lot of data in memory.
I have this spring data repository method
public FileActivity findTopByFileIDOrderByCreatedDesc(String fileID);
This works fine . But how do I make it work for list of parameters ?
This doesn't work (I can have many FileActivity for file id - but I want only the last one) :
public List<FileActivity> findTopByFileIDOrderByCreatedDesc(List<String> fileIDs);
Spring Data's support for derived queries is useful but for anything other than simple queries it is probably easier and clearer just to define your own JPQL query.
#Query("select f from File f where f.id in :fileIds order by f.created desc")
public Page<FileActivity> findTopFilesById(
#Param("fileIDs") List<String> fileIDs, Pageable pageable);
As JPQL does not have a limit keyword you can simply pass in a Page.
List<String> fileIds = //;
Page<File> page = repository.findTopFilesById(fileIds, new PageRequest(0, fileIds.size());
List<File> files = page.getContent();
You could also dynamically specify the sort order in the PageRequest rather than in the JPQL giving a bit more flexibility:
https://docs.spring.io/spring-data/commons/docs/current/api/org/springframework/data/domain/PageRequest.html
public FileActivity findTopByFileIDInOrderByCreatedDesc(Collection<String> fileIDs);
See more examples here
Note - if you use 'Top' then you get only one record. To get more than one record you have to add a number to 'Top' for example: 'Top10'.
More info here
I'm new with Spring Data and Spring MVC and I don't understand why am I getting empty content:
#RequestMapping(value="/pages", method=RequestMethod.GET)
#ResponseBody
public Page<Client> contactsPages(#RequestParam int page, #RequestParam int size) {
Pageable pageable = new PageRequest(page, size, new Sort("id"));
Page<Client> pageResult = clientRepository.findAll(pageable);
return pageResult;
}
The result of my json when I test the url is:
{"content":[],"last":true,"totalElements":2,"totalPages":1,"size":5,"number":1,"sort":[{"direction":"ASC","property":"id","ignoreCase":false,"nullHandling":"NATIVE","ascending":true}],"first":false,"numberOfElements":0}
And if you have good example making pageable request using Spring Data and Spring MVC and AngularJS; It will be a big help for me.
Ensure that your PageRequest object is requesting 0 for small sets, not 1.
The pagination begins from 0.
This is a common mistake for beginners and is a common redherring when using #Query in conjunction with Spring pagination. If your #Query works without pagination and then returns nothing when using it, check the page number.
I had a similar issue and with mine it was due to the fact that I modify the data after doing a select such that the modified field is the same field I use for searching. This means each time I run the query it will match less and less records so page zero will always have new data, so in that instance using page 0 always will not hurt as new records will always fill page zero. In the other instance I did not modify the search field and my query will match the same records each time I execute my select statement so in this instance requesting the exact page was important.
Dharman's answer helps me find out the reason.
If you use pagination to get data from DB, and also you update the search field after you retrieve it, then the page info will get messed up.
I have a Spring CrudRepository called 'ImportReceiptRepository'.
I'm simply trying to compose a method which grabs the first row in an order by clause.
Here's what I'm using currently:
ImportReceipt importReceipt = this.importReceiptRepository.getOneByImportTypeOrderByTimestampDesc(importType);
The issue is that when there is more than a single row returned, Spring throws a:
org.springframework.dao.IncorrectResultSizeDataAccessException: result returns more than one elements; nested exception is javax.persistence.NonUniqueResultException: result returns more than one elements
How should I rename this CrudRepository function to simply grab the first row when there are 0-n rows returned?
Simply using Pageable was the key:
List<ImportReceipt> findByImportType(String importType, Pageable pageable);
and calling it like:
List<ImportReceipt> importReceipts = this.importReceiptRepository.findByImportType(importType, new PageRequest(0, 1, Direction.DESC, "Timestamp"));
ImportReceipt importReceipt = importReceipts.get(0);
Credit: How to combine pagination with a criteria query in Spring Data JPA?
You can make it even simpler by findFirst:
ImportReceipt findFirstByImportType(String importType);
For more info: Limiting Query Results
As of the upcoming version 1.7 of Spring Data JPA, the query method as you originally declared it works out of the box. See this ticket for details. The support has already been published in the first milestone of the release.
Hi all question I'm having problems finding an answer for...
Use Case:
Reading in an Excel spreadsheet in a controller. A 4 row sheet was processed and row 2 and 3 had some errors in it, so I skip them and move on with the rest of the processing. But I want to retain these rows to display to the user after the processing is complete. I'd like to retain all the values in the row to display back to the user just to provide enough information.
So what I tried to do was this:
#RequestMapping(value = "/bulk-create", method = RequestMethod.POST)
public String bulkCreate(Model model, SpreadSheetFile spreadSheetFile){
...some code...
List<Row> errorRows = new LinkedList<Row>();
...some code to process rows...
//Error Found
errorRows.add(row);
...more code to wrap up processing (possibly more errors)...
model.addAttribute("erroRows", errorRows);
return "redirect:/bulk-review"
}
#RequestMapping(value = "/bulk-review", method = RequestMethod.GET)
public String bulkReview(Model model,
#ModelAttribute("errorRows")
LinkedList<Row> errorRows){
model.addAttribute("errorRows", errorRows);
return "bulk-review";
}
So basically I'm trying to pass a LinkedList from one MVC method to another without actually hitting a page first (unless I'm misunderstanding how "redirect:" works... it calls the controller method first correct?)
When I but a break point into the bulkReview method, errorRows is empty... So how do I retain this value between the two methods?
Thanks for the help! Let me know if I need to clarify anything! :)
Kris
I don't think the redirect mechanism works the way you think it does. The first controller sends back a URL to the user's browser which the browser then follows. And by default, Spring places any data in the model at the time of a redirect in the redirect URL, so that the page-being-redirected to can "see" the model data. However, this doesn't work well when you have complex data in the model.
So, you need to think about how the data will be passed in the redirect URL that the browser sends back to the user. How does a List get serialized to the URL? probably not very well.
Another option would be to put the data in session from one controller and access it in the second.