Spring boot skipping and duplicating items in pagination - java

I have a Users endpoint to get all the users in MySQL sorted by a boolean value using pagination, as the number keeps growing I found out it's skipping some users and duplicating others here's some screenshots in postman where a user with the same id (95) is duplicating in 2 different pages
Also sometimes i can't find some users in all the pages
Page 2
Page 3
ServiceImpl
#Override
public PagedResponse<?> findAll(UserPrincipal currentUser, int page, int size) {
AppUtils.validatePageNumberAndSize(page, size);
//return users with 'isBoard' true first
Pageable pageable = PageRequest.of(page, size, Sort.Direction.DESC, "isBoard");
if (currentUser.getAuthorities().contains(new SimpleGrantedAuthority(RoleName.ROLE_ADMIN.toString()))
|| currentUser.getAuthorities().contains(new SimpleGrantedAuthority(RoleName.ROLE_SUPER_ADMIN.toString()))) {
Page<MEUser> contentPage = userRepository.findAll(pageable);
List<MEUser> list = contentPage.getNumberOfElements() == 0 ? Collections.emptyList() : contentPage.getContent();
return new PagedResponse<>(list.stream()
.map(user -> mapToAllUsersDTO(user, new AllUserDTO()))
.collect(Collectors.toList()), contentPage.getNumber(), contentPage.getSize(), contentPage.getTotalElements(), contentPage.getTotalPages(), contentPage.isLast());
} else {
Page<MEUser> contentPage = userRepository.findByMemberTrue(pageable);
List<MEUser> list = contentPage.getNumberOfElements() == 0 ? Collections.emptyList() : contentPage.getContent();
return new PagedResponse<>(list.stream()
.map(user -> mapToAllUsersDTO(user, new AllUserDTO()))
.collect(Collectors.toList()), contentPage.getNumber(), contentPage.getSize(), contentPage.getTotalElements(), contentPage.getTotalPages(), contentPage.isLast());
}
}

This is a database problem rather than in your code. The database does not ensure consistent results when there's multiple rows with the same value of an ordered field.
When using pagination with LIMIT and OFFSET you have to supply an order by field with unique values, so in your case you have to sort by isBoard and id to ensure consistent result.

This is because the if condition is validating only ADMIN and SUPER_USER. user 95 may have other roles that come in else conditions. Please check once and provide the database info for Roles and Users.

i added the id condition to the pagination:
Pageable pageable =
PageRequest.of(page, size, Sort.by("isBoard").descending().and(Sort.by("id")));

Related

How to do pagination with DynamoDBMapper?

I'm developing an application in Quarkus that integrates with the DynamoDB database. I have a query method that returns a list and I'd like this list to be paginated, but it would have to be done manually by passing the parameters.
I chose to use DynamoDBMapper because it gives more possibilities to work with lists of objects and the level of complexity is lower.
Does anyone have any idea how to do this pagination manually in the function?
DynamoDBScanExpression scanExpression = new DynamoDBScanExpression()
.withLimit(pageSize)
.withExclusiveStartKey(paginationToken);
PaginatedScanList<YourModel> result = mapper.scan(YourModel.class, scanExpression);
String nextPaginationToken = result.getLastEvaluatedKey();
You can pass the pageSize and paginationToken as parameters to your query method. The nextPaginationToken can be returned along with the results, to be used for the next page.
DynamoDB Mapper paginates by iterating over the results, by lazily loading the dataset:
By default, the scan method returns a "lazy-loaded" collection. It initially returns only one page of results, and then makes a service call for the next page if needed. To obtain all the matching items, iterate over the result collection.
Ref
For example:
List<Customer> result = mapper.scan(Customer.class, scanExpression);
for ( Customer cust : result ) {
System.out.println(cust.getId());
}
To Scan manually page by page you can use ScanPage
final DynamoDBScanExpression scanPageExpression = new DynamoDBScanExpression()
.withLimit(limit);
do {
ScanResultPage<MyClass> scanPage = mapper.scanPage(MyClass.class, scanPageExpression);
scanPage.getResults().forEach(System.out::println);
System.out.println("LastEvaluatedKey=" + scanPage.getLastEvaluatedKey());
scanPageExpression.setExclusiveStartKey(scanPage.getLastEvaluatedKey());
} while (scanPageExpression.getExclusiveStartKey() != null);
Ref
Ref

Spring Jpa Pageable get content using Java stream

I am doing a search to bring all registered cars 100 by 100 in a paginated way.
Pageable pageable = PageRequest.of(0, 100);
Page<Car> listOfcars = carService.findAll(pageable);
For me to get to the next page, I'm doing it via while:
while(listOfcars.hasNext()){
listOfcars = carService.findAll(listOfcars.nextPageable());
carService.start(listOfcars.getContent());
log.info("New batch sent" + listOfcars.getTotalElements());
}
There is a way to do this via the java stream, searching for the data on a page and right after that already searching for the sequences without having to do it via while ?
you can try by knowing total of available pages (maybe a total count will be needed first) and store each page numeric representation in a collection (let's say pages = [1, 2, ... n] ) then you can create a method OperateListOfCars and call your logic for each page with pages.stream()... maybe create a CustomPageResult wrapper and put there the pages collection will help to have the code more clean.
You don't need stream or loop here, you can use getContent method to return all content in a List from Page
Returns the page content as List.
Pageable pageable = PageRequest.of(0, 100);
Page<Car> listOfcars = carService.findAll(pageable);
List<Car> cars = listOfCars.getContent();

implement tree iterator with JPA

I'm writing spring boot application. I have the HashTree structure, sorted by specific field. And I have a specific method that iterates through the tree, so if it finds a value (used as comparator) it returns it and does not check for other values.
With code it will look something like this:
class Storage {
TreeSet<Book> tree = new TreeSet<>(Comparator.comparingInt(Book::getNumberOfPages));
public Book getBookWithSmallestNumberOfPagesButGreaterThanSpecific(int number) {
iterator = tree.iterator();
while (iterator.hasNext()) {
Book book = iterator.next();
if (book.getNumberOfPages() > number) {
return book;
}
}
return null;
}
}
Please don't jurge that code, it's just for example, real code is much more complicated.
So I want to do the same thing using Spring JPA, and store Books in database.
My plan is:
create index on numberOfPages column in order to iterate it in asc order
get a coursor to the first database record
return current element or get next element in a cursor.
The thing is I dont know how to get a cursor, all JPA methods return all elements in a List<>.
Please challenge my solution. I don't know if I move in a right direction. I'm pretty new to JPA.
You can use Pageable,
Pageable pageable = PageRequest.of(page, size, direction, property);
List<ENTITY> findByProjectId(String projectId, Pageable pageable);

Spring Data Pagination returns wrong number of content as it is requested

currently I am developing an api which uses Spring Data Pagination and I came to the problem that I passing a Pageable object as a request to my repository with which page I want to receive and how many elements should be on it. And with that my object looks like: Pageable pageable = PageRequest.of(0, 2); - so I want first page with two elements. In my database are 3 objects so therefore there will be 2 pages. But what ide shows me is: screenshot. Can anyone tell me why it shows that content is an array of 3 elements but actually I asked for 2 elements?
#Override
public List<NotificationDto> getLast30NotificationsFor(ApplicationUser user) {
Pageable pageable = PageRequest.of(0, 2);
Page<Notification> first30ByOwnerIdOrderByCreatedAtDesc = notificationRepository.findFirst30ByOwnerIdOrderByCreatedAtDesc(user.getId(), pageable);
List<Notification> notifications = new ArrayList<>(first30ByOwnerIdOrderByCreatedAtDesc.getContent());
return notifications.stream()
.map(NotificationToDtoMapper::map)
.collect(toList());
}
Try:
#Override
public List<NotificationDto> getLast30NotificationsFor(ApplicationUser user) {
Pageable page = PageRequest.of(0,2, Sort.by("createdAt").descending());
return notificationRepository.findByOwnerId(user.getId(), page)
.getContent()
.stream()
.map(NotificationToDtoMapper::map)
.collect(toList());
}

Creating Pagination in Spring Data JPA

I am trying to implement pagination feature in Spring Data JPA.
I am referring this Blog
My Controller contains following code :
#RequestMapping(value="/organizationData", method = RequestMethod.GET)
public String list(Pageable pageable, Model model){
Page<Organization> members = this.OrganizationRepository.findAll(pageable);
model.addAttribute("members", members.getContent());
float nrOfPages = members.getTotalPages();
model.addAttribute("maxPages", nrOfPages);
return "members/list";
}
My DAO is following :
#Query(value="select m from Member m", countQuery="select count(m) from Member m")
Page<Organization> findMembers(Pageable pageable);
I am able to show first 20 records, how do I show next 20???
Is there any other pagination example that I can refer??
The constructors of Pageable are deprecated, use of() instead:
Pageable pageable = PageRequest.of(0, 20);
I've seen similar problem last week, but can't find it so I'll answer directly.
Your problem is that you specify the parameters too late. Pageable works the following way: you create Pageable object with certain properties. You can at least specify:
Page size,
Page number,
Sorting.
So let's assume that we have:
PageRequest p = new PageRequest(2, 20);
the above passed to the query will filter the results so only results from 21th to 40th will be returned.
You don't apply Pageable on result. You pass it with the query.
Edit:
Constructors of PageRequest are deprecated. Use Pageable pageable = PageRequest.of(2, 20);
Pageable object by default comes with size 20, page 0, and unsorted
So if you want the next page in front end the url can be sent with query params page, size,sort and these u can test it on postman.
You can use Page, List or Slice.
If you dont need the number of pages, and only need to know if the next page exists, use Slice, since it does not do the "count" query:
for (int i = 0; ; i++) {
Slice<Organization> pageOrganization = organizationRepository.find(PageRequest.of(0, 100));
List<Organization> organizationList = pageOrganization.getContent();
for (Organization org : organizationList) {
// code
}
if (!pageOrganization.hasNext()) {
break;
}
}

Categories

Resources