Develop Reference

Pagination in CosmosDB Java SDK with continuation token

I'm trying to create from an async client a method to retrieve items from a CosmosDB but I'm afraid I'm full of questions and little to no documentation from Microsoft side
I've created a function that will read from a cosmosDB a list of items, page by page, which continuation will depend on a continuityToken. The methos looks like this. Please, be aware there could be some minor mistakes non related to the core functionality which is reading page by page:
#FunctionName("Feed")
public HttpResponseMessage getFeed(
#HttpTrigger(
name = "get",
methods = { HttpMethod.GET },
authLevel = AuthorizationLevel.ANONYMOUS,
route = "Feed"
) final HttpRequestMessage<Optional<String>> request,
#CosmosDBInput(
name = "Feed",
databaseName = Constants.DATABASE_NAME,
collectionName = Constants.LOG_COLLECTION_NAME,
sqlQuery = "SELECT * FROM c", // This won't be used actually as we use our own query
connectionStringSetting = Constants.CONNECTION_STRING_KEY
) final LogEntry[] logEntryArray,
final ExecutionContext context
) {
context
.getLogger()
.info("Query with paging and continuation token");
String query = "SELECT * FROM c"
int pageSize = 10; //No of docs per page
int currentPageNumber = 1;
int documentNumber = 0;
String continuationToken = null;
double requestCharge = 0.0;
// First iteration (continuationToken = null): Receive a batch of query response pages
// Subsequent iterations (continuationToken != null): Receive subsequent batch of query response pages, with continuationToken indicating where the previous iteration left off
do {
context
.getLogger()
.info("Receiving a set of query response pages.");
context
.getLogger()
.info("Continuation Token: " + continuationToken + "\n");
CosmosQueryRequestOptions queryOptions = new CosmosQueryRequestOptions();
Flux<FeedResponse<LogEntry>> feedResponseIterator =
container.queryItems(query, queryOptions, LogEntry.class).byPage(continuationToken,pageSize);
try {
feedResponseIterator.flatMap(fluxResponse -> {
context
.getLogger()
.info("Got a page of query result with " +
fluxResponse.getResults().size() + " items(s)"
+ " and request charge of " + fluxResponse.getRequestCharge());
context
.getLogger()
.info("Item Ids " + fluxResponse
.getResults()
.stream()
.map(LogEntry::getDate)
.collect(Collectors.toList()));
return Flux.empty();
}).blockLast();
} catch (Exception e) {
}
} while (continuationToken != null);
context
.getLogger()
.info(String.format("Total request charge: %f\n", requestCharge));
return request
.createResponseBuilder(HttpStatus.OK)
.header("Content-Type", "application/json")
.body("ALL READ")
.build();
}
For simplicity the read items are merely logged.
First question: We are using an async document client that returns a Flux. Will the client keep track of the token? It is a stateless client in principle. I understand that the sync client could take easily care of this case, but wouldn't the async client reset its memory of tokens after the first page and token has been generated?
Second: Is the while loop even appropriated? My assumption is a big no, as we need to send back the token in a header and the frontend UI will need to send the token to the Azure Function in a header or other similar fashion. The token should be extracted from the context then
Third: Is the flatMap and blockList way to read the flux appropriate? I was trying to play with the subscribe method but again I don't see how it could work for an async client.
Thanks a lot,
Alex.
UPDATE:
I have observed that Flux only uses the items per page value to set the number of items to be retrieved per batch, but after retrieval of one page it doesn't stop and keeps retrieving pages! I don't know how to stop it. I have tried substituting the Flux.empty() per Mono.empty() and setting a LIMIT clause in the sql query. The first option does the same and the second freezes the query and never returns apparently. How can I return one page an only one page along with the continuation token to do the following query once the user clicks on the next page button?

Adding roles to a mentioned user in JDA

I want to be able to run a command that begins with !suspend, mentions a user, then determines a time length and adds a role named 'Suspended' to the mentioned user for the specified time length.
Message message = event.getMessage(); //Sets the message equal to the variable type 'Message' so it can be modified within the program.
String content = message.getContentRaw(); //Gets the raw content of the message and sets it equal to a string (best way to convert types Message to String).
MessageChannel channel = event.getChannel(); //Gets the channel the message was sent in (useful for sending error messages).
Guild guild = event.getGuild();
//Role group = content.matches("((Suspended))") ? guild.getRolesByName("Suspended", true).get(0) : null;
if(content.startsWith("!suspend"))//Pretty self explanatory. Enters the loop if the message begins with !suspend.
{
String[] spliced = content.split("\\s+"); //Splits the message into an array based on the spaces in the message.
TextChannel textChannel = event.getGuild().getTextChannelsByName("ranked-ms_punishments",true).get(0); //If there is a channel called ranked-ms_punishments which there should be set the value of it equal to the variable.
int length = spliced.length;//Sets 'length' equal to the number of items in the array.
if(length == 3)//If the number of items in the array is 3 then...
{
if(spliced[1].startsWith("<"))
{
list.add(spliced[1]);
String tempuser = spliced[1];
textChannel.sendMessage(tempuser + " you have been suspended for " + spliced[2] + " days.").queue();//Sends the message in the quotations.
//add role to the member
}
}else {
channel.sendMessage("Please use the following format for suspending a user: '!suspend' <#user> (length)").queue(); //If length doesn't equal 3 then it sends the message in quotations.
}
}
Not sure how to do this, as I'm too unfamiliar with JDA to make it work. I have everything working except for the actual addition of the role named 'Suspended'.

Adding a role to the mentioned members in a message can be done as follows:
Role role = event.getGuild().getRolesByName("Suspended", false).get(0);
List<Member> mentionedMembers = message.getMentionedMembers();
for (Member mentionedMember : mentionedMembers) {
event.getGuild().getController().addRolesToMember(mentionedMember, role).queue();
}
Note that your bot will need the MANAGE_ROLES permission to add roles.

You can do this: 👍
event.getGuild().addRoleToMember(memberId, jda.getRoleById(yourRole));

event.getGuild().addRoleToMember(member, event.getGuild().getRoleById(yourRole)).queue();

How to pull more than 500 user objects from my domain?

I am trying to pull about 20,000 users from my Google domain. However, i know that Google only has a limit of about 500 users for a pull request. I know about the pageToken stuff, but the documentation for it online is terrible. Can someone show me how to use the pageToken? Please keep in mind i am using the google client libraries. This is what my code looks like so far:
#Test
public void paginationTest() throws IOException, NullPointerException, GeneralSecurityException {
try {
Directory directory = GCAuthentication.getDirectoryService("xxx", "vvv", dddd);
Directory.Users.List list = directory.users().list().setOrderBy("email").setMaxResults(500).setDomain("dev.royallepage.ca");
do {
com.google.api.services.admin.directory.model.Users users = list.execute();
java.util.List<User> uL = users.getUsers();
//uL.addAll(users.getUsers());
//list.setPageToken(list.getPageToken());
System.out.println(uL.size());
}while (list.getPageToken() != null && list.getPageToken().length() > 0);
}catch(NullPointerException e) {
}
}
Please advise what i am doing wrong! Thanks,
Mesam

You will have to create a function that will get the pageToken variable then call another request including the nextPageToken.
Use the pageToken query string for responses with large number of groups. In the case of pagination, the response returns the nextPageToken property which gives a token for the next page of response results. Your next request uses this token as the pageToken query string value.
Sample Code Request:
GET https://www.googleapis.com/admin/directory/v1/users
?domain=primary domain name&pageToken=token for next results page
&maxResults=max number of results per page
&orderBy=email, givenName, or familyName
&sortOrder=ascending or descending
&query=email, givenName, or familyName:the query's value*
Hope this helps!

Retrieve multiple messages from SQS

I have multiple messages in SQS. The following code always returns only one, even if there are dozens visible (not in flight). setMaxNumberOfMessages I thought would allow multiple to be consumed at once .. have i misunderstood this?
CreateQueueRequest createQueueRequest = new CreateQueueRequest().withQueueName(queueName);
String queueUrl = sqs.createQueue(createQueueRequest).getQueueUrl();
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages) {
// i'm a message from SQS
}
I've also tried using withMaxNumberOfMessages without any such luck:
receiveMessageRequest.withMaxNumberOfMessages(10);
How do I know there are messages in the queue? More than 1?
Set<String> attrs = new HashSet<String>();
attrs.add("ApproximateNumberOfMessages");
CreateQueueRequest createQueueRequest = new CreateQueueRequest().withQueueName(queueName);
GetQueueAttributesRequest a = new GetQueueAttributesRequest().withQueueUrl(sqs.createQueue(createQueueRequest).getQueueUrl()).withAttributeNames(attrs);
Map<String,String> result = sqs.getQueueAttributes(a).getAttributes();
int num = Integer.parseInt(result.get("ApproximateNumberOfMessages"));
The above always is run prior and gives me an int that is >1
Thanks for your input

AWS API Reference Guide: Query/QueryReceiveMessage
Due to the distributed nature of the queue, a weighted random set of machines is sampled on a ReceiveMessage call. That means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
and
MaxNumberOfMessages: Maximum number of messages to return. SQS never returns more messages than this value but might return fewer.

There is a comprehensive explanation for this (arguably rather idiosyncratic) behaviour in the SQS reference documentation.
SQS stores copies of messages on multiple servers and receive message requests are made to these servers with one of two possible strategies,
Short Polling : The default behaviour, only a subset of the servers (based on a weighted random distribution) are queried.
Long Polling : Enabled by setting the WaitTimeSeconds attribute to a non-zero value, all of the servers are queried.
In practice, for my limited tests, I always seem to get one message with short polling just as you did.

I had the same problem. What is your Receive Message Wait Time for your queue set to? When mine was at 0, it only returned 1 message even if there were 8 in the queue. When I increased the Receive Message Wait Time, then I got all of them. Seems kind of buggy to me.

I was just trying the same and with the help of these two attributes setMaxNumberOfMessages and setWaitTimeSeconds i was able to get 10 messages.
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
receiveMessageRequest.setWaitTimeSeconds(20);
Snapshot of o/p:
Receiving messages from TestQueue.
Number of messages:10
Message
MessageId: 31a7c669-1f0c-4bf1-b18b-c7fa31f4e82d
...

receiveMessageRequest.withMaxNumberOfMessages(10);
Just to be clear, the more practical use of this would be to add to your constructor like this:
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueUrl).withMaxNumberOfMessages(10);
Otherwise, you might as well just do:
receiveMessageRequest.setMaxNumberOfMessages(10);
That being said, changing this won't help the original problem.

Thanks Caoilte!
I faced this issue also. Finally solved by using long polling follow the configuration here:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-long-polling-for-queue.html
Unfortunately, to use long polling, you must create your queue as FIFO one. I tried standard queue with no luck.
And when receiving, need also set MaxNumberOfMessages. So my code is like:
ReceiveMessageRequest receive_request = new ReceiveMessageRequest()
.withQueueUrl(QUEUE_URL)
.withWaitTimeSeconds(20)
.withMaxNumberOfMessages(10);
Although solved, still feel too wired. AWS should definitely provide a more neat API for this kind of basic receiving operation.
From my point, AWS has many many cool features but not good APIs. Like those guys are rushing out all the time.

For small task list I use FIFO queue like stackoverflow.com/a/55149351/13678017
for example modified AWS tutorial
// Create a queue.
System.out.println("Creating a new Amazon SQS FIFO queue called " + "MyFifoQueue.fifo.\n");
final Map<String, String> attributes = new HashMap<>();
// A FIFO queue must have the FifoQueue attribute set to true.
attributes.put("FifoQueue", "true");
/*
* If the user doesn't provide a MessageDeduplicationId, generate a
* MessageDeduplicationId based on the content.
*/
attributes.put("ContentBasedDeduplication", "true");
// The FIFO queue name must end with the .fifo suffix.
final CreateQueueRequest createQueueRequest = new CreateQueueRequest("MyFifoQueue4.fifo")
.withAttributes(attributes);
final String myQueueUrl = sqs.createQueue(createQueueRequest).getQueueUrl();
// List all queues.
System.out.println("Listing all queues in your account.\n");
for (final String queueUrl : sqs.listQueues().getQueueUrls()) {
System.out.println(" QueueUrl: " + queueUrl);
}
System.out.println();
// Send a message.
System.out.println("Sending a message to MyQueue.\n");
for (int i = 0; i < 4; i++) {
var request = new SendMessageRequest()
.withQueueUrl(myQueueUrl)
.withMessageBody("message " + i)
.withMessageGroupId("userId1");
;
sqs.sendMessage(request);
}
for (int i = 0; i < 6; i++) {
var request = new SendMessageRequest()
.withQueueUrl(myQueueUrl)
.withMessageBody("message " + i)
.withMessageGroupId("userId2");
;
sqs.sendMessage(request);
}
// Receive messages.
System.out.println("Receiving messages from MyQueue.\n");
var receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
receiveMessageRequest.setMaxNumberOfMessages(10);
receiveMessageRequest.setWaitTimeSeconds(20);
// what receive?
receiveMessageRequest.withMessageAttributeNames("userId2");
final List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (final Message message : messages) {
System.out.println("Message");
System.out.println(" MessageId: "
+ message.getMessageId());
System.out.println(" ReceiptHandle: "
+ message.getReceiptHandle());
System.out.println(" MD5OfBody: "
+ message.getMD5OfBody());
System.out.println(" Body: "
+ message.getBody());
for (final Entry<String, String> entry : message.getAttributes()
.entrySet()) {
System.out.println("Attribute");
System.out.println(" Name: " + entry
.getKey());
System.out.println(" Value: " + entry
.getValue());
}
}

Here's a workaround, you can call receiveMessageFromSQS method asynchronously.
bulkReceiveFromSQS (queueUrl, totalMessages, asyncLimit, batchSize, visibilityTimeout, waitTime, callback) {
batchSize = Math.min(batchSize, 10);
let self = this,
noOfIterations = Math.ceil(totalMessages / batchSize);
async.timesLimit(noOfIterations, asyncLimit, function(n, next) {
self.receiveMessageFromSQS(queueUrl, batchSize, visibilityTimeout, waitTime,
function(err, result) {
if (err) {
return next(err);
}
return next(null, _.get(result, 'Messages'));
});
}, function (err, listOfMessages) {
if (err) {
return callback(err);
}
listOfMessages = _.flatten(listOfMessages).filter(Boolean);
return callback(null, listOfMessages);
});
}
It will return you an array with a given number of messages

Elasticsearch - Assigning Shards

I have recently discovered Elasticsearch and I decided to have a play. Unfortunately I am having trouble with adding indexes.
The code used to add an index is as follows and runs every time a new index is attempted to be added:
public void index ( String index, String type, String id, String json ){
Node node = null;
try{
node = nodeBuilder().node();
Client client = node.client();
IndexResponse response = client.prepareIndex( index, type, id )
.setSource( json )
.execute()
.actionGet();
}
catch ( Exception e ){
Logger.error( e, " Error indexing JSON file: " + json );
}
finally {
if( node != null)
node.close();
}
}
No indexes appear to be added and my Cluster helath is currently red (as one of the shards is red), but I have no idea how to resolve this. I am receiveing confirmation that my index is being added each time but they do not show up when searched or in es-admin.
All help or ideas are greatly appreciated.

When starting a Node, one of the common settings to consider is if it should hold data or not. In other words, should indices and shards be allocated to it. Many times we would like to have the clients just be clients, without shards being allocated to them [1].
If you want to set up your client as being a non-data client (no shards) try setting it up like so by replacing this:
node = nodeBuilder().node();
with this:
node = nodeBuilder().client(true).node();
[1] http://www.elasticsearch.org/guide/reference/java-api/client.html