Develop Reference

Unable to parse 2022-10-04T19:24:50Z format in ElasticSearch Java Implemnetation

SearchRequest searchRequest = Requests.searchRequest(indexName);
SearchSourceBuilder builder = new SearchSourceBuilder();
Gson gson = new Gson();
QueryBuilder querybuilder = QueryBuilders.wrapperQuery(query);
query : {
"range": {
"timecolumn": {
"gte":"2022-10-07T09:45:13Z",
"lte":"2022-10-07T09:50:50Z"
}
}
}
While passing the above Query I am getting Parser Exception , I cannot change the date format as data in DB is getting inserted in same format .
Need Advice on :
How can we parse this kind of timestamp in ElasticSearch Java , if not
How can we control pattern updation during data insertion like my column in defined as text which takes date format "2022-10-07T09:45:13Z" as text .
either I have to pass this format in ES Parser or I have to change format to 2022-10-07 09:45:13 during insertion itself .
I cannot convert for each row after inserting because we have lakhs of data

As you are mentioning, Elasticsearch storing timecolumn as text type hence i will suggest to change mapping of timecolumn to date type and you will be able to use range query with date. Because if you store date as text and applied range then it will not return a expected result.
{
"mappings": {
"properties": {
"timecolumn": {
"type": "date"
}
}
}
}
Now coming to your Java code issue, You can use below code for creating range query in Java as you are using Java client.
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
QueryBuilder query = QueryBuilders.rangeQuery("timecolumn").gte("2022-10-07T09:45:13Z").lte("2022-10-07T09:50:50Z");
searchSourceBuilder.query(query);
searchRequest.source(searchSourceBuilder);
Regarding your concern about reindexing data:
I cannot convert for each row after inserting because we have lakhs of
data
You can use Reindex API to move data from original index to temp index and delete your original index. Then, defined the index with proper mapping and again use same Reindex API to copy data from temp index to original index with new mapping.

Coinbase pagination returning {"errors":[{"id":"not_found","message":"Not found"}]}

I am trying to iterate over a paginated list of accounts but when I send a request using the value from "next_uri" I receive an error from the server:
{"errors":[{"id":"not_found","message":"Not found"}]}
I am correctly adding headers etc as all other API calls work fine, its just the request using the "next_uri" that is not working. I think I am following the api spec correctly so I am unsure what is the issue and how to fix it. Does anyone know what is wrong with the code / logic please?
Simplified code:
ArrayList<X> results = new ArrayList<>();
String uri = "/v2/accounts";
javax.ws.rs.client.Client client = getClient();
while(uri != null){
T response = client.target("https://api.coinbase.com")
.path(uri).request(MediaType.APPLICATION_JSON).get(responseType);
results.addAll(response.getData());
uri = response.getPagination()==null ? null :response.getPagination().getNextUri();
}
return results;
The results are this:
Request 1:
https://api.coinbase.com/v2/accounts
Response 1: pagination":
{"ending_before":null,"starting_after":null,"previous_ending_before":null,"next_starting_after":"ef35df6c-a45b-5858-b755-f12a709cf26e","limit":25,"order":"desc","previous_uri":null,"next_uri":"/v2/accounts?starting_after=ef35df6c-a45b-5858-b755-f12a709cf26e"},"data":[{....}]
Request 2:
https://api.coinbase.com/v2/accounts%3Fstarting_after=ef35df6c-a45b-5858-b755-f12a709cf26e
Response 2:
{"errors":[{"id":"not_found","message":"Not found"}]}

This was related to how the jax-rs library needs the query params adding. Just relying on the uri is not enough, the parameters also need adding specifically:
target = target.queryParam(e.getKey(), e.getValue());
so the final code is something like
WebTarget target = client.target(e"https://api.coinbase.com");
if(params !=null){
for(Map.Entry<String, String> e : params.entrySet()){
target = target.queryParam(e.getKey(), e.getValue());
}
}
target = target.path(path);
return target.request(MediaType.APPLICATION_JSON).get(responseType);

Pagination on Google Analytics using Java

I'm implementing a data extraction from Google Analytics using Java and I'm following this example: https://developers.google.com/analytics/devguides/reporting/core/v4/quickstart/service-java
I managed to extract the data I need but I can't figure out how to set the start-index using its client. Below you can see the changed I made to the default implementation. I can set the page size but I can't find out how to set the start-index.
public GetReportsResponse getReport(String dateStart, String dateEnd) throws IOException {
String[] metricsArr = {"ga:users", "ga:newUsers", "ga:sessions", "ga:totalEvents"};
String[] dimensionsArr = {"ga:eventLabel","ga:eventCategory","ga:eventAction", "ga:country", "ga:countryIsoCode", "ga:dateHourMinute"};
// Create the DateRange object.
DateRange dateRange = new DateRange();
dateRange.setStartDate(dateStart);
dateRange.setEndDate(dateEnd);
// Create the Metrics object.
ArrayList<Metric> metrics = new ArrayList<Metric>();
for(String item : metricsArr){
Metric m = new Metric().setExpression(item).setAlias(item.replace("ga:", ""));
metrics.add(m);
}
ArrayList<Dimension> dimensions = new ArrayList<Dimension>();
for(String item : dimensionsArr){
Dimension d = new Dimension().setName(item);
dimensions.add(d);
}
// Create the ReportRequest object.
ReportRequest request = new ReportRequest()
.setViewId(this.VIEW_ID)
.setDateRanges(Arrays.asList(dateRange))
.setMetrics(metrics)
.setDimensions(dimensions)
.setFiltersExpression("ga:eventCategory=#NOTICE,ga:eventCategory==Document,ga:eventCategory==Document reader")
.setPageSize(10000);
ArrayList<ReportRequest> requests = new ArrayList<ReportRequest>();
requests.add(request);
// Create the GetReportsRequest object.
GetReportsRequest getReport = new GetReportsRequest().setReportRequests(requests);
// Call the batchGet method.
GetReportsResponse response = service.reports().batchGet(getReport).execute();
// Return the response.
return response;
}
How can I achieve that so I can navigate through all pages and extract all items?

The Reporting API V4 uses page tokens. The reply from the reporting API will return the next page's token, see nextPageToken. Using that you can make the exact same call but updating the pageToken in the request with the nextpagetoken from the previous reply. Note that the first call you make the reporting API will not have a page token attached to the request and the last page will not have the nextpagetoken set.
I hope that helps.

How to parse protobuf data of Envelope.Payload.Data?

I’m using Hyperledger Fabric Java SDK to get a transaction by txId. The return object includes Transaction Information.
TransactionInfo txInfo = channel.queryTransactionByID(txId);
Common.Envelope envelope = txInfo.getEnvelope();
Common.Payload payload = Common.Payload.parseFrom(envelope.getPayload());
The Payload message includes headers and data. I can parse headers by using Common.Header.ChannelHeader and Common.Header.SignatureHeader.
Common.ChannelHeader channelHeader = Common.ChannelHeader.parseFrom(payload.getHeader().getChannelHeader());
Common.SignatureHeader signatureHeader = Common.SignatureHeader.parseFrom(payload.getHeader().getSignatureHeader());
The problem is, I cannot see any message type to get data from Payload.
My expectation would be like,
SomeMessage someMsg = SomeMessage.parseFrom(payload.getData());
What is the ideal approach to get a data object?

Thanks to Driden Myung's tip, Finally found a way to parse QSCC responses into TxReadWriteSet or even KVRWSet !!
Here is an example:
TransactionInfo txInfo = channel.queryTransactionByID(txId);
Common.Envelope envelope = txInfo.getEnvelope();
Common.Payload payload = Common.Payload.parseFrom(envelope.getPayload());
FabricTransaction.Transaction transaction = FabricTransaction.Transaction.parseFrom(payload.getData());
FabricTransaction.TransactionAction action = transaction.getActionsList().get(0); // 0 is a index
FabricTransaction.ChaincodeActionPayload chaincodeActionPayload = FabricTransaction.ChaincodeActionPayload.parseFrom(action.getPayload());
FabricProposalResponse.ProposalResponsePayload prp = FabricProposalResponse.ProposalResponsePayload.parseFrom(chaincodeActionPayload.getAction().getProposalResponsePayload());
FabricProposal.ChaincodeAction ca = FabricProposal.ChaincodeAction.parseFrom(prp.getExtension());
Rwset.TxReadWriteSet txrws = Rwset.TxReadWriteSet.parseFrom(ca.getResults());
TxReadWriteSetInfo txrwsInfo = new TxReadWriteSetInfo(txrws);
KvRwset.KVRWSet kvrwSet = txrwsInfo.getNsRwsetInfo(0).getRwset();
KvRwset.KVWrite kvWrite = kvrwSet.getWrites(0);
String writeVal = kvWrite.getValue().toStringUtf8();

I found the answer.
FabricTransaction.Transaction transaction = FabricTransaction.Transaction.parseFrom(payload.getData());
After that,
FabricTransaction.TransactionAction action = transaction.getActionsList().get(index);
FabricTransaction.ChaincodeActionPayload chaincodeActionPayload = FabricTransaction.ChaincodeActionPayload.parseFrom(action.getPayload());
chaincodeActionPayload.getAction().getEndorsementsList().forEach(endorsement -> {
// This is my current point
???? endorser = ????.parseFrom(endorsement.getEndorser());
});
Let me add if I can find more. Anybody add comments welcome.

We have faced a similar problem to get the request data from a transaction.
The following code will help to get the transaction request data of a transaction
Fabric SDK version : 2.1.4
// get transaction from transaction ID
TransactionInfo txInfo = channel.queryTransactionByID(txId);
// transaction is stored inside the envelope containing the payload and signature
Common.Envelope envelope = txInfo.getEnvelope();
// parse payload from the envelope
Common.Payload payload = Common.Payload.parseFrom(envelope.getPayload());
// payload contains Header and Data. We are parsing data to get the transaction
TransactionPackage.Transaction transaction = TransactionPackage.Transaction.parseFrom(payload.getData());
// get first action from the transaction action list. it contains input and other details
TransactionPackage.TransactionAction action = transaction.getActionsList().get(0); // 0 is a index
// chaincode action payload contains input parameters. So we are taking the action payload
TransactionPackage.ChaincodeActionPayload chaincodeActionPayload = TransactionPackage.ChaincodeActionPayload.parseFrom(action.getPayload());
// chaincode ProposalPayload contains Input and TransientMap. We are parsing actionPayload to proposalPayload
ProposalPackage.ChaincodeProposalPayload prp = ProposalPackage.ChaincodeProposalPayload.parseFrom(chaincodeActionPayload.getChaincodeProposalPayload());
// parse the input to chaincodeInvocationSpec so that we can unmarshal the input
Chaincode.ChaincodeInvocationSpec chaincodeInvocationSpec = Chaincode.ChaincodeInvocationSpec.parseFrom(prp.getInput());
// get the input and parse the arg list and get input arguments
chaincodeInvocationSpec.getChaincodeSpec().getInput().getArgsList().get(ChaincodeInput.ARGS_FIELD_NUMBER).toStringUtf8();

AWS SSM parameter store not fetching all key/values

Could someone let me know why the below code only fetching few entries from the parameter store ?
GetParametersByPathRequest getParametersByPathRequest = new GetParametersByPathRequest();
getParametersByPathRequest.withPath("/").setRecursive(true);
getParametersByPathRequest.setWithDecryption(true);
GetParametersByPathResult result = client.getParametersByPath(getParametersByPathRequest);
result.getParameters().forEach(parameter -> {
System.out.println(parameter.getName() + " - > " + parameter.getValue());
});

GetParametersByPath is a paged operation. After each call you must retrieve NextToken from the result object, and if it's not null and not empty you must make another call with it added to the request.
Here's an example using DescribeParameters, which has the same behavior:
DescribeParametersRequest request = new DescribeParametersRequest();
DescribeParametersResult response;
do
{
response = client.describeParameters(request);
for (ParameterMetadata param : response.getParameters())
{
// do something with metadata
}
request.setNextToken(response.getNextToken());
}
while ((response.getNextToken() != null) && ! respose.getNextToken.isEmpty());

Here is the code, based on the code above, for the new 2.0 version of AWS SSM manager. Notice I have set the maxResults to 1 to prove out the loop. You will want to remove that. AWS has mentioned that in the new code they wanted to emphasize immutability.
Using this dependency:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>ssm</artifactId>
<version>2.10.32</version>
</dependency>
I came up with this code:
private void refreshCache() {
StopWatch sw = StopWatch.createStarted();
GetParametersByPathRequest request = GetParametersByPathRequest.builder()
.path(prefix)
.withDecryption(useDecryption)
.maxResults(1)
.build();
GetParametersByPathResponse response;
do {
response = ssm.getParametersByPath(request);
for (Parameter p : response.parameters()) {
//do something with the values.
}
request = GetParametersByPathRequest.builder()
.path(prefix)
.withDecryption(useDecryption)
.nextToken(response.nextToken())
.maxResults(1)
.build();
}
while (StringUtils.isNotBlank(response.nextToken()));
LOG.trace("Refreshed parameters in {}ms", sw.getTime());
}

private void getSsmParams() {
AWSSimpleSystemsManagement client = AWSSimpleSystemsManagementClientBuilder.defaultClient();
GetParametersByPathRequest request = new GetParametersByPathRequest();
request.withRecursive(true);
request.withPath('/your/path/parameterName').setWithDecryption(true);
GetParametersByPathResult response;
do {
response = client.getParametersByPath(request);
for (Parameter p : response.parameters()) {
//do something with the values. maybe add to a list
}
request.setNextToken(response.getNextToken())
}
while (StringUtils.isNotBlank(response.getNextToken()));
}
Above piece of code worked for me .ssm only sends 10 parameters at a time, so if you want to fetch more than 10 parameters from ssm parameter store programatically you will have to use multiple calls to fetch them. here the token is important , if there are more values in the path (request.withPath('/your/path/parameterName')) you have given, it will send a token indicating that there are more values in the given path ,and you will have to make the following request with the token received from the previous request in order to get the rest of the values.