Develop Reference

AWS IoT MQTT only works on the example topic?

I've just gotten started in AWS and IoT. Using the documentation and the tutorial I managed to get a working publish app ripped from the sample classes:
public static void main(String[] args) throws AWSIotException, InterruptedException {
String clientEndpoint = "<prefix>-ats.iot.us-west-2.amazonaws.com"; // replace <prefix> and <region> with your own
String clientId = "sdk-java-23"; // replace with your own client ID. Use unique client IDs for concurrent connections.
String certificateFile = "athing.cert.pem"; // X.509 based certificate file
String privateKeyFile = "athing.private.key"; // PKCS#1 or PKCS#8 PEM encoded private key file
// SampleUtil.java and its dependency PrivateKeyReader.java can be copied from the sample source code.
// Alternatively, you could load key store directly from a file - see the example included in this README.
SampleUtil.KeyStorePasswordPair pair = SampleUtil.getKeyStorePasswordPair(certificateFile, privateKeyFile);
AWSIotMqttClient client = new AWSIotMqttClient(clientEndpoint, clientId, pair.keyStore, pair.keyPassword);
// optional parameters can be set before connect()
client.connect();
String topic = "sdk/test/java";
String payload = "[\n" +
"{\n" +
" \"id\": \"1231231234123\",\n" +
" \"value\": \"25\",\n" +
" \"unit\": \"°C\",\n" +
" \"timestamp\": \"1585954728\"\n" +
"},\n" +
"{\n" +
" \"id\": \"121231231233\",\n" +
" \"value\": \"26\",\n" +
" \"unit\": \"°B\",\n" +
" \"timestamp\": \"1585254728\"\n" +
"}"+
"]";
System.out.println(payload);
while (true) {
client.publish(topic, AWSIotQos.QOS0, payload);
System.out.println("message sent");
Thread.sleep(2000);
}
}
And I can see the messages coming through successfully on the aws console:
But if I change JUST the publish topic from:
String topic = "sdk/test/java";
to:
String topic = "sensors/temperature";
Now it no longer works. I don't see anything appear in the AWS console and the java program is showing some kind of connection error. My first instinct is some kind of security issue where it's not allowed to publish to any topic other than the one that is used in the sample program. I have no experience with IAM, cognito etc. so I would require some guidance (if that is the cause)
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionSuccess
INFO: Connection successfully established
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionSuccess
INFO: Client connection active: sdk-java
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionFailure
INFO: Connection temporarily lost
Apr 04, 2020 4:29:05 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionFailure
INFO: Client connection lost: sdk-java
Apr 04, 2020 4:29:08 PM com.amazonaws.services.iot.client.core.AwsIotConnection$1 run
INFO: Connection is being retried
Apr 04, 2020 4:29:11 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionSuccess
INFO: Connection successfully established
Apr 04, 2020 4:29:11 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionSuccess
INFO: Client connection active: sdk-java

So it turns out it was just a policy issue, I had no idea you had to define which ClientIDs and which topics are allowed to published/subscribed etc.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Receive"
],
"Resource": [
"arn:aws:iot:us-west-2:<>:topic/sensors/realtime",
"arn:aws:iot:us-west-2:<>:topic/sdk/test/java",
"arn:aws:iot:us-west-2:<>:topic/sdk/test/Python",
"arn:aws:iot:us-west-2:<>:topic/topic_1",
"arn:aws:iot:us-west-2:<>:topic/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:us-west-2:<>:topicfilter/sensors/realtime",
"arn:aws:iot:us-west-2:<>:topicfilter/sdk/test/java",
"arn:aws:iot:us-west-2:<>:topicfilter/sdk/test/Python",
"arn:aws:iot:us-west-2:<>:topicfilter/topic_1",
"arn:aws:iot:us-west-2:<>:topicfilter/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:us-west-2:<>:client/JavaClient2",
"arn:aws:iot:us-west-2:<>:client/sdk-java",
"arn:aws:iot:us-west-2:<>:client/basicPubSub",
"arn:aws:iot:us-west-2:<>:client/sdk-nodejs-*"
]
}
]
}

Mongo: "Waiting for replication not allowed"

I have a tiny groovy script to run .mongo file against a replica set of Mongo DB ver. 3.6:
ScriptOperations scriptOps = new MongoTemplate( mongoClient, 'mydb' ).scriptOps()
[ 'inx.mongo', 'roles.mongo' ].each{
try{
ExecutableMongoScript s = new ExecutableMongoScript( new File( it ).text )
def res = scriptOps.execute s
log.info "processed $it >> $res"
}catch( Throwable t ){
log.error "$it --> $t"
}
}
When the script is run, I'm getting the log:
10:22:23.017 [main] INFO Migrator - processed inx.mongo >> [retval:[createdCollectionAutomatically:false, numIndexesBefore:4, numIndexesAfter:4, note:all indexes already exist, ok:1.0, operationTime:Timestamp{value=6608816253102456876, seconds=1538734942, inc=44}, $clusterTime:[clusterTime:Timestamp{value=6608816253102456876, seconds=1538734942, inc=44}, signature:[hash:org.bson.types.Binary#c98f581, keyId:0]]], ok:1.0, operationTime:Timestamp{value=6608816253102456876, seconds=1538734942, inc=44}, $clusterTime:[clusterTime:Timestamp{value=6608816253102456876, seconds=1538734942, inc=44}, signature:[hash:org.bson.types.Binary#c98f581, keyId:0]]]
10:22:23.087 [main] ERROR Migrator - roles.mongo --> com.mongodb.MongoCommandException: Command failed with error 139: 'Error: Waiting for replication not allowed while holding a lock :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
DB.prototype.updateRole#src/mongo/shell/db.js:1713:19
#:1:1
' on server mozaiq_mongo2_1:27002. The full response is { "operationTime" : { "$timestamp" : { "t" : 1538734942, "i" : 44 } }, "ok" : 0.0, "errmsg" : "Error: Waiting for replication not allowed while holding a lock :\n_getErrorWithCode#src/mongo/shell/utils.js:25:13\nDB.prototype.updateRole#src/mongo/shell/db.js:1713:19\n#:1:1\n", "code" : 139, "codeName" : "JSInterpreterFailure", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1538734942, "i" : 44 } }, "signature" : { "hash" : { "$binary" : "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type" : "00" }, "keyId" : { "$numberLong" : "0" } } } }
So the 1st file inx.mongo seems to be processed fine, but the 2nd one roles.mongo throws the Error: Waiting for replication not allowed while holding a lock.
How can I fix the issue?

confused sending json as a parameter

I've the below java program that prints a json string.
System.out.println(objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(objectsToSerialise));
I tried to find the type of objectsToSerialise with the below line of code.
System.out.println(objectsToSerialise.getClass().getTypeName());
this returned java.util.ArrayList
the output that I get is as below.
[ {
"EntityLabels" : [ {
"StartToken" : 8,
"EntityType" : "Personname",
"EndToken" : 16
}, {
"StartToken" : 24,
"EntityType" : "Amount::Spent",
"EndToken" : 31
} ],
"ExampleText" : "What is Frede's limit? ",
"SelectedIntentName" : "GiftLimit"
} ]
And I'm trying to post the same to my API . And the code is as below.
And luisUrl ="https://westus.api.cognitive.microsoft.com"
HttpPost httpPost = new HttpPost(luisUrl + "/luis/v1.0/prog/apps/" + appId + "/examples?");
httpPost.setHeader("Ocp-Apim-Subscription-Key", subsriptionId);
StringEntity params = new StringEntity(objectsToSerialise.toString());
httpPost.setHeader("Content-Type", "application/json");
httpPost.setEntity(params);
HttpResponse response = httpclient.execute(httpPost);
/* Debug Info */
System.out.println("---------------Start2----------------");
HttpEntity entity = response.getEntity();
String responseString = EntityUtils.toString(entity, "UTF-8");
System.out.println(responseString);
System.out.println("---------------End----------------");
/*End Debug Info*/
if (response.getStatusLine().getStatusCode() == 201) {
success = true;
System.out.println("Success");
} else {
System.out.println("Block");
success = false;
}
And this is always printing else content. i.e. Block. I thought this might be an issue with my calling and tried the same in Postman. with the below details.
URL:https://westus.api.cognitive.microsoft.com/luis/v1.0/prog/apps/{appId}/examples?
Method: post
Headers:
Content-Type:application/json
Ocp-Apim-Subscription-Key:{mySubscriptionKey}
Body:**raw->JSON
**Content:
[ {
"EntityLabels" : [ {
"StartToken" : 8,
"EntityType" : "Personname",
"EndToken" : 16
}, {
"StartToken" : 24,
"EntityType" : "Amount::Spent",
"EndToken" : 31
} ],
"ExampleText" : "What is Frede's limit? ",
"SelectedIntentName" : "GiftLimit"
} ]
To my surprise, when I post this, it is returning me 201, where as my Java code is returning me 400.
This is very confusing. Please let me know where am I going wrong and how can I fix this.
Thanks

objectsToSerialize seems to be an ArrayList, but in your code you construct request body with new StringEntity(objectsToSerialise.toString());. That is, you don't convert your ArrayList to JSON, but instead you just get its string representation which is not a JSON usually.
Try changing
new StringEntity(objectsToSerialise.toString());
to
new StringEntity(objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(objectsToSerialise));

Solrj to query solr : dataimport.properties file

How can I use Solrj to query Solr using the following api:
http://localhost:8983/solr/admin/zookeeper?detail=true&path=%2Fconfigs%2Fmy-search%2Fdataimport.properties
the above api gives the content of the dataimport.properties file.
{
"znode": {
"path": "/configs/my-search/dataimport.properties",
"prop": {
"version": 186,
"aversion": 0,
"children_count": 0,
"ctime": "Sun Oct 16 10:24:04 UTC 2016 (1476613444895)",
"cversion": 0,
"czxid": 479,
"ephemeralOwner": 0,
"mtime": "Fri Mar 24 09:48:50 UTC 2017 (1490348930211)",
"mzxid": 31451,
"pzxid": 479,
"dataLength": 111
},
"data": "#Fri Mar 24 09:48:50 UTC 2017\nname.last_index_time=2017-03-24 09\\:48\\:49\nlast_index_time=2017-03-24 09\\:48\\:49\n"
},
"tree": [
{
"data": {
"title": "dataimport.properties",
"attr": {
"href": "admin/zookeeper?detail=true&path=%2Fconfigs%2Fmy-search%2Fdataimport.properties"
}
}
}
]
}
btw, my Solr is configured in cloud mode.

Given that your Solr is running in Cloud mode the file /configs/my-search/dataimport.properties is into Zookeeper.
SolrJ does not have native API to easily read from Zookeeper.
To read into the Zookeeper I suggest to use use Apache Curator Framework
String dataPath = "/configs/my-search/dataimport.properties";
String connectionString = "zookeeper-ensemble:2181";
CuratorFramework client = CuratorFrameworkFactory.newClient(connectionString, new ExponentialBackoffRetry(1000, 3));
client.start();
byte[] barray = client.getData().forPath(dataPath);
if (barray != null) {
String data = new String(barray);
System.out.println(data);
}

google distance matrix api strange

I am facing a strange problem here,
when I run below url from web browser or from java command line
http://maps.googleapis.com/maps/api/distancematrix/json?origins=416%2063,Sweden&destinations=424%2069,Stor%C3%A5s%20Industrigatan%2020,Angered,G%C3%B6teborg&sensor=false
I get below results.
{
"destination_addresses" : [ "Storås Industrigata 20, 424 69
Angered, Sweden" ],
"origin_addresses" : [ "Gothenburg, Sweden" ],
"rows" : [
{
"elements" : [
{
"distance" : {
"text" : "10.4 km",
"value" : 10388
},
"duration" : {
"text" : "15 mins",
"value" : 924
},
"status" : "OK"
}
]
}
],
"status" : "OK"
}
but when I run the same url from glassfish server I mean sending an
http request from
form submit I get below strange response
{ "destination_addresses" : [ "" ],
"origin_addresses" : [ "Gothenburg, Sweden" ],
"rows" : [
{
"elements" : [
{
"status" : "NOT_FOUND"
}
]
}
],
"status" : "OK"
}
please not destination_addresses is empty in this case and status in
NOT_FOUND.
Java code I used to get the response is
private String getResponse(String URL) throws Exception {
InputStream stream = new URL(URL).openStream();
byte[] array = new byte[stream.available()];
stream.read(array);
return new String(array);
}
please guide me to resolve this issue,
thanks....

Use https
unless google will not let your software to connect.
and
get a api key.
read this:
https://developers.google.com/maps/documentation/distance-matrix/start
https://maps.googleapis.com/maps/api/distancematrix/json?origins=416%2063,Sweden&destinations=424%2069,Stor%C3%A5s%20Industrigatan%2020,Angered,G%C3%B6teborg&sensor=false&key=YOUR_API_KEY
be sure to replace YOUR_API_KEY with your actual API key

I had a similar problem when using the Distance Matrix API in Java. Setting the language in my request seems to make it work:
DistanceMatrixElement distanceMatrixElement;
DistanceMatrix matrix;
try
{
matrix = distanceMatrixApiRequest
.origins(...)
.destinations(...)
.language("en")
.await();
}