Develop Reference

Cumulocity microservice deployment: access is denied

My goal (background info)
I want to develop a java microservice on cumulocity. It should be able to do the following:
I would send "112233" to the microservice as follows:
https://myTenant.cumulocity.com/service/my-application-name/decode?data=112233
The microservice should then split the data into "11" for the first measurement and "22" for the second measurement etc. These measurements would be POSTed to cumulocity.
My problem
I am now stuck on getting the Hello, microservice tutorial to work. I can't deploy the microservice (zip file) to cumulocity.
"error":"security/Forbidden","info":"https://www.cumulocity.com/guides/reference-guide/#error_reporting","message":"Access is denied"} (I am an admin user.)
I also tried to upload the zip file via the website, this created a HOSTED application instead of a MICROSERVICE. Uploading my zip via a POST request to a HOSTED application actually works (which I obviously don't need).
I suspect that I get the "access denied" error cause cumulocity thinks that I upload a HOSTED application to a MICROSERVICE.
What I've done so far
Code side
I downloaded the hello-world-microservice example from the cumulocity bitbucket development branch. (This code is not available on the default branch).
I changed the cumulocity versions to 9.3.0, only this version seems to exist.
The HelloWorldMain.java is unedited
This is my cumulocity.json manifest file: (the roles make no difference)
{
"apiVersion":"1",
"type":"MICROSERVICE",
"version":"#project.version#",
"availability":"PRIVATE",
"provider":{
"name":"Cumulocity GmbH"
},
"isolation":"MULTI_TENANT",
"requiredRoles":[
"ROLE_APPLICATION_MANAGEMENT_ADMIN",
"ROLE_MEASUREMENT_ADMIN",
"ROLE_INVENTORY_ADMIN"
],
"roles":[
"ROLE_APPLICATION_MANAGEMENT_ADMIN",
"ROLE_MEASUREMENT_ADMIN",
"ROLE_INVENTORY_ADMIN"
],
"livenessProbe":{
"httpGet":{
"path":"/health",
"port":80
},
"initialDelaySeconds":15,
"periodSeconds":10
},
"readinessProbe":{
"httpGet":{
"path":"/health",
"port":80
}
}
}
This is my application.properties file
application.name=my-application-name
server.port=80
C8Y.baseURL=https://myTenant.cumulocity.com
C8Y.bootstrap.tenant=myTenant
C8Y.bootstrap.user=servicebootstrap_my-application-name
C8Y.bootstrap.password={SECRET_BOOTSTRAP_PASSW}
C8Y.user={MY_USERNAME}
C8Y.password={SECRET_PASSW}
C8Y.bootstrap.register=true
C8Y.microservice.isolation=MULTI_TENANT
C8Y.bootstrap.initialDelay=10000
Cumulocity side
I successfully created a microservice application,
GET https://myTenant.cumulocity.com/application/applications/5886 returns:
{
"availability":"PRIVATE",
"id":"5886",
"key":"my-application-key",
"manifest":{
"imports":[
],
"noAppSwitcher":true
},
"name":"my-application-name",
"owner":{
"self":"https://myTenant.cumulocity.com/tenant/tenants/myTenant",
"tenant":{
"id":"myTenant"
}
},
"requiredRoles":[
"ROLE_APPLICATION_MANAGEMENT_ADMIN",
"ROLE_MEASUREMENT_ADMIN",
"ROLE_INVENTORY_ADMIN"
],
"roles":[
"ROLE_APPLICATION_MANAGEMENT_ADMIN",
"ROLE_MEASUREMENT_ADMIN",
"ROLE_INVENTORY_ADMIN"
],
"self":"https://myTenant.cumulocity.com/application/applications/5886",
"type":"MICROSERVICE"
}
I also successfully subscribed to this application.
When I try to upload the zip file to cumulocity, I get this error:
"error":"security/Forbidden","info":"https://www.cumulocity.com/guides/reference-guide/#error_reporting","message":"Access is denied"}
(Uploading to a HOSTED type application works fine, but I don't want that.)
note: I also tried to use the microservice deploy script, this gave the same result as doing everything manually.
Trying to run it locally
Since I couldn't get it to work on the cumulocity platform, I tried to run it locally via docker. I ran it with this command:
docker run -e "C8Y_MICROSERVICE_ISOLATION=MULTI_TENANT" 10aa0b73ddb3
note: I had to add the "C8Y_MICROSERVICE_ISOLATION=MULTI_TENANT" environment variable. if I didn't add this, I'd get credential/permission issues. This seems weird to me, since all other info is read from the application.properties file except for this one.
I have no errors when running this image on a local docker.
According to the Hello, microservice tutorial, I should be able to request curl -H "Authorization: {AUTHORIZATION}" https://myTenant.cumulocity.com/service/my-application-name/hello?who=me
This returns:
{"error":"microservice/Not Found","info":"https://www.cumulocity.com/guides/reference-guide/#error_reporting","message":"Microservice my-application-name not found."}
Back to the questions
Has anyone else had difficulties with setting up a microservice on cumulocity?
Is there something I'm totally overseeing?

The microservice hosting needs to be assigned to your tenant otherwise it won't work and the API in that case will return forbidden. So it might be that it is no issue with your user but that your tenant has the feature not activated.

Conversation-with-Discovery "Exception on Query:null" - No Response from Chat App

I have successfully been able to deploy and run this demo for using conversation-with-discovery.
I have spent the last 2 weeks trying to run the application with my own data, and provide a query on the Chat App localhost that I know should provide results from my data from Watson Discovery, I get the following:
http://localhost:9081/dist/index.html: "Service seems to be down.Please try again
after sometime or Please check the logs." "No response from Watson"
On Terminal Command Line: "10:40:03.549 [Default Executor-thread-79]
INFO com.ibm.watson.apis.conversation_with_discovery.discovery.DiscoveryQuery - Query: searchText:CO2 Absorption
10:40:04.360 [Default Executor-thread-79]
INFO com.ibm.watson.apis.conversation_with_discovery.discovery.DiscoveryClient - Creating Discovery Payload
10:40:04.361 [Default Executor-thread-79]
ERROR com.ibm.watson.apis.conversation_with_discovery.rest.ProxyResource - Exception on Query:null"
Now if I run a query on the Local Chat App that doesn't make sense to Watson Discovery I get the following:
http://localhost:9081/dist/index.html: "Sorry I haven't learned answers to questions like this" " "log_messages": [],
"CEPayload": [
{
"text": "empty",
"textSnippet": "empty",
"confidence": "0.0",
"sourceUrl": "empty",
"title": "No results found"
}
]
},
"input": {
"text": "CO#""
On Terminal Command Line: "10:52:39.614 [Default Executor-thread-228]
INFO com.ibm.watson.apis.conversation_with_discovery.discovery.DiscoveryQuery - Query: searchText:CO#
10:52:40.385 [Default Executor-thread-228]
INFO com.ibm.watson.apis.conversation_with_discovery.discovery.DiscoveryClient - Creating Discovery Payload"
My intuition tells me that something is not right when Discovery API tries to send back a response to the Chat App via the Java application I am using but when I ask something that Discovery cannot answer I get "empty" fields for my Payload, as shown above.
Based on what I have found regarding this type of set up tried the following:
I changed the fields on the Java Application to match with what is shown on discovery as fields on the data schema and on the API
I modified the manifest.yml file: buildpack: liberty-for-java_v3_7-20170118-2046" and "env: JVM: openjdk as suggested on a few forums
I downloaded java-sdk-java-sdk-4.0.0 but am not sure how to update my current Java SDK. Not sure if it's the actual problem so haven't deeply approached this route.
I am running on macOS Sierra version 10.12.6. Perhaps something is incompatible with my OS?
I just need some guidance before I continue down a rabbit hole that holds no solution. Could it be the Java Application or is this an internal Java system issue on my laptop? I have searched forums and seen both situations.

YCSB Error with scan operation on mysql db

I am trying to run a scan-only workload using YCSB client on MySQL database (a 10GB setup). I am using the following command to run the workload
./bin/ycsb run jdbc -P workloads/workloademod -p db.driver=com.mysql.jd bc.Driver -p db.url=jdbc:mysql://localhost:3306/ycsbtest -p db.user=username -p maxscanlength=100 -p recordcount=5000 -p operationcount=1000 -p maxexecutiontime=1800 > try_ycsb_emod
But I get java GC out of memory exception and the workload gets killed as follows:-
Loading workload...
Starting test.
Maximum execution time specified as: 1800 secs
Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3380)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3327)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1928)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3278)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:462)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2997)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2245)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2638)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2030)
at com.yahoo.ycsb.db.JdbcDBClient.scan(JdbcDBClient.java:348)
at com.yahoo.ycsb.DBWrapper.scan(DBWrapper.java:107)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionScan(CoreWorkload.java:605)
at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:491)
at com.yahoo.ycsb.ClientThread.run(Client.java:233)
Could not wait until max specified time, TerminatorThread interrupted.
I have tried increasing the java heap size for ycsb client by doing "export YCSB_HEAP_SIZE=8000" to make it 8G, still I get the same error.
I checked the mysql query that the ycsb client sends to mysql server and I don't see a reference/implication to the max_scan_length metric. Also, running this mysql query using a mysql client instead of the ycsb client works fine.
I am not sure what can I do to fix the same. Can someone help me out here?
Thanks,
Rene

It's a bug with the jdbc implementation:
https://github.com/brianfrankcooper/YCSB/issues/171

Monitoring concurrent user in Zabbix

Say I want to monitor number of concurrent user at this moment in our production server using Zabbix http://www.zabbix.com/. Can anyone let me know how can I do that? The servers are written in java.

First question to answer is: how you can query the number of users on the server? Is this something you can grep out of some log file? Or maybe your 'server written in java' allows to query the user count in some standardized way (i.e. via http)?
You might be able to use one of standard zabbix checks, but most probably you'll have to configure zabbix agent running at your server to monitor a custom User Parameter: https://www.zabbix.com/documentation/1.8/manual/config/user_parameters.
So, needed value should be parsed out of json available via http. Starting with version 2.2 you'll be able to do it with web.page.regexp[] but since it's not production ready yet you'll have to stick to custom user items.
Supposedly zabbix agent is running on linux. Open the zabbix_agentd.conf and add the following line:
UserParameter=json.query[*], value=$(curl -s $1 | grep -Eo "\"$2\":(\w*)"| cut -d: -f2); if [ -z "$value" ]; then echo "not found"; else echo $value; fi
Restart the agent. After this new user parameter json.query will become available as "Zabbix Agent" type item on this server. It takes two arguments: address of json to fetch and name of variable to extract. I was testing on the following sample:
"usercount":30
You might need to alter the regex if your json is formatted differently.
Now you need to add the new zabbix item: Configuration/Hosts/Items - Create Item
Description: Concurrent user count
Type: Zabbix Agent
Key: json.query[myserver.com/status.json,usercount]
Data Type: numeric
That's it. User count is now monitored and you can reference it in zabbix triggers the usual way.

How can I monitor/log Tomcat's thread pool?

I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?
Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.
Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?
Any help much appreciated :D
Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.

Direct JMX access
Try adding this to catalina.sh/bat:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
UPDATE: Alex P suggest that the following settings might also be required in some situations:
-Dcom.sun.management.jmxremote.host=localhost
This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.
What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.
JMX proxy servlet
Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:
$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool
A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.

You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true
If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true
{
request: {
mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
attribute: [
"currentThreadsBusy",
"maxThreads",
"currentThreadCount",
"connectionCount"
],
type: "read"
},
value: {
currentThreadsBusy: 1,
connectionCount: 4,
currentThreadCount: 10,
maxThreads: 200
},
timestamp: 1490396960,
status: 200
}
Note: This example works on Tomcat7 +.

For a more enterprise solution. I have been using New Relic in our production environment.
This provides a graph of the changes to the threadpool over time.

There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm
You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.