Elastic APM elastic.apm.ignore_exceptions does not work - java

I want to use elastic.apm.ignore_exceptions setting to ignore temporary logs from APM Agent when queue in APM Server is full or when APM Server is not working so it receive 503 error.
So exceptions from co.elastic.apm.agent.report.IntakeV2ReportingEventHandler
Based on doc:https://www.elastic.co/guide/en/apm/agent/java/master/config-core.html
I added elastic.apm.ignore_exceptions in elasticapm.properties, then as linux ENV, then as JAVA_OPT (-Delastic.apm.ignore_exceptions=co.elastic.apm.agent.report.IntakeV2ReportingEventHandler) but nothing changed.
During startup my application I can see
INFO co.elastic.apm.agent.configuration.StartupInfo - ignore_exceptions: 'co.elastic.apm.agent.report.IntakeV2ReportingEventHandler' (source:
but still I can se ERROR logs from IntakeV2ReportingEventHandler and still some functions in my app does not work because of that.
I use elastic-apm-java agent in 1.35 version
Can someone explain how this setting should work?

Related

After deploy Google App Engine returns HTTP response code 403

I've done another deploys and all was fine, but after finishing the app, I'm getting this error. And the page request keeps loading.
Do I need to configure something in "IAM"?
Java 11
Standard Environment
h2 DB
Spring boot
The stack trace from Google Cloud:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://clouddebugger.googleapis.com/v2/controller/debuggees/register at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1919) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at com.google.devtools.cdbg.debuglets.java.GcpHubClient.registerDebuggee (Unknown Source)
I've got new data using Stackdriver debug.
"message": "Stackdriver Debugger API has not been used in project
929024293238 before or it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/clouddebugger.googleapis.com/overview?project=929024293238
then retry. If you enabled this API recently, wait a few minutes for
the action to propagate to our systems and retry.",
Just a note if somebody else stumbles across this error: I had the same message in Google's App Engine dashboard popping up up to 60 times in a couple of minutes.
Enabling the Stackdriver Debugging API, as linked above, solved it. No more error logs (of that kind) were being produced. The weired thing is that the Stackdriver Debugging API should have been turned on by default for my standard environment.
The log wasn't very informative.
I change the application port from 8000 to 8080.
application.properties:
server.port=${PORT:8080}
Now the app is running fine.

How to fix error "Enrollment failed" when trying WSO2 IoT 3.3.0 android enrollment?

I set up a new server windows 2012R2 in a VM machine. The latest active-mq and Apache-ant are downloaded. Oracle JDK 8u192 is used. After installation and setup of the WSO2 IoT application, I cannot enroll an android device.
I installed the WSO2IOT by following the instructions in the online documentations for version 3.3.0.
CARBON_HOME and JAVA_HOME are set up in environment variables already.
The process of steps that I go are as follows:
I start the activemq by running it inside a cmd window as instructed by the documentation.
I start broker.bat in another cmd window
I start iot-server.bat in another cmd window
I start analytics.bat in another cmd window
Everything starts and I end up with the text "Carbon .. has stared in ** milliseconds.
I open my management console on https port 9443\devicemgt and login as admin no problem.
After that I connect my galaxy A5 2016 model with android 7.0 to the same network via WiFi. Download the app via QR code from my PC screen to the mobile phone. Install the app and try to register. The IP which I enter is static in the VM and of course I add 8280 port in the wso2 management app on the phone. Then I login with admin/admin credentials which logins successfully but right after that it posts an message on the phone that it is receiving policy and here it stops. The little circle spins (loading) and nothing happens for few minutes after which I get an error "Enrollment failed. Contact the administrator. Enrollment failed." no other message or anything.
On the analytics cmd window I got the following error inside my VM:
[2018-12-20 11:00:00,037] [IoT-Analytics] ERROR {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in executing task: nulljava.lang.NullPointerException
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:256)
at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:206)
at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:60)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
And that is basically it I cant move on forward from here. I tried with different java jdk versions as old as 8u144 and I tried with openJDK 8 LTSu191 to no avail.
Other things like creating policies in the devicemgt portal, creating users and browsing the portal works fine.
Did anyone encounter this issue and manage to solve it?
Thanks.
I used WM windows10. The situation fully corresponded to that described by you, with only one difference - I also had an error on the page with the preservation of policies for Android. I also saw that the default policies for windows are reflected in the platform configuration section, and the android policies are missing and cannot be saved due to the error you described. However, in the wso2 home directory I found the Android configuration file in which the default policy !!! was done.
Next, I used ADB to watch the log from the Android device, at the time of binding (enrollment). He only confirmed the previously received information. I saw a 404 http error at the time the device attempted to get a policy agreement. Point.
Unexpected outcome. I downloaded version 3.1.0 instead of the current version 3.3.0. I started the batch file. I went to the platform configuration section, to the Android configuration - the default agreement policy was displayed correctly and you can change it and save the changes. The binding procedure (enrollment) passed without problems - the policy was obtained by the device and then everything went according to official instructions. I fully realize that this is not the answer to the question you raised, but at the same time it can help with a quick start. I hope.
p.s. I tried versions 3.3.1 and 3.3.0 - in both there is a similar problem with the enrollment. Version 3.1.0 does not have this problem. Versions tested on Ubuntu and WM Windows10.
Of course it would be great if the developer representatives tell you where to dig, to use a more current version. Have a nice day.
additionaly - also good work with 3.2.0

How can I monitor/log Tomcat's thread pool?

I have a Tomcat installation where I suspect the thread pool may be decreasing over time due to threads not being properly released. I get an error in catalina.out when maxthreads is reached, but I would like to log the number of threads in use to a file every five minutes so I can verify this hypothesis. Would anyone please be able to advise how this can be be done?
Also in this installation there is no Tomcat manager, it appears whoever did the original installation deleted the manager webapp for some reason. I'm not sure if manager would be able to do the above or if I can reinstall it without damaging the existing installation? All I really want to do is keep track of the thread pool.
Also, I noticed that maxthreads for Tomcat is 200, but the max number of concurrent connections for Apache is lower (Apache is using mod_proxy and mod_proxy_ajp (AJP 1.3) to feed Tomcat). That seems wrong too, what is the correct relationship between these numbers?
Any help much appreciated :D
Update: Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Just a quick update to say the direct JMX access worked. However I also had to set Dcom.sun.management.jmxremote.host. I set it to localhost and it worked, however without it no dice. If anyone else has a similar problem trying to enable JMX I recommend you set this value also, even if you are connecting from the local machine. Seems it is required with some versions of Tomcat.
Direct JMX access
Try adding this to catalina.sh/bat:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=5005
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
UPDATE: Alex P suggest that the following settings might also be required in some situations:
-Dcom.sun.management.jmxremote.host=localhost
This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin.
What you are looking for is Catalina -> ThreadPool -> http-bio-8080 -> various interesting metrics.
JMX proxy servlet
Easier method might be to use Tomcat's JMX proxy servlet under: http://localhost:8080/manager/jmxproxy. For instance try this query:
$ curl --user tomcat:tomcat http://localhost:8080/manager/jmxproxy?qry=Catalina:name=%22http-bio-8080%22,type=ThreadPool
A little bit of grepping and scripting and you can easily and remotely monitor your application. Note that tomcat:tomcat is the username/password of user having manager-jmx role in conf/tomcat-users.xml.
You can deploy jolokia.war and then retrieve mbeans values in JSON (without the manager):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool?ignoreErrors=true
If you want only some values (currentThreadsBusy, maxThreads, currentThreadCount, connectionCount):
http://localhost:8080/jolokia/read/Catalina:name=*,type=ThreadPool/currentThreadsBusy,maxThreads,currentThreadCount,connectionCount?ignoreErrors=true
{
request: {
mbean: "Catalina:name="http-nio-8080",type=ThreadPool",
attribute: [
"currentThreadsBusy",
"maxThreads",
"currentThreadCount",
"connectionCount"
],
type: "read"
},
value: {
currentThreadsBusy: 1,
connectionCount: 4,
currentThreadCount: 10,
maxThreads: 200
},
timestamp: 1490396960,
status: 200
}
Note: This example works on Tomcat7 +.
For a more enterprise solution. I have been using New Relic in our production environment.
This provides a graph of the changes to the threadpool over time.
There are cheaper tools out meanwhile: I am using this jar here: https://docs.cyclopsgroup.org/jmxterm
You can automate it via shell/batch scripts. I regexed the output and let prometheus poll it for displaying it in grafana.

How can I make "jconsole" work with Websphere 6.1?

I've deployed some Managed Beans on WebSphere 6.1 and I've managed to invoke them through a standalone client, but when I try to use the application "jconsole" distributed with the standard JDK can can't make it works.
Has anyone achieved to connect the jconsole with WAS 6.1?
IBM WebSphere 6.1 it's supossed to support JSR 160 JavaTM Management Extensions (JMX) Remote API. Furthermore, it uses the MX4J implementation (http://mx4j.sourceforge.net). But I can't make it works with neither "jconsole" nor "MC4J".
I have the Classpath and the JAVA_HOME correctly setted, so the issue it's not there.
WebSphere's support for JMX is crap. Particularly, if you need to connect to any secured JMX beans. Here's an interesting tidbit, their own implementation of jConsole will not connect to their own JVM. I have had a PMR open with IBM for over a year to fix this issue, and have gotten nothing but the runaround. They clearly don't want to fix this issue.
The only way I have been able to invoke remote secured JMX beans hosted on WebSphere has been to implement a client using the "WebSphere application client". This is basically a stripped down app server used for stuff like this.
Open a PMR with IBM. Perhaps if more people report this issue, they will actually fix it.
Update: You can run your application as a WebSphere Application Client in RAD. Open the run menu, then choose "Run...". In the dialog that opens, towards the bottom on the left hand side, you will see "WebSphere v6.1 Application Client". I'm not sure how to start and Application Client outside of RAD.
IT WORKS !
http://issues.apache.org/jira/browse/GERONIMO-4534;jsessionid=FB20DD5973F01DD2D470FB9A1B45D209?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
1) Change the config.xml and start the server.
-see here how to change config.xml: http://publib.boulder.ibm.com/wasce/V2.1.0/en/working-with-jconsole.html
2) start the jconsole with : jconsole -J-Djavax.net.ssl.keyStore=%GERONIMO_HOME%\var\security\keystores\geronimo-default -J-Djavax.net.ssl.keyStorePassword=secret -J-Djavax.net.ssl.trustStore=%GERONIMO_HOME%\var\security\keystores\geronimo-default -J-Djavax.net.ssl.trustStorePassword=secret -J-Djava.class.path=%JAVA_HOME%\lib\jconsole.jar;%JAVA_HOME%\lib\tools.jar;%GERONIMO_HOME%\repository\org\apache\geronimo\framework\geronimo-kernel\2.1.4\geronimo-kernel-2.1.4.jar
[or your version of geronimo-kernel jar]
3) in the jconsole interface->advanced, input:
JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/JMXSecureConnector
user name: system
password: manager
4) click the connect button.
If you want the WebSphere MBeans this one works for me:
The key is to configure the classpath and the security properly.
in one line:
jconsole -J-Dwas.install.root=C:/was61 -J-Djava.ext.dirs=C:/was61/plugins;C:/was61/plugins/com.ibm.ws.security.crypto_6.1.0;C:/was61/lib;C:/was61/java/jre/lib/ext -J-Dcom.ibm.SSL.ConfigURL="file:../../properties/ssl.client.props" -J-Dcom.ibm.CORBA.ConfigURL="file:../../properties/sas.client.props" service:jmx:iiop://host:port/jndi/JMXConnector
where port = bootstrap port ex: (2809)
Be careful when setting the sas and the ssl props.
Robert
I have successfully connected to ActiveMQ and ServiceMix using the JConsole. Does WAS 6.1 use Java Management Extension (JMX) technology? JMX is required for JConsole.
If your path is set correctly it should work fine. On windows you go to System Properties -> Advanced Tab -> Environment Variables. Have your JAVA_HOME System variable set to the path of your JDK or JRE and your Path variable with %JAVA_HOME%/bin added somewhere in there. Then all you need to do is go to Start->Run->JConsole. Select the correct Process Name and your done.
Where are you having problems at? I hope this helps.
Edit:
Here is the Java Doc's on JConsole.
Hmm... I know that WebSphere is kind of hard to configure. Thats part of the reason we used ServiceMix for our ESB. Maybe its not enabled by default in WebSphere and you would have to turn it on in the config somewhere.
Websphere 6.1 does not support the JConsole for some reason even though it fully implements the JMS specs. Seems to be a week area at the moment. Your best bet is to look at the Admin client to implement you own console.
You all seem to be incorrect. I am running Websphere 6.1.041 , using JDK 1.5 , and I just started up Jconsole and used the "simple connect" tab to connect to localhost with port=0 and without a username and password and it works fine.

NPE in JBossWS on JBoss 4.2.2 with jmxremote enabled

I am trying to set up JBoss 4.2.2 and JConsole for remote monitoring. As per many of the how-to's I have found on the web to do this you need to enable jmxremote by setting the following options in run.conf. (I realize the other two opts disable authentication)
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=11099"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=false"
Which results in the following exception:
13:06:56,418 INFO [TomcatDeployer] performDeployInternal :: deploy, ctxPath=/services, warUrl=.../tmp/deploy/tmp34585xxxxxxxxx.ear-contents/mDate-Services-exp.war/
13:06:57,706 WARN [AbstractServerConfig] getWebServicePort :: Unable to calculate 'WebServicePort', using default '8080'
13:06:57,711 WARN [AbstractServerConfig] getWebServicePort :: Unable to calculate 'WebServicePort', using default '8080'
13:06:58,070 WARN [AbstractServerConfig] getWebServicePort :: Unable to calculate 'WebServicePort', using default '8080'
13:06:58,071 WARN [AbstractServerConfig] getWebServicePort :: Unable to calculate 'WebServicePort', using default '8080'
13:06:58,138 ERROR [MainDeployer] start :: Could not start deployment: file:/opt/jboss-4.2.2.GA/server/default/tmp/deploy/tmp34585xxxxxxxxx.ear-contents/xxxxx-Services.war
java.lang.NullPointerException
at org.jboss.wsf.stack.jbws.WSDLFilePublisher.getPublishLocation(WSDLFilePublisher.java:303)
at org.jboss.wsf.stack.jbws.WSDLFilePublisher.publishWsdlFiles(WSDLFilePublisher.java:103)
at org.jboss.wsf.stack.jbws.PublishContractDeploymentAspect.create(PublishContractDeploymentAspect.java:52)
at org.jboss.wsf.framework.deployment.DeploymentAspectManagerImpl.deploy(DeploymentAspectManagerImpl.java:115)
at org.jboss.wsf.container.jboss42.ArchiveDeployerHook.deploy(ArchiveDeployerHook.java:97)
...
My application uses JWS which according to this bug:
https://jira.jboss.org/jira/browse/JBWS-1943
Suggests this workaround:
JAVA_OPTS="$JAVA_OPTS -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl"
JAVA_OPTS="$JAVA_OPTS -Djboss.platform.mbeanserver"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"
(https://developer.jboss.org/wiki/JBossWS-FAQ#jive_content_id_How_to_use_JDK_JMX_JConsole_with_JBossWS)
I've tried that however that then throws the following exception while trying to deploy a sar file in my ear which only contains on class which implements Schedulable for a couple of scheduled jobs my application requires:
Caused by: java.lang.NullPointerException
at EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap.hash(ConcurrentReaderHashMap.java:298)
at EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap.get(ConcurrentReaderHashMap.java:410)
at org.jboss.mx.server.registry.BasicMBeanRegistry.getMBeanMap(BasicMBeanRegistry.java:959)
at org.jboss.mx.server.registry.BasicMBeanRegistry.contains(BasicMBeanRegistry.java:577)
Any suggestions on where to go from here?
EDIT:
I have also tried the following variation:
JAVA_OPTS="$JAVA_OPTS -DmbipropertyFile=../server/default/conf/mbi.properties -DpropertyFile=../server/default/conf/mdate.properties -Dwicket.configuration=DEVELOPMENT"
JAVA_OPTS="$JAVA_OPTS -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl"
JAVA_OPTS="$JAVA_OPTS -Djboss.platform.mbeanserver"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"
I'm using JDK 1.6.0_01-b06
I have honestly never tried this remoting approach. But, if both your client machine and the server happen to both be linux boxes or similar *nixes with SSH, then you can ssh -XCA to the server and start JConsole on the server and have the GUI display on your client machine with X port forwarding. A JConsole running locally to the server JVM you want to monitor should not have any trouble connecting.
I personally think that's a nifty trick but I realize that it dosn't really solve the problem of getting JConsole to connect remotely through JWS.
First thing I would do is to delete both /tmp and /work directories under JBoss /default and redeploy the WAR. If that doesn't, I would upgrade the JDK to use a more recent version of 1.6. 1.6.0_01 is pretty old.
I'm not sure if there's a specific reason you're trying to use WS to access the mbean server, but with JConsole you can directly access a remote JVM. To do this use "service:jmx:rmi:///jndi/rmi://<remote-machine>:<port>/jmxrmi" (where <remote-machine> is whatever machine your trying to connect to and <port> is 11099) as the remote process.
I have used this to connect to any 1.6 JVM that exposes an mbean server (JBoss, ActiveMQ, etc).
I don't know if this is related, but JBoss has a tendency to redirect to itself. If you connect to a host, say jboss.localdomain:3873, wanting to connect to a ejb, JBoss might lookup its own hostname and redirect to the address it gets from there. If you have a public hostname, it might find that instead (say jboss.publicdomain.com), and tell the client to reconnect to jboss.publicdomain.com:1099. Depending on your DNS, this might or might not be a reachable address from your client.
There are various varations of this problem, and as a bonus, sometimes the initial "connection check" works, so the client app deploys, but fails later on connect.
Had a similar issue, but with JBoss Seam: take a look at JBSEAM-4029. As one of the workarounds it suggests to override the class running into the NPE - in Seam's case the JBossClusterMonitor.
I bet the JWS code is running into exact same issue, i.e. ending up calling MBeanServerFactory.findMBeanServer(null) at some point in time. The stack trace should reveal which exact class does this.

Categories

Resources