We have a Cloud Foundry (Java) application running on IBM Bluemix and we are looking for a way of health check for it. We mainly would like to monitor memory usage (both CF instance memory and JVM heap). We know that Auto-Scaling can do a similar thing but we think it keeps memory usages for recent 2 hours. (Please correct us if we are misunderstanding.) We prefer to monitor memory usage at least recent 24 hours. Any suggestions or comments must be appreciated. Thank you.
From a platform standpoint, you don't have a lot of options:
You can configure an HTTP-based health check for your app. Instead of just monitoring the port your application is listening to, this will actually send an HTTP request and check that it gets a valid response. If it does not, then your application will get automatically restarted by the platform. This does not keep track of any of the metrics that you listed. It's just purely a check to determine if your application is still alive.
Ex: cf set-health-check my-super-cool-app http --endpoint /health
https://docs.cloudfoundry.org/devguide/deploy-apps/healthchecks.html
You can connect to the firehose and pull metrics. This will include the container metrics of CPU, RAM & Disk usage. The firehose is just a method to obtain this information though, the whole problem of storage and pretty graphs is one that you'd still need to solve.
The firehose plugin is the best example of doing this: https://github.com/cloudfoundry-community/firehose-plugin
Beyond the platform itself, you might want to look at an APM (application performance monitoring) tool. I'm not going to list examples here. There are many which you can find with a quick Internet search. Some even integrate nicely with the Java buildpack, those are listed here.
This is probably the solution you want as it will give you all sorts of metrics, including the ones you mentioned above.
Hope that helps!
Related
My application is just a bigger version of the default Jhipster app.. I even have no Cache.
I deployed it successfully on an Amazon free tier t1.micro instance.
I experienced some random 503 errors. I checked the health of the instance and it sometimes said "no data sent" some other times "93% of memory is in use". Now it's down (red).
I cloned the environment, then terminated the original one. I get those various errors.
I deployed the war with Dev spring profile but I believe it's not what is causing this much horror.
Do I need to configure the java memory usage ? Why could the app be this memory hungry?
I posted the question on StackOverflow as I am caring more about performance tuning of the deployed Jhipster war but if you think it's more a problem with Amazon please let me know why you think that.
Thanks
Deploy the application on a instance with much more memory ie an t2.large (8GB)
The size on an existing instance can be altered by using the console "stop", find the console "instance settings" "instance type" change and start again
Ensure that your application has a method for attaching jconsole to it available (apparently the development version does, with jmx). See http://docs.oracle.com/javase/8/docs/technotes/guides/management/jconsole.html for more information on jconsole
Run the application and monitor the nice graphs in jconsole
See what the peak is over a few days of normal use. Also log on to the server with ssh and use free -m to see the system memory use ( see http://www.linuxatemyram.com/ for a guide to interpreting the data )
Once you know the actual amount of RAM it uses choose an appropriate instance size, see http://www.ec2instances.info/
You might need to adjust the -Xmx setting, I don't know the specifics with jhipster but this is a common requirement for java applications
We have a java application hosted at m.large ec2 server which in future requires high number of computations to be done, To be precise it is expected to have 10 million computations per day and each unit computation requires around 100ms, computation is of type of multiple processing at java code and then dumping to db, not like Write Once, Read Many. Which is the best way to balance the load? Options we have been going across is to instantiate ec2 servers by script as the load increases, but before we implement it, we want to have a solid suggestions from experts. Kindly provide any suggestions.
This may not be the right answer for you but certainly worth a consideration. You should define the metric and threshold for your application.
instantiate ec2 servers by script as the load increases
In above statement, you need to define what do you mean by "load"? what is the threshold that load should cross in order to say the load has "increased"
Once you have this information handy, check whether you can monitor these metrics and thresholds using AWS cloudwatch. If yes, then you can create an autscalling group where the autscalling will spin-up a new instance as soon as it told by the cloudwatch alarm that "the load has increased".
If you find that cloudwatch does not support your metrics, then write your own custom metric and have cloudwatch monitor it. Click here for custom cloudwatch metrics documentations.
Once you have your custom metric, then again integrate autoscalling and cloudwatch alarms to manage creation of new EC2 instances when your load increases.
In short, explore AWS cloudwatch alarms and AWS autoscalling.
This whole process assumes that you do have a fully-automated way of creating an EC2 instance along with your software stack. You can create a pre-baked AMI with your application stack OR you can install the application stack on the fly using tools like Opscode Chef.
Use a Platform-as-a-Service (PaaS) instead of manually creating machines and scaling them. This is more flexible for long-term use, and requires less scripting. There are some PaaS recommendations here: Looking for PaaS providers recommendations
Disclaimer: I work for Gigaspaces, developer of the Cloudify open-source PaaS stack
We have distributed web application developed few years back on JDK 5.
How JMX will help this application?
1) Will it help me to monitor performance (Memory, CPU and Network & Disk IO)?
2) If so then application is deployed in multiple servers,how can I monitor in one single dashboard?
3) Do I have to make any new code changes to the existing application or can I monitor without code changes?
4) What else can we do apart from performance monitoring? Because name is Management Extensions (MX) what can we manage and how? It seems primary objective is not monitoring, it is management?
I couldn’t get much about JMX from the tutorial provided in Oracle website http://docs.oracle.com/javase/tutorial/jmx/overview/why.html. It seems some business representative prepared this tutorial not a techie.
Can anyone help me in understanding this JMX stuff, your efforts are sincerely appreciated.
Thanks,
Your Friend.
JMX is a standardized way of getting information out of a running system and to invoke operations on it. The JVM gives you a set of MBeans through which you can access runtime data like memory consumption, GC stats and some more data. You can also invoke a number of operations. Your app server will also give you a number of MBeans which you can use to control the server and installed applications.
Yes, it will give you some performance data. See http://docs.oracle.com/javase/1.5.0/docs/guide/management/overview.html for a list of resources.
This depends on your application server. If it support clustered environments, it will probably give you a dashboard.
If you are fine with the standard JVM and app server MBeans, you won't need do any changes. But you may want to write your own MBean to give specific application status and statistics or ways to control your application. See http://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
Well, you can do anything you like in a standardized way.
Take a look at jconsole (included in the JDK) to see what JMX offers for you!
I want to find or develop an application that can run as a daemon, notify the administrator by email or sms when the Java applications running on a host get any exceptions or errors. I know JVMTI can achieve part of my goal, but it will impact performance of the monitored applications(I don't know how much will it be, it will be acceptable if it's slight), besides it seems to be a troublesom job to develop a JVMTI agent and I'm not sure what would happen if several applications running at the same time using the same agent. Is there any better solutions? Thanks in advance.
One way would be to use a logging system like log4j that publishes all errors occuring on system A to a logging server on system B from which you can monitor the errors occured. This isn't a completely generic solutation however, since only exceptions propagated to log4j (or any other logging system) would be handled - but it may be a good start.
The best solution is to have the Java application send its errors via email/sms. The problem is that programs will generate exceptions and handle correctly in normal operation. You only want particular exception.
Failing this you could write a log reader, which reads the logs of the application. This is tricky to get right, but it can be done.
An application can generate 1000+ exception per days and still be behaving normally because the application knows how to handle these exceptions. e.g. every time a socket connection is closed an exception can be thrown.
IMO, the best approach is to deploy an external monitoring system. This can:
monitor multiple applications
monitor infrastructure services
monitor network availability and machine accessibility,
monitor resources such as processor and file system usage.
Applications can be monitored in a variety of ways, including:
by processing log events,
by watching for application restarts,
by "pinging" the application's web apis to check service liveness, and
by using the application's JMX interfaces.
This information can be filtered and prioritized in an intelligent fashion, and critical events can be reported by whatever means is most appropriate.
You don't want individual applications sending emails, because they don't have sufficient information to do a decent job. Furthermore, putting the reporting logic into individual applications is likely to lead to inconsistent implementation, poor configurability, and so on.
There is a nearby alternative to JVMTI : JPDA. This infrastructure allows you to create a remote "debugger" (yes, that's what you're planning to do) using Java code, and connect it to the VM using either local or remote connection.
There will be, like for JVMTI, an overhead to program execution. However, as the Trace.java example shows, it's quite simple to both implement and connect to target VM.
Finally, notice if you want to instrument code run by application server (JBoss, Glassfish, Tomcat, you name it) there are various other means available.
I follow the pattern where every exception gets logged to a table.
Then an RSS feed selects from that table.
I subscribe to the RSS feed in MS Outlook at work and also on my Android phone with a program called NewsRob. NewsRob let me set my phone to alert me when there is something new.
I blog about how to do this HERE. It is in .net, but you get the idea.
As a related step I found a way to notify myself when something DIDN'T happen. That blog is HERE.
There are loads of applications out there that do what you are looking for in a way that does not impact performance. Have you had a look at Kibana/ElasticSearch, or Splunk or Logscape for enterprise solutions ( they both also have free versions).
I'm going to echo what has already been said and highlight what java already provides and what you can do with an external monitoring system. Java already provides:
log4j - log ERRORS, WARNINGS, FATAL and Exceptions to a file
JMX - Create custom application metrics and you also have access to java.lang/* which will give you heap memory usage , garbage collection, thread counters etc.
JVM gc logging - you can log all your garbage collection events to a file and watch for any long Full GC collections.
An external monitoring system will allow you to set alerts triggered off different operational scenarios. You will also get visualisation of your system performance through charts. I've used Logscape's java app in the past to monitor 30 java processes spread out over3 hosts.
A program I am working on takes forever to complete (~3days, everytime).
Is there some place on the internet where I can leave the code, some robot might run it for me and I can come back and collect the results? Some online judge that offers this capability?
[I am not talking abt optimisations here.]
You may need to go to something like this:
Amazon Elastic Compute Cloud (Amazon
EC2) is a web service that provides
resizable compute capacity in the
cloud. It is designed to make
web-scale computing easier for
developers.
http://aws.amazon.com/ec2/
If you really cannot run it on your own machines, you can run in on Amazon EC2 cloud. You would need to maintain a virtual machine, but Amazon provides some preconfigured settings.
The pricing starts with $0.085/hour (~$6 for three days). The actual price is determined by the duration of use and your CPU needs. Higher CPU capability is more expensive.