Develop Reference

How to get the CPU utilization for Java8 in AppEngine

I have a Google AppEngine application written in Java using JDK8.
I want to get the CPU utilization. How can I achieve that?
I have tried two approaches:
final OperatingSystemMXBean os = ManagementFactory.getOperatingSystemMXBean(); // this is the java.lang version
final double systemCpuLoadRatio = Math.max(0.0, os.getSystemLoadAverage() / os.getAvailableProcessors());
final long systemCpuLoad = (long) (systemCpuLoadRatio * 100);
OperatingSystemMXBean os = (OperatingSystemMXBean)ManagementFactory.getOperatingSystemMXBean() // This is the sun.management version.
final long systemCpuLoad = os.getAvailableProcessors();
Both approaches always yield zero as the result. The CPU utilization cannot be 0 as there are real requests ongoing.

Try multiplying by 100.0 then let the cast remove the fractional part...
You may also try the oshi framework: https://github.com/oshi

How can I get the CPU usage of a process with "tasklist" in Windows

I am writing a program in Java to periodically display the CPU and memory usage of a given process ID. My implementation invokes tasklist. It is pretty straightforward to get the memory usage by the following command:
tasklist /fi "memusage ge 0" /fi "pid eq 2076" /v
This will return the memory usage of process id 2076 and i can use this for my task. By invoking the following command, I can extract the CPU Time.
tasklist /fi "pid eq 2076" /fi "CPUTIME ge 00:00:00" /v
My question is, how would I go about getting the CPU usage of this process?
I found a post on StackOverflow for my question but the answer isn't clear and I don't understand what to type in the command to get what I need. The question was answered in 2008 and someone asked for clarification in 2013 but the person that answered the question hasn't replied.
Here is the post that I have found.

Memory is like a tea cup, it maybe full or empty, an instantaneous look at the cup allows you to see how full of tea it is (that is your "memusage" command).
CPU is like a ski lift. It moves at a reasonably constant rate irrespective of whether your are riding the lift or not. It is not possible to determine your usage in a single instantaneous observation - we need to know how long you were riding it for (that is your "cputime" command). You have to use the "cputime" command at least twice!
For example:
At 7:09 pm, you run the cputime command on your process, and it returns "28 minutes"
At 7:17 pm, you run the cputime command on your process again, and it returns "32 minutes"
From the first time you ran the cputime command to the second time, the usage has increased from 28 minutes to 32 minutes -- the process has used 4 minutes of CPU time.
From 7:09pm to 7:17pm is a duration of 8 minutes -- A total of 8 minutes of time were available, but your process just used 4 minutes: 4 / 8 = 50% average system usage.
If your system has multiple processors, then you can divide by the total number of CPUs to get an average per CPU - e.g. 50% / 2 = 25% average in a dual cpu system.
I used minutes above for ease of writing - in reality you may be looking at how many nanoseconds of CPU time the process used during a time window that is just milliseconds long.

tasklist does not provide the information you are looking for. I would suggest using Get-Counter. A comment on an answer from the SuperUser site looks to be on track for what you're after.
Get-Counter '\Process(*)\% Processor Time' | Select-Object -ExpandProperty countersamples| Select-Object -Property instancename, cookedvalue| ? {$_.instanceName -notmatch "^(idle|_total|system)$"} | Sort-Object -Property cookedvalue -Descending| Select-Object -First 25| ft InstanceName,#{L='CPU';E={($_.Cookedvalue/100/$env:NUMBER_OF_PROCESSORS).toString('P')}} -AutoSize

I once wrote a class:
private static class PerformanceMonitor {
private int availableProcessors = ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();
private long lastSystemTime = 0;
private long lastProcessCpuTime = 0;
/**
* Get's the cpu usage of the jvm
*
* #return the cpu usage a double of percentage
*/
private synchronized double getCpuUsage() {
if (lastSystemTime == 0) {
baselineCounters();
return 0d;
}
long systemTime = System.nanoTime();
long processCpuTime = 0;
if (getOperatingSystemMXBean() instanceof com.sun.management.OperatingSystemMXBean) {
processCpuTime = ((com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean()).getProcessCpuTime();
}
double cpuUsage = ((double) (processCpuTime - lastProcessCpuTime)) / ((double) (systemTime - lastSystemTime));
lastSystemTime = systemTime;
lastProcessCpuTime = processCpuTime;
return cpuUsage / availableProcessors;
}
private void baselineCounters() {
lastSystemTime = System.nanoTime();
if (getOperatingSystemMXBean() instanceof com.sun.management.OperatingSystemMXBean) {
lastProcessCpuTime = ((com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean()).getProcessCpuTime();
}
}
}
Which is used like:
private static final PerformanceMonitor _MONITOR = new PerformanceMonitor();
_MONITOR.getCpuUsage();
This prints out the usage of the cpu consumed by the process of this JVM.

Track native memory usage from Java?

Is there any way I can log native memory usage from Java, i.e., either native memory directly or the total memory the process is using (e.g., ask the OS)?
I'd like to run this on user's machines behind the scenes, so the NativeMemoryTracking command line tool isn't really the most appealing option. I already log free/max/total heap sizes.
Background: A user of my software reported an exception (below) and I have no idea why. My program does use SWIG'd native code, but it's a simple API, I don't think it has a memory leak, and wasn't on the stacktrace (or run immediately before the error). My log indicated there was plenty of heap space available when the error occurred. So I'm really at a loss for how to track this down.
java.lang.OutOfMemoryError: null
at java.io.RandomAccessFile.writeBytes0(Native Method) ~[na:1.7.0_45]
at java.io.RandomAccessFile.writeBytes(Unknown Source) ~[na:1.7.0_45]
at java.io.RandomAccessFile.write(Unknown Source) ~[na:1.7.0_45]
The error occurred on Windows (7 or 10)(?) from within webstart, configured with these parameters:
<java href="http://java.sun.com/products/autodl/j2se" initial-heap-size="768m" java-vm-args="" max-heap-size="900m" version="1.7+"/>

If you want tp track down the JVM memory on your certain method or lines of code you can use the Runtime API.
Runtime runtime = Runtime.getRuntime();
NumberFormat format = NumberFormat.getInstance();
long maxMemory = runtime.maxMemory();
long allocatedMemory = runtime.totalMemory();
long freeMemory = runtime.freeMemory();
System.out.println("free memory: " + format.format(freeMemory / 1024));
System.out.println("allocated memory: " + format.format(allocatedMemory / 1024));
System.out.println("max memory: " + format.format(maxMemory / 1024));
System.out.println("total free memory: " + format.format((freeMemory + (maxMemory - allocatedMemory)) / 1024));

I ended up using this code which asks the OS for RSS and Peak memory usage. It was straightforward for me to add since I already have a SWIG module set up. The code might not be threadsafe since I hit a random malloc exception when I was testing, meaning I'm not sure I want to keep it in there.
I'm really surprised the JVM doesn't provide a way to do this. Please someone let me know if there's a way.

Here is a snippet of code that sets the string equal to the amount of memory used(mb)/total memory(mb) You can then use this to log however you want!
Runtime instance = Runtime.getRuntime();
String mem = "Memory Used: "+ (instance.totalMemory() - instance.freeMemory()) / mb +"MB ("+
(int)((instance.totalMemory() - instance.freeMemory())*1.0/instance.totalMemory()*100.0)+"%)"

public final void writeBytes(String s) throws IOException {
int len = s.length();
byte[] b = new byte[len];
s.getBytes(0, len, b, 0);
writeBytes(b, 0, len);
}
Looking at the source, it is possible that a sufficiently large String would have caused out of memory error. I suspect that your heap log was done before this happened which explains the free heap space you saw. I suggest you verify if this is the case and if yes, limit the String size and/or increase the heap size.

Java Out of Memory exception in Ubuntu when using Flume/Hadoop

I'm getting an out of memory exception due to lack of Java heap space when I try and download tweets using Flume and pipe them into Hadoop.
I have set the heap space currently to 4GB in the mapred-site.xml of Hadoop, like so:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx4096m</value>
</property>
I am hoping to download tweets continually for two days but can't get past 45 minutes without errors.
Since I do have the disk space to hold all of this, I am assuming the error is coming from Java having to handle so many things at once. Is there a way for me to slow down the speed at which these tweets are downloaded, or do something else to solve this problem?
Edit: flume.conf included
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = <required>
TwitterAgent.sources.Twitter.consumerSecret = <required>
TwitterAgent.sources.Twitter.accessToken = <required>
TwitterAgent.sources.Twitter.accessTokenSecret = <required>
TwitterAgent.sources.Twitter.keywords = manchester united, man united, man utd, man u
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:50070/user/flume/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100
Edit 2
I've tried increasing the memory to 8GB which still doesn't help. I am assuming I am placing too many tweets in Hadoop at once and need to write them to disk and release the space again (or something to that effect). Is there a guide anywhere on how to do this?

Set JAVA_OPTS value at flume-env.sh and start flume agent.

It appears the problem had to do with the batch size and transactionCapacity. I changed them to the following:
TwitterAgent.sinks.HDFS.hdfs.batchSize = 100
TwitterAgent.channels.MemChannel.transactionCapacity = 1000
This works without me even needing to change the JAVA_OPTS value.

Resolving java.lang.OutOfMemoryError: Java heap space

As the title suggests I'm getting this error inside a thread.
The offending LOCs looks like this:
for (int i = 0; i < objectListSize; i++) {
logger.INFO("Loop repeat: "+i+" ...", true);
final Double discreteScore = sp.getDouble(superPeerSocket);
final int expectedObjectIDs = sp.getInteger(superPeerSocket);
final String discreteObjects[] = new String[expectedObjectIDs];
for ( int j = 0; j < expectedObjectIDs; j++)
discreteObjects[j] = sp.getString(superPeerSocket);
htPlus.attachInitialDiscreteList2L1(discreteScore, discreteObjects);
}
The final String discreteObjects[] declaration is where I get the error. I am running this code inside a thread. I have two threads currently active when I get this. I also tried using the MAT tool from eclipse. here is a link with some chart files inside:
PLC chart files (dropbox URL)
If anyone has any idea for this problem I would be grateful.
P.S.: I am thinking to remove the loop although it just fails in the first loop pass.
(I get this in the output when the program fails)
Expected data size: 10
Repeat: 0 ...
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid3793.hprof ...
Heap dump file created [1404020 bytes in 0.015 secs]
Exception in thread "1" java.lang.OutOfMemoryError: Java heap space
at planetlab.app.factory.Worker$15.run(Worker.java:796)
at java.lang.Thread.run(Thread.java:662)
Something irrelevant:
What's with the code not properly formatted/intended error when making posts in stack overflow? It took me 15 minutes to figure out what to do :# :S :#

Every Java program runs in a sandbox. While your OS might have 10 GB of RAM available to it your app might only have 128 MB.
You need to make sure you app has enough ram allocated to the JVM by using the -Xms -Xmx arguments. -Xms indicates the minimum and -Xmx the maximum
It's been suggested in the comments that your expectedObjectIDs seem kinda high. I would certainly check that out. However, you can use the following code to get an idea as you to memory usage and available memory. Using that info you can adjust your -Xms -Xmx accordingly.
Good luck!
Runtime runtime = Runtime.getRuntime();
long maxMemory = runtime.maxMemory();
long allocatedMemory = runtime.totalMemory();
long freeMemory = runtime.freeMemory();
System.out.println("free memory: " + freeMemory / 1024);
System.out.println("allocated memory: " + allocatedMemory / 1024);
System.out.println("max memory: " + maxMemory /1024);
System.out.println("total free memory: " +
(freeMemory + (maxMemory - allocatedMemory)) / 1024);