I'm investigating some unit test failures. The tests pass on an old build server that's been hand-configured (and not documented). I'm trying to run them in a clean virtual machine.
My latest problem is a unit test that creates 10K threads.
for (int i = 0; i < 10000; i++) {
final Thread thread = new Thread(new Runnable() { ... });
threads.add(thread);
thread.start();
}
Well, the max user processes in the clean environment is only 4K.
$ ulimit -u
4096
I was wondering if there's some way for Java to get at that limit. The test really doesn't need 10K, it just needs some arbitrarily large number.
You could call ulimit directly:
Runtime.getRuntime().exec(command)
Related
I want to speed up the Jenkins job through parallelizing the test stages. The idea is to move each test stage to a separate node. What I've found is the parallel keyword. When I used this I ran into several problems:
The timing seems to be different. I had to adapt several tests.
Mocking seems to have issues in parallel mode.
For me it seems that the parallel stages are running on the same machine/node. Is there an opportunity to force the different stages to run on different nodes so that they don't influence each other?
One of the errors I get:
org.mockito.exceptions.misusing.UnfinishedMockingSessionException:
Unfinished mocking session detected.
Previous MockitoSession was not concluded with 'finishMocking()'.
For examples of correct usage see javadoc for MockitoSession class.
The Jenkins file:
pipeline {
options {
timeout(time: 120, unit: 'MINUTES')
buildDiscarder(logRotator(numToKeepStr: env.BRANCH_NAME == 'master' ? '30' : '5'))
skipDefaultCheckout()
}
agent {
label 'win10'
}
stages {
stage('cleanWs and checkoutCode') {
steps {
cleanWs()
script {
checkoutCode()
}
}
}
stage('build') {
steps {
bat 'gradlew showJavaVersion'
bat 'gradlew printVersion'
bat 'gradlew compileKotlin compileTestKotlin compileJava compileTestJava compileIntTestJava --parallel'
bat 'git status'
}
}
stage('tests') {
parallel {
stage("unit tests without ui") {
environment {
// give each build testing its own random port
BDP_HTTP_MOCK_PORT = 0
}
steps {
bat 'gradlew test -x :ui:test -x :bdp-ui:test -x :bdp-mock:test -P headless=true -P maxParallelIntegrationTests=3 -P os=win --parallel'
}
}
stage("mock tests") {
environment {
// give each build testing its own random port
BDP_HTTP_MOCK_PORT = 0
}
steps {
bat 'gradlew :bdp-mock:test -P headless=true -P maxParallelIntegrationTests=3 -P os=win --parallel'
}
}
stage("unit tests with ui") {
environment {
// give each build testing its own random port
BDP_HTTP_MOCK_PORT = 0
}
steps {
bat 'gradlew :ui:test :bdp-ui:test -P headless=true -P maxParallelIntegrationTests=3 -P os=win --parallel'
}
}
} // parallel
} // stage('tests')
}
}
Several ways of fixing this:
Use only a single slot on the test machines:
Pros: Very simple, foolproof way of avoiding interference between tests
Cons: Using a single slot might be wasteful if the machine could actually handle more
use locks, make sure you have the NODE_NAME be part of the lock's name
Pros: Fine-grained control over which parts of the pipeline need locking
Cons: locks won't influence where the stage will be scheduled, just make processes wait who want the lock
Apply a different scheduling strategy, for example Least Load, or round robbin. This will distribute load better and make clashes less likely.
Pros: Better load balancing
Cons: Doesn't actually prevent clashes
In my opinion, I would say a combination of using locks, and changing the scheduler, is the right way to do this (unless someone comes up with a better way I don't know). Locks guarantee that no tests will interfere, and round-robbin scheduling / Least Load will make it less likely a process needs to wait for the lock to be released.
Currently my batch file will launch 20 jars with different arguments, after an hour, a taskkill command kills all java, the batch file will now launch 20 different jars.
My problem is, max cpu usage on startup, and potentially wasted cpu later. I could start the jar files at different times, but then they won't run for equal amounts of time. Takes up to 5 minutes for the cpu usage to half.
I need to launch a jar file, then kill it in an hour, without touching the other 19 jar files, and without knowing the PID.
I have been browsing the web and I see some stuff about, making it a background process, then getting the PID that way, can someone help me out with that?
This is what it looks like now
java -jar file.jar -a first
timeout 3
java -jar file.jar -a second
timeout 3
java -jar file.jar -a third
timeout 3
Use jps.exe utility (which is part of standard JDK) to learn the PID of the just started Java process. Then use taskkill /pid to kill this one process.
I'm just going to have some fun here :)
If you don't have ownership of the jars you might try something like:
public class KillTask extends TimerTask {
private Process process;
public KillTask(Process process) {
this.process = process;
}
#Override
public void run() {
// Prepare to die!
process.destroy();
}
}
public class KillDriver {
public KillDriver(String command) {
try {
Process process = Runtime.getRuntime().exec(command);
new Timer().schedule(new KillTask(process), 60 * 60 * 1000);
} catch (IOException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
for (String s : args)
new KillDriver(s);
}
}
This example assumes multiple jars will be started but you can alter that for a single or possibly pass in the delay as an argument to add some flexibility.
It appears you are giving the Jars their own process names ("first", "second", "third"), so I assume you could use a VBscript like this that does not require the PID to kill it.
ProcessName = "FIRST"
Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate,authenticationLevel=pktPrivacy,(Debug)}!\\.\root\cimv2")
Set colProcessList = objWMIService.ExecQuery("Select * From Win32_Process")
For Each objProcess in colProcessList
If InStr(UCase(objProcess.Name), ProcessName) Then
WScript.Echo "Killing " & objProcess.Name
objProcess.Terminate()
End If
Next
I have written a simple factorial program, with arbitrary precision:
public class Fac {
public static void main(String[] args) {
int stop = 100000;
long start = System.currentTimeMillis();
BigInteger integer = new BigInteger("1");
for(int i = 2; i <= stop; i++){
integer = integer.multiply(new BigInteger(i +""));
}
System.out.println("It took: " + (System.currentTimeMillis() - start) + "ms");
//System.out.println(integer);
}
}
When i run it in IntelliJ:
It took: 5392ms
When i run it in commandline:
It took: 17919ms
The commandline is run by:
javac Fac.java
java Fac
I know this is not the best way to measure time but the gap is soo huge that it does not matter.
Why is the performence that different?
Other people has noticed similar difference, however, as far as i can tell, their conclusions seem unrelated to my situation.
Why is my application running faster in IntelliJ compared to command-line?
http://grails.1312388.n4.nabble.com/Why-does-IntelliJ-IDEA-runs-faster-than-Windows-command-line-td3894823.html
It's because you are launching the jvm to run your program with different classpath, arguments, etc.
If you run the program in IntelliJ, you will see the first line of the Run window something like "C:\Program ..."
Click on it to expand it, and you will see all the arguments used when intellij runs your program (I am splitting an example over several lines here).
"C:\Program Files (x86)\Java\jdk1.8.0_40\bin\java"
-Didea.launcher.port=7532
"-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA 14.0.3\bin"
-Dfile.encoding=UTF-8
-classpath "C:\Program Files (x86)\Java\jdk1.8.0_40\jre\lib\charsets.jar;...etc..."
Fac
If you duplicate the exact same arguments (using the exact same jvm) then you will likely see similar performance when you run your application manually.
Your system settings for PATH, JAVA_HOME and CLASSPATH are used by default for launching your program if you don't specify them fully.
In Java, suppose if I start 1000 threads with a loop as below, is there any way I can monitor the number of threads actually running and the CPU resources that the threads consume with task manager?
for(int j=0; j<=1000; j++)
{
MyThread mt = new MyThread ();
mt.start ();
}
You can use VisualVM or JConsole or any other monitoring tool
If you mean the Windows task manager then yes, you can customize the columns shown in the process tab:
Menu View > Select Columns > Threads
EDIT
A quick test shows that creating an additional thread does increment that counter by one - and when that thread terminates, the counter decrements.
But it starts with more than one thread, because it probably includes the various JVM threads too (it starts with 19 threads). Note that jconsole shows 10 threads on a mono-thread program too.
If you use visual VM, you can see the split between daemon and non daemon threads (all JVM threads are daemon).
Test code:
public static void main(String[] args) throws InterruptedException {
Thread.sleep(3000);
Runnable r = new Runnable() {
#Override
public void run() {
try {
Thread.sleep(10000);
} catch (InterruptedException ex) {}
}
};
for (int i = 0; i < 5; i++) {
new Thread(r).start();
Thread.sleep(1000);
}
Thread.sleep(10000);
}
In code you can use Thread.activeCount() method
I think Visual VM is a better tool for this purpose. You'll get threads and a lot more information if you download and install all the plugins.
You can use managed bean for this perpers (MXBean). For example ThreadMXBean.
To get MXBean just call
ManagementFactory.getThreadMXBean()
The methods getThreadCount() and getCurrentThreadCpuTime() will help you.
If you just want to check the thread count in Windows 10
Task manager -> Details -> (right click) Select columns -> (check) Threads
I have a Java program that needs to call the same external executable 6 times. This executable produces an output file and once all 6 runs are complete I "merge" these files together. I did just have a for-loop where I ran the code, waited for the first run of the external executable to end then I called it again, etc.
I found this highly time consuming, averaging 52.4s for it to run 6 times... I figured it would be pretty easy to speed up by running the external executable 6 times all at once, especially since they aren't dependent on one another. I used ExecutorService and Runnable, etc. to achieve this.
With my current implementation, I shave about ~5s off my time, making it only ~11% faster.
Here is some (simplified) code that explains what I'm doing:
private final List<Callable<Object>> tasks = new ArrayList<Callable<Object>>();
....
private void setUpThreadsAndRun() {
ExecutorService executor = Executors.newFixedThreadPool(6);
for (int i = 0; i < 6; i++) {
//create the params object
tasks.add(Executors.callable(new RunThread(params)));
}
try {
executor.invokeAll(tasks);
} catch (InterruptedException ex) {
//uh-oh
}
executor.shutdown();
System.out.println("Finished all threads!");
}
private class RunThread implements Runnable {
public RunThread(ModelParams params) {
this.params = params;
}
#Override
public void run()
{
//NOTE: cmdarray is constructed from the params object
ProcessBuilder pb = new ProcessBuilder(cmdarray);
pb.directory(new File(location));
p = pb.start();
}
}
I'm hoping there is a more efficient way to do this...or maybe I'm "clogging" my computer's resources by trying to run this process 6 times at once. This process does involve file I/O and writes files that are about 30mb in size.
The only time that forking the executable 6 times will earn a performance boost is if you have at least 6 CPU cores and your application is CPU bound -- i.e. mostly doing processor operations. Since each application writes a 30mb file, it sounds like it is doing a large amount of IO and the applications are IO bound instead -- limited by your hardware's ability to service the IO requests.
To speed up your program, you might try 2 concurrent processes to see if you get an improvement. However, if you program is IO bound, then you will never get much of a speed improvement by forking multiple copies.