In my scenario, the methods in benchmark should run sequentially in one thread and modify the state in order.
For example, there is a List<Integer> called num in the benchmark class. What I want is: first, run add() to append a number into the list. Then, run remove() to remove the number from the list.
The calling sequence must be add() --> remove(). If remove() runs before add() or they run concurrently, they would raise exceptions because there's no element in the list.
That is, add() and remove() must be called sequentially and in one thread.
In Control the order of methods using JMH, I learned that the methods run in the lexicographical order. I tried the code below:
#State(Scope.Group)
#OutputTimeUnit(TimeUnit.MILLISECONDS)
#BenchmarkMode(Mode.AverageTime)
#Fork(value = 10)
public class ListBenchmark {
private List<Integer> num;
public static void main(String[] args) throws RunnerException {
Options options = new OptionsBuilder()
.verbosity(VerboseMode.NORMAL)
.syncIterations(true)
.threads(1)
.include(".*" + ListBenchmark.class.getCanonicalName() + ".*")
.build();
new Runner(options).run();
}
#Setup(Level.Invocation)
public void setup() throws Exception {
num = new ArrayList<>();
}
#Benchmark
#BenchmarkMode(Mode.SingleShotTime)
#Group("num")
public void add() throws Exception {
num.add(1);
}
#Benchmark
#BenchmarkMode(Mode.SingleShotTime)
#Group("num")
public void remove() throws Exception {
num.remove(0);
}
}
But it doesn't work, because the add method and the remove method run concurrently. In some cases, the remove runs before add and raises IndexOutOfBoundsException.
How to run the methods in benchmarks sequentially with JMH?
You start with a wrong precondition and everything fails short because of that. You can see a broader explanation from the authors here. You want symmetry where asymmetry is implied.
If you want to see how much it takes add -> remove place them both in the same #Benchmark, and same for individual add or remove via different State. For example:
#State(Scope.Thread)
public static class BothAddAndRemove {
List<Integer> num;
#Setup(Level.Invocation)
public void setup() throws Exception {
num = new ArrayList<>();
}
}
#State(Scope.Thread)
public static class RemoveOnly {
List<Integer> num;
#Setup(Level.Invocation)
public void setup() throws Exception {
num = new ArrayList<>();
num.add(1);
}
}
#Fork(25)
#Benchmark
#BenchmarkMode(Mode.SingleShotTime)
public int add(BothAddAndRemove both) {
both.num.add(1);
return both.num.remove(0);
}
#Fork(25)
#Benchmark
#BenchmarkMode(Mode.SingleShotTime)
public int removeOnly(RemoveOnly removeOnly) {
return removeOnly.num.remove(0);
}
Related
A version of the static method append() below, is sometimes called simultaneously by various threads:
public class CMEDemo {
private static final Logger LOG = Logger.getLogger(CMEDemo.class.getName());
private static final String[] SOME_DEMO = {"albino", "mosquito", "libido"};
private static final Set<String> SET_SOURCE = new LinkedHashSet<>(Arrays.asList(SOME_DEMO));
public static void append() {
//ConcurrentModificationException is thrown in the constructor for modifiableCopy.
LinkedHashSet<String> modifiableCopy = new LinkedHashSet<>(getBuiltInFunctions());
//Exception is not thown here, or later.
Set<String> doomed = modifiableCopy.stream()
.filter(f -> f.contains("quit")).collect(Collectors.toSet());
for (String found : doomed) {
LOG.log(Level.INFO, found);
}
}
public static Set<String> getBuiltInFunctions() {
return Collections.unmodifiableSet(SET_SOURCE);
}
}
Normally, all works as expected, but sometimes the LinkedHashSet constructor throws a ConcurrentModificationException:
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042)
at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
at java.util.LinkedHashSet.<init>(LinkedHashSet.java:169)
My JVM is Java 1.8_212. If I setup a test case to spawn multiple threads and let it run for a while, append() eventually throws ConcurrentModificationException. Why is this exception thrown, and how do I safely get a LinkedHashSet?
Other code was modifying the Set SET_SOURCE which was wrapped, not copied, by Collections.unmodifiableSet(..), and that caused intermittent exceptions during the construction of the copy in LinkedHashSet.
So I replaced getBuiltInFunctions() with the equivalent of
public static Set<String> getBuiltInFunctions() {
synchronized (SET_SOURCE) {
return Collections.unmodifiableSet(new HashSet<>(SET_SOURCE));
}
}
private static Set<String> getFunctionsRW() {
synchronized (SET_SOURCE ) {
return SET_SOURCE;
}
}
I've got a class which runs my tests in parallel with parameters. But to check for concurrency problems I want to execute the tests in a random order.
Eg. test1 - test2 - test3 on the first thread and test2 - test1 - test3 on the second thread. (I think you got the point)
This is the code I'm currently working with and I already found some examples using BlockJUnit4ClassRunnerWithParameters but changing Parameterized to BlockJUnit4ClassRunnerWithParameters did obviously not work for me.
Thus my question is: how can I make each thread execute the test in a random order?
Hope you guys can give me some pointers...
public class Parallelized extends Parameterized
{
private static class ThreadPoolScheduler implements RunnerScheduler
{
private ExecutorService executor;
public ThreadPoolScheduler()
{
String threads = System.getProperty("junit.parallel.threads", "16");
int numThreads = Integer.parseInt(threads);
executor = Executors.newFixedThreadPool(numThreads);
}
#Override
public void finished()
{
executor.shutdown();
try
{
executor.awaitTermination(10, TimeUnit.MINUTES);
}
catch (InterruptedException exc)
{
throw new RuntimeException(exc);
}
}
#Override
public void schedule(Runnable childStatement)
{
executor.submit(childStatement);
}
}
public Parallelized(Class klass) throws Throwable
{
super(klass);
setScheduler(new ThreadPoolScheduler());
}
}
Okay, I found a solution which is fairly simple:
All you need is a ParametersRunnerFactory which creates Runners of type BlockJUnit4ClassRunnerWithParameters.
To randomize the test order for each runner you only have to override computeTestMethods() and shuffle the list of methods.
To use your created RunnerFactory, you have to add #Parameterized.UseParametersRunnerFactory(Parallelized.RunnerFactory.class) right below the #RunWith(Parallelized.class) in each class you would like to use the randomized execution order.
Note: If don't add this annotation junit will use the default runner and not your custom runner -> no change.
public static class RunnerFactory implements ParametersRunnerFactory {
#Override
public org.junit.runner.Runner createRunnerForTestWithParameters(TestWithParameters test) throws InitializationError {
return new CustomRunnerWithParameters(test);
}
}
public static class CustomRunnerWithParameters extends BlockJUnit4ClassRunnerWithParameters {
private final Object[] parameters;
#Override
protected List<FrameworkMethod> computeTestMethods() {
List<FrameworkMethod> tests = new ArrayList<FrameworkMethod>(super.computeTestMethods());
Collections.shuffle(tests);
return tests;
}
public CustomRunnerWithParameters(TestWithParameters test) throws InitializationError {
super(test);
parameters = test.getParameters().toArray(new Object[test.getParameters().size()]);
}
#Override
public Object createTest() throws Exception {
return getTestClass().getOnlyConstructor().newInstance(parameters);
}
}
Edit after Aroidans comment:
I've forgot to add that I'm using an overwritten getChildren method in my custom runner extending Parameterized to shuffle the execution order of the tests. But his answer is valid, too.
#Override
protected List<Runner> getChildren() {
ArrayList<Runner> runner = new ArrayList<>(super.getChildren());
if (NUM_THREADS > 1) {
Collections.shuffle(runner);
}
return runner;
}
The easiest way I found to randomize the order of the parameterized tests is to do a shuffle on the collection with the parameters.
Using the Fibonacci sample test as an example.
#Parameters(name= "{index}: fib[{0}]={1}")
public static Iterable<Object[]> data() {
Iterable<Object[]> tests = Arrays.asList(new Object[][] { { 0, 0 }, { 1, 1 }, { 2, 1 },{ 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 } });
Collections.shuffle((List<?>) tests);
return tests;
}
This randomizes the order in which the tests using the same method are run. Using the computeTestMethod() does not work as there is only one method executing all of the tests.
I have been staring at this code for quite some time now and I really can't figure out how and what to do for its JUnit testing.
static void printList(OrderedIntList list) {
System.out.print("[");
for (int i = 0; i < list.orderedIntegerList.length; i++) {
System.out.print(list.orderedIntegerList[i]);
if (i != list.orderedIntegerList.length - 1) {
System.out.print(", ");
}
}
System.out.println("]");
}
This code is from a utility class called orderedIntList, which is an array list.
Well... think about the contracts of this method: What does it promise? It promises to print a list of integers, separated by a comma, and inside of brackets. So, you could specify this in a unit test.
Edit2: Just in case you fancy unit testing whether the list is printed in order: I would not consider this a property of the printList method. This property should be tested in the methods that modify the OrderedIntList.
Write a fixture for some Lists with the string that you expect to be printed. Don't forget the corner case: An empty list. Then, maybe a list of size 1 and one of size 10.
To check what has been printed, you can create a new PrintStream, and set the value of System.out, by calling System.setOut. Do this at the beginning of each test, and don't forget to reset System.out (so don't forget to store its original value). The PrintStream you use need not be fancy: You simply must be able to compare the printed stream. You may want to consider using Mockito for that matter.
Edit:
For example, the following code tests whether the list containing the sole element 4, actually prints the string "[4]\n":
#Test
public void aListOfSizeOneShouldPrintTheElementWithinBrackets() {
/* Setup */
final StringBuffer printStream = new StringBuffer();
PrintStream mockedOut = mock(PrintStream.class);
Mockito.doAnswer(new Answer() {
#Override
public Object answer(InvocationOnMock invocation) throws Throwable {
printStream.append(invocation.getArguments()[0].toString());
return null;
}
}).when(mockedOut).print(any());
Mockito.doAnswer(new Answer() {
#Override
public Object answer(InvocationOnMock invocation) throws Throwable {
printStream.append(invocation.getArguments()[0].toString());
return null;
}
}).when(mockedOut).print(anyString());
Mockito.doAnswer(new Answer() {
#Override
public Object answer(InvocationOnMock invocation) throws Throwable {
printStream.append(invocation.getArguments()[0].toString()).append("\n");
return null;
}
}).when(mockedOut).println(any());
Mockito.doAnswer(new Answer() {
#Override
public Object answer(InvocationOnMock invocation) throws Throwable {
printStream.append(invocation.getArguments()[0].toString()).append("\n");
return null;
}
}).when(mockedOut).println(anyString());
PrintStream originalOut = System.out;
System.setOut(mockedOut);
/* the actual test */
ArrayList<Integer> listWithOneElement = new ArrayList<Integer>();
listWithOneElement.add(4);
OrderedIntList list = new OrderedIntList(listWithOneElement);
OrderedIntList.printList(list);
MatcherAssert.assertThat(printStream.toString(), is("[4]\n"));
/* Teardown - reset System.out */
System.setOut(originalOut);
}
Note that you probably want to extract the setup- and the teardown part to use it in other tests and to make your test readable. If I am not mistaken, JUnit provides functionality to add code that gets invoked before and after each test execution if you specify it with an #Before and #After annotation.
Obviously, you do not necessarily need Mockito to achieve this. You can also write a class that implements PrintStream and stores the printed strings. In this case, Mockito simply allows you to ignore all the methods you do not care about (i.e., you do not need to implement the whole PrintStream interface)
A small sidenote on the duplication of the stubing: We need to stub both methods, print and println, because both are used. Also, because of overriding, print(String x) is not the same as print(Object x), so we need to stub them both.
#Kulu Limpa answer is correct but much more complicated than the actual implementation, because you need some mocking.
if you refactor your code to
static void printList(OrderedIntList list) {
System.out.println(toString(list));
}
static String toString(OrderedIntList list) {
StringBuilder result = new StringBuilder();
System.out.println(toString(list));
result.append("[");
for (int i = 0; i < list.orderedIntegerList.length; i++) {
result.append(list.orderedIntegerList[i]);
if (i != list.orderedIntegerList.length - 1) {
result.append(", ");
}
}
result.append("]");
return result.toString();
}
testing should be much easier:
#Test
public void aListOfSizeOneShouldPrintTheElementWithinBrackets() {
ArrayList<Integer> listWithOneElement = new ArrayList<Integer>();
listWithOneElement.add(4);
OrderedIntList list = new OrderedIntList(listWithOneElement);
String result = OrderedIntList.toString(list);
MatcherAssert.assertThat(result, is("[4]"));
}
The library System Rules has a JUnit rule called StandardOutputStreamLog. With this rule you are able to test your code, that writes to System.out:
public void MyTest {
#Rule
public final StandardOutputStreamLog log = new StandardOutputStreamLog();
#Test
public void test() {
OrderedIntList list = ...
printList(list);
assertEquals("[1,2,3,4]", log.getLog());
}
}
Hello all
I have a piece of software that I would like to run many different times, each for a particular value of a class field that is set in the class's constructor.
E.g, somewhere in the code is something along the lines of
public class Stuff
{
private double importantVal;
public Stuff(double val)
{
this.importantval = val;
}
public double doStuff()
{
return 4 * importantVal;
}
}
This class and method is very far down in the program/call-stack, so I can't merely call doStuff several times by itself.
I would like to test the program for various values of importantVal, perhaps by placing them in a file and iterating over them. I worked out the easy bit of running the program many times , but I have no good idea of how to substitute different values of importantVal. If all else fails I can always write a script that modifies the source code, but that feels ugly and ad-hoc. Is there a more elegant solution involving injection, or something along those lines?
To illustrate what the folks are trying to tell you here, here's how the testcases might look like:-
public class StuffTest {
#Test
public void testDoStuff_Zero(){
Stuff stuff = new Stuff(0);
assertEquals(0, stuff.doStuff());
}
#Test
public void testDoStuff_One(){
Stuff stuff = new Stuff(1);
assertEquals(4, stuff.doStuff());
}
#Test
public void testDoStuff_NegativeValue(){
Stuff stuff = new Stuff(-10);
assertEquals(-40, stuff.doStuff());
}
#Test
public void testDoStuff_PositiveValue(){
Stuff stuff = new Stuff(10);
assertEquals(40, stuff.doStuff());
}
#Test
public void testDoStuff_DecimalValue(){
Stuff stuff = new Stuff(1.1);
assertEquals(4.4, stuff.doStuff());
}
}
public class StuffRunner {
public static void main(String[] args) {
double x = 0.0d;
for (int i = 0; i < 100; ++i) {
Stuff s = new Stuff(x);
if (s.doStuff() != 4 * x) {
System.out.print("Error, unexpected value. X=");
System.out.println(x);
}
x = x + 1.22;
}
}
}
do you have multiple instances of the Stuff class? if not, perhaps you could try "injecting" the values by making importantVal static? or to inject multiple values use a List?
public class Stuff{
private static List<Double> testVals = new LinkedList()<Double>;
private double importantVal;
public Stuff(double val)
{
this.importantval = val;
}
public static addTest(double test){
testVals.add(test);
}
public double doStuff()
{
return 4 * testVals.removeFirst();
}
}
In JUnit4 you can write parameterized unit tests by providing parameters collection in one method, which will be passed to the constructor of the test and testing in another method. If I have a parameter for which I expect an exception to be thrown, how do I specify that?
this is how i use junit parameterized test with expected exceptions:
#RunWith(Parameterized.class)
public class CalcDivTest {
#Parameter(0)
public int num1;
#Parameter(1)
public int num2;
#Parameter(2)
public int expectedResult;
#Parameter(3)
public Class<? extends Exception> expectedException;
#Parameter(4)
public String expectedExceptionMsg;
#Rule
public ExpectedException thrown = ExpectedException.none();
#Parameters
public static Iterable<Object[]> data() {
return Arrays.asList(new Object[][] {
// calculation scenarios:
{ 120, 10, 12, null, null }, // simple div
{ 120, 0, -1, ArithmeticException.class, "/ by zero" }, // div by zero
});
}
#Test
public void testDiv() throws CCalculationException {
//setup expected exception
if (expectedException != null) {
thrown.expect(expectedException);
thrown.expectMessage(expectedExceptionMsg);
}
assertEquals("calculation result is not as", expectedResult, div(num1, num2) );
}
private int div(int a, int b) {
return a/b;
}
}
In contrast to what other suggest, I would not introduce any kind of logic to tests - even simple ifs!
What you should have are two testing methods:
first one takes valid parameters (and expects some output)
second takes invalid parameters (and expects exceptions)
Not sure if JUnit with its constructor-based parametrized testing is able to do this. Probably you would have to create two test classes for this. Go with JUnit Params or TestNG which offer much more convenient solution.
I agree with Tomek, and would go with two tests. The first tests for cases where no exceptions are expected. The second tests for values that should result in exceptions being thrown (i.e., and fails if they are not thrown).
Below is a simple example, where the implementation of ExceptionThrower.throwAnInstanceException(int) simply throws an IllegalArgumentException when the supplied int is less-than-1. In your implementation, all supplied values should trigger the exception.
#ParameterizedTest
#ValueSource(ints = {0, 1})
public void parameterizedIntExceptionTest(int testValue) {
ExceptionThrower exceptionThrower = new ExceptionThrower();
assertThrows(IllegalArgumentException.class, () -> {
exceptionThrower.throwAnInstanceException(testValue);
});
}
If you wanted to supply multiple arguments, then you'd be looking at using a MethodSource vice a ValueSource for the test.
if (parameter == EXCEPTION_EXPECTED) {
try {
method(parameter);
fail("didn't throw an exception!");
} catch (ExpectedException ee) {
// Test succeded!
}
}
Gabriel, please look at TestWatcher rule (since JUnit 4.9). Here is the sample code quoted from http://junit-team.github.io/junit/javadoc/4.11/org/junit/rules/TestWatcher.html:
public static class WatchmanTest {
private static String watchedLog;
#Rule
public TestWatcher watchman= new TestWatcher() {
#Override
protected void failed(Throwable e, Description description) {
watchedLog+= description + "\n";
}
#Override
protected void succeeded(Description description) {
watchedLog+= description + " " + "success!\n";
}
};
#Test
public void fails() {
fail();
}
#Test
public void succeeds() {
}
}
Another approach would be to use ErrorCollector from JUnit 4.7:
#Rule
public ExpectedException thrown = ExpectedException.none();
#Test
public void testCollectingErrors() {
thrown.handleAssertionErrors();
thrown.expect(MultipleFailureException.class); // or #expectMessage()/#expectCause()
collector.checkThat("a", equalTo("b"));
//...
}
If you used catch-exception instead of the corresponding annotations and rules of JUnit4, then your code would look like this:
catchException(obj).method(parameter);
if (parameter != EXCEPTION_EXPECTED) {
assert caughtException() instanceof ExpectedException;
}
// more assertions
#Test(expected = Exception.class)
#Parameters(value = { "invalidInput1", "invalidInput2" })
public void shouldThrowOnInvalidInput(String input) {
ClassToTest.methodToTest(input);
}
Using junitparams.Parameters from junitparams library.