I figured out how to use the Ant API to run a JUnit Test and create an XML of the result.
String pathToReports = "/tmp/junitreports";
Project project = new Project();
JUnitTest test = null;
try
{
new File(pathToReports).mkdir();
JUnitTask task = new JUnitTask();
project.setProperty("java.io.tmpdir",pathToReports);
task.setProject(project);
FormatterElement.TypeAttribute type = new FormatterElement.TypeAttribute();
type.setValue("xml");
FormatterElement formater = new FormatterElement();
formater.setType(type);
task.addFormatter(formater);
test = new JUnitTest(TestClass.class.getName());
test.setTodir(new File(pathToReports));
task.addTest(test);
task.execute();
}
...
TestClass:
public class TestClass
{
#Test
public void test()
{
fail("failed");
}
}
The Code works just fine. The XML is created and I can see that the test "failed".
Now my question: is there any way to also get the test results programatically? I expected to get an updated version of the JUnitTest object somehow where I can call the method "failureCount()".
test.failurecount() after execution of the task returns 0 of course. Parsing the XML seems odd to me as the number of failures should already be stored somewhere.
You could have a variable (int failedTests) increment each time a test fails. You could do this by using a TestWatcher rule.
After all the tests have run you could print it out (or do whatever you wanna do with it...) with:
#AfterClass
public static void printFailedTestsCount() {
System.out.println(failedTests + " tests failed.");
}
Related
I am trying to port a project from JUnit 4 to JUnit 5. The project includes a custom runner that has a listener that detects whether a test has a certain annotation (#GradedTest) and accesses the annotation's key-value pairs. For example, it would be able to access the values associated with name and points in this code:
#Test
#GradedTest(name = "greet() test", points = "1")
public void defaultGreeting() {
assertEquals(GREETING, unit.greet());
}
The existing JUnit 4 code has a listener that extends RunListener and overrides testStarted():
#Override
public void testStarted(Description description) throws Exception {
super.testStarted(description);
this.currentGradedTestResult = null;
GradedTest gradedTestAnnotation = description.getAnnotation(GradedTest.class);
if (gradedTestAnnotation != null) {
this.currentGradedTestResult = new GradedTestResult(
gradedTestAnnotation.name(),
gradedTestAnnotation.number(),
gradedTestAnnotation.points(),
gradedTestAnnotation.visibility()
);
}
}
Note that this makes use of Description.getAnnotation().
I am trying to switch to the JUnit Platform Launcher API. I can use a LauncherDiscoveryRequestBuilder to select the tests I want to run, and I can create listeners that extend SummaryGeneratingListener and override executionStarted(TestIdentifier testIdentifier). I see no way, however, to get an annotation and its values from a TestIdentifier.
What is the JUnit 5 equivalent of Description.getAnnotation() or the new way of getting a test annotation's values?
I did find a way to get annotations, but I do not know how robust it is. This is how I overrode SummaryGeneratingListener.executionStarted(TestIdentifier identifier):
#Override
public void executionStarted(TestIdentifier identifier) {
super.executionStarted(identifier);
this.currentGradedTestResult = null;
// Check if this is an atomic test, not a container.
if (identifier.isTest()) {
// Check if the test's source is provided.
TestSource source = identifier.getSource().orElse(null);
// If so, and if it's a MethodSource, get and use the annotation if present.
if (source != null && source instanceof MethodSource) {
GradedTest gradedTestAnnotation = ((MethodSource) source).getJavaMethod().getAnnotation(GradedTest.class);
if (gradedTestAnnotation != null) {
this.currentGradedTestResult = new GradedTestResult(
gradedTestAnnotation.name(),
gradedTestAnnotation.number(),
gradedTestAnnotation.points(),
gradedTestAnnotation.visibility()
);
this.currentGradedTestResult.setScore(gradedTestAnnotation.points());
}
}
}
this.testOutput = new ByteArrayOutputStream();
System.setOut(new PrintStream(this.testOutput));
}
The weak link is TestIdentifier.getSource(). The documentation says it gets "the source of the represented test or container, if available." It works for my tests, but I don't know under what circumstances the source is (not) available.
I am trying to test a method which has to print a specific string to System.out.
I have used this answer to get this working with JUnit.
This works fine with JUnit, but the tests are then run in a CI environment, by Maven surefire. The test then fails, supposedly because Maven catches System.out and it is not available to the test ?
Can someone please help me to find a way to use the console output in this Surefire test ?
Here is my test :
private final ByteArrayOutputStream outContent = new ByteArrayOutputStream();
private final ByteArrayOutputStream errContent = new ByteArrayOutputStream();
private final PrintStream originalOut = System.out;
private final PrintStream originalErr = System.err;
#Before
public void setUpStreams() {
System.setOut(new PrintStream(outContent));
System.setErr(new PrintStream(errContent));
}
#After
public void restoreStreams() {
System.setOut(originalOut);
System.setErr(originalErr);
}
#Test
public void testUpdateData() throws Exception {
MyData data = new MyData();
DataUtil.updateData(data);
Assert.assertTrue(outContent.toString().contains("is updating data within last 24h")); // assert that warning issued
}
The tested class :
public final class DataUtil {
public static void updateData(Data d) {
/// do stuff
if (/*condition*/) {
System.out.println("data " + d.id + "is updating data within last 24h");
}
}
}
Output with Maven testing :
testUpdateData Time elapsed: 0.035 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
I fixed this with changing the System.out.println() to logging with log4j.
Following Raedwalds comment and link7 I decided to discard the old system output. But instead of using PrintStreams I went one step further and refactored the code to use Log4j logging.
Then I used this answer to read the outputted logs from within the unit test.
I'm trying to write a unit test that checks if methods were invoked in an order. To do that I'm using Mockito's inOrder.verify() like this:
#Test
public void shouldExecuteAllFileCommandsOnAFileInFIFOOrder() {
// Given
ProcessFileListCommand command = new ProcessFileListCommand();
FileCommand fileCommand1 = mock(FileCommand.class, "fileCommand1");
command.addCommand(fileCommand1);
FileCommand fileCommand2 = mock(FileCommand.class, "fileCommand2");
command.addCommand(fileCommand2);
File file = mock(File.class, "file");
File[] fileArray = new File[] { file };
// When
command.executeOn(fileArray);
// Then
InOrder inOrder = Mockito.inOrder(fileCommand1, fileCommand2);
inOrder.verify(fileCommand1).executeOn(file);
inOrder.verify(fileCommand2).executeOn(file);
}
However, the second verify() fails with the following error:
org.mockito.exceptions.verification.VerificationInOrderFailure:
Verification in order failure
Wanted but not invoked:
fileCommand2.executeOn(file);
-> at (...)
Wanted anywhere AFTER following interaction:
fileCommand1.executeOn(file);
-> at (...)
If I change .executeOn(file) to .executeOn(any(File.class)) the test passes, but I want to make sure that the methods are invoked using the same argument.
Here's the class I'm testing:
public class ProcessFileListCommand implements FileListCommand {
private List<FileCommand> commands = new ArrayList<FileCommand>();
public void addCommand(final FileCommand command) {
this.commands.add(command);
}
#Override
public void executeOn(final File[] files) {
for (File file : files) {
for (FileCommand command : commands) {
file = command.executeOn(file);
}
}
}
}
The test fails because the argument to the second executeOn() method call is not the same file as the argument of the first one, since the first file is replaced by another one in
file = command.executeOn(file);
I am testing with the wonderful TestNG-Framework. My question is if it is possible to set the annotations for #Test-annotation in the testng.xml-configuration file?
I don't want to hard-code the #Test-annotation like
#Test(dataProvider = "dataFileProvider", dataProviderClass = TestDataProvider.class)
I want to configure it in the testng.xml
I have got two ideas on this case:
Workaraound 1: StaticProvider
You can easily change the Static Provider if needed
Workaraound 2: Annotation Transformer
Never tried that but should work even if have to grab the XML- data manually
Looking forward to Mr. Beust's answer... ;)
The short answer is: no, you can't add annotations to your code from testng.xml.
You can modify existing annotations with an Annotation Transformer, as explained by Frank.
Sometimes, you just really want to do something and you can't, like accessing private variables to fix memory leaks. Figuring out how to do things like this, despite the fact that you can't are fun. In case, you really want to, I might suggest trying to run your suite using the TestNG object and before running loading the testng.xml file.
Personally, I like using 'mvn test' and unfortunately, adding the pom.xml code to run from a testng xml file will require that you supply a testng.xml file, so 'mvn test' won't work. Always make sure what 95% of programmers use works, then allow overridding.
So, I might suggest extending the testng.xml file yourself and writing some code to read the testng.xml file and configure annotations using the annotation transformer class.
Here is some code to get you started:
public class TestNGSuite {
public static void main(String[] args) {
System.out.println("main start");
try {
new TestNGSuite(new Class[]{ Demo.class });
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("main finish");
}
public TestNGSuite(Class[] classes) throws Exception {
// Create Suite List
List<XmlSuite> suites = new ArrayList<XmlSuite>();
// Add Suite to Suite List
XmlSuite suite = new XmlSuite();
suites.add(suite);
suite.setName("MyTestSuite");
// Add Test to Suite
XmlTest test = new XmlTest(suite);
test.setName("MyTest");
// Add Class List to Test
List<XmlClass> xmlClasses = new ArrayList<XmlClass>();
test.setXmlClasses(xmlClasses);
// Add Class to Class List
for(Class clazz: classes) {
XmlClass xmlClass = new XmlClass(clazz);
xmlClasses.add(xmlClass);
}
// Run TestNG
TestNG testNG = new TestNG();
testNG.setXmlSuites(suites);
testNG.addListener(new TestNGAnnotationTransformer(methodName));
testNG.addListener(new TestNGSuiteConsoleLogger());
testNG.run();
if(testNG.hasFailure()) { // Throw an exception to make mvn goal fail
throw new Exception("Failed Tests");
}
}
public static class TestNGSuiteConsoleLogger extends TestListenerAdapter{
#Override
public void onTestFailure(ITestResult tr) {
Console.log(TestNGSuiteConsoleLogger.class, "FAILURE:"+tr.getMethod());
tr.getThrowable().printStackTrace();
}
}
public static class TestNGAnnotationTransformer implements IAnnotationTransformer{
String methodToRun;
public TestNGAnnotationTransformer(String methodName) {
methodToRun = methodName;
}
public void transform(ITestAnnotation annotation, Class arg1,
Constructor arg2, Method testMethod) {
if (methodToRun.equals(testMethod.getName())) {
annotation.setEnabled(true);
}
}
}
}
If you want to run Demo.class, make sure there is a method there with the TestNG annotation "#Test".
Is it possible to create a report from JUnit without Ant or Maven? Because I call the tests with velocitycode, and the velocitycodes calls a method. And that method calls all the tests. So I can get a response from it, the failures/errors/runs etc. But I want to create a report with it.. Or do I need to create html stuff by myself?
I created the methods and testmethods in Java, so I will do everything in Java, except the call, thats in Velocity code.
Velocitycode:
${custom.test}
Java code:
public void getTest(){
junit.textui.TestRunner runner = new junit.textui.TestRunner();
TestResult testresult = Junit.textui.TestRunner.run(runner.getTest(MyTestClass.class.getName()));
}
You will need the ant library. But with this code you can create an XML report and use it in other pograms. Such as Jenkins.
public static void getTest(){
String pathToReports = "C:\\path\\to\\the\\Reports";
Project project = new Project();
try {
new File(pathToReports).mkdir();
JUnitTask task = new JUnitTask();
project.setProperty("java.io.tmpdir",pathToReports);
task.setProject(project);
FormatterElement.TypeAttribute type = new FormatterElement.TypeAttribute();
type.setValue("xml");
FormatterElement formater = new FormatterElement();
formater.setType(type);
task.addFormatter(formater);
JUnitTest test = new JUnitTest(YOURTEST.class.getName());
test.setTodir(new File(pathToReports));
task.addTest(test);
task.execute();
} catch (Exception e) {
}
}
Don't think so. But you might be able to use ant as a library instead of a tool, and use the same code that the tool uses to generate these reports.