I'm new to BDD and particularly Cucumber.
Can I get a features and its steps from a variable? Also, I want to get a feature and its steps from a test tracker (TestRail) before running tests by the special selection of this tests, and put it in a list, then one by one get a scenario and run it.
Is there such a possibility? Should I use Cucumber or another framework for this?
No, you can't define a Cucumber scenario in code (or at least not in a supported way). But if you were going to write code to get a scenario and its steps from your test tracker and run it, you could equally well write code to put the scenario and its steps in files and run the scenario with the cucumber executable.
I don't know of a Java testing framework in which you can define tests dynamically. You could do that in Ruby with RSpec or (less cleanly) minitest. But I don't know whether a Ruby test framework would be acceptable, or whether it would be OK for the people writing entries in your test tracker to have to read and/or write RSpec examples. (It seems strange to have Cucumber step definitions in a test tracker, too; having features in a test tracker seems more reasonable, aside from the question of how to run them.)
I'm working with a large project with 50,000 source files about 5000 tests spread over 2000 classes. These are mostly JUnit 3 with a smattering of JUnit 4. These tests are run on a frequent (daily or weekly) basis from batch files to ensure the product is working and not suffering from regressions or other issues.
I want to enumerate through the source or class files and produce a list of classes and methods that represent the test cases in the project. I can compare this list to determine which tests are not being run by the batch files.
Is there any simple way to do this such as functionality in JUnit? Or is there functionality in JUnit I could programatically drive to get it, e.g. to scan a dir full of class files and figure out which are tests. I realise that I could write code to individually load each class and start examining it for certain characteristics but if something exists I'd like to be able to use it.
The second part of the question, is there any commonly used way to to annotate tests? If I could annotate tests I could potentially filter them and generate batch files on the fly that run them according to the filter criteria. e.g. run all tests which need a network connection, run all tests that use subsystem A and so on.
Is there anything which would help me do this? I realise I could roll my own annotation with a comma separated list of values or something, but perhaps this sort of thing has been formalised and there are tools that work with it.
Tests are classes that extend TestCase (in case of JUnit 3) or annotated as #Test (in case of JUnit 4). So it is not a problem to do either using IDE or programmatically: write program that finds all *class files recursively into your classes folder(s) and tests whether class extends TestCase or is annotated with #Test.
Maybe my question is a newbie one, but I can not really understand the circumstances under which I would use junit?
Whether I write simple applications or larger ones I test them with the System.out statements and it seams quite easy to me.
Why create test-classes with JUnit, unnecessary folders in the project if we still have to call the same methods, check what they return and we then have an overhead of annotating everything?
Why not write a class and test it at once with System.out but not create Test-classes?
PS. I have never worked on large projects I am just learning.
So what is the purpose?
That's not testing, that's "looking manually at output" (known in the biz as LMAO). More formally it's known as "looking manually for abnormal output" (LMFAO). (See note below)
Any time you change code, you must run the app and LMFAO for all code affected by those changes. Even in small projects, this is problematic and error-prone.
Now scale up to 50k, 250k, 1m LOC or more, and LMFAO any time you make a code change. Not only is it unpleasant, it's impossible: you've scaled up the combinations of inputs, outputs, flags, conditions, and it's difficult to exercise all possible branches.
Worse, LMFAO might mean visiting pages upon pages of web app, running reports, poring over millions of log lines across dozens of files and machines, reading generated and delivered emails, checking text messages, checking the path of a robot, filling a bottle of soda, aggregating data from a hundred web services, checking the audit trail of a financial transaction... you get the idea. "Output" doesn't mean a few lines of text, "output" means aggregate system behavior.
Lastly, unit and behavior tests define system behavior. Tests can be run by a continuous integration server and checked for correctness. Sure, so can System.outs, but the CI server isn't going to know if one of them is wrong–and if it does, they're unit tests, and you might as well use a framework.
No matter how good we think we are, humans aren't good unit test frameworks or CI servers.
Note: LMAO is testing, but in a very limited sense. It isn't repeatable in any meaningful way across an entire project or as part of a process. It's akin to developing incrementally in a REPL, but never formalizing those incremental tests.
We write tests to verify the correctness of a program's behaviour.
Verifying the correctness of a program's behaviour by inspecting the content of output statements using your eyes is a manual, or more specifically, a visual process.
You could argue that
visual inspection works, I check that the code does what it's meant to
do, for these scenarios and once I can see it's correct we're good to
go.
Now first up, it's great to that you are interested in whether or not the code works correctly. That's a good thing. You're ahead of the curve! Sadly, there are problems with this as an approach.
The first problem with visual inspection is that you're a bad welding accident away from never being able to check your code's correctness again.
The second problem is that the pair of eyes used is tightly coupled with the brain of the owner of the eyes. If the author of the code also owns the eyes used in the visual inspection process, the process of verifying correctness has a dependency on the knowledge about the program internalised in the visual inspector's brain.
It is difficult for a new pair of eyes to come in and verify the correctness of the code simply because they are not partnered up with brain of the original coder. The owner of the second pair of eyes will have to converse with original author of the code in order to fully understand the code in question. Conversation as a means of sharing knowledge is notoriously unreliable. A point which is moot if the Original Coder is unavailable to the new pair eyes. In that instance the new pair of eyes has to read the original code.
Reading other people's code that is not covered by unit tests is more difficult than reading code that has associated unit tests. At best reading other peoples code is tricky work, at its worst this is the most turgid task in software engineering. There's a reason that employers, when advertising job vacancies, stress that a project is a greenfield (or brand new) one. Writing code from scratch is easier than modifying existing code and thereby makes the advertised job appear more attractive to potential employees.
With unit testing we divide code up into its component parts. For each component we then set out our stall stating how the program should behave. Each unit test tells a story of how that part of the program should act in a specific scenario. Each unit test is like a clause in a contract that describes what should happen from the client code's point of view.
This then means that a new pair of eyes has two strands of live and accurate documentation on the code in question.
First they have the code itself, the implementation, how the code was done; second they have all of the knowledge that the original coder described in a set of formal statements that tell the story of how this code is supposed to behave.
Unit tests capture and formally describe the knowledge that the original author possessed when they implemented the class. They provide a description of how that class behaves when used by a client.
You are correct to question the usefulness of doing this because it is possible to write unit tests that are useless, do not cover all of the code in question, become stale or out of date and so on. How do we ensure that unit tests not only mimics but improves upon the process of a knowledgeable, conscientious author visually inspecting their code's output statements at runtime? Write the unit test first then write the code to make that test pass. When you are finished, let the computers run the tests, they're fast they are great at doing repetitive tasks they are ideally suited to the job.
Ensure test quality by reviewing them each time you touch off the code they test and run the tests for each build. If a test fails, fix it immediately.
We automate the process of running tests so that they are run each time we do a build of the project. We also automate the generation of code coverage reports that details what percentage of code that is covered and exercised by tests. We strive for high percentages. Some companies will prevent code changes from being checked in to source code control if they do not have sufficient unit tests written to describe any changes in behaviour to the code. Typically a second pair of eyes will review code changes in conjunction with the author of the changes. The reviewer will go through the changes ensure that the changes understandable and sufficiently covered by tests. So the review process is manual, but when the tests (unit and integration tests and possibly user acceptance tests) pass this manual review process the become part of the automatic build process. These are run each time a change is checked in. A continuous-integration server carries out this task as part of the build process.
Tests that are automatically run, maintain the integrity of the code's behaviour and help to prevent future changes to the code base from breaking the code.
Finally, providing tests allows you to aggressively re-factor code because you can make big code improvements safe in the knowledge that your changes do not break existing tests.
There is a caveat to Test Driven Development and that is that you have to write code with an eye to making it testable. This involves coding to interfaces and using techniques such as Dependency Injection to instantiate collaborating objects. Check out the work of Kent Beck who describes TDD very well. Look up coding to interfaces and study design-patterns
When you test using something like System.out, you're only testing a small subset of possible use-cases. This is not very thorough when you're dealing with systems that could accept a near infinite amount of different inputs.
Unit tests are designed to allow you to quickly run tests on your application using a very large and diverse set of different data inputs. Additionally, the best unit tests also account for boundary cases, such as the data inputs that lie right on the edge of what is considered valid.
For a human being to test all of these different inputs could take weeks whereas it could take minutes for a machine.
Think of it like this: You're also not "testing" something that will be static. Your application is most likely going through constant changes. Therefore, these unit tests are designed to run at different points in the compile or deployment cycle. Perhaps the biggest advantage is this:
If you break something in your code, you'll know about it right now, not after you deployed, not when a QA tester catches a bug, not when your clients have cancelled. You'll also have a better chance of fixing the glitch immediately, since it's clear that the thing that broke the part of the code in question most likely happened since your last compile. Thus, the amount of investigative work required to fix the problem is greatly reduced.
I added some other System.out can NOT do:
Make each test cases independent (It's important)
JUnit can do it: each time new test case instance will be created and #Before is called.
Separate testing code from source
JUnit can do it.
Integration with CI
JUnit can do it with Ant and Maven.
Arrange and combine test cases easily
JUnit can do #Ignore and test suite.
Easy to check result
JUnit offers many Assert methods (assertEquals, assertSame...)
Mock and stub make you focus on the test module.
JUnit can do: Using mock and stub make you setup correct fixture, and focus on the test module logic.
Unit tests ensure that code works as intended. They are also very helpful to ensure that the code still works as intended in case you have to change it later to build new functionalities to fix a bug. Having a high test coverage of your code allows you to continue developing features without having to perform lots of manual tests.
Your manual approach by System.out is good but not the best one.This is one time testing that you perform. In real world, requirements keep on changing and most of the time you make a lot of modificaiotns to existing functions and classes. So… not every time you test the already written piece of code.
there are also some more advanced features are in JUnit like like
Assert statements
JUnit provides methods to test for certain conditions, these methods typically start with asserts and allow you to specify the error message, the expected and the actual result
Some of these methods are
fail([message]) - Lets the test fail. Might be used to check that a certain part of the code is not reached. Or to have failing test before the test code is implemented.
assertTrue(true) / assertTrue(false) - Will always be true / false. Can be used to predefine a test result, if the test is not yet implemented.
assertTrue([message,] condition) - Checks that the boolean condition is true.
assertEquals([message,] expected, actual) - Tests whether two values are equal (according to the equals method if implemented, otherwise using == reference comparison). Note: For arrays, it is the reference that is checked, and not the contents, use assertArrayEquals([message,] expected, actual) for that.
assertEquals([message,] expected, actual, delta) - Tests whether two float or double values are in a certain distance from each other, controlled by the delta value.
assertNull([message,] object) - Checks that the object is null
and so on. See the full Javadoc for all examples here.
Suites
With Test suites, you can in a sense combine multiple test classes into a single unit so you can execute them all at once. A simple example, combining the test classes MyClassTest and MySecondClassTest into one Suite called AllTests:
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
#RunWith(Suite.class)
#SuiteClasses({ MyClassTest.class, MySecondClassTest.class })
public class AllTests { }
The main advantage of JUnit is that it is automated rather than you manually having to check with your print outs. Each test you write stays with your system. This means that if you make a change that has an unexpected side effect your test will catch it and fail rather than you having to remember to manually test everything after each change.
JUnit is a unit testing framework for the Java Programming Language. It is important in the test driven development, and is one of a family of unit testing frameworks collectively known as xUnit.
JUnit promotes the idea of "first testing then coding", which emphasis on setting up the test data for a piece of code which can be tested first and then can be implemented . This approach is like "test a little, code a little, test a little, code a little..." which increases programmer productivity and stability of program code that reduces programmer stress and the time spent on debugging.
Features
JUnit is an open source framework which is used for writing & running tests.
Provides Annotation to identify the test methods.
Provides Assertions for testing expected results.
Provides Test runners for running tests.
JUnit tests allow you to write code faster which increasing quality
JUnit is elegantly simple. It is less complex & takes less time.
JUnit tests can be run automatically and they check their own results and provide immediate feedback. There's no need to manually comb through a report of test results.
JUnit tests can be organized into test suites containing test cases and even other test suites.
Junit shows test progress in a bar that is green if test is going fine and it turns red when a test fails.
I have slightly different perspective of why JUnit is needed.
You can actually write all test cases yourself but it's cumbersome. Here are the problems:
Instead of System.out we can add if(value1.equals(value2)) and return 0 or -1 or error message. In this case, we need a "main" test class which runs all these methods and checks results and maintains which test cases failed and which are passed.
If you want to add some more tests you need to add them to this "main" test class as well. Changes to existing code. If you want to auto detect test cases from test classes, then you need to use reflection.
All your tests and your main class to run tests are not detected by eclipse and you need to write custom debug/run configurations to run these tests. You still don't see those pretty green/red colored outputs though.
Here is what JUnit is doing:
It has assertXXX() methods which are useful for printing helpful error messages from the conditions and communicating results to "main" class.
"main" class is called runner which is provided by JUnit, so we don't have to write any. And it detects the test methods automatically by reflection. If you add new tests with #Test annotation then they are automatically detected.
JUnit has eclipse integration and maven/gradle integration as well, so it is easy to run tests and you will not have to write custom run configurations.
I'm not an expert in JUnit, so that's what I understood as of now, will add more in future.
You cannot write any test case without using testing framework or else you will have to write your testing framewok to give justice to your test cases.
Here are some info about JUnit Framework apart from that you can use TestNG framework .
What is Junit?
Junit is widely used testing framework along with Java Programming Language. You can use this automation framework for both unit testing and UI testing.It helps us define the flow of execution of our code with different Annotations. Junit is built on idea of "first testing and then coding" which helps us to increase productivity of test cases and stability of the code.
Important Features of Junit Testing -
It is open source testing framework allowing users to write and run test cases effectively.
Provides various types of annotations to identify test methods.
Provides different Types of Assertions to verify the results of test case execution.
It also gives test runners for running tests effectively.
It is very simple and hence saves time.
It provides ways to organize your test cases in form of test suits.
It gives test case results in simple and elegant way.
You can integrate jUnit with Eclipse, Android Studio, Maven & Ant, Gradle and Jenkins
JUNIT is the method that is usually accepted by java developer.
Where they can provide similar expected input to the function and decide accordingly that written code is perfectly written or if test case fails then different approach may also need to implement.
JUNIT will make development fast and will ensure the 0 defects in the function.
JUNIT : OBSERVE AND ADJUST
Here is my perspective of JUNIT.
JUNIT can be used to,
1)Observe a system behaviour when a new unit is added in that system.
2)Make adjustment in the system to welcome the "new" unit in the system.
What? Exactly.
Real life eg.
When your relative visits your college hostel room,
1) You will pretend to be more responsible.
2) You will keep all things where they should be, like shoes in shoe rack not on chair, clothes in cupboard not on chair.
3) You will get rid of all the contraband.
4) you will start cleanUp in every device you posses.
In programming terms
System: Your code
UNIT: new functionality.
As JUNIT framework is used for JAVA language so JUNIT = JAVA UNIT (May be).
Suppose a you already have a well bulletproof code, but a new requirement came and you have to add the new requirement in your code. This new requirement may break your code for some input(testcase).
Easy way to adapt this change is using unit testing (JUNIT).
For that you should write multiple testcases for your code when you are building your codebase. And whenever a new requirement comes you just run all the test cases to see if any test case fails.
If No then you are a BadA** artist and you are ready to deploy the new code.
If any of the testcases fail then you change your code and again run testcases until you get the green status.
I am very new to TDD in general so please forgive me if my question does not make lots of sense.
After looking looking around for a bit, it seems that jUnit is capable of implement integration test. I am hoping that the community can provide me some guidance on how to write integration test. Here is a simple overview of my design.
I have Main1, that accept a list of zip files. Main will extract the zip files, edit the content of the pdf inside the zip files, and put the final pdf files into folder X. If the number of pdf reach a THRESHOLD, then Main2Processor (not a main class) will get invoked and zip all the pdf files, and also create a report text file with the same name as the newly create zip file.
If I run Main2, it also will kick off Main2Processor, which will zip the pdf file and create text file reports (even though the number of pdf in folder X did not reach a THRESHOLD).
How do I write integration test testing the correctness for my above design?
You're right; JUnit can be used to write tests that would be called integration tests. All you have to do is relax the rules regarding tests not touching external resources.
First, I would refactor the main() of your application to do as little as you can possibly make it do; there isn't a really good way to test the code in a main() function. Have it construct and run an object (that object can be the object containing main(), if you wish), passing that new object your list of ZIP files. That object can now be tested using JUnit by just instantiating it.
Now, you just have to architect the test to set up a constant test environment and then perform a repeatable test. Create, or clear out, a temp directory somewhere, then copy over some test ZIP files into that directory. Then, run your main processor.
To detect that the proper behavior occurs when the threshold is reached, you just test for the existence of a zip file (and/or its absence if the threshold isn't reached).
Do you really want an "integration test" (that term is overloaded beyond comprehension now so if you could state your end-goals that'd help) ? How about an acceptance test where you use this console/GUI app like a real user with specific input and check for the expected output?
JUnit is just a test runner and is oblivious of what the test actually does. So yes, you could use it to write any test. However it has been built for unit-testing and that will leak sometimes.. e.g. the fact that the test shuts down on the first error / the first assert that does not succeed. Usually coarse-tests like to push ahead and get a bunch of errors at the end.
If you want an integration test - you would have to redesign your app to be callable from tests (if it is not already so). Raise specific obstacles to writing this test and I can offer more specific advice.
I think that you should create some utility methods for your tests. For example running applicaiton, checking directory, clearing directory etc.
Then you will be able to implement test like the following:
#Test
public mytest1() {
exec(Main1.class, "f1.zip", "f2.zip");
Assert.assertTrue(getFileCount(OUTPUT_DIR) < THRESHOLD);
// perform verification of your files etc...
}
First of all you might describe your above specification, but from a "test sequence" point of view. For example, test one would provide to Main1 a set of N pdf files, N being under the threshold. Then your test code, after Main1 returned, would check X folder content, as well as reports, to verify that your expectations are met.
JUnit itself just helps running test case, it does not really help writing the tests.
And JUnit is "unit test" oriented (but you can use it as well for integration tests, despite some situations does not suit well; when a global setup is required for example, or when the test cases are expected to be run in a specific order...).
Some additional libraries can help greatly to interact easily with the rest of your code : dbunit, httpunit and so on.
I am continuing the development of a serialization layer generator. The user enters a description of types (currently in XSD or in WSDL), and the software produces code in a certain target language (currently, Java and ansi C89) which is able to represent the types described and which is also able to serialize (turn into a byte-sequence) and deserialize these values.
Since generating code is tricky (I mean, writing code is hard. Writing code that writes code is writing code to do a hard thing, which is a whole new land of hardness :) ). Thus, in the project which preceded my master thesis, we decided that we want some system tests in place.
These system tests know a type and a number of pairs of values and byte sequences. In order to execute a system test in a certain language, the type is run through the syste, resulting in code as described above. This code is then linked with some handwritten host-code, which is capable of reading these pairs of a byte sequence and a value and functions to read values of the given value from a string. The resulting executable is then run and the byte-value-pairs are fed into this executable and it is overall checked if all such bindings result in the output "Y". If this is the case, then these example values for the type serialize into the previously defined byte sequence and we can conclude that the generated code compiles and runs correctly, and thus, overall, that the part of the system handling this type is correct. This is a very good thing.
However, right now I am a bit unhappy with the current implementation. Currently, I have written a custom junit runner which uses quite a lot of reflection sorcery in order to read these byte-value-bindings from a classes attributes. Also, the overall stack to generate the code requires a lot of boilerplate code and boilerplate classes which do little more than to contain two or three strings. Even worse, it is quite hard to get a good integration with all tools which base on Junits descriptions and which generate test failure reports. It is quite hard to actually debug what is happening if the helpful maven Junit testrunner or the eclipse test runner gobble up whatever errors the compiler threw, just because the format of this error is different from junits own assertion errors.
Even worse, a single failed test in the generated code causes the maven build to fail. This is very annoying. I like it if the maven build fails if a certain test of a different unit fails, because (for example), if a certain depth first preorder calculation fails for some reason, everything will go haywire. However, if I just want to show someone some generated code for a type I know working, then it is very annoying if I cannot quickly build my application because the type I am working on right now is not finished.
So, given this background, how can I get a nice automated system which checks these generation specifications? Possibilities I have considererd:
A Junit integrated solution appears to be less than ideal, unless I can improve the integration of maven and junit and junit with my runner and everything else.
We used fitnesse earlier, but overall ditched it, because it caused more problems than it solved. The major issues we had were integration into maven and hudson.
A solution using texttest. I am not entirely convinced, because this mostly wants an executable, strings to put on stdin and strings to expect on stdout. Adding the whole "run application, link with host code and THEN run the generated executable" seems kinda complicated.
Writing my own solution. This will of course work and do what I want. However, this will be the most time consuming task, as usual.
So... do you see another possible way to do this while avoiding to write something own?
You can run Maven with -Dmaven.test.skip=true. Netbeans has a way to set this automatically unless you explicitly hit one of the commands to test the project, I don't know about Eclipse.