Current Situation
As a pre-build test, I am trying to check that any commits/changes to java fixtures will not break any fitnesse tests I have in my fitnesse server.
The approach I have taken is to grab all the tests (context.txt) files that I want to verify the commit will not break, and parse it the best I can and compare it with the available methods I can grab using reflection from my projects.
Currently, I have a HashMap of 'class name' to 'class object' for all my available java fixtures. I also have been able to get access to all the fitnesse tests as File objects. It looks a little like this:
HashMap<String, Class<?>> availableJavaClassesMap = initJavaMap();
List<File> allFitnesseTestFiles = initTestFiles();
Goal
Now I want to be able to do something like this:
HashMap<String, Method> parsedMethodsFromFitnesseMap;
For(File file: allFitnesseTestFiles) {
parsedMethodsFromFitnesseMap.addAll( parseFile( file ) );
}
Then I would simply compare the two HashMaps and make sure the parsedMethodsFromFitnesseMap HashMap is a subset of the availableJavaClassesMap HashMap.
Worries
Include files: how to handle parsing those first/other approaches
Scenarios: create my own list of known scenarios and how they work
Ideal Solution
Is there an already made parser that will do this?
I have found the Slim Code and think it could be refactored to my needs.
Is this the best approach for this pre-build check?
Note
Each test is very expensive to run, therefore simply running all the tests to see if they still work is not an option for my needs.
Ideas? Thanks.
My Solution
The approach I settled on was using SVNKit to programmatically pull from svn into the systems temp folder and then did my parsing and comparing between methods and then deleted the directory when I was done.
For execution time, if I pulled and parsed 200 tests, it took about 20 seconds to do so. However, about 90% of the time is taken up by simply pulling the tests from the svn repo.
Holding on to the repo instead of deleting it would solve this timing issue (besides the first pull obviously), however for my jUnit style approach, I will probably take the longer time hit for sake of encapsulation.
I made sure that the import files were parsed before hand so that the scenarios in those could be used when parsing the actual tests.
I ended up not using any of the slim code (However if you want to give it a shot, the InstructionExecutor class was what I thought was most useful/close to a parser.
This was a fairly easy approach and I would recommend it.
Note
I never tried the restful approach, which might of yielded a faster execution time, but I did enjoy the simplicity once the repo was successfully pulled onto the machine.
Related
Suppose I want to write a test(s) for a Java class that would provide a method for reading and parsing external files (to be precise, files would be JSON and I would be using Jackson).
Moreover, I've got some examples of JSON files I'd parse, and I also have a vague idea what kind of Java object this SomeMagicalReader.readPony("path/to/location/pony.json") method should return; if I manage to getreadPony to return a some kind of PonyObject, I think have an idea how to test that the produced PonyObject is what I expect.
The question I have concerns providing the readPony function with the test data. I'm probably thinking about this way too much, but (1) is there an idiomatic "Java + Junit" way of doing that? (= testing a method that reads external files?) Copypaste the contents of the example file as a String variable in the test code? (They're fairly short, but that would still end up looking ugly quite fast.) Place the example JSONs just ...somewhere and call readPony with the path? (This sounds more sensible.) (2) What would then be a canonical place to put such external JSON test files, if my tests are organized in a Maven style test package hierarchy, e.g. src/test/java/com/stuff/app/package/SomeMagicalReaderTest.java?
As per maven's standard directory layout, I would advise you to put you JSON test files in src/test/resources since they are test resources. Note that you can (and should) organize your own folders hierarchy under the resources folder (other developers would find it easier to locate specific test resources when fixing or adding some other tests).
So yes you would end up with your JSON files somewhere but not anywhere if your own test resources hierarchy is good enough (for example, if you think your package structure is well organized with meaningful package names, following it for your test resources hierarchy isn't a bad idea at all).
You should ask yourself what the mission critical code is for your project - reading files or parsing their content. For my projects, parsing was the interesting part, so I placed files to parse as test resources, read them in the unit test to string and pass them to the parser to unit test the parser. It is also possible to include the contents directly in unit tests as big ugly strings, but when you have a dedicated place for test resources, why not use it.
Copypaste the contents of the example file as a String variable in the test code
I suggest against doing this as it makes modifying the input for your tests more difficult. Also, using an external file makes your tests more flexible. For example, reading an external file allows you to create multiple tests while reusing the basic framework for each test. Of course, this also means that you will need to take some time to design the methods that actually perform the tests.
Background:
I'm working with (for me) a reasonably large codebase (eg: I've only got a few of the related projects checked out at the moment, and its > 11000 classes).
Build is ant, Tests are JUnit, CI is Jenkins.
Running all tests before checkin is not an option, it takes Jenkins hours. Even for some of the individual apps it can be 45 minutes.
There are some tests that don't reference individual methods based on reflection, and in some cases don't even directly reference the class of the tested methods, as they interrogate an aggregator class, and are aware of the patterns of pass-through methods in use here. As it's a big codebase, > 10 developers, and I'm not in charge, this is something I can not change for now.
What I want, is the ability to before check-in print out a list of all test classes that are two degrees away (Kevin-Bacon-wise) from any class in the git diff list. This way I can run them all and cut down on angry emails from Jenkins when something I missed eventually gets run and has an error.
The easiest way I can think of to achieve this is to code it myself with a Ruby script or similar, which allows me to account for some of the patterns we're using, but to do it I need to be able to query "which classes reference class X?"
I could parse .java or (easier) .class files to get this info, but I'd rather not :) Is there a way I can make Javac export it in a simple format as it compiles?
Is there a way I can make Javac export it in a simple format as it compiles?
AFAIK, no.
However, there are other ways to get a list of the dependencies:
How do I get a list of Java class dependencies for a main class?.
(Note however that you are unlikely to get a static tool to extract dependencies resulting from Class.forName(), etcetera. Also note that you cannot infer the complete set of dependencies from bytecode files because of the way that "compile time constants" are handled.)
It strikes me that there are a few problems here:
It sounds to me like your build, and indeed your project structure is monolithic. If you could restructure the code base into large-scale modules that build separately (according to their dependencies), and version controlled separately, then you only need to do a full build and run all unit tests when there is a change high up ... in a module that everything else depends on. (Can I suggest the "Maven" word. It really helps for a large codebase, and 11,000 classes is large.)
It sounds like you may be suffering from the "branches are hard" problem of classic VCS systems.
It sounds like you may need a beefier CI system. If you've got more cores and the build framework is right, you should be able to get faster CI builds. (And if you modularize so that you rebuild less ...)
I think it might be easier to address your slow build/test cycle that way rather than via extra (possibly bespoke) tooling to do dependency analysis.
But I recognize that it may not be up to you to make those decisions.
I am continuing the development of a serialization layer generator. The user enters a description of types (currently in XSD or in WSDL), and the software produces code in a certain target language (currently, Java and ansi C89) which is able to represent the types described and which is also able to serialize (turn into a byte-sequence) and deserialize these values.
Since generating code is tricky (I mean, writing code is hard. Writing code that writes code is writing code to do a hard thing, which is a whole new land of hardness :) ). Thus, in the project which preceded my master thesis, we decided that we want some system tests in place.
These system tests know a type and a number of pairs of values and byte sequences. In order to execute a system test in a certain language, the type is run through the syste, resulting in code as described above. This code is then linked with some handwritten host-code, which is capable of reading these pairs of a byte sequence and a value and functions to read values of the given value from a string. The resulting executable is then run and the byte-value-pairs are fed into this executable and it is overall checked if all such bindings result in the output "Y". If this is the case, then these example values for the type serialize into the previously defined byte sequence and we can conclude that the generated code compiles and runs correctly, and thus, overall, that the part of the system handling this type is correct. This is a very good thing.
However, right now I am a bit unhappy with the current implementation. Currently, I have written a custom junit runner which uses quite a lot of reflection sorcery in order to read these byte-value-bindings from a classes attributes. Also, the overall stack to generate the code requires a lot of boilerplate code and boilerplate classes which do little more than to contain two or three strings. Even worse, it is quite hard to get a good integration with all tools which base on Junits descriptions and which generate test failure reports. It is quite hard to actually debug what is happening if the helpful maven Junit testrunner or the eclipse test runner gobble up whatever errors the compiler threw, just because the format of this error is different from junits own assertion errors.
Even worse, a single failed test in the generated code causes the maven build to fail. This is very annoying. I like it if the maven build fails if a certain test of a different unit fails, because (for example), if a certain depth first preorder calculation fails for some reason, everything will go haywire. However, if I just want to show someone some generated code for a type I know working, then it is very annoying if I cannot quickly build my application because the type I am working on right now is not finished.
So, given this background, how can I get a nice automated system which checks these generation specifications? Possibilities I have considererd:
A Junit integrated solution appears to be less than ideal, unless I can improve the integration of maven and junit and junit with my runner and everything else.
We used fitnesse earlier, but overall ditched it, because it caused more problems than it solved. The major issues we had were integration into maven and hudson.
A solution using texttest. I am not entirely convinced, because this mostly wants an executable, strings to put on stdin and strings to expect on stdout. Adding the whole "run application, link with host code and THEN run the generated executable" seems kinda complicated.
Writing my own solution. This will of course work and do what I want. However, this will be the most time consuming task, as usual.
So... do you see another possible way to do this while avoiding to write something own?
You can run Maven with -Dmaven.test.skip=true. Netbeans has a way to set this automatically unless you explicitly hit one of the commands to test the project, I don't know about Eclipse.
Here's the scenario. I have VO (Value Objects) or DTO objects that are just containers for data. When I take those and split them apart for saving into a DB that (for lots of reasons) doesn't map to the VO's elegantly, I want to test to see if each field is successfully being created in the database and successfully read back in to rebuild the VO.
Is there a way I can test that my tests cover every field in the VO? I had an idea about using reflection to iterate through the fields of the VO's as part of the solution, but maybe you guys have solved the problem before?
I want this test to fail when I add fields in the VO, and don't remember to add checks for it in my tests.
dev environment:
Using JUnit, Hibernate/Spring, and Eclipse
Keep it simple: write one test per VO/DTO:
fill the VO/DTO with test data
save it
(optional: check everything has been correctly save at the database level, using pure JDBC)
load it
check that the loaded VO/DTO and the original one matches
Productive code will evolve and tests will need to be maintained as well. Making tests the simplest as possible, even if they are repetitive, is IMHO the best approach. Over-engineering the tests or testing framework itself to make tests generic (e.g. by reading fields with reflection and filling VO/DTO automatically) leads to several problems:
time spent to write the test is higher
bug might be introduced in the test themselves
maintenance of the test is harder because they are more sophisticated
tests are harder to evolve, e.g. the generic code will maybe not work for new kinds of VO/DTO that differ slightly from the other and will be introduced later (it's just an example)
tests can not be used easily as example of how the productive code works
Test and productive code are very different in nature. In productive code, you try to avoid duplication and maximize reuse. Productive code can be complicated, because it is tested. On the other hand, you should try to have tests as simple as possible, and duplication is ok. If a duplicated portion is broken, the test will fail anyway.
When productive code change, this may require several tests to be trivially changed. With the problem that tests are seen as boring piece of code. But I think that's the way they should be.
If I however got your question wrong, just let me know.
I would recommend cobertura for this task.
You will get a complete code coverage report after you run your tests and if you use the cobertura-check ant task you can add checks for the coverage and stop the ant call with the property haltonfailure.
You could make it part of the validation of the VO. If the fields aren't set when you use a getter it can throw an exception.
I have a web tool which when queried returns generated Java classes based upon the arguments in the URL.
The classes we retrieve from the webserver change daily and we need to ensure that they still can process known inputs.
Note these classes do not test the webserver, they run locally and transform xml into a custom format. I am not testing the webserver.
These classes must then be placed in specific package structure compiled and run against a known set of input data and compared against known output data.
I would like to do this automatically each night to make sure that the generated classes are correct.
What is the best way to achieve this?
Specifically whats the best way to:
retrieve the code from a webserver and place it in a file
compile the code and then call it
I'm sure a mix of junit and ant will be able to achieve this but is there and standard solution / approach for this?
First up, to answer your question: No, I do not think that there is a standard approach for this. This sounds like quite an unusual situation ;-)
Given that, what I would do is to write your JUnit tests to all call a class GeneratedCode, and then, once you download the code, rename the class to GeneratedCode, compile, and run your unit tests.
You have the same goal as continuous integration ;-)
Maybe a bit overkill for this simple task, but this is the standard way to get something, compile something and test something regularly.
E.g. you could try hudson.
You should be creating a "mock" interface for your web service that (a) behaves the same way and (b) returns a known answer.
You should then do some other integration testing with the live web service where a person looks at the results and decides if they worked.
Can you only test the generated classes after they were published on the webservice ? You have no way to test during or just after the generation ?
One idea, if the generated code isn't to complex, is to load it via the GroovyClassLoader and to run your tests against it. See this page for examples.