Writing a test I expect the tested method to return certain outputs. Usually I'm checking that for a given database operation I get a certain output. My practice has usually been that of writing an array as a quick map/properties file in the test itself.
This solution is quick, and is not vulnerable to run-time changes of an external file to load the expected results from.
A solution is to place the data in a java source file, so I bloat less the test and still get a compile-time checked test. How about this?
Or is loading the exepected results as resources a better approach? A .properties file is not good enough since I can have only one value per key. Is commons-config the way to go?
I'd prefer a simple solution where I name the properties per key, so for each entry I might have a doc-length and numFound property value (sounds like the elements of an xml node)?
How do you go about this?
You must remember about maintaining such tests. After writing several web services tests with Spring-WS test support I must admit that storing requests (test setup) and expected responses in external XML files wasn't such a good idea. Each request-response pair had the same name prefix as test case so everything was automated and very clean. But still refactoring and diagnosing test failures becomes painful. After a while I realized that embedding XML in test case as String, although ugly, is much easier to maintain.
In your case, I assume you invoke some database query and you get a list of maps in response. What about writing some nice DSL to make assertions on these structures? Actually, FEST-Assert is quite good for that.
Let's say you test the following query (I know it's an oversimplification):
List<Map<String, Object>> rs = db.query("SELECT id, name FROM Users");
then you can simply write:
assertThat(rs).hasSize(1);
assertThat(rs.get(0))
.hasSize(2)
.includes(
entry("id", 7),
entry("name", "John")
)
);
Of course it can and should be further simplified to fit your needs better. Isn't it easier to have a full test scenario on one screen rather than jump from one file to another?
Or maybe you should try Fitnesse (looks like you are no longer doing unit testing, so acceptance testing framework should be fine), where tests are stored in wiki-like documents, including tables?
Yes, using resources for expected results (and also setup data) works well and is pretty common.
XML may well be a useful format for you - being hierarchical can certainly help (one element per test method). It depends on the exact situation, but it's definitely an option. Alternatively, JSON may be easier for you. What are you comfortable with, in terms of serialization APIs?
Given that you mention you are usually testing that a certain DB operation returns expected output, you may want to take a look at using DBUnit:
// Load expected data from an XML dataset
IDataSet expectedDataSet = new FlatXmlDataSetBuilder().build(new File("expectedDataSet.xml"));
ITable expectedTable = expectedDataSet.getTable("TABLE_NAME");
// Assert actual database table match expected table
Assertion.assertEquals(expectedTable, actualTable);
DBUnit handles comparing the state of a table after some operation has completed and asserting that the data in the table matches an expected DataSet. The most common format for the DataSet that you compare the actual table state with is probably using an XmlDataSet, where the expected data is loaded from an XML file, but there are other subclasses as well.
If you are already doing testing like this, then it sounds like you may have written most of the same logic already - but DBUnit may give you additional features you haven't implemented on your own yet for free.
Related
I have a use case when I need to capture the data flow from one API to another. For example my code reads data from database using hibernate and during the data processing I convert one POJO to another and perform some more processing and then finally convert into final result hibernate object. In a nutshell something like POJO1 to POJO2 to POJO3.
In Java is there a way where I can deduce that an attribute from POJO3 was made/transformed from this attribute of POJO1. I want to look something where I can capture data flow from one model to another. This tool can be either compile time or runtime, I am ok with both.
I am looking for a tool which can run in parallel with code and provide data lineage details on each run basis.
Now instead of Pojos I will call them States! You are having a start position you iterate and transform your model through different states. At the end you have a final terminal state that you would like to persist to the database
stream(A).map(P1).map(P2).map(P3)....-> set of B
If you use a technic known as Event sourcing you can deduce it yes. How would this look like then? Instead of mapping directly A to state P1 and state P1 to state P2 you will queue all your operations that are necessary and enough to map A to P1 and P1 to P2 and so on... If you want to recover P1 or P2 at any time, it will be just a product of the queued operations. You can at any time rewind forward or rewind backwards as long as you have not yet chaged your DB state. P1,P2,P3 can act as snapshots.
This way you will be able to rebuild the exact mapping flow for this attribute. How fine grained you will queue your oprations, if it is going to be as fine as attribute level , or more course grained it is up to you.
Here is a good article that depicts event sourcing and how it works: https://kickstarter.engineering/event-sourcing-made-simple-4a2625113224
UPDATE:
I can think of one more technic to capture the attribute changes. You can instument your Pojo-s, it is pretty much the same technic used by Hibernate to enhance Pojos and same technic profiles use to for tracing. Then you can capture and react to each setter invocation on the Pojo1,Pojo2,Pojo3. Not sure if I would have gone that way though....
Here is some detiled readin about the byte code instrumentation if https://www.cs.helsinki.fi/u/pohjalai/k05/okk/seminar/Aarniala-instrumenting.pdf
I would imagine two reasons, either the code is not developed by you and therefore you want to understand the flow of data along with combinations to convert input to output OR your code is behaving in a way that you are not expecting.
I think you need to log the values of all the pojos, inputs and outputs to any place that you can inspect later for each run.
Example: A database table if you might need after hundred of runs, but if its one time may be to a log in appropriate form. Then you need to yourself manually use those data values layer by later to map to the next layer. I think with availability of code that would be easy. If you have a different need pls. explain.
Please accept and like if you appreciate my gesture to help with my ideas n experience.
There are "time travelling debuggers". For Java, a quick search did only spill this out:
Chronon Time Travelling Debugger, see this screencast how it might help you .
Since your transformations probably use setters and getters this tool might also be interesting: Flow
Writing your own java agent for tracking this is probably not what you want. You might be able to use AspectJ to add some stack trace logging to getters and setters. See here for a quick introduction.
We have a test where basically we need to input a specific value in a web site and make sure another value comes out. The data of the input-output for this is stored in an XML file.
Now we can create a single Scenario that runs once and loops through, submitting each value however we run into some reporting problems, if 2 out of 100 pairs fail we want to know which ones and not just have an assertion error for the whole scenario.
We would get much clearer reporting using a Scenario Outline where all the values are in the examples table. then the scenario itself runs repeatedly and we can fail an individual set as an assertion error and have that kick back clearly in a report.
Problem: we do not want to hard code all the values from the xml into the .feature. it's noisy but also if the values change it's slow to update. we would rather just provide the XML parse it and go, if things change we just drop in an updated XML.
Is there a way to create dynamic examples where we can run the scenario repeatedly, one for each data case, without explicitly defining it in the examples table ?
Using Cucumber for this is a bad idea. You should test this functionality lower down your stack with a unit test.
At some point in your code, after the user has input their value, the value will be passed to a method/function that will return your answer. This is the place to do this sort of testing.
A cucumber test going through the whole stack will upwards of 3 orders of magnitude slower than a well written unit tests. So you could test thousands of pairs of values in your unit test in the time it takes to run one single cuke.
If you do this sort of testing in Cucumber you will quickly end up with a test suite that takes far too long to run, or that can only be run quickly at great expense. This is very damaging to a project.
Cuking should be about one happy path (The user can enter a value and see the result) and maybe a sad path (the user enters a bad value and sees an error/explanation). Anything else needs to be pushed down to unit tests.
The NoraUi framework does exactly what you want to do in your project. The NoraUi code is open source. If you have questions about this framework, you can post an issue with the tag "Question"
Background
I am developing a new approach to automated testing using Selenium/TestNG/POM. So far, we have very promising results and automated large parts of our regression testing.
Now, we came to the step of expanding our test set by parametrizing them. We have end2end flows defined over multiple pages, where the pages are modeled into page objects. Most of these pages are forms, and in the end we get a resulting product/report that combines all the information.
Now my problem statement
We want to use Excel sheets so our business analysts can input the parameters and the expected values. But I am struggling to find a good way to structure the sheets. The naive approach would be to have a column for every parameter, and a line for every test case. A slightly better approach would probably be to use a different sheet for each page object, in a workbook, having a different workbook for each flow.
But my fear is that, by doing this, we are undoing all the good the POM brings us. If we need to change something (add a field in one of the screens for example), we need to regenerate all the excel sheets, or at least manually update each one. This is not that much better than having to update each scenario. In fact, we probably still have to edit all the scenario's (for example, if the new field is "name", add everywhere a "fillName('John')" step.
Is there a better way? Or is the manual work unavoidable? Or am I implementing POM incorrectly if I still need to adapt most scenarios everytime I need to add a field?
Trying to create yet another solution which use Excel as page object source you are bringing youself back to stone age. Use modern frameworks to handle Page Objects issues e.g. JDI with elements typification or Selenide wich provides more concise way.
I have a class, which parses data from html page and turns it into a collection of String. Basically I have a URL of online-shop and I want to have a list of its items.
My class have the following signature:
public static List<String> getShopItems()
usually method returns very large List (4k items or more).
My question is, how can I test this method?
I think that I have to assert that returned list has correct size and contains all the items needed. But it would be very tedious to create List with 4k items and compare actual and expected Lists. Furthermore, items can change in the future, and my test will fail.
To sum it up, I can get actual data from my method getShopItems(), but I have no idea how to get expected data for assertion in test.
Thanks in advance.
You're testing data extraction from a webpage, not the webpage itself. As a consequence, you can (and should) make your own test page. This way, you can
reduce the list of needed items (I assume there aren't 4k different cases ?)
ensure the data won't change
Firstly, static methods like this are almost always a bad idea they make testing difficult.
Secondly, a Resource interface can really help testing cases like this. eg:
public interface Resource {
InputStream getStream();
}
Then you could refactor the class to be something like:
public class ShopItemProvider {
private final Resource resource;
public ShopItemProvider(Resource resource) {
this.resource = resource;
}
#Override
public List<String> getShopItems() {
try (InputStream in = resource.getStream()) {
return someFancyParseMethod(in);
}
}
}
Now, you've got the hook you need for testing. In a test case you can mock up an InputStream with just a few records in it, possibly sourcing a test file from the test classpath.
In production, the InputStream can come from a URL or File or maybe some production classpath resource.
A simple way is to keep the expected items in a local file. Then you can load the file contents, store all items in a sorted list and compare it to the (sorted) list of your shop items.
I normally split these tests up into 2 parts:
Testing the function thoroughly with basic / small data sets
Testing some basic properties with real data
You can achieve the different inputs by creating an extra parameter in the constructor (possibly optional), for example a location of a HTML file with test data. This allows you to inject fake data into the class.
Basic Test
Try to get 100% code coverage with this test, test for error handling, and for invalid data. Use several data sets for this test, and verify all output is correct.
Real data test
Try to use data from a real source, this could be the live site, but that might not always be available when running the test. Also it might fluctuate too much too write a proper unit test for. At least aim to write a test that will only fail if you have made an error writing the system (not if the data in the site changes). So most of the time I go for a data set that is saved earlier, so I can assert a bit more on the data set.
Now you have to be creative in your tests: try to assert for properties of the data. Examples could be (try to pick several!):
All prices in a price list must be positive.
There are no errors coming from your exhaustive error checking in the function.
Check the number of items. For the real data do this vaguely, for fake data do check the exact number.
separate method that gets the data from the method that parse the data so you can test different inputs (parse method should take a string or a stream rather than an url)
do very detailed tests using custom, small inputs. this should be your main testing part
you can take a snapshot of the webpage, save it locally and use it to test real data. update this file when page structure changes
last part is integration tests. connect to real webpage and check if your parser doesn't throw exceptions, if it still gives some reasonable output (like a list greater than 1k elements), check if page structure didn't change. don't test for exact list content as the page may change. also don't include integration tests into your unit tests as sometimes page may be down, network may be down etc
I am going to use a template engine in Java (probably FreeMarker). Now I wonder how to write a unit test for it.
I can prepare an expected output as a text file but I will have to change it manually whenever I change the template. I would like to parse the output using the template to get the model data and it with the original model. Can I do that?
Depends on what your template will output. Of course if it produced a well formed format like XML or JSON you could easily parse it so I'm left to assume it's probably HTML. You might be able to use JTidy to parse it into a DOM model that can be used from Java.
http://jtidy.sourceforge.net/
Another option might be to use good old fashion grep routine, then build primitives like assertContains, assertDoesNotContain, etc. I have a JSON library I write unit tests for and I took that route to just do old fashion grep, and it's worked well in that it catches bugs. I worried about parsing the JSON code directly because I'd be using the code I wrote in my tests to test that very code. Your case might be different so parsing to a DOM model might make more sense.
The big thing, to consider, is you're testing the view. So as things change visually you might have a very hard time keeping your tests up to date. The view gets changed a lot in most programs because your client, PM, person in control of what you're doing, changes their mind frequently.