We have REST services created in RestEasy and running in wildfly server. We are running Postman test cases to test the Rest URLs.
Is there a way to get a code coverage of the services when we execute postman test suite?
We use SonarQube to analyse the code coverage.
I think no, a similar question was asked here:
Generate Sonar code coverage report from Postman tests
The original poster commented further down:
In fact, after a bit of googling, as a work-around we could use remote
Jacoco agent hooked in the java application server. We'll try to run
jacoco maven goals before and after the tests execution in order to
generate jacoco coverage report. See: link I'll update the post if we
have some progress.
Also, newman seems to have aticket about it:
https://github.com/postmanlabs/newman/issues/408
Though this might help
Karate is the answer to your problem, provided you are willing to switch to another testing framework.
Here is the link to the demo-example which has code-coverage working: https://github.com/intuit/karate/tree/master/karate-demo#code-coverage-using-jacoco. Since Karate is a JVM implementation it is straightforward, and I recommend you keep Karate tests in the same Maven module (or equivalent) for the easiest option. Otherwise it is possible, but just harder - and you will need to fiddle with a Maven profile etc or do some instrumentation synchronization gymnastics.
I guess if you already have a lot of tests in Postman, the advice here may not be practical. But I'm posting this answer for the benefit of others who will come across this question in the future.
If you are lucky, you may be able to quickly port your tests to Karate using the experimental converter built into the UI: https://github.com/intuit/karate/wiki/Karate-UI#postman-import
Perhaps you can contribute to making that feature prod-ready.
There is nothing available that provides code coverage for postman tests.
In the end we chose rest assured and started replacing all postman tests.
Related
So this is my situation:
I am fairly new to gitlab-ci. I don't host my own gitlab instance but rather push everything to gitab itself. I am not using and am not familiar with any build tools like Maven. I usually work and run my programms from an IDE rather than the terminal.
This is my problem:
When I push my Java project I want my pipeline to start the Junit tests I wrote. Whereas I've found various simple commands for other languages than Java to run unit tests I didn't come across anything for Junit. I've just found people using Maven, running the test locally and then pushing the test reports to gitlab. Is it even possible to easily run Junit tests on the gitlab server with the pipeline without build tools like Maven? Do I have to run them locally? Do I have to learn to start them with a Java terminal command? I've beeen searching for days now.
The documentation is clear:
To enable the Unit test reports in merge requests, you need to add artifacts:reports:junit in .gitlab-ci.yml, and specify the path(s) of the generated test reports.
The reports must be .xml files, otherwise GitLab returns an Error 500.
You then have various example in Ruby, Gio, Java (Gradle or Maven), and other languages.
But with GitLab 13.12 (May 2021), this gets better:
Failed test screenshots in test report
GitLab makes it easy for teams to set up end-to-end testing with automation tools like Selenium that capture screenshots of failed tests as artifacts.
This is great until you have to sort through a huge archive of screenshots looking for the specific one you need to debug a failing test.
Eventually, you may give up due to frustration and just re-run the test locally to try and figure out the source of the issue instead of wasting more time.
Now, you can link directly to the captured screenshot from the details screen in the Unit Test report on the pipeline page.
This lets you quickly review the captured screenshot alongside the stack trace to identify what failed as fast as possible.
See Documentation and Issue.
Our team is starting a JUnit 5 project with karate tests.
Currently we are using this as a template for our Karate test runner https://github.com/intuit/karate#junit-5-parallel-execution.
It allows us to pass in the "target/surefire-reports" and then before the test finishes we call ReportBuilder.generateReports(). It is basically identical to this code https://github.com/intuit/karate/blob/b50202b3c8a8916a7db0f3d5196d42086ab80a04/karate-junit4/src/test/java/com/intuit/karate/mock/MockServerTest.java.
This works well, but while I was looking at how to set up JUnit 5 I noticed this very slick fluent api https://github.com/intuit/karate#junit-5.
It would be nice to use that syntax, but I can't get the Cucumber report generated like I can with Runner.parallel. I made sure the maven-surefire-plugin was in build.gradle(although I could have messed that up) but it didn't seem to help.
I also tried doing ReportBuilder.generateReports() and the related logic from the parallel execution example in the #AfterAll function, but couldn't get that working either. The errors suggested that the target/surefire-reports folder didn't exist.
Is the cucumber report supported in the second example? If so, is there a trick to getting it setup?
Great question. The reason we de-couple the JUnit execution and the parallel-runner - is JUnit is more useful in development mode, and you expect detailed pass/fail stats in the IDE for example. But this will be an un-necessary overhead in "CI mode".
That said, we have put in some work on making the Parallel runner a fluent interface, so great timing :) You can find an example on line 57 here.
May I request you to try the develop branch and see if you are missing anything ? Building is easy, here are some instructions: https://github.com/intuit/karate/wiki/Developer-Guide
I'm new to BDD and particularly Cucumber.
Can I get a features and its steps from a variable? Also, I want to get a feature and its steps from a test tracker (TestRail) before running tests by the special selection of this tests, and put it in a list, then one by one get a scenario and run it.
Is there such a possibility? Should I use Cucumber or another framework for this?
No, you can't define a Cucumber scenario in code (or at least not in a supported way). But if you were going to write code to get a scenario and its steps from your test tracker and run it, you could equally well write code to put the scenario and its steps in files and run the scenario with the cucumber executable.
I don't know of a Java testing framework in which you can define tests dynamically. You could do that in Ruby with RSpec or (less cleanly) minitest. But I don't know whether a Ruby test framework would be acceptable, or whether it would be OK for the people writing entries in your test tracker to have to read and/or write RSpec examples. (It seems strange to have Cucumber step definitions in a test tracker, too; having features in a test tracker seems more reasonable, aside from the question of how to run them.)
I'm just getting started with ReadyAPI(SoapUI NG) to do automated testing for our Spring based project. So using the ReadyAPI documentation I successfully tested REST URI calls using below steps:
Created a project: File > New Project > Create a Project using REST URI
It creates a project structure like below: REST Project > URI > createAccount < Request 1
In the Request window I added my parameters by using xml/json.
After Running the Request I'm getting my desired response.
I also checked Assertion that also giving desired results.
I did all this using ReadyAPI Documentation
My questions are below:
How to achieve Automated Testing using ReadyAPI(SoapUI NG)?
What features of ReadAPI are generally used? How to use them effectively?
What are the differences between SoapUI NG, LoadUI NG, SoapUI, ReadyAPI? (I thinks these are just different versions, not sure though)
Let me try to answer them to my best of abilities.
How to achieve Automated Testing using ReadyAPI(SoapUI NG)?
Ready API or SoapUI are the automation tools themselves. To manually
test the APIs, there are other tools like swagger. However if
you are planning to automate the flow of your whole project and
planning to use some sort of architecture/framework which will enable
you to do a lot more than just execute the tests at once then you can
try using tools like Jenkins.
What features of ReadAPI are generally used? How to use them
effectively?
One of the most talked about things in Ready API is Data Driven
Testing. If you follow this approach, then you may use test steps like
DataGen, DataSource, DataSink, PropertyTransfer, Groovy Assertions,
Groovy Script Step, JDBC Step. Those are the steps that are used
widely in Ready API Projects.
What are the differences between SoapUI NG, LoadUI NG, SoapUI,
ReadyAPI? (I thinks these are just different versions, not sure
though)
Ready API is a collection of different API testing solutions, like
Performance Testing(LoadUI), Security testing(Secure), Functional
testing (SoapUI). When you install Ready API, you install all the
solutions together. And then you can chose to buy licence for each of
those solutions separately.
Hope that answers your question.
I am at the stage now where I have a fairly good understanding of programming/development, using Java.
Could anyone tell me the best way for me to start using testing packages? I have looked at Hibernate but not surer where to go with it...
I use Eclipse 3.5 on Mac OS X. Is it a case of writing scripts to test methods? What is unit-testing? etc.
Where do I begin?
Many thanks. Alex
What is Unit Testing
Unit testing is writing code (i.e. test code) that passes known inputs into code under test and then validating the code under test returns expected outputs. It's the most granular testing you can perform on an application. To make it easier, usually a unit testing framework is used. For Java, JUnit is the most popular, but TestNG is also notable.
Getting Started
Unit testing frameworks provide tools for test execution, validation and results reporting. For your setup, Eclipse has built in support for JUnit. Eclipse is able to automatically detect tests, compile tests and code under test, execute tests, and report results within the IDE. Furthermore, failures are reported as clickable stack trace information that loads the corresponding file at the given line number.
Mock Objects
That you're also working with Hibernate, suggests you also investigate a mock object framework as well - such as jMock. Mock objects are usually substituted as part of a code under tests's composition and serve two purposes: (1) returning known outputs and (2) recording they've been called and how so that unit tests can introspect that information as part of validation.
The ability to use Mock objects to make testing easier is predicated on dependency injection. That is other entities that compose the object under test. The idea is decoupling dependencies (e.g. Hibernate) to focus on testing algorithms that manipulate that data you're working with.
Database
However, if you've got code that is not easily refactored, or perhaps you want to validate database code, you can also test Hibernate interaction as well. In that case you want a database in a known state. Three approaches come to mind:
Restoring a database backup at the beginning of each test execution.
Use dbunit, which provides its own mechanisms for maintain state.
Transactional locking with rollback. Wrap the entire case is wrapped with a try{} finally{}, where the latter always rolls back the transaction.
James Shore ("a thought leader in the Agile software development community") has a series of screen casts of him demonstrating Test Driven Development, using Eclipse.
http://jamesshore.com/Blog/Lets-Play/
While there are many ways to start testing, there is no "best" way so there's no point in looking for that as a starting point.
Search the web for a good tutorial on junit and do it. That will be the absolute best way to get started IMO. Don't get sidetracked with code coverage or integrating with Hudson or any of the other tasks that are on the periphery to testing. Focus on writing a handful (or 10) if tests first.
Once you understand the basics you can start looking at other tools to see if they meet your needs any better or worse than junit.
First up: Hibernate is not a testing package.
Now that's out of the way, I'd suggest you take a look at JUnit. Read up on unit testing first so you know what it is (the Wikipedia entry is a good place to start), then try the JUnit cookbook. Write some unit tests for a small piece of your code to see how it works, then move on to bigger chunks.
While you are at it, take a look at other development tools like Cobertura (for finding out how good your test coverage is) and static analysis tools like Findbugs and Checkstyle. These all integrate nicely with Ant and probably Eclipse, too.
If you are interested in improving your coding standards and build systems then I highly recommend using Ant, JUnit, Cobertura, Checkstyle and Findbugs together with a continuous integration server (e.g. Hudson or CruiseControl) and a version control system (e.g. git). With a toolkit like that you can't go wrong.
There are other frameworks out there (TestNG, Mockito etc) so take a look at them, too, and decide which you prefer (EDIT: And which work nicely together. Mockito + JUnit is a good combination.)