I'm about to start work on a large project that will involve providing a significant number of web services.
We'll be using the Java platform, so, of course, we'll be making heavy use of JUnit, Hudson, etc. (although I'm not sure that matters at all.)
We're looking for a set of best practices and/or tools for testing the web services. We'll have several goals in mind:
Obviously, we need to prove that the services behave correctly. This means issuing web service calls, receiving the responses (checking them for correctness), and then, probably issuing another web service call to ensure that the system is in the new state that we want it to be in post-original call (and, obviously, checking that response for correctness.)
We'll need to collect response times.
We must be able to execute the test cases automatically (jUnit style, but against a running system, not against individual code units.)
It would be incredibly cool if non-developers (technical business users) could "author" the test cases. They aren't XML experts (and will never be,) so expecting them to create the XML files is not realistic. But they are subject matter experts, so giving them a "form" to fill out that would generate the XML file is realistic. However; there will likely be far too many disparate services for us to custom-build each of these "forms."
Any thoughts, best practices, or lessons learned would be appreciated.
SOAPUI
This is the best open source tool for testing web services. You can create unit tests, it even has a form interface that will read the wsdl and create a form to invoke the services.
Fiddler
You will need to look at the request and responses on the wire. This is crucial since most toolkits don't make it easy to see what exactly is coming back and forth.
XMLSPY
This is a paid program, but it has very very clean interface. IT make it easy to debug the xml files and will let you invoke services. It reformat even malformed XML and is a lifesaver in many situations. If you can afford it GET IT!
We use SOAPUI to do the functional testing initially. However, SOAPUI is not good for regression and load testing.
Next step, we load the XML into JMeter.
It can drive load from multiple machines and we can easily stress our servers to the limits.
Related
I'm quite new to performance testing and am looking to be pointed in the right direction.
I have a Java project which contains two parts, deployed seperately:
A service-broker, published as a webservice; which has service and db wrappers.
A front end, which has a service-broker facade, business logic and a Spring MVC UI.
It is deployed on tomcat, which is running on a fresh install of Windows server 2008.
I need to do basic soak testing on this project, to highlight major memory leaks performance issues.
I've been told SOAP UI is the tool I need to do this.
Now for my questions:
Soap UI (Load UI) is only appropriate as the load-generator for testing the service-broker aspect of the project, right?
What additional tools would be helpful (Something to visualize garbage collection, memory use, heap/stack size etc?)
Can I use Load UI as a load generator for a Spring MVC Front end? If not, what's an appropriate alternative?
Thanks a lot.
Here is my opinions
SoapUI is good enough for microbenchmark test but not good at huge scale of testing. So i recommend to use other load testing tool. LoadUI can be a solution. But i want to recommend nGrinder. I used it, it works very well. Apache Jmeter is common tool. But it is JVM based so Jmeter itself needs a tuning.
To monitoring application during perfomance testing. easiest way is use VisualVM. It can monitor all that u mentioned. But it can show u a data in just Java Virtual machine perspective. I rather to recommend to use APM (Application Performance Monitoring). AppDynamic will be good solution.
About UX testing, big difference is, it needs record and play feature. U can do it by using Load UI. but nGrinder can also cover that by implementing HTTP resquest in coding. (It is a reason why we use such a expensive tools like LoadRunner etc).
I hope this will be useful to u.
Cheers
There are four sets of requirements you need to cover in a tool
Can it exercise my interface (Any HTTP test tool will do this for a web services application)
Can it monitor my infrastructure. Now you are getting into the details of if your underlying OS and Virtual Machine can be monitored in an integrated fashion. Not all tools allow for this and you need to be very explicit as to the level of detail you are interested in.
Will it report appropriate to my requirements and in a way which allows for easier identification of system bottlenecks? This is a mix of objective and subjective items. You have not indicated what level of reporting you need
Does my user community have the skills to use the tool? Get the top three right and miss this one and even a free as in beer tool goes to a negative ROI almost immediately.
It's time to button up the requirements or just hire a firm with a set of tools included to do the job.
I need to write integrations to multiple external web services. Some of them are SOAP (have WSDL), some of them pretty much ad hoc - HTTP(s), authentication either by basic auth or parameters in URL (!), natural-language like XML which does not really map nicely to domain classes..
For now, I've done the spike integrations using Spring Web 3.0 RestTemplate and binding using JAXB2 (Jaxb2Marshaller). Some kind of binding is needed because domain classes need to be cleaner than the XML.
It works, but it kind of feels bad. Obviously this partially just because how the services are built. And one minor issue I have is naming of RestTemplate as services have nothing to do with REST. This I can live with. JAXB2 feels a bit heavy though.
So, I'm looking for some other alternatives. Ideas? I'd like to have a simple solution (so RestTemplate is fine), not too enterprisey..
While some of your services may be schemaless XML, they will still probably have a well-documented API. One of the techniques that the Spring folks seem to be pushing, at least from the web-service server side, is to use XPath/XQuery for retrieving only the information you really need from a request. I know that this may only end up being part of your solution, but I'm not sure that this is a situation where one particular binding framework is going to meet all your needs.
If I understand correctly you have 1 application that has to make calls to various external (web) services by use of different technologies. The first thing that comes to mind is to have some intermediate level. While this could be something as elaborate as en ESB-solution, my guess is that is not what you're looking for.
You could for example achieve this intermediate level by having a class hierarchy with at its top an interface 'Consumer'. Method to be implemented: doConsume() and so on.
If you look into it you'll probably have the opportunity to make use of several design patterns like Strategy or Template. Remember to be pro-active and try to ask a few times 'What if ..' (As in: what if they need me to consume yet another service? etc.)
If JAXB feels too heavy there are other API's to be found:
Axis
JAX-WS
CXF
other
It'll depend on the situation which one would be better. If you run into troubles with any of them I'm sure you'll be able to find help here on SO (and from people who have more hands-on experience with them than me ;-)
we use spring-remoting for a Client-Server-Application. Now we are in doubt how to realize a real load test with serialized objects.
The problem is that many load testing toolkits are based on plain text HTTP communication, so it's very easy to parameterize the http-requests. Because spring-remoting is based on serialized objects we can't easily parameterize the http-communication to the server.
My first idea was to encapsulate the endpoint of spring-remoting and to call the service over client-side service classes. But how can I realize calling the remote-service from multiply clients to gain a real load test scenario?
Are there any solution for wrapping the client-side service in an easy manner?
Has anybody some experiences how to realize it?
Thanks for advice.
There is a discussion on the Spring forums about how to use JMeter to do testing. To use JMeter you'll need to write a custom Sampler. They have examples on the JMeter site.
Alternatively, you could try Grinder. Testing scripts are written using jython but it is arguably easier to create tests for non-standard protocols.
There is a commercial JMeter plugin which allows this, see:
http://www.ubik-ingenierie.com/blog/load-testing-java-serialization-applications-with-jmeter/
To make your tests realistic, you will need to variabilize content in the
serialized objects.
This Java Serialization plugin will allow the following:
Easy recording of traffic with JMeter Proxy Server, a Test Plan using
custom Sampler will be created
Easy variabilization of requests (which will appear as XML) through as
easy syntax as for example ${searchedWord} where searchedWord can come from
a CSV or any user defined variable.
Easy extraction of data from responses using JMeter standard Post
Processors
Easy debugging of Request/Responses through standard JMeter View Results
Tree element
Disclaimer : I am member of the company that distributes this solution
We are removing our XSL-> HTML solution annd replacing it with a Flex UI for our userbase. One of the concerns is that the BlazeDS (amf endpoint, so streaming or polling) won't be able to handle the load as well as the EJB -> XML -> XSLT -> HTML stack. My theory is that it should perform better, but I need a way to statistically prove this.
Are there any suggestions for a product and/or method for stress-testing BlazeDS? Adobe has a stress test framework for LCDS, but it says it is not compatible with BlazeDS. I think that it might be simple to test given that I am not planning on using it as a messeging solution right now, and simple want it as an endpoint for RemoteObject connections.
I would look at using JMeter to run some automated tests against the backend:
http://jmeter.apache.org/
You should be able to write a test against the backend which takes some Java objects and uses the API available in Blaze to encode them in AMF and set the result as the body of the HTTP request.
Rob de Boer has a good article on implementing such an approach here http://blog.jteam.nl/2009/07/14/performance-testing-a-flex-blazeds-application/
To expand upon Cliff's answer, there is now a JMeter plugin for handling AMF messages. It will deserialize the message to XML so that it can be edited. You can use variables to generate unique session ids that are inserted into each AMF sample.
http://github.com/steeltomato/jmeter-amf
Full disclosure: I am the project author.
Several frameworks come to mind:
jmeter
perf4j
TPTP.
, but I need a way to statistically prove this.
Hm, if you want really expressive statistics (e.g. besides average, max, min also standard deviation, or even 95% confidence interval or serial correlation), you'll have to take a close look at the tools, since most of them are weak on statistics.
Hopefully this list explaining the strength and weakness of several performance testing tools can help you.
My Company (UBIK-INGENIERIE) distributes a commercial Apache JMeter plugin to load test AMF based applications and by extension BlazeDS ones.
You can easily record, variabilize and replay Flex/AMF based application with standard knowledge of Apache JMeter.
See:
http://www.ubik-ingenierie.com/-Solutions-
http://www.ubik-ingenierie.com/blog/ubik-load-pack/
My company is considering using web service as mean of ETL process. However I don't think web service fit into this purpose, for several reasons:
1. web service could possibly consume a lot of memory when generating large xml.
2. xml is a bloated format.
3. possibly time-out if the server takes huge amount of time to generate data
4. file size limitation? (for windows, it's 2Gb, if my memory serves me right)
I am not a web service expert, so I need your opinions. :)
Thanks.
There are plenty of technologies in the Web Services tool shed that circumvent all the problems you elaborate. There is stream oriented XML shredding, there are XML compression formats for delivery, protocols that deal with fragmentation and fairness and there are many a storage systems that can hold terabytes upon terabytes of data.
If by web service you imagine some college freshmen homework concoction of an interface that accepts a single glop argument with a 2GB serialized table in it then all your arguments are valid. But if you give your requirements to an experienced team with knowledge of the concepts involved in WS-ReliableMessaging and WS-Transaction then there is no reason not to have an ETL process around Web Services. Note that I do not advocate the SOAP protocols per-se, but I do advocate knowledge and understanding of the concepts involved.
Now that being said, whether an Web Service oriented ETL process makes sense for you or not it depends on a whole set of other reasons. However, your rebuttal of the Web Service technologies does not hold water.
I would not use a web service for an ETL task. There are specialized tools for that task (e.g., Ab Initio, Informatica, etc.) that are better suited.
If you have a large amount of data, I'd say that the price of the extra latency that the network would introduce would be prohibitive.
It really does depend on what you are doing and how you are trying to accomplish it. In general webservices require more care and feeding than you would normally put into an ETL process, but they can be surprisingly effective at the task as well. I did not get enough specifics for your scenario to say whether it would work.
I have worked on Webservices which transmit and recieve 100+ MB documents, some encoded in XML some not, and do it in seconds (on a closed local network). These services required a good deal of tuning and planning, but they did work well for our scenario and they allowed a wide variety of clients to connect and transmit differing amounts of data through a fairly standard interface. This differed from some of the other ETL jobs we had were the job was specific to each client and had to be setup and maintained for each client.
It all depends on what you are doing and what your constraints are.
If you are going to pursue this route sit down and draft out the process from beginning to end, including how you want clients to connect, verify that the data was received and verify that the job is finished. Consider some of the scenarios, the clients and the types of data being transmitted and then work out what would be needed. Contrast that with what is already available in other tools, and how much time you have to get it done.
I'm really wondering why your company is not considering using a real ETL tool like like those mentioned by duffymo in his answer or, Talend or CloverETL if open source is an option.
They are in general good for ETL purpose :)
Building your own solution sounds like reinventing the wheel.
Many of them have web services oriented features (see Export a job as webservice in Talend's wiki or CloverETL Server HTTP Launch Services for example).
I'm not an ETL product expert and I didn't check them all but I'm pretty sure this is something to consider.
Look up MTOM, to start with, which allows arbitrary non-XML data to be streamed in a web service.
Web services are just fine for ETL tasks. Remember that each task is going to get handled in its own thread for free, and you're guaranteed proper cleanup between requests. Using web services inside something like Tomcat wouldn't be nearly as heavy as you think.
If you're concerned over the bloat of XML, consider JSON format.